Zeek is a powerful network analysis framework that is much different from the typical IDS you may know.
This release fixes the following security issues:
A specially-crafted series of FTP packets with a CMD command with a large path followed by a very large number of replies could cause Zeek to spend a long time processing the data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included prevents Zeek from reusing the CMD command if it was already consumed by path-traversal logic.
A specially-crafted with a truncated header can cause Zeek to overflow memory and potentially crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. This overflow requires implementing the raw_packet event handler which isn’t implemented by default, which makes the risk of this issue low. The fix included adds additional length checking during handling of raw_packet events.
A specially-crafted series of SMTP packets can cause Zeek to generate a very large number of events and take a long time to process them. Zeek correctly disables the SMTP analyzer while processing these packets but continues to feed packets to it, generating more events. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included prevents an analyzer from calling another analyzer that has already been disabled for a connection.
A specially-crafted series of POP3 packets containing MIME data can cause Zeek to spend a long time dealing with each individual file ID. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included attempts to reuse an existing file ID for a connection instead of recreating it each pass through the MIME analyzer.
This release fixes the following bugs:
The config parser implements handling of commas at the end of input files in a safer way now, avoiding some crashes on Linux systems during parsing.
The AF_Packet plugin wasn't properly masking the tp_vlan_tci values received from the kernel, and so could return invalid values for the VLAN ID reported to Zeek. The value is now correctly masked.
The AF_Packet plugin now checks whether the interface is up during setup, ensuring that a more useful error message is reported.
This release fixes the following security issues:
A specially-crafted series of FTP packets with a CMD command with a large path followed by a very large number of replies could cause Zeek to spend a long time processing the data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included prevents Zeek from reusing the CMD command if it was already consumed by path-traversal logic.
A specially-crafted with a truncated header can cause Zeek to overflow memory and potentially crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. This overflow requires implementing the raw_packet event handler which isn’t implemented by default, which makes the risk of this issue low. The fix included adds additional length checking during handling of raw_packet events.
A specially-crafted series of SMTP packets can cause Zeek to generate a very large number of events and take a long time to process them. Zeek correctly disables the SMTP analyzer while processing these packets but continues to feed packets to it, generating more events. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included prevents an analyzer from calling another analyzer that has already been disabled for a connection.
A specially-crafted series of POP3 packets containing MIME data can cause Zeek to spend a long time dealing with each individual file ID. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included attempts to reuse an existing file ID for a connection instead of recreating it each pass through the MIME analyzer.
This release fixes the following bugs:
This release fixes the following security issues:
A specially-crafted stream of FTP packets containing a command reply with many intermediate lines can cause Zeek to spend a large amount of time processing data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to ignore intermediate lines of multiline replies, as they do not contain status data and can’t generally be matched to a corresponding command.
A specially-crafted set of packets containing extremely large file offsets cause cause the reassembler code to allocate large amounts of memory. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. This was seen originally with the SMB analyzer, but could be reproduced with the HTTP analyzer as well. The fix included adds some additional length checking to avoid integer overflows in the calculations involving those offsets.
The DNS manager does not correctly expire responses that don’t contain any data, such those containing NXDOMAIN or NODATA status codes. This can lead to Zeek allocating large amounts of memory for these responses and never deallocating them. This is a DoS risk as it is possible to cause Zeek to repeatedly make DNS lookups to hostnames that will always return these responses. The fix included removes the check for whether empty responses should be expired, instead always allowing them to expire once they have hit their TTL.
A specially-crafted stream of RDP packets can cause Zeek to spend large amounts of time processing data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included here is two parts. The first adds some additional length checking to the RDP analyzer for protocol validation. The second is to remove the use of the PIA analyzer when processing internal packets, instead sending all encrypted packet data directly to the SSL analyzer.
A specially-crafted stream of SMTP packets can cause Zeek to spend large amounts of time processing data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds some validation of the order that SMTP commands are received, ensuring that they are received in the proper order.
This release fixes the following bugs:
Data stores used by the known-{hosts,certs,services} policies now default to using local stores instead of Broker stores. This change was made to avoid some performance issues with those stores on high-traffic networks.
Zeekctl now assigns network ports to workers starting at port 27760. This fixes an issue where workers were starting up with ports within Linux's ephemeral port range, and were potentially failing to startup due the ports already being in use. This change may require changes in firewall/routing configurations between hosts in a Zeek cluster. This should not affect clusters running on FreeBSD, as that OS uses a different range for ephemeral ports.
The languages reported for the keyboard_layout field in rdp.log were
updated to match the current standardized set of languages. Unknown layout
values now attempt to fallback to a "parent" layout if one is available.
Replies to HTTP/0.9 requests are now handled in a separate fashion from HTTP/1.x in order to avoid analyzer violations.
The VXLAN analyzer no longer reports analzyer violations if the tunneled packets were not able to be parsed.
The VXLAN and Geneve report analyzer confirmations once their protocols have been fully parsed, but before attempting to forward to the tunneled packets to other analyzers. This should make the confirmation of these protocols more consistent.
New wierds were added to the AYIYA, Geneve, and VXLAN analyzers
(ayiya_empty_packet, geneve_empty_packet, and
vxlan_empty_packet). These are reported when packets containing these
protocols do not have any further data after the tunnel header.
A new script-level option Pcap::non_fd_timeout was added to allow
fine-tuning the amount of time to sleep on each IO loop when using a packet
source that doesn't provide a file descriptor (e.g. Myricom). The option
defaults to 20 microseconds. Tuning this option can greatly reduce the amount
of CPU time used by Zeek on low-traffic networks, but may also increase the
number of dropped packets if set to too high of a value.
This release fixes the following security issues:
A specially-crafted stream of FTP packets containing a command reply with many intermediate lines can cause Zeek to spend a large amount of time processing data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to ignore intermediate lines of multiline replies, as they do not contain status data and can’t generally be matched to a corresponding command.
A specially-crafted set of packets containing extremely large file offsets cause cause the reassembler code to allocate large amounts of memory. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. This was seen originally with the SMB analyzer, but could be reproduced with the HTTP analyzer as well. The fix included adds some additional length checking to avoid integer overflows in the calculations involving those offsets.
The DNS manager does not correctly expire responses that don’t contain any data, such those containing NXDOMAIN or NODATA status codes. This can lead to Zeek allocating large amounts of memory for these responses and never deallocating them. This is a DoS risk as it is possible to cause Zeek to repeatedly make DNS lookups to hostnames that will always return these responses. The fix included removes the check for whether empty responses should be expired, instead always allowing them to expire once they have hit their TTL.
A specially-crafted stream of RDP packets can cause Zeek to spend large amounts of time processing data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included here is two parts. The first adds some additional length checking to the RDP analyzer for protocol validation. The second is to remove the use of the PIA analyzer when processing internal packets, instead sending all encrypted packet data directly to the SSL analyzer.
A specially-crafted stream of SMTP packets can cause Zeek to spend large amounts of time processing data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds some validation of the order that SMTP commands are received, ensuring that they are received in the proper order.
This release fixes the following bugs:
Data stores used by the known-{hosts,certs,services} policies now default to using local stores instead of Broker stores. This change was made to avoid some performance issues with those stores on high-traffic networks.
The VXLAN and Geneve report analyzer confirmations once their protocols have been fully parsed, but before attempting to forward to the tunneled packets to other analyzers. This should make the confirmation of these protocols more consistent.
New wierds were added to the AYIYA, Geneve, and VXLAN analyzers
(ayiya_empty_packet, geneve_empty_packet, and
vxlan_empty_packet). These are reported when packets containing these
protocols do not have any further data after the tunnel header.
A new script-level option Pcap::non_fd_timeout was added to allow
fine-tuning the amount of time to sleep on each IO loop when using a packet
source that doesn't provide a file descriptor (e.g. Myricom). The option
defaults to 20 microseconds. Tuning this option can greatly reduce the amount
of CPU time used by Zeek on low-traffic networks, but may also increase the
number of dropped packets if set to too high of a value.
Avoid attempting to retrieve packets during every loop for a packet source, instead switching to a predictive approach that keeps track of whether or not that packet source has previously seen traffic. This improves performance somewhat, especially on traffic links and on certain packet sources that may block when traffic isn't available.
This release fixes the following security issues:
Receiving DNS responses from async DNS requests (via the lookup_addr, etc BIF methods) with the TTL set to zero could cause the DNS manager to eventually stop being able to make new requests. This would lead to a memory leak that causes Zeek to crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to more carefully handle these types of responses to allow DNS resolution to continue.
Specially-crafted FTP packets with excessively long usernames, passwords, or other fields could cause log writes to use large amounts of disk space. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds new script-level options for limiting the possible input length for these fields, reporting weirds when those limits are exceeded.
The find_all and find_all_ordered BIF methods could take extremely large amounts of time to process incoming data depending on the size of the input. This has been encountered with packet input to multiple analyzers, but most notably the SMTP analyzer. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds a new argument to those BIF methods allowing callers to a set a maximum size for an input string to be processed. If the limit is exceeded, a weird is reported.
This release fixes the following bugs:
Various issues with signed/unsigned character discrepancies on arm64 builds are fixed. These caused problems in the HTTP and NetBIOS analyzers, as well as Base64 decoding.
A performance degredation in debug builds involving hashing large keys for Dictionaries was fixed.
This release fixes the following security issues:
Receiving DNS responses from async DNS requests (via the lookup_addr, etc BIF methods) with the TTL set to zero could cause the DNS manager to eventually stop being able to make new requests. This would lead to a memory leak that causes Zeek to crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to more carefully handle these types of responses to allow DNS resolution to continue.
Specially-crafted FTP packets with excessively long usernames, passwords, or other fields could cause log writes to use large amounts of disk space. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds new script-level options for limiting the possible input length for these fields, reporting weirds when those limits are exceeded.
The find_all and find_all_ordered BIF methods could take extremely large amounts of time to process incoming data depending on the size of the input. This has been encountered with packet input to multiple analyzers, but most notably the SMTP analyzer. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds a new argument to those BIF methods allowing callers to a set a maximum size for an input string to be processed. If the limit is exceeded, a weird is reported.
Zeekctl now assigns network ports to workers starting at port 27760. This fixes an issue where workers were starting up with ports within Linux's ephemeral port range, and were potentially failing to startup due the ports already being in use. This change may require changes in firewall/routing configurations between hosts in a Zeek cluster. This should not affect clusters running on FreeBSD, as that OS uses a different range for ephemeral ports.
Zeekctl support for the AF_PACKET plugin specific options (af_packet_*) has been integrated into zeekctl directly. Upgrading to Zeek 5.2 with a builtin AF_PACKET packet source (default on Linux) requires an upgrade of zeekctl to the version bundled with Zeek to continue using these options.
The blank identifier _ cannot be used in expressions and options anymore.
Outside of obfuscation exercises, this should have little real-world impact.
A new mysql_eof event has been introduced and the mysql_ok event
is not raised in its place or artificially anymore. The base scripts were
adapted accordingly. Users of mysql_ok() likely need to switch to
mysql_eof().
Zeek will now exit at startup if an external plugin (e.g. from a package) is discovered to have the same name as a built-in plugin. See below for the change regarding the AF_PACKET plugin now being built-in for an example of this potentially being triggered.
DNS query type strings were updated to match the current standardized list of strings. This changes the string reported for a small subset of query types:
30: Changed from "EID" to "NXT" 31: Changed from "NIMLOC" to "EID" 32: Changed from "NB" to "NIMLOC"
The --with-caf option for the configure script was removed. Broker now
requires specific versions of CAF per Zeek release, and passing an
externally-built version of CAF often lead to build failures.
Experimental support added for building and running Zeek on Microsoft Windows environments. This is considered experimental due to the fact that our standard testing setup (btest) doesn't run properly on Windows. This will be fixed in the future. In the meantime we have done some basic testing against builds done with Visual Studio 2019. Information on how to build on Windows is available in the Zeek documentation. Note also that Spicy is currently unsupported and will be fixed in the future.
The feature as checked into the repository is not considered production-ready. There are many bugs to squash and features to improve, and we will be steadily fixing things over the next few months.
The Zeek team wants to give a huge thank you to the team at Microsoft for all of their effort in completing this port.
Zeek container images are now being published to zeek/zeek and zeek/zeek-dev rather than zeekurity/zeek and zeekurity/zeek-dev on Docker Hub (and continue to be published to public.ecr.aws) Further, container images for amd64 and arm64 platforms are now available. Main driver for the latter was to allow usage of the official container images on Apple's M1 systems.
Zeekctl support for using af_packet as lb_method has been added.
New analyzer_confirmation_info and analyzer_violation_info events with
accompanying record types AnalyzerConfirmationInfo and
AnalyzerViolationInfo have been added. These supersede
analyzer_confirmation and analyzer_violation, which have been
deprecated.
Added helpers to determine protocol, packet or file analyzer based on
AllAnalyzers::Tag values named is_protocol_analyzer(),
is_packet_analyzer() and is_file_analyzer().
File analyzers can now raise analyzer violations to the script-layer via the
new AnalyzerViolation() method.
Packet and file analyzers can now be disabled and enabled at runtime using the
Analyzer::enable_analyzer() and Analyzer::disable_analyzer()
wrappers. While initially for protocol analyzers only, these have been
extended to work for packet and file analyzers. This now allows to leverage
Analyzer::disabled_analyzers for these kinds of analyzers.
The blank identifier _ can now be used to ignore loop variables of
different types without type clash errors. This allows to do the following
within the same scope:
local vec = vector("a", "b", "c");
for ( _, v in vec )
print v;
for ( i, _ in vec )
print v;
Iterating over only the values of a table can be done by ignoring the full index with a single blank identifier. Due to the internal structure of Zeek tables, this can result in a performance improvement.
local tab = table(["a", 1, T] = "a1T", ["b", 2, F] = "b2f");
for ( _, v in tab )
print v;
It's also possible ignore individual indices of different types with the blank
identifier _ as follows:
for ( [_, i, _], v in tab )
print i, v;
As noted under breaking changes, the blank identifier _ cannot be
referenced in expression anymore.
It is now possible to put trailing commas within table, vector, set and record construction. For example, the following code is now valid, which can make for more uniform style and smaller diffs.
local vec = vector(
"1",
"2",
);
local tab: table[string] of count = [
["a"] = 1,
["b"] = 2,
];
Function calls and record constructors can have a trailing comma after the last argument.
Analyzer::schedule_analyzer(
chan$orig_h,
chan$resp_h,
chan$resp_p,
Analyzer::ANALYZER_FTP_DATA,
5mins,
);
Re-introduce event groups. Allow the &group attribute on event and hook
handlers for annotating them with one or more event groups. These groups can
be disabled and enable during runtime. Disabling an event group implies
disabling all event and hook handlers that are part of it.
The main difference to a previous implementation in (very) old Zeek versions is its granularity: It is now possible to toggle individual event handlers (event handler bodies). The original implementation worked at the level of events, disabling or enabling all event handlers for a given event at once. Additionally, support for hooks was added as these are structurally similar to events.
The BIFs disable_event_group() and enable_event_group() are
re-instantiated and allow controlling event groups based on the group
attribute.
Additionally, event and hook handlers are implicitly placed into event module
groups based on the module they are implemented in. All events implemented in
a given module can be toggled with disable_module_events() and
enable_module_events().
Extend the Logging::Stream record with an event_groups field and
toggle these during Log::disable_stream and Log::enable_stream
invocations. This allows for explicit/manual opt-in performance optimizations
by turning off event handlers at runtime that are only needed for log
generation.
On Linux, the AF_PACKET packet source plugin
(https://github.com/zeek/zeek-af_packet-plugin) is included as a builtin
plugin by default. To select this packet source, prefix the interface name
with af_packet.
zeek -i af_packet::eth0
Usage of break and next statements is now validated. It was previously
possible to place these outside of for, while or switch statements
without any error indication.
Add two BIFs get_identifier_declaring_script() and
get_record_field_declaring_script() to query the declaring scripts for
identifiers and record fields from Zeek scripts.
Extend the SSH analyzer to produce new events (ssh2_ecc_init,
ssh2_gh_gex_init, ssh2_gss_init, ssh2_rsa_secret``) to detect when SSH
client and server roles are reversed.
Analyzers found in the new Analyzer::requested_analyzers set will be
enabled at zeek_init() time. The set can be populated via
:zeek:see:redef. This change only has an effect in settings where
Analyzer::disable_all is changed to T. By default, all analyzers
continue to be enabled.
A new analyzer.log was added to log all analyzer violations and optionally
analyzer confirmations. This log can be useful during development of new
analyzers as well as for collecting operational data in production
environments.
Expose configurability of for SQLite's synchronous and journal_mode PRAGMAs for SQLite backed Broker data stores. Setting these to synchronous=normal and journal_mode=wal can significantly improve throughput at the cost of some durability in the presence of power loss or OS crash. In the context of Zeek, this is likely more than acceptable.
Additionally, add integrity_check and failure_mode options to support detecting and deleting corrupted SQLite database at store initialization.
A new join_string_set BIF was added, replacing the existing script-level
version from utils/strings.zeek.
A new &ordered attribute for tables and sets was added. This attribute
causes iteration over a table/set to return elements in the order of their
insertion.
A new -D argument was added to the configure script to allow passing
parameters directly to the underlying CMake call.
Added parsing for the challenge and response fields to the NTLM analyzer.
A new FTP::max_command_length value was added to script-land, defaulting
to 100. This value is used by the FTP analyzer to limit the size of commands
accepted by the analyzer. A FTP_max_command_length_exceeded weird is
raised for any violations of that length.
The MySQL analyzer has been extended to detect when client and server negotiate to use a SSL encrypted session. This allows analysis of the subsequent SSL handshake. The service field for encrypted MySQL connections in the conn.log will have entries for both, mysql and ssl.
Violations for packet analyzers that have sessions attached with them will be raised once only. Further, analyzer confirmations are not raised after a violation.
The parameter given to enum_names() can now be a string naming the enum
type, rather than the type itself.
The type_name of enum types produced by record_fields() now includes
the actual type name rather than just "enum".
Passing non-string sep and def arguments to cat_sep() isn't a
fatal error anymore. More descriptive error messages are produced, too.
The number of analyzer violation events that can be raised by protocol
analyzer instances is now capped by the const
max_analyzer_violation_events.
The number of analyzer violation events that can be raised by protocol and
file analyzer instances is now capped by the const
max_analyzer_violation_events. Its default is 1000 and the main purpose
is to prevent analyzers from scheduling too many analyzer_violation_info
events before the DPD max_violations script-level logic has a chance to
run and disable the problematic analyzer.
The TCP analyzer now continues processing payload for some connections missing initial packets where it would previously have stopped. This fixes a few cases where we already had the logic to continue in place, but we still ended up considering them partial.
Count underflows via --c or subtract from statements (c = c - 1) are
now consistently warned about. Previously, underflows through --c were
treated as runtime errors, while "subtract from" underflows were silently
accepted. The following (surprising behavior) now causes a warning, too:
$ zeek -e 'print 1 - 2'
expression warning in <command line>, line 1: count underflow (1 - 2)
18446744073709551615
The MQTT scripts registering the analyzer and DPD signatures have been moved from the policy folder to base and are loaded by default.
Notices created for files transferred over multiple connections will now be associated with one of the connections rather than none.
The MySQL analyzer has been switched to parse in little endian. This avoids analyzer violations due to out of bound errors for length encoded strings.
Non-fatal errors when setting up BPF filtering will no longer cause Zeek to exit, but instead will log the error in reporter.log and continue processing.
The languages reported for the keyboard_layout field in rdp.log were
updated to match the current standardized set of languages. Unknown layout
values now attempt to fallback to a "parent" layout if one is available.
In the cluster management framework, the controller now supports Broker's
WebSocket data transport for communication with clients. It listens on TCP
port 2149 for this purpose. zeek-client now likewise uses the WebSocket
transport, removing its runtime dependency on the Broker library and enabling
standalone installation. The client still bundles with Zeek by default but is
now also available on PyPI and installable via pip install zeek-client.
The documentation provides additional details.
The global disabling_analyzer() hook has been deprecated and replaced
with Analyzer::disabling_analyzer() that has the same semantics.
The analyzer_confirmation and analyzer_violation events have been
deprecated in favor of the more generic analyzer_confirmation_info and
analyzer_violation_info events.
The const values for toggling individual tunnel packet analyzers have been
deprecated in favor of using Analyzer::disable_analyzer() directly. This
affects:
Tunnel::enable_ip
Tunnel::enable_ayiya
Tunnel::enable_teredo
Tunnel::enable_gtpv1
Tunnel::enable_gre
Setting these explicitly to F can be achieved by leveraging
Analyzers::disabled_analyzers, for example:
redef Analyzer::disabled_analyzers += { PacketAnalyzer::ANALYZER_GRE };
The zeek::merge_type_list() function has been deprecated. Please consider
the partially compatible and saner zeek::maximal_type() instead. See
GH-2604 for context.
The pre-authentication data field (pa_data) available in certain Kerberos events now exposes the (encrypted) PA-ENC-TIMESTAMP field (padata-type=2).
The SupressWeirds() method in the ContentLine analyzer was deprecated in
favor of the correctly-spelled SuppressWeirds() method.
The bro symlink has finally been removed.
Zeekctl now assigns network ports to workers starting at port 27760. This fixes an issue where workers were starting up with ports within Linux's ephemeral port range, and were potentially failing to startup due the ports already being in use. This change may require changes in firewall/routing configurations between hosts in a Zeek cluster. This should not affect clusters running on FreeBSD, as that OS uses a different range for ephemeral ports.
The blank identifier _ cannot be used in expressions and options anymore.
Outside of obfuscation exercises, this should have little real-world impact.
A new mysql_eof event has been introduced and the mysql_ok event
is not raised in its place or artificially anymore. The base scripts were
adapted accordingly. Users of mysql_ok() likely need to switch to
mysql_eof().
Zeek will now exit at startup if an external plugin (e.g. from a package) is discovered to have the same name as a built-in plugin. See below for the change regarding the AF_PACKET plugin now being built-in for an example of this potentially being triggered.
DNS query type strings were updated to match the current standardized list of strings. This changes the string reported for a small subset of query types:
30: Changed from "EID" to "NXT" 31: Changed from "NIMLOC" to "EID" 32: Changed from "NB" to "NIMLOC"
The --with-caf option for the configure script was removed. Broker now
requires specific versions of CAF per Zeek release, and passing an
externally-built version of CAF often lead to build failures.
Experimental support added for building and running Zeek on Microsoft Windows environments. This is considered experimental due to the fact that our standard testing setup (btest) doesn't run properly on Windows. This will be fixed in the future. In the meantime we have done some basic testing against builds done with Visual Studio 2019. Information on how to build on Windows is available in the Zeek documentation. Note also that Spicy is currently unsupported and will be fixed in the future.
The feature as checked into the repository is not considered production-ready. There are many bugs to squash and features to improve, and we will be steadily fixing things over the next few months.
The Zeek team wants to give a huge thank you to the team at Microsoft for all of their effort in completing this port.
Zeek container images are now being published to zeek/zeek and zeek/zeek-dev rather than zeekurity/zeek and zeekurity/zeek-dev on Docker Hub (and continue to be published to public.ecr.aws) Further, container images for amd64 and arm64 platforms are now available. Main driver for the latter was to allow usage of the official container images on Apple's M1 systems.
New analyzer_confirmation_info and analyzer_violation_info events with
accompanying record types AnalyzerConfirmationInfo and
AnalyzerViolationInfo have been added. These supersede
analyzer_confirmation and analyzer_violation, which have been
deprecated.
Added helpers to determine protocol, packet or file analyzer based on
AllAnalyzers::Tag values named is_protocol_analyzer(),
is_packet_analyzer() and is_file_analyzer().
File analyzers can now raise analyzer violations to the script-layer via the
new AnalyzerViolation() method.
Packet and file analyzers can now be disabled and enabled at runtime using the
Analyzer::enable_analyzer() and Analyzer::disable_analyzer()
wrappers. While initially for protocol analyzers only, these have been
extended to work for packet and file analyzers. This now allows to leverage
Analyzer::disabled_analyzers for these kinds of analyzers.
The blank identifier _ can now be used to ignore loop variables of
different types without type clash errors. This allows to do the following
within the same scope:
local vec = vector("a", "b", "c"); for ( _, v in vec ) print v;
for ( i, _ in vec ) print v;
Iterating over only the values of a table can be done by ignoring the full index with a single blank identifier. Due to the internal structure of Zeek tables, this can result in a performance improvement.
local tab = table(["a", 1, T] = "a1T", ["b", 2, F] = "b2f"); for ( _, v in tab ) print v;
It's also possible ignore individual indices of different types with the blank
identifier _ as follows:
for ( [_, i, _], v in tab ) print i, v;
As noted under breaking changes, the blank identifier _ cannot be
referenced in expression anymore.
It is now possible to put trailing commas within table, vector, set and record construction. For example, the following code is now valid, which can make for more uniform style and smaller diffs.
local vec = vector( "1", "2", );
local tab: table[string] of count = [ ["a"] = 1, ["b"] = 2, ];
Function calls and record constructors can have a trailing comma after the last argument.
Analyzer::schedule_analyzer( chan$orig_h, chan$resp_h, chan$resp_p, Analyzer::ANALYZER_FTP_DATA, 5mins, );
Re-introduce event groups. Allow the &group attribute on event and hook
handlers for annotating them with one or more event groups. These groups can
be disabled and enable during runtime. Disabling an event group implies
disabling all event and hook handlers that are part of it.
The main difference to a previous implementation in (very) old Zeek versions is its granularity: It is now possible to toggle individual event handlers (event handler bodies). The original implementation worked at the level of events, disabling or enabling all event handlers for a given event at once. Additionally, support for hooks was added as these are structurally similar to events.
The BIFs disable_event_group() and enable_event_group() are
re-instantiated and allow controlling event groups based on the group
attribute.
Additionally, event and hook handlers are implicitly placed into event module
groups based on the module they are implemented in. All events implemented in
a given module can be toggled with disable_module_events() and
enable_module_events().
Extend the Logging::Stream record with an event_groups field and
toggle these during Log::disable_stream and Log::enable_stream
invocations. This allows for explicit/manual opt-in performance optimizations
by turning off event handlers at runtime that are only needed for log
generation.
On Linux, the AF_PACKET packet source plugin
(https://github.com/zeek/zeek-af_packet-plugin) is included as a builtin
plugin by default. To select this packet source, prefix the interface name
with af_packet.
zeek -i af_packet::eth0
Usage of break and next statements is now validated. It was previously
possible to place these outside of for, while or switch statements
without any error indication.
Add two BIFs get_identifier_declaring_script() and
get_record_field_declaring_script() to query the declaring scripts for
identifiers and record fields from Zeek scripts.
Extend the SSH analyzer to produce new events (ssh2_ecc_init,
ssh2_gh_gex_init, ssh2_gss_init, ssh2_rsa_secret``) to detect when SSH
client and server roles are reversed.
Analyzers found in the new Analyzer::requested_analyzers set will be
enabled at zeek_init() time. The set can be populated via
:zeek:see:redef. This change only has an effect in settings where
Analyzer::disable_all is changed to T. By default, all analyzers
continue to be enabled.
A new analyzer.log was added to log all analyzer violations and optionally
analyzer confirmations. This log can be useful during development of new
analyzers as well as for collecting operational data in production
environments.
Expose configurability of for SQLite's synchronous and journal_mode PRAGMAs for SQLite backed Broker data stores. Setting these to synchronous=normal and journal_mode=wal can significantly improve throughput at the cost of some durability in the presence of power loss or OS crash. In the context of Zeek, this is likely more than acceptable.
Additionally, add integrity_check and failure_mode options to support detecting and deleting corrupted SQLite database at store initialization.
A new join_string_set BIF was added, replacing the existing script-level
version from utils/strings.zeek.
A new &ordered attribute for tables and sets was added. This attribute
causes iteration over a table/set to return elements in the order of their
insertion.
A new -D argument was added to the configure script to allow passing
parameters directly to the underlying CMake call.
Added parsing for the challenge and response fields to the NTLM analyzer.
A new FTP::max_command_length value was added to script-land, defaulting
to 100. This value is used by the FTP analyzer to limit the size of commands
accepted by the analyzer. A FTP_max_command_length_exceeded weird is
raised for any violations of that length.
The MySQL analyzer has been extended to detect when client and server negotiate to use a SSL encrypted session. This allows analysis of the subsequent SSL handshake. The service field for encrypted MySQL connections in the conn.log will have entries for both, mysql and ssl.
Violations for packet analyzers that have sessions attached with them will be raised once only. Further, analyzer confirmations are not raised after a violation.
The parameter given to enum_names() can now be a string naming the enum
type, rather than the type itself.
The type_name of enum types produced by record_fields() now includes
the actual type name rather than just "enum".
Passing non-string sep and def arguments to cat_sep() isn't a
fatal error anymore. More descriptive error messages are produced, too.
The number of analyzer violation events that can be raised by protocol
analyzer instances is now capped by the const
max_analyzer_violation_events.
The number of analyzer violation events that can be raised by protocol and
file analyzer instances is now capped by the const
max_analyzer_violation_events. Its default is 1000 and the main purpose
is to prevent analyzers from scheduling too many analyzer_violation_info
events before the DPD max_violations script-level logic has a chance to
run and disable the problematic analyzer.
The TCP analyzer now continues processing payload for some connections missing initial packets where it would previously have stopped. This fixes a few cases where we already had the logic to continue in place, but we still ended up considering them partial.
Count underflows via --c or subtract from statements (c = c - 1) are
now consistently warned about. Previously, underflows through --c were
treated as runtime errors, while "subtract from" underflows were silently
accepted. The following (surprising behavior) now causes a warning, too:
$ zeek -e 'print 1 - 2'
expression warning in
The MQTT scripts registering the analyzer and DPD signatures have been moved from the policy folder to base and are loaded by default.
Notices created for files transferred over multiple connections will now be associated with one of the connections rather than none.
The MySQL analyzer has been switched to parse in little endian. This avoids analyzer violations due to out of bound errors for length encoded strings.
Non-fatal errors when setting up BPF filtering will no longer cause Zeek to exit, but instead will log the error in reporter.log and continue processing.
The languages reported for the keyboard_layout field in rdp.log were
updated to match the current standardized set of languages. Unknown layout
values now attempt to fallback to a "parent" layout if one is available.
The global disabling_analyzer() hook has been deprecated and replaced
with Analyzer::disabling_analyzer() that has the same semantics.
The analyzer_confirmation and analyzer_violation events have been
deprecated in favor of the more generic analyzer_confirmation_info and
analyzer_violation_info events.
The const values for toggling individual tunnel packet analyzers have been
deprecated in favor of using Analyzer::disable_analyzer() directly. This
affects:
Tunnel::enable_ip Tunnel::enable_ayiya Tunnel::enable_teredo Tunnel::enable_gtpv1 Tunnel::enable_gre
Setting these explicitly to F can be achieved by leveraging
Analyzers::disabled_analyzers, for example:
redef Analyzer::disabled_analyzers += { PacketAnalyzer::ANALYZER_GRE };
The zeek::merge_type_list() function has been deprecated. Please consider
the partially compatible and saner zeek::maximal_type() instead. See
GH-2604 for context.
The pre-authentication data field (pa_data) available in certain Kerberos events now exposes the (encrypted) PA-ENC-TIMESTAMP field (padata-type=2).
The SupressWeirds() method in the ContentLine analyzer was deprecated in
favor of the correctly-spelled SuppressWeirds() method.
The bro symlink has finally been removed.
This release fixes the following security issues:
A missing field in the SMB FSControl script-land record could cause a heap buffer overflow when receiving packets containing those header types. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to add the missing field to the record type.
Receiving a series of packets that start with HTTP/1.0 and then switch to HTTP/0.9 could cause Zeek to spend a large amount of time processing the packets. Due to the possiblity of receiving these packets from remote hosts, this is a DoS risk. The fix included is to ensure that the HTTP analyzer switches versions correctly.
Receiving large numbers of FTP commands sequentially from the network with bad data in them could cause Zeek to spend a large amount of time processing the packets, and generate a large amount of events. Due to the possiblity of receiving these packets from remote hosts, this is a DoS risk. The fix included adds better validation for FTP command strings and request codes.
This release fixes the following bugs:
Zeek could throw a scripting error when receiving SMB1 packets containing
connect_andx_response messages prior to receiving an associated request. A
new weird smb_tree_connect_andx_response_without_tree will now be raised
in these cases and the packet will be ignored.
A performance regression from 4.2 to 5.0 when reading pcap files related to Broker's internal clock was fixed.
Notices created for files transferred over multiple connections will now be associated with one of the connections rather than none.
A new file_over_new_connection event was added to the Intel framework, for
use when receiving files over established connections (for example, HTTP).
This ensures that the filename in the file info record is populated correctly.
The error message returned when trying use invalid enums in scripts now correctly includes the script location.
This release fixes the following security issues:
A missing field in the SMB FSControl script-land record could cause a heap buffer overflow when receiving packets containing those header types. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to add the missing field to the record type.
Receiving a series of packets that start with HTTP/1.0 and then switch to HTTP/0.9 could cause Zeek to spend a large amount of time processing the packets. Due to the possiblity of receiving these packets from remote hosts, this is a DoS risk. The fix included is to ensure that the HTTP analyzer switches versions correctly.
Receiving large numbers of FTP commands sequentially from the network with bad data in them could cause Zeek to spend a large amount of time processing the packets, and generate a large amount of events. Due to the possiblity of receiving these packets from remote hosts, this is a DoS risk. The fix included adds better validation for FTP command strings and request codes.
This release fixes the following bugs:
Zeek could throw a scripting error when receiving SMB1 packets containing
connect_andx_response messages prior to receiving an associated request. A
new weird smb_tree_connect_andx_response_without_tree will now be raised
in these cases and the packet will be ignored.
A performance regression from 4.2 to 5.0 when reading pcap files related to Broker's internal clock was fixed.
Notices created for files transferred over multiple connections will now be associated with one of the connections rather than none.
A new file_over_new_connection event was added to the Intel framework, for
use when receiving files over established connections (for example, HTTP).
This ensures that the filename in the file info record is populated correctly.
The error message returned when trying use invalid enums in scripts now correctly includes the script location.
Analyzer confirmation events are no longer raised per-packet for tunnels. This greatly reduces the amount of events/log messages on networks containing a lot of tunnel traffic.
The Teredo packet analyzer is now less greedy when matching packets. This should greatly reduce the amount of log messages from that analyzer, as well as reduce the number of analyzer violations reported by it.