Zeek is a powerful network analysis framework that is much different from the typical IDS you may know.
The methods Dispatcher::Lookup() and Analyzer::Lookup() in the packet_analysis
namespace were changed to return a reference to a std::shared_ptr instead of a copy
for performance reasons.
Zeek's OPENSSL_INCLUDE_DIR is not automatically added to an external plugin's
include path anymore. A plugin using OpenSSL functionality directly can use the
following explicit entry to re-use Zeek's OPENSSL_INCLUDE_DIR:
zeek_add_plugin( Namespace Name INCLUDE_DIRS "${OPENSSL_INCLUDE_DIR}" SOURCES ... )
The "segment_profiling" functionality and load_sample event have been removed
without deprecation. This functionality was unmaintained and not known to be used.
Certain ldap.log and ldap_search.log fields have been renamed from
plural to singular and their types changed to scalars. This maps better onto
the expected request-response protocol used between client and server. Additionally,
it removes the burden of working with non-scalar columns from downstream systems.
Specifically, for ldap.log:
arguments: vector of string is now argument: string
diagnostic_messages: vector of string is now diagnostic_message: string
objects: vector of string is now object: string
opcodes: set[string] is now opcode: string
results: set[string] is now result: string
For ldap_search.log, the following fields were changed:
base_objects: vector of string is now base_object: string
derefs: set[string] is now deref_aliases: string
diagnostic_messages: vector of string is now diagnostic_message: string
results: set[string] is now result: string
scopes: set[string] is now scope: string
In the unlikely scenario that a request-response pair with the same message
identifier is observed, containing different values for certain fields, new
weirds are raised and will appear in weird.log, including the old and new
values as well as the LDAP message identifier. The value within the LDAP logs
will be the most recently observed one.
BIF methods now return a ValPtr directly instead of a BifReturnVal object
which was just a thin wrapper around ValPtr. This may cause compilation errors
in C++ code that was calling BIF methods directly.
The table type was extended to allow parallel regular expression matching when a table's index is a pattern. Indexing such tables yields a vector containing all values of matching patterns for keys of type string.
As an example, the following snippet outputs [a, a or b], [a or b].
global tbl: table[pattern] of string; tbl[/a/] = "a"; tbl[/a|b/] = "a or b"; tbl[/c/] = "c"; print tbl["a"], tbl["b"];
Depending on the patterns and input used for matching, memory growth may
be observed over time as the underlying DFA is constructed lazily. Users are
advised to test with realistic and adversarial input data with focus on
memory growth. The DFA's state can be reset by removal/addition of a single
pattern. For observability, a new bif table_pattern_matcher_stats()
can be used to gather MatcherStats.
Support for delaying log writes.
The logging framework offers two new functions Log::delay() and Log::delay_finish()
to delay a Log::write() operation. This new functionality allows delaying of
a specific log record within the logging pipeline for a variable but bounded
amount of time. This can be used, for example, to query and wait for additional
information to attach to the pending record, or even change its final verdict.
Conceptually, delaying a log record happens after the execution of the global
Log::log_stream_policy hook for a given Log::write() and before the
execution of filter policy hooks. Any mutation of the log record within the
delay period will be visible to filter policy hooks. Calling Log::delay()
is currently only allowed within the context of the Log::log_stream_policy hook
for the active Log::write()` operation (or during the execution of post delay callbacks). While this may appear restrictive, it makes it explicit which Log::write()``
operation is subject to the delay.
Interactions, semantics and conflicts of this feature when writing the same log record multiple times to the same or different log streams need to be taken into consideration by script writers.
Given this is the first iteration of this feature, feedback around usability and use-cases that aren't covered are more than welcome.
A WebSocket analyzer has been added together with a new websocket.log.
The WebSocket analyzer is instantiated when a WebSocket handshake over HTTP is recognized. By default, the payload of WebSocket messages is fed into Zeek's dynamic protocol detection framework, possibly discovering and analyzing tunneled protocols.
The format of the log and the event semantics should be considered preliminary until the arrival of the next long-term-stable release (7.0).
To disable the analyzer in case of fatal errors or unexpected resource usage,
use the Analyzer::disabled_analyzers pattern:
redef Analyzer::disabled_analyzers += { Analyzer::ANALYZER_WEBSOCKET, };
The SMTP analyzer was extended to recognize and properly handle the BDAT command from RFC 3030. This improves visibility into the SMTP protocol when mail agents and servers support and use this extension.
The event keyword in signatures was extended to support choosing a custom event
to raise instead of signature_match(). This can be more efficient in certain
scenarios compared to funneling every match through a single event.
The new syntax is to put the name of the event before the string used for the
msg argument. As an extension, it is possible to only provide an event name,
skipping msg. In this case, the framework expects the event's parameters to
consist of only state and data as follows:
signature only-event { payload /.*root/ event found_root }
event found_root(state: signature_state, data: string) { }
Using the msg parameter with a custom event looks as follows. The custom
event's parameters need to align with those for ``signature_match()` event:
signature event-with-msg { payload /.*root/ event found_root_with_msg "the-message" }
event found_root_with_msg(state: signature_state, msg: string, data: string) { }
Note, the message argument can currently still be specified as a Zeek identifier
referring to a script-level string value. If used, this is disambiguated behind
the scenes for the first variant. Specifying msg as a Zeek identifier has
been deprecated with the new event support and will be removed in the future.
Note that matches for signatures with custom events will not be recorded in
signatures.log. This log is based on the generation of signature_match()
events.
The QUIC analyzer has been extended to support analyzing QUIC Version 2
INITIAL packets (RFC 9369). Additionally, prior draft and some of
Facebook's mvfst versions are supported. Unknown QUIC versions will now be
reported in quic.log as an entry with a U history field.
Conditional directives (@if, @ifdef, @ifndef, @else and
@endif) can now be placed within a record's definition to conditionally
define or extend a record type's fields.
type r: record { c: count; @if ( cond ) d: double; @else d: count; @endif };
Note that generally you should prefer record extension in conditionally loaded scripts rather than using conditional directives in the original record definition.
The 'X' code can now appear in a connection's history. It is meant to indicate
situations where Zeek stopped analyzing traffic due to exceeding certain limits or
when encountering unknown/unsupported protocols. Its first use is to indicate
Tunnel::max_depth being exceeded.
A new Intel::seen_policy hook has been introduced to allow intercepting
and changing ``Intel::seen` behavior:
hook Intel::seen_policy(s: Intel::Seen, found: bool)
A new NetControl::rule_added_policy hook has been introduced to allow modification
of NetControl rules after they have been added.
The IP geolocation / ASN lookup features in the script layer provide better
configurability. The file names of MaxMind databases are now configurable via
the new mmdb_city_db, mmdb_country_db, and mmdb_asn_db constants,
and the previously hardwired fallback search path when not using an
mmdb_dir value is now adjustable via the mmdb_dir_fallbacks
vector. Databases opened explicitly via the mmdb_open_location_db and
mmdb_open_asn_db functions now behave more predictably when updated or
removed. For details, see:
https://docs.zeek.org/en/master/customizations.html#address-geolocation-and-as-lookups
The zeek-config script now provides a set of --have-XXX checks for
features optionally compiled in. Each check reports "yes"/"no" to stdout and
exits with 0/1, respectively.
The split_string family of functions now respect the beginning-of-line ^ and
end-of-line $ anchors. Previously, an anchored pattern would be matched anywhere
in the input string.
The sub() and ``gsub()` functions now respect the beginning-of-line ^ and
end-of-line $ anchors. Previously, an anchored pattern would be matched anywhere
in the input string.
Ed25519 and Ed448 DNSKEY and RRSIG entries do not cause weirds anymore.
The OpenSSL references in digest.h and OpaqueVal.h headers have been
hidden to avoid unneeded dependencies on OpenSSL headers. Plugins using the
detail API from digest.h to compute hashes likely need to accommodate for
this change.
The Tunnel::max_depth default was changed from 2 to 4 allowing for more than
two encapsulation layers. Two layers are already easily reached in AWS GLB
environments.
Nested MIME message analysis is now capped at a maximum depth of 100 to prevent
unbounded MIME message nesting. This limit is configurable with MIME::max_depth.
A new weird named exceeded_mime_max_depth is reported when reached.
The netcontrol_catch_release.log now contains a plugin column that shows which
plugin took an action. The logs also contain information when errors or existing
rules are encountered.
The Cluster::PoolSpec record no longer provides default values for its
topic and node_type fields, since defaults don't fit their intended
use and looked confusing in generated documentation.
The virtual functions DoSerializeand DoUnserialize of the OpaqueVal
(and BloomFilter) class will be removed with Zeek 7.1. Unfortunately, code
implementing the deprecated methods does not produce compiler warnings.
Plugin authors implementing an OpaqueVal subclass need to convert to
DoSerializeData and DoUnserializeData:
std::optional<BrokerData> OpaqueVal::DoSerializeData() const
bool OpaqueVal::DoUnserializeData(BrokerDataView data)
When overriding DoSerializeData(), return std::nullopt (or a
default-constructed optional) for values that cannot be serialized.
Otherwise, the canonical way to create a BrokerData for serialization is
by using a BrokerListBuilder. For example, creating a BrokerData that
contains true and the count 42 could be implemented as follows:
BrokerListBuilder builder; builder.Add(true); builder.AddCount(42u); return std::move(builder).Build();
Please refer to the respective class documentation for a full list of member
functions on BrokerListBuilder and BrokerDataView.
For plugins that are using the macro DECLARE_OPAQUE_VALUE to generate the
function prototypes for the serialization functions: please use
DECLARE_OPAQUE_VALUE_DATA instead to generate prototypes for the new API.
Plugin authors that need to support multiple Zeek versions can use the
ZEEK_VERSION_NUMBER macro to conditionally implement the new and old
methods. Provide the new versions with Zeek 6.2 (60200) or later, otherwise
keep the old signature. The default implementations for the new functions
as used by Zeek will call the old signatures and convert the results.
The Cluster::Node$interface field has been deprecated. It's essentially
unneeded, unused and not a reliable way to gather the actual interface used
by a worker. In Zeekctl deployments the field will be populated until its
removal. The packet_source() bif should be used on worker processes to
gather information about the interface.
The policy/misc/load-balancing script has been deprecated in favor of
AF_PACKET PF_RING, Netmap or other NIC specific load balancing approaches.
Time machine related enums, options and fields have been marked for removal.
The check_for_unused_event_handlers options the related UsedHandlers(),
UnusedHandlers() and their related SetUsed() and Used() methods
have been marked for removal. The feature of finding unused event handlers is
provided by default via the UsageAnalyzer component.
Using a Zeek identifier for the msg argument within a signatures's event
keyword has been deprecated.
We would like to thank Anthony Verez (netantho), Bijoy Das (mute019), Jan Grashöfer (J-Gras), Matti Bispham (mbispham), Phil Rzewski (philrz), and xb-anssi for their contributions to this release.
The methods Dispatcher::Lookup() and Analyzer::Lookup() in the packet_analysis
namespace were changed to return a reference to a std::shared_ptr instead of a copy
for performance reasons.
Zeek's OPENSSL_INCLUDE_DIR is not automatically added to an external plugin's
include path anymore. A plugin using OpenSSL functionality directly can use the
following explicit entry to re-use Zeek's OPENSSL_INCLUDE_DIR:
zeek_add_plugin( Namespace Name INCLUDE_DIRS "${OPENSSL_INCLUDE_DIR}" SOURCES ... )
The "segment_profiling" functionality and load_sample event have been removed
without deprecation. This functionality was unmaintained and not known to be used.
Certain ldap.log and ldap_search.log fields have been renamed from
plural to singular and their types changed to scalars. This maps better onto
the expected request-response protocol used between client and server. Additionally,
it removes the burden of working with non-scalar columns from downstream systems.
Specifically, for ldap.log:
arguments: vector of string is now argument: string
diagnostic_messages: vector of string is now diagnostic_message: string
objects: vector of string is now object: string
opcodes: set[string] is now opcode: string
results: set[string] is now result: string
For ldap_search.log, the following fields were changed:
base_objects: vector of string is now base_object: string
derefs: set[string] is now deref_aliases: string
diagnostic_messages: vector of string is now diagnostic_message: string
results: set[string] is now result: string
scopes: set[string] is now scope: string
In the unlikely scenario that a request-response pair with the same message
identifier is observed, containing different values for certain fields, new
weirds are raised and will appear in weird.log, including the old and new
values as well as the LDAP message identifier. The value within the LDAP logs
will be the most recently observed one.
BIF methods now return a ValPtr directly instead of a BifReturnVal object
which was just a thin wrapper around ValPtr. This may cause compilation errors
in C++ code that was calling BIF methods directly.
The table type was extended to allow parallel regular expression matching when a table's index is a pattern. Indexing such tables yields a vector containing all values of matching patterns for keys of type string.
As an example, the following snippet outputs [a, a or b], [a or b].
global tbl: table[pattern] of string; tbl[/a/] = "a"; tbl[/a|b/] = "a or b"; tbl[/c/] = "c"; print tbl["a"], tbl["b"];
Depending on the patterns and input used for matching, memory growth may
be observed over time as the underlying DFA is constructed lazily. Users are
advised to test with realistic and adversarial input data with focus on
memory growth. The DFA's state can be reset by removal/addition of a single
pattern. For observability, a new bif table_pattern_matcher_stats()
can be used to gather MatcherStats.
Support for delaying log writes.
The logging framework offers two new functions Log::delay() and Log::delay_finish()
to delay a Log::write() operation. This new functionality allows delaying of
a specific log record within the logging pipeline for a variable but bounded
amount of time. This can be used, for example, to query and wait for additional
information to attach to the pending record, or even change its final verdict.
Conceptually, delaying a log record happens after the execution of the global
Log::log_stream_policy hook for a given Log::write() and before the
execution of filter policy hooks. Any mutation of the log record within the
delay period will be visible to filter policy hooks. Calling Log::delay()
is currently only allowed within the context of the Log::log_stream_policy hook
for the active Log::write()` operation (or during the execution of post delay callbacks). While this may appear restrictive, it makes it explicit which Log::write()``
operation is subject to the delay.
Interactions, semantics and conflicts of this feature when writing the same log record multiple times to the same or different log streams need to be taken into consideration by script writers.
Given this is the first iteration of this feature, feedback around usability and use-cases that aren't covered are more than welcome.
A WebSocket analyzer has been added together with a new websocket.log.
The WebSocket analyzer is instantiated when a WebSocket handshake over HTTP is recognized. By default, the payload of WebSocket messages is fed into Zeek's dynamic protocol detection framework, possibly discovering and analyzing tunneled protocols.
The format of the log and the event semantics should be considered preliminary until the arrival of the next long-term-stable release (7.0).
To disable the analyzer in case of fatal errors or unexpected resource usage,
use the Analyzer::disabled_analyzers pattern:
redef Analyzer::disabled_analyzers += { Analyzer::ANALYZER_WEBSOCKET, };
The SMTP analyzer was extended to recognize and properly handle the BDAT command from RFC 3030. This improves visibility into the SMTP protocol when mail agents and servers support and use this extension.
The event keyword in signatures was extended to support choosing a custom event
to raise instead of signature_match(). This can be more efficient in certain
scenarios compared to funneling every match through a single event.
The new syntax is to put the name of the event before the string used for the
msg argument. As an extension, it is possible to only provide an event name,
skipping msg. In this case, the framework expects the event's parameters to
consist of only state and data as follows:
signature only-event { payload /.*root/ event found_root }
event found_root(state: signature_state, data: string) { }
Using the msg parameter with a custom event looks as follows. The custom
event's parameters need to align with those for ``signature_match()` event:
signature event-with-msg { payload /.*root/ event found_root_with_msg "the-message" }
event found_root_with_msg(state: signature_state, msg: string, data: string) { }
Note, the message argument can currently still be specified as a Zeek identifier
referring to a script-level string value. If used, this is disambiguated behind
the scenes for the first variant. Specifying msg as a Zeek identifier has
been deprecated with the new event support and will be removed in the future.
Note that matches for signatures with custom events will not be recorded in
signatures.log. This log is based on the generation of signature_match()
events.
The QUIC analyzer has been extended to support analyzing QUIC Version 2
INITIAL packets (RFC 9369). Additionally, prior draft and some of
Facebook's mvfst versions are supported. Unknown QUIC versions will now be
reported in quic.log as an entry with a U history field.
Conditional directives (@if, @ifdef, @ifndef, @else and
@endif) can now be placed within a record's definition to conditionally
define or extend a record type's fields.
ype r: record {
c: count;
if ( cond )
d: double;
else
d: count;
endif
;
Note that generally you should prefer record extension in conditionally loaded scripts rather than using conditional directives in the original record definition.
The 'X' code can now appear in a connection's history. It is meant to indicate
situations where Zeek stopped analyzing traffic due to exceeding certain limits or
when encountering unknown/unsupported protocols. Its first use is to indicate
Tunnel::max_depth being exceeded.
A new Intel::seen_policy hook has been introduced to allow intercepting
and changing ``Intel::seen` behavior:
hook Intel::seen_policy(s: Intel::Seen, found: bool)
A new NetControl::rule_added_policy hook has been introduced to allow modification
of NetControl rules after they have been added.
The IP geolocation / ASN lookup features in the script layer provide better
configurability. The file names of MaxMind databases are now configurable via
the new mmdb_city_db, mmdb_country_db, and mmdb_asn_db constants,
and the previously hardwired fallback search path when not using an
mmdb_dir value is now adjustable via the mmdb_dir_fallbacks
vector. Databases opened explicitly via the mmdb_open_location_db and
mmdb_open_asn_db functions now behave more predictably when updated or
removed. For details, see:
https://docs.zeek.org/en/master/customizations.html#address-geolocation-and-as-lookups
The zeek-config script now provides a set of --have-XXX checks for
features optionally compiled in. Each check reports "yes"/"no" to stdout and
exits with 0/1, respectively.
The split_string family of functions now respect the beginning-of-line ^ and
end-of-line $ anchors. Previously, an anchored pattern would be matched anywhere
in the input string.
The sub() and ``gsub()` functions now respect the beginning-of-line ^ and
end-of-line $ anchors. Previously, an anchored pattern would be matched anywhere
in the input string.
Ed25519 and Ed448 DNSKEY and RRSIG entries do not cause weirds anymore.
The OpenSSL references in digest.h and OpaqueVal.h headers have been
hidden to avoid unneeded dependencies on OpenSSL headers. Plugins using the
detail API from digest.h to compute hashes likely need to accommodate for
this change.
The Tunnel::max_depth default was changed from 2 to 4 allowing for more than
two encapsulation layers. Two layers are already easily reached in AWS GLB
environments.
Nested MIME message analysis is now capped at a maximum depth of 100 to prevent
unbounded MIME message nesting. This limit is configurable with MIME::max_depth.
A new weird named exceeded_mime_max_depth is reported when reached.
The netcontrol_catch_release.log now contains a plugin column that shows which
plugin took an action. The logs also contain information when errors or existing
rules are encountered.
The Cluster::PoolSpec record no longer provides default values for its
topic and node_type fields, since defaults don't fit their intended
use and looked confusing in generated documentation.
The virtual functions DoSerializeand DoUnserialize of the OpaqueVal
(and BloomFilter) class will be removed with Zeek 7.1. Unfortunately, code
implementing the deprecated methods does not produce compiler warnings.
Plugin authors implementing an OpaqueVal subclass need to convert to
DoSerializeData and DoUnserializeData:
std::optional<BrokerData> OpaqueVal::DoSerializeData() const
bool OpaqueVal::DoUnserializeData(BrokerDataView data)
When overriding DoSerializeData(), return std::nullopt (or a
default-constructed optional) for values that cannot be serialized.
Otherwise, the canonical way to create a BrokerData for serialization is
by using a BrokerListBuilder. For example, creating a BrokerData that
contains true and the count 42 could be implemented as follows:
BrokerListBuilder builder; builder.Add(true); builder.AddCount(42u); return std::move(builder).Build();
Please refer to the respective class documentation for a full list of member
functions on BrokerListBuilder and BrokerDataView.
For plugins that are using the macro DECLARE_OPAQUE_VALUE to generate the
function prototypes for the serialization functions: please use
DECLARE_OPAQUE_VALUE_DATA instead to generate prototypes for the new API.
Plugin authors that need to support multiple Zeek versions can use the
ZEEK_VERSION_NUMBER macro to conditionally implement the new and old
methods. Provide the new versions with Zeek 6.2 (60200) or later, otherwise
keep the old signature. The default implementations for the new functions
as used by Zeek will call the old signatures and convert the results.
The Cluster::Node$interface field has been deprecated. It's essentially
unneeded, unused and not a reliable way to gather the actual interface used
by a worker. In Zeekctl deployments the field will be populated until its
removal. The packet_source() bif should be used on worker processes to
gather information about the interface.
The policy/misc/load-balancing script has been deprecated in favor of
AF_PACKET PF_RING, Netmap or other NIC specific load balancing approaches.
Time machine related enums, options and fields have been marked for removal.
The check_for_unused_event_handlers options the related UsedHandlers(),
UnusedHandlers() and their related SetUsed() and Used() methods
have been marked for removal. The feature of finding unused event handlers is
provided by default via the UsageAnalyzer component.
Using a Zeek identifier for the msg argument within a signatures's event
keyword has been deprecated.
This release fixes the following security issues:
This release fixes the following bugs:
CMake correctly passes along third-party package information when building plugins. This ensures that, for example, the same paths to OpenSSL used in a Zeek build are provided to a plugin build.
Fix a problem with the HTTP analyzer where a signature regex ending in '$' used to match against 'http-request-body' or 'http-reply-bdoy' will never succeed. Thank you to GitHub user xb-anssi for this fix.
The DNS analyzer now understands the Ed25519 and Ed448 signature algorithms.
The SMB::State$recent_files field was not correctly expiring entries, leading to unbounded state growth. This is fixed to correctly follow the &read_expire condition on the field. Thank you to Slack user ya-sato for reporting this.
The &create_expire attribute is now kept valid after clearing a table. After switching the known scripts away from broker stores, the &create_expire value of the local tables/sets of the known scripts wasn't in effect due to Cluster::node_up() and Cluster::node_down() re-assigning these without keeping the &create_expire attribute intact. This broke the "log hosts every 24h" behavior.
Zeek builds using the --binary-package argument and including Spicy will now include all necessary Spicy symbols.
This release fixes the following security issues:
This release fixes the following bugs:
CMake correctly passes along third-party package information when building plugins. This ensures that, for example, the same paths to OpenSSL used in a Zeek build are provided to a plugin build.
Fix a problem with the HTTP analyzer where a signature regex ending in '$' used to match against 'http-request-body' or 'http-reply-bdoy' will never succeed. Thank you to GitHub user xb-anssi for this fix.
The DNS analyzer now understands the Ed25519 and Ed448 signature algorithms.
The SMB::State$recent_files field was not correctly expiring entries, leading to unbounded state growth. This is fixed to correctly follow the &read_expire condition on the field. Thank you to Slack user ya-sato for reporting this.
The &create_expire attribute is now kept valid after clearing a table. After switching the known scripts away from broker stores, the &create_expire value of the local tables/sets of the known scripts wasn't in effect due to Cluster::node_up() and Cluster::node_down() re-assigning these without keeping the &create_expire attribute intact. This broke the "log hosts every 24h" behavior.
Zeek builds using the --binary-package argument and including Spicy will now include all necessary Spicy symbols.
This release fixes the following security issues:
A specially-crafted SSL packet could cause Zeek to leak memory and potentially crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds additional memory cleanup to the x509 file analyzer.
A specially-crafted series of FTP packets could cause Zeek to log entries for requests that have already been completed, using resources unnecessarily and potentially causing Zeek to lose other traffic. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included changes the way that we track the pending FTP commands, avoiding possibly reusing the same value for subsequent commands.
A specially-crafted series of SSL packets could cause Zeek to output a very large number of unnecessary alerts for the same record. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds a new option SSL::max_alerts_per_record that caps the number of alerts that can be generated for an SSL record. For TLS 1.3 this is capped at 1 as defined in RFC 8446. For non-TLS 1.3 it is a configurable value. A SSL_excessive_alerts_in_record weird will be raised if the cap is exceeded.
A specially-crafted series of SSL packets could cause Zeek to generate very long ssl_history fields in the ssl.log, potentially using a large amount of memory due to unbounded state growth. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds a new option SSL::max_ssl_history_length that caps this to 100 characters by default. A SSL_max_ssl_history_length_reached weird will be raised if the cap is exceeded.
A specially-crafted IEEE802.11 packet could cause Zeek to overflow memory and potentially crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds additional bounds checking to the IEEE802.11 packet analyzer.
This release fixes the following bugs:
Fixed Spicy type names from causing collisions with existing Zeek types.
On some systems with low values for the maximum number of file descriptors, it was possible to run into crashes when doing DNS lookups if all of the file descriptors were used. This is now avoided with better checking for the number of available file descriptors before trying a lookup. Thank you to Zeek Slack user h-mikami for reporting this issue.
Tables backed by a Broker backend now correctly support deletion if they have complex index types. Zeek previously reported an error when trying to delete elements from these tables.
A significant performance issue with Zeek's supervisor code was fixed, revolving around the re-initialization of the Event Manager object used to track events. Thank you to Jan Grashoefer for reporting this issue.
The MaxMind DB code now cleans up after itself, resolving a memory leak with the loaded database files.
The ZeekJS submodule was updated to version 0.9.6, bringing fixes for zeek.invoke and zeek.event crashes, garbage collection, and an issue where Zeek may stop executing events from ZeekJS.
assert is now a reserved keyword for the new assert statement.
The __bro_plugin__ file that gets generated as part of plugin builds was
renamed to __zeek_plugin__. This will affect the ability for older
versions of zkg to use the zkg unload and zkg load commands. This
should only cause breakage for people using a version of ``zkg` that doesn't
come bundled with Zeek (which we generally don't recommend doing).
Zeek does not traverse into dot directories to find plugins or hlto files anymore. Any dot directories found below the directories specified in ZEEK_PLUGIN_PATH or ZEEK_SPICY_MODULE_PATH are now skipped. Dot directories explicitly listed in ZEEK_PLUGIN_PATH or ZEEK_SPICY_MODULE_PATH are not skipped.
External plugins will fail to configure if their minimum required CMake version is below 3.15. This was a warning with Zeek 6.0, but has caused user confusion due to unhelpful error messages around the IN_LIST operator policy.
The FindBISON, FindOpenSSL, FindPackageHandleStandardArgs, FindPackageMessage, and SelectLibraryConfigurations cmake files were removed from our cmake repository in favor of the versions that come with CMake. This should not cause any breakage, but it is possible in the case that someone was using these in a plugin.
Zeek now includes the LDAP protocol analyzer from the zeek/spicy-ldap project
(https://github.com/zeek/spicy-ldap). This analyzer is enabled by default. The
analyzer's events and its ldap.log and ldap_search.log should be
considered preliminary and experimental until the arrival of Zeek's next
long-term-stable release (7.0).
If you observe unusually high CPU consumption or other issues due to this
analyzer being enabled by default, the easiest way to disable it is via the
Analyzer::disabled_analyzers const as follows:
redef Analyzer::disabled_analyzers += { Analyzer::ANALYZER_LDAP_UDP, Analyzer::ANALYZER_LDAP_TCP, };
Please do report issues to us including diagnostic information in case this is necessary in your environment. We're also open to general feedback about the structure of the new logs.
Zeek now includes the QUIC protocol analyzer from the zeek/spicy-quic project (https://github.com/zeek/spicy-quic). This project is a fork of Fox-IT's initial implementation (https://github.com/fox-ds/spicy-quic).
As for the LDAP analyzer, the analyzer's events and the new quic.log
should be considered preliminary and experimental until the arrival of Zeek's
next long-term-stable release (7.0). As above, any feedback and contributions
to this analyzer and the new log are welcome.
The analyzer's functionality is limited to decryption of the INITIAL packets
of QUIC version 1. If decryption of these packets is successful, the handshake
data is forwarded to Zeek's SSL analyzer. An ssl.log entry will appear in
ssl.log for QUIC connections. The entry in the conn.log will contain
quic and ssl in the service field.
To disable the analyzer in case of issues, use the following snippet:
redef Analyzer::disabled_analyzers += { Analyzer::ANALYZER_QUIC, };
Added a new assert statement for assertion based testing and asserting
runtime state.
assert <expr: bool>[, <message: string>];
This statement comes with two hooks. First, assertion_failure() that is
invoked for every failing assert statement. Second, assertion_result()
which is invoked for every assert statement and its outcome. The latter allows
to construct a summary of failing and passing assert statements. Both hooks
receive the location and call stack for the assert statement via a
Backtrace vector.
A failing assert will abort execution of the current event handler similar to
scripting errors. By default, a reporter error message is logged. Using the
break statement within assertion_failure() or assertion_result()
allows to suppress the default message.
Add a new &default_insert attribute for tables. This behaves as
&default with the addition that the default value is inserted into the
table upon a failed lookup. Particularly for tables with nested container
values, the &default behavior of not inserting the value can be of little
use.
The from_json() function now takes an optional key_func argument to
normalize JSON object key names. This can be useful if the keys in a JSON
object are not valid Zeek identifiers or reserved keywords.
Module names are now included in global_ids(). Their key in the returned
table is prefixed with "module " and their value will have the type_name
field set to "module".
Identifiers in the global scope can now be referenced and defined from within
modules by prefixing their names with ::. Previously, these required an
explicit GLOBAL:: prefix to be used. Using GLOBAL:: has been
deprecated.
The as keyword now supports casting between set and vector values
with the same element type. Converting set values with multiple index
values is not supported. We plan to extend the use of the as keyword to
support more type conversions in the future.
Added new packet analyzer to handle PCAP files DLT_PPP link type.
Fixed appending of any to vector of any.
The ModBus analyzer's function support was expanded, with new handling of the
Encapsulation Interface Transport (function 28) And Diagnostics (function 8)
functions. This adds new modbus_encap_interface_transport_{request,response}
and modbus_diagnostics_{request,response} events.
The ModBus file record read and write events now provide the full data from the request and response messages as part of the event data.
The full PDU length was added to the ModBusHeader record type passed with
all of the ModBus events.
A connection's value is now updated in-place when its directionality is
flipped due to Zeek's heuristics (for example, SYN/SYN-ACK reversal or
protocol specific approaches). Previously, a connection's value was discarded
when flipped, including any values set in a new_connection() handler. A
new connection_flipped() event is added to allow updating custom state in
script-land.
Loading policy/frameworks/notice/community-id.zeek now also automatically
community ID logging. In the past, loading the script had no effect unless
policy/protocols/conn/community-id-logging.zeek was loaded before. This
was fairly unusual and hard to debug behavior.
Connections to broadcast addresses are not flipped based on
likely_server_ports anymore. Previously, broadcast packets originating
from a likely server port resulted in 255.255.255.255 being the originator in
conn.log.
When too many HTTP requests are pending, Zeek will now log them at once and
reset request/response correlation instead of running into unbounded state
growth. This behavior is configurable via a new option
HTTP::max_pending_requests. The default is 100.
Fix deferred initialization of nested records containing non-const &default attributes.
Parameter lists for functions, events and hooks now use commas instead of semicolons in error messages or when printing such functions.
The IO buffer size used for PCAP file reading is now always 128kb. This new
default can be changed via Pcap::bufsize_offline_bytes.
The input framework now provides better information in error messages when encountering missing non-optional field while loading data.
The SSL analyzer will now parse a configurable maximum of 10 SSL Alerts per
SSL message. For TLS 1.3, the maximum is implicitly 1 as defined by RFC 8446.
If there are more alerts, a new weird "SSL_excessive_alerts_in_record" is raised.
For non-TLS 1.3, the maximum can be redefined via SSL::max_alerts_per_record.
The ssl_history field in the ssl.log is now capped at a configurable
limit of 100 characters prevent unbounded growth. The limit can be changed
via the option SSL::max_ssl_history_length. When reached, a new weird
named "SSL_max_ssl_history_length_reached" is raised.
Accessing globals with GLOBAL::name has been deprecated and will be
removed with Zeek 7.1. Use ::name instead.
The original trigger::Trigger constructor has been deprecated and will be
removed with Zeek 7.1. Use the new alternative constructor (per
src/Trigger.h) instead, including replacing any use of new ... with
make_intrusive<...>. The new constructor differs only in the placement of
the timeout parameter, and in that - unlike the original - it always
returns a valid pointer, which must be Unref()'d after construction, either
explicitly (if using new) or implicitly (if using
make_intrusive<...>).
This release fixes the following security issues:
File extraction limits were not correctly enforced for files containing large amounts of missing bytes. Crafting files with large amounts of missing bytes in them could cause Zeek to spend a long time processing data, allocate a lot of main memory, and write a lot of data to disk. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included makes Zeek correctly enforce file size limits. We also added a new option (default_limit_includes_missing) which allows to customize the behavior when encountering large amounts of missed bytes in file. This issue was discovered by Luca Cigarini.
Sessions are sometimes not cleaned up completely within Zeek during shutdown, potentially causing a crash when using the -B dpd flag for debug logging. This is low priority because it only happens at shutdown and only when using that flag. The fix included is to reorder shutdown to cleanup all the sessions prior to tearing down the analyzers.
A specially-crafted HTTP packet can cause Zeek’s filename extraction code to take a long time to process the data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adjusts the regular expression used in the extract_filename_from_content_disposition() script function to more-correctly match the data.
A specially-crafted series of FTP packets made up of a CWD request followed by a large amount of ERPT requests may cause Zeek to spend a long time logging the commands. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to prevent logging of pending commands for FTP packets.
A specially-crafted VLAN packet can cause Zeek to overflow memory and potentially crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to add some additional length checking to the VLAN analyzer.
This release fixes the following bugs:
Fixed a base64 decoding issue with the authorization field of HTTP request headers that was sometimes causing Zeek to output error messages. Thank you to GitHub user @progmboy for reporting and providing a fix for this issue.
Ensure that Zeek builds use the internal version of Spicy instead of external installations, unless specifically configured for that mode.
Support was added for switch fields when exporting Spicy types to Zeek.
A number of fixes were added to protect against potential unbounded state
growth with the SMB and DCE-RPC analyzers. SMB close requests will properly
tear down an related DCE-RPC analyzers. A new SMB::max_dce_rpc_analyzers
script variable was added that allows finer control of how many DCE-RPC
analyzers are allowed to be created per SMB connection. Thanks to Zeek Slack
user Takomi Sugawara for reporting this issue.
Fixed a regression in the UDP and TCP analyzers that was causing more data than necessary to be forwarded to the next analyzer in the chain. Thanks to Zeek Slack user Hiep Long Tan for reporting this issue.
A connection's value is now updated in-place when its directionality is
flipped due to Zeek's heuristics (for example, SYN/SYN-ACK reversal or
protocol specific approaches). Previously, a connection's value was discarded
when flipped, including any values set in a new_connection() handler. A
new connection_flipped() event is added to allow updating custom state in
script-land.
Fixed undefined symbols being reported from Spicy when building some of the binary packages for Zeek.
Loading policy/frameworks/notice/community-id.zeek now also automatically
community ID logging. In the past, loading the script had no effect unless
policy/protocols/conn/community-id-logging.zeek was loaded before. This
was fairly unusual and hard to debug behavior.
Spicy no longer registers an extra port for every port registered in a plugin's .evt file.
Timeouts in DNS resolution no longer cause uncontrolled memory growth.
Fix check to skip DNS hostname lookups for notices that are not delivered via
email in policy/frameworks/notice/extend-email/hostnames. Due to that
policy script being loaded in the Zeek's default local.zeek, this
previously caused unneeded DNS lookups for every generated notice instead of
just those delivered via email.
This release fixes the following security issues:
File extraction limits were not correctly enforced for files containing large amounts of missing bytes. Crafting files with large amounts of missing bytes in them could cause Zeek to spend a long time processing data, allocate a lot of main memory, and write a lot of data to disk. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included makes Zeek correctly enforce file size limits. We also added a new option (default_limit_includes_missing) which allows to customize the behavior when encountering large amounts of missed bytes in file. This issue was discovered by Luca Cigarini.
Sessions are sometimes not cleaned up completely within Zeek during shutdown, potentially causing a crash when using the -B dpd flag for debug logging. This is low priority because it only happens at shutdown and only when using that flag. The fix included is to reorder shutdown to cleanup all the sessions prior to tearing down the analyzers.
A specially-crafted HTTP packet can cause Zeek’s filename extraction code to take a long time to process the data. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adjusts the regular expression used in the extract_filename_from_content_disposition() script function to more-correctly match the data.
A specially-crafted series of FTP packets made up of a CWD request followed by a large amount of ERPT requests may cause Zeek to spend a long time logging the commands. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included is to prevent logging of pending commands for FTP packets.
A specially-crafted series of SMB packets containing DCE-RPC traffic may cause Zeek to use a large amount of memory and potentially crash. Due to the possibility of receiving these packets from remote hosts, this is a DoS risk. The fix included adds additional cleanup for DCE-RPC analyzers when closing SMB connections and limits on the number of DCE-RPC analyzers that can be created as part of an SMB connection.
This release fixes the following bugs:
Fixed the DPD signature for the SOCKS analyzer to actually match the format for those packets.
Timeouts in DNS resolution no longer cause uncontrolled memory growth.
Fix check to skip DNS hostname lookups for notices that are not delivered via
email in policy/frameworks/notice/extend-email/hostnames. Due to that
policy script being loaded in the Zeek's default local.zeek, this
previously caused unneeded DNS lookups for every generated notice instead of
just those delivered via email.
Zeek now treats private address space (i.e., non-routable IP address ranges)
as local by default, matching the intuition of many users that e.g. a
192.168/16 IP address should show up as local in the logs. To do this, Zeek
automatically adds Site::private_address_space to Site::local_nets at
startup. Subsequent runtime updates to Site::private_address_space
propagate to Site::local_nets, while updates to the latter don't affect
the former.
You're free to define Site::local_nets as before and do not need to update
your configurations. If you added standard private address space to
Site::local_nets in the past, you no longer need to do so. This also
applies to zeekctl's networks.cfg file.
The new global Boolean Site::private_address_space_is_local, true by
default, controls the behavior. A redef to false brings back Zeek's prior
behavior of considering private address space an unrelated concept, which will
come in handy for example when working with tests that compare results against
log baselines that have not yet been updated.
Telemetry centralization and Prometheus exposition is not enabled by default anymore. Previously, the manager node would open port 9911/tcp by default and import all metrics from other nodes. For large clusters, the current implementation introduces significant processing overhead on the manager even if the Prometheus functionality is not used. While inconvenient, disable this functionality (assumed to be used by few as of now) by default to preserve resources.
The script to enable centralization and the Prometheus endpoint is now
located in the policy/ folder. Re-enable the old functionality with:
@load frameworks/telemetry/prometheus
You may experiment with increasing Broker::metrics_export_interval
(default 1s) to reduce the extra overhead and communication at the expense
of stale metrics.
Custom source tarballs require a repo-info.json file.
Note, should you be using official Zeek release tarballs only, or build Zeek solely from git checkouts, this does not affect you.
However, if you're building your own Zeek source tarballs, it is now required
that a repo-info.json file exists at the top-level. The dist target was
extended to add this file and official Zeek release source tarballs will
contain it going forward.
The following command can be used to produce repo-info.json:
python3 ./ci/collect-repo-info.py --only-git > ../path/to/tarballdir/repo-info.json
This is required to support the new -V / --build-info option that
provides information about git submodules and included plugins used during
the build. The ci/collect-repo-info.py tool runs at ./configure time
and either collects the required information from a git clone (when git is
installed), or otherwise uses the content of a file named repo-info.json.
If you see opportunities to extend repo-info.json with further information,
please get in touch.
Plugin authors should raise the minimum required CMake version to 3.15 to ensure compatibility with new CMake scaffolding included in this release. Older versions will trigger a warning at configuration time and, depending on the functionality included in the plugin, may trigger subsequent errors during configuration or build.
Zeek container images are not pushed to the zeekurity organization anymore.
Please switch to using the zeek/zeek image on DockerHub, or the images
published to public.ecr.aws/zeek/zeek.
The IRC_Data analyzer declaration has been moved to protocols/irc/IRC.h.
The error message returned when using bro_init, bro_done, and
bro_script_loaded events is now removed. removed. Usage of these events
has returned that error during script parsing for a few years, and time has
come to finally remove it.
Zeek now features experimental JavaScript support:
/* hello.js */ zeek.on('zeek_init', () => { console.log('Hello, Zeek!'); });
$ zeek ./hello.js Hello, Zeek!
When building Zeek on a system that features a recent (16.13+) version of the
libnode package with development headers, Zeek automatically includes the
externally-maintained ZeekJS plugin (https://github.com/corelight/zeekjs) as a
builtin plugin. This allows Zeek to load and execute JavaScript code located
in .js or .cjs files. When no such files are passed to Zeek, the
JavaScript engine and Node.js environment aren't initialized and there is no
runtime impact.
The Linux distributions Fedora 37 & 38, Ubuntu 22.10, and the upcoming Debian
12 release provide suitable packages. On other platforms, Node.js can be built
from source with the --shared option.
To disable this functionality, pass --disable-javascript to configure.
Zeek now comes with Spicy support built in, meaning it can now leverage any analyzers written in Spicy out of the box. While the interface layer connecting Zeek and Spicy used to be implemented through an external Zeek plugin, that code has now moved into the Zeek code base itself. We also added infrastructure to Zeek that enables its built-in standard analyzers to use Spicy instead of Binpac. As initial (simple) examples, Zeek's Syslog and Finger analyzers are now implemented in Spicy. While their legacy versions remain available as fallbacks for now in case Spicy gets explicitly disabled at build time, their use is deprecated and their code won't be maintained any further. (Some of these Spicy updates were part of Zeek 5.2 already, but hadn't been included in its NEWS section.)
Zeek events now hold network timestamps. For scheduled events, the timestamp
represents the network time for which the event was scheduled for, otherwise
it is the network time at event creation. A new bif current_event_time()
allows to retrieve the current event's network timestamp within the script-layer.
When Zeek sends events via Broker to other nodes in a cluster, an event's network
timestamp is attached to the Broker messages. On a receiving Zeek node executing a
handler for a remote event, current_event_time() returns the network time of
the sending node at the time the event was created.
The Broker level implementation allows to exchange arbitrary event metadata, but Zeek's script and C++ APIs currently only expose network timestamp functionality.
A new bif from_json() can be used to parse JSON strings into records.
type A: record { a: addr; }; local p = from_json({"a": "192.168.0.1"}", A); if ( p$valid ) print (p$v as A)
Implicit conversion from JSON to Zeek types is implemented for bool, int, count, real, interval (number as seconds) and time (number as unix timestamp), port (strings in "80/tcp" notation), patterns, addr, subnet, enum, sets, vectors and records similar to the rules of the input framework. Optional or default record fields are allowed to be missing or null in the input.
Zeek now provides native "Community ID" support with a new bif called
community_id_v1(). Two policy scripts protocols/conn/community-id-logging
and frameworks/notice/community-id extend the respective logs with a
community_id field the same way as the external zeek-community-id plugin
provides. A main difference to the external hash_conn() bif is that the
community_id_v1() takes a conn_id record instead of a connection.
Loading the new policy scripts and using the external zeek-community-id plugin at the same time is unsupported.
ZeekControl is now multi-logger aware. When multiple logger nodes are configured in ZeekControl's node.cfg, by default the log archival logic adds a logger's name as suffix to the rotated file name:
stats.11:18:57-11:19:00-logger-1.log.gz stats.11:18:57-11:19:00-logger-2.log.gz
Previously, in a multi-logger setup, individual logger processes would overwrite each other's log files during rotation, causing data loss.
For setups with a single logger, there's no change in behavior. The naming
of the final logs can be customized by providing an alternative
make-archive-name script and using the new ZEEK_ARG_LOG_SUFFIX
environment variable.
A supervisor controlled Zeek cluster is now multi-logger aware. This avoids loggers overwriting each other's log files within a single log-queue directory. By default, a logger's name is appended to the rotated logs by zeek-archiver.
Introduce a new command-line option -V / --build-info. It produces
verbose output in JSON format about the repository state and any included
plugins.
The X.509 certificate parser now exposes the signature type that is given inside the signed portion of the certificate.
The SSL parser now parses the CertificateRequest handshake message. There is a new
ssl_certificate_request event and a new parse_distinguished_name function.
We also added the protocols/ssl/certificate-request-info policy script, that
adds some additional information to ssl.log.
Add logging metrics for streams (zeek-log-stream-writes) and writers
(zeek-log-writer-writes-total).
Add networking metrics via the telemetry framework. These are enabled
when the misc/stats script is loaded.
zeek-net-dropped-packets zeek-net-link-packets zeek-net-received-bytes zeek-net-packet-lag-seconds zeek-net-received-packets-total
Except for lag, metrics originate from the get_net_stats() bif and are
updated through the Telemetry::sync() hook every 15 seconds by default.
The DNS analyzer now parses RFC 2535's AD ("authentic data") and CD ("checking
disabled") flags from DNS requests and responses, making them available in
the dns_msg record provided by many of the dns_* events. The existing
Z field remains unchanged and continues to subsume the two flags, for
backward compatibility.
The supervisor framework can now start worker nodes that read from a trace file.
Zeek can be prevented from updating network_time() to the current time
by setting allow_network_time_forward=F. Together with set_network_time()
or a custom plugin, this allows control of network_time() without Zeek
interfering.
The setting Pcap::non_fd_timeout can be used to configure the timeout
used by non-selectable packet sources in the idle case (default 20usec).
This value has previously been hard-coded, but increasing it can significantly
reduce idle CPU usage in low packet rate deployments.
Zeek now supports a new @pragma directive. It currently allows suppressing
deprecation warnings in Zeek scripts by opening with
@pragma push ignore-deprecations and closing with @pragma pop.
This particularly helps in situations where use of the Zeek base scripts, for
example to populate a deprecated field for API compatibility, would otherwise
trigger deprecation warnings.
The Reporter class was extended by a Deprecation() method to use
for logging deprecations rather than using ad-hoc Warning() calls.
The network statistics record type features a new pkts_filtered field for
reporting the number of packets that the interface filtered before hand-off to
Zeek. Packet source implementations are free to fill this field as
feasible. The default pcap packet source does not provide this information
because its availability depends on the libpcap version.
Packet statistics (packets received, packets dropped, bytes received, packets
seen on link, and packets filtered) are now reported to the Telemetry
framework, under the zeek_net prefix.
Zeek's cluster framework provides the new get_node_count(node_type: NodeType)
function to obtain the number of nodes for a given node type as defined in the
cluster layout. Furthermore, broadcast_topics was added as a collection of
broker topics that can be used to reach all nodes in a cluster.
The new Cluster::Experimental namespace has been introduced to Zeek's cluster
framework to provide experimental features. Based on practical experiences and the
adoption of an experimental feature, it may become a regular feature or be removed
in future releases. Experimental features are loaded via:
@load policy/frameworks/cluster/experimental
Zeek's cluster framework provides two new experimental events:
cluster_started: This event will be broadcasted from the manager once all
cluster-level connections have been established based on the given cluster layout.
If any node restarts (including the manager itself), the event will neither be
rebroadcasted nor raised locally for the restarted node.
node_fully_connected: This event will be sent to the manager and raised
locally once a cluster node has successfully conducted cluster-level handshakes
for all its outgoing connections to other cluster nodes based on the given cluster
layout.
Note: There is no tracking of cluster node connectivity. Thus, there is no guarantee that all peerings still exist at the time of these events being raised.
The IEEE 802.11 packet analyzer gains the ability to parse encapsulated A-MSDU packets, instead of just dropping them. It also gains the ability to properly recognize CCMP-encrypted packets. These encrypted packets are currently dropped to Zeek's inability to do anything with them.
Add packet analzyers for LLC, SNAP, and Novell 802.3, called from the Ethernet and VLAN analyzers by default.
Environment variables for the execution of log rotation postprocessors can
be set via Log::default_rotation_postprocessor_cmd_env.
The record_field record was extended by optional and record_fields()
can now be used to determine the optionality of record fields.
The ip4_hdr record was extended by DF, MF, offset and sum
to aid packet-level analysis use-cases.
Zeek now supports parsing the recently standardized DTLS 1.3. Besides the protocol
messages being correctly parsed and raising the typical SSL/TLS events, the biggest
visible change is the newly added ssl_extension_connection_id event.
The NTP analyzer now recognizes when client and server mode messages disagree
with the notion of "originator" and "responder" and flips the connection. This
can happen in packet loss or packet re-ordering scenarios. Such connections will
have a ^ added to their history.
New bifs for ceil() and log2() have been added.
Seeds for deterministic processing can now also be set through a new environment
variable called ZEEK_SEED_VALUES. The format is expected to contain 21
positive numbers separated by spaces.
The base distribution of the Zeek container images has been upgraded to Debian 12 "bookworm" and JavaScript support was enabled.
When get_file_handle() is invoked for an analyzer that did not register
an appropriate callback function, log a warning and return a generic handle
value based on the analyzer and connection information.
The &on_change attribute of set and tables is propagated through copy().
Revert back to old method of preallocating PortVal objects for all valid
port numbers, as it was implemented prior to the Windows port. Not
preallocating these objects saves a minor amount of memory for short runs of
Zeek, but comes at a performance cost for having to allocate the objects every
time a new port is seen plus do map lookups for each port. This memory savings
is mostly lost for long runs of Zeek, since all of the ports will likely end
up allocated in time.
If the version from the Windows port is desired, a new configure option
--disable-port-prealloc will disable the preallocation and enable the map
lookup version.
The main-loop has been changed to process all ready IO sources with a zero timeout in the same loop iteration. Previously, two zero-timeout sources would require two main-loop iterations. Further, when the main-loop is polling IO sources with file descriptors, zero timeout IO sources are added to the list of sources to be processed as well.
The intervals to decide when Zeek checks FD-based IO sources for readiness
have been made configurable through io_poll_interval_default and
io_poll_interval_live for ease of testing, development and debugging
of the main-loop.
Zeek does not arbitrarily update network_time() to current time anymore.
When a packet source is providing a constant stream of packets, packets
drive network time. Previously, Zeek updated network time to current
time in various situations, disregarding timestamps of network packets.
Zeek will now update network_time() only when a packet source has been
inactive/idle for an interval of packet_source_inactivity_timeout
(default 100msec). When a worker process suddenly observes no packets, timer
expiration may initially be delayed by packet_source_inactivity_timeout.
Calling suspend_processing() when reading traces does not update network
time to the current time anymore. Instead, Zeek keeps network_time()
according to the trace file. This causes scheduled events to not fire once
suspend_processing() is called, which seems more reasonable than
arbitrarily setting network_time() to current time. Processing can still
be continued from broker events or input readers.
Previously, Zeek would process and dispatch events for the very first packet
in a trace file in order to initialize time, even if suspend_processing()
was called in a zeek_init() handler. This has been changed such that the
first packet will only be processed once continue_processing() has been
invoked again. Some background around the previous behavior can be found
in GH-938. Given that the network_time_init() event explicitly signals
initialization of network time, this behavior seems more reasonable.
If an event is scheduled with a 0.0sec timeout from a zeek_init() handler
that also invokes suspend_processing(), the scheduled event will fire
immediately with network_time() still yielding 0.0. Previously,
network_time() was set to the current time. The new behavior provides
more deterministic operation and aligns with timers stopping during a
suspend_processing().
Broker no longer initializes network time to current time when processing input. Particularly in combination with pcap processing this was not desirable behavior.
The IO loop's poll interval is now correctly reduced from 100 to 10 for live packet sources. This should lower CPU usage for deployments with non-selectable packet sources.
Zeek's CMake scaffolding has received an overhaul for modernizing the build system and to make it easier to maintain going forward. Plugins can now use a declarative interface for adding all sources, BIFs, etc. in one block instead of using the previous begin/end functions. While the old plugin functions still exist for backward compatibility, the underlying codebase requires newer CMake features. Plugin authors should raise their minimum required CMake version to 3.15, to match Zeek's.
The IRC data analyzer does not extract DCC acknowledgements to files anymore.
Instead, irc_dcc_send_ack is raised with the bytes acknowledged by the
recipient.
The IRC base script now use file_sniff() instead of file_new() for
DCC file transfers to capture fuid and inferred MIME type in irc.log.
The ignore_checksums script variable now reflects the correct value
when using the -C command-line flag.
Support for ARUBA GRE tunnels now covers all of the known protocol type values for those tunnels.
The vlan field reported by the AF_PACKET packet source is now properly masked to exclude PCP and DEI bits. Previously, these bits were included and could cause invalid vlan values > 4095 to be reported.
Libpcap based packet source now avoids the 32bit wraparound of link and dropped packet counters as reported by users.
The ssl_history field in ssl.log indicates that the letter j is reserved
for hello retry requests. However, this logging was never fully implemented;
instead, hello retry requests were logged like as a server hello (with the letter
s). This oversight was fixed, and hello retry requests are now correctly logged.
When per-connection SMB parser state (read offsets, tree ids, ...) exceeds
SMB::max_pending_messages (default 1000), Zeek discards such per-connection
state and raises a new smb2_discarded_messages_state() event. This event is
used to reset script-layer SMB state. This change provides protection against
unbounded state growth due to partial or one-sided SMB connections.
Setting SMB::max_pending_messages to 0 can be used to switch back to the
previous behavior of not discarding state. Setting SMB::enable_state_clear
to F skips the script-layer state clearing logic.
Fix disable_analyzer() builtin function crashing when attempting to
disable connection's root analyzers.
Zeek script vectors now support negative indices.
local v = vector(1, 2, 3); print v[-1]; # prints 3
Function parameters are rendered by Zeekygen as :param x rather than just
:x:. This allows to group parameters Zeek's documentation.
Mixing vector and scalar operands for binary expressions, like addition, multiplication, etc., is now an error.
Using deprecated when semantics without capturing variables is now an error.
Referencing local variables in a more outer scope than where they were declared is now an error
worker_count has been deprecated in favor of the
new function get_active_node_count(node_type: NodeType) that can be used
to obtain the number of nodes of a given type the calling node is currently
connected to.Zeek now treats private address space (i.e., non-routable IP address ranges)
as local by default, matching the intuition of many users that e.g. a
192.168/16 IP address should show up as local in the logs. To do this, Zeek
automatically adds Site::private_address_space to Site::local_nets at
startup. Subsequent runtime updates to Site::private_address_space
propagate to Site::local_nets, while updates to the latter don't affect
the former.
You're free to define Site::local_nets as before and do not need to update
your configurations. If you added standard private address space to
Site::local_nets in the past, you no longer need to do so. This also
applies to zeekctl's networks.cfg file.
The new global Boolean Site::private_address_space_is_local, true by
default, controls the behavior. A redef to false brings back Zeek's prior
behavior of considering private address space an unrelated concept, which will
come in handy for example when working with tests that compare results against
log baselines that have not yet been updated.
Telemetry centralization and Prometheus exposition is not enabled by default anymore. Previously, the manager node would open port 9911/tcp by default and import all metrics from other nodes. For large clusters, the current implementation introduces significant processing overhead on the manager even if the Prometheus functionality is not used. While inconvenient, disable this functionality (assumed to be used by few as of now) by default to preserve resources.
The script to enable centralization and the Prometheus endpoint is now
located in the policy/ folder. Re-enable the old functionality with:
@load frameworks/telemetry/prometheus
You may experiment with increasing Broker::metrics_export_interval
(default 1s) to reduce the extra overhead and communication at the expense
of stale metrics.
Custom source tarballs require a repo-info.json file.
Note, should you be using official Zeek release tarballs only, or build Zeek solely from git checkouts, this does not affect you.
However, if you're building your own Zeek source tarballs, it is now required
that a repo-info.json file exists at the top-level. The dist target was
extended to add this file and official Zeek release source tarballs will
contain it going forward.
The following command can be used to produce repo-info.json:
python3 ./ci/collect-repo-info.py --only-git > ../path/to/tarballdir/repo-info.json
This is required to support the new -V / --build-info option that
provides information about git submodules and included plugins used during
the build. The ci/collect-repo-info.py tool runs at ./configure time
and either collects the required information from a git clone (when git is
installed), or otherwise uses the content of a file named repo-info.json.
If you see opportunities to extend repo-info.json with further information,
please get in touch.
Plugin authors should raise the minimum required CMake version to 3.15 to ensure compatibility with new CMake scaffolding included in this release. Older versions will trigger a warning at configuration time and, depending on the functionality included in the plugin, may trigger subsequent errors during configuration or build.
Zeek container images are not pushed to the zeekurity organization anymore.
Please switch to using the zeek/zeek image on DockerHub, or the images
published to public.ecr.aws/zeek/zeek.
The IRC_Data analyzer declaration has been moved to protocols/irc/IRC.h.
The error message returned when using bro_init, bro_done, and
bro_script_loaded events is now removed. removed. Usage of these events
has returned that error during script parsing for a few years, and time has
come to finally remove it.
Zeek now features experimental JavaScript support:
/* hello.js */ zeek.on('zeek_init', () => { console.log('Hello, Zeek!'); });
$ zeek ./hello.js Hello, Zeek!
When building Zeek on a system that features a recent (16.13+) version of the
libnode package with development headers, Zeek automatically includes the
externally-maintained ZeekJS plugin (https://github.com/corelight/zeekjs) as a
builtin plugin. This allows Zeek to load and execute JavaScript code located
in .js or .cjs files. When no such files are passed to Zeek, the
JavaScript engine and Node.js environment aren't initialized and there is no
runtime impact.
The Linux distributions Fedora 37 & 38, Ubuntu 22.10, and the upcoming Debian
12 release provide suitable packages. On other platforms, Node.js can be built
from source with the --shared option.
To disable this functionality, pass --disable-javascript to configure.
Zeek now comes with Spicy support built in, meaning it can now leverage any analyzers written in Spicy out of the box. While the interface layer connecting Zeek and Spicy used to be implemented through an external Zeek plugin, that code has now moved into the Zeek code base itself. We also added infrastructure to Zeek that enables its built-in standard analyzers to use Spicy instead of Binpac. As initial (simple) examples, Zeek's Syslog and Finger analyzers are now implemented in Spicy. While their legacy versions remain available as fallbacks for now in case Spicy gets explicitly disabled at build time, their use is deprecated and their code won't be maintained any further. (Some of these Spicy updates were part of Zeek 5.2 already, but hadn't been included in its NEWS section.)
Zeek events now hold network timestamps. For scheduled events, the timestamp
represents the network time for which the event was scheduled for, otherwise
it is the network time at event creation. A new bif current_event_time()
allows to retrieve the current event's network timestamp within the script-layer.
When Zeek sends events via Broker to other nodes in a cluster, an event's network
timestamp is attached to the Broker messages. On a receiving Zeek node executing a
handler for a remote event, current_event_time() returns the network time of
the sending node at the time the event was created.
The Broker level implementation allows to exchange arbitrary event metadata, but Zeek's script and C++ APIs currently only expose network timestamp functionality.
A new bif from_json() can be used to parse JSON strings into records.
type A: record { a: addr; }; local p = from_json({"a": "192.168.0.1"}", A); if ( p$valid ) print (p$v as A)
Implicit conversion from JSON to Zeek types is implemented for bool, int, count, real, interval (number as seconds) and time (number as unix timestamp), port (strings in "80/tcp" notation), patterns, addr, subnet, enum, sets, vectors and records similar to the rules of the input framework. Optional or default record fields are allowed to be missing or null in the input.
Zeek now provides native "Community ID" support with a new bif called
community_id_v1(). Two policy scripts protocols/conn/community-id-logging
and frameworks/notice/community-id extend the respective logs with a
community_id field the same way as the external zeek-community-id plugin
provides. A main difference to the external hash_conn() bif is that the
community_id_v1() takes a conn_id record instead of a connection.
Loading the new policy scripts and using the external zeek-community-id plugin at the same time is unsupported.
ZeekControl is now multi-logger aware. When multiple logger nodes are configured in ZeekControl's node.cfg, by default the log archival logic adds a logger's name as suffix to the rotated file name:
stats.11:18:57-11:19:00-logger-1.log.gz stats.11:18:57-11:19:00-logger-2.log.gz
Previously, in a multi-logger setup, individual logger processes would overwrite each other's log files during rotation, causing data loss.
For setups with a single logger, there's no change in behavior. The naming
of the final logs can be customized by providing an alternative
make-archive-name script and using the new ZEEK_ARG_LOG_SUFFIX
environment variable.
A supervisor controlled Zeek cluster is now multi-logger aware. This avoids loggers overwriting each other's log files within a single log-queue directory. By default, a logger's name is appended to the rotated logs by zeek-archiver.
Introduce a new command-line option -V / --build-info. It produces
verbose output in JSON format about the repository state and any included
plugins.
The X.509 certificate parser now exposes the signature type that is given inside the signed portion of the certificate.
The SSL parser now parses the CertificateRequest handshake message. There is a new
ssl_certificate_request event and a new parse_distinguished_name function.
We also added the protocols/ssl/certificate-request-info policy script, that
adds some additional information to ssl.log.
Add logging metrics for streams (zeek-log-stream-writes) and writers
(zeek-log-writer-writes-total).
Add networking metrics via the telemetry framework. These are enabled
when the misc/stats script is loaded.
zeek-net-dropped-packets zeek-net-link-packets zeek-net-received-bytes zeek-net-packet-lag-seconds zeek-net-received-packets-total
Except for lag, metrics originate from the get_net_stats() bif and are
updated through the Telemetry::sync() hook every 15 seconds by default.
The DNS analyzer now parses RFC 2535's AD ("authentic data") and CD ("checking
disabled") flags from DNS requests and responses, making them available in
the dns_msg record provided by many of the dns_* events. The existing
Z field remains unchanged and continues to subsume the two flags, for
backward compatibility.
The supervisor framework can now start worker nodes that read from a trace file.
Zeek can be prevented from updating network_time() to the current time
by setting allow_network_time_forward=F. Together with set_network_time()
or a custom plugin, this allows control of network_time() without Zeek
interfering.
The setting Pcap::non_fd_timeout can be used to configure the timeout
used by non-selectable packet sources in the idle case (default 20usec).
This value has previously been hard-coded, but increasing it can significantly
reduce idle CPU usage in low packet rate deployments.
Zeek now supports a new @pragma directive. It currently allows suppressing
deprecation warnings in Zeek scripts by opening with
@pragma push ignore-deprecations and closing with @pragma pop.
This particularly helps in situations where use of the Zeek base scripts, for
example to populate a deprecated field for API compatibility, would otherwise
trigger deprecation warnings.
The Reporter class was extended by a Deprecation() method to use
for logging deprecations rather than using ad-hoc Warning() calls.
The network statistics record type features a new pkts_filtered field for
reporting the number of packets that the interface filtered before hand-off to
Zeek. Packet source implementations are free to fill this field as
feasible. The default pcap packet source does not provide this information
because its availability depends on the libpcap version.
Packet statistics (packets received, packets dropped, bytes received, packets
seen on link, and packets filtered) are now reported to the Telemetry
framework, under the zeek_net prefix.
Zeek's cluster framework provides the new get_node_count(node_type: NodeType)
function to obtain the number of nodes for a given node type as defined in the
cluster layout. Furthermore, broadcast_topics was added as a collection of
broker topics that can be used to reach all nodes in a cluster.
The new Cluster::Experimental namespace has been introduced to Zeek's cluster
framework to provide experimental features. Based on practical experiences and the
adoption of an experimental feature, it may become a regular feature or be removed
in future releases. Experimental features are loaded via:
@load policy/frameworks/cluster/experimental
Zeek's cluster framework provides two new experimental events:
cluster_started: This event will be broadcasted from the manager once all
cluster-level connections have been established based on the given cluster layout.
If any node restarts (including the manager itself), the event will neither be
rebroadcasted nor raised locally for the restarted node.
node_fully_connected: This event will be sent to the manager and raised
locally once a cluster node has successfully conducted cluster-level handshakes
for all its outgoing connections to other cluster nodes based on the given cluster
layout.
Note: There is no tracking of cluster node connectivity. Thus, there is no guarantee that all peerings still exist at the time of these events being raised.
The IEEE 802.11 packet analyzer gains the ability to parse encapsulated A-MSDU packets, instead of just dropping them. It also gains the ability to properly recognize CCMP-encrypted packets. These encrypted packets are currently dropped to Zeek's inability to do anything with them.
Add packet analzyers for LLC, SNAP, and Novell 802.3, called from the Ethernet and VLAN analyzers by default.
Environment variables for the execution of log rotation postprocessors can
be set via Log::default_rotation_postprocessor_cmd_env.
The record_field record was extended by optional and record_fields()
can now be used to determine the optionality of record fields.
The ip4_hdr record was extended by DF, MF, offset and sum
to aid packet-level analysis use-cases.
Zeek now supports parsing the recently standardized DTLS 1.3. Besides the protocol
messages being correctly parsed and raising the typical SSL/TLS events, the biggest
visible change is the newly added ssl_extension_connection_id event.
The NTP analyzer now recognizes when client and server mode messages disagree
with the notion of "originator" and "responder" and flips the connection. This
can happen in packet loss or packet re-ordering scenarios. Such connections will
have a ^ added to their history.
New bifs for ceil() and log2() have been added.
Seeds for deterministic processing can now also be set through a new environment
variable called ZEEK_SEED_VALUES. The format is expected to contain 21
positive numbers separated by spaces.
The base distribution of the Zeek container images has been upgraded to Debian 12 "bookworm" and JavaScript support was enabled.
When get_file_handle() is invoked for an analyzer that did not register
an appropriate callback function, log a warning and return a generic handle
value based on the analyzer and connection information.
The &on_change attribute of set and tables is propagated through copy().
Revert back to old method of preallocating PortVal objects for all valid
port numbers, as it was implemented prior to the Windows port. Not
preallocating these objects saves a minor amount of memory for short runs of
Zeek, but comes at a performance cost for having to allocate the objects every
time a new port is seen plus do map lookups for each port. This memory savings
is mostly lost for long runs of Zeek, since all of the ports will likely end
up allocated in time.
If the version from the Windows port is desired, a new configure option
--disable-port-prealloc will disable the preallocation and enable the map
lookup version.
The main-loop has been changed to process all ready IO sources with a zero timeout in the same loop iteration. Previously, two zero-timeout sources would require two main-loop iterations. Further, when the main-loop is polling IO sources with file descriptors, zero timeout IO sources are added to the list of sources to be processed as well.
The intervals to decide when Zeek checks FD-based IO sources for readiness
have been made configurable through io_poll_interval_default and
io_poll_interval_live for ease of testing, development and debugging
of the main-loop.
Zeek does not arbitrarily update network_time() to current time anymore.
When a packet source is providing a constant stream of packets, packets
drive network time. Previously, Zeek updated network time to current
time in various situations, disregarding timestamps of network packets.
Zeek will now update network_time() only when a packet source has been
inactive/idle for an interval of packet_source_inactivity_timeout
(default 100msec). When a worker process suddenly observes no packets, timer
expiration may initially be delayed by packet_source_inactivity_timeout.
Calling suspend_processing() when reading traces does not update network
time to the current time anymore. Instead, Zeek keeps network_time()
according to the trace file. This causes scheduled events to not fire once
suspend_processing() is called, which seems more reasonable than
arbitrarily setting network_time() to current time. Processing can still
be continued from broker events or input readers.
Previously, Zeek would process and dispatch events for the very first packet
in a trace file in order to initialize time, even if suspend_processing()
was called in a zeek_init() handler. This has been changed such that the
first packet will only be processed once continue_processing() has been
invoked again. Some background around the previous behavior can be found
in GH-938. Given that the network_time_init() event explicitly signals
initialization of network time, this behavior seems more reasonable.
If an event is scheduled with a 0.0sec timeout from a zeek_init() handler
that also invokes suspend_processing(), the scheduled event will fire
immediately with network_time() still yielding 0.0. Previously,
network_time() was set to the current time. The new behavior provides
more deterministic operation and aligns with timers stopping during a
suspend_processing().
Broker no longer initializes network time to current time when processing input. Particularly in combination with pcap processing this was not desirable behavior.
The IO loop's poll interval is now correctly reduced from 100 to 10 for live packet sources. This should lower CPU usage for deployments with non-selectable packet sources.
Zeek's CMake scaffolding has received an overhaul for modernizing the build system and to make it easier to maintain going forward. Plugins can now use a declarative interface for adding all sources, BIFs, etc. in one block instead of using the previous begin/end functions. While the old plugin functions still exist for backward compatibility, the underlying codebase requires newer CMake features. Plugin authors should raise their minimum required CMake version to 3.15, to match Zeek's.
The IRC data analyzer does not extract DCC acknowledgements to files anymore.
Instead, irc_dcc_send_ack is raised with the bytes acknowledged by the
recipient.
The IRC base script now use file_sniff() instead of file_new() for
DCC file transfers to capture fuid and inferred MIME type in irc.log.
The ignore_checksums script variable now reflects the correct value
when using the -C command-line flag.
Support for ARUBA GRE tunnels now covers all of the known protocol type values for those tunnels.
The vlan field reported by the AF_PACKET packet source is now properly masked to exclude PCP and DEI bits. Previously, these bits were included and could cause invalid vlan values > 4095 to be reported.
Libpcap based packet source now avoids the 32bit wraparound of link and dropped packet counters as reported by users.
The ssl_history field in ssl.log indicates that the letter j is reserved
for hello retry requests. However, this logging was never fully implemented;
instead, hello retry requests were logged like as a server hello (with the letter
s). This oversight was fixed, and hello retry requests are now correctly logged.
When per-connection SMB parser state (read offsets, tree ids, ...) exceeds
SMB::max_pending_messages (default 1000), Zeek discards such per-connection
state and raises a new smb2_discarded_messages_state() event. This event is
used to reset script-layer SMB state. This change provides protection against
unbounded state growth due to partial or one-sided SMB connections.
Setting SMB::max_pending_messages to 0 can be used to switch back to the
previous behavior of not discarding state. Setting SMB::enable_state_clear
to F skips the script-layer state clearing logic.
Fix disable_analyzer() builtin function crashing when attempting to
disable connection's root analyzers.
Zeek script vectors now support negative indices.
local v = vector(1, 2, 3); print v[-1]; # prints 3
Function parameters are rendered by Zeekygen as :param x rather than just
:x:. This allows to group parameters Zeek's documentation.
Mixing vector and scalar operands for binary expressions, like addition, multiplication, etc., is now an error.
Using deprecated when semantics without capturing variables is now an error.
Referencing local variables in a more outer scope than where they were declared is now an error
worker_count has been deprecated in favor of the
new function get_active_node_count(node_type: NodeType) that can be used
to obtain the number of nodes of a given type the calling node is currently
connected to.