syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.
This is the combination of the news entries of 4.0.0 and 4.0.1.
This is a new major version of syslog-ng, ending the 3.x series which started roughly 13 years ago, on 17th February 2009.
Like all releases in the 3.x series, 4.0.0 is not a breaking change either. Long-term compatibility has been and continues to be an essential objective of syslog-ng; thus, you can still run unchanged configurations that were originally created for syslog-ng 3.0.0.
You can safely upgrade to 4.0.0 if you followed along 3.x, and you should probably also consider upgrading if you are stuck with an older 3.x release.
The new version number primarily indicates that this version of syslog-ng is much more than the software we released 13 years ago. While it does have certain "big-bang" items in its feature list, new features were continuously introduced throughout our 3.x series as well. Our engineering practices have not changed simply because we were working on a new major release: this is the continuation of our previous releases in every respect, produced in the same manner, just with a more catchy version number.
For this reason, there is no separate deprecation or support period for 3.x releases, similarly with our existing practice. We support earlier syslog-ng releases by providing maintenance and fixes in the new release track. Fixes to problems are not backported to earlier releases by the syslog-ng project.
syslog-ng uses a data model where a log message contains an unordered set of name-value pairs. The values stored in these name-value pairs are usually textual, so syslog-ng has traditionally stored these values in text format.
With the increase of JSON-based message sources and destinations, types became more important. If we encounter a message where a name-value pair originates from a JSON document, and this document contains a member that is numeric, we may want to reproduce that as we send this data to a consumer.
For example, sometimes we extract a numerical metric from a log message, and we need to send this to a consumer, again with the correct type.
To be able to do this, we added runtime type information to the syslog-ng message model: each name-value pair becomes a (name, type, value) triplet.
We introduced the following types:
Apart from the syslog-ng core supporting the notion of types, its use is up to the sources, filters, rewrite rules, parsers and destinations that set or make use of them in any way it makes the most sense for the component in question.
syslog-ng uses filter expressions to make routing decisions and during the transformation of messages. These filter expressions are used in filter {} or if {} statements, for example.
In these expressions, you can use comparison operators. This example, for instance, uses the '>' operator to check for HTTP response codes greater-or-equal than 500:
if ("${apache.response}" >= 500) {
};
Earlier, we had two sets of operators, one for numeric (==, !=, <, >) and the other for string-based comparisons (eq, ne, gt, lt).
The separate operators were cumbersome to use. Users often forgot which operator was the right one for a specific case.
Typing allows us to do the right thing in most cases automatically, and a syntax that allows the user to override the automatic decisions in the rare case.
With that, starting with 4.0, the old-numeric operators have been converted to be type-aware operators. It would compare as strings if both sides of the comparisons are strings. It would compare numerically if at least one side is numeric. A great deal of inspiration was taken from JavaScript, which was considered to be a good model, since the problem space is similar.
See this blog post for more details: https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/
When using json-parser(), syslog-ng converts all members of a JSON object to syslog-ng name-value pairs. Prior to the introduction of type support, these name-value pairs were all stored as strings. Any type information originally present in the incoming JSON object was lost.
This meant that if you regenerated the JSON from the name-value pairs using the $(format-json) template function, all numbers, booleans and other types became strings in the output.
There has been a feature in syslog-ng that alleviated the loss of types. This feature was called "type-hints". Type-hints tell $(format-json) to use a specific type on output, independently of a name-value pair's original type, but this type conversion needed to be explicit in the configuration.
An example configuration that parses JSON on input and produces a JSON on output:
log {
source { ... };
parser { json-parser(prefix('.json.')); };
destination { file(... template("$(format-json .json.*)\n")); };
};
To augment the above with type hinting, you could use:
log {
source { ... };
parser { json-parser(prefix('.json.')); };
destination { file(... template("$(format-json .json.* .json.value=int64(${.json.value})\n")); };
};
NOTE the presence of the int64() type hint in the 2nd example.
The new feature introduced with typing is that syslog-ng would automatically store the JSON type information as a syslog-ng type, thus it will transparently carry over types from inputs to output, without having to be explicit about them.
Typing is a feature throughout syslog-ng, and although the gust of it has been explained in the highlights section, some further details are documented in the list down below:
type-aware comparisons in filter expressions: as detailed above, the previously numeric operators become type-aware, and the exact comparison performed will be based on types associated with the values we compare.
json-parser() and $(format-json): JSON support is massively improved with the introduction of types. For one: type information is retained across input parsing->transformation->output formatting. JSON lists (arrays) are now supported and are converted to syslog-ng lists so they can be manipulated using the $(list-*) template functions. There are other important improvements in how we support JSON.
set(), groupset(): in any case where we allow the use of templates, support for type-casting was added, and the type information is properly promoted.
db-parser() type support: db-parser() gets support for type casts,
<value name="foobar" type="integer">$PID</value>. The “integer” is a type-cast that
associates $foobar with an integer type. db-parser()’s internal parsers
(e.g. @NUMBER@) will also associate type information with a name-value
pair automatically.
add-contextual-data() type support: any new name-value pair that is populated using add-contextual-data() will propagate type information, similarly to db-parser().
map-value-pairs() type support: propagate type information
SQL type support: the sql() driver gained support for types, so that columns with specific types will be stored as those types.
template type support: templates can now be casted explicitly to a specific type, but they also propagate type information from macros/template functions and values in the template string
value-pairs type support: value-pairs form the backbone of specifying a set of name-value pairs and associated transformations to generate JSON or a key-value pair format. It also gained support for types, the existing type-hinting feature that was already part of value-pairs was adapted and expanded to other parts of syslog-ng.
python() typing: support for typing was added to all Python components
(sources, destinations, parsers and template functions), along with more
documentation & examples on how the Python bindings work. All types except
json() are supported as they are queried- or changed by Python code.
on-disk serialized formats (e.g. disk buffer/logstore): we remain compatible with messages serialized with an earlier version of syslog-ng, and the format we choose remains compatible for “downgrades” as well. E.g. even if a new version of syslog-ng serialized a message, the old syslog-ng and associated tools will be able to read it (sans type information of course)
For syslog-ng, everything is traditionally a string. A convention was
started with syslog-ng in v3.10, where a comma-separated format
could be used as a kind of array using the $(list-*) family of template
functions.
For example, $(list-head) takes off the first element in a list, while $(list-tail) takes the last. You can index and slice list elements using the $(list-slice) and $(list-nth) functions and so on.
syslog-ng has started to return such lists in various cases, so they can be manipulated using these list-specific template functions. These include the xml-parser(), or the $(explode) template function, but there are others.
Here is an example that has worked since syslog-ng 3.10:
# MSG contains foo:bar:baz
# - the $(list-head) takes off the first element of a list
# - the $(explode) expression splits a string at the specified separator, ':' in this case.
$(list-head $(explode : $MSG))
New functions that improve these features:
JSON arrays are converted to lists, making it a lot easier to slice and extract information from JSON arrays. Of course, $(format-json) will take lists and convert them back to arrays.
The $* is a new macro that converts the internal list of match variables ($1, $2, $3 and so on) to a list, usable with $(list-*) template functions. These match variables have traditionally been filled by regular expressions when a capture group in a regexp matches.
The set-matches() rewrite operation performs the reverse; it assigns the match variables to list elements, making it easier to use list elements in template expressions by assigning them to $1, $2, $3 and so on.
Top-level JSON arrays (e.g. ones where the incoming JSON data is an array and not an object) are now accepted, and the array elements are assigned to the match variables.
syslog-ng has had support for Python-based processing elements since 3.7, released in 2015, which was greatly expanded early 2017 (3.9, LogParser) and late 2018 (3.18, LogSource and LogFetcher).
This support has now been improved in a number of ways to make its use both easier and its potential more powerful.
A framework was added to syslog-ng that allows seamless implementation of
syslog-ng features in Python, with a look and feel of that of a native
implementation. An example for using this framework is available in the
modules/python-modules/example directory, as well as detailed
documentation in the form of modules/python-modules/README.md that is
installed to /etc/syslog-ng/python.
The framework consists of these changes:
syslogng Python package: native code provided by the syslog-ng core
has traditionally been exported in the syslogng Python module. An
effort was made to make these native classes exported by the C layer
more discoverable and more intuitive. As a part of this effort, the
interfaces for all key Python components (LogSource, LogFetcher,
LogDestination, LogParser) were exposed in the syslogng module, along
with in-line documentation.
/etc/syslog-ng/python: syslog-ng now automatically adds this directory to
the PYTHONPATH so that you have an easy place to add Python modules required
by your configuration.
Python virtualenv support for production use: more sophisticated Python
modules usually have 3rd party dependencies, which either needed to be
installed from the OS repositories (using the apt-get or yum/dnf tools) or
PyPI (using the pip tool). syslog-ng now acquired support for an embedded
Python virtualenv (/var/lib/syslog-ng/python-venv or similar, depending on
the installation layout), meaning that these requirements can be installed
privately, without deploying them in the system PYTHONPATH where it might
collide with other applications. The base set of requirements that
syslog-ng relies on can be installed via the syslog-ng-update-virtualenv
script, which has been added to our rpm/deb postinst scripts.
Our mod-python module validates this virtualenv at startup and activates it automatically if the validation is successful. You can disable this behaviour by loading the Python module explicitly with the following configuration statement:
@module mod-python use-virtualenv(no)
You can force syslog-ng to use a specific virtualenv by activating it first, prior to executing syslog-ng. In this case, syslog-ng will not try to use its private virtualenv, rather it would use the one activated when it was started. It assumes that any requirements needed for syslog-ng functionality implemented in Python are deployed by the user. These requirements are listed in the /usr/lib/syslog-ng/python/requirements.txt file.
SCL snippets in Python plugins: by adding an scl/whatever.conf file to
your Python-based syslog-ng plugin, you can easily wrap a Python-based
log processing functionality with a syslog-ng block {}, so the user can
use a syntax very similar to native plugins in their main configuration.
confgen in Python: should a simple block {} statement not be enough to
wrap the functionality implemented in Python, the mod-python module now
supports confgen functions to be implemented in Python. confgen
has been a feature in syslog-ng for a long time that allows you to
generate configuration snippets dynamically by executing an external
program or script. This has now been ported to Python, e.g.
syslog-ng can invoke a Python function to generate parts of its
configuration.
Example:
@version: 4.0
python {
from syslogng import register_config_generator
def generate_foobar(args):
print(args)
return "tcp(port(2000))"
#
# this registers a plugin in the "source" context named "foobar"
# which would invoke the generate_foobar() function when a foobar() source
# reference is encountered.
#
register_config_generator("source", "foobar", generate_foobar)
};
log {
# we are actually calling the generate_foobar() function in this
# source, passing all parameters as values in the "args" dictionary
source { foobar(this(is) a(value)); };
destination { file("logfile"); };
};
kubernetes() source and kubernetes-metadata-parser(): these two
components gained the ability to enrich log messages with Kubernetes
metadata. When reading container logs, syslog-ng would query the Kubernetes
API for the following fields and add them to the log-message. The returned
meta-data is cached in memory, so not all log messages trigger a new query.
.k8s.pod_uuid
.k8s.labels.<label_name>
.k8s.annotations.<annotation_name>
.k8s.namespace_name
.k8s.pod_name
.k8s.container_name
.k8s.container_image
.k8s.container_hash
.k8s.docker_id
java() destinations: fixed compatibility with newer Java versions,
syslog-ng is now able to compile up to Java 18.
disk-buffer: Added prealloc() option to preallocate new disk-buffer
files.
(#4056)
disk-buffer: The default value of truncate-size-ratio() has been changed to 1,
which means truncation is disabled by default. This means that by default, the
disk-buffer files will gradually become larger and will never reduce in size.
This improves performance.
(#4056)
log-level(): added a new global option to control syslog-ng's own internal
log level. This augments the existing support for doing the same via the
command line (via -d, -v and -t options) and via syslog-ng-ctl. This change
also causes higher log-levels to include messages from lower log-levels,
e.g. "trace" also implies "debug" and "verbose". By adding this capability
to the configuration, it becomes easier to control logging in containerized
environments where changing command line options is more challenging.
syslog-ng-ctl log-level: this new subcommand in syslog-ng-ctl allows
setting the log level in a more intuitive way, compared to the existing
syslog-ng-ctl verbose|debug|trace -s syntax.
syslog-ng --log-level: this new command line option for the syslog-ng
main binary allows you to set the desired log-level similar to how you
can control it from the configuration or through syslog-ng-ctl.
(#4091)
network/syslog/tls context options: SSL_CONF_cmd support
SSL_CONF_cmd TLS configuration support for network() and syslog() driver has been added.
OpenSSL offers an alternative, software-independent configuration mechanism through the SSL_CONF_cmd interface to support a common solution for setting the so many various SSL_CTX and SSL options that can be set earlier via multiple, separated openssl function calls only. This update implements that similar to the mod_ssl in Apache.
IMPORTANT: The newly introduced openssl-conf-cmds always has the
highest priority, its content parsed last, so it will override any other
options that can be found in the tls() section, does not matter if
they appear before or after openssl-conf-cmds.
As described in the SSL_CONF_cmd documentation, the order of operations within openssl-conf-cmds() is significant and the commands are executed in top-down order. This means that if there are multiple occurrences of setting the same option then the 'last wins'. This is also true for options that can be set multiple ways (e.g. used cipher suites and/or protocols).
Example config:
source source_name {
network (
ip(0.0.0.0)
port(6666)
transport("tls")
tls(
ca-dir("/etc/ca.d")
key-file("/etc/cert.d/serverkey.pem")
cert-file("/etc/cert.d/servercert.pem")
peer-verify(yes)
openssl-conf-cmds(
# For system wide available cipher suites use: /usr/bin/openssl ciphers -v
# For formatting rules see: https://www.openssl.org/docs/man1.1.1/man3/SSL_CONF_cmd.html
# For quick and dirty testing try: https://github.com/rbsec/sslscan
#
"CipherString" => "ECDHE-RSA-AES128-SHA", # TLSv1.2 and bellow
"CipherSuites" => "TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384", # TLSv1.3+ (OpenSSl 1.1.1+)
"Options" => "PrioritizeChaCha",
"Protocol" => "-ALL,TLSv1.3",
)
)
);
};
network/syslog/http destination: OCSP stapling support
OCSP stapling support for network destinations and for the http() module has been added.
When OCSP stapling verification is enabled, the server will be requested to send back OCSP status responses.
This status response will be verified using the trust store configured by the user (ca-file(), ca-dir(), pkcs12-file()).
Note: RFC 6961 multi-stapling and TLS 1.3-provided multiple responses are currently not validated, only the peer certificate is verified.
Example config:
destination {
network("test.tld" transport(tls)
tls(
pkcs12-file("/path/to/test.p12")
peer-verify(yes)
ocsp-stapling-verify(yes)
)
);
http(url("https://test.tld") method("POST") tls(peer-verify(yes) ocsp-stapling-verify(yes)));
};
(#4082)
Python LogMessage class: get_pri() and get_timestamp() methods were added that
allow the query of the syslog-style priority and the message timestamp,
respectively. The return value of get_pri() is an integer, while
get_timestamp() returns a Python datetime.datetime instance. Some macros
that were previously unavailable from Python (e.g. the STAMP, R_STAMP and
C_STAMP macros) are now made available.
Python Logger: the low-level Logger class exported by syslog-ng was
wrapped by a logging.LogHandler class so that normal Python APIs for logging
can now be used.
db-parser() and grouping-by(): added a prefix() option to both
db-parser() and grouping-by() that allows specifying an extra prefix
to be prepended to all name-value pairs that get extracted from messages
using patterns or
csv-parser(): add a new dialect, called escape-backslash-with-sequences
which uses "" as an escape character but also supports C-style escape
sequences, like "\n" or "\r".
tcp(), network() or syslog() destinations: fixed a crash that could
happen after reload when a kept-alive connection is terminated, in case
the target server is configured using a hostname (and not an IP address)
and that name becomes unresolvable (e.g. dropped from DNS or /etc/hosts)
(#4044)
python() destination: Fixed a crash, when trying to resolve the
"R_STAMP", "P_STAMP" or "STAMP" macros from Python code.
(#4057)
Python LogSource & LogFetcher: a potential deadlock was fixed in
acknowledgement tracking.
Python LogTemplate: the use of template functions in templates
instantiated from Python caused a crash, which has been fixed.
grouping-by() persist-name() option: fixed a segmentation fault in the
grammar.
(#4180)
$(format-json): fix a bug in the --key-delimiter option introduced in
3.38, which causes the generated JSON to contain multiple values for the
same key in case the key in question contains a nested object and
key-delimiter specified is not the dot character.
(#4127)
add-contextual-data(): add compatibility warnings and update advise in
case of the value field of the add-contextual-data() database contains an
expression that resembles the new type-hinting syntax: type(value).
syslog-ng --help screen: the output for the --help command line option has
included sample paths to various files that contained autoconf style
directory references (e.g. ${prefix}/etc for instance). This is now fixed,
these paths will contain the expanded path. Fixes Debian Bug report #962839:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962839
(#4143)
csv-parser(): fixed the processing of the dialect() parameter, which was
not taken into consideration.
apache-accesslog-parser(): Apache may use backslash-style escapes in the
request field, so support it by setting the csv-parser() dialect to
escape-backslash-with-sequences. Also added validation that the
rawrequest field contains a valid HTTP request and only extract verb,
request and httpversion if this is the case.
riemann: fixed severity levels of Riemann diagnostics messages, the error
returned by riemann_communicate() was previously only logged at the trace
level and was even incomplete: not covering the case where
riemann_communicate() returns NULL.
(#4238)
python: python2 support is now completely removed. syslog-ng can no
longer be configured with --with-python=2.
(#4057)
python: Python 2 support is now completely removed from the syslog-ng
functional test framework, called Light, too. Light will support only Python 3
from now.
(#4174)
Python virtualenv support for development use: syslog-ng is now capable of using a build-time virtualenv, where all Python development tools are automatically deployed by the build system. You can control if you want to use this using the --with-python-packages configure option. There are three possible values for this parameter:
venv: denoting that you want to use the virtualenv and install
all these requirements automatically using pip, into the venv.system: meaning that you want to rely on the system Python
without using a virtualenv. syslog-ng build scripts would install
requirements automatically to the system Python path usually
/usr/local/lib/pythonX.Y
none: disable deploying packages automatically. All
dependencies are assumed to be present in the system Python before
running the syslog-ng build process.Please note that syslog-ng has acquired quite a number of these
development time dependencies with the growing number of functionality
the Python binding offers, so using the system or none settings are
considered advanced usage, meant to be used for distro packaging.
make dist: fixed make dist of FreeBSD so that source tarballs can
easily be produced even if running on FreeBSD.
(#4163)
Debian and derivatives: The syslog-ng-mod-python package is now built with python3 on the following platforms:
debian-stretch
debian-buster
ubuntu-bionic
(#4057)dbld: Removed support for ubuntu-xenial.
(#4057)
dbld: Updated support from Fedora 35 to Fedora 37
Leaner production docker image: the balabit/syslog-ng docker image stops pulling in logrotate and its dependencies into the image. logrotate recursively pulled in cron and exim4 which are inoperable within the image anyway and causes the image to be larger as well as increasing the potential attack surface.
Debian packaging: logrotate became Suggested instead of Recommended to avoid installing logrotate by default.
scl: To match the way scls are packaged in debian, we have added a syslog-ng-scl package.
This makes it possible to upgrade from the official debian syslog-ng package to the ose-repo provided one.
(#4252) (#4256)
sumologic-http() improvements
Improved defaults: sumologic-http() originally sent incomplete
messages (only the $MESSAGE part) to Sumo Logic by default. The new
default is a JSON object, containing all name-value pairs. This is a
breaking change if you used the default value as it was, but this is not
really anticipated. To override the new message format or revert to the
old default, the template() option can be used.
sumologic-http() enables batching by default to significantly increase
the destination's performance.
The tls() block has become optional, Sumo Logic servers will be
verified using the system's certificate store by default.
(#4124)
Debian/Ubuntu Packages https://github.com/syslog-ng/syslog-ng#debianubuntu
RedHat, CentOS and Fedora Packages https://www.syslog-ng.com/community/b/blog/posts/rpm-packages-from-syslog-ng-git-head/
docker image: Nightly production docker images are now available as balabit/syslog-ng:nightly
(#4117)
docker image: added jemalloc to the production image, which improves performance, decreases memory fragmentation and makes syslog-ng to return memory to the system much more aggressively.
Removed support for Debian stretch.
syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.
Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.
We would like to thank the following people for their contribution:
Andras Mitzki, Attila Szakacs, Attila Szalay, Balazs Scheidler, Bálint Horváth, Gabor Nagy, István Hoffmann, Joshua Root, László Várady, Szilárd Parrag
This is a new major version of syslog-ng, ending the 3.x series which started roughly 13 years ago, on 17th February 2009.
Like all releases in the 3.x series, 4.0.0 is not a breaking change either. Long-term compatibility has been and continues to be an essential objective of syslog-ng; thus, you can still run unchanged configurations that were originally created for syslog-ng 3.0.0.
You can safely upgrade to 4.0.0 if you followed along 3.x, and you should probably also consider upgrading if you are stuck with an older 3.x release.
The new version number primarily indicates that this version of syslog-ng is much more than the software we released 13 years ago. While it does have certain "big-bang" items in its feature list, new features were continuously introduced throughout our 3.x series as well. Our engineering practices have not changed simply because we were working on a new major release: this is the continuation of our previous releases in every respect, produced in the same manner, just with a more catchy version number.
For this reason, there is no separate deprecation or support period for 3.x releases, similarly with our existing practice. We support earlier syslog-ng releases by providing maintenance and fixes in the new release track. Fixes to problems are not backported to earlier releases by the syslog-ng project.
syslog-ng uses a data model where a log message contains an unordered set of name-value pairs. The values stored in these name-value pairs are usually textual, so syslog-ng has traditionally stored these values in text format.
With the increase of JSON-based message sources and destinations, types became more important. If we encounter a message where a name-value pair originates from a JSON document, and this document contains a member that is numeric, we may want to reproduce that as we send this data to a consumer.
For example, sometimes we extract a numerical metric from a log message, and we need to send this to a consumer, again with the correct type.
To be able to do this, we added runtime type information to the syslog-ng message model: each name-value pair becomes a (name, type, value) triplet.
We introduced the following types:
Apart from the syslog-ng core supporting the notion of types, its use is up to the sources, filters, rewrite rules, parsers and destinations that set or make use of them in any way it makes the most sense for the component in question.
syslog-ng uses filter expressions to make routing decisions and during the transformation of messages. These filter expressions are used in filter {} or if {} statements, for example.
In these expressions, you can use comparison operators. This example, for instance, uses the '>' operator to check for HTTP response codes greater-or-equal than 500:
if ("${apache.response}" >= 500) {
};
Earlier, we had two sets of operators, one for numeric (==, !=, <, >) and the other for string-based comparisons (eq, ne, gt, lt).
The separate operators were cumbersome to use. Users often forgot which operator was the right one for a specific case.
Typing allows us to do the right thing in most cases automatically, and a syntax that allows the user to override the automatic decisions in the rare case.
With that, starting with 4.0, the old-numeric operators have been converted to be type-aware operators. It would compare as strings if both sides of the comparisons are strings. It would compare numerically if at least one side is numeric. A great deal of inspiration was taken from JavaScript, which was considered to be a good model, since the problem space is similar.
See this blog post for more details: https://syslog-ng-future.blog/syslog-ng-4-progress-3-38-1-release/
When using json-parser(), syslog-ng converts all members of a JSON object to syslog-ng name-value pairs. Prior to the introduction of type support, these name-value pairs were all stored as strings. Any type information originally present in the incoming JSON object was lost.
This meant that if you regenerated the JSON from the name-value pairs using the $(format-json) template function, all numbers, booleans and other types became strings in the output.
There has been a feature in syslog-ng that alleviated the loss of types. This feature was called "type-hints". Type-hints tell $(format-json) to use a specific type on output, independently of a name-value pair's original type, but this type conversion needed to be explicit in the configuration.
An example configuration that parses JSON on input and produces a JSON on output:
log {
source { ... };
parser { json-parser(prefix('.json.')); };
destination { file(... template("$(format-json .json.*)\n")); };
};
To augment the above with type hinting, you could use:
log {
source { ... };
parser { json-parser(prefix('.json.')); };
destination { file(... template("$(format-json .json.* .json.value=int64(${.json.value})\n")); };
};
NOTE the presence of the int64() type hint in the 2nd example.
The new feature introduced with typing is that syslog-ng would automatically store the JSON type information as a syslog-ng type, thus it will transparently carry over types from inputs to output, without having to be explicit about them.
Typing is a feature throughout syslog-ng, and although the gust of it has been explained in the highlights section, some further details are documented in the list down below:
type-aware comparisons in filter expressions: as detailed above, the previously numeric operators become type-aware, and the exact comparison performed will be based on types associated with the values we compare.
json-parser() and $(format-json): JSON support is massively improved with the introduction of types. For one: type information is retained across input parsing->transformation->output formatting. JSON lists (arrays) are now supported and are converted to syslog-ng lists so they can be manipulated using the $(list-*) template functions. There are other important improvements in how we support JSON.
set(), groupset(): in any case where we allow the use of templates, support for type-casting was added, and the type information is properly promoted.
db-parser() type support: db-parser() gets support for type casts,
<value name="foobar" type="integer">$PID</value>. The “integer” is a type-cast that
associates $foobar with an integer type. db-parser()’s internal parsers
(e.g. @NUMBER@) will also associate type information with a name-value
pair automatically.
add-contextual-data() type support: any new name-value pair that is populated using add-contextual-data() will propagate type information, similarly to db-parser().
map-value-pairs() type support: propagate type information
SQL type support: the sql() driver gained support for types, so that columns with specific types will be stored as those types.
template type support: templates can now be casted explicitly to a specific type, but they also propagate type information from macros/template functions and values in the template string
value-pairs type support: value-pairs form the backbone of specifying a set of name-value pairs and associated transformations to generate JSON or a key-value pair format. It also gained support for types, the existing type-hinting feature that was already part of value-pairs was adapted and expanded to other parts of syslog-ng.
python() typing: support for typing was added to all Python components
(sources, destinations, parsers and template functions), along with more
documentation & examples on how the Python bindings work. All types except
json() are supported as they are queried- or changed by Python code.
on-disk serialized formats (e.g. disk buffer/logstore): we remain compatible with messages serialized with an earlier version of syslog-ng, and the format we choose remains compatible for “downgrades” as well. E.g. even if a new version of syslog-ng serialized a message, the old syslog-ng and associated tools will be able to read it (sans type information of course)
For syslog-ng, everything is traditionally a string. A convention was
started with syslog-ng in v3.10, where a comma-separated format
could be used as a kind of array using the $(list-*) family of template
functions.
For example, $(list-head) takes off the first element in a list, while
$(list-tail) takes the last. You can index and slice list elements using
the $(list-slice) and $(list-nth) functions and so on.
syslog-ng has started to return such lists in various cases, so they can be manipulated using these list-specific template functions. These include the xml-parser(), or the $(explode) template function, but there are others.
Here is an example that has worked since syslog-ng 3.10:
# MSG contains foo:bar:baz
# - the $(list-head) takes off the first element of a list
# - the $(explode) expression splits a string at the specified separator, ':' in this case.
$(list-head $(explode : $MSG))
New functions that improve these features:
JSON arrays are converted to lists, making it a lot easier to slice and extract information from JSON arrays. Of course, $(format-json) will take lists and convert them back to arrays.
The $* is a new macro that converts the internal list of match variables ($1, $2, $3 and so on) to a list, usable with $(list-*) template functions. These match variables have traditionally been filled by regular expressions when a capture group in a regexp matches.
The set-matches() rewrite operation performs the reverse; it assigns the match variables to list elements, making it easier to use list elements in template expressions by assigning them to $1, $2, $3 and so on.
Top-level JSON arrays (e.g. ones where the incoming JSON data is an array and not an object) are now accepted, and the array elements are assigned to the match variables.
syslog-ng has had support for Python-based processing elements since 3.7, released in 2015, which was greatly expanded early 2017 (3.9, LogParser) and late 2018 (3.18, LogSource and LogFetcher).
This support has now been improved in a number of ways to make its use both easier and its potential more powerful.
A framework was added to syslog-ng that allows seamless implementation of
syslog-ng features in Python, with a look and feel of that of a native
implementation. An example for using this framework is available in the
modules/python-modules/example directory, as well as detailed
documentation in the form of modules/python-modules/README.md that is
installed to /etc/syslog-ng/python.
The framework consists of these changes:
syslogng Python package: native code provided by the syslog-ng core
has traditionally been exported in the syslogng Python module. An
effort was made to make these native classes exported by the C layer
more discoverable and more intuitive. As a part of this effort, the
interfaces for all key Python components (LogSource, LogFetcher,
LogDestination, LogParser) were exposed in the syslogng module, along
with in-line documentation.
/etc/syslog-ng/python: syslog-ng now automatically adds this directory to
the PYTHONPATH so that you have an easy place to add Python modules required
by your configuration.
Python virtualenv support for production use: more sophisticated Python
modules usually have 3rd party dependencies, which either needed to be
installed from the OS repositories (using the apt-get or yum/dnf tools) or
PyPI (using the pip tool). syslog-ng now acquired support for an embedded
Python virtualenv (/var/lib/syslog-ng/python-venv or similar, depending on
the installation layout), meaning that these requirements can be installed
privately, without deploying them in the system PYTHONPATH where it might
collide with other applications. The base set of requirements that
syslog-ng relies on can be installed via the syslog-ng-update-virtualenv
script, which has been added to our rpm/deb postinst scripts.
Our mod-python module validates this virtualenv at startup and activates it automatically if the validation is successful. You can disable this behaviour by loading the Python module explicitly with the following configuration statement:
`@module mod-python use-virtualenv(no)`
You can force syslog-ng to use a specific virtualenv by activating it first, prior to executing syslog-ng. In this case, syslog-ng will not try to use its private virtualenv, rather it would use the one activated when it was started. It assumes that any requirements needed for syslog-ng functionality implemented in Python are deployed by the user. These requirements are listed in the /usr/lib/syslog-ng/python/requirements.txt file.
SCL snippets in Python plugins: by adding an scl/whatever.conf file to
your Python-based syslog-ng plugin, you can easily wrap a Python-based
log processing functionality with a syslog-ng block {}, so the user can
use a syntax very similar to native plugins in their main configuration.
confgen in Python: should a simple block {} statement not be enough to
wrap the functionality implemented in Python, the mod-python module now
supports confgen functions to be implemented in Python. confgen
has been a feature in syslog-ng for a long time that allows you to
generate configuration snippets dynamically by executing an external
program or script. This has now been ported to Python, e.g.
syslog-ng can invoke a Python function to generate parts of its
configuration.
Example:
@version: 4.0
python {
from syslogng import register_config_generator
def generate_foobar(args):
print(args)
return "tcp(port(2000))"
#
# this registers a plugin in the "source" context named "foobar"
# which would invoke the generate_foobar() function when a foobar() source
# reference is encountered.
#
register_config_generator("source", "foobar", generate_foobar)
};
log {
# we are actually calling the generate_foobar() function in this
# source, passing all parameters as values in the "args" dictionary
source { foobar(this(is) a(value)); };
destination { file("logfile"); };
};
kubernetes() source and kubernetes-metadata-parser(): these two
components gained the ability to enrich log messages with Kubernetes
metadata. When reading container logs, syslog-ng would query the Kubernetes
API for the following fields and add them to the log-message. The returned
meta-data is cached in memory, so not all log messages trigger a new query.
.k8s.pod_uuid
.k8s.labels.<label_name>
.k8s.annotations.<annotation_name>
.k8s.namespace_name
.k8s.pod_name
.k8s.container_name
.k8s.container_image
.k8s.container_hash
.k8s.docker_id
java() destinations: fixed compatibility with newer Java versions,
syslog-ng is now able to compile up to Java 18.
disk-buffer: Added prealloc() option to preallocate new disk-buffer
files.
(#4056)
disk-buffer: The default value of truncate-size-ratio() has been changed to 1,
which means truncation is disabled by default. This means that by default, the
disk-buffer files will gradually become larger and will never reduce in size.
This improves performance.
(#4056)
log-level(): added a new global option to control syslog-ng's own internal
log level. This augments the existing support for doing the same via the
command line (via -d, -v and -t options) and via syslog-ng-ctl. This change
also causes higher log-levels to include messages from lower log-levels,
e.g. "trace" also implies "debug" and "verbose". By adding this capability
to the configuration, it becomes easier to control logging in containerized
environments where changing command line options is more challenging.
syslog-ng-ctl log-level: this new subcommand in syslog-ng-ctl allows
setting the log level in a more intuitive way, compared to the existing
syslog-ng-ctl verbose|debug|trace -s syntax.
syslog-ng --log-level: this new command line option for the syslog-ng
main binary allows you to set the desired log-level similar to how you
can control it from the configuration or through syslog-ng-ctl.
(#4091)
network/syslog/tls context options: SSL_CONF_cmd support
SSL_CONF_cmd TLS configuration support for network() and syslog() driver has been added.
OpenSSL offers an alternative, software-independent configuration mechanism through the SSL_CONF_cmd interface to support a common solution for setting the so many various SSL_CTX and SSL options that can be set earlier via multiple, separated openssl function calls only. This update implements that similar to the mod_ssl in Apache.
IMPORTANT: The newly introduced openssl-conf-cmds always has the
highest priority, its content parsed last, so it will override any other
options that can be found in the tls() section, does not matter if
they appear before or after openssl-conf-cmds.
As described in the SSL_CONF_cmd documentation, the order of operations within openssl-conf-cmds() is significant and the commands are executed in top-down order. This means that if there are multiple occurrences of setting the same option then the 'last wins'. This is also true for options that can be set multiple ways (e.g. used cipher suites and/or protocols).
Example config:
source source_name {
network (
ip(0.0.0.0)
port(6666)
transport("tls")
tls(
ca-dir("/etc/ca.d")
key-file("/etc/cert.d/serverkey.pem")
cert-file("/etc/cert.d/servercert.pem")
peer-verify(yes)
openssl-conf-cmds(
# For system wide available cipher suites use: /usr/bin/openssl ciphers -v
# For formatting rules see: https://www.openssl.org/docs/man1.1.1/man3/SSL_CONF_cmd.html
# For quick and dirty testing try: https://github.com/rbsec/sslscan
#
"CipherString" => "ECDHE-RSA-AES128-SHA", # TLSv1.2 and bellow
"CipherSuites" => "TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384", # TLSv1.3+ (OpenSSl 1.1.1+)
"Options" => "PrioritizeChaCha",
"Protocol" => "-ALL,TLSv1.3",
)
)
);
};
network/syslog/http destination: OCSP stapling support
OCSP stapling support for network destinations and for the http() module has been added.
When OCSP stapling verification is enabled, the server will be requested to send back OCSP status responses.
This status response will be verified using the trust store configured by the user (ca-file(), ca-dir(), pkcs12-file()).
Note: RFC 6961 multi-stapling and TLS 1.3-provided multiple responses are currently not validated, only the peer certificate is verified.
Example config:
destination {
network("test.tld" transport(tls)
tls(
pkcs12-file("/path/to/test.p12")
peer-verify(yes)
ocsp-stapling-verify(yes)
)
);
http(url("https://test.tld") method("POST") tls(peer-verify(yes) ocsp-stapling-verify(yes)));
};
(#4082)
Python LogMessage class: get_pri() and get_timestamp() methods were added that
allow the query of the syslog-style priority and the message timestamp,
respectively. The return value of get_pri() is an integer, while
get_timestamp() returns a Python datetime.datetime instance. Some macros
that were previously unavailable from Python (e.g. the STAMP, R_STAMP and
C_STAMP macros) are now made available.
Python Logger: the low-level Logger class exported by syslog-ng was
wrapped by a logging.LogHandler class so that normal Python APIs for logging
can now be used.
db-parser() and grouping-by(): added a prefix() option to both
db-parser() and grouping-by() that allows specifying an extra prefix
to be prepended to all name-value pairs that get extracted from messages
using patterns or
csv-parser(): add a new dialect, called escape-backslash-with-sequences
which uses "" as an escape character but also supports C-style escape
sequences, like "\n" or "\r".
tcp(), network() or syslog() destinations: fixed a crash that could
happen after reload when a kept-alive connection is terminated, in case
the target server is configured using a hostname (and not an IP address)
and that name becomes unresolvable (e.g. dropped from DNS or /etc/hosts)
(#4044)
python() destination: Fixed a crash, when trying to resolve the
"R_STAMP", "P_STAMP" or "STAMP" macros from Python code.
(#4057)
Python LogSource & LogFetcher: a potential deadlock was fixed in
acknowledgement tracking.
Python LogTemplate: the use of template functions in templates
instantiated from Python caused a crash, which has been fixed.
grouping-by() persist-name() option: fixed a segmentation fault in the
grammar.
(#4180)
$(format-json): fix a bug in the --key-delimiter option introduced in
3.38, which causes the generated JSON to contain multiple values for the
same key in case the key in question contains a nested object and
key-delimiter specified is not the dot character.
(#4127)
add-contextual-data(): add compatibility warnings and update advise in
case of the value field of the add-contextual-data() database contains an
expression that resembles the new type-hinting syntax: type(value).
syslog-ng --help screen: the output for the --help command line option has
included sample paths to various files that contained autoconf style
directory references (e.g. ${prefix}/etc for instance). This is now fixed,
these paths will contain the expanded path. Fixes Debian Bug report #962839:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962839
(#4143)
csv-parser(): fixed the processing of the dialect() parameter, which was
not taken into consideration.
apache-accesslog-parser(): Apache may use backslash-style escapes in the
request field, so support it by setting the csv-parser() dialect to
escape-backslash-with-sequences. Also added validation that the
rawrequest field contains a valid HTTP request and only extract verb,
request and httpversion if this is the case.
riemann: fixed severity levels of Riemann diagnostics messages, the error
returned by riemann_communicate() was previously only logged at the trace
level and was even incomplete: not covering the case where
riemann_communicate() returns NULL.
(#4238)
python: python2 support is now completely removed. syslog-ng can no
longer be configured with --with-python=2.
(#4057)
python: Python 2 support is now completely removed from the syslog-ng
functional test framework, called Light, too. Light will support only Python 3
from now.
(#4174)
Python virtualenv support for development use: syslog-ng is now capable of using a build-time virtualenv, where all Python development tools are automatically deployed by the build system. You can control if you want to use this using the --with-python-packages configure option. There are three possible values for this parameter:
venv: denoting that you want to use the virtualenv and install
all these requirements automatically using pip, into the venv.system: meaning that you want to rely on the system Python
without using a virtualenv. syslog-ng build scripts would install
requirements automatically to the system Python path usually
/usr/local/lib/pythonX.Y
none: disable deploying packages automatically. All
dependencies are assumed to be present in the system Python before
running the syslog-ng build process.Please note that syslog-ng has acquired quite a number of these
development time dependencies with the growing number of functionality
the Python binding offers, so using the system or none settings are
considered advanced usage, meant to be used for distro packaging.
make dist: fixed make dist of FreeBSD so that source tarballs can
easily be produced even if running on FreeBSD.
(#4163)
Debian and derivatives: The syslog-ng-mod-python package is now built with python3 on the following platforms:
debian-stretch
debian-buster
ubuntu-bionic
(#4057)dbld: Removed support for ubuntu-xenial.
(#4057)
dbld: Updated support from Fedora 35 to Fedora 37
Leaner production docker image: the balabit/syslog-ng docker image stops pulling in logrotate and its dependencies into the image. logrotate recursively pulled in cron and exim4 which are inoperable within the image anyway and causes the image to be larger as well as increasing the potential attack surface.
Debian packaging: logrotate became Suggested instead of Recommended to avoid installing logrotate by default.
sumologic-http() improvements
Improved defaults: sumologic-http() originally sent incomplete
messages (only the $MESSAGE part) to Sumo Logic by default. The new
default is a JSON object, containing all name-value pairs. This is a
breaking change if you used the default value as it was, but this is not
really anticipated. To override the new message format or revert to the
old default, the template() option can be used.
sumologic-http() enables batching by default to significantly increase
the destination's performance.
The tls() block has become optional, Sumo Logic servers will be
verified using the system's certificate store by default.
(#4124)
Debian/Ubuntu Packages https://github.com/syslog-ng/syslog-ng#debianubuntu
RedHat, CentOS and Fedora Packages https://www.syslog-ng.com/community/b/blog/posts/rpm-packages-from-syslog-ng-git-head/
docker image: Nightly production docker images are now available as balabit/syslog-ng:nightly
(#4117)
docker image: added jemalloc to the production image, which improves performance, decreases memory fragmentation and makes syslog-ng to return memory to the system much more aggressively.
Removed support for Debian stretch.
syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.
Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.
We would like to thank the following people for their contribution:
Andras Mitzki, Attila Szakacs, Attila Szalay, Balazs Scheidler, Bálint Horváth, Gabor Nagy, István Hoffmann, Joshua Root, László Várady, Szilárd Parrag
syslog-ng v4.0 is right around the corner.
This release (v3.38.1) contains all major changes, however, they are
currently all hidden behind a feature flag.
To enable and try those features, you need to specify @version: 4.0 at the
top of the configuration file.
You can find out more about the 4.0 changes and features here.
Read our practical introduction to typing at syslog-ng-future.blog.
grouping-by(): added inject-mode(aggregate-only)
This inject mode will drop individual messages that make up the correlation
context (key() groups) and would only yield the aggregate messages
(e.g. the results of the correlation).
(#3998)
add-contextual-data(): add support for type propagation, e.g. set the
type of name-value pairs as they are created/updated to the value returned
by the template expression that we use to set the value.
The 3rd column in the CSV file (e.g. the template expression) now supports specifying a type-hint, in the format of "type-hint(template-expr)".
Example line in the CSV database:
selector-value,name-value-pair-to-be-created,list(foo,bar,baz) (#4051)
$(format-json): add --key-delimiter option to reconstruct JSON objects
using an alternative structure separator, that was created using the
key-delimiter() option of json-parser().
(#4093)
json-parser(): add key-delimiter() option to extract JSON structure
members into name-value pairs, so that the names are flattened using the
character specified, instead of dot.
Example: Input: {"foo":{"key":"value"}}
Using json-parser() without key-delimiter() this is extracted to:
foo.key="value"
Using json-parser(key-delimiter("~")) this is extracted to:
foo~key="value"
This feature is useful in case the JSON keys contain dots themselves, in those cases the syslog-ng representation is ambigious. (#4093)
Fixed buffer handling of syslog and timestamp parsers (CVE-2022-38725)
Multiple buffer out-of-bounds issues have been fixed, which could cause hangs, high CPU usage, or other undefined behavior. (#4110)
Fixed building with LibreSSL (#4081)
network(): Fixed a bug, where syslog-ng halted the input instead of skipping a character
in case of a character conversion error.
(#4084)
redis(): Fixed bug where using redis driver without the batch-lines option caused program crash.
(#4114)
pdbtool: fix a SIGABRT on FreeBSD that was triggered right before pdbtool
exits. Apart from being an ugly crash that produces a core file,
functionally the tool behaved correctly and this case does not affect
syslog-ng itself.
(#4037)
regexp-parser(): due to a change introduced in 3.37, named capture groups
are stored indirectly in the LogMessage to avoid copying of the value. In
this case the name-value pair created with the regexp is only stored as a
reference (name + length of the original value), which improves performance
and makes such name-value pairs use less memory. One omission in the
original change in 3.37 is that syslog-ng does not allow builtin values to
be stored indirectly (e.g. $MESSAGE and a few of others) and this case
causes an assertion to fail and syslog-ng to crash with a SIGABRT. This
abort is now fixed. Here's a sample config that reproduces the issue:
regexp-parser(patterns('(?<MESSAGE>.*)'));
(#4043)
set-tag: fix cloning issue when string literal were used (see #4062) (#4065)
add-contextual-data(): fix high memory usage when using large CSV files
(#4067)
The json-c library is no longer bundled in the syslog-ng source tarball
Since all known OS package managers provide json-c packages nowadays, the json-c submodule has been removed from the source tarball.
The --with-jsonc=internal option of the configure script has been removed
accordingly, system libraries will be used instead. For special cases, the JSON
support can be disabled by specifying --with-jsonc=no.
(#4078)
platforms: Dropped support for ubuntu-impish as it became EOL (#4088)
syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.
Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.
We would like to thank the following people for their contribution:
Alvin Šipraga, Andras Mitzki, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Daniel Klauer, Fabrice Fontaine, Gabor Nagy, HenryTheSir, László Várady, Parrag Szilárd, Peter Kokai, Shikhar Vashistha, Szilárd Parrag, Vivin Peris
kubernetes source: A new source for Kubernetes CRI (Container Runtime Interface) format.
By default it tails the /var/log/containers folder which can be overriden with the base-dir() parameter.
Example configuration:
source {
kubernetes();
# or specifying the directory:
# kubernetes(base-dir("/dir/to/tail"));
};
mariadb-audit-parser: A new parser for mariadb/mysql audit plugin logs have been added.
The parser supports the syslog output type's format, see mariadb page for details.
(#3947)internal(): add rcptid tag to all trace messages that relate to incoming
log messages. This makes it easier to correlate parsing, rewriting and
routing actions with incoming log messages.
(#3972)
syslog-parser(): allow comma (e.g. ',') to separate the seconds and the fraction of a
second part as some devices use that character. This change applies to both
to syslog-parser() and the builtin syslog parsing functionality of network
source drivers (e.g. udp(), tcp(), network() and syslog()).
(#3949)
cisco-parser: add ISO 8601 timestamp support
(#3934)
network(), syslog() sources and destinations: added new TLS options sigalgs() and client-sigalgs()
They can be used to restrict which signature/hash pairs can be used in digital signatures. It sets the "signature_algorithms" extension specified in RFC5246 and RFC8446.
Example configuration:
destination {
network("test.host" port(4444) transport(tls)
tls(
pkcs12-file("/path/to/tls/test.p12")
peer-verify(yes)
sigalgs("RSA-PSS+SHA256:ed25519")
)
);
};
(#4000)
set-matches() and unset-matches(): these new rewrite operations allow
the setting of match variables ($1, $2, ...) in a single operation, based
on a syslog-ng list expression.
Example:
# set $1, $2 and $3 respectively
set-matches("foo,bar,baz");
# likewise, but using a list function
set-matches("$(explode ':' 'foo:bar:baz')");
(#3948)
$* macro: the $* macro in template expressions convert the match variables
(e.g. $1, $2, ...) into a syslog-ng list that can be further manipulated
using the list template functions, or turned into a list in type-aware
destinations.
(#3948)
set-tag(): add support for using template expressions in set-tag() rewrite
operations, which makes it possible to use tag names that include macro
references.
(#3962)
http() and other threaded destinations: fix $SEQNUM processing so that
only local messages get an associated $SEQNUM, just like normal
syslog()-like destinations. This avoids a [meta sequenceId="XXX"] SD-PARAM
being added to $SDATA for non-local messages.
(#3928)grouping-by(): fix grouping-by() use through parser references.
Originally if a grouping-by() was part of a named parser statement and was
referenced from multiple log statements, only the first grouping-by()
instance behaved properly, 2nd and subsequent references were ignoring all
configuration options and have reverted using defaults instead.
(#3957)db-parser(): similarly to grouping-by(), db-parser() also had issues
propagating some of its options to 2nd and subsequent references of a parser
statement. This includes drop-unmatched(), program-template() and
template() options.
(#3957)match(), subst() and regexp-parser(): fixed storing of numbered
(e.g. $1,$2, $3 and so on) and named capture groups in regular expressions
in case the input of the regexp is the same as one of the match variables being
stored. In some cases the output of the regexp was clobbered and an invalid
value stored.
(#3948)threaded(no) related crash: if threaded mode is disabled for
asynchronous sources and destinations (all syslog-like drivers such as
tcp/udp/syslog/network qualify), a use-after-free condition can happen due
to a reference counting bug in the non-threaded code path. The
threaded(yes) setting has been the default since 3.6.1 so if you are using
a more recent version, you are most probably unaffected. If you are using
threaded(no) a use-after-free condition happens as the connection closes.
The problem is more likely to surface on 32 bit platforms due to pointer
sizes and struct layouts where this causes a NULL pointer dereference.
(#3997)set(): make sure that template formatting options (such as time-zone() or
frac-digits()) are propagated to all references of the rewrite rule
containing a set(). Previously the clone() operation used to implement
multiple references missed the template related options while cloning set(),
causing template formatting options to be set differently, depending on
where the set() was referenced from.
(#3962)csv-parser(): fix flags(strip-whitespace) and null-value handling
for greedy column
(#4028)java()/python() destinations: the $SEQNUM macro (and "seqnum" attribute in
Python) was erroneously for both local and non-local logs, while it should
have had a value only in case of local logs to match RFC5424 behavior
(section 7.3.1). This bug is now fixed, but that means that all non-local
logs will have $SEQNUM set to zero from this version on, e.g. the $SEQNUM
macro would expand to an string, to match the syslog() driver behaviour.
(#3928)dbld: add support for Fedora 35 in favour of Fedora 33
(#3933)$(md4)) is no longer available when compiling syslog-ng with OpenSSL v3.0.
MD4 is now deprecated, it will be removed completely in future versions.
(#4012)syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.
Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.
We would like to thank the following people for their contribution:
Andras Mitzki, Attila Szakacs, Balazs Scheidler, Ben Burrows, Fᴀʙɪᴇɴ Wᴇʀɴʟɪ, Gabor Nagy, László Várady, mohitvaid, Parrag Szilárd, Peter Kokai, Peter Viskup, Roffild, Ryan Faircloth, Scott Parlane, Zoltan Pallagi
system() source: added basic support for reading macOS system logs
The current implementation processes the output of the original macOS syslogd:
/var/log/system.log.
(#3710)
$(values) and $(names): these new template functions can be used to
query a list of name-value pairs in the current message. The list of name
value pairs queried are specified by a value-pairs expression, just like
with $(format-json).
Examples:
This expression sets the JSON array values to contain the list of SDATA
values, while the JSON array names would contain the associated names, in
the same order.
$(format-json values=list($(values .SDATA.*)) names=list($(names .SDATA.*)))
The resulting name-value pairs are always sorted by their key, regardless of the argument order. (#3911)
rename(): added a new rewrite rule, called rename()
Example usage:
rewrite {
rename( "renamed-from" "renamed-to" );
};
(#3841)
network() drivers: added TLS keylog support
syslog-ng dumps TLS secrets for a given source/destination, which can be used for debugging purposes to decrypt data with, for example, Wireshark.
This should be used for debugging purposes only!
Example usage:
source tls_source{
network(
port(1234)
transport("tls"),
tls(
key-file("/path/to/server_key.pem"),
cert-file("/path/to/server_cert.pem"),
ca-dir("/path/to/ca/")
keylog-file("/path/to/keylog_file")
)
);
};
(#3792)
tls() block: added option for restricting TLS 1.3 ciphers
The network(), syslog(), and the http() modules now support specifying TLS 1.3 cipher suites,
Example usage:
network(
transport("tls")
tls(
pkcs12-file("test.p12")
cipher-suite(
tls12-and-older("ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256"),
tls13("TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384")
)
)
);
tls12-and-older() can be used to specify TLS v1.2-and-older ciphers,
tls13() can be used for TLS v1.3 ciphers only.
Note: The old cipher-suite("list:of:ciphers") option restricts only the TLS v1.2-and-older cipher suite
for backward compatibility.
(#3907)
file() destination: added a new option: symlink-as()
This feature allows one to maintain a persistent symlink to a log file when a
template is used (for example: /var/log/cron -> /var/log/cron.${YEAR}${MONTH}).
Example usage:
destination d_file_cron {
file("/var/log/cron.${YEAR}${MONTH}" symlink-as("/var/log/cron"));
};
From a functional perspective, the symlink-as file inherits both
create-dirs and file ownership from its file destination (permissions are not
applicable to symlinks, at least on linux).
The symlink is adjusted at the time a new destination file is opened (in the
example above, if ${YEAR} or ${MONTH} changes).
Although not specific to time macros, that's where the usefulness is. If the
template contains something like ${PROGRAM} or ${HOST}, the configuration wouldn't
necessarily be invalid, but you'd get an ever-changing symlink of dubious
usefulness.
(#3855)
flags(no-rfc3164-fallback): added a new flag to sources that parse
incoming syslog data and operate in RFC5424 mode (e.g. syslog-protocol is
also set). With the new flag the automatic fallback to RFC3164 format
is disabled. In this case if the parsing in RFC5424 fails, the
syslog parser would result in an error message. In the case of
syslog-parser(drop-invalid(yes)), the message would be dropped.
(#3891)
syslog-format: accept ISO timestamps that incorrectly use a space instead of
a 'T' to delimit the date from the time portion. For example, a
"2021-01-01T12:12:12" timestamp is well formed according to RFC5424 (which
uses a subset of ISO8601, see https://datatracker.ietf.org/doc/html/rfc5424#section-6.2.3).
Some systems simply use a space instead of a 'T'. The same format is
accepted for both RFC3164 (e.g. udp(), tcp() and network() sources) and
RFC5424 (e.g. syslog() source).
(#3893)
transport(text-with-nuls): added a new transport mechanism for
the network() driver that allows NUL characters within the message.
Note: syslog-ng does not support embedded NUL characters everywhere, so it is
recommended that you also use flags(no-multi-line) that causes NUL
characters to be replaced by space.
(#3913)
filter: fixed the not operator in filter expressions (regression in v3.35.1)
Reusing a filter that contains the not operator more than once, or
referencing a complex expression containing not might have caused invalid results
in the previous syslog-ng version (v3.35.1). This has been fixed.
(#3863)
throttle() filter: support negation
(#3863)
disk-buffer(): fixed a crash which could happen in very rare cases, while a corrupted disk-buffer was getting replaced
(#3845)
disk-buffer(): fixed a memory leak issue and inconsistent buffer handling in rare cases
(#3887)
disk-buffer(): fixed underflowing queued stats counter
(#3887)
disk-buffer(): fixed queued stats were not adjusted when a disk-buffer became corrupt
(#3851)
disk-buffer(): fixed a disk-buffer corruption issue
A completely filled and then emptied disk-buffer may have been recognised as corrupt. (#3874)
amqp(): fixed a minor error reporting problem.
(#3869)
amqp(): syslog-ng now drops messages that are too large to send
(#3869)
amqp(): fixed a crash, which happened with librabbitmq v0.9.0 or v0.10.0, while using the tls() block.
(#3929)
file() source: fixed invalid buffer handling when encoding() is used
A bug has been fixed that - under rare circumstances - could cause message duplication or partial message loss when non-fixed length or less known fixed-length encodings are used. (#3892)
syslog-ng: fixed a SIGSEGV triggered by an incorrectly formatted "CONFIG"
command, received on the syslog-ng control socket. The only known
implementation of the control protocol is syslog-ng-ctl itself, which always
sends a correct command, but anyone with access to the UNIX domain socket
syslog-ng.ctl (root only by default) can trigger a crash.
(#3900)
credit-card-mask(): fixed visa, mastercard and jcb card regex pattern
(#3853)
cisco-parser(): allow a leading dot in the timestamp (not synced clocks)
(#3843)
plugins: we have made it easier to implement filter plugins
An example can be found under modules/rate-limit-filter.
(#3866)
dev-utils: various fixes for the plugin skeleton generator script (#3866)
throttle() filter: renamed to rate-limit()
(#3866)python: support Python 3.10
(#3865)java: upgraded from old log4j v1.x line to log4j v2.17.2
(#3861)
(#3927)syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.
Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.
We would like to thank the following people for their contribution:
Andras Mitzki, Andrea Biardi, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Gabor Nagy, Janos SZIGETVARI, Laszlo Budai, Laszlo Szemere, László Várady, Mikel Olasagasti Uranga, Norbert Takacs, Parrag Szilárd, Peter Kokai, Szilárd Parrag, Zoltan Pallagi, Stanislav Osipov, Yash Mathne
From now on, Ubuntu and Debian packages will be published with every syslog-ng release in the form of an APT repository.
We, syslog-ng developers, provide these packages and the APT repository "as is" without warranty of any kind, on a best-effort level.
Currently, syslog-ng packages are released for the following distribution versions (x86-64):
For instructions on how to install syslog-ng on Debian/Ubuntu distributions, see the README.
throttle(): added a new filter that allows rate limiting messages based on arbitrary keys in each message.
Note: messages over the rate limit are dropped (just like in any other filter).
filter f_throttle {
throttle(
template("$HOST")
rate(5000)
);
};
(#3781)
mqtt(): added a new source that can be used to receive messages using the MQTT protocol.
Supported transports: tcp, ws, ssl, wss
Example config:
source {
mqtt{
topic("sub1"),
address("tcp://localhost:4445")
};
};
(#3809)
afsocket: Socket options, such as ip-ttl() or tcp-keepalive-time(), are
traditionally named by their identifier defined in socket(7) and unix(7) man
pages. This was not the case with the pass-unix-credentials() option, which -
unlike other similar options - was also possible to set globally.
A new option called so-passcred() is now introduced, which works similarly how other socket related options do, which also made possible a nice code cleanup in the related code sections. Of course the old name remains supported in compatibility modes.
The PR also implements a new source flag ignore-aux-data, which causes
syslog-ng not to propagate transport-level auxiliary information to log
messages. Auxiliary information includes for example the pid/uid of the
sending process in the case of UNIX based transports, OR the X.509
certificate information in case of SSL/TLS encrypted data streams.
By setting flags(ignore-aux-data) one can improve performance at the cost of making this information unavailable in the log messages received through affected sources. (#3670)
network: add support for PROXY header before TLS payload
This new transport method called proxied-tls-passthrough is capable of detecting the
PROXY header before the TLS payload.
Loggen has been updated with the--proxied-tls-passthrough option for testing purposes.
source s_proxied_tls_passthrough{
network(
port(1234)
transport("proxied-tls-passthrough"),
tls(
key-file("/path/to/server_key.pem"),
cert-file("/path/to/server_cert.pem"),
ca-dir("/path/to/ca/")
)
);
};
(#3770)
mqtt() destination: added client-id option. It specifies the unique client ID sent to the broker.
(#3809)
unset(), groupunset(): fix unwanted removal of values on different log paths
Due to a copy-on-write bug, unset() and groupunset() not only removed values
from the appropriate log paths, but from all the others where the same message
went through. This has been fixed.
(#3803)
regexp-parser(): fix storing unnamed capture groups under prefix()
(#3810)
loggen: cannot detect plugins on platforms with non .so shared libs (osx)
(#3832)
debian/control: Added libcriterion-dev as a build dependency, where it is available from APT.
(debian-bullseye, debian-testing, debian-sid)
(#3794)
centos-7: kafka and mqtt modules are now packaged.
The following packages are used as dependencies:
librdkafka-devel from EPEL 7paho-c-devel from copr:copr.fedorainfracloud.org:czanik:syslog-ng-githead
(#3797)debian: Added bullseye support.
(#3794)
bison: support build with bison 3.8
(#3784)
dbld: As new distributions use python3 by default it makes sense to explicitly state older platforms which use python2
instead of the other way around, so it is not necessary to add that new platform to the python3 case.
(#3780)
dbld: move dbld image cache from DockerHub to GitHub
In 2021, GitHub introduced the GitHub Packages service. Among other repositories - it provides a standard Docker registry. DBLD uses this registry, to avoid unnecessary rebuilding of the images. (#3782)
syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.
Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.
We would like to thank the following people for their contribution:
Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Colin Douch, Gabor Nagy, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Czanik (CzP), Peter Kokai, Robert Paschedag, Ryan Faircloth, Szilárd Parrag, Thomas Klausner, Zoltan Pallagi
regexp-parser(): new parser that can parse messages with regular expressions
Example:
regexp-parser(
template("${MESSAGE}")
prefix(".regexp.")
patterns("(?<DN>foo)", "(?<DN>ball)")
);
regexp-parser() can be used as an intuitive replacement for regexp filters
that had their store-matches flag set in order to save those matches.
(#3702)
redis(): workers() and batching support
The Redis driver now support the workers() option, which specifies the
number of parallel workers, and the batch-lines() option.
This could drastically increase the throughput of the Redis destination driver.
Example:
redis(
host("localhost")
port(6379)
command("HINCRBY", "hosts", "$HOST", "1")
workers(8)
batch-lines(100)
batch-timeout(10000)
log-fifo-size(100000)
);
mqtt(): TLS and WebSocket Secure support
The MQTT destination now supports TLS and WSS.
Example config:
mqtt(
address("ssl://localhost:8883")
topic("syslog/$HOST")
fallback-topic("syslog/fallback")
tls(
ca-file("/path/to/ca.crt")
key-file("/path/to/client.key")
cert-file("/path/to/client.crt")
peer-verify(yes)
)
);
(#3747)
system() source: added support for NetBSD
(#3761)
stats: new statistics counter
The following statistics are now available for the HTTP destination, and other file and network based sources/destinations:
msg_size_max/msg_size_avg: Shows the largest/average message size of the given source/destination that has
been measured so far.
batch_size_max/batch_size_avg: When batching is enabled, then this shows the
largest/average batch size of the given source/destination that has been measured so far.
eps_last_1h, eps_last_24h, eps_since_start: Events per second, measured for the last hour,
for the last 24 hours, and since syslog-ng startup, respectively.
Notes:
mqtt(): username/password authentication
Example config:
mqtt(
address("tcp://localhost:1883")
topic("syslog/messages")
username("user")
password("passwd")
);
Note: The password is transmitted in cleartext without using ssl:// or wss://.
(#3747)
mqtt(): new option http-proxy() for specifying HTTP/HTTPS proxy for WebSocket connections
(#3747)
syslog-ng-ctl: new flag for pruning statistics
syslog-ng-ctl stats --remove-orphans can be used to remove "orphaned" statistic counters.
It is useful when, for example, a templated file destination ($YEAR.$MONTH.$DAY) produces a lot of stats,
and one wants to remove those abandoned counters occasionally/conditionally.
(#3760)
disk-buffer(): added a new option to reliable disk-buffer: qout-size().
This option sets the number of messages that are stored in the memory in addition to storing them on disk. The default value is 1000.
This serves performance purposes and offers the same no-message-loss guarantees as before.
It can be used to maintain a higher throughput when only a small number of messages are waiting in the disk-buffer. (#3754)
network(), syslog(): fixed network sources on NetBSD
On NetBSD, TCP-based network sources closed their listeners shortly after startup due to a non-portable TCP keepalive setting. This has been fixed. (#3751)
disk-buffer(): fixed a very rare case, where the reliable disk-buffer never resumed
after triggering flow-control.
(#3752)
disk-buffer(): fixed a rare memory leak that occurred when mem-buf-length()
or mem-buf-size() was configured incorrectly
(#3750)
redis(): fixed command errors that were not detected and marked as successful delivery
(#3748)
Light framework: new proxy-related options are supported with loggen:
--proxy-src-ip, --proxy-dst-ip, --proxy-src-port, --proxy-dst-port
(#3766)
log-threaded-dest: descendant drivers from LogThreadedDest no longer inherit
batch-lines() and batch-timeout() automatically. Each driver have to opt-in for
these options with log_threaded_dest_driver_batch_option.
log_threaded_dest_driver_option has been renamed to log_threaded_dest_driver_general_option,
and log_threaded_dest_driver_workers_option have been added similarly to the
batch-related options.
(#3741)
disk-buffer(): performance improvements
Based on our measurements, the following can be expected compared to the previous syslog-ng release (v3.33.1):
disk-buffer(): the default value of the following options has been changed for performance reasons:
truncate-size-ratio(): from 0.01 to 0.1 (from 1% to 10%)qout-size(): from 64 to 1000 (this affects only the non-reliable disk buffer)
(#3757)kafka-c(): properties-file() option is removed
Please list librdkafka properties in the config() option in syslog-ng's configuration.
See librdkafka configuration here.
(#3704)
syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.
Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.
We would like to thank the following people for their contribution:
Andras Mitzki, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Fabrice Fontaine, Gabor Nagy, Laszlo Szemere, LittleFish33, László Várady, Norbert Takacs, Parrag Szilárd, Peter Czanik, Peter Kokai, Zoltan Pallagi
disk-buffer: fixed a bug, which was introduced in 3.33.1, where we
sometimes corrupted the disk-buffer file when it reached full size.
(#3726)syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.
Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.
We would like to thank the following people for their contribution:
Andras Mitzki, Antal Nemes, Attila Szakacs, Balázs Barkó, Balazs Scheidler, Benedek Cserhati, Gabor Nagy, Josef Schlehofer, Laszlo Budai, Laszlo Szemere, László Várady, Norbert Takacs, Parrag Szilárd, Peter Kokai, Zoltan Pallagi
MQTT destination
The new mqtt() destination can be used to publish messages using the MQTT protocol.
Currently MQTT 3.1.1 and 3.1 are supported.
Supported transports: tcp, ws.
Example config:
destination {
mqtt(
address("tcp://localhost:1883"),
topic("syslog/$HOST"),
fallback-topic("syslog/fallback")
);
};
Note: MQTT 5.0 and TLS (ssl://, wss://) are currently not supported.
(#3703)
discord() destination
syslog-ng now has a webhook-based Discord destination. Example usage:
destination {
discord(url("https://discord.com/api/webhooks/x/y"));
};
The following options can be used to customize the destination further:
avatar-url(), username("$HOST-bot"), tts(true), template("${MSG:-[empty message]}").
(#3717)
kafka-c: batching support in case of sync-send(yes)
kafka-c(
bootstrap-server("localhost:9092")
topic("syslog-ng")
sync-send(yes)
batch-lines(10)
batch-timeout(10000)
);
Note1: batch-lines are accepted in case of sync-send(no), but no batching is done. Note2: messages are still sent one at a time to kafka, the batch yields multiple message per transaction. (#3699)
kafka-c: sync-send(yes) enables synchronous message delivery, reducing the possibility of message loss.
kafka-c(
bootstrap-server("localhost:9092")
topic("syslog-ng")
sync-send(yes)
);
Warning: this option also reduces significantly the performance of kafka-c driver. (#3681)
disk-buffer: Now we optimize the file truncating frequency of disk-buffer.
The new behavior saves IO time, but loses some disk space, which is configurable with a new option. The new option in the config is settable at 2 places:
truncate-size-ratio() in the disk-buffer() block, which affects the given disk-buffer.disk-buffer(truncate-size-ratio()) in the global options block, which affects every disk-buffer
which did not set truncate-size-ratio() itself.
The default value is 0.01, which operates well with most disk-buffers.If the possible size reduction of the truncation does not reach truncate-size-ratio() x disk-buf-size(),
we do not truncate the disk-buffer.
To completely turn off truncating (maximal disk space loss, maximal IO time saved) set truncate-size-ratio(1),
or to mimic the old behavior (minimal disk space loss, minimal IO time saved) set truncate-size-ratio(0).
(#3689)
syslog-format: fixing the check-hostname(yes|no) option
The check-hostname(yes|no) option detected every value as invalid, causing a parse error when enabled. (#3690)
disk-buffer(): fix crash when switching between disk-based and memory queues
When a disk-buffer was removed from the configuration and the new config was applied by reloading syslog-ng, a crash occurred. (#3700)
logpath: Fixed a message write protection bug, where message modifications (rewrite rules, parsers, etc.)
leaked through preceding path elements. This may have resulted not only in unwanted/undefined message modification,
but in certain cases crash as well.
(#3708)
mongodb(): fix crash with older mongo-c-driver versions
syslog-ng crashed (was aborted) when the mongodb() destination was used with
older mongo-c-driver versions (< v1.11.0).
(#3677)
java(): fix debug logging of Java-based destinations
Java debug logging was not enabled previously when syslog-ng was started in debug/trace mode. This has been fixed. (#3679)
kafka-c: fixed a hang during shutdown/reload, when multiple workers is used (workers() option is set to 2 or higher) and the librdkafka internal queue is filled.
(error message was kafka: failed to publish message; topic='test-topic', error='Local: Queue full')
(#3711)
smtp(): libesmtp is now detected via pkg-config
(#3669)syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.
Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.
We would like to thank the following people for their contribution:
Andras Mitzki, Antal Nemes, Attila Szakacs, Balazs Scheidler, Balázs Barkó, Benedek Cserhati, Gabor Nagy, L4rS6, Laszlo Budai, Laszlo Szemere, LittleFish33, László Várady, Norbert Takacs, Peter Czanik, Peter Kokai, Todd C. Miller, Tomáš Mózes, Zoltan Pallagi
mongodb(): add workers() support (multi-threaded connection pooling)
The MongoDB driver now supports the workers() option, which specifies the
number of parallel workers to be used.
Workers are based on the connection pooling feature of the MongoDB C library.
This increases the throughput of the MongoDB destination driver.
Example:
destination {
mongodb(
uri("mongodb://hostA,hostB/syslog?replicaSet=my_rs&wtimeoutMS=10000&socketTimeoutMS=10000&connectTimeoutMS=10000&serverSelectionTimeoutMS=5000")
collection("messages")
workers(8)
);
};
(#3621)
mongodb(): template support for the collection() option
The collection() option of the MongoDB destination driver now accepts
templates, for example:
destination {
mongodb(
uri("mongodb://host/syslog")
collection("${HOST}_messages")
);
};
(#3621)
time-reopen: Support the time-reopen() option on the driver level for the following drivers:
example-diskq-source, python-fetcher
amqp, example-destination, file, http, mongodb, network, pipe,
program, pseudofile, python, redis, riemann, smtp, sql, stomp,
syslog, tcp, tcp6, udp, udp6, unix-dgram, unix-stream, usertty
(#3585)csv-parser(): add drop-invalid() option along with the already existing
flag with the same name. This is to improve the consistency of the
configuration language.
(#3547)usertty() destination: Support changing the terminal disable timeout with the time-reopen() option.
Default timeout change to 60 from 600. If you wish to use the old 600 timeout, add time-reopen(600)
to your config in the usertty() driver.
(#3585)syslog-parser(): add a new drop-invalid() option that allows the use of
syslog-parser() in if statements. Normally a syslog-parser() injects an
error message instead of failing.
(#3565)date-parser: if the timestamp pattern did not covered a field (for example seconds) that field had undefined value
The missing fields are initialized according to the following rules:
Fix compilation issues on OpenBSD
syslog-ng can now be compiled on OpenBSD. (#3661)
loggen: debug message printed wrong plugin name (ssl-plugin instead of socket_plugin) (#3624)
tls: fixup EOF detection issue in tls (before 3.0 version)
syslog-ng error message: "I/O error occurred while reading; fd='13', error='Success (0)'" (#3618)
kafka: the config() block couldn't contain option that is already a keyword in syslog-ng (example: retries) (#3658)
templates: fixed error reporting when invalid templates were specified
The amqp(), file() destination, sql(), stomp(), pdbtool, and
graphite() plugins had template options that did not report errors at startup
when invalid values were specified.
(#3660)
syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.
Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.
We would like to thank the following people for their contribution:
Andras Mitzki, Attila Szakacs, Balazs Scheidler, Gabor Nagy, Janos SZIGETVARI, Laszlo Budai, Laszlo Szemere, LittleFish33, László Várady, Ming Liu, Norbert Takacs, Peter Kokai, Todd C. Miller, Yi Fan Yu, Zoltan Pallagi