4.7.1

This is the combination of the news entries of 4.7.0 and 4.7.1. 4.7.1 hotfixed two crashes related to configuration reload.

Read Axoflow's blog post for more details. You can read more about the new features in the AxoSyslog documentation.

Highlights

Collecting Jellyfin logs

The new jellyfin() source, reads Jellyfin logs from its log file output.

Example minimal config:

source s_jellyfin {
  jellyfin(
    base-dir("/path/to/my/jellyfin/root/log/dir")
    filename-pattern("log_*.log")
  );
};

For more details about Jellyfin logging, see:

• https://jellyfin.org/docs/general/administration/configuration/#main-configuration
• https://jellyfin.org/docs/general/administration/configuration/#log-directory

As the jellyfin() source is based on a wildcard-file() source, all of the wildcard-file() source options are applicable, too. (#4802)

Collecting *arr logs

Use the newly added *arr() sources to read various *arr logs:

• lidarr()
• prowlarr()
• radarr()
• readarr()
• sonarr()
• whisparr()

Example minimal config:

source s_radarr {
  radarr(
    dir("/path/to/my/radarr/log/dir")
  );
};

The logging module is stored in the <prefix><module> name-value pair, for example: .radarr.module => ImportListSyncService. The prefix can be modified with the prefix() option. (#4803)

Features

• opentelemetry(), syslog-ng-otlp() source: Added concurrent-requests() option.

  This option configures the maximal number of in-flight gRPC requests per worker. Setting this value to the range of 10s or 100s is recommended when there are a high number of clients sending simultaneously.

  Ideally, workers() * concurrent-requests() should be greater or equal to the number of clients, but this can increase the memory usage. (#4827)

• loki(): Support multi-tenancy with the new tenant-id() option (#4812)

• s3(): Added support for authentication from environment.

  The access-key() and secret-key() options are now optional, which makes it possible to use authentication methods originated from the environment, e.g. AWS_... environment variables or credentials files from the ~/.aws/ directory.

  For more info, see: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html (#4881)

• gRPC based drivers: Added channel-args() option.

  Affected drivers are:

  • bigquery() destination
  • loki() destination
  • opentelemetry() source and destination
  • syslog-ng-otlp() source and destination

  The channel-args() option accepts name-value pairs and sets channel arguments defined in https://grpc.github.io/grpc/core/group__grpc__arg__keys.html

  Example config:

    opentelemetry(
      channel-args(
        "grpc.loadreporting" => 1
        "grpc.minimal_stack" => 0
      )
    );
  

  (#4827)

• ${TRANSPORT} macro: Added support for locally created logs.

  New values are:

  • "local+unix-stream"
  • "local+unix-dgram"
  • "local+file"
  • "local+pipe"
  • "local+program"
  • "local+devkmsg"
  • "local+journal"
  • "local+afstreams"
  • "local+openbsd" (#4777)

• tags: Added new built-in tags that help identifying parse errors.

  New tags are:

  • "message.utf8_sanitized"
  • "message.parse_error"
  • "syslog.missing_pri"
  • "syslog.missing_timestamp"
  • "syslog.invalid_hostname"
  • "syslog.unexpected_framing"
  • "syslog.rfc3164_missing_header"
  • "syslog.rfc5424_unquoted_sdata_value" (#4804)

• mqtt() source: Added ${MQTT_TOPIC} name-value pair.

  It is useful for the cases where topic() contains wildcards.

  Example config:

  log {
    source { mqtt(topic("#")); };
    destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); };
  };
  

  (#4824)

• template(): Added a new template function: $(tags-head)

  This template function accepts multiple tag names, and returns the first one that is set.

  Example config:

  # resolves to "bar" if "bar" tag is set, but "foo" is not
  template("$(tags-head foo bar baz)")
  

  (#4804)

• s3(): Use default AWS URL if url() is not set. (#4813)

• opentelemetry(), syslog-ng-otlp() source: Added log-fetch-limit() option.

  This option can be used to fine tune the performance. To minimize locking while moving messages between source and destination side queues, syslog-ng can move messages in batches. The log-fetch-limit() option sets the maximal size of the batch moved by a worker. By default it is equal to log-iw-size() / workers(). (#4827)

• dqtool: add option for truncating (compacting) abandoned disk-buffers (#4875)

Bugfixes

• opentelemetry(): fix crash when an invalid configuration needs to be reverted (#4910)

• gRPC drivers: fixed a crash when gRPC drivers were used and syslog-ng was reloaded (#4909)

• opentelemetry(), syslog-ng-otlp() source: Fixed a crash.

  It occurred with multiple workers() during high load. (#4827)

• rename(): Fixed a bug, which always converted the renamed NV pair to string type. (#4847)

• With IPv6 disabled, there were linking errors (#4880)

Metrics

• http(): Added a new counter for HTTP requests.

  It is activated on stats(level(1));.

  Example metrics:

  syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="200",driver="http",id="#anon-destination0#0"} 16
  syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="401",driver="http",id="#anon-destination0#0"} 2
  syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="502",driver="http",id="#anon-destination0#0"} 1
  syslogng_output_http_requests_total{url="http://localhost:8888/foo",response_code="200",driver="http",id="#anon-destination0#0"} 24
  

  (#4805)

• gRPC based destination drivers: Added gRPC request related metrics.

  Affected drivers:

  • opentelemetry()
  • syslog-ng-otlp()
  • bigquery()
  • loki()

  Example metrics:

  syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="ok"} 49
  syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="unavailable"} 11
  

  (#4811)

• New metric to monitor destination reachability

  syslogng_output_unreachable is a bool-like metric, which shows whether a destination is reachable or not.

  sum() can be used to count all unreachable outputs, hence the negated name.

  It is currently available for the network(), syslog(), unix-*() destinations, and threaded destinations (http(), opentelemetry(), redis(), mongodb(), python(), etc.). (#4876)

• destinations: Added "syslogng_output_event_retries_total" counter.

  This counter is available for the following destination drivers:

  • amqp()
  • bigquery()
  • http() and all http based drivers
  • java()
  • kafka()
  • loki()
  • mongodb()
  • mqtt()
  • opentelemetry()
  • python() and all python based drivers
  • redis()
  • riemann()
  • smtp()
  • snmp()
  • sql()
  • stomp()
  • syslog-ng-otlp()

  Example metrics:

  syslogng_output_event_retries_total{driver="http",url="http://localhost:8888/${path}",id="#anon-destination0#0"} 5
  

  (#4807)

• syslogng_memory_queue_capacity

  Shows the capacity (maximum possible size) of each queue. Note that this metric publishes log-fifo-size(), which only limits non-flow-controlled messages. Messages coming from flow-controlled paths are not limited by log-fifo-size(), their corresponding source log-iw-size() is the upper limit. (#4831)

Other changes

• opentelemetry(), syslog-ng-otlp() source: Changed the backpressure behavior.

  syslog-ng no longer returns UNAVAILABLE to the gRPC request, when it cannot forward the received message because of backpressure. Instead, syslog-ng will block until the destination can accept more messages. (#4827)

• opentelemetry(), syslog-ng-otlp() source: log-iw-size() is now split between workers. (#4827)

• APT packages: Dropped Debian Buster support.

  Old packages are still available, but new syslog-ng versions will not be available on Debian Buster (#4840)

• dbld: AlmaLinux 8 support (#4902)

syslog-ng Discord

For a bit more interactive discussion, join our Discord server:

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Arpad Kunszt, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Hofi, Kovács, Gergő Ferenc, László Várady, Peter Marko, shifter

4.7.0

Read Axoflow's blog post for more details. You can read more about the new features in the AxoSyslog documentation.

Highlights

Collecting Jellyfin logs

The new jellyfin() source, reads Jellyfin logs from its log file output.

Example minimal config:

source s_jellyfin {
  jellyfin(
    base-dir("/path/to/my/jellyfin/root/log/dir")
    filename-pattern("log_*.log")
  );
};

For more details about Jellyfin logging, see:

• https://jellyfin.org/docs/general/administration/configuration/#main-configuration
• https://jellyfin.org/docs/general/administration/configuration/#log-directory

As the jellyfin() source is based on a wildcard-file() source, all of the wildcard-file() source options are applicable, too. (#4802)

Collecting *arr logs

Use the newly added *arr() sources to read various *arr logs:

• lidarr()
• prowlarr()
• radarr()
• readarr()
• sonarr()
• whisparr()

Example minimal config:

source s_radarr {
  radarr(
    dir("/path/to/my/radarr/log/dir")
  );
};

The logging module is stored in the <prefix><module> name-value pair, for example: .radarr.module => ImportListSyncService. The prefix can be modified with the prefix() option. (#4803)

Features

• opentelemetry(), syslog-ng-otlp() source: Added concurrent-requests() option.

  This option configures the maximal number of in-flight gRPC requests per worker. Setting this value to the range of 10s or 100s is recommended when there are a high number of clients sending simultaneously.

  Ideally, workers() * concurrent-requests() should be greater or equal to the number of clients, but this can increase the memory usage. (#4827)

• loki(): Support multi-tenancy with the new tenant-id() option (#4812)

• s3(): Added support for authentication from environment.

  The access-key() and secret-key() options are now optional, which makes it possible to use authentication methods originated from the environment, e.g. AWS_... environment variables or credentials files from the ~/.aws/ directory.

  For more info, see: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html (#4881)

• gRPC based drivers: Added channel-args() option.

  Affected drivers are:

  • bigquery() destination
  • loki() destination
  • opentelemetry() source and destination
  • syslog-ng-otlp() source and destination

  The channel-args() option accepts name-value pairs and sets channel arguments defined in https://grpc.github.io/grpc/core/group__grpc__arg__keys.html

  Example config:

    opentelemetry(
      channel-args(
        "grpc.loadreporting" => 1
        "grpc.minimal_stack" => 0
      )
    );
  

  (#4827)

• ${TRANSPORT} macro: Added support for locally created logs.

  New values are:

  • "local+unix-stream"
  • "local+unix-dgram"
  • "local+file"
  • "local+pipe"
  • "local+program"
  • "local+devkmsg"
  • "local+journal"
  • "local+afstreams"
  • "local+openbsd" (#4777)

• tags: Added new built-in tags that help identifying parse errors.

  New tags are:

  • "message.utf8_sanitized"
  • "message.parse_error"
  • "syslog.missing_pri"
  • "syslog.missing_timestamp"
  • "syslog.invalid_hostname"
  • "syslog.unexpected_framing"
  • "syslog.rfc3164_missing_header"
  • "syslog.rfc5424_unquoted_sdata_value" (#4804)

• mqtt() source: Added ${MQTT_TOPIC} name-value pair.

  It is useful for the cases where topic() contains wildcards.

  Example config:

  log {
    source { mqtt(topic("#")); };
    destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); };
  };
  

  (#4824)

• template(): Added a new template function: $(tags-head)

  This template function accepts multiple tag names, and returns the first one that is set.

  Example config:

  # resolves to "bar" if "bar" tag is set, but "foo" is not
  template("$(tags-head foo bar baz)")
  

  (#4804)

• s3(): Use default AWS URL if url() is not set. (#4813)

• opentelemetry(), syslog-ng-otlp() source: Added log-fetch-limit() option.

  This option can be used to fine tune the performance. To minimize locking while moving messages between source and destination side queues, syslog-ng can move messages in batches. The log-fetch-limit() option sets the maximal size of the batch moved by a worker. By default it is equal to log-iw-size() / workers(). (#4827)

• dqtool: add option for truncating (compacting) abandoned disk-buffers (#4875)

Bugfixes

• opentelemetry(), syslog-ng-otlp() source: Fixed a crash.

  It occurred with multiple workers() during high load. (#4827)

• rename(): Fixed a bug, which always converted the renamed NV pair to string type. (#4847)

• With IPv6 disabled, there were linking errors (#4880)

Metrics

• http(): Added a new counter for HTTP requests.

  It is activated on stats(level(1));.

  Example metrics:

  syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="200",driver="http",id="#anon-destination0#0"} 16
  syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="401",driver="http",id="#anon-destination0#0"} 2
  syslogng_output_http_requests_total{url="http://localhost:8888/bar",response_code="502",driver="http",id="#anon-destination0#0"} 1
  syslogng_output_http_requests_total{url="http://localhost:8888/foo",response_code="200",driver="http",id="#anon-destination0#0"} 24
  

  (#4805)

• gRPC based destination drivers: Added gRPC request related metrics.

  Affected drivers:

  • opentelemetry()
  • syslog-ng-otlp()
  • bigquery()
  • loki()

  Example metrics:

  syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="ok"} 49
  syslogng_output_grpc_requests_total{driver="syslog-ng-otlp",url="localhost:12345",response_code="unavailable"} 11
  

  (#4811)

• New metric to monitor destination reachability

  syslogng_output_unreachable is a bool-like metric, which shows whether a destination is reachable or not.

  sum() can be used to count all unreachable outputs, hence the negated name.

  It is currently available for the network(), syslog(), unix-*() destinations, and threaded destinations (http(), opentelemetry(), redis(), mongodb(), python(), etc.). (#4876)

• destinations: Added "syslogng_output_event_retries_total" counter.

  This counter is available for the following destination drivers:

  • amqp()
  • bigquery()
  • http() and all http based drivers
  • java()
  • kafka()
  • loki()
  • mongodb()
  • mqtt()
  • opentelemetry()
  • python() and all python based drivers
  • redis()
  • riemann()
  • smtp()
  • snmp()
  • sql()
  • stomp()
  • syslog-ng-otlp()

  Example metrics:

  syslogng_output_event_retries_total{driver="http",url="http://localhost:8888/${path}",id="#anon-destination0#0"} 5
  

  (#4807)

• syslogng_memory_queue_capacity

  Shows the capacity (maximum possible size) of each queue. Note that this metric publishes log-fifo-size(), which only limits non-flow-controlled messages. Messages coming from flow-controlled paths are not limited by log-fifo-size(), their corresponding source log-iw-size() is the upper limit. (#4831)

Other changes

• opentelemetry(), syslog-ng-otlp() source: Changed the backpressure behavior.

  syslog-ng no longer returns UNAVAILABLE to the gRPC request, when it cannot forward the received message because of backpressure. Instead, syslog-ng will block until the destination can accept more messages. (#4827)

• opentelemetry(), syslog-ng-otlp() source: log-iw-size() is now split between workers. (#4827)

• APT packages: Dropped Debian Buster support.

  Old packages are still available, but new syslog-ng versions will not be available on Debian Buster (#4840)

• dbld: AlmaLinux 8 support (#4902)

syslog-ng Discord

For a bit more interactive discussion, join our Discord server:

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Arpad Kunszt, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Hofi, Kovács, Gergő Ferenc, László Várady, Peter Marko, shifter

4.6.0

Read Axoflow's blog post for more details. You can read more about the new features in the AxoSyslog documentation.

Highlights

Forwarding logs to Google BigQuery

The bigquery() destination inserts logs to a Google BigQuery table via the high-performance gRPC API.

Authentication is done via Application Default Credentials.

You can locate your BigQuery table with the project() dataset() and table() options.

There are two ways to configure your table's schema.

• You can set the columns and their respective type and template with the schema() option. The available types are: STRING, BYTES, INTEGER, FLOAT, BOOLEAN, TIMESTAMP, DATE, TIME, DATETIME, JSON, NUMERIC, BIGNUMERIC, GEOGRAPHY, RECORD, INTERVAL.
• Alternatively you can import a .proto file with the protobuf-schema() option, and map the templates for each column.

The performance can be further improved with the workers(), batch-lines(), batch-bytes(), batch-timeout() and compression() options. By default the messages are sent with one worker, one message per batch and without compression.

Keepalive can be configured with the keep-alive() block and its time(), timeout() and max-pings-without-data() options.

Example config:

bigquery(
    project("test-project")
    dataset("test-dataset")
    table("test-table")
    workers(8)

    schema(
        "message" => "$MESSAGE"
        "app" STRING => "$PROGRAM"
        "host" STRING => "$HOST"
        "pid" INTEGER => int("$PID")
    )

    on-error("drop-property")

    # or alternatively instead of schema():
    # protobuf-schema("/tmp/test.proto"
    #                 => "$MESSAGE", "$PROGRAM", "$HOST", "$PID")

    # keep-alive(time(20000) timeout(10000) max-pings-without-data(0))
);

Example .proto schema:

syntax = "proto2";
​
message CustomRecord {
  optional string message = 1;
  optional string app = 2;
  optional string host = 3;
  optional int64 pid = 4;
}

(#4733) (#4770) (#4756)

Collecting native macOS system logs

Two new sources have been added on macOS: darwin-oslog(), darwin-oslog-stream(). darwin-oslog() replaced the earlier file source based solution with a native OSLog framework based one, and is automatically used in the system() source on darwin platform if the darwinosl plugin is presented.

This plugin is available only on macOS 10.15 Catalina and above, the first version that has the OSLog API.

darwin-oslog()

This is a native OSLog Framework based source to read logs from the local store of the unified logging system on darwin OSes. For more info, see https://developer.apple.com/documentation/oslog?language=objc

The following parameters can be used for customization:

• filter-predicate()
  • string value, which can be used to filter the log messages natively
  • default value: (eventType == 'logEvent' || eventType == 'lossEvent' || eventType == 'stateEvent' || eventType == 'userActionEvent') && (logType != 'debug')
  • for more details, see
    • man log
    • https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/Predicates/Articles/pSyntax.html
• go-reverse()
  • boolean value, setting to yes will provide a reverse-ordered log list (from latest to oldest)
  • default value: no
• do-not-use-bookmark()
  • boolean value, setting to yes will prevent syslog-ng from continuing to feed the logs from the last remembered position after a (re-)start, which means, depending on the other settings, the feed will always start from the end/beginning of the available log list
  • default value: no, which means syslog-ng will attempt to continue feeding from the last remembered log position after a (re-)start
• max-bookmark-distance()
  • integer value, maximum distance in seconds that far an earlier bookmark can point backward, e.g. if syslog-ng was stopped for 10 minutes and max-bookmark-distance is set to 60 then syslog-ng will start feeding the logs only from the last 60 seconds at startup, 9 minutes of logs 'will be lost'
  • default value: 0, which means no limit
• read-old-records()
  • boolean value, controls if syslog-ng should start reading logs from the oldest available at first start (or if no bookmark can be found)
  • default value: no
• fetch-delay()
  • integer value, controls how much time syslog-ng should wait between reading/sending log messages, this is a fraction of a second, where wait_time = 1 second / n, so, e.g. n=1 means that only about 1 log will be read and sent in each second, and n=1 000 000 means only 1 microsecond (the allowed minimum value now!) will be the delay between read/write attempts
  • Use with care, though lower delay time can increase log feed performance, at the same time could lead to a heavy system load!
  • default value: 10 000
• fetch-retry-delay()
  • integer value, controls how many seconds syslog-ng will wait before a repeated attempt to read/send once it's out of available logs
  • default value: 1
• log-fetch-limit()
  • Warning: This option is now disabled due to an OSLog API bug (https://openradar.appspot.com/radar?id=5597032077066240), once it's fixed it will be enabled again
  • integer value, that limits the number of logs syslog-ng will send in one run
  • default value: 0, which means no limit

NOTE: the persistent OSLog store is not infinite, depending on your system setting usually, it keeps about 7 days of logs on disk, so it could happen that the above options cannot operate the way you expect, e.g. if syslog-ng was stopped for about more then a week it could happen that will not be able to restart from the last saved bookmark position (as that might not be presented in the persistent log anymore)

darwin-oslog-stream()

This is a wrapper around the OS command line "log stream" command that can provide a live log stream feed. Unlike in the case of darwin-oslog() the live stream can contain non-persistent log events too, so take care, there might be a huge number of log events every second that could put an unusual load on the device running syslog-ng with this source. Unfortunately, there's no public API to get the same programmatically, so this one is implemented using a program() source.

Possible parameters:

• params()
  • a string that can contain all the possible params the macOS log tool can accept
  • see log --help stream for full reference, and man log for more details
  • IMPORTANT: the parameter --style is used internally (defaults to ndjson), so it cannot be overridden, please use other sysylog-ng features (templates, rewrite rules, etc.) for final output formatting
  • default value: --type log --type trace --level info --level debug, you can use ``def-osl-stream-params\ for referencing it if you wish to keep the defaults when you add your own

(#4423)

Collecting qBittorrent logs

The new qbittorrent() source, reads qBittorrent logs from its log file output.

Example minimal config:

source s_qbittorrent {
  qbittorrent(
    dir("/path/to/my/qbittorrent/root/log/dir")
  );
};

The root dir of the qBittorrent logs can be found in the "Tools" / "Preferences" / "Behavior" / "Log file" / "Save path" field.

As the qbittorrent() source is based on a file() source, all of the file() source options are applicable, too.

(#4760)

Collecting pihole FTL logs

The new pihole-ftl() source reads pihole FTL (Faster Than Light) logs, which are usually accessible in the "Tools" / "Pi-hole diagnosis" menu.

Example minimal config:

source s_pihole_ftl {
  pihole-ftl();
};

By default it reads the /var/log/pihole/FTL.log file. You can change the root dir of Pi-hole's logs with the dir() option, where the FTL.log file can be found.

As the pihole-ftl() source is based on a file() source, all of the file() source options are applicable, too.

(#4760)

Parsing Windows Eventlog XMLs

The new windows-eventlog-xml-parser() introduces parsing support for Windows Eventlog XMLs.

Its parameters are the same as the xml() parser.

Example config:

parser p_win {
    windows-eventlog-xml-parser(prefix(".winlog."));
};

(#4793)

Features

• cloud-auth(): Added support for user-managed-service-account() gcp() auth method.

  This authentication method can be used on VMs in GCP to use the linked service.

  Example minimal config, which tries to use the "default" service account:

  cloud-auth(
    gcp(
      user-managed-service-account()
    )
  )
  

  Full config:

  cloud-auth(
    gcp(
      user-managed-service-account(
        name("[email protected]")
        metadata-url("my-custom-metadata-server:8080")
      )
    )
  )
  

  This authentication method is extremely useful with syslog-ng's google-pubsub() destination, when it is running on VMs in GCP, for example:

  destination {
    google-pubsub(
      project("syslog-ng-test-project")
      topic("syslog-ng-test-topic")
      auth(user-managed-service-account())
    );
  };
  

  For more info about this GCP authentication method, see:

  • https://cloud.google.com/compute/docs/access/authenticate-workloads#curl
  • https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances (#4755)

• opentelemetry(), syslog-ng-otlp() sources: Added workers() option.

  This feature enables processing the OTLP messages on multiple threads, which can greatly improve the performance. By default it is set to workers(1). (#4774)

• opentelemetry(), syslog-ng-otlp() destinations: Added compression() option.

  This boolean option can be used to enable gzip compression in gRPC requests. By default it is set to compression(no). (#4765)

• opentelemetry(), syslog-ng-otlp() destinations: Added batch-bytes() option.

  This option lets the user limit the bytes size of a batch. As there is a default 4 MiB batch limit by OTLP, it is necessary to keep the batch size smaller, but it would be hard to configure without this option.

  Please note that the batch can be at most 1 message larger than the set limit, so consider this when setting this value.

  The default value is 4 MB, which is a bit below 4 MiB.

  The calculation of the batch size is done before compression, which is the same as the limit is calculated on the server.

  Example config:

    syslog-ng-otlp(
      url("localhost:12345")
      workers(16)
      log-fifo-size(1000000)
  
      batch-timeout(5000) # ms
      batch-lines(1000000) # Huge limit, batch-bytes() will limit us sooner
  
      batch-bytes(1MB) # closes and flushes the batch after the last message pushed it above the 1 MB limit
      # not setting batch-bytes() defaults to 4 MB, which is a bit below the default 4 MiB limit
    );
  

  (#4772)

• opentelemetry(), syslog-ng-otlp(): Added syslog-ng style list support. (#4794)

• $(tag) template function: expose bit-like tags that are set on messages.

  Syntax: $(tag <name-of-the-tag> <value-if-set> <value-if-unset>)

  Unless the value-if-set/unset arguments are specified $(tag) results in a boolean type, expanding to "0" or "1" depending on whether the message has the specified tag set.

  If value-if-set/unset are present, $(tag) would return a string, picking the second argument <value-if-set> if the message has <tag> and picking the third argument <value-if-unset> if the message does not have <tag> (#4766)

• set-severity() support for aliases: widespread aliases to severity values produced by various applications are added to set-severity(). (#4763)

• flags(seqnum-all): available in all destination drivers, this new flag changes $SEQNUM behaviour, so that all messages get a sequence number, not just local ones. Previously syslog-ng followed the logic of the RFC5424 meta.sequenceId structured data element, e.g. only local messages were to get a sequence number, forwarded messages retained their original sequenceId that we could potentially receive ourselves.

  For example, this destination would include the meta.sequenceId SDATA element even for non-local logs and increment that value by every message transmitted:

  destination { syslog("127.0.0.1" port(2001) flags(seqnum-all)); };

  This generates a message like this on the output, even if the message is not locally generated (e.g. forwarded from another syslog sender):

    <13>1 2023-12-09T21:51:30+00:00 localhost sdff - - [meta sequenceId="1"] f sdf fsd
    <13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="2"] f sdf fsd
    <13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="3"] f sdf fsd
    <13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="4"] f sdf fsd
    <13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="5"] f sdf fsd
  

  (#4745)

• loggen: improve loggen performance for synthetic workloads, so we can test for example up to 650k msg/sec on a AMD Ryzen 7 Pro 6850U CPU. (#4476)

Bugfixes

• metrics-probe(): Fixed not cleaning up dynamic labels for each message if no static labels are set. (#4750)

• regexp-parser(): Fixed a bug, which stored some values incorrectly if ${MESSAGE} was changed with a capture group. (#4759)

• network() source: fix marking originally valid utf-8 messages when sanitize-utf8 is enabled (#4744)

• python(): Fixed a memory leak in list typed LogMessage values. (#4790)

Packaging

• VERSION renamed to VERSION.txt: due to a name collision with C++ based builds on MacOS, the file containing our version number was renamed to VERSION.txt. (#4775)

• Added gperf as a build dependency. (#4763)

Notes to developers

• LogThreadedSourceDriver: Added multi-worker API, which is a breaking change.

  Check the Pull Request for inspiration on how to follow up these changes. (#4774)

Other changes

• network()/syslog() sources: support UTF-8 sanitization/validation of RFC 5424 and no-parse messages

  The sanitize-utf8, validate-utf8 flags are now supported when parsing RFC 5424 messages or when parsing is disabled. (#4744)

• APT packages: Added Ubuntu Mantic Minotaur. (#4737)

syslog-ng Discord

For a bit more interactive discussion, join our Discord server:

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Attila Szakacs, Balazs Scheidler, Hofi, László Várady, Romain Tartière

4.5.0

Read Axoflow's blog post for more details. You can read more about the new features in the AxoSyslog documentation.

Highlights

Sending log messages to OpenObserve

The openobserve-log() destination feeds OpenObserve via the JSON API.

Example config:

openobserve-log(
    url("http://openobserve-endpoint")
    port(5080)
    stream("default")
    user("[email protected]")
    password("V2tsn88GhdNTKxaS")
);

(#4698)

Sending messages to Google Pub/Sub

The google-pubsub() destination feeds Google Pub/Sub via the HTTP REST API.

Example config:

google-pubsub(
  project("syslog-ng-project")
  topic("syslog-ng-topic")
  auth(
    service-account(
      key("/path/to/service-account-key.json")
    )
  )
);

See the Google Pub/Sub documentation to learn more about configuring a service account. (#4651)

Parsing PostgreSQL logs

The postgresql-csvlog-parser(): add a new parser to process CSV log formatted by PostgreSQL (https://www.postgresql.org/docs/current/runtime-config-logging.html). The CSV format is extracted into a set of name-value pairs. (#4586)

Features

• http(): Added support for using templates in the url() option.

  In syslog-ng a template can only be resolved on a single message, as the same template might have different resolutions on different messages. A http batch consists of multiple messages, so it is not trivial to decide which message should be used for the resolution.

  When batching is enabled and multiple workers are configured it is important to only batch messages which generate identical URLs. In this scenario one must set the worker-partition-key() option with a template that contains all the templates used in the url() option, otherwise messages will be mixed.

  For security reasons, all the templated contents in the url() option are getting URL encoded automatically. Also the following parts of the url cannot be templated:

  • scheme
  • host
  • port
  • user
  • password (#4663)

• $TRANSPORT: this is a new name-value pair that syslog-ng populates automatically. It indicates the "transport" mechanism used to retrieve/receive the message. It is up to the source driver to determine the value. Currently the following values were implemented:

  BSD syslog drivers: tcp(), udp() & network()

  • rfc3164+tls
  • rfc3164+tcp
  • rfc3164+udp
  • rfc3164+proxied-tls
  • rfc3164+<custom logproto like altp>

  UNIX domain drivers: unix-dgram(), unix-stream()

  • unix-stream
  • unix-dgram

  RFC5424 style syslog: syslog():

  • rfc5426: syslog over udp
  • rfc5425: syslog over tls
  • rfc6587: syslog over tcp
  • rfc5424+<custom logproto like altp>: syslog over a logproto plugin

  Other drivers:

  • otlp: otel() driver
  • mqtt: mqtt() driver
  • hypr-api: hypr-audit-source() driver

  $IP_PROTO: indicate the IP protocol version used to retrieve/receive the message. Contains either "4" to indicate IPv4 and "6" to indicate IPv6. (#4673)

• network() and syslog() drivers: Added ignore-validity-period as a new flag to ssl-options().

  By specifying ignore-validity-period, you can ignore the validity periods of certificates during the certificate validation process. (#4642)

• tls() in udp()/tcp()/network() and syslog() drivers: add support for a new http() compatible ssl-version() option. This makes the TLS related options for http() and other syslog-like drivers more similar. This requires OpenSSL 1.1.0. (#4682)

• cloud-auth(): Added a new plugin for drivers, which implements different cloud related authentications.

  Currently the only supported authentication is GCP's Service Account for the http() destination.

  Example config:

  http(
    cloud-auth(
      gcp(
        service-account(
          key("/path/to/service-account-key.json")
          audience("https://pubsub.googleapis.com/google.pubsub.v1.Publisher")
        )
      )
    )
  );
  

  (#4651)

• csv-parser(): allow parsing the extracted values into matches ($1, $2, $3 ...) by omitting the columns() parameter, which normally specifies the column names. (#4678)

• --check-startup: a new command line option for syslog-ng along with the existing --syntax-only. This new option will do a complete configuration initialization and then exit with exit code indicating the result. Since this also initializes things like network listeners, it will probably not work when there is another syslog-ng instance running in the background. The recommended use of this option is a dedicated config check container, as explained in #4592. (#4646)

Bugfixes

• s3: Fixed an ImportError.

  ImportError: cannot import name 'SharedBool' from 'syslogng.modules.s3.s3_object' (#4700)

• loki(): fixed mixing non-related label values (#4713)

• type hinting: Parsing and casting fractions are now done locale independently. (#4702)

• metrics-probe(): Fixed a crash.

  This crash occurred when a metrics-probe() instance was used in multiple source threads, like a network() source with multiple connections. (#4685)

• flags() argument to various drivers: fix a potential crash in case a flag with at least 32 characters is used. No such flag is defined by syslog-ng, so the only way to trigger the crash is to use an invalid configuration file. (#4689)

• Fix $PROTO value for transport(tls) connections, previously it was set to "0" while in reality these are tcp connections (e.g. "6").

  Fix how syslog-ng sets $HOST for V4-mapped addresses in case of IPv6 source drivers (e.g. udp6()/tcp6() or when using ip-protocol(6) for tcp()/udp()). Previously V4-mapped addresses would be represented as "::ffff:<ipv4 address>". This is not wrong per-se, but would potentially cause the same host to be represented in multiple ways. With the fix, syslog-ng would just use "<ipv4 address>" in these cases. (#4673)

• db-parser(): support nested match characters in @QSTRING@ pattern parser (#4717)

Other changes

• LogSource and LogFetcher: additional documentation was added to these Python classes to cover explicit source-side batching functionalities (e.g. the auto_close_batch attribute and the close_batch() method). (#4673)

• rate-limit(): Renamed the template() option to key(), which better communicates the intention. (#4679)

• templates: The template-escape() option now only escapes the top-level template function.

  Before syslog-ng 4.5.0 if you had embedded template functions, the template-escape(yes) setting escaped the output of each template function, so the parent template function received an already escaped string. This was never the intention of the template-escape() option.

  Although this is a breaking change, we do not except anyone having a config that is affected. If you have such a config, make sure to follow-up this change. If you need help with it, feel free to open an issue or discussion on GitHub, or contact us on the Axoflow Discord server. (#4666)

• loki(): The timestamp() option now supports quoted strings.

  The valid values are the following, with or without quotes, case insensitive:

  • "current"
  • "received"
  • "msg" (#4688)

syslog-ng Discord

For a bit more interactive discussion, join our Discord server:

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Attila Szakacs, Balazs Scheidler, Cedric Arickx, Fabrice Fontaine, Hofi, László Várady, Romain Tartière, Szilard Parrag, yashmathne

4.4.0

Read Axoflow's blog post for more details. You can read more about the new features in the AxoSyslog documentation.

Highlights

Sending messages between syslog-ng instances via OTLP/gRPC

The syslog-ng-otlp() source and destination helps to transfer the internal representation of a log message between syslog-ng instances. In contrary to the syslog-ng() (ewmm()) drivers, syslog-ng-otlp() does not transfer the messages on simple TCP connections, but uses the OpenTelemetry protocol to do so.

It is easily scalable (workers() option), uses built-in application layer acknowledgement, out of the box supports google service authentication (ADC or ALTS), and gives the possibility of better load balancing.

The performance is currently similar to ewmm() (OTLP is ~30% quicker) but there is a source side limitation, which will be optimized. We measured 200-300% performance improvement with a PoC optimized code using multiple threads, so stay tuned.

Note: The syslog-ng-otlp() source is only an alias to the opentelemetry() source. This is useful for not needing to open different ports for the syslog-ng messages and other OpenTelemetry messages. The syslog-ng messages are marked with a @syslog-ng scope name and the current syslog-ng version as the scope version. Both sources will handle the incoming syslog-ng messages as syslog-ng messages, and all other messages as simple OpenTelemetry messages. (#4564)

Grafana Loki destination

The loki() destination sends messages to Grafana Loki using gRPC. The message format conforms to the documented HTTP endpoint: https://grafana.com/docs/loki/latest/reference/api/#push-log-entries-to-loki

Example config:

loki(
    url("localhost:9096")
    labels(
        "app" => "$PROGRAM",
        "host" => "$HOST",
    )

    workers(16)
    batch-timeout(10000)
    batch-lines(1000)
);

Loki requires monotonic timestamps within the same label-set, which makes it difficult to use the original message timestamp without the possibility of message loss. In case the monotonic property is violated, Loki discards the problematic messages with an error. The source of the timestamps can be configured with the timestamp() option (current, received, msg).

(#4631)

S3 destination

The s3() destination stores log messages in S3 objects.

Minimal config:

s3(
    url("http://localhost:9000")
    bucket("syslog-ng")
    access-key("my-access-key")
    secret-key("my-secret-key")
    object-key("${HOST}/my-logs")
    template("${MESSAGE}\n")
);

Compression

Setting compression(yes) enables gzip compression, and implicitly adds a .gz suffix to the created object's key. Use the compresslevel() options to set the level of compression (0-9).

Rotation based on object size

The max-object-size() option configures syslog-ng to finish an object if it reaches a certain size. syslog-ng will append an index ("-1", "-2", ...) to the end of the object key when starting a new object after rotation.

Rotation based on timestamp

The object-key-timestamp() option can be used to set a datetime related template, which gets appended to the end of the object (e.g. "${R_MONTH_ABBREV}${R_DAY}" => "-Sep25"). When a log message arrives with a newer timestamp template resolution, the previous timestamped object gets finised and a new one is started with the new timestamp. Backfill messages do not reopen and append the old object, but starts a new object with the key having an index appended to the old object.

Rotation based on timeout

The flush-grace-period() option sets the number of minutes to wait for new messages to arrive to objects, if the timeout expires the object is finished, and a new message will start a new with an index appended.

Upload options

The objects are uploaded with the multipart upload API. Chunks are composed locally. When a chunk reaches a certain size (by default 5 MiB), the chunk is uploaded. When an object is finished, the multipart upload gets completed and the chunks are merged by S3.

Upload parameters can be configured with the chunk-size(), upload-threads() and max-pending-uploads() options.

Additional options

Additional options include region(), storage-class() and canned-acl().

(#4624)

Features

• http(): Added compression ability for use with metered egress/ingress

  The new features can be accessed with the following options:

  • accept-encoding() for requesting the compression of HTTP responses form the server. (These are currently not used by syslog-ng, but they still contribute to network traffic.) The available options are identity (for no compression), gzip or deflate. If you want the driver to accept multiple compression types, you can list them separated by commas inside the quotation mark, or write all, if you want to enable all available compression types.
  • content-compression() for compressing messages sent by syslog-ng. The available options are identity for no compression, gzip, or deflate.

  Below you can see a configuration example:

  destination d_http_compressed{
    http(url("127.0.0.1:80"), content-compression("deflate"), accept-encoding("all"));
  };
  

  (#4137)

• opensearch: Added a new destination.

  It is similar to elasticsearch-http(), with the difference that it does not have the type() option, which is deprecated and advised not to use. (#4560)

• Added metrics for message delays: a new metric is introduced that measures the delay the messages accumulate while waiting to be delivered by syslog-ng. The measurement is sampled, e.g. syslog-ng would take the very first message in every second and expose its delay as a value of the new metric.

  There are two new metrics:

  • syslogng_output_event_delay_sample_seconds -- contains the latency of outgoing messages
  • syslogng_output_event_delay_sample_age_seconds -- contains the age of the last measurement, relative to the current time. (#4565)

• metrics-probe: Added dynamic labelling support via name-value pairs

  You can use all value-pairs options, like key(), rekey(), pair() or scope(), etc...

  Example:

  metrics-probe(
    key("foo")
    labels(
      "static-label" => "bar"
      key(".my_prefix.*" rekey(shift-levels(1)))
    )
  );
  

  syslogng_foo{static_label="bar",my_prefix_baz="almafa",my_prefix_foo="bar",my_prefix_nested_axo="flow"} 4
  

  (#4610)

• systemd-journal(): Added support for enabling multiple systemd-journal() sources

  Using multiple systemd-journal() sources are now possible as long as each source uses a unique systemd namespace. The namespace can be configured with the namespace() option, which has a default value of "*". (#4553)

• stdout(): added a new destination that allows you to write messages easily to syslog-ng's stdout. (#4620)

• network(): Added ignore-hostname-mismatch as a new flag to ssl-options().

  By specifying ignore-hostname-mismatch, you can ignore the subject name of a certificate during the validation process. This means that syslog-ng will only check if the certificate itself is trusted by the current set of trust anchors (e.g. trusted CAs) ignoring the mismatch between the targeted hostname and the certificate subject. (#4628)

Bugfixes

• syslog-ng: fix runtime undefined symbol: random_choice_generator_parser' when executing syslog-ng -V or using an example plugin (#4615)

• Fix threaded destination crash during a configuration revert

  Threaded destinations that do not support the workers() option crashed while syslog-ng was trying to revert to an old configuration. (#4588)

• redis(): fix incrementing seq_num (#4588)

• python(): fix crash when using Persist or LogTemplate without global python{} code block in configuration (#4572)

• mqtt() destination: fix template option initialization (#4605)

• opentelemetry: Fixed error handling in case of insert failure. (#4583)

• pdbtool: add validation for types of <value> tags

  In patterndb, you can add extra name-value pairs following a match with the tags. But the actual value of these name-value pairs were never validated against their types, meaning that an incorrect value could be set using this construct. (#4621)

• grouping-by(), group-lines(): Fixed a persist name generating error. (#4478)

Packaging

• debian: Added tzdata-legacy to BuildDeps for recent debian versions.

  In the recent debian packaging some of the timezone info files moved to a new tzdata-legacy package from the standard tzdata package. (#4643)

• rhel: contrib/vim has been removed from the source. (#4607)

Other changes

• APT packages: Dropped support for Ubuntu Bionic. (#4648)

• vim: Syntax highlight file is no longer packaged.

  vim syntax files where previously installed by the RedHat packages of syslog-ng (but not the Debian ones). These files where sometime lagging behind, so in order to provide a more up-to-date experience on all platforms, regardless of the installation of the syslog-ng package, the vim syntax files have been moved to a dedicated repository syslog-ng/vim-syslog-ng that can be used using a plugin manager such as vim-plug, vim-pathogen or vundle. (#4607)

syslog-ng Discord

For a bit more interactive discussion, join our Discord server:

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Alex Becker, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Hofi, László Várady, Romain Tartière, Szilard Parrag

4.3.1

This is the combination of the news entries of 4.3.0 and 4.3.1. 4.3.1 hotfixed a python-parser() related crash and a metrics related memory leak. It also added Ubuntu 23.04 and Debian 12 support for APT packages and the opensearch() destination.

Read Axoflow's blog post for more details.

Highlights

parallelize() support for pipelines

syslog-ng has traditionally performed processing of log messages arriving from a single connection sequentially. This was done to ensure message ordering as well as most efficient use of CPU on a per message basis. This mode of operation is performing well as long as we have a relatively large number of parallel connections, in which case syslog-ng would use all the CPU cores available in the system.

In case only a small number of connections deliver a large number of messages, this behaviour may become a bottleneck.

With the new parallelization feature, syslog-ng gained the ability to re-partition a stream of incoming messages into a set of partitions, each of which is to be processed by multiple threads in parallel. This does away with ordering guarantees and adds an extra per-message overhead. In exchange it will be able to scale the incoming load to all CPUs in the system, even if coming from a single, chatty sender.

To enable this mode of execution, use the new parallelize() element in your log path:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4));

  # from this part on, messages are processed in parallel even if
  # messages are originally coming from a single connection

  parser { ... };
  destination { ... };
};

The config above will take all messages emitted by the tcp() source and push the work to 4 parallel threads of execution, regardless of how many connections were in use to deliver the stream of messages to the tcp() driver.

parallelize() uses round-robin to allocate messages to partitions by default. You can however retain ordering for a subset of messages with the partition-key() option.

You can use partition-key() to specify a message template. Messages that expand to the same value are guaranteed to be mapped to the same partition.

For example:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4) partition-key("$HOST"));

  # from this part on, messages are processed in parallel if their
  # $HOST value differs. Messages with the same $HOST will be mapped
  # to the same partition and are processed sequentially.

  parser { ... };
  destination { ... };
};

NOTE: parallelize() requires a patched version of libivykis that contains this PR https://github.com/buytenh/ivykis/pull/25. syslog-ng source releases bundle this version of ivykis in their source trees, so if you are building from source, be sure to use the internal version (--with-ivykis=internal). You can also use Axoflow's cloud native container image for syslog-ng, named AxoSyslog (https://github.com/axoflow/axosyslog-docker) which also incorporates this change.

(#3966)

Receiving and sending OpenTelemetry (OTLP) messages

The opentelemetry() source, parser and destination are now available to receive, parse and send OTLP/gRPC messages.

syslog-ng accepts logs, metrics and traces.

The incoming fields are not available through syslog-ng log message name-value pairs for the user by default. This is useful for forwarding functionality (the opentelemetry() destination can access and format them). If such functionality is required, you can configure the opentelemetry() parser, which maps all the fields with some limitations.

The behavior of the opentelemetry() parser is the following:

The name-value pairs always start with .otel. prefix. The type of the message is stored in .otel.type (possible values: log, metric and span). The resource info is mapped to .otel.resource.<...> (e.g.: .otel.resource.dropped_attributes_count, .otel.resource.schema_url ...), the scope info is mapped to .otel.scope.<...> (e.g.: .otel.scope.name, .otel.scope.schema_url, ...).

The fields of log records are mapped to .otel.log.<...> (e.g. .otel.log.body, .otel.log.severity_text, ...).

The fields of metrics are mapped to .otel.metric.<...> (e.g. .otel.metric.name, .otel.metric.unit, ...), the type of the metric is mapped to .otel.metric.data.type (possible values: gauge, sum, histogram, exponential_histogram, summary) with the actual data mapped to .otel.metric.data.<type>.<...> (e.g.: .otel.metric.data.gauge.data_points.0.time_unix_nano, ...).

The fields of traces are mapped to .otel.span.<...> (e.g. .otel.span.name, .otel.span.trace_state, ...).

repeated fields are given an index (e.g. .otel.span.events.5.time_unix_nano).

The mapping of AnyValue type fields is limited. string, bool, int64, double and bytes values are mapped with the respective syslog-ng name-value type (e.g. .otel.resource.attributes.string_key => string_value), however ArrayValue and KeyValueList types are stored serialized with protobuf type. protobuf and bytes types are not directly available for the user, unless an explicit type cast is added (e.g. "bytes(${.otel.log.span_id})") or --include-bytes is passed to name-value iterating template functions (e.g. $(format-json .otel.* --include-bytes), which will base64 encode the bytes content).

Three authentication methods are available in the source auth() block: insecure() (default), tls() and alts(). tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values: required-trusted, required-untrusted, optional-trusted and optional-untrusted) options. ALTS is a simple to use authentication, only available within Google's infrastructure.

The same methods are available in the destination auth() block, with two differences: tls(peer-verify()) is not available, and there is a fourth method, called ADC, which accepts the target-service-account() option, where a list of service accounts can be configured to match against when authenticating the server.

Example configs:

log otel_forward_mode_alts {
  source {
    opentelemetry(
      port(12345)
      auth(alts())
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12345")
      auth(alts())
    );
  };
};

log otel_to_non_otel_insecure {
  source {
    opentelemetry(
      port(12345)
    );
  };

  parser {
    opentelemetry();
  };

  destination {
    network(
      "my-network-server"
      port(12345)
      template("$(format-json .otel.* --shift-levels 1 --include-bytes)\n")
    );
  };
};

log non_otel_to_otel_tls {
  source {
    network(
      port(12346)
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
};

(#4523) (#4510)

Sending messages to CrowdStrike Falcon LogScale (Humio)

The logscale() destination feeds LogScale via the Ingest API.

Minimal config:

destination d_logscale {
  logscale(
    token("my-token")
  );
};

Additional options include:

• url()
• rawstring()
• timestamp()
• timezone()
• attributes()
• extra-headers()
• content-type()

(#4472)

Features

• afmongodb: Bulk MongoDB insert is added via the following options

  • bulk (yes/no) turns on/off bulk insert usage, no forces the old behavior (each log is inserted one by one into the MongoDB)
  • bulk_unordered (yes/no) turns on/off unordered MongoDB bulk operations
  • bulk_bypass_validation (yes/no) turns on/off MongoDB bulk operations validation
  • write_concern (unacked/acked/majority/n > 0) sets write concern mode of the MongoDB operations, both bulk and single

  NOTE: Bulk sending is only efficient if the used collection is constant (e.g. not using templates) or the used template does not lead to too many collections switching within a reasonable time range. (#4483)

• sql: Added 2 new options

  • quote_char to aid custom quoting for table and index names (e.g. MySQL needs sometimes this for certain identifiers) NOTE: Using a back-tick character needs a special formatting as syslog-ng uses it for configuration parameter names, so for that use: quote_char("``") (double back-tick)
  • dbi_driver_dir to define an optional DBI driver location for DBD initialization

  NOTE: libdbi and libdbi-drivers OSE forks are updated, afsql now should work nicely both on ARM and X86 macOS systems too (tested on macOS 13.3.1 and 12.6.4)

  Please do not use the pre-built ones (e.g. 0.9.0 from Homebrew), build from the master of the following

  • https://github.com/balabit-deps/libdbi-drivers
  • https://github.com/balabit-deps/libdbi

  (#4460)

• opensearch: Added a new destination.

  It is similar to elasticsearch-http(), with the difference that it does not have the type() option, which is deprecated and advised not to use. (#4560)

Bugfixes

• network(),syslog(),tcp() destination: fix TCP keepalive

  tcp-keepalive-*() options were broken on the destination side since v3.34.1. (#4559)

• Fixed a hang, which happend when syslog-ng received exremely low CPU time. (#4524)

• $(format-json): Fixed a bug where sometimes an unnecessary comma was added in case of a type cast failure. (#4477)

• Fix flow-control when fetch-limit() is set higher than 64K

  In high-performance use cases, users may configure log-iw-size() and fetch-limit() to be higher than 2^16, which caused flow-control issues, such as messages stuck in the queue forever or log sources not receiving messages. (#4528)

• int32() and int64() type casts: accept hex numbers as proper number representations just as the @NUMBER@ parser within db-parser(). Supporting octal numbers were considered and then rejected as the canonical octal representation for numbers in C would be ambigious: a zero padded decimal number could be erroneously considered octal. I find that log messages contain zero padded decimals more often than octals. (#4535)

• Fixed compilation on platforms where SO_MEMINFO is not available (#4548)

• python: InstantAckTracker, ConsecutiveAckTracker and BatchedAckTracker are now called properly.

  Added proper fake classes for the InstantAckTracker, ConsecutiveAckTracker and BatchedAckTracker classes, and the wapper now calls the super class' constructor. Previusly the super class' constructor was not called which caused the python API to never call into the C API, which's result was that that the callback was never called. (#4549)

• python: Fixed a crash when reloading with a config, which uses a python parser with multiple references. (#4552) (#4567)

• mqtt(): Fixed the name of the stats instance (mqtt-source) to conform to the standard comma-separated format. (#4551)

• metrics: Fixed a memory leak which happened during reload, and was introduced in 4.3.0. (#4568)

Packaging

• scl.conf: The scl.conf file has been moved to /share/syslog-ng/include/scl.conf (#4534)

• C++ plugins: Some of syslog-ng's plugins now contain C++ code.

  By default they are being built if a C++ compiler is available. Disabling it is possible with --disable-cpp.

  Affected plugins:

  • lib/syslog-ng/libexamples.so
    • --disable-cpp will only disable the C++ part (random-choice-generator())
  • lib/syslog-ng/libotel.so

  (#4484)

• debian: A new module is added, called syslog-ng-mod-grpc.

  Its dependencies are: protobuf-compiler, protobuf-compiler-grpc, libprotobuf-dev, libgrpc++-dev. Building the module can be toggled with --enable-grpc. (#4510)

• pcre: syslog-ng now uses pcre2 (8 bit) as a dependency instead of pcre.

  The minimum pcre2 version is 10.0. (#4537)

Notes to developers

• lib/logmsg: Public field LogMessage::protected has been renamed to LogMessage::write_protected.

  Direct usage of this field is discouraged, instead use the following functions:

  • log_msg_is_write_protected()
  • log_msg_write_protect() (#4484)

• lib/templates: Public field LogTemplate::template has been renamed to LogTemplate::template_str. (#4484)

Other changes

• syslog-ng-cfg-db: Moved to a separate repository.

  It is available at: https://github.com/alltilla/syslog-ng-cfg-helper (#4475)

• disk-buffer: Added alternative option names

  disk-buf-size() -> capacity-bytes() qout-size() -> front-cache-size() mem-buf-length() -> flow-control-window-size() mem-buf-size() -> flow-control-window-bytes()

  Old option names are still available.

  Example configs:

  tcp(
    "127.0.0.1" port(2001)
    disk-buffer(
      reliable(yes)
      capacity-bytes(1GiB)
      flow-control-window-bytes(200MiB)
      front-cache-size(1000)
    )
  );
  
  tcp(
    "127.0.0.1" port(2001)
    disk-buffer(
      reliable(no)
      capacity-bytes(1GiB)
      flow-control-window-size(10000)
      front-cache-size(1000)
    )
  );
  

  (#4526)

• selinux: Added RHEL9 support for the selinux policies

  Added RHEL9 support for the selinux policies at contrib/selinux (#4509)

• metrics: replace driver_instance (stats_instance) with metric labels

  The new metric system had a label inherited from legacy: driver_instance.

  This non-structured label has been removed and different driver-specific labels have been added instead, for example:

  Before:

  syslogng_output_events_total{driver_instance="mongodb,localhost:27017,defaultdb,,coll",id="#anon-destination1#1",result="queued"} 4
  

  After:

  syslogng_output_events_total{driver="mongodb",host="localhost:27017",database="defaultdb",collection="coll",id="#anon-destination1#1",result="queued"} 4
  

  This change may affect legacy stats outputs (syslog-ng-ctl stats), for example, persist-name()-based naming is no longer supported in this old format. (#4551)

• APT packages: Added Ubuntu Lunar Lobster and Debian Bookworm support. (#4561)

syslog-ng Discord

For a bit more interactive discussion, join our Discord server:

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andreas Friedmann, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Chuck Silvers, Evan Rempel, Hofi, Kovacs, Gergo Ferenc, László Várady, Romain Tartière, Ryan Faircloth, vostrelt

4.3.0

Read Axoflow's blog post for more details.

Highlights

parallelize() support for pipelines

syslog-ng has traditionally performed processing of log messages arriving from a single connection sequentially. This was done to ensure message ordering as well as most efficient use of CPU on a per message basis. This mode of operation is performing well as long as we have a relatively large number of parallel connections, in which case syslog-ng would use all the CPU cores available in the system.

In case only a small number of connections deliver a large number of messages, this behaviour may become a bottleneck.

With the new parallelization feature, syslog-ng gained the ability to re-partition a stream of incoming messages into a set of partitions, each of which is to be processed by multiple threads in parallel. This does away with ordering guarantees and adds an extra per-message overhead. In exchange it will be able to scale the incoming load to all CPUs in the system, even if coming from a single, chatty sender.

To enable this mode of execution, use the new parallelize() element in your log path:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4));

  # from this part on, messages are processed in parallel even if
  # messages are originally coming from a single connection

  parser { ... };
  destination { ... };
};

The config above will take all messages emitted by the tcp() source and push the work to 4 parallel threads of execution, regardless of how many connections were in use to deliver the stream of messages to the tcp() driver.

parallelize() uses round-robin to allocate messages to partitions by default. You can however retain ordering for a subset of messages with the partition-key() option.

You can use partition-key() to specify a message template. Messages that expand to the same value are guaranteed to be mapped to the same partition.

For example:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4) partition-key("$HOST"));

  # from this part on, messages are processed in parallel if their
  # $HOST value differs. Messages with the same $HOST will be mapped
  # to the same partition and are processed sequentially.

  parser { ... };
  destination { ... };
};

NOTE: parallelize() requires a patched version of libivykis that contains this PR https://github.com/buytenh/ivykis/pull/25. syslog-ng source releases bundle this version of ivykis in their source trees, so if you are building from source, be sure to use the internal version (--with-ivykis=internal). You can also use Axoflow's cloud native container image for syslog-ng, named AxoSyslog (https://github.com/axoflow/axosyslog-docker) which also incorporates this change.

(#3966)

Receiving and sending OpenTelemetry (OTLP) messages

The opentelemetry() source, parser and destination are now available to receive, parse and send OTLP/gRPC messages.

syslog-ng accepts logs, metrics and traces.

The incoming fields are not available through syslog-ng log message name-value pairs for the user by default. This is useful for forwarding functionality (the opentelemetry() destination can access and format them). If such functionality is required, you can configure the opentelemetry() parser, which maps all the fields with some limitations.

The behavior of the opentelemetry() parser is the following:

The name-value pairs always start with .otel. prefix. The type of the message is stored in .otel.type (possible values: log, metric and span). The resource info is mapped to .otel.resource.<...> (e.g.: .otel.resource.dropped_attributes_count, .otel.resource.schema_url ...), the scope info is mapped to .otel.scope.<...> (e.g.: .otel.scope.name, .otel.scope.schema_url, ...).

The fields of log records are mapped to .otel.log.<...> (e.g. .otel.log.body, .otel.log.severity_text, ...).

The fields of metrics are mapped to .otel.metric.<...> (e.g. .otel.metric.name, .otel.metric.unit, ...), the type of the metric is mapped to .otel.metric.data.type (possible values: gauge, sum, histogram, exponential_histogram, summary) with the actual data mapped to .otel.metric.data.<type>.<...> (e.g.: .otel.metric.data.gauge.data_points.0.time_unix_nano, ...).

The fields of traces are mapped to .otel.span.<...> (e.g. .otel.span.name, .otel.span.trace_state, ...).

repeated fields are given an index (e.g. .otel.span.events.5.time_unix_nano).

The mapping of AnyValue type fields is limited. string, bool, int64, double and bytes values are mapped with the respective syslog-ng name-value type (e.g. .otel.resource.attributes.string_key => string_value), however ArrayValue and KeyValueList types are stored serialized with protobuf type. protobuf and bytes types are not directly available for the user, unless an explicit type cast is added (e.g. "bytes(${.otel.log.span_id})") or --include-bytes is passed to name-value iterating template functions (e.g. $(format-json .otel.* --include-bytes), which will base64 encode the bytes content).

Three authentication methods are available in the source auth() block: insecure() (default), tls() and alts(). tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values: required-trusted, required-untrusted, optional-trusted and optional-untrusted) options. ALTS is a simple to use authentication, only available within Google's infrastructure.

The same methods are available in the destination auth() block, with two differences: tls(peer-verify()) is not available, and there is a fourth method, called ADC, which accepts the target-service-account() option, where a list of service accounts can be configured to match against when authenticating the server.

Example configs:

log otel_forward_mode_alts {
  source {
    opentelemetry(
      port(12345)
      auth(alts())
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12345")
      auth(alts())
    );
  };
};

log otel_to_non_otel_insecure {
  source {
    opentelemetry(
      port(12345)
    );
  };

  parser {
    opentelemetry();
  };

  destination {
    network(
      "my-network-server"
      port(12345)
      template("$(format-json .otel.* --shift-levels 1 --include-bytes)\n")
    );
  };
};

log non_otel_to_otel_tls {
  source {
    network(
      port(12346)
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
};

(#4523) (#4510)

Sending messages to CrowdStrike Falcon LogScale (Humio)

The logscale() destination feeds LogScale via the Ingest API.

Minimal config:

destination d_logscale {
  logscale(
    token("my-token")
  );
};

Additional options include:

• url()
• rawstring()
• timestamp()
• timezone()
• attributes()
• extra-headers()
• content-type()

(#4472)

Features

• afmongodb: Bulk MongoDB insert is added via the following options

  • bulk (yes/no) turns on/off bulk insert usage, no forces the old behavior (each log is inserted one by one into the MongoDB)
  • bulk_unordered (yes/no) turns on/off unordered MongoDB bulk operations
  • bulk_bypass_validation (yes/no) turns on/off MongoDB bulk operations validation
  • write_concern (unacked/acked/majority/n > 0) sets write concern mode of the MongoDB operations, both bulk and single

  NOTE: Bulk sending is only efficient if the used collection is constant (e.g. not using templates) or the used template does not lead to too many collections switching within a reasonable time range. (#4483)

• sql: Added 2 new options

  • quote_char to aid custom quoting for table and index names (e.g. MySQL needs sometimes this for certain identifiers) NOTE: Using a back-tick character needs a special formatting as syslog-ng uses it for configuration parameter names, so for that use: quote_char("``") (double back-tick)
  • dbi_driver_dir to define an optional DBI driver location for DBD initialization

  NOTE: libdbi and libdbi-drivers OSE forks are updated, afsql now should work nicely both on ARM and X86 macOS systems too (tested on macOS 13.3.1 and 12.6.4)

  Please do not use the pre-built ones (e.g. 0.9.0 from Homebrew), build from the master of the following

  • https://github.com/balabit-deps/libdbi-drivers
  • https://github.com/balabit-deps/libdbi

  (#4460)

Bugfixes

• network(),syslog(),tcp() destination: fix TCP keepalive

  tcp-keepalive-*() options were broken on the destination side since v3.34.1. (#4559)

• Fixed a hang, which happend when syslog-ng received exremely low CPU time. (#4524)

• $(format-json): Fixed a bug where sometimes an unnecessary comma was added in case of a type cast failure. (#4477)

• Fix flow-control when fetch-limit() is set higher than 64K

  In high-performance use cases, users may configure log-iw-size() and fetch-limit() to be higher than 2^16, which caused flow-control issues, such as messages stuck in the queue forever or log sources not receiving messages. (#4528)

• int32() and int64() type casts: accept hex numbers as proper number representations just as the @NUMBER@ parser within db-parser(). Supporting octal numbers were considered and then rejected as the canonical octal representation for numbers in C would be ambigious: a zero padded decimal number could be erroneously considered octal. I find that log messages contain zero padded decimals more often than octals. (#4535)

• Fixed compilation on platforms where SO_MEMINFO is not available (#4548)

• python: InstantAckTracker, ConsecutiveAckTracker and BatchedAckTracker are now called properly.

  Added proper fake classes for the InstantAckTracker, ConsecutiveAckTracker and BatchedAckTracker classes, and the wapper now calls the super class' constructor. Previusly the super class' constructor was not called which caused the python API to never call into the C API, which's result was that that the callback was never called. (#4549)

• python: Fixed a crash when reloading with a config, which uses a python parser with multiple references. (#4552)

• mqtt(): Fixed the name of the stats instance (mqtt-source) to conform to the standard comma-separated format. (#4551)

Packaging

• scl.conf: The scl.conf file has been moved to /share/syslog-ng/include/scl.conf (#4534)

• C++ plugins: Some of syslog-ng's plugins now contain C++ code.

  By default they are being built if a C++ compiler is available. Disabling it is possible with --disable-cpp.

  Affected plugins:

  • lib/syslog-ng/libexamples.so
    • --disable-cpp will only disable the C++ part (random-choice-generator())
  • lib/syslog-ng/libotel.so

  (#4484)

• debian: A new module is added, called syslog-ng-mod-grpc.

  Its dependencies are: protobuf-compiler, protobuf-compiler-grpc, libprotobuf-dev, libgrpc++-dev. Building the module can be toggled with --enable-grpc. (#4510)

• pcre: syslog-ng now uses pcre2 (8 bit) as a dependency instead of pcre.

  The minimum pcre2 version is 10.0. (#4537)

Notes to developers

• lib/logmsg: Public field LogMessage::protected has been renamed to LogMessage::write_protected.

  Direct usage of this field is discouraged, instead use the following functions:

  • log_msg_is_write_protected()
  • log_msg_write_protect() (#4484)

• lib/templates: Public field LogTemplate::template has been renamed to LogTemplate::template_str. (#4484)

Other changes

• syslog-ng-cfg-db: Moved to a separate repository.

  It is available at: https://github.com/alltilla/syslog-ng-cfg-helper (#4475)

• disk-buffer: Added alternative option names

  disk-buf-size() -> capacity-bytes() qout-size() -> front-cache-size() mem-buf-length() -> flow-control-window-size() mem-buf-size() -> flow-control-window-bytes()

  Old option names are still available.

  Example configs:

  tcp(
    "127.0.0.1" port(2001)
    disk-buffer(
      reliable(yes)
      capacity-bytes(1GiB)
      flow-control-window-bytes(200MiB)
      front-cache-size(1000)
    )
  );
  
  tcp(
    "127.0.0.1" port(2001)
    disk-buffer(
      reliable(no)
      capacity-bytes(1GiB)
      flow-control-window-size(10000)
      front-cache-size(1000)
    )
  );
  

  (#4526)

• selinux: Added RHEL9 support for the selinux policies

  Added RHEL9 support for the selinux policies at contrib/selinux (#4509)

• metrics: replace driver_instance (stats_instance) with metric labels

  The new metric system had a label inherited from legacy: driver_instance.

  This non-structured label has been removed and different driver-specific labels have been added instead, for example:

  Before:

  syslogng_output_events_total{driver_instance="mongodb,localhost:27017,defaultdb,,coll",id="#anon-destination1#1",result="queued"} 4
  

  After:

  syslogng_output_events_total{driver="mongodb",host="localhost:27017",database="defaultdb",collection="coll",id="#anon-destination1#1",result="queued"} 4
  

  This change may affect legacy stats outputs (syslog-ng-ctl stats), for example, persist-name()-based naming is no longer supported in this old format. (#4551)

syslog-ng Discord

For a bit more interactive discussion, join our Discord server:

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Andreas Friedmann, Attila Szakacs, Balazs Scheidler, Bálint Horváth, Chuck Silvers, Evan Rempel, Hofi, Kovacs, Gergo Ferenc, László Várady, Romain Tartière, Ryan Faircloth, vostrelt

4.2.0

Read Axoflow's blog post for more details.

Highlights

Sending messages to Splunk HEC

The splunk-hec-event() destination feeds Splunk via the HEC events API.

Minimal config:

destination d_splunk_hec_event {
  splunk-hec-event(
    url("https://localhost:8088")
    token("70b6ae71-76b3-4c38-9597-0c5b37ad9630")
  );
};

Additional options include:

• event()
• index()
• source()
• sourcetype()
• host()
• time()
• default-index()
• default-source()
• default-sourcetype()
• fields()
• extra-headers()
• extra-queries()
• content-type()

The splunk-hec-raw() destination feeds Splunk via the HEC raw API.

Minimal config:

destination d_splunk_hec_raw {
  splunk-hec-raw(
    url("https://localhost:8088")
    token("70b6ae71-76b3-4c38-9597-0c5b37ad9630")
    channel("05ed4617-f186-4ccd-b4e7-08847094c8fd")
  );
};

(#4462)

Smart multi-line for recognizing backtraces

multi-line-mode(smart): With this multi-line mode, the inherently multi-line data backtrace format is recognized even if they span multiple lines in the input and are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby and Dart.

The regular expressions to recognize these programming languages are specified by an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.

group-lines() parser: this new parser correlates multi-line messages received as separate, but subsequent lines into a single log message. Received messages are first collected into streams related messages (using key()), then collected into correlation contexts up to timeout() seconds. The identification of multi-line messages are then performed on these message contexts within the time period.

  group-lines(key("$FILE_NAME")
              multi-line-mode("smart")
        template("$MESSAGE")
        timeout(10)
        line-separator("\n")
  );

(#4225)

HYPR Audit Trail source

hypr-audit-trail() & hypr-app-audit-trail() source drivers are now available to monitor the audit trails for HYPR applications.

See the README.md file in the driver's directory to see usage information.

(#4175)

ebpf() plugin and reuseport packet randomizer

A new ebpf() plugin was added as a framework to leverage the kernel's eBPF infrastructure to improve performance and scalability of syslog-ng.

Example:

source s_udp {
        udp(so-reuseport(yes) port(2000) persist-name("udp1")
                ebpf(reuseport(sockets(4)))
        );
        udp(so-reuseport(yes) port(2000) persist-name("udp2"));
        udp(so-reuseport(yes) port(2000) persist-name("udp3"));
        udp(so-reuseport(yes) port(2000) persist-name("udp4"));
};

NOTE: The ebpf() plugin is considered advanced usage so its compilation is disabled by default. Please don't use it unless all other avenues of configuration solutions are already tried. You will need a special toolchain and a recent kernel version to compile and run eBPF programs.

(#4365)

Features

• network source: During a TLS handshake, syslog-ng now automatically sets the certificate_authorities field of the certificate request based on the ca-file() and ca-dir() options. The pkcs12-file() option already had this feature. (#4412)

• metrics-probe(): Added level() option to set the stats level of the generated metrics. (#4453)

• metrics-probe(): Added increment() option.

  Users can now set a template, which resolves to a number that modifies the increment of the counter. If not set, the increment is 1. (#4447)

• python: Added support for typed custom options.

  This applies for python source, python-fetcher source, python destination, python parser and python-http-header inner destination.

  Example config:

  python(
    class("TestClass")
    options(
      "string_option" => "example_string"
      "bool_option" => True  # supported values are: True, False, yes, no
      "integer_option" => 123456789
      "double_option" => 123.456789
      "string_list_option" => ["string1", "string2", "string3"]
      "template_option" => LogTemplate("${example_template}")
    )
  );
  

  Breaking change! Previously values were converted to strings if possible, now they are passed to the python class with their real type. Make sure to follow up these changes in your python code! (#4354)

• mongodb destination: Added support for list, JSON and null types. (#4437)

• add-contextual-data(): significantly reduce memory usage for large CSV files. (#4444)

• python(): new LogMessage methods for querying as string and with default values

  • get(key[, default]) Return the value for key if key exists, else default. If default is not given, it defaults to None, so that this method never raises a KeyError.

  • get_as_str(key, default=None, encoding='utf-8', errors='strict', repr='internal'): Return the string value for key if key exists, else default. If default is not given, it defaults to None, so that this method never raises a KeyError.

    The string value is decoded using the codec registered for encoding. errors may be given to set the desired error handling scheme.

    Note that currently repr='internal' is the only available representation. We may implement another more Pythonic representation in the future, so please specify the repr argument explicitly if you want to avoid future representation changes in your code. (#4410)

• kubernetes() source: Added support for json-file logging driver format. (#4419)

• The new $RAWMSG_SIZE hard macro can be used to query the original size of the incoming message in bytes.

  This information may not be available for all source drivers. (#4440)

• syslog-ng configuration identifier

  A new syslog-ng configuration keyword has been added, which allows specifying a config identifier. For example:

  @config-id: cfg-20230404-13-g02b0850fc
  

  This keyword can be used for config identification in managed environments, where syslog-ng instances and their configuration are deployed/generated automatically.

  syslog-ng-ctl config --id can be used to query the active configuration ID and the SHA256 hash of the full "preprocessed" syslog-ng configuration. For example:

  $ syslog-ng-ctl config --id
  cfg-20230404-13-g02b0850fc (08ddecfa52a3443b29d5d5aa3e5114e48dd465e195598062da9f5fc5a45d8a83)
  

  (#4420)

• syslog-ng: add --config-id command line option

  Similarly to --syntax-only, this command line option parses the configuration and then prints its ID before exiting.

  It can be used to query the ID of the current configuration persisted on disk. (#4435)

• Health metrics and syslog-ng-ctl healthcheck

  A new syslog-ng-ctl command has been introduced, which can be used to query a healthcheck status from syslog-ng. Currently, only 2 basic health values are reported.

  syslog-ng-ctl healthcheck --timeout <seconds> can be specified to use it as a boolean healthy/unhealthy check.

  Health checks are also published as periodically updated metrics. The frequency of these checks can be configured with the stats(healthcheck-freq()) option. The default is 5 minutes. (#4362)

• $(format-json) and template functions which support value-pairs expressions: new key transformations upper() and lower() have been added to translate the caps of keys while formatting the output template. For example:

  template("$(format-json test.* --upper)\n")
  

  Would convert all keys to uppercase. Only supports US ASCII. (#4452)

• python(), python-fetcher() sources: Added a mapping for the flags() option.

  The state of the flags() option is mapped to the self.flags variable, which is a Dict[str, bool], for example:

  {
      'parse': True,
      'check-hostname': False,
      'syslog-protocol': True,
      'assume-utf8': False,
      'validate-utf8': False,
      'sanitize-utf8': False,
      'multi-line': True,
      'store-legacy-msghdr': True,
      'store-raw-message': False,
      'expect-hostname': True,
      'guess-timezone': False,
      'header': True,
      'rfc3164-fallback': True,
  }
  

  (#4455)

Metrics

• network(), syslog(): TCP connection metrics

  syslogng_socket_connections{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 3
  syslogng_socket_max_connections{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 10
  syslogng_socket_rejected_connections_total{id="tcp_src#0",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:5555))",direction="input"} 96
  

  internal(): internal_events_queue_capacity metric

  syslog-ng-ctl healthcheck: new healthcheck value syslogng_internal_events_queue_usage_ratio (#4411)

• metrics: new network (TCP, UDP) metrics are available on stats level 1

  # syslog-ng-ctl stats prometheus
  
  syslogng_socket_receive_buffer_used_bytes{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 0
  syslogng_socket_receive_buffer_max_bytes{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 268435456
  syslogng_socket_receive_dropped_packets_total{id="#anon-source0#3",direction="input",driver_instance="afsocket_sd.udp4"} 619173
  
  syslogng_socket_connections{id="#anon-source0#0",direction="input",driver_instance="afsocket_sd.(stream,AF_INET(0.0.0.0:2000))"} 1
  

  (#4374)

• New configuration-related metrics:

  syslogng_last_config_reload_timestamp_seconds 1681309903
  syslogng_last_successful_config_reload_timestamp_seconds 1681309758
  syslogng_last_config_file_modification_timestamp_seconds 1681309877
  

  (#4420)

• destination: Introduced queue metrics.

  • The corresponding driver is identified with the "id" and "driver_instance" labels.
  • Available counters are "memory_usage_bytes" and "events".
  • Memory queue metrics are available with "syslogng_memory_queue_" prefix, disk-buffer metrics are available with "syslogng_disk_queue_" prefix.
  • disk-buffer metrics have an additional "path" label, pointing to the location of the disk-buffer file and a "reliable" label, which can be either "true" or "false".
  • Threaded destinations, like http, python, etc have an additional "worker" label.

  Example metrics

  syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true",worker="0"} 80
  syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00001.rqf",reliable="true",worker="1"} 7
  syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00002.rqf",reliable="true",worker="2"} 7
  syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00003.rqf",reliable="true",worker="3"} 7
  syslogng_disk_queue_events{driver_instance="tcp,localhost:1235",id="d_network_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.qf",reliable="false"} 101
  syslogng_disk_queue_memory_usage_bytes{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true",worker="0"} 3136
  syslogng_disk_queue_memory_usage_bytes{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00001.rqf",reliable="true",worker="1"} 2776
  syslogng_disk_queue_memory_usage_bytes{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00002.rqf",reliable="true",worker="2"} 2760
  syslogng_disk_queue_memory_usage_bytes{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00003.rqf",reliable="true",worker="3"} 2776
  syslogng_disk_queue_memory_usage_bytes{driver_instance="tcp,localhost:1235",id="d_network_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.qf",reliable="false"} 39888
  syslogng_memory_queue_events{driver_instance="http,http://localhost:1236",id="d_http#0",worker="0"} 15
  syslogng_memory_queue_events{driver_instance="http,http://localhost:1236",id="d_http#0",worker="1"} 14
  syslogng_memory_queue_events{driver_instance="tcp,localhost:1234",id="d_network#0"} 29
  syslogng_memory_queue_memory_usage_bytes{driver_instance="http,http://localhost:1236",id="d_http#0",worker="0"} 5896
  syslogng_memory_queue_memory_usage_bytes{driver_instance="http,http://localhost:1236",id="d_http#0",worker="1"} 5552
  syslogng_memory_queue_memory_usage_bytes{driver_instance="tcp,localhost:1234",id="d_network#0"} 11448
  

  (#4392)

• network(), syslog(), file(), http(): new byte-based metrics for incoming/outgoing events

  These metrics show the serialized message sizes (protocol-specific header/framing/etc. length is not included).

  syslogng_input_event_bytes_total{id="s_network#0",driver_instance="tcp,127.0.0.1"} 1925529600
  syslogng_output_event_bytes_total{id="d_network#0",driver_instance="tcp,127.0.0.1:5555"} 565215232
  syslogng_output_event_bytes_total{id="d_http#0",driver_instance="http,http://127.0.0.1:8080/"} 1024
  

  (#4440)

• disk-buffer: Added metrics for monitoring the available space in disk-buffer dir()s.

  Metrics are available from stats(level(1)).

  By default, the metrics are generated every 5 minutes, but it can be changed in the global options:

  options {
    disk-buffer(
      stats(
        freq(10)
      )
    );
  };
  

  Setting freq(0) disabled this feature.

  Example metrics:

  syslogng_disk_queue_dir_available_bytes{dir="/var/syslog-ng"} 870109413376
  

  (#4399)

• disk-buffer: Added metrics for abandoned disk-buffer files.

  Availability is the same as the disk_queue_dir_available_bytes metric.

  Example metrics:

  syslogng_disk_queue_capacity_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 104853504
  syslogng_disk_queue_disk_allocated_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 273408
  syslogng_disk_queue_disk_usage_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 269312
  syslogng_disk_queue_events{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 860
  

  (#4402)

• disk-buffer: Added capacity, disk_allocated and disk_usage metrics.

  • "capacity_bytes": The theoretical maximal useful size of the disk-buffer. This is always smaller, than disk-buf-size(), as there is some reserved space for metadata. The actual full disk-buffer file can be larger than this, as syslog-ng allows to write over this limit once, at the end of the file.

  • "disk_allocated_bytes": The current size of the disk-buffer file on the disk. Please note that the disk-buffer file size does not strictly correlate with the number of messages, as it is a ring buffer implementation, and also syslog-ng optimizes the truncation of the file for performance reasons.

  • "disk_usage_bytes": The serialized size of the queued messages in the disk-buffer file. This counter is useful for calculating the disk usage percentage (disk_usage_bytes / capacity_bytes) or the remaining available space (capacity_bytes - disk_usage_bytes).

  Example metrics:

  syslogng_disk_queue_capacity_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 104853504
  syslogng_disk_queue_disk_allocated_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 17284
  syslogng_disk_queue_disk_usage_bytes{driver_id="d_network#0",driver_instance="tcp,localhost:1235",path="/var/syslog-ng-00000.rqf",reliable="true"} 13188
  

  (#4356)

• kubernetes(): Added input_events_total and input_event_bytes_total metrics.

  syslogng_input_events_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 25
  syslogng_input_event_bytes_total{cluster="k8s",driver="kubernetes",id="#anon-source0",namespace="default",pod="log-generator-1682517834-7797487dcc-49hqc"} 1859
  

  (#4447)

Bugfixes

• pdbtool test: fix two type validation bugs:

  1. When pdbtool test validates the type information associated with a name-value pair, it was using string comparisons, which didn't take type aliases into account. This is now fixed, so that "int", "integer" or "int64" can all be used to mean the same type.

  2. When type information is missing from a <test_value/> tag, don't validate it against "string", rather accept any extracted type.

  In addition to these fixes, a new alias "integer" was added to mean the same as "int", simply because syslog-ng was erroneously using this term when reporting type information in its own messages. (#4405)

• $(format-json): fix RFC8259 number violation

  $(format-json) produced invalid JSON output when it contained numeric values with leading zeros or + signs. This has been fixed. (#4415)

• grouping-by(): fix persist-name() option not taken into account (#4390)

• python(), db-parser(), grouping-by(), add-contextual-data(): fix typing compatibility with <4.0 config versions (#4394)

• python: Fixed a crash which occurred at reloading after registering a confgen plugin. (#4459)

• date-parser(): fix %z when system timezone has no daylight saving time (#4401)

• Consider messages consumed into correlation states "matching": syslog-ng's correlation functionality (e.g. grouping-by() or db-parser() with such rules) drop individual messages as they are consumed into a correlation contexts and you are using inject-mode(aggregate-only). This is usually happens because you are only interested in the combined message and not in those that make up the combination. However, if you are using correlation with conditional processing (e.g. if/elif/else or flags(final)), such messages were erroneously considered as unmatching, causing syslog-ng to take the alternative branch.

  Example:

  With a configuration similar to this, individual messages are consumed into a correlation state and dropped by grouping-by():

  log {
      source(...);
  
      if {
          grouping-by(... inject-mode(aggregate-only));
      } else {
          # alternative branch
      };
  };
  

  The bug was that these individual messages also traverse the else branch, even though they were successfully processed with the inclusion into the correlation context. This is not correct. The bugfix changes this behaviour. (#4370)

• netmask6(): fix crash when user specifies too long mask (#4429)

• afprog: Fixed possible freezing on some OSes (#4438)

• network(), syslog(), syslog-parser(): fix null termination of SDATA param names (#4429)

• python(): fix LogMessage subscript not raising KeyError on non-existent keys

  When message fields were queried (msg["key"]) and the given key did not exist, None or an empty string was returned (depending on the version of the config).

  Neither was correct, now a KeyError occurs in such cases. (#4410)

• $(python): fix template function prefix being overwritten when using datetime types (#4410)

• disk-buffer: Fixed queued messages stats counting, when a disk-buffer became corrupted. (#4385)

• $(format-json): fix escaping control characters

  $(format-json) produced invalid JSON output when a string value contained control characters. (#4417)

• disk-buffer(): fix deinitialization when starting syslog-ng with invalid configuration (#4418)

• python(): fix exception handling when LogMessage value conversion fails (#4410)

• json-parser(): Fixed parsing non-string arrays.

  syslog-ng now no longer parses non-string arrays to list of strings, losing the original type information of the array's elements. (#4396)

• disk-buffer: Fixed a rare race condition when calculating disk-buffer filename. (#4381)

• python-persist: fix off-by-one overflow (#4429)

Packaging

• The --with-python-venv-dir=path configure option can be used to modify the location of syslog-ng's venv. The default is still ${localstatedir}/python-venv. (#4465)

Other changes

• The sdata-prefix() option does not accept values longer than 128 characters. (#4429)

• grouping-by(): Remove setting of the ${.classifier.context_id} name-value pair in all messages consumed into a correlation context. This functionality is inherited from db-parser() and has never been documented for grouping-by(), has of limited use, and any uses can be replaced by the use of the built-in macro named $CONTEXT_ID. Modifying all consumed messages this way has significant performance consequences for grouping-by() and removing it outweighs the small incompatibility this change introduces. The similar functionality in db-parser() correlation is not removed with this change. (#4424)

• config: Added internal() option to sources, destinations, parsers and rewrites.

  Its main usage is in SCL blocks. Drivers configured with internal(yes) register their metrics on level 3. This makes developers of SCLs able to create metrics manually with metrics-probe() and "disable" every other metrics, they do not need. (#4451)

• The following Prometheus metrics have been renamed:

  log_path_{in,e}gress -> route_{in,e}gress_total internal_source -> internal_events_total

  The internal_queue_length stats counter has been removed. It was deprecated since syslog-ng 3.29. (#4411)

syslog-ng Discord

For a bit more interactive discussion, join our Discord server:

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Alex Becker, Attila Szakacs, Balazs Scheidler, Hofi, László Várady, Muhammad Shanif, Ricfilipe, Romain Tartière

4.1.1

This is the combination of the news entries of 4.1.0 and 4.1.1. 4.1.1 hotfixed a grouping-by() and db-parser() related crash.

Highlights

PROXY protocol v2 support (#4211)

We've added support for PROXY protocol v2 (transport(proxied-tcp)), a protocol used by network load balancers, such as Amazon Elastic Load Balancer and HAProxy, to carry original source/destination address information, as described in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Metrics revised

Prometheus metric format (#4325)

A new metric system has been introduced to syslog-ng, where metrics are identified by names and partitioned by labels, which is similar to the Prometheus data model.

The syslog-ng-ctl stats prometheus command can be used to query syslog-ng metrics in a format that conforms to the Prometheus text-based exposition format.

syslog-ng-ctl stats prometheus --with-legacy-metrics displays legacy metrics as well. Legacy metrics do not follow Prometheus' metric and label conventions.

Classification (metadata-based metrics) (#4318)

metrics-probe(), a new parser has also been added, which counts messages passing through based on the metadata of each message. The parser creates labeled metrics based on the fields of the message.

Both the key and labels can be set in the config, the values of the labels can be templated. E.g.:

parser p_metrics_probe {
  metrics-probe(
    key("custom_key")  # adds "syslogng_" prefix => "syslogng_custom_key"
    labels(
      "custom_label_name_1" => "foobar"
      "custom_label_name_2" => "${.custom.field}"
    )
  );
};

With this config, it creates counters like these:

syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3

The minimal config creates counters with the key syslogng_classified_events_total and labels app, host, program and source. E.g.:

parser p_metrics_probe {
  metrics-probe();
};

With this config, it creates counters like these:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1

Named log paths (path ingress/egress metrics) (#4344)

It is also possible to create named log paths, for example:

log top-level {
    source(s_local);

    log inner-1 {
        filter(f_inner_1);
        destination(d_local_1);
    };

    log inner-2 {
        filter(f_inner_2);
        destination(d_local_2);
    };
};

Each named log path counts its ingress and egress messages:

syslogng_log_path_ingress{id="top-level"} 114
syslogng_log_path_ingress{id="inner-1"} 114
syslogng_log_path_ingress{id="inner-2"} 114
syslogng_log_path_egress{id="top-level"} 103
syslogng_log_path_egress{id="inner-1"} 62
syslogng_log_path_egress{id="inner-2"} 41

Note that the egress statistics only count the messages which have been have not been filtered out from the related log path, it does care about whether there are any destinations in it or that any destination delivers or drops the message.

The above three features are experimental; the output of stats prometheus (names, labels, etc.) and the metrics created by metrics-probe() and named log paths may change in the next 2-3 releases.

Features

• $(format-date): add a new template function to format time and date values

  $(format-date [options] format-string [timestamp])

  $(format-date) takes a timestamp in the DATETIME representation and formats it according to an strftime() format string. The DATETIME representation in syslog-ng is a UNIX timestamp formatted as a decimal number, with an optional fractional part, where the seconds and the fraction of seconds are separated by a dot.

  If the timestamp argument is missing, the timestamp of the message is used.

  Options: --time-zone <TZstring> -- override timezone of the original timestamp (#4202)

• syslog-parser() and all syslog related sources: accept unquoted RFC5424 SD-PARAM-VALUEs instead of rejecting them with a parse error.

  sdata-parser(): this new parser allows you to parse an RFC5424 style structured data string. It can be used to parse this relatively complex format separately. (#4281)

• system() source: the system() source was changed on systemd platforms to fetch journal messages that relate to the current boot only (e.g. similar to journalctl -fb) and to ignore messages generated in previous boots, even if those messages were succesfully stored in the journal and were not picked up by syslog-ng. This change was implemented as the journald access APIs work incorrectly if time goes backwards across reboots, which is an increasingly frequent event in virtualized environments and on systems that lack an RTC. If you want to retain the old behaviour, please bypass the system() source and use systemd-journal() directly, where this option can be customized. The change is not tied to @version as we deemed the new behaviour fixing an actual bug. For more information consult #2836.

  systemd-journald() source: add match-boot() and matches() options to allow you to constrain the collection of journal records to a subset of what is in the journal. match-boot() is a yes/no value that allows you to fetch messages that only relate to the current boot. matches() allows you to specify one or more filters on journal fields.

  Examples:

  source s_journal_current_boot_only {
    systemd-source(match-boot(yes));
  };
  
  source s_journal_systemd_only {
    systemd-source(matches(
      "_COMM" => "systemd"
      )
    );
  };
  

  (#4245)

• date-parser(): add value() parameter to instruct date-parser() to store the resulting timestamp in a name-value pair, instead of changing the timestamp value of the LogMessage.

  datetime type representation: typed values in syslog-ng are represented as strings when stored as a part of a log message. syslog-ng simply remembers the type it was stored as. Whenever the value is used as a specific type in a type-aware context where we need the value of the specific type, an automatic string parsing takes place. This parsing happens for instance whenever syslog-ng stores a datetime value in MongoDB or when $(format-date) template function takes a name-value pair as parameter. The datetime() type has stored its value as the number of milliseconds since the epoch (1970-01-01 00:00:00 GMT). This has now been enhanced by making it possible to store timestamps up to nanosecond resolutions along with an optional timezone offset.

  $(format-date): when applied to name-value pairs with the datetime type, use the timezone offset if one is available. (#4319)

• stats: Added syslog-stats() global stats() group option.

  E.g.:

  options {
    stats(
      syslog-stats(no);
    );
  };
  

  It changes the behavior of counting messages based on different syslog-proto fields, like SEVERITY, FACILITY, HOST, etc...

  Possible values are:

  • yes => force enable
  • no => force disable
  • auto => let stats(level()) decide (old behavior) (#4337)

• kubernetes source: Added key-delimiter() option.

  Some metadata fields can contain .-s in their name. This does not work with syslog-ng-s macros, which by default use . as a delimiter. The added key-delimiter() option changes this behavior by storing the parsed metadata fields with a custom delimiter. In order to reach the fields, the accessor side has to use the new delimiter format, e.g. --key-delimiter option in $(format-json). (#4213)

Bugfixes

• Fix conditional evaluation with a dangling filter

  We've fixed a bug that caused conditional evaluation (if/else/elif) and certain logpath flags (final, fallback) to occasionally malfunction. The issue only happened in certain logpath constructs; examples can be found in the PR description. (#4058)

• python: Fixed a bug, where PYTHONPATH was ignored with python3.11. (#4298)

• disk-buffer: Fixed disk-queue file becoming corrupt when changing disk-buf-size().

  syslog-ng now continues with the originally set disk-buf-size(). Note that changing the disk-buf-size() of an existing disk-queue was never supported, but could cause errors, which are fixed now. (#4308)

• dqtool: fix dqtool assign (#4355)

• example-diskq-source: Fixed failing to read the disk-queue content in some cases. (#4308)

• default-network-drivers(): Added support for the log-iw-size() option with a default value of 1000. Making it possible to adjust the log-iw-size() for the TCP/TLS based connections, when changing the max-connections() option. (#4328)

• apache-accesslog-parser(): fix rawrequest escaping binary characters (#4303)

• dqtool: Fixed dqtool cat failing to read the content in some cases. (#4308)

• Fixed a rare main loop related crash on FreeBSD. (#4262)

• Fix a warning message that was displayed incorrectly: "The actual number of worker threads exceeds the number of threads estimated at startup." (#4282)

• Fix minor memory leak related to tznames (#4334)

• db-parser(), grouping-by(): Fixed a crash introduced in 4.1.0. (#4366)

Packaging

• dbparser: libdbparser.so has been renamed to libcorrelation.so. (#4294)
• systemd-journal: Fixed a linker error, which occurred, when building with --with-systemd-journal=optional. (#4304) (#4302)

Notes to developers

• LogThreadedSourceDriver and Fetcher: implement source-side batching support on the input path by assigning a thread_id to dynamically spawned input threads (e.g. those spawned by LogThreadedSourceDriver) too. To actually improve performance the source driver should disable automatic closing of batches by setting auto_close_batches to FALSE and calling log_threaded_source_close_batch() explicitly. (#3969)

Other changes

• stats related options: The stats related options have been groupped to a new stats() block.

  This affects the following global options:

  • stats-freq()
  • stats-level()
  • stats-lifetime()
  • stats-max-dynamics()

  These options have been kept for backward compatibility, but they have been deprecated.

  Migrating from the old stats options to the new ones looks like this.

  @version: 4.0
  
  options {
      stats-freq(1);
      stats-level(1);
      stats-lifetime(1000);
      stats-max-dynamics(10000);
  };
  

  @version: 4.1
  
  options {
      stats(
          freq(1)
          level(1)
          lifetime(1000)
          max-dynamics(10000)
      );
  };
  

  Breaking change For more than a decade stats() was a deprecated alias to stats-freq(), now it is used as the name of the new block. If you have been using stats(xy), use stats(freq(xy)) instead. (#4337)

• kubernetes source: Improved error logging, when the pod was unreachable through the python API. (#4305)

• APT repository: Added .gz, .xz and .bz2 compression to the Packages file. (#4313)

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Attila Szakacs, Balazs Scheidler, Bálint Horváth, Gergo Ferenc Kovacs, Hofi, László Várady, Ronny Meeus, Szilard Parrag

4.1.0

Highlights

PROXY protocol v2 support (#4211)

We've added support for PROXY protocol v2 (transport(proxied-tcp)), a protocol used by network load balancers, such as Amazon Elastic Load Balancer and HAProxy, to carry original source/destination address information, as described in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Metrics revised

Prometheus metric format (#4325)

A new metric system has been introduced to syslog-ng, where metrics are identified by names and partitioned by labels, which is similar to the Prometheus data model.

The syslog-ng-ctl stats prometheus command can be used to query syslog-ng metrics in a format that conforms to the Prometheus text-based exposition format.

syslog-ng-ctl stats prometheus --with-legacy-metrics displays legacy metrics as well. Legacy metrics do not follow Prometheus' metric and label conventions.

Classification (metadata-based metrics) (#4318)

metrics-probe(), a new parser has also been added, which counts messages passing through based on the metadata of each message. The parser creates labeled metrics based on the fields of the message.

Both the key and labels can be set in the config, the values of the labels can be templated. E.g.:

parser p_metrics_probe {
  metrics-probe(
    key("custom_key")  # adds "syslogng_" prefix => "syslogng_custom_key"
    labels(
      "custom_label_name_1" => "foobar"
      "custom_label_name_2" => "${.custom.field}"
    )
  );
};

With this config, it creates counters like these:

syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3

The minimal config creates counters with the key syslogng_classified_events_total and labels app, host, program and source. E.g.:

parser p_metrics_probe {
  metrics-probe();
};

With this config, it creates counters like these:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1

Named log paths (path ingress/egress metrics) (#4344)

It is also possible to create named log paths, for example:

log top-level {
    source(s_local);

    log inner-1 {
        filter(f_inner_1);
        destination(d_local_1);
    };

    log inner-2 {
        filter(f_inner_2);
        destination(d_local_2);
    };
};

Each named log path counts its ingress and egress messages:

syslogng_log_path_ingress{id="top-level"} 114
syslogng_log_path_ingress{id="inner-1"} 114
syslogng_log_path_ingress{id="inner-2"} 114
syslogng_log_path_egress{id="top-level"} 103
syslogng_log_path_egress{id="inner-1"} 62
syslogng_log_path_egress{id="inner-2"} 41

Note that the egress statistics only count the messages which have been have not been filtered out from the related log path, it does care about whether there are any destinations in it or that any destination delivers or drops the message.

The above three features are experimental; the output of stats prometheus (names, labels, etc.) and the metrics created by metrics-probe() and named log paths may change in the next 2-3 releases.

Features

• $(format-date): add a new template function to format time and date values

  $(format-date [options] format-string [timestamp])

  $(format-date) takes a timestamp in the DATETIME representation and formats it according to an strftime() format string. The DATETIME representation in syslog-ng is a UNIX timestamp formatted as a decimal number, with an optional fractional part, where the seconds and the fraction of seconds are separated by a dot.

  If the timestamp argument is missing, the timestamp of the message is used.

  Options: --time-zone <TZstring> -- override timezone of the original timestamp (#4202)

• syslog-parser() and all syslog related sources: accept unquoted RFC5424 SD-PARAM-VALUEs instead of rejecting them with a parse error.

  sdata-parser(): this new parser allows you to parse an RFC5424 style structured data string. It can be used to parse this relatively complex format separately. (#4281)

• system() source: the system() source was changed on systemd platforms to fetch journal messages that relate to the current boot only (e.g. similar to journalctl -fb) and to ignore messages generated in previous boots, even if those messages were succesfully stored in the journal and were not picked up by syslog-ng. This change was implemented as the journald access APIs work incorrectly if time goes backwards across reboots, which is an increasingly frequent event in virtualized environments and on systems that lack an RTC. If you want to retain the old behaviour, please bypass the system() source and use systemd-journal() directly, where this option can be customized. The change is not tied to @version as we deemed the new behaviour fixing an actual bug. For more information consult #2836.

  systemd-journald() source: add match-boot() and matches() options to allow you to constrain the collection of journal records to a subset of what is in the journal. match-boot() is a yes/no value that allows you to fetch messages that only relate to the current boot. matches() allows you to specify one or more filters on journal fields.

  Examples:

  source s_journal_current_boot_only {
    systemd-source(match-boot(yes));
  };
  
  source s_journal_systemd_only {
    systemd-source(matches(
      "_COMM" => "systemd"
      )
    );
  };
  

  (#4245)

• date-parser(): add value() parameter to instruct date-parser() to store the resulting timestamp in a name-value pair, instead of changing the timestamp value of the LogMessage.

  datetime type representation: typed values in syslog-ng are represented as strings when stored as a part of a log message. syslog-ng simply remembers the type it was stored as. Whenever the value is used as a specific type in a type-aware context where we need the value of the specific type, an automatic string parsing takes place. This parsing happens for instance whenever syslog-ng stores a datetime value in MongoDB or when $(format-date) template function takes a name-value pair as parameter. The datetime() type has stored its value as the number of milliseconds since the epoch (1970-01-01 00:00:00 GMT). This has now been enhanced by making it possible to store timestamps up to nanosecond resolutions along with an optional timezone offset.

  $(format-date): when applied to name-value pairs with the datetime type, use the timezone offset if one is available. (#4319)

• stats: Added syslog-stats() global stats() group option.

  E.g.:

  options {
    stats(
      syslog-stats(no);
    );
  };
  

  It changes the behavior of counting messages based on different syslog-proto fields, like SEVERITY, FACILITY, HOST, etc...

  Possible values are:

  • yes => force enable
  • no => force disable
  • auto => let stats(level()) decide (old behavior) (#4337)

• kubernetes source: Added key-delimiter() option.

  Some metadata fields can contain .-s in their name. This does not work with syslog-ng-s macros, which by default use . as a delimiter. The added key-delimiter() option changes this behavior by storing the parsed metadata fields with a custom delimiter. In order to reach the fields, the accessor side has to use the new delimiter format, e.g. --key-delimiter option in $(format-json). (#4213)

Bugfixes

• Fix conditional evaluation with a dangling filter

  We've fixed a bug that caused conditional evaluation (if/else/elif) and certain logpath flags (final, fallback) to occasionally malfunction. The issue only happened in certain logpath constructs; examples can be found in the PR description. (#4058)

• python: Fixed a bug, where PYTHONPATH was ignored with python3.11. (#4298)

• disk-buffer: Fixed disk-queue file becoming corrupt when changing disk-buf-size().

  syslog-ng now continues with the originally set disk-buf-size(). Note that changing the disk-buf-size() of an existing disk-queue was never supported, but could cause errors, which are fixed now. (#4308)

• dqtool: fix dqtool assign (#4355)

• example-diskq-source: Fixed failing to read the disk-queue content in some cases. (#4308)

• default-network-drivers(): Added support for the log-iw-size() option with a default value of 1000. Making it possible to adjust the log-iw-size() for the TCP/TLS based connections, when changing the max-connections() option. (#4328)

• apache-accesslog-parser(): fix rawrequest escaping binary characters (#4303)

• dqtool: Fixed dqtool cat failing to read the content in some cases. (#4308)

• Fixed a rare main loop related crash on FreeBSD. (#4262)

• Fix a warning message that was displayed incorrectly: "The actual number of worker threads exceeds the number of threads estimated at startup." (#4282)

• Fix minor memory leak related to tznames (#4334)

Packaging

• dbparser: libdbparser.so has been renamed to libcorrelation.so. (#4294)
• systemd-journal: Fixed a linker error, which occurred, when building with --with-systemd-journal=optional. (#4304) (#4302)

Notes to developers

• LogThreadedSourceDriver and Fetcher: implement source-side batching support on the input path by assigning a thread_id to dynamically spawned input threads (e.g. those spawned by LogThreadedSourceDriver) too. To actually improve performance the source driver should disable automatic closing of batches by setting auto_close_batches to FALSE and calling log_threaded_source_close_batch() explicitly. (#3969)

Other changes

• stats related options: The stats related options have been groupped to a new stats() block.

  This affects the following global options:

  • stats-freq()
  • stats-level()
  • stats-lifetime()
  • stats-max-dynamics()

  These options have been kept for backward compatibility, but they have been deprecated.

  Migrating from the old stats options to the new ones looks like this.

  @version: 4.0
  
  options {
      stats-freq(1);
      stats-level(1);
      stats-lifetime(1000);
      stats-max-dynamics(10000);
  };
  

  @version: 4.1
  
  options {
      stats(
          freq(1)
          level(1)
          lifetime(1000)
          max-dynamics(10000)
      );
  };
  

  Breaking change For more than a decade stats() was a deprecated alias to stats-freq(), now it is used as the name of the new block. If you have been using stats(xy), use stats(freq(xy)) instead. (#4337)

• kubernetes source: Improved error logging, when the pod was unreachable through the python API. (#4305)

• APT repository: Added .gz, .xz and .bz2 compression to the Packages file. (#4313)

Credits

syslog-ng is developed as a community project, and as such it relies on volunteers, to do the work necessarily to produce syslog-ng.

Reporting bugs, testing changes, writing code or simply providing feedback are all important contributions, so please if you are a user of syslog-ng, contribute.

We would like to thank the following people for their contribution:

Attila Szakacs, Balazs Scheidler, Bálint Horváth, Gergo Ferenc Kovacs, Hofi, László Várady, Ronny Meeus, Szilard Parrag