Fancy stream processing made operationally mundane
For installation instructions check out the getting started guide.
nats_kv
cache type.nats_jetstream
input now supports last_per_subject
and new
deliver fallbacks.error_patterns
added to the drop_on
output.redis_scan
input type.auto_replay_nacks
added to all inputs that traditionally automatically retry nacked messages as a toggle for this behaviour.retry
processor.noop
cache.targets_input
added to the azure_blob_storage
input.reject_errored
output.nats_request_reply
processor.json_documents
scanner.unarchive
processor no longer yields linting errors when the format csv:x
is specified. This is a regression introduced in v4.25.0.sftp
input will no longer consume files when the watcher cache returns an error. Instead, it will reattempt the file upon the next poll.aws_sqs
input no longer logs error level logs for visibility timeout refreshing errors.nats_kv
processor now allows nats wildcards for the keys
operation.nats_kv
processor keys
operation now returns a single message with an array of found keys instead of a batch of messages.nats_kv
processor history
operation now returns a single message with an array of objects containing the record fields instead of a batch of messages.timeout
added to the nats_kv
processor to specify the maximum period to wait on an operation before aborting and returning an error.>
, <
, <=
, >=
) now match the precision of the compared integers when applicable.parse_form_url_encoded
Bloblang method no longer produces results with an unknown data type for repeated query parameters.echo
CLI command no longer fails to sanitise configs when encountering an empty password
field.split
processor (or custom plugins) no longer results in downstream error handling loops around nacks. This was previously implemented as a feature to ensure unbounded expanded and split batches don't flood downstream services in the event of a minority of errors. However, introducing more clever origin tracking of errored messages has eliminated the need for this undocumented behaviour.The full change log can be found here.
For installation instructions check out the getting started guide.
credit
added to the amqp_1
input to specify the maximum number of unacknowledged messages the sender can transmit.if
statements.sql
cache.batch_size
, sort
and limit
added to the mongodb
input.idemponent_write
added to the kafka
output.amqp_1.credit
input has changed from 1
to 64
.mongodb
processor and output now support extended JSON in canonical form for document, filter and hint mappings.open_telemetry_collector
tracer has had the url
field of gRPC and HTTP collectors deprecated in favour of address
, which more accurately describes the intended format of endpoints. The old style will continue to work, but eventually will have its default value removed and an explicit value will be required.%
characters were being incorrectly parsed during unit test execution. This was a regression introduced in v4.25.0.%
characters were being incorrectly parsed. This was a regression introduced in v4.25.0.The full change log can be found here.
For installation instructions check out the getting started guide.
The full change log can be found here.
For installation instructions check out the getting started guide.
address_cache
added to the socket_server
input.read_header
added to the amqp_1
input.codec
field now support a new field scanner
to replace it. Scanners are more powerful as they are configured in a structured way similar to other component types rather than via a single string field, for more information check out the scanners page.diff
and patch
Bloblang methods.processors
processor.read_header
added to the amqp_1
input./debug/pprof/allocs
has been added for profiling allocations.cockroachdb_changefeed
input.open_telemetry_collector
tracer now supports sampling.aws_kinesis
input and output now support specifying ARNs as the stream target.azure_cosmosdb
input, processor and output.sql_*
components now support the gocosmos
driver.opensearch
output.javascript
processor now handles module imports correctly.if
statements now provide explicit errors when query expressions resolve to non-boolean values.amqp_1
input were always empty due to type mismatch, this should no longer be the case.zip
Bloblang method no longer fails when executed without arguments.amqp_0_9
output no longer prints bogus exchange name when connecting to the server.generate
input no longer adds an extra second to interval: '@every x'
syntax.nats_jetstream
input no longer fails to locate mirrored streams.period
, where data arrives in low volumes and is sporadic.parse_parquet
Bloblang function, parquet_decode
, parquet_encode
processors and the parquet
input have all been upgraded to the latest version of the underlying Parquet library. Since this underlying library is experimental it is likely that behaviour changes will result. One significant change is that encoding numerical values that are larger than the column type (float64
into FLOAT
, int64
into INT32
, etc) will no longer be automatically converted.parse_log
processor field codec
is now deprecated.The full change log can be found here.
For installation instructions check out the getting started guide.
NOTE: This is a release candidate, you can download a binary from this page or pull a docker image from https://github.com/benthosdev/benthos/pkgs/container/benthos with the specific tag of the release candidate.
address_cache
added to the socket_server
input.read_header
added to the amqp_1
input.codec
field now support a new field scanner
to replace it. Scanners are more powerful as they are configured in a structured way similar to other component types rather than via a single string field, for more information check out the scanners page.diff
and patch
Bloblang methods.processors
processor.read_header
added to the amqp_1
input./debug/pprof/allocs
has been added for profiling allocations.cockroachdb_changefeed
input.open_telemetry_collector
tracer now supports sampling.aws_kinesis
input and output now support specifying ARNs as the stream target.javascript
processor now handles module imports correctly.if
statements now provide explicit errors when query expressions resolve to non-boolean values.amqp_1
input were always empty due to type mismatch, this should no longer be the case.zip
Bloblang method no longer fails when executed without arguments.amqp_0_9
output no longer prints bogus exchange name when connecting to the server.generate
input no longer adds an extra second to interval: '@every x'
syntax.nats_jetstream
input no longer fails to locate mirrored streams.period
, where data arrives in low volumes and is sporadic.parse_parquet
Bloblang function, parquet_decode
, parquet_encode
processors and the parquet
input have all been upgraded to the latest version of the underlying Parquet library. Since this underlying library is experimental it is likely that behaviour changes will result. One significant change is that encoding numerical values that are larger than the column type (float64
into FLOAT
, int64
into INT32
, etc) will no longer be automatically converted.parse_log
processor field codec
is now deprecated.The full change log can be found here.
For installation instructions check out the getting started guide.
address_cache
added to the socket_server
input.read_header
added to the amqp_1
input.codec
field now support a new field scanner
to replace it. Scanners are more powerful as they are configured in a structured way similar to other component types rather than via a single string field, for more information check out the scanners page.diff
and patch
Bloblang methods.processors
processor.read_header
added to the amqp_1
input./debug/pprof/allocs
has been added for profiling allocations.cockroachdb_changefeed
input.javascript
processor now handles module imports correctly.if
statements now provide explicit errors when query expressions resolve to non-boolean values.amqp_1
input were always empty due to type mismatch, this should no longer be the case.zip
Bloblang method no longer fails when executed without arguments.amqp_0_9
output no longer prints bogus exchange name when connecting to the server.generate
input no longer adds an extra second to interval: '@every x'
syntax.nats_jetstream
input no longer fails to locate mirrored streams.period
, where data arrives in low volumes and is sporadic.parse_parquet
Bloblang function, parquet_decode
, parquet_encode
processors and the parquet
input have all been upgraded to the latest version of the underlying Parquet library. Since this underlying library is experimental it is likely that behaviour changes will result. One significant change is that encoding numerical values that are larger than the column type (float64
into FLOAT
, int64
into INT32
, etc) will no longer be automatically converted.parse_log
processor field codec
is now deprecated.The full change log can be found here.
For installation instructions check out the getting started guide.
idempotent_write
added to the kafka_franz
output.idle_timeout
added to the read_until
input.delay_seconds
added to the aws_sqs
output.discard_unknown
and use_proto_names
added to the protobuf
processors.root = ...
.protobuf
processor now supports targetting nested types from proto files.schema_registry_encode
and schema_registry_decode
processors should no longer double escape URL unsafe characters within subjects when querying their latest versions.The full change log can be found here.
For installation instructions check out the getting started guide.
amqp_0_9
output now supports dynamic interpolation functions within the exchange
field.custom_topic_creation
added to the kafka
output.ts_sub
.abs
now supports integers in and integers out.extract_tracing_map
field added to the nats
, nats_jetstream
and nats_stream
inputs.inject_tracing_map
field added to the nats
, nats_jetstream
and nats_stream
outputs._fail_fast
variants for the broker
output fan_out
and fan_out_sequential
patterns.summary_quantiles_objectives
added to the prometheus
metrics exporter.metric
processor now supports floating point values for counter_by
and gauge
types.log/slog
loggers injected into a stream builder via StreamBuilder.SetLogger
should now respect formatting strings.kafka_franz
input now provides properly typed metadata values.trino
driver for the various sql_*
components no longer panics when trying to insert nulls.http_client
input no longer sends a phantom request body on subsequent requests when an empty payload
is specified.schema_registry_encode
and schema_registry_decode
processors should no longer fail to obtain schemas containing slashes (or other URL path unfriendly characters).parse_log
processor no longer extracts structured fields that are incompatible with Bloblang mappings.float32
values.The full change log can be found here.
For installation instructions check out the getting started guide.
-e/--env-file
cli flag for importing environment variable files now supports glob patterns.-e/--env-file
cli flags now support triple quoted strings.counter
function added to Bloblang. It is recommended that this function, although experimental, should be used instead of the now deprecated count
function.schema_registry_encode
and schema_registry_decode
processors now support JSONSchema.metadata
added to the nats
and nats_jetstream
outputs.cached
processor field ttl
now supports interpolation functions.amqp_0_9
output.command
added to the redis_list
input and output.generate
input with a descriptor interval (@hourly
, etc) had a chance of firing twice.redis_streams
input that is rejected from read attempts enters a reconnect loop without backoff.sqs
input now periodically refreshes the visibility timeout of messages that take a significant amount of time to process.ts_add_iso8601
and ts_sub_iso8601
bloblang methods now return the correct error for certain invalid durations.discord
output no longer ignores structured message fields containing underscores.kafka_franz
input was ignoring batching periods and stalling.random_int
Bloblang function now prevents instantiations where either the max
or min
arguments are dynamic. This is in order to avoid situations where the random number generator is re-initialised across subsequent mappings in a way that surprises map authors.The full change log can be found here.
For installation instructions check out the getting started guide.
client_id
and rack_id
added to the kafka_franz
input and output.command
processor.no_cache
added to the file
and env
Bloblang functions.file_rel
function added to Bloblang.endpoint_params
added to the oauth2
section of HTTP client components.azure_blob_storage
input no longer adds blob_storage_content_type
and blob_storage_content_encoding
metadata values as string pointer types, and instead adds these values as string types only when they are present.http_server
input now returns a more appropriate 503 service unavailable status code during shutdown instead of the previous 404 status.pusher
output that was never initialised.sftp
output now reconnects upon being disconnected by the Azure idle timeout.switch
output now produces error logs when messages do not pass at least one case with strict_mode
enabled, previously these rejected messages were potentially re-processed in a loop without any logs depending on the config. An inaccuracy to the documentation has also been fixed in order to clarify behaviour when strict mode is not enabled.log
processor fields_mapping
field should no longer reject metadata queries using @
syntax.zip
method no longer produces values that yield an "Unknown data type".The full change log can be found here.