Features

• Re-adds the fetch_read_strategy cluster config property to select between polling and non-polling fetch implementations. Uses the non-polling fetch implementation by default. by @StephanDollberg in #18176
• #18163 rpk container start: now starts a Redpanda Console container connected with the cluster. by @r-vasquez in #18164
• rpk container now has a set of flags to specify ports for node to start on. by @r-vasquez in #18148

Bug Fixes

• Fix a bug validating WebAssembly when global constants are specific values that have the encoded byte 0x0B. by @rockwotj in #18108
• Fix a bug where an invalid buffer passed into the WebAssembly host from the guest could cause Redpanda to abort. by @rockwotj in #18234
• Fix a scenario where list_offset with a timestamp could return a lower offset than partition start after a trim-prefix command. This could lead to consumers being stuck with an out-of-range-offset exception if they began consuming from an offset below the one which was used in the trim-prefix command. by @nvartolomei in #18281
• #18100 Better mapping of REST error codes by @mmaslankaprv in #18102
• #18158 Fix issuing timequeries to cloud storage if remote.read is not enabled. by @WillemKauf in #18159
• #18240 Fixes a crash caused by a race between a client disconnect and a segment reader in tiered storage. by @andrwng in #18241
• #18317 Fixes expiration for transactions that have begun and not produced any data batches. This prevents a stalling LSO. by @bharathv in #18324
• PR #18051 [v24.1.x] Address oversized allocs across kafka API and schema registry by @oleiman
• PR #18125 [v24.1.x] cluster_recovery_backend_test: fix unsafe iteration by @andrwng
• PR #18141 [v24.1.x] Fixes for wait_ms cpu profiler mode by @StephanDollberg
• PR #18216 [v24.1.x] controller_backend: prevent busy-looping when removing partitions by @ztlpn
• PR #18222 [v24.1.x] tx/tm_stm: fix unboundedness of _pid_tx_id by @bharathv
• PR #18328 [v24.1.x] Change information stored in _topic_node_index to avoid oversized alloc by @ballard26
• PR #18406 [v24.1.x] Fix some concurrent memory access problems in partition balancer by @ztlpn

Improvements

• Improve cloud storage cache to prevent readers from being blocked during cache eviction. by @Lazin in #18134
• #18150 rpk container start: You can now select the subnet and gateway to create your 'redpanda' network. by @r-vasquez in #18151
• allow interpreting 'retention_duration' = -1 in a topic_manifest.json file as infinite time retention by @andijcr in #18243
• rpk container now starts the seed broker using the default listener ports. by @r-vasquez in #18148
• PR #18117 [v24.1.x] wasm/parser: better global support by @rockwotj
• PR #18128 [v24.1.x] c/balancer_backend: first initialize planner and then call plan by @mmaslankaprv
• PR #18194 [v24.1.x] configuration to enable delete retention for consumer offsets by @bharathv
• PR #18228 [v24.1.x] CORE-1752: cst: Downgrade error logs to debug by @abhijat
• PR #18269 [v24.1.x] [CORE-2581] cst: move chunk downloads to remote segment bg loop by @abhijat
• PR #18321 [v24.1.x] rpk: stop using args[0] in cloud cluster select by @r-vasquez
• PR #18318 [v24.1.x] offline_log_viewer: fix get_control_record_type by @bharathv

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v24.1.1...v24.1.2

Bug Fixes

• Fix a bug where an invalid buffer passed into the WebAssembly host from the guest could cause Redpanda to abort. by @rockwotj in #18235
• Fixes expiration for transactions that have begun and not produced any data batches. This prevents a stalling LSO. by @bharathv in #18248
• #18237 Fixes a crash caused by a race between a client disconnect and a segment reader in tiered storage. by @andrwng in #18238
• PR #18223 [v23.3.x] tx/tm_stm: fix unboundedness of _pid_tx_id by @bharathv

Improvements

• allow interpreting 'retention_duration' = -1 in a topic_manifest.json file as infinite time retention by @andijcr in #18242

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.14...v23.3.15

Bug Fixes

• fixed a problem leading to UAF error while calculating cloud stage usage by @mmaslankaprv in #17981

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.2.28...v23.2.29

Features

• rpk: rpk cluster partitions list now supports filtering with broker IDs. by @daisukebe in [#18104](https://github.com/redpanda-data/ redpanda/pull/18104)

Bug Fixes

• Fix a bug validating WebAssembly when global constants are specific values that have the encoded byte 0x0B. by @rockwotj in #18109
• #18081 Fixes a crash that could happen when reading from local storage with a large number of segments that all do not contain user data. by @andrwng in #18088 * #18101 Better mapping of REST error codes by @mmaslankaprv in [#18103](https:// github.com/redpanda-data/redpanda/pull/18103)
• #18155 Fix issuing timequeries to cloud storage if remote.read is not enabled. by @WillemKauf in #18156
• PR #18123 [v23.3.x] cluster_recovery_backend_test: fix unsafe iteration by @andrwng

Improvements

• #18133 Improve cloud storage cache to prevent readers from being blocked during cache eviction. by @Lazin in #18138
• PR #18111 [v23.3.x] gh: fix lint-cpp for ubuntu noble by @dotnwat
• PR #18118 [v23.3.x] wasm/parser: better global support by @rockwotj
• PR #18154 [v23.3.x] c/topic_table: do not log duplicated lifecycle marker command by @mmaslankaprv
• PR #18166 [v23.3.x] rpk: bump docker version by @r-vasquez
• PR #18186 [v23.3.x] rpk: bump go deps by @r-vasquez
• PR #18191 [backport][23.3.x] configuration to enable delete retention for consumer offsets #18140 by @bharathv

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.13...v23.3.14

Bug Fixes

• fixed a problem leading to UAF error while calculating cloud stage usage by @mmaslankaprv in [#17980](https://github.com/ redpanda-data/redpanda/pull/17980)
• prevents partial consumer group recovery by @mmaslankaprv in #18016

Improvements

• Changes what the kafka_latency_fetch_latency metric measures to be the time the first fetch_ntps_in_parallel takes. by @ ballard26 in #17977
• largely reduced number of health report copies by @mmaslankaprv in #18017## None No release notes explicitly specified.
• PR #17917 [v23.3.x] archival: Start housekeeping jobs only on a leader by @Lazin
• PR #17927 [v23.3.x] c/hm_backend: cache the collected report by @ mmaslankaprv
• PR #17960 [v23.3.x] rptest: fix test_exceed_broker_limit flake by @ travisdowns
• PR #17979 [v23.3.x] [CORE-2400] kafka/server: Disable quota balancer by @BenPope
• PR #18019 [v23.3.x] CORE-1752: cst: Downgrade error logs to debug by @ abhijat
• PR #18023 [v23.3.x] Backport of #16243 by @mmaslankaprv
• PR #18039 [v23.3.x] c/controller_backend: try to force-abort reconfiguration only on leaders by @ztlpn
• PR #18053 [v23.3.x] Address oversized allocs across kafka API and schema registry by @oleiman

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.12...v23.3.13

Bug Fixes

• Fix a race condition between suffix truncation / delete records and adjacent segment compaction that can lead to crashes and data-loss. by @ nvartolomei in #17254
• Fix initial_leader_epoch/KIP-320 handling in fetch requests. It was ignored until now which prevented consumers to correctly detect suffix truncation. For Redpanda (and Raft), this is a minor problem since suffix truncation is a very improbable event. by @nvartolomei in #17727
• Fixes a bug of config_frontend methods getting called on shards other than the controller shard. by @pgellert in [#17211](https://github.com/ redpanda-data/redpanda/pull/17211)
• Prevent detecting leader epoch advancement when state is not up to date by @mmaslankaprv in [#17882](https://github.com/redpanda-data/ redpanda/pull/17882)
• Reduces maximum log line size from 1MiB to 128KiB to reduce occurrences of memory allocation failures by @michael-redpanda in #17924
• #16612 fixes small inconsistency between Kafka and Redpanda when trying to query end_offset of an empty log by @mmaslankaprv in #17881
• #17238 Fixes a bug in CreateTopicsResponse to now return all the configs of the topic, not just the topic-specific override configs. by @pgellert in #17241
• #17790 Fix a bug that could lead to raft log inconsistencies when 2 out of 3 nodes in a configuration are changed. by @ztlpn in #17797
• prevents partial consumer group recovery by @mmaslankaprv in #17882
• PR #17160 [v23.2.x] compression: Allocate memory for LZ4_compressEnd by @abhijat
• PR #17826 [v23.2.x] CORE-1722: compression: Use preallocated decompression buffers for lz4 by @abhijat
• PR #17881 [v23.2.x] k/replicated_partition: fixed querying end offset of an empty log by @nvartolomei
• PR #17882 Backport of #17673 #17498 #16560 by @mmaslankaprv

Improvements

• Adds a new public metric redpanda_raft_recovery_partition_movement_consumed_bandwidth that tracks how much bandwidth is currently in use for raft recovery. This helps tune raft_learner_recovery_rate. by @bharathv in #17217

• PR #17397 [v23.2.x] k/group: recover leader epoch on leader change by @nvartolomei * PR #17448 [v23.2.x] tx: fix param ordering in log statement by @nvartolomei

• PR #17577 [v23.2.x] c/topics_frontend: break the loop when dispatching to current leader by @mmaslankaprv

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.2.27...v23.2.28

Bug Fixes

• Fix an issue where rpk transform logs waits for records without the --follow flag specified. by @rockwotj in #17837
• Fixes rpk transform init --install-deps so that an explicit true value is not needed. by @rockwotj in [#17867](https:// github.com/redpanda-data/redpanda/pull/17867)
• Fixes a crash when data transforms error and restart by @rockwotj in [#17696](https://github.com/redpanda-data/redpanda/pull/ 17696)
• Reduces maximum log line size from 1MiB to 128KiB to reduce occurrences of memory allocation failures by @michael-redpanda in #17923
• #16612 fixes small inconsistency between Kafka and Redpanda when trying to query end_offset of an empty log by @mmaslankaprv in #17809
• #17718 Fix reported config source for cleanup.policy by reporting DEFAULT_CONFIG instead of DYNAMIC_TOPIC_CONFIG for the default value. by @pgellert in [#17719](https://github.com/redpanda-data/ redpanda/pull/17719)
• #17791 Fix a bug that could lead to raft log inconsistencies when 2 out of 3 nodes in a configuration are changed. by @ztlpn in #17796
• #17817 Fix problem in Tiered-Storage that could potentially cause consumers to get stuck by @Lazin in #17818
• #17891 fix a race between eviction and producer registration that results in an invalid transaction state. by @bharathv in #17900

Improvements

• Handle missing data transform logs topic in rpk transform logs by @rockwotj in [#17835](https://github.com/redpanda-data/ redpanda/pull/17835)
• #17197 more accurate node status reporting by @mmaslankaprv in #17698
• skipping overhead of collecting node health report for each node separately. by @mmaslankaprv in [#17864](https://github.com/ redpanda-data/redpanda/pull/17864)
• PR #17756 [v23.3.x] kafka: chunked_vector for config responses by @ pgellert
• PR #17792 [v23.3.x] CORE-1752: cst: improved logging by @abhijat
• PR #17825 [v23.3.x] CORE-1722: compression: Use preallocated decompression buffers for lz4 by @abhijat
• PR #17888 [v23.3.x] CORE-2365: storage: increase size of offset key map fragment size by @dotnwat

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.11...v23.3.12

Features

• Introduce "trust_file_crc32c" metric to export a checksum for each trust file in the system. by @oleiman in #17587

Bug Fixes

• #16650 Fix oversized allocation in storage. by @Lazin in #17541
• #17459 Fixes a bug with TLS metrics where expiration timestamps would not advance on certificate reload by @oleiman in #17460
• rpk: prevent a segfault when creating a profile from a cloud that is not in ready state. by @r-vasquez in #17585
• PR #17435 [v23.3.x] c/frag_vector: added get_allocator() method to fragmented vector by @mmaslankaprv
• PR #17449 [v23.3.x] tx: fix param ordering in log statement by @ nvartolomei
• PR #17572 [v23.3.x] Fixed node_hash_map caused oversized allocations in cluster module by @mmaslankaprv
• PR #17573 [v23.3.x] use chunked vector as batches cache in raft:: replicate_batcher by @mmaslankaprv
• PR #17576 [v23.3.x] c/topics_frontend: break the loop when dispatching to current leader by @mmaslankaprv
• PR #17578 [v23.3.x] rm_stm: do not hold producer lock for the duration of the barrier by @bharathv
• PR #17584 [v23.3.x] k/group_manager: used chunked_vector when cleaning groups by @mmaslankaprv

Improvements

• #17428 Improves error feedback when Redpanda is given an invalid number of partitions during either topic creation or when the partition count for a topic is increased. by @michael-redpanda in #17431
• PR #17574 [v23.3.x] Improved validation of Fetch requests when reading from follower by @mmaslankaprv

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.10...v23.3.11

Features

• #16075 Data Transform's Rust SDK now supports a Schema Registry Client. by @rockwotj in #17174
• #17235 rpk: support running rpk transform against Redpanda cloud clusters. by @r-vasquez in #17252

Bug Fixes

• Fix a bug where logging in a transform could cause the transform to not make progress. by @rockwotj in #17192
• Fix a race condition between suffix truncation / delete records and adjacent segment compaction that can lead to crashes and data-loss. by @nvartolomei in #17253
• Fixes a bug of config_frontend methods getting called on shards other than the controller shard. by @pgellert in #17184
• Fixes a bug that would prevent the transaction coordinator topic from reclaiming disk space. by @andrwng in #17380
• Fixes a concurrency issue in transform offset commits pertaining to taking/applying snapshots. by @bharathv in #17400
• #17237 Fixes a bug in CreateTopicsResponse to now return all the configs of the topic, not just the topic-specific override configs. by @pgellert in #17239
• #17419 fixed enabling cloud storage in existing clusters by @mmaslankaprv in #17420

Improvements

• Adds a new public metric redpanda_raft_recovery_partition_movement_consumed_bandwidth that tracks how much bandwidth is currently in use for raft recovery. This helps tune raft_learner_recovery_rate. by @bharathv in #17216
• less overhead of health report collection by @mmaslankaprv in #17360

Full Changelog: https://github.com/redpanda-data/redpanda/compare/v23.3.9...v23.3.10