Quickwit Versions Save

• improved merge execution #4741
• automatic pruning of metastore checkpoints #4777
• exposes all native and ES compatible read endpoints (e.g field capabilities) #4805

Added

• Add support for OTLP logs in Kafka source (#4807)

Fixed

• Bug in the chitchat digest message serialization (chitchat#144)
• Fix parsing of date times without year (#4811)

Full Changelog: https://github.com/quickwit-oss/quickwit/compare/v0.8.0...v0.8.1

Added

• Remove some noisy logs (#4447)
• Add /{index}/_stats and /_stats ES API (#4442)
• Use search_after in ES scroll API (#4280)
• Add support for wildcard exclusion in index patterns (#4458)
• Add . support in DSL indentifiers (#3989)
• Add cat indices ES API (#4465)
• Limit concurrent merges (#4473)
• Add Index Template API and auto create index (#4456) (only available with ingest V2)
• Add support for compressed ES _bulk requests (#4506)
• Add support for slash / character in field names (#4510)
• Handle SIGTERM shutdown signal (#4539)
• Add start_timestamp and end_timestamp filter to ES _field_caps API (#4547)
• Limit the number of merge pipelines that can be spawned concurrently (#4574)
• Add support for _source_excludes and _source_includes query parameters in ES API (#4572)
• Add gRPC metrics layer to clients and servers (#4591)
• Add additional cluster metrics (#4597)
• Add index patterns query param on GET /indexes endpoint (#4600)
• Add support for GCS file backed metastore (#4604)
• Add default search fields for OTEL traces index (#4602)
• Add support for delete index in ES API (#4606)
• Add a handler to dynamically change the log level (#4662)
• Add REST endpoint to parse a query into a query AST (#4652)
• Add postgresql index and use IN instead of many OR (#4670)
• Add support for _source_excludes, _source_includes, extra_filters in _msearch ES API (#4696)
• Handle track_total_size on request ES body (#4710)
• Add a metric for the number number of indexes (#4711)
• Add various performance optimizations in Quickwit and Tantivy

More details in tantivy's changelog.

Fixed

• Fix aggregation result on empty index (#4449)
• Fix Gzip file source (#4457)
• Rate limit noisy logs (#4483)
• Prevent the exponential backoff from overflowing after 64 attempts (#4501)
• Remove field presence in ES _field_caps API (#4492)
• Remove source in ES parameter, remove unsupported field fields in response (#4590)
• Fix aggregation split_size parameter, add docs and test (#4627)
• Various fixes in chitchat (gossip): more details in chitchat commit history
• Various fixes in mrecordlog (WAL): more details in mrecordlog commit history

Changed

• (Breaking) Add ZSTD compression to chitchat's Deltas

Removed

Fixed

• bug in gzip file source

Added

• Add es _count API (#4410)
• Add _elastic/_field_caps API (#4350)
• Make gRPC message size configurable (#4388)
• Add API endpoint to get some control-plan internal info (#4339)
• Add Google Cloud Storage Implementation available for storage paths starting with gs:// (#4344)

Changed

• Return 404 on index not found in ES Bulk API (#4425)
• Allow $ and @ characters in field names (#4413)

Fixed

• Assign all sources/shards, even if this requires exceeding the indexer #4363
• Fix traces doc mapping (service name set as fast) and update default otel logs index ID to otel-logs-v0_7 (#4401)
• Fix parsing multi-line queries (#4409)
• Fix range query for optional fast field panics with Index out of bounds (#4362)

Migration from 0.70 to 0.7.1

Quickwit 0.7.1 will create the new index otel-logs-v0_7 which is now used by default when ingesting data with the OTEL gRPC and HTTP API.

In the traces index otel-traces-v0_7, the service_name field is now fast. No migration is done if otel-traces-v0_7 already exists. If you want service_name field to be fast, you have to delete first the existing otel-traces-v0_7 index or you need to create your own index.

Beta release of Quicwkit Lambdas.

Added

• Elasticsearch-compatible API
  • Added scroll and search_after APIs and support for multi-index search queries
  • Added exists, multi-match, match phrase prefix, match bool prefix, bool queries
  • Added _field_caps API
• Added support for OTLP over HTTP API (Protobuf only) (#4335)
• Added Jaeger REST endpoints for Grafana tracing support (#4197)
• Added support for injecting custom HTTP headers and moved REST config parameters into REST config section (#4198)
• Added support for OTLP trace data in arbitrary sources
• Commit Kafka offsets on suggest truncate (#3638)
• Honor auto.offset.reset parameter in Kafka source (#4095)
• Added exact count optimization (#4019)
• Added stream splits gRPC (#4109)
• Adding a split cache in Searchers (#3857)
• Added coerce and output_format options for numeric fields (#3704)
• Added PhraseMatchQuery and MultiMatchQuery (#3727)
• Added Elasticsearch's TermsQuery (#3747)
• Added GCP PubSub source (#3720)
• Parse timestamp strings (#3639)
• Added Digital Ocean storage flavor (#3632)
• Added new tokenizers: source_code_default, source_code, multilang (#3647, #3655, #3608)

Fixed

• Fixed dates in UI (#4277)
• Fixed duplicate splits planned on pipeline crash-respawn (#3854)
• Fixed sorting (#3799)

More details in tantivy's changelog.

Changed

• Improve OTEL traces index config (#4311)

  • OTEL endpoints are now using by default indexes otel-logs-v0_7 and otel-traces-v0_7 instead of otel-logs-v0_6 and otel-traces-v0_6
  • OTEL indexes have more fields stored as "fast" and have Trace and Span ID bytes field in hex format

• Increased the gRPC payload limits from 10MiB to 20MiB (#4227)

• Reject malformed Elasticsearch API requests (#4175)

• Better logging when doc processing fails (#4323)

• Search performance improvements

• Indexing performance improvements

Removed

Migration from 0.6.x to 0.7

The format of the index and internal objects stored in the metastore of 0.7 is backward compatible with 0.6.

If you are using the OTEL indexes and ingesting data into indexes the otel-logs-v0_6 and otel-traces-v0_6, you must stop indexing before upgrading. Indeed, the first time you start Quickwit 0.7, it will update the doc mapping fields of Trace ID and Span ID of those two indexes by changing their input/output formats from base64 to hex. This is automatic: you don't have to perform any manual operation. Quickwit 0.7 will create new indexes otel-logs-v0_7 and otel-traces-v0_7, which are now used by default when ingesting data with the OTEL gRPC and HTTP API. The Jaeger gRPC and HTTP APIs will query both otel-traces-v0_6 and otel-traces-v0_7 by default. It's possible to define the index ID you want to use for OTEL gRPC endpoints and Jaeger gRPC API by setting the request header qw-otel-logs-index or qw-otel-traces-index to the index ID you want to target.

Fixed

• Increase the gRPC payload limits of the metastore gRPC client to 20MiB in commit f76fcee to fix #4256. It only partially solves the problem, as it will appear if the user has 4k or 5k indexes. This will be adequately solved in the 0.7 release.