OpenSearch Versions Save

2024-03-21 Version 2.13.0 Release Notes

[2.13.0]

Added

• [Tiered caching] Introducing cache plugins and exposing Ehcache as one of the pluggable disk cache option (#11874)
• Add support for dependencies in plugin descriptor properties with semver range (#11441)
• Add community_id ingest processor (#12121)
• Introduce query level setting index.query.max_nested_depth limiting nested queries (#3268
• Add toString methods to MultiSearchRequest, MultiGetRequest and CreateIndexRequest (#12163)
• Fix error in RemoteSegmentStoreDirectory when debug logging is enabled (#12328)
• Support for returning scores in matched queries (#11626)
• Add shard id property to SearchLookup for use in field types provided by plugins (#1063)
• [Tiered caching] Make IndicesRequestCache implementation configurable [EXPERIMENTAL] (#12533)
• Force merge API supports performing on primary shards only (#11269)
• Add kuromoji_completion analyzer and filter (#4835)
• [Admission Control] Integrate IO Usage Tracker to the Resource Usage Collector Service and Emit IO Usage Stats (#11880)
• The org.opensearch.bootstrap.Security should support codebase for JAR files with classifiers (#12586)
• Remote reindex: Add support for configurable retry mechanism (#12561)
• Tracing for deep search path (#12103)
• [Metrics Framework] Adds support for asynchronous gauge metric type. (#12642)
• [Tiered caching] Add Stale keys Management and CacheCleaner to IndicesRequestCache (#12625)
• Make search query counters dynamic to support all query types (#12601)
• [Tiered caching] Add policies controlling which values can enter pluggable caches [EXPERIMENTAL] (#12542)
• [Tiered caching] Add serializer integration to allow ehcache disk cache to use non-primitive values (#12709)
• [Admission Control] Integrated IO Based AdmissionController to AdmissionControl Framework (#12583)
• Add Remote Store Migration Experimental flag and allow mixed mode clusters under same (#11986)
• Built-in secure transports support (#12435)
• Lightweight Transport action to verify local term before fetching cluster-state from remote (#12252)
• Integrate with admission controller for cluster-manager Read API. (#12496)
• Introduce a new setting index.check_pending_flush.enabled to expose the ability to disable the check for pending flushes by write threads (#12710)

Dependencies

• Bump com.squareup.okio:okio from 3.7.0 to 3.8.0 (#12290)
• Bump org.bouncycastle:bcprov-jdk15to18 to org.bouncycastle:bcprov-jdk18on version 1.77 (#12326)
• Bump org.bouncycastle:bcmail-jdk15to18 to org.bouncycastle:bcmail-jdk18on version 1.77 (#12326)
• Bump org.bouncycastle:bcpkix-jdk15to18 to org.bouncycastle:bcpkix-jdk18on version 1.77 (#12326)
• Bump gradle/wrapper-validation-action from 1 to 2 (#12367)
• Bump netty from 4.1.106.Final to 4.1.107.Final (#12372)
• Bump opentelemetry from 1.34.1 to 1.36.0 (#12388, #12618)
• Bump Apache Lucene from 9.9.2 to 9.10.0 (#12392)
• Bump org.apache.logging.log4j:log4j-core from 2.22.1 to 2.23.1 (#12464, #12587)
• Bump antlr4 from 4.11.1 to 4.13.1 (#12445)
• Bump com.netflix.nebula.ospackage-base from 11.8.0 to 11.8.1 (#12461)
• Bump peter-evans/create-or-update-comment from 3 to 4 (#12462)
• Bump lycheeverse/lychee-action from 1.9.1 to 1.9.3 (#12521)
• Bump com.azure:azure-core from 1.39.0 to 1.47.0 (#12520)
• Bump ch.qos.logback:logback-core from 1.2.13 to 1.5.3 (#12519)
• Bump codecov/codecov-action from 3 to 4 (#12585)
• Bump org.apache.zookeeper:zookeeper from 3.9.1 to 3.9.2 (#12580)
• Bump org.codehaus.woodstox:stax2-api from 4.2.1 to 4.2.2 (#12579)
• Bump Jackson version from 2.16.1 to 2.17.0 (#12611, #12662)
• Bump reactor-netty from 1.1.15 to 1.1.17 (#12633)
• Bump reactor from 3.5.14 to 3.5.15 (#12633)
• Bump aws-sdk-java from 2.20.55 to 2.20.86 (#12251)

Changed

• Allow composite aggregation to run under a parent filter aggregation (#11499)
• Quickly compute terms aggregations when the top-level query is functionally match-all for a segment (#11643)
• Mark fuzzy filter GA and remove experimental setting (12631)
• Keep the election scheduler open until cluster state has been applied (#11699)

Fixed

• [Revert] [Bug] Check phase name before SearchRequestOperationsListener onPhaseStart (#12035)
• Add support of special WrappingSearchAsyncActionPhase so the onPhaseStart() will always be followed by onPhaseEnd() within AbstractSearchAsyncAction (#12293)
• Add a system property to configure YamlParser codepoint limits (#12298)
• Prevent read beyond slice boundary in ByteArrayIndexInput (#10481)
• Fix the "highlight.max_analyzer_offset" request parameter with "plain" highlighter (#10919)
• Prevent unnecessary fetch sub phase processor initialization during fetch phase execution (#12503)
• Fix terms query on float field when doc_values are turned off by reverting back to FloatPoint from FloatField (#12499)
• Fix get task API does not refresh resource stats (#11531)
• Fix for deserilization bug in weighted round-robin metadata (#11679)
• onShardResult and onShardFailure are executed on one shard causes opensearch jvm crashed (#12158)
• Avoid overflow when sorting missing last on epoch_millis datetime field (#12676)

2024-03-01 Version 1.3.15 Release Notes

Upgrades

• Bump netty from 4.1.100.Final to 4.1.104.Final (#11294, #11775), #12034)
• Bump netty from 4.1.106.Final to 4.1.107.Final (#12372)
• Bump org.apache.kerby:kerb-admin from 1.0.1 to 2.0.3 (#12194)

2024-02-09 Version 2.12.0 Release Notes

[2.12.0]

Added

• [Admission control] Add Resource usage collector service and resource usage tracker (#9890)
• [Admission control] Add enhancements to FS stats to include read/write time, queue size and IO time (#10541)
• [Remote cluster state] Change file names for remote cluster state (#10557)
• [Search Pipelines] Add request-scoped state shared between processors (and three new processors) (#9405)
• Per request phase latency (#10351)
• [Remote Store] Add repository stats for remote store(#10567)
• [Remote cluster state] Upload global metadata in cluster state to remote store(#10404)
• [Remote cluster state] Download functionality of global metadata from remote store (#10535)
• [Remote cluster state] Restore global metadata from remote store when local state is lost after quorum loss (#10404)
• [Remote cluster state] Make index and global metadata upload timeout dynamic cluster settings (#10814)
• Add search query categorizer (#10255)
• Per request phase latency (#10351)
• Add cluster state stats (#10670)
• Remove ingest processor supports excluding fields (#10967, #11983)
• [Tiered caching] Enabling serialization for IndicesRequestCache key object (#10275)
• [Tiered caching] Defining interfaces, listeners and extending IndicesRequestCache with Tiered cache support (#10753)
• [Remote cluster state] Restore cluster state version during remote state auto restore (#10853)
• Update the indexRandom function to create more segments for concurrent search tests (10247)
• Add support for query profiler with concurrent aggregation (#9248)
• Introduce ConcurrentQueryProfiler to profile query using concurrent segment search path and support concurrency during rewrite and create weight (10352)
• Implement on behalf of token passing for extensions (#8679)
• Provide service accounts tokens to extensions (#9618)
• [Streaming Indexing] Introduce new experimental server HTTP transport based on Netty 4 and Project Reactor (Reactor Netty) (#9672)
• Enable must_exist parameter for update aliases API (#11210)
• Add back half_float BKD based sort query optimization (#11024)
• Request level coordinator slow logs (#10650)
• Add template snippets support for field and target_field in KV ingest processor (#10040)
• Allowing pipeline processors to access index mapping info by passing ingest service ref as part of the processor factory parameters (#10307)
• Add experimental SIMD implementation of B-tree to round down dates (#11194)
• Make number of segment metadata files in remote segment store configurable (#11329)
• Allow changing number of replicas of searchable snapshot index (#11317)
• Adding slf4j license header to LoggerMessageFormat.java (#11069)
• [BWC and API enforcement] Introduce checks for enforcing the API restrictions (#11175)
• Maintainer approval check (#11378)
• Create separate transport action for render search template action (#11170)
• Add additional handling in SearchTemplateRequest when simulate is set to true (#11591)
• Introduce cluster level setting cluster.index.restrict.replication.type to prevent replication type setting override during index creations(#11583)
• Add match_only_text field that is optimized for storage by trading off positional queries performance (#6836)
• Add copy ingest processor (#11870)
• Introduce new feature flag "WRITEABLE_REMOTE_INDEX" to gate the writeable remote index functionality (#11717)
• Bump OpenTelemetry from 1.32.0 to 1.34.1 (#11891)
• Add remove_by_pattern ingest processor (#11920)
• Support index level allocation filtering for searchable snapshot index (#11563)
• Add org.opensearch.rest.MethodHandlers and RestController#getAllHandlers (11876)
• New DateTime format for RFC3339 compatible date fields (#11465)
• Add community_id ingest processor (#12121)
• Add support for Google Application Default Credentials in repository-gcs (#8394)
• Remove concurrent segment search feature flag for GA launch (#12074)
• Enable Fuzzy codec for doc id fields using a bloom filter (#11022)
• [Metrics Framework] Adds support for Histogram metric (#12062)
• Support for returning scores in matched queries (#11626)

Dependencies

• Bumps jetty version to 9.4.52.v20230823 to fix GMS-2023-1857 (#9822)
• Bump Lucene from 9.7.0 to 9.8.0 (10276)
• Bump commons-io:commons-io from 2.13.0 to 2.15.1 (#10294, #11001, #11002, #11446, #11554, #11560, #11796)
• Bump com.google.api.grpc:proto-google-common-protos from 2.10.0 to 2.25.1 (#10208, #10298)
• Bump com.netflix.nebula.ospackage-base from 11.4.0 to 11.8.0 (#10295, #11630, #12167)
• Bump org.apache.zookeeper:zookeeper from 3.9.0 to 3.9.1 (#10506)
• Bump de.thetaphi:forbiddenapis from 3.5.1 to 3.6 (#10508)
• Bump org.codehaus.woodstox:stax2-api from 4.2.1 to 4.2.2 (#10639)
• Bump org.bouncycastle:bc-fips from 1.0.2.3 to 1.0.2.4 (#10297)
• Bump com.google.http-client:google-http-client from 1.43.2 to 1.43.3 (#10635)
• Bump com.squareup.okio:okio from 3.5.0 to 3.7.0 (#10637, #11632)
• Bump org.apache.logging.log4j:log4j-core from 2.20.0 to 2.22.1 (#10858, #11000, #11270, #11695)
• Bump aws-actions/configure-aws-credentials from 2 to 4 (#10504)
• Bump stefanzweifel/git-auto-commit-action from 4 to 5 (#11171)
• Bump actions/github-script from 6 to 7.0.1 (#11271, #12166)
• Bump jackson and jackson_databind from 2.15.2 to 2.16.0 (#11273)
• Bump netty from 4.1.100.Final to 4.1.106.Final (#11294, #11775), #12034)
• Bump com.avast.gradle:gradle-docker-compose-plugin from 0.16.12 to 0.17.6 (#10163, #11692)
• Bump com.squareup.okhttp3:okhttp from 4.11.0 to 4.12.0 (#10861)
• Bump org.apache.commons:commons-text from 1.10.0 to 1.11.0 (#11344)
• Bump reactor-netty-core from 1.1.12 to 1.1.15 (#11350), (#12042)
• Bump com.gradle.enterprise from 3.14.1 to 3.16.2 (#11339, #11629, #12056)
• Bump actions/setup-java from 3 to 4 (#11447)
• Bump commons-net:commons-net from 3.9.0 to 3.10.0 (#11450)
• Bump org.apache.maven:maven-model from 3.9.4 to 3.9.6 (#11445)
• Bump org.apache.xmlbeans:xmlbeans from 5.1.1 to 5.2.0 (#11448)
• Bump logback-core and logback-classic to 1.2.13 (#11521)
• Bumps jetty version from 9.4.52.v20230823 to 9.4.53.v20231009 (#11539)
• Bump org.wiremock:wiremock-standalone from 3.1.0 to 3.3.1 (#11555)
• Bump org.apache.commons:commons-compress from 1.24.0 to 1.25.0 (#11556)
• Bump actions/stale from 8 to 9 (#11557)
• Bump com.netflix.nebula:nebula-publishing-plugin from 20.3.0 to 21.0.0 (#11671)
• Bump commons-cli:commons-cli from 1.5.0 to 1.6.0 (#10996)
• Bump com.maxmind.geoip2:geoip2 from 4.1.0 to 4.2.0 (#11559)
• Bump org.apache.commons:commons-lang3 from 3.13.0 to 3.14.0 (#11691)
• Bump com.maxmind.db:maxmind-db from 3.0.0 to 3.1.0 (#11693)
• Bump net.java.dev.jna:jna from 5.13.0 to 5.14.0 (#11798)
• Bump lycheeverse/lychee-action from 1.8.0 to 1.9.1 (#11795, #11887)
• Bump Lucene from 9.8.0 to 9.9.2 (#11421), (#12063)
• Bump com.networknt:json-schema-validator from 1.0.86 to 1.2.0 (#11886, #11963)
• Bump com.google.api:gax-httpjson from 0.103.1 to 2.42.0 (#11794, #12165)
• Bump com.google.oauth-client:google-oauth-client from 1.34.1 to 1.35.0 (#11960)
• Bump com.diffplug.spotless from 6.23.2 to 6.25.0 (#11962, #12055)
• Bump com.google.cloud:google-cloud-core from 2.5.10 to 2.30.0 (#11961)
• Bump reactor-core from 3.5.11 to 3.5.14 (#12042)
• Bump org.apache.shiro:shiro-core from 1.11.0 to 1.13.0 (#12200)
• Bump com.google.http-client:google-http-client-jackson2 from 1.43.3 to 1.44.1 (#12059)
• Bump peter-evans/create-issue-from-file from 4 to 5 (#12057)
• Bump org.gradle.test-retry from 1.5.4 to 1.5.8 (#12168)
• Bump org.apache.kerby:kerb-admin from 1.0.1 to 2.0.3 (#12194)

Changed

• Mute the query profile IT with concurrent execution (#9840)
• Force merge with only_expunge_deletes honors max segment size (#10036)
• Add the means to extract the contextual properties from HttpChannel, TcpCChannel and TrasportChannel without excessive typecasting (#10562), (#11751)
• Introduce new dynamic cluster setting to control slice computation for concurrent segment search (#9107)
• Search pipelines now support asynchronous request and response processors to avoid blocking on a transport thread (#10598)
• [Remote Store] Add Remote Store backpressure rejection stats to _nodes/stats (#10524)
• [BUG] Fix java.lang.SecurityException in repository-gcs plugin (#10642)
• Add telemetry tracer/metric enable flag and integ test. (#10395)
• Performance improvement for Datetime field caching (#4558)
• Add instrumentation for indexing in transport bulk action and transport shard bulk action. (#10273)
• Disallow removing some metadata fields by remove ingest processor (#10895, #11607)
• Performance improvement for MultiTerm Queries on Keyword fields (#7057)
• Refactor common parts from the Rounding class into a separate 'round' package (#11023)
• Performance improvement for date histogram aggregations without sub-aggregations (#11083)
• Apply the fast filter optimization to composite aggregation of date histogram source (#11505)
• Disable concurrent aggs for Diversified Sampler and Sampler aggs (#11087)
• Made leader/follower check timeout setting dynamic (#10528)
• Improved performance of numeric exact-match queries (#11209)
• Change error message when per shard document limit is breached (#11312)
• Improve boolean parsing performance (#11308)
• Interpret byte array as primitive using VarHandles (#11362)
• Automatically add scheme to discovery.ec2.endpoint (#11512)
• Restore support for Java 8 for RestClient (#11562)
• Add deleted doc count in _cat/shards (#11678)
• Capture information for additional query types and aggregation types (#11582)
• Use slice_size == shard_size heuristic in terms aggs for concurrent segment search and properly calculate the doc_count_error (#11732)
• Added Support for dynamically adding SearchRequestOperationsListeners with SearchRequestOperationsCompositeListenerFactory (#11526)
• Ensure Jackson default maximums introduced in 2.16.0 do not conflict with OpenSearch settings (#11890)
• Extract cluster management for integration tests into JUnit test rule out of OpenSearchIntegTestCase (#11877), (#12000)
• Workaround for https://bugs.openjdk.org/browse/JDK-8323659 regression, introduced in JDK-21.0.2 (#11968)
• Updates IpField to be searchable when only doc_values are enabled (#11508)
• [Query Insights] Query Insights Framework which currently supports retrieving the most time-consuming queries within the last configured time window (#11903)
• [Query Insights] Implement Top N Queries feature to collect and gather information about high latency queries in a window (#11904)
• Add override support for sampling based on action (#9621)
• Added custom sampler support based on transport action in request (#9621)
• Disable concurrent search for composite aggregation(#12375)

Removed

• Remove deprecated classes for Rounding (#10956)

Fixed

• Fix failure in dissect ingest processor parsing empty brackets (#9225)
• Fix class_cast_exception when passing int to _version and other metadata fields in ingest simulate API (#10101)
• Fix Segment Replication ShardLockObtainFailedException bug during index corruption (10370)
• Fix some test methods in SimulatePipelineRequestParsingTests never run and fix test failure (#10496)
• Fix passing wrong parameter when calling newConfigurationException() in DotExpanderProcessor (#10737)
• Delegating CachingWeightWrapper#count to internal weight object (#10543)
• Fix per request latency last phase not tracked (#10934)
• Fix SuggestSearch.testSkipDuplicates by forcing refresh when indexing its test documents (#11068)
• [BUG] Fix the thread context that is not properly cleared and messes up the traces (#10873)
• Handle canMatchSearchAfter for frozen context scenario (#11249)
• Fix the issue with DefaultSpanScope restoring wrong span in the TracerContextStorage upon detach (#11316)
• Remove shadowJar from lang-painless module publication (#11369)
• Fix remote shards balancer and remove unused variables (#11167)
• Fix parsing of flat object fields with dots in keys (#11425)
• Fix bug where replication lag grows post primary relocation (#11238)
• Fix noop_update_total metric in indexing stats cannot be updated by bulk API (#11485,#11917)
• Fix for stuck update action in a bulk with retry_on_conflict property (#11152)
• Fix template setting override for replication type (#11417)
• Fix Automatic addition of protocol broken in #11512 (#11609)
• Fix issue when calling Delete PIT endpoint and no PITs exist (#11711)
• Fix tracing context propagation for local transport instrumentation (#11490)
• Fix parsing of single line comments in lang-painless (#11815)
• Fix memory leak issue in ReorganizingLongHash (#11953)
• Prevent setting remote_snapshot store type on index creation (#11867)
• [BUG] Fix remote shards balancer when filtering throttled nodes (#11724)
• Add advance(int) for numeric values in order to allow point based optimization to kick in (#12089)

2023-11-20 Version 2.11.1 Release Notes

[2.11.1]

Changed

• Use iterative approach to evaluate Regex.simpleMatch (#11060)

Fixed

• [BUG] Disable sort optimization for HALF_FLOAT (#10999)
• Adding version condition while adding geoshape doc values to the index, to ensure backward compatibility.(#11095)
• Remove shadowJar from lang-painless module publication (#11369)

2023-10-12 Version 2.11.0 Release Notes

[2.11]

Added

• Add coordinator level stats for search latency (#8386)
• Add metrics for thread_pool task wait time (#9681)
• Async blob read support for S3 plugin (#9694)
• [Telemetry-Otel] Added support for OtlpGrpcSpanExporter exporter (#9666)
• Async blob read support for encrypted containers (#10131)
• Implement Visitor Design pattern in QueryBuilder to enable the capability to traverse through the complex QueryBuilder tree. (#10110)
• Add capability to restrict async durability mode for remote indexes (#10189)
• Add Doc Status Counter for Indexing Engine (#4562)
• Add unreferenced file cleanup count to merge stats (#10204)
• Configurable merge policy for index with an option to choose from LogByteSize and Tiered merge policy (#9992)
• [Remote Store] Add support to restrict creation & deletion if system repository and mutation of immutable settings of system repository (#9839)
• Improve compressed request handling (#10261)

Dependencies

• Bump JNA version from 5.5 to 5.13 (#9963)
• Bump peter-evans/create-or-update-comment from 2 to 3 (#9575)
• Bump actions/checkout from 2 to 4 (#9968)
• Bump OpenTelemetry from 1.26.0 to 1.30.1 (#9950)
• Bump org.apache.commons:commons-compress from 1.23.0 to 1.24.0 ([#9973, #9972](https://github.com/opensearch-project/OpenSearch/pull/9973, https://github.com/opensearch-project/OpenSearch/pull/9972))
• Bump com.google.cloud:google-cloud-core-http from 2.21.1 to 2.23.0 (#9971)
• Bump mockito from 5.4.0 to 5.5.0 (#10022)
• Bump bytebuddy from 1.14.3 to 1.14.7 (#10022)
• Bump com.zaxxer:SparseBitSet from 1.2 to 1.3 (#10098)
• Bump tibdex/github-app-token from 1.5.0 to 2.1.0 (#10125)
• Bump org.wiremock:wiremock-standalone from 2.35.0 to 3.1.0 (#9752)
• Bump org.eclipse.jgit from 6.5.0 to 6.7.0 (#10147)
• Bump codecov/codecov-action from 2 to 3 (#10209)
• Bump com.google.http-client:google-http-client-jackson2 from 1.43.2 to 1.43.3 (#10126)
• Bump org.xerial.snappy:snappy-java from 1.1.10.3 to 1.1.10.5 (#10206, #10299)
• Bump org.bouncycastle:bcpkix-jdk15to18 from 1.75 to 1.76 (10219)`
• Bump org.bouncycastle:bcprov-jdk15to18 from 1.75 to 1.76 (10219)`
• Bump org.bouncycastle:bcmail-jdk15to18 from 1.75 to 1.76 (10219)`
• Bump asm from 9.5 to 9.6 (#10302)
• Bump netty from 4.1.97.Final to 4.1.99.Final (#10303)
• Bump peter-evans/create-pull-request from 3 to 5 (#10301)
• Bump org.apache.avro:avro from 1.11.2 to 1.11.3 (#10210)
• Bump netty from 4.1.99.Final to 4.1.100.Final (#10564)

Changed

• Add instrumentation in rest and network layer. (#9415)
• Allow parameterization of tests with OpenSearchIntegTestCase.SuiteScopeTestCase annotation (#9916)
• Add instrumentation in transport service. (#10042)
• [Tracing Framework] Add support for SpanKind. (#10122)
• Pass parent filter to inner query in nested query (#10246)
• Disable concurrent segment search when terminate_after is used (#10200)
• Add instrumentation in Inbound Handler. (#100143)
• Enable remote segment upload backpressure by default (#10356)
• [Remote Store] Add support to reload repository metadata inplace (#9569)
• [Metrics Framework] Add Metrics framework. (#10241)
• Updating the separator for RemoteStoreLockManager since underscore is allowed in base64UUID url charset (#10379)

Removed

• Remove spurious SGID bit on directories (#9447)

Fixed

• Fix ignore_missing parameter has no effect when using template snippet in rename ingest processor (#9725)
• Fix broken backward compatibility from 2.7 for IndexSorted field indices (#10045)
• Fix concurrent search NPE when track_total_hits, terminate_after and size=0 are used (#10082)
• Fix remove ingest processor handing ignore_missing parameter not correctly (10089)
• Fix registration and initialization of multiple extensions (10256)
• Fix circular dependency in Settings initialization (10194)
• Fix Segment Replication ShardLockObtainFailedException bug during index corruption (10370)

2023-09-08 Version 2.10.0 Release Notes

[2.10]

Added

• Add server version as REST response header #6583
• Start replication checkpointTimers on primary before segments upload to remote store. (#8221https://github.com/opensearch-project/OpenSearch/pull/8221)
• Introduce new static cluster setting to control slice computation for concurrent segment search. (#8847)
• Add configuration for file cache size to max remote data ratio to prevent oversubscription of file cache (#8606)
• Disallow compression level to be set for default and best_compression index codecs (#8737https://github.com/opensearch-project/OpenSearch/pull/8737)
• [distribution/archives] [Linux] [x64] Provide the variant of the distributions bundled with JRE (#8195https://github.com/opensearch-project/OpenSearch/pull/8195)
• Prioritize replica shard movement during shard relocation (#8875)
• Introducing Default and Best Compression codecs as their algorithm name (#9123)
• Make SearchTemplateRequest implement IndicesRequest.Replaceable (#9122)
• [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them (#9223)
• [Remote Store] Add Segment download stats to remotestore stats API (#8718)
• [Remote Store] Add remote segment transfer stats on NodesStats API (#9168 #9393 #9454)
• [Segment Replication] Support realtime reads for GET requests (#9212)
• Allow test clusters to run with TLS (#8900)
• Add jdk.incubator.vector module support for JDK 20+ (#8601)
• [Feature] Expose term frequency in Painless script score context (#9081)
• Add support for reading partial files to HDFS repository (#9513)
• [Remote Store] Rate limiter integration for remote store uploads and downloads(#9448)
• [BWC and API enforcement] Decorate the existing APIs with proper annotations (part 1) (#9520)
• Add support for extensions to search responses using SearchExtBuilder (#9379)
• [Remote State] Create service to publish cluster state to remote store (#9160)
• Core crypto library to perform encryption and decryption of source content (#8466)
• Expose DelimitedTermFrequencyTokenFilter to allow providing term frequencies along with terms (#9479)
• APIs for performing async blob reads and async downloads from the repository using multiple streams (#9592)
• Add concurrent segment search related metrics to node and index stats (#9622)
• Add average concurrency metric for concurrent segment search (#9670)
• Introduce cluster default remote translog buffer interval setting (#9584)
• Added encryption-sdk lib to provide encryption and decryption capabilities (#8466 #9289)
• [Segment Replication] Adding segment replication statistics rolled up at index, node and cluster level (#9709)
• Added crypto-kms plugin to provide AWS KMS based key providers for encryption/decryption. (#8465)
• [Remote state] Integrate remote cluster state in publish/commit flow (#9665)
• [Remote Store] Changes to introduce repository registration during bootstrap via node attributes. (#9105)
• [Remote state] Auto restore index metadata from last known cluster state (#9831)

Dependencies

• Bump org.apache.logging.log4j:log4j-core from 2.17.1 to 2.20.0 (#8307)
• Bump io.grpc:grpc-context from 1.46.0 to 1.57.1 (#8726, #9145)
• Bump com.netflix.nebula:gradle-info-plugin from 12.1.5 to 12.1.6 (#8724)
• Bump commons-codec:commons-codec from 1.15 to 1.16.0 (#8725)
• Bump org.apache.zookeeper:zookeeper from 3.8.1 to 3.9.0 (#8844, #9146)
• Bump org.gradle.test-retry from 1.5.3 to 1.5.4 (#8842)
• Bump com.netflix.nebula.ospackage-base from 11.3.0 to 11.4.0 (#8838)
• Bump com.google.http-client:google-http-client-gson from 1.43.2 to 1.43.3 (#8840)
• OpenJDK Update (July 2023 Patch releases) (#8869)
• Bump hadoop libraries from 3.3.4 to 3.3.6 (#6995)
• Bump com.gradle.enterprise from 3.13.3 to 3.14.1 (#8996)
• Bump org.apache.commons:commons-lang3 from 3.12.0 to 3.13.0 (#8995)
• Bump com.google.cloud:google-cloud-core-http from 2.21.0 to 2.21.1 (#8999)
• Bump com.maxmind.geoip2:geoip2 from 4.0.1 to 4.1.0 (#8998)
• Bump org.apache.commons:commons-lang3 from 3.12.0 to 3.13.0 in /plugins/repository-hdfs (#8997)
• Bump netty from 4.1.94.Final to 4.1.96.Final (#9030)
• Bump com.google.jimfs:jimfs from 1.2 to 1.3.0 (#9080)
• Bump io.projectreactor.netty:reactor-netty-http from 1.1.8 to 1.1.9 (#9147)
• Bump org.apache.maven:maven-model from 3.9.3 to 3.9.4 (#9148)
• Bump com.azure:azure-storage-blob from 12.22.3 to 12.23.0 (#9231)
• Bump com.diffplug.spotless from 6.19.0 to 6.20.0 (#9227)
• Bump org.xerial.snappy:snappy-java from 1.1.8.2 to 1.1.10.3 (#9252)
• Bump com.squareup.okhttp3:okhttp from 4.9.3 to 4.11.0 (#9252)
• Bump com.squareup.okio:okio from 2.8.0 to 3.5.0 (#9252)
• Bump com.google.code.gson:gson from 2.9.0 to 2.10.1 (#9230)
• Bump lycheeverse/lychee-action from 1.2.0 to 1.8.0 (#9228)
• Bump snakeyaml from 2.0 to 2.1 (#9269)
• Bump aws-actions/configure-aws-credentials from 1 to 2 (#9302)
• Bump com.github.luben:zstd-jni from 1.5.5-3 to 1.5.5-5 (#9431
• Bump netty from 4.1.96.Final to 4.1.97.Final (#9553)
• Bump io.grpc:grpc-api from 1.57.1 to 1.57.2 (#9578)
• Add Encryption SDK dependencies (#8466)

Changed

• Default to mmapfs within hybridfs (#8508)
• Perform aggregation postCollection in ContextIndexSearcher after searching leaves (#8303)
• Make Span exporter configurable (#8620)
• Perform aggregation postCollection in ContextIndexSearcher after searching leaves (#8303)
• [Refactor] StreamIO from common to core.common namespace in core lib (#8157)
• [Refactor] Remaining HPPC to java.util collections (#8730)
• Remote Segment Store Repository setting moved from index.remote_store.repository to index.remote_store.segment.repository and cluster.remote_store.repository to cluster.remote_store.segment.repository respectively for Index and Cluster level settings (#8719)
• Change InternalSignificantTerms to sum shard-level superset counts only in final reduce (#8735)
• Exclude 'benchmarks' from codecov report (#8805)
• Create separate SourceLookup instance per segment slice in SignificantTextAggregatorFactory (#8807)
• Replace the deprecated IndexReader APIs with new storedFields() & termVectors() (#7792)
• [Remote Store] Add support to restore only unassigned shards of an index (#8792)
• Add safeguard limits for file cache during node level allocation (#8208)
• Performance improvements for BytesRefHash (#8788)
• Add support for aggregation profiler with concurrent aggregation (#8801)
• [Remove] Deprecated Fractional ByteSizeValue support #9005 (#9005)
• Add support for aggregation profiler with concurrent aggregation (#8801)
• [Remote Store] Restrict user override for remote store index level settings (#8812)
• [Refactor] MediaTypeParser to MediaTypeParserRegistry (#8636)
• Make MultiBucketConsumerService thread safe to use across slices during search (#9047)
• Removed blocking wait in TransportGetSnapshotsAction which was exhausting generic threadpool (#8377)
• Adds support for tracing runnable scenarios (#8831)
• Change shard_size and shard_min_doc_count evaluation to happen in shard level reduce phase (#9085)
• Add attributes to startSpan methods (#9199)
• [Refactor] Task foundation classes to core library - pt 1 (#9082)
• Add support for wrapping CollectorManager with profiling during concurrent execution (#9129)
• Add base class for parameterizing the search based tests #9083 (#9083)
• Add support for wrapping CollectorManager with profiling during concurrent execution (#9129)
• Rethrow OpenSearch exception for non-concurrent path while using concurrent search (#9177)
• Improve performance of encoding composite keys in multi-term aggregations (#9412)
• Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility (#9262)
• Fix sort related ITs for concurrent search (#9177
• [Remote Store] Implicitly use replication type SEGMENT for remote store clusters (#9264)
• Add support to use trace propagated from client (#9506)
• Separate request-based and settings-based concurrent segment search controls and introduce AggregatorFactory method to determine concurrent search support (#9469)
• [Remote Store] Rate limiter integration for remote store uploads and downloads(#9448)
• [Remote Store] Implicitly use replication type SEGMENT for remote store clusters (#9264)
• Redefine telemetry context restoration and propagation (#9617)
• Use non-concurrent path for sort request on timeseries index and field(#9562)
• Added sampler based on Blanket Probabilistic Sampling rate and Override for on demand (#9621)
• Decouple replication lag from logic to fail stale replicas (#9507)
• Improve performance of rounding dates in date_histogram aggregation (#9727)
• [Remote Store] Add support for Remote Translog Store stats in _remotestore/stats/ API (#9263)
• Removing the vec file extension from INDEX_STORE_HYBRID_NIO_EXTENSIONS, to ensure the no performance degradation for vector search via Lucene Engine.(#9528))
• Cleanup Unreferenced file on segment merge failure (#9503)
• Move zstd compression codec to external custom-codecs repository (#9422
• [Remote Store] Add support for Remote Translog Store upload stats in _nodes/stats/ API (#8908)
• [Remote Store] Removing feature flag to mark feature GA (#9761)

Removed

• Remove provision to create Remote Indices without Remote Translog Store (#8719)

Fixed

• Fix flaky ResourceAwareTasksTests.testBasicTaskResourceTracking test (#8993)
• Fix null_pointer_exception when creating or updating ingest pipeline (#9259)
• Fix memory leak when using Zstd Dictionary (#9403)
• Fix condition to remove index create block (#9437)
• Add support to clear archived index setting (#9019)
• Fix range reads in respository-s3 (9512)
• [Segment Replication] Fixed bug where replica shard temporarily serves stale data during an engine reset (#9495)
• Disable shard/segment level search_after short cutting if track_total_hits != false (#9683)
• [Segment Replication] Fixed bug where bytes behind metric is not accurate (#9686)
• Fix broken backward compatibility from 2.7 for IndexSorted field indices (#10045)

2023-09-14 Version 1.3.13 Release Notes

Upgrades

• Bump netty from 4.1.96.Final to 4.1.97.Final (#9553)
• Bump org.xerial.snappy:snappy-java from 1.1.8.2 to 1.1.10.3 (#9252)
• Bump com.squareup.okhttp3:okhttp from 4.9.3 to 4.11.0 (#9252)
• Bump com.squareup.okio:okio from 2.8.0 to 3.5.0 (#9252)

2023-07-13 Version 2.9.0 Release Notes

[2.9]

Added

• [SearchPipeline] Add new search pipeline processor type, SearchPhaseResultsProcessor, that can modify the result of one search phase before starting the next phase.(#7283)
• Add task cancellation monitoring service (#7642)
• Add TokenManager Interface (#7452)
• Add Remote store as a segment replication source (#7653)
• Implement concurrent aggregations support without profile option (#7514)
• Add dynamic index and cluster setting for concurrent segment search (#7956)
• Add descending order search optimization through reverse segment read. (#7967)
• [Search pipelines] Added search pipelines output to node stats (#8053)
• Update components of segrep backpressure to support remote store. (#8020)
• Make remote cluster connection setup in async (#8038)
• Add API to initialize extensions (#8029https://github.com/opensearch-project/OpenSearch/pull/8029)
• Add distributed tracing framework (#7543)
• Enable Point based optimization for custom comparators (#8168)
• [Extensions] Support extension additional settings with extension REST initialization (#8414)
• Adds mock implementation for TelemetryPlugin (#7545)
• Support transport action names when registering NamedRoutes (#7957)
• Create concept of persistent ThreadContext headers that are unstashable (#8291https://github.com/opensearch-project/OpenSearch/pull/8291)
• [Search pipelines] Add Global Ignore_failure options for Processors (#8373)
• Enable Partial Flat Object (#7997)
• Add jdk.incubator.vector module support for JDK 20+ (#8601)
• Introduce full support for Search Pipeline (#8613)
• Add partial results support for concurrent segment search (#8306)

Dependencies

• Bump com.azure:azure-storage-common from 12.21.0 to 12.21.1 (#7566, #7814)
• Bump com.google.guava:guava from 30.1.1-jre to 32.1.1-jre (#7565, #7811, #7807, #7808, #8402, #8400, #8401, #8581)
• Bump net.minidev:json-smart from 2.4.10 to 2.4.11 (#7660, #7812)
• Bump org.gradle.test-retry from 1.5.2 to 1.5.3 (#7810)
• Bump com.diffplug.spotless from 6.17.0 to 6.18.0 (#7896)
• Bump jackson from 2.15.1 to 2.15.2 (#7897)
• Add com.github.luben:zstd-jni version 1.5.5-3 (#2996)
• Bump netty from 4.1.91.Final to 4.1.93.Final (#7901)
• Bump com.amazonaws 1.12.270 to software.amazon.awssdk 2.20.55 (7372)
• Add org.reactivestreams 1.0.4 (7372)
• Bump com.networknt:json-schema-validator from 1.0.81 to 1.0.85 ([7968], #8255)
• Bump com.netflix.nebula:gradle-extra-configurations-plugin from 9.0.0 to 10.0.0 in /buildSrc (#7068)
• Bump com.google.guava:guava from 32.0.0-jre to 32.0.1-jre (#8009)
• Bump commons-io:commons-io from 2.12.0 to 2.13.0 (#8014, #8013, #8010)
• Bump com.diffplug.spotless from 6.18.0 to 6.19.0 (#8007)
• Bump 'com.azure:azure-storage-blob to 12.22.2 from 12.21.1 (#8043)
• Bump org.jruby.joni:joni from 2.1.48 to 2.2.1 (#8015, #8254)
• Bump com.google.guava:guava from 32.0.0-jre to 32.0.1-jre (#8011, #8012, #8107)
• Bump io.projectreactor:reactor-core from 3.4.18 to 3.5.6 in /plugins/repository-azure (#8016)
• Bump spock-core from 2.1-groovy-3.0 to 2.3-groovy-3.0 (#8122)
• Bump com.networknt:json-schema-validator from 1.0.83 to 1.0.84 (#8141)
• Bump com.netflix.nebula:gradle-info-plugin from 12.1.3 to 12.1.5 (#8139, #8568)
• Bump commons-io:commons-io from 2.12.0 to 2.13.0 in /plugins/discovery-azure-classic (#8140)
• Bump mockito from 5.2.0 to 5.4.0 (#8181)
• Bump netty from 4.1.93.Final to 4.1.94.Final (#8191)
• Bump org.apache.hadoop:hadoop-minicluster from 3.3.5 to 3.3.6 (#8257)
• Bump io.projectreactor.netty:reactor-netty-http from 1.1.7 to 1.1.8 (#8256)
• [Upgrade] Lucene 9.7.0 release (#8272)
• Bump org.jboss.resteasy:resteasy-jackson2-provider from 3.0.26.Final to 6.2.4.Final in /qa/wildfly (#8209)
• Bump com.google.api-client:google-api-client from 1.34.0 to 2.2.0 (#8276)
• Update Apache HttpCore/ HttpClient and Apache HttpCore5 / HttpClient5 dependencies (#8434)
• Bump org.apache.maven:maven-model from 3.9.2 to 3.9.3 (#8403)
• Bump io.projectreactor.netty:reactor-netty and io.projectreactor.netty:reactor-netty-core from 1.1.7 to 1.1.8 (#8405)
• Bump com.azure:azure-storage-blob from 12.22.2 to 12.22.3 (#8572)
• Bump net.minidev:json-smart from 2.4.11 to 2.5.0 (#8575, #8576)
• Bump com.google.jimfs:jimfs from 1.2 to 1.3.0 (#8577, #8571)
• Bump com.networknt:json-schema-validator from 1.0.85 to 1.0.86 (#8573)
• Bump com.google.cloud:google-cloud-core-http from 2.17.0 to 2.21.0 (#8586)
• Bump com.google.jimfs:jimfs from 1.2 to 1.3.0 (#8585)

Changed

• Replace jboss-annotations-api_1.2_spec with jakarta.annotation-api (#7836)
• Reduce memory copy in zstd compression (#7681)
• Add min, max, average and thread info to resource stats in tasks API (#7673)
• Add ZSTD compression for snapshotting (#2996)
• Change com.amazonaws.sdk.ec2MetadataServiceEndpointOverride to aws.ec2MetadataServiceEndpoint (7372)
• Change com.amazonaws.sdk.stsEndpointOverride to aws.stsEndpointOverride (7372)
• Add new query profile collector fields with concurrent search execution (#7898)
• Align range and default value for deletes_pct_allowed in merge policy (#7730)
• Rename QueryPhase actors like Suggest, Rescore to be processors rather than phase (#8025)
• Compress and cache cluster state during validate join request (#7321)
• [Snapshot Interop] Add Changes in Create Snapshot Flow for remote store interoperability. (#7118)
• Allow insecure string settings to warn-log usage and advise to migration of a newer secure variant (#5496)
• Add self-organizing hash table to improve the performance of bucket aggregations (#7652)
• Check UTF16 string size before converting to String to avoid OOME (#7963)
• Move ZSTD compression codecs out of the sandbox (#7908)
• Update ZSTD default compression level (#8471)
• [Search Pipelines] Pass pipeline creation context to processor factories (#8164)
• Enabling compression levels for zstd and zstd_no_dict (#8312)
• Optimize Metadata build() to skip redundant computations as part of ClusterState build (#7853)
• Add safeguard limits for file cache during node level allocation (#8208)
• Move span actions to Scope (#8411)
• Add wrapper tracer implementation (#8565)
• Improved performance of parsing floating point numbers (#7909)

Removed

• Remove COMPRESSOR variable from CompressorFactory and use DEFLATE_COMPRESSOR instead (7907)
• Remove concurrency based minimum file cache size restriction (#8294)

Fixed

• Fixing error: adding a new/forgotten parameter to the configuration for checking the config on startup in plugins/repository-s3 #7924
• Enforce 512 byte document ID limit in bulk updates (#8039)
• With only GlobalAggregation in request causes unnecessary wrapping with MultiCollector (#8125)
• Fix mapping char_filter when mapping a hashtag (#7591)
• Fix NPE in multiterms aggregations involving empty buckets (#7318)
• Precise system clock time in MasterService debug logs (#7902)
• Improve indexing performance for flat_object type (#7855)
• Adds log4j configuration for telemetry LogSpanExporter (#8393)
• Fix painless casting bug, which crashes the OpenSearch process (#8315)
• Add missing validation/parsing of SearchBackpressureMode of SearchBackpressureSettings (#7541)

2023-06-06 Version 2.8.0 Release Notes

[2.8]

Added

• [Extensions] Moving Extensions APIs to support cross versions via protobuf. (#7402)
• [Extensions] Add IdentityPlugin into core to support Extension identities (#7246)
• Add connectToNodeAsExtension in TransportService (#6866)
• [Search Pipelines] Accept pipelines defined in search source (#7253)
• [Search Pipelines] Add default_search_pipeline index setting (#7470)
• [Search Pipelines] Add RenameFieldResponseProcessor for Search Pipelines (#7377)
• [Search Pipelines] Split search pipeline processor factories by type (#7597)
• [Search Pipelines] Add script processor (#7607)
• Add 'unsigned_long' numeric field type (#6237)
• Add back primary shard preference for queries (#7375)
• Add task cancellation timestamp in task API (#7455)
• Adds ExtensionsManager.lookupExtensionSettingsById (#7466)
• SegRep with Remote: Add hook for publishing checkpoint notifications after segment upload to remote store (#7394)
• Add search_after query optimizations with shard/segment short cutting (#7453)
• Provide mechanism to configure XContent parsing constraints (after update to Jackson 2.15.0 and above) (#7550)
• Support to clear filecache using clear indices cache API (#7498)
• Create NamedRoute to map extension routes to a shortened name (#6870)
• Added @dbwiddis as on OpenSearch maintainer (#7665)
• [Extensions] Add ExtensionAwarePlugin extension point to add custom settings for extensions (#7526)
• Add new cluster setting to set default index replication type (#7420)

Dependencies

• Bump com.netflix.nebula:gradle-info-plugin from 12.0.0 to 12.1.3 (#7564)
• Bump com.netflix.nebula:nebula-publishing-plugin from 19.2.0 to 20.2.0
• Bump com.google.protobuf:protobuf-java from 3.22.2 to 3.22.3
• Bump jackson from 2.14.2 to 2.15.0 (#7286)
• Bump com.netflix.nebula:nebula-publishing-plugin from 20.2.0 to 20.3.0
• Bump com.netflix.nebula.ospackage-base from 11.0.0 to 11.3.0
• Bump gradle.plugin.com.github.johnrengelman:shadow from 7.1.2 to 8.0.0
• Bump jna from 5.11.0 to 5.13.0
• Bump commons-io:commons-io from 2.7 to 2.12.0 (#7661, #7658, #7656)
• Bump org.apache.shiro:shiro-core from 1.9.1 to 1.11.0 (#7397)
• Bump jetty-server in hdfs-fixture from 9.4.49.v20220914 to 9.4.51.v20230217 (#7405)
• OpenJDK Update (April 2023 Patch releases) (#7448
• Bump org.apache.commons:commons-compress from 1.22 to 1.23.0 (#7462)
• Bump com.azure:azure-core from 1.34.0 to 1.39.0
• Bump com.networknt:json-schema-validator from 1.0.78 to 1.0.81 (#7460)
• Bump Apache Lucene to 9.6.0 (#7505)
• Bump com.google.cloud:google-cloud-core-http from 1.93.3 to 2.17.0 (#7488)
• Bump com.google.guava:guava from 30.1.1-jre to 32.0.0-jre (#7565, #7811, #7808, #7807)
• Bump com.azure:azure-storage-common from 12.20.0 to 12.21.1 (#7566, #7814)
• Bump org.apache.commons:commons-compress from 1.22 to 1.23.0 (#7563)
• Bump jackson from 2.15.0 to 2.15.1 (#7603)
• Bump net.minidev:json-smart from 2.4.10 to 2.4.11 (#7660, #7812)
• Bump io.projectreactor.netty:reactor-netty-core from 1.1.5 to 1.1.7 (#7657)
• Bump org.apache.maven:maven-model from 3.9.1 to 3.9.2 (#7655)
• Bump com.google.api:gax from 2.17.0 to 2.27.0 (#7697)
• Bump io.projectreactor.netty:reactor-netty from 1.1.4 to 1.1.7 (#7725)
• Bump io.projectreactor.netty:reactor-netty-http from 1.1.4 to 1.1.7 (#7725)
• Bump com.google.http-client:google-http-client-appengine from 1.41.8 to 1.43.2 (#7813)
• Bump org.gradle.test-retry from 1.5.2 to 1.5.3 (#7810)

Changed

• Enable ./gradlew build on MacOS by disabling bcw tests (#7303)
• Moved concurrent-search from sandbox plugin to server module behind feature flag (#7203)
• Allow access to indices cache clear APIs for read only indexes (#7303)
• Changed concurrent-search threadpool type to be resizable and support task resource tracking (#7502)
• Default search preference to _primary for searchable snapshot indices (#7628)
• [Segment Replication] Remove codec name string match check for checkpoints (#7741)

Fixed

• Add more index blocks check for resize APIs (#6774)
• Replaces ZipInputStream with ZipFile to fix Zip Slip vulnerability (#7230)
• Add missing validation/parsing of SearchBackpressureMode of SearchBackpressureSettings (#7541)
• [Search Pipelines] Better exception handling in search pipelines (#7735)
• Fix input validation in segments and delete pit request (#6645)

Release Notes: https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-1.3.10.md