VictoriaMetrics: fast, cost-effective monitoring solution and time series database
Released at 2024-04-11
Update note 1: When upgrading to this release from v1.99.0 or v1.100.0 it is recommended to reset caches stored on disk according to these docs.
FEATURE: vmbackupmanager: allow specifying custom backup interval via -backupInterval command-line flag. See this feature request.
BUGFIX: properly register new entries in IndexDB when many new time series are ingested into VictoriaMetrics in a short period of time. See this and this issues. The bug has been introduced in v1.99.0.
BUGFIX: vmbackupmanager: fix panic when performing a restore from GCS. Previously, filters for GCS objects were not properly set which caused objects to be returned instead of prefixes.
Released at 2024-04-11
Released at 2024-04-04
OR operators. See this issue.DEBUG: start trimLines and DEBUG: end trimLines. This bug has been introduced in v0.5.0 in this commit.Released at 2024-04-04
Update note 1: the -datasource.lookback command-line flag at vmalert is no-op starting from this release. This flag will be removed in the future, so please switch to eval_delay option. See this issue for more details.
SECURITY: upgrade Go builder from Go1.21.7 to Go1.22.2. See the list of issues addressed in Go1.22.1 and the list of issues addressed in Go1.22.2.
FEATURE: downsampling: add ability to configure distinct downsampling per distinct sets of time series and/or tenants. See this feature request and these docs.
FEATURE: vmauth: allow discovering ip addresses for backend instances hidden behind a shared hostname, via discover_backend_ips: true option. This allows evenly spreading load among backend instances. See these docs and this feature request.
FEATURE: vmauth: allow routing incoming requests based on HTTP query args via src_query_args option at url_map. See these docs and this feature request.
FEATURE: vmauth: allow routing incoming requests based on HTTP request headers via src_headers option at url_map. See these docs.
FEATURE: vmauth: add ability to read auth tokens from arbitrary HTTP request headers via -httpAuthHeader command-line flag. Previously auth tokens were read only from Authorization HTTP request header. See these docs for details.
FEATURE: vmauth: add ability to authorize by opaque HTTP request header value via auth_token option in -auth.config.
FEATURE: stream aggregation: reduce memory usage by up to 5x when aggregating over big number of unique time series. The memory usage reduction is most visible when stream deduplication is enabled.
FEATURE: stream aggregation: allow using -streamAggr.dedupInterval and -remoteWrite.streamAggr.dedupInterval command-line flags without the need to specify -streamAggr.config and -remoteWrite.streamAggr.config. See these docs.
FEATURE: stream aggregation: add -streamAggr.dropInputLabels command-line flag, which can be used for dropping the listed labels from input samples before applying stream de-duplication and aggregation. This is faster and easier to use alternative to input_relabel_configs. See these docs.
FEATURE: stream aggregation: add dedup_interval option, which allows configuring individual deduplication intervals per each stream aggregation config.
FEATURE: stream aggregation: use the same logic in stream deduplication as in the deduplication at VictoriaMetrics. See this issue.
FEATURE: stream aggregation: ignore out of order samples when calculating increase, increase_prometheus, total and total_prometheus outputs. Thanks to @edma2 for the pull request.
FEATURE: stream aggregation: add an ability to ignore input samples with old timestamps outside the current aggregation interval. See these docs for details.
FEATURE: stream aggregation: add keep_metric_names option, which can be set at stream aggregation config in order to keep the original metric names in the output aggregated samples instead of using the default output metric naming scheme.
FEATURE: stream aggregation: align the time of aggregated data flush to the specified aggregation interval. For example, if interval is set to 1m, then the aggregated data will be flushed at the end of every minute. The alginment can be disabled by setting no_align_flush_to_interval: true option at stream aggregation config. See these docs for details.
FEATURE: stream aggregation: add unique_samples output, which can be used for calculating the number of unique sample values over the given interval.
FEATURE: stream aggregation: add increase_prometheus and total_prometheus outputs, which can be used for increase and total aggregations when the first sample of every new time series must be ignored.
FEATURE: stream aggregation: expose vm_streamaggr_flush_timeouts_total and vm_streamaggr_dedup_flush_timeouts_total counters at /metrics page, which can be used for detecting flush timeouts for stream aggregation states. Expose also vm_streamaggr_flush_duration_seconds and vm_streamaggr_dedup_flush_duration_seconds histograms for monitoring the real flush durations of stream aggregation states.
FEATURE: vmui: improve trace display for better visual separation of branches. See this pull request.
FEATURE: vmagent: ability to limit the ingestion rate via -maxIngestionRate command-line flag. See this pull request.
FEATURE: vmagent: use the provided -remoteWrite.tlsServerName as Host header in requests to -remoteWrite.url. This allows sending data to https remote storage by IP address instead of hostname. Thanks to @minor-fixes for initial idea and the pull request.
FEATURE: vmagent: add -remoteWrite.shardByURL.ignoreLabels command-line flag, which can be used for specifying the ignored list of labels when sharding by -remoteWrite.url is enabled. Thanks to @edma2 for the idea and the pull request.
FEATURE: vmagent: automatically reload updated root CA certificates from files without the need to restart vmagent. See this issue.
FEATURE: optimize /api/v1/labels and /api/v1/label/.../values when match[] filters contains metric name. For example, /api/v1/label/instance/values?match[]=up now works much faster than before. See this issue.
FEATURE: vmagent: accept Prometheus remote write protocol data at /api/v1/push endpoint. This simplifies writing data to vmagent from Grafana Mimir. Thanks to @edma2 for the pull request.
FEATURE: vmctl: support client-side TLS configuration for native protocol. See this feature request. Thanks to @khushijain21 for the pull request.
FEATURE: vmctl: support client-side TLS configuration for VictoriaMetrics destination specified via --vm-* cmd-line flags used in InfluxDB, Remote Read protocol, OpenTSDB, Prometheus and Promscale migration modes.
FEATURE: vmctl: split explore phase in vm-native mode by time intervals when --vm-native-step-interval is specified. This should reduce probability of exceeding complexity limits for number of selected series during explore phase. See this issue.
FEATURE: vmgateway: add -logInvalidAuthTokens command-line flag, which can be used for logging invalid auth tokens. This is useful for debugging of auth token format issues. See this feature request.
FEATURE: graphite: add support for aggregateSeriesLists, diffSeriesLists, multiplySeriesLists and sumSeriesLists functions. Thanks to @rbizos for the pull request.
FEATURE: OpenTelemetry: add -opentelemetry.usePrometheusNaming command-line flag, which can be used for enabling automatic conversion of the ingested metric names and labels into Prometheus-compatible format. See these docs and this issue.
BUGFIX: prevent from automatic deletion of newly registered time series when it is queried immediately after the addition. The probability of this bug has been increased significantly after v1.99.0 because of optimizations related to registering new time series. See this and this issue.
BUGFIX: vmagent: properly set Host header in requests to scrape targets if it is specified via headers option. Thanks to @fholzer for the bugreport and the fix.
BUGFIX: vmagent: properly set Host header in requests to scrape targets when server_name option at tls_config is set. Previously the Host header was set incorrectly to the target hostname in this case.
BUGFIX: do not drop match[] filter at /api/v1/series if -search.ignoreExtraFiltersAtLabelsAPI command-line flag is set, since missing match[] filter breaks /api/v1/series requests.
BUGFIX: vmagent: return proper responses for AWS Firehose requests according to these docs. See this pull request and this issue.
BUGFIX: vmctl: properly parse TLS key and CA files for InfluxDB and OpenTSDB migration modes.
BUGFIX: vmui: fix VictoriaLogs UI query handling to correctly apply _time filter across all queries. See this issue.
BUGFIX: vmui: fix issue where step parameter wasn't updated after moving the focus outside the step input field in Firefox.
BUGFIX: vmui: fix freezing when pasting a query with autocomplete enabled. See this issue.
BUGFIX: vmui: fix auto-suggestion trigger issue after whitespace char. See this issue.
BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: limit duration of requests to /api/v1/labels, /api/v1/label/.../values or /api/v1/series with -search.maxLabelsAPIDuration duration. Before, -search.maxExportDuration value was used by mistake. The bug has been introduced in v1.99.0. Thanks to @kbweave for the pull request.
BUGFIX: properly wait for force merge to be completed during the shutdown. See this issue for the details.
BUGFIX: vmalert: set correct endsAt value in notifications sent to the Alertmanager. Previously, a rule with evaluation intervals lower than 10s could never be triggered. See this issue for details.
BUGFIX: vmalert: properly account for -rule.resendDelay for alerting rules that are constantly switching state from inactive to firing. Before, notifications for such rules could have been skipped if state change happened more often than -rule.resendDelay. See this pull request for details.
BUGFIX: vmalert: respect -remoteWrite.maxBatchSize at shutdown period. See this issue. Thanks to @jiekun for the pull request.
Unpack the victoria-metrics-*.tar.gz archive and read these docs.
vmutils-*.tag.gz archive contains the following tools:
vmutils-*-enterprise.tar.gz archive contains the following additional enterprise tools:
The corresponding docker images are available here. Cluster version is available here.
Released at 2024-03-01
This release contains the issue, which can prevent from storing data for new time series under high rate of search queries. Please upgrade to v1.100.0.
FEATURE: MetricsQL: propagate label filters via all the label manipulation functions. For example, label_del(some_metric{job="foo"}, "instance") + other_metric{pod="bar"} is now transformed to label_del(some_metric{job="foo",pod="bar"}, "instance") + other_metric{job="foo",pod="bar"}. This should reduce the amounts of time series processed during query execution.
FEATURE: MetricsQL: add count_values_over_time function. See this feature request.
FEATURE: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: expose vm_last_partition_parts metrics, which show the number of parts in the latest partition. These metrics may help debugging query performance slowdown related to the increased number of parts in the last partition, since usually all the ingested data is written to the last partition and all the queries are performed over the recently ingested data, e.g. the last partition.
FEATURE: Single-node VictoriaMetrics and vmstorage in VictoriaMetrics cluster: expose vm_snapshots metric, which shows the current number of snapshots created via snapshot API.
FEATURE: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: add -search.ignoreExtraFiltersAtLabelsAPI command-line flag, which can be used for reducing load on VictoriaMetrics when /api/v1/labels, /api/v1/label/.../values or /api/v1/series are queried with too broad extra_filters or extra_label, which match many time series. See these docs for details.
FEATURE: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: allow limiting CPU and RAM usage at /api/v1/labels, /api/v1/label/.../values and /api/v1/series on systems with high churn rate via -search.maxLabelsAPIDuration and -search.maxLabelsAPISeries command-line flags. See these docs for details.
FEATURE: vmagent: support reading Amazon CloudWatch metrics in OpenTelemetry format from Amazon Data Firehose.
FEATURE: vmagent: add support for client_id option into kuma_sd_configs in the same way as Prometheus does. See this pull request.
FEATURE: vmagent: add support for enable_compression option in scrape_configs in order to be compatible with Prometheus scrape configs. See this pull request and this feature request. Note that vmagent was always supporting disable_compression option before Prometheus added enable_compression option.
FEATURE: vmctl: support client-side TLS configuration for InfluxDB, Remote Read protocol and OpenTSDB. See this feature request. Thanks to @khushijain21 for pull requests 1, 2, 3.
FEATURE: vmui: preserve WITH templates when clicking the prettify query button at the right side of query input field. See this feature request.
FEATURE: vmui: allow filling gaps on graphs with interpolated lines as Grafana does. See this feature request and this pull request.
FEATURE: vmalert: support filtering by group, rule or labels in vmalert's UI for /groups and /alerts pages. See the pull request by @victoramsantos.
FEATURE: docker-compose: create a separate docker-compose environment for VictoriaLogs installation, including fluentbit and VictoriaLogs Grafana datasource.
FEATURE: vmbackupmanager: wait for up 30 seconds before making a snapshot for backup if vmstorage is temporarily unavailalbe. This should prevent from vmbackupmanager termination in this case. See this feature request.
BUGFIX: downgrade Go builder from 1.22.0 to 1.21.7, since 1.22.0 contains the bug, which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at vmagent. This may result in incorrect service discovery, target scraping and failed sending samples to remote storage.
BUGFIX: all VictoriaMetrics components: return back periodic closing of incoming connections to -httpListenAddr every 2 minutes, which was disabled in v1.98.0 when addressing this issue. See this comment for details on why the periodic closing of incoming connections has been returned back.
BUGFIX: vmagent: accept OpenTelemetry data at /opentelemetry/v1/metrics, since the /v1/metrics suffix is hardcoded at OpenTelemetry protocol specification.
BUGFIX: vmagent: fix possible deadlock when sharding among remote storages is enabled with -remoteWrite.shardByURL command-line flag. Thanks to @penguinlav for the fix for this issue.
BUGFIX: fix the misleading error 0ms is out of allowed range [0 ... when passing step=0 to /api/v1/query or /api/v1/query_range. See this issue.
BUGFIX: Single-node VictoriaMetrics and vmselect in VictoriaMetrics cluster: fixed floating-point error when parsing time in RFC3339 format. See this issue for details.
BUGFIX: vmalert: consistently sort groups by name and filename on /groups page in UI. This should prevent non-deterministic sorting for groups with identical names.
BUGFIX: vmui: fix Popper display issues. See this pull request.
BUGFIX: vmselect: format time with milliseconds precision when __timestamp__:rfc3339 is specified for /api/v1/export/csv API response. See this issue for details.
BUGFIX: MetricsQL: properly construct the destination label in label_join if it is used as source label. See this issue for details.
Unpack the victoria-metrics-*.tar.gz archive and read these docs.
vmutils-*.tag.gz archive contains the following tools:
vmutils-*-enterprise.tar.gz archive contains the following additional enterprise tools:
The corresponding docker images are available here. Cluster version is available here.
Released at 2024-03-01
v1.97.x is a line of LTS releases. It contains important up-to-date bugfixes for VictoriaMetrics enterprise. All these fixes are also included in the latest community release. The v1.97.x line will be supported for at least 12 months since v1.97.0 release
1.22.0 to 1.21.7, since 1.22.0 contains the bug, which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at vmagent. This may result in incorrect service discovery, target scraping and failed sending samples to remote storage.-search.maxUniqueTimeseries limit at /api/v1/labels, /api/v1/label/.../values and /api/v1/series. This limit was ignored at these endpoints starting from v1.97.0 when trying to address this issue. It has been appeared that this can result in high CPU usage when heaviweight queries hit these endpoints. This can prevent from executing lightweight queries to these endpoints. It is better to return limit exceeded error to heavyweight queries instead of burning CPU.-httpListenAddr every 2 minutes, which was disabled in v1.97.2 when addressing this issue. See this comment for details on why the periodic closing of incoming connections has been returned back./opentelemetry/v1/metrics, since the /v1/metrics suffix is hardcoded at OpenTelemetry protocol specification.-remoteWrite.shardByURL command-line flag. Thanks to @penguinlav for the fix for this issue.0ms is out of allowed range [0 ... when passing step=0 to /api/v1/query or /api/v1/query_range. See this issue.vmselect in VictoriaMetrics cluster: fixed floating-point error when parsing time in RFC3339 format. See this issue for details./groups page in UI. This should prevent non-deterministic sorting for groups with identical names.Unpack the victoria-metrics-*.tar.gz archive and read these docs.
vmutils-*.tag.gz archive contains the following tools:
vmutils-*-enterprise.tar.gz archive contains the following additional enterprise tools:
The corresponding docker images are available here. Cluster version is available here.
Released at 2024-03-01
v1.93.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release
1.22.0 to 1.21.7, since 1.22.0 contains the bug, which can lead to deadlocked HTTP connections to remote storage systems, scrape targets and service discovery endpoints at vmagent. This may result in incorrect service discovery, target scraping and failed sending samples to remote storage./opentelemetry/v1/metrics, since the /v1/metrics suffix is hardcoded at OpenTelemetry protocol specification.-remoteWrite.shardByURL command-line flag. Thanks to @penguinlav for the fix for this issue.0ms is out of allowed range [0 ... when passing step=0 to /api/v1/query or /api/v1/query_range. See this issue.vmselect in VictoriaMetrics cluster: fixed floating-point error when parsing time in RFC3339 format. See this issue for details./groups page in UI. This should prevent non-deterministic sorting for groups with identical names.Unpack the victoria-metrics-*.tar.gz archive and read these docs.
vmutils-*.tag.gz archive contains the following tools:
vmutils-*-enterprise.tar.gz archive contains the following additional enterprise tools:
The corresponding docker images are available here. Cluster version is available here.
Released at 2024-03-01
FEATURE: support the ability to limit the number of returned log entries from HTTP querying API by passing limit query arg. Previously all the matching log entries were returned until closing the response stream. See this feature request. Thanks to @dmitryk-dk for the pull request.
BUGFIX: do not panic on incorrect regular expression in stream filter. Thanks to @XLONG96 for the bugfix.
BUGFIX: properly determine when the assisted merge is needed. Previously the logs for determining whether the assisted merge is needed was broken. This could lead to too big number of parts under high data ingestion rate. Thanks to @lujiajing1126 for the fix.
BUGFIX: properly stop execution of aborted query when the query doesn't contain _stream filter. Previously such a query could continue consuming resources after being aborted by the client. Thanks to @z-anshun for the fix.
Released at 2024-02-14
v1.93.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release
SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7, plus the changelog for Go1.22.0.
BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example, sum({job="foo"}, {job="bar"}) by (job) + a was improperly optimized to sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"} before being executed. This could lead to unexpected results. See this issue.
BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.
BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.
BUGFIX: prevent from possible too big indexBlockSize panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics.
Unpack the victoria-metrics-*.tar.gz archive and read these docs.
vmutils-*.tag.gz archive contains the following tools:
vmutils-*-enterprise.tar.gz archive contains the following additional enterprise tools:
The corresponding docker images are available here. Cluster version is available here.
Released at 2024-02-14
v1.97.x is a line of LTS releases (e.g. Long-Time Support). It contains important up-to-date bugfixes for VictoriaMetrics enterprise. Note to our Community users: All these fixes are also included in the latest community release. Please upgrade to it in order to benefit from them. The v1.97.x line will be supported for at least 12 months since v1.97.0 release. We recommend to regularly upgrade to the latest VictoriaMetrics release, which always contains all the latest fixes.
SECURITY: upgrade Go builder from Go1.21.6 to Go1.22.0. See the list of issues addressed in Go1.21.7, plus the changelog for Go1.22.0.
BUGFIX: vmagent: reduce CPU usage when -promscrape.dropOriginalLabels command-line flag is set. This issue has been introduced in v1.96.0 when addressing this feature request.
BUGFIX: vmauth: properly release memory during config reload. See this issue.
BUGFIX: vmauth: properly expose vmauth_unauthorized_user_concurrent_requests_capacity, vmauth_unauthorized_user_concurrent_requests_current, vmauth_user_concurrent_requests_capacity and vmauth_user_concurrent_requests_current metrics after config reload. Previously these metrics didn't work after config reload.
BUGFIX: MetricsQL: properly propagate label filters from multiple arguments passed to aggregate functions. For example, sum({job="foo"}, {job="bar"}) by (job) + a was improperly optimized to sum({job="foo"}, {job="bar"}) by (job) + a{job="foo"} before being executed. This could lead to unexpected results. See this issue.
BUGFIX: MetricsQL: properly handle precision errors when calculating changes, changes_prometheus, increases_over_time and resets functions. See this issue.
BUGFIX: properly store staleness markers for self-scraped metrics on single-node VictoriaMetrics shutdown. See this issue.
BUGFIX: prevent from possible too big indexBlockSize panic when samples with too long label values (~64Kb) are ingested into VictoriaMetrics.
Unpack the victoria-metrics-*.tar.gz archive and read these docs.
vmutils-*.tag.gz archive contains the following tools:
vmutils-*-enterprise.tar.gz archive contains the following additional enterprise tools:
The corresponding docker images are available here. Cluster version is available here.