Agent

Prelude

Release on: 2024-04-30

• Please refer to the 7.53.0 tag on integrations-core for the list of changes on the Core Checks

New Features

• Support database-monitoring autodiscovery for Aurora cluster instances. Adds a new configuration listener to poll for a specific set of Aurora cluster IDs and then create a new database-monitoring supported check configuration for each endpoint. This allows for monitoring of endpoints that scale dynamically.
• Add new core check orchestrator_ecs to collect running ECS tasks
• APM stats now include an is_trace_root field to indicate if the stats are from the root span of a trace.
• The cluster-agent now collects network policies from the cluster.
• Enable 'host_benchmarks' by default when running the security-agent compliance module.
• OTLP ingest now has a feature flag to identify top-level spans by span kind. This new logic can be enabled by adding enable_otlp_compute_top_level_by_span_kind in DD_APM_FEATURES.
  • With this new logic, root spans and spans with a server or consumer span.kind will be marked as top-level. Additionally, spans with a client or producer span.kind will have stats computed.
  • Enabling this feature flag may increase the number of spans that generate trace metrics, and may change which spans appear as top-level in Datadog.
• Experimental: The process-agent checks (process, container, and process-discovery) can be run from the Core Agent in Linux. This feature can be toggled on by setting the process_config.run_in_core_agent.enabled flag to true in the datadog.yaml file. This feature is disabled by default.

Enhancement Notes

• Add the container image and container lifecycle checks to the output of the Agent status command.
• Add kubelet_core_check_enabled flag to Agent config to control whether the kubelet core check should be loaded.
• Added LastSuccessfulTime to cronjob status payload.
• Add a retry mechanism to Software Bill of Materials (SBOM) collection for container images. This will help to avoid intermittent failures during the collection process.
• Add startup timestamp to the Agent metadata payload.
• Agents are now built with Go 1.21.9.
• Adds image repo digest string to the container payload when present
• CWS: Add selftests report on Windows and platforms with no eBPF support.
• CWS: Add visibility for cross container program executions on platforms with no eBPF support.
• APM: Enable credit card obfuscation by default. There is a small chance that numbers that are similar to valid credit cards may be redacted, this feature can be disabled by using apm_config.obfuscation.credit_cards.enabled. Alternatively, it can be made more accurate through luhn checksum verification by using apm_config.obfuscation.credit_cards.luhn, however, this increases the performance penalty of this check.
• logs_config.expected_tags_duration now works for journald logs.
• [oracle] Adds oracle.can_query service check.
• [oracle] Automatically fall back to deprecated Oracle integration mode if privileges are missing.
• [oracle] Add service configuration parameter.
• The connections check no longer relies on the process/container check as it can now fetch container data independently.
• The performance of Remote Config has been significantly improved when large amounts of configurations are received.
• Send ECS task lifecycle events in the container lifecycle check.
• dbm: add new SQL obfuscation mode normalize_only to support normalizing SQL without obfuscating it. This mode is useful for customers who want to view unobfuscated SQL statements. By default, ObfuscationMode is set to obfuscate_and_normalize and every SQL statement is obfuscated and normalized.
• USM: Handle the HTTP TRACE method.

Deprecation Notes

• [oracle] Deprecating Oracle integration code. The functionality is fully implemented in the oracle-dbm check which is now renamed to oracle.

Bug Fixes

• The windows_registry check can be run with the check sub-command.
• CWS: Fix very rare event corruption.
• Fixes issue where processes for ECS Fargate containers would sometimes not be associated with the correct container.
• Fixed a bug in the Dual Shipping feature where events were not being emitted on endpoint recovery.
• Fix issue with display_container_name being tagged as N/A when container_name information is available.
• Fix a Windows process handle leak in the Process Agent, which was introduced in 7.52.0 when process_collection is enabled.
• Fixes a bug where the tagger server did not properly handle a closed channel.
• [oracle] Set the default for metric_prefix in custom_queries to oracle.
• [oracle] Fix global_custom_queries bug.
• [oracle] Adds the oracle.process.pga_maximum_memory metric for backward compatibility.
• Stop sending systemd metrics when they are not set

Datadog Cluster Agent

Prelude

Released on: 2024-04-30 Pinned to datadog-agent v7.53.0: CHANGELOG.

New Features

• APM library injection now works on EKS Fargate when the admission controller is configured to add an Agent sidecar in EKS Fargate.
• Cluster Agent now supports activating Application Security Management, Code Vulnerabilities, and Software Composition Analysis via Helm charts.

Enhancement Notes

• Add the mutation_webhook tag to admission_webhooks.webhooks_received and admission_webhooks.response_duration Cluster Agent telemetry.
• When using the admission controller to inject an Agent sidecar on EKS Fargate, shareProcessNamespace is now set to true automatically. This is to ensure that the process collection feature works.

Agent

Prelude

Release on: 2024-04-04

Enhancement Notes

• Add a check to the Windows installer to verify that the caller has the correct membership to install the Agent.
• Ensure the metadata requests are delayed at Agent startup to reduce host tag delays.

Agent

Prelude

Release on: 2024-03-21

• Please refer to the 7.52.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• To prevent misconfigurations, the Windows Datadog Agent installer now raises an error if the user account running the installer MSI is provided as the ddagentuser (DDAGENTUSER_NAME) account. If the account is a service account, such as LocalSystem or a gMSA account, no action is needed. If the account is a regular account, configure a different Datadog Agent service account.

New Features

• Add device_type to the device metadata.
• Attach host tags to metrics for expected_tags_duration amount of time.
• APM stats will now include, if present, the Git commit SHA from traces (or container tags) and the image tag from container tags.
• Creation of a new packageSigning component to collect Linux package signature information and improve signature rotation process. More information can be found in DataDog documentation at 2024 linux key rotation.
• Adds support for span links in the trace agent. This field contains a list of casual relationships between spans and is only populated when v0.4 of the Trace API is used.
• The Windows Agent now supports CWS for process and network threats.
• CWS: Add chdir event to allow recent container escape detection.
• CWS: [BETA] Add File Integrity Monitoring support on Windows, supporting both files and registry.
• CWS: The Agent now automatically suppresses benign security events if they have already been reported for a particular container image.
• Updating process agent discovery configuration to include a Data Scrubber for obfuscating sensitive information such as passwords, API keys, or tokens.
• Add support for pinging network devices in the SNMP integration.
• [oracle] Add oracle.locks.transaction_duration metric.
• APM: Add support for Single Step Instrumentation remote configuration
• Headless agent installation support on macOS 14 and later

Enhancement Notes

• [DBM] Increase the DBM dbm-metrics-intake endpoint's defaultInputChanSize value to 500.
• Add debug level logs when files are evicted from registry.json after their TTL expires.
• Add the instance ID returned by the IMDSv2 metadata endpoint to the list of EC2 host aliases.
• This change adds journald permissions to the flare in the logs_file_permissions.log file, in the form of either the journald directory or a specific file (if specified by the Agent journald configuration).
• The Logs Agent now creates a file in the flare, called logs_file_permissions.log, which lists every file and that file's permissions that the Logs Agent can detect.
• Add the SBOM check to the output of the Agent status command and the Agent flare.
• Add the Software Bill of Materials (SBOM) for container images to the output of the flare command.
• Add repo_digest to containerd ContainerImage to remove duplicate images in container images UI.
• Agents are now built with Go 1.21.7.
• Agents are now built with Go 1.21.8.
• CWS: Improved coverage on platforms with no eBPF support.
• CWS: Send context of variables in events.
• Add DD_APM_DEBUGGER_DIAGNOSTICS_DD_URL, DD_APM_DEBUGGER_DIAGNOSTICS_API_KEY, and DD_APM_DEBUGGER_DIAGNOSTICS_ADDITIONAL_ENDPOINTS to allow sending Live Debugger / Dynamic Instrumentation diagnostic data to multiple intakes.
• Added config that allows user to toggle on and off the collection of zombie processes in the Process Agent.
• [oracle] Add ddagenthostname tag.
• [oracle]: Add oracle.tablespace.maxsize metric.
• OTLP ingest supports stable Java runtime metrics introduced in opentelemetry-java-instrumentation v2.0.0. OTLP ingest supports Kafka metrics mapping. This allows users of the JMX Receiver/JMX Metrics Gatherer and Kafka metrics receiver to have access to the OOTB Kafka Dashboard.
• Modified the process check to populate process with the newly created field "ProcessContext"
• Rename the kubelet_core check to kubelet and change the metrics prefix from kubernetes_core to kubernetes so that it can replace the Python kubelet check.
• APM: Adds msgp_short_bytes reason for trace payloads dropped to distinguish them from EOF errors.
• When getting resource tags from an ECS task with zero containers, print a warn log instead of error log.

Deprecation Notes

• Removal of the pod check from the process agent. The current check will run from the core agent.
• This release drops support for Red Hat Enterprise Linux 6 and its derivatives.
• [oracle] Deprecate the configuration parameter instant_client. Replacing it with oracle_client.
• Removed the system-probe configuration value data_streams_config.enabled and replaced it with service_monitoring_config.enable_kafka_monitoring. This also implies that the DsmEnabled field in the AgentConfiguration proto will consistently be set to false.

Bug Fixes

• Upgrade dependencies for systemd core check. This silences excessive warning logs on systemd v252.
• oracle: Fix wrong tablespace metrics.
• APM: Stop dropping incoming OTel payloads when the processing channel is full and eliminate OOM issues in the trace agent and collector component in high load scenarios, making the OTel pipeline more reliable.
• Fix dogstatsd-capture. Message PID was not set after the 7.50 release.
• Fix a memory exception where the flare controller tries to stat a file that doesn't exist.
• Fleet Automation filters in the Datadog UI now accurately reflect which products are enabled when deployed with the official DataDog Helm chart on Kubernetes.
• Corrected a problem where the ignore_autodiscovery_tags parameter was not functioning correctly with pod annotations or autodiscovery version 2 (adv2) annotations. This fix ensures that when this parameter is set to true, autodiscovery tags are ignored as intended. Example:

yaml ad.datadoghq.com/redis.checks: | { "redisdb": { "ignore_autodiscovery_tags": true, "instances": [ { "host": "%%host%%", "port": "6379" } ] } } Moving forward, configurations that attempt to use hybrid setups—combining adv2 for check specification while also employing `adv1 for ignore_autodiscovery_tags—are no longer supported by default. Users should set the configuration parameter cluster_checks.support_hybrid_ignore_ad_tags to true to enable this behavior.

• [oracle]: Add support for more Asian character sets.
• Prevention of OOMs when collecting a large number of zombie processes.
• Fixed race conditions caused by concurrent execution of etw.StartEtw() and etw.StopEtw() functions which may concurrently access and modify a global map.
• Fix recent PR #22664 which in turn fixes a race condition in the ETW package. The previous PR introduced a minor error addressed in this PR.
• [oracle] Add resource_manager configuration to conf.yaml.example.
• [oracle] Fix multi-tagging bug.
• Fixes a bug in OTLP ingest where empty histograms were not being sent to the backend in the distributions mode. Empty histograms are now mapped as if they had a single (min, max) bucket.
• Scrub authentication bearer token of any size, even invalid, from integration configuration (when being printed through the checksconfig CLI command or other).
• Empty UDS payloads no longer cause the DogStatsD server to close the socket.

Other Notes

• The version of Python required for tooling in README matches that which the CI uses.

Datadog Cluster Agent

New Features

• Add agent sidecar injection webhook in cluster-agent Kubernetes admission controller. This new webhook adds the Agent as sidecar container in applicative Pods when it is required by the environment. For example with the EKS Fargate environment.

Enhancement Notes

• Introduces a new config option in the Cluster Agent to set the rebalance period when advanced dispatching is enabled: cluster_checks.rebalance_period. The default value is 10 min.

Bug Fixes

• Fix an issue where the admission controller would remove the field restartPolicy from native sidecar containers, preventing pod creation on Kubernetes 1.29+.
• Fix missing kube_api_version tag on HPA and VPA resources.

Agent

Prelude

Release on: 2024-02-29

• Please refer to the 7.51.1 tag on integrations-core for the list of changes on the Core Checks

New Features

• Add the chdir event type to CWS.

Security Notes

• Bump embedded Python version to 3.11.8 to address CVE-2023-5678 on Windows.

Bug Fixes

• Fix a crash in the win32_event_log check that occurs when processing an event that has a missing publisher and no EventData.

Agent

Prelude

Release on: 2024-02-19

• Please refer to the 7.51.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• The orchestrator check is moving from the Process Agent to the Core Agent. Any orchestrator configuration set on the Process Agent will need to be moved to the Core Agent. No other changes are required. If you need to go back to the old check, you can do so temporarily by manually setting the environment variable DD_ORCHESTRATOR_EXPLORER_RUN_ON_NODE_AGENT to false. The Process Agent pod check will be deprecated in the following release.
• Upgrade the Python version from 3.9 to 3.11.

New Features

• Add support for ARM64 SLES flavor of datadog-agent
• Add support for multiple users when listening for SNMP traps.
• Add check_delay metric in Agent telemetry
• Add an ETW component for ETW tracing.
• Add an ETW APM tracer component to forward .Net ETW events to the Tracer Agent.
• DBM: Add configuration options to SQL obfuscator to customize the normalization of SQL statements:
  • KeepTrailingSemicolon - disable removing trailing semicolon. This option is only valid when ObfuscationMode is obfuscate_and_normalize.
  • KeepIdentifierQuotation - disable removing quotation marks around identifiers. This option is only valid when ObfuscationMode is obfuscate_and_normalize.
• CWS: [BETA] early support based on ptrace for platforms with no eBPF support. Only processes and files are currently supported.
• Add msodbcsql18 linux dependency needed for SQL Server to run in Docker Agent.
• Add timestamps to the logs HTTP client
• Add support for Oracle Active Data Guard.
• Re-enable Aerospike in SUSE packages.
• The Windows Registry integration can now send the registry values as logs.

Enhancement Notes

• Updated the ntp check to support the default location of chrony.conf on Ubuntu (/etc/chrony/chrony.conf).

• Agents are now built with Go 1.21.5.

• CWS: Reloading the datadog-agent-sysprobe systemd service now reloads the runtime security policies.

• CWS: Added ssdeep file hashing algorithm support.

• USM will report the actual status code of the HTTP traffic, instead of reporting only the status code family (2xx, 3xx, etc.).

• Improved performance of the activity sampling query on RDS and Oracle Cloud databases.

• OTLP ingest log timestamps (i.e. '@timestamp') now include milliseconds.

• Always report the following telemetry metrics about the retry queue capacity:

  • datadog.agent.retry_queue_duration.capacity_secs
  • datadog.agent.retry_queue_duration.bytes_per_sec
  • datadog.agent.retry_queue_duration.capacity_bytes

• Support container metrics for kata containers using containerd.

• System Probe can now expose its healthcheck on a dedicated HTTP port. The Kubernetes daemonset uses this by default on port 5558.

Deprecation Notes

• The config value ipc_address is deprecated in favor of cmd_host.
• service_monitoring_config.process_service_inference.enabled is deprecated and replaced by system_probe_config.process_service_inference.enabled service_monitoring_config.process_service_inference.use_windows_service_name is deprecated and replaced by system_probe_config.process_service_inference.use_windows_service_name
• Removes freetds and msodbcsql18 dependencies for py2.
• Removes postgresql dependency after upgrading psycopg2 to v2.9 in integrations-core. psycopg2 now comes with pre-built wheel for arm architecture.
• An error will now be logged if replace tags are used to change the Agent "env", since this could have negative side effects. At this time, an error is logged, but future versions may explicitly disallow this to avoid bugs. See https://docs.datadoghq.com/getting_started/tracing/#environment-name for instructions on setting the env, and https://github.com/DataDog/datadog-agent/issues/21253 for more details about this issue.

Bug Fixes

• CWS/CSPM: Fixes the hostname value attached to CWS and CSPM events, which in rare cases the security agent computed incorrectly.
• Fix file_handle core check on Darwin by using sysctl system call.
• Fix spikes for bandwidth usage metric when interface speed is auto-adjusted.
• Fixes Agent startup script when enabling OOM Kill and TCP Queue Length checks to prevent crashes when restarting the container.
• Fix a spewing error message ("DCA Client not initialized by main provider, cannot post heartbeat") in the cluster check runner log during CLC initialization.
• Fixed Logs Agent additional endpoints to respect their logs_no_ssl setting.
• [DBM] Add Oracle broken connection handling on Windows
• Fix indentation in conf.yaml.example.
• Bug fix for empty database names in query samples.
• Bug fix for the Korean character set for Windows.
• Fixing the issue with a Korean character set for Windows.
• Fix missing sysmetrics, such as shared pool and library cache.
• Bug fix for missing tags.
• Fixed obfuscation error false positive when the access or filter predicates are empty.
• Fix resource manager metrics collection bugs.
• Pause containers from the Rancher image-mirror repository (rancher/mirrored-pause.*) are now excluded by default for containers and metrics collection.
• Error messages from Go checks are now shown on the Agent GUI status page instead of UNKNOWN ERROR.

Other Notes

• Update s6-overlay version used in Datadog Agent container images to v2.2.0.3
• Added a warning when logs_no_ssl is set and dd_url contains an https prefix. logs_no_ssl will take precedence over the prefix in a future version.

Datadog Cluster Agent

Prelude

Released on: 2024-02-19 Pinned to datadog-agent v7.51.0: CHANGELOG.

New Features

• Enable Horizontal Pod Autoscaler collection for the Orchestrator by default
• Add isolate command to clusterchecks to make it easier to pinpoint a check that that is causing high CPU/memory usage. Command can be run in the cluster agent with: datadog-cluster-agent clusterchecks isolate --checkID=<checkID>

Enhancement Notes

• Enable CRD collection by default in the orchestrator check.

Bug Fixes

• Fixes a bug that would trigger unnecessary APIServer List requests from the Cluster Agent or Cluster Checks Runner.

Prelude

Release on: 2024-01-11

Bug Fixes

• Fix incorrect metadata about system-probe being sent to Inventory and Fleet Automation products.

Prelude

Release on: 2024-01-04

• Please refer to the 7.50.2 tag on integrations-core for the list of changes on the Core Checks

Enhancement Notes

• Agents are now built with Go 1.20.12.

Bug Fixes

• The CWS configuration parameter to enable anomaly detection is now working and taken into account by the Agent.
• Fix issue introduced in 7.47 that allowed all users to start/stop the Windows Datadog Agent services. The Windows installer now, as in versions before 7.47, grants this permission explicitly to ddagentuser.

Agent and Datadog Cluster Agent

Prelude

Release on: 2023-12-21

Bug Fixes

• Fixes a bug introduced in 7.50.0 preventing DD_TAGS to be added to kubernetes_state.* metrics.

Agent

Prelude

Release on: 2023-12-19

• Please refer to the 7.50.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• The win32_event_log check has moved from Python (integrations-core#16108) to Go (#20701 <https://github.com/DataDog/datadog-agent/pull/20701>). All legacy_mode: false configuration options are backwards compatible except for some regular expressions used in the included_messages and excluded_messages options. For example, Go regular expressions do not support lookahead or lookbehind assertions. If you do not use these options, then no configuration changes are necessary. See the Python regular expression docs and the Go regular expression docs for more information on the supported regular expression syntax. Set legacy_mode_v2: true to revert to the Python implementation of the check. The Python implementation may be removed in a future version of the Agent.

New Features

• The orchestrator check is moving from the Process Agent to the Node Agent. In the next release, this new check will replace the current pod check in the Process Agent. You can start using this new check now by manually setting the environment variable DD_ORCHESTRATOR_EXPLORER_RUN_ON_NODE_AGENT to true.

• Adds the following CPU manager metrics to the kubelet core check: kubernetes_core.kubelet.cpu_manager.pinning_errors_total, kubernetes_core.kubelet.cpu_manager.pinning_requests_total.

• Add a diagnosis for connecting to the agent logs endpoints. This is accessible through the agent diagnose command.

• Add FIPS mode support for Network Device Monitoring products

• Added support for collecting Cloud Foundry container names without the Cluster Agent.

• The Kubernetes State Metrics Core check now collects kubernetes_state.ingress.tls.

• APM: Added a new endpoint tracer_flare/v1/. This endpoint acts as a proxy to forward HTTP POST request from tracers to the serverless_flare endpoint, allowing tracer flares to be triggered via remote config, improving the support experience by automating the collection of logs.

• CWS: Ability to send a signal to a process when a rule was triggered. CWS: Add Kubernetes user session context to events, in particular the username, UID and groups of the user that ran the commands remotely.

• Enable container image collection by default.

• Enable container lifecycle events collection by default. This feature helps stopped containers to be cleaned from Datadog faster.

• [netflow] Allow collecting configurable fields for Netflow V9/IPFIX

• Add support for Oracle 12.1 and Oracle 11.

• Add monitoring of Oracle ASM disk groups.

• Add metrics for monitoring Oracle resource manager.

• [corechecks/snmp] Load downloaded profiles

• DBM: Add configuration option to SQL obfuscator to use go-sqllexer package to run SQL obfuscation and normalization

• Support filtering metrics from endpoint and service checks based on namespace when the DD_CONTAINER_EXCLUDE_METRICS environment variable is set.

• The Windows Event Log tailer saves its current position in an event log and resumes reading from that location when the Agent restarts. This allows the Agent to collect events created before the Agent starts.

Enhancement Notes

• [corechecks/snmp] Support symbol modifiers for global metric tags and metadata tags.
• Update the go-systemd package to the latest version (22.5.0).
• Added default peer tags for APM stats aggregation which can be enabled through a new flag (peer_tags_aggregation).
• Add a stop timeout to the Windows Agent services. If an Agent service does not cleanly stop within 15 seconds after receiving a stop command from the Service Control Manager, the service will hard stop. The timeout can be configured by setting the DD_WINDOWS_SERVICE_STOP_TIMEOUT_SECONDS environment variable. Agent stop timeouts are logged to the Windows Event Log and can be monitored and alerted on.
• APM: OTLP: Add support for custom container tags via resource attributes prefixed by datadog.container.tag.*.
• Agents are now built with Go 1.20.11.
• CWS: Support for Ubuntu 23.10. CWS: Reduce memory usage of ring buffer on machines with more than 64 CPU cores. CSPM: Move away from libapt to run Debian packages compliance checks.
• DBM: Bump the minimum version of the go-sqllexer library to 0.0.7 to support collecting stored procedure names.
• Add subcommand diagnose show-metadata gohai for gohai data
• Upgraded JMXFetch to 0.49.0 which adds some more telemetry and contains some small fixes.
• Netflow now supports the datadog-agent status command, providing configuration information. Any configuration errors encountered will be listed.
• Emit database_instance tag with the value host/cdb. The goal is to show each database separately in the DBM entry page. Currently, the backend initializes database_instance to host. Also, the Agent will emit the new db_server tag because we have to initialize the host tag to host/cdb.
• Improve obfuscator formatting. Prevent spaces after parentheses. Prevent spaces before # when # is a part of an identifier.
• Emit query metrics with zero executions to capture long runners spanning over several sampling periods.
• Impose a time limit on query metrics processing. After exceeding the default limit of 20s, the Agent stops emitting execution plans and fqt events.
• Add oracle.inactive_seconds metric. Add tags with session attributes to oracle.process_pga* metrics.
• Stop override peer.service with other attributes in OTel spans.
• Process-Agent: Improved parsing performance of the '/proc/pid/stat' file (Linux only)
• [snmp_listener] Enable collect_topology by default.
• dbm: add SQL obfuscation options to give customer more control over how SQL is obfuscated and normalized.
  • RemoveSpaceBetweenParentheses - remove spaces between parentheses. This option is only valid when ObfuscationMode is obfuscate_and_normalize.
  • KeepNull` - disable obfuscating null values with ?. This option is only valid whenObfuscationModeis "obfuscate_only" orobfuscate_and_normalize``.
  • KeepBoolean - disable obfuscating boolean values with ?. This option is only valid when ObfuscationMode is obfuscate_only or obfuscate_and_normalize.
  • KeepPositionalParameter - disable obfuscating positional parameters with ?. This option is only valid when ObfuscationMode is obfuscate_only or obfuscate_and_normalize.
• Add logic to support multiple tags created by a single label/annotaion. For example, add the following config to extract tags for chart_name and app_chart_name. podLabelsAsTags: chart_name: chart_name, app_chart_name Note: the format must be a comma-separated list of tags.
• The logs collection pipeline has been through a refactor to support processing only the message content (instead of the whole raw message) in the journald and Windows events tailers. This feature is experimental and off by default since it changes how existing log_processing_rules behaves with journald and Windows events tailer. Note that it will be switched on by default in a future release of the Agent. A warning notifying about this is shown when the journald and Windows events tailers are used with some log_processing_rules.
• The Datadog agent container image is now using Ubuntu 23.10 mantic as the base image.
• The win32_event_log check now continuously collects and reports events instead of waiting for min_collection_interval to collect. min_collection_interval now controls how frequently the check attempts to reconnect when the event subscription is in an error state.

Deprecation Notes

• Installing the Agent on Windows Server versions lower than 2016 and client versions lower than 10 is now deprecated.
• The timeout option for the win32_event_log check is no longer applicable and can be removed. If the option is set, the check logs a deprecation warning and ignores the option.

Security Notes

• Fix CVE-2023-45283 and CVE-2023-45284
• Update OpenSSL from 3.0.11 to 3.0.12. This addresses CVE-2023-5363.

Bug Fixes

• On Windows, uninstalling the Agent should not fail if the Datadog Agent registry key is missing.
• APM: OTLP: Only extract DD container tags from resource attributes. Previously, container tags were also extracted from span attributes.
• APM: OTLP: Only add container tags in tag _dd.tags.container. Previously, container tags were also added as span tags.
• Resolved an issue in the containerd collector where the SBOM collection did not correctly attach RepoTags and RepoDigests to the SBOM payload.
• Add a workaround for a bug in a Windows API that can cause the Agent to crash when collecting forwarded events from the Windows Event Log.
• Resolve the issue with hostname resolution in the kube_apiserver provider when the useHostNetwork setting is enabled.
• Fix an issue that prevented process ID (PID) from being associated with containers in Live Container View when the Agent is deployed in AWS Fargate.
• APM: Fixed trace-agent not forwarding errors from remote configuration and reporting them all as 500s
• On Windows, the SE_DACL_AUTO_INHERITED flag is reset on %PROJECTLOCATION% during upgrades and uninstalls.
• Fixes a bug in the Windows NPM driver where NPM displays byte overcounts.
• For USM on Windows, fixes the problem where paths were being erroneously reported as truncated
• Fixes journald log's Seek function to be set at the beginning or end upon initialization.
• Fixed the cause of some crashes related to CPU instruction incompatibility happening under certain CPUs when making calls to the included libgmp library.
• [kubelet] The Kubelet client no longer fails to initialize when the parameter kubelet_tls_verify is set to false with a misconfigured root certificate authority.
• Fixes a bug where the process-agent process check command would fail to run when language detection was enabled.
• Document query metrics metric_prefix parameter.
• Set the tag dd.internal.resource:database_instance to host instead of host/cdb.
• Switch to the new obfuscator where bugs such as getting an error when obfuscating @! and where comments on DMLs weren't being removed are fixed.
• Fixes wrong values in Oracle query metrics data. Extreme cases had inflated statistics and missing statements. The affected were pure DML and PL/SQL statements.
• Fix the bug that prevented Oracle DBM working properly on AWS RDS non-multitenant instances.
• Fix an issue that caused the win32_event_log check to not stop running when the rate of incoming event records was higher than the timeout option. The timeout option is now deprecated.
• The Windows Event Log tailer automatically recovers and is able to resume collecting events when a log provider is reinstalled, which sometimes happens during Windows updates.

Datadog Cluster Agent

New Features

• Add language detection API handler to the cluster-agent.
• Report rate_limit_queries_remaining_min telemetry from external-metrics server.
• Added a new --force option to the datadog-cluster-agent clusterchecks rebalance command that allows you to force clustercheck rebalancing with utilization.
• [Beta] Enable APM library injection in cluster-agent admission controller based on automatic language detection annotations.

Enhancement Notes

• Show Autodiscovery information in the output of datadog-cluster-agent status.
• Added CreateContainerConfigError wait reason to the kubernetes_state.container.status_report.count.waiting metric reported by the kubernetes_state_core check.
• Release the Leader Election Lock on shutdown to make the initialization of future cluster-agents faster.
• The Datadog cluster-agent container image is now using Ubuntu 23.10 mantic as the base image.

Bug Fixes

• Fixed a bug in the kubernetes_state_core check that caused tag corruption when telemetry was set to true.
• Fix stale metrics being reported by kubernetes_state_core check in some rare cases.
• Fixed a bug in the rebalancing of cluster checks. Checks that contained secrets were never rebalanced when the Cluster Agent was configured to not resolve check secrets (option secret_backend_skip_checks set to true).

Prelude

Release on: 2023-11-16

• Please refer to the 7.49.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• CWS: add arch field into agent context included in CWS events.
• APM: Fix a deadlock issue which can prevent the trace-agent from shutting down.
• CWS: Fix the broken lineage check for process activity in CWS.
• APM: fix a regression in the Trace Agent that caused container tagging with UDS and cgroup v2 to fail.