[gardener/gardener]

📰 Noteworthy

• [OPERATOR] Five minutes Infrastructure Cleanup Wait Period during shoot deletion was removed. Shoot annotation shoot.gardener.cloud/infrastructure-cleanup-wait-period-seconds which could be used to configure this period was removed, too. by @oliver-goetz [#9632]
• [DEVELOPER] The tools installed via the tools.mk make file are now by default installed in an OS and arch specific folder to allow running make targets from different platforms sharing the same source code.
  The previous behavior can be achieved by setting the variable TOOLS_BIN_DIR to hack/tools/bin to any make target. by @vpnachev [#9589]
• [DEVELOPER] Today's method of providing Plutono dashboards for garden or shoot clusters is deprecated and will be removed in a future release. Migrate to the new approach (see this document) for details. by @rfranzke [#9624]

✨ New Features

• [OPERATOR] gardener-operator is now managing the Gardener Dashboard web terminal controller manager when .spec.virtualCluster.gardener.gardenerDashboard.terminal is set in the Garden resource. Read more about it here by @rfranzke [#9646]
• [OPERATOR] gardener-node-agent no longer watches all Nodes in the cluster but restricts to only the Node it is responsible for (with the help of label/field selectors). This should lead to a significant reduction of network I/O, especially for shoot clusters with many nodes. by @rfranzke [#9672]
• [OPERATOR] gardener-operator now deploys two more Prometheus replicas into the garden namespace for storing long-term metrics. Read more about it here. by @rfranzke [#9606]
• [OPERATOR] A new feature gate named VPAForETCD is now introduced for gardenlet and gardener-operator. When enabled, VPA for etcd is used, regardless of the HVPA feature gate setting. The new VPA limits scaling down to a Shoot's maintenance window or even entirely based on the ShootClass in the same way as it is currently done for HVPA. by @voelzmo [#8984]
• [OPERATOR] gardener-operator is now managing the Gardener Dashboard when .spec.virtualCluster.gardener.gardenerDashboard is set in the Garden resource. Read more about it here by @rfranzke [#9583]
• [USER] It is now possible to define a higher number of maximum worker count in a shoot than pods and nodes networks allow. cluster-autoscaler ensures that not more nodes than the networking settings allow will be created. by @oliver-goetz [#9599]

🐛 Bug Fixes

• [OPERATOR] gardener-operator is now capable of reconciling shoot cluster-specific NetworkPolicys in case the garden cluster is a seed cluster at the same time. by @rfranzke [#9658]
• [OPERATOR] Fixed prometheus alerting rules for Seeds with unhealthy control-planes by @voelzmo [#9692]
• [OPERATOR] In the migrate flow of control plane migration the Deleting extensions before kube-apiserver task now depends on the Waiting until extension resources have been deleted task. by @plkokanov [#9651]
• [OPERATOR] Only update network policy allow-to-runtime-apiserver after resolver has been synced. by @MartinWeindel [#9644]

🏃 Others

• [OPERATOR] Updated VPA to 1.1.1 by @voelzmo [#8984]
• [OPERATOR] If a previous file copy attempt failed gardener-node-agent now deletes leftover *.tmp files instead of returning an error. by @oliver-goetz [#9630]
• [OPERATOR] extension library: An issue causing the backup.gardener.cloud/created-by annotation not being added on existing etcd-backup Secrets is now fixed. by @ialidzhikov [#9613]
• [OPERATOR] Added a cleanup function to gardenlet which is executed at startup and deletes orphaned VPAs with label role: vali-vpa that were previously managed by the HVPA deployed for vali. by @plkokanov [#9681]
• [OPERATOR] The gardenlet now runs as nonroot user and group 65532. by @AleksandarSavchev [#9669]
• [OPERATOR] A new plutono dashboard named Resource usage by container is added to garden/plutono. It shows aggregated CPU/memory usage vs requests/limits and utilization per container (currently only metrics for kube-apiserver containers are federated). by @ialidzhikov [#9643]
• [OPERATOR] Containers, configured to run as non-root, are now validated to start with non-root user by the kubelet. by @AleksandarSavchev [#9640]
• [OPERATOR] The fluent-operator component now runs as nonroot user and group 65532. by @AleksandarSavchev [#9640]
• [OPERATOR] The kube-controller-manager's (H)VPA minAllowed memory is reduced from 100Mi to 50Mi. The kube-apiserver's HVPA minAllowed memory is reduced from 400M to 200M. by @ialidzhikov [#9654]

Docker Images

• admission-controller: europe-docker.pkg.dev/gardener-project/releases/gardener/admission-controller:v1.94.0
• apiserver: europe-docker.pkg.dev/gardener-project/releases/gardener/apiserver:v1.94.0
• controller-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/controller-manager:v1.94.0
• gardenlet: europe-docker.pkg.dev/gardener-project/releases/gardener/gardenlet:v1.94.0
• node-agent: europe-docker.pkg.dev/gardener-project/releases/gardener/node-agent:v1.94.0
• operator: europe-docker.pkg.dev/gardener-project/releases/gardener/operator:v1.94.0
• resource-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/resource-manager:v1.94.0
• scheduler: europe-docker.pkg.dev/gardener-project/releases/gardener/scheduler:v1.94.0

[gardener/gardener]

🐛 Bug Fixes

• [USER] A bug which mounted the kubelet data volume to /var/lib instead of /var/lib/kubelet when kubeletDataVolumeName was set has been fixed. by @oliver-goetz [#9615]
• [OPERATOR] A bug in gardener-node-agent which prevented copying files between different block devices has been fixed. by @oliver-goetz [#9615]

Docker Images

• admission-controller: europe-docker.pkg.dev/gardener-project/releases/gardener/admission-controller:v1.92.2
• apiserver: europe-docker.pkg.dev/gardener-project/releases/gardener/apiserver:v1.92.2
• controller-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/controller-manager:v1.92.2
• gardenlet: europe-docker.pkg.dev/gardener-project/releases/gardener/gardenlet:v1.92.2
• node-agent: europe-docker.pkg.dev/gardener-project/releases/gardener/node-agent:v1.92.2
• operator: europe-docker.pkg.dev/gardener-project/releases/gardener/operator:v1.92.2
• resource-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/resource-manager:v1.92.2
• scheduler: europe-docker.pkg.dev/gardener-project/releases/gardener/scheduler:v1.92.2

[gardener/gardener]

🐛 Bug Fixes

• [OPERATOR] A bug in gardener-node-agent which prevented copying files between different block devices has been fixed. by @oliver-goetz [#9618]
• [USER] A bug which mounted the kubelet data volume to /var/lib instead of /var/lib/kubelet when kubeletDataVolumeName was set has been fixed. by @oliver-goetz [#9618]

Docker Images

• admission-controller: europe-docker.pkg.dev/gardener-project/releases/gardener/admission-controller:v1.90.8
• apiserver: europe-docker.pkg.dev/gardener-project/releases/gardener/apiserver:v1.90.8
• controller-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/controller-manager:v1.90.8
• gardenlet: europe-docker.pkg.dev/gardener-project/releases/gardener/gardenlet:v1.90.8
• node-agent: europe-docker.pkg.dev/gardener-project/releases/gardener/node-agent:v1.90.8
• operator: europe-docker.pkg.dev/gardener-project/releases/gardener/operator:v1.90.8
• resource-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/resource-manager:v1.90.8
• scheduler: europe-docker.pkg.dev/gardener-project/releases/gardener/scheduler:v1.90.8

[gardener/gardener]

🐛 Bug Fixes

• [USER] A bug which mounted the kubelet data volume to /var/lib instead of /var/lib/kubelet when kubeletDataVolumeName was set has been fixed. by @oliver-goetz [#9617]
• [OPERATOR] A bug in gardener-node-agent which prevented copying files between different block devices has been fixed. by @oliver-goetz [#9617]

Docker Images

• admission-controller: europe-docker.pkg.dev/gardener-project/releases/gardener/admission-controller:v1.91.4
• apiserver: europe-docker.pkg.dev/gardener-project/releases/gardener/apiserver:v1.91.4
• controller-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/controller-manager:v1.91.4
• gardenlet: europe-docker.pkg.dev/gardener-project/releases/gardener/gardenlet:v1.91.4
• node-agent: europe-docker.pkg.dev/gardener-project/releases/gardener/node-agent:v1.91.4
• operator: europe-docker.pkg.dev/gardener-project/releases/gardener/operator:v1.91.4
• resource-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/resource-manager:v1.91.4
• scheduler: europe-docker.pkg.dev/gardener-project/releases/gardener/scheduler:v1.91.4

[gardener/gardener]

📰 Noteworthy

• [OPERATOR] Set kube-apiserver maxReplicas=3 for all Shoots that are not annotated with alpha.control-plane.scaling.shoot.gardener.cloud/scale-down-disabled=true. by @voelzmo [#9605]

✨ New Features

• [OPERATOR] gardener-operator now deploys two Prometheus replicas into the garden namespace. Read more about it here. by @rfranzke [#9543]
• [OPERATOR] A new gardenlet feature gate called ShootManagedIssuer was introduced. This feature gate guards the functionality described in GEP-24 until all of the components mentioned in the enhancement proposal are implemented by Gardener. by @dimityrmirchev [#9489]
• [OPERATOR] A new admission plugin ShootResourceReservation has been added to gardener-apiserver. It supports calculating resource reservations (memory/CPU/PID) for the kubelet.kubeReserved fields in Shoots based on the available resources of a machine type. This only applies when useGKEFormula is set to true. Otherwise, the old static values remain to be used. by @MichaelEischer [#9449]
• [OPERATOR] Support for proxy protocol is added to the istio ingress gateway to preserve the client source IP addresses. by @DockToFuture [#9526]

🐛 Bug Fixes

• [OPERATOR] Fix kube-apiserver advertise address for ipv6 local setup. by @axel7born [#9555]
• [OPERATOR] When vali is disabled in the GardenletConfiguration its fluentbit ClusterOutputs are no longer deployed. by @maboehm [#9525]
• [OPERATOR] Istio-ingress gateway dashboard now shows the correct sent tcp traffic metric and the correct memory usage. by @ScheererJ [#9596]
• [OPERATOR] A bug in gardener-node-agent which prevented copying files between different block devices has been fixed. by @oliver-goetz [#9614]
• [USER] A bug which mounted the kubelet data volume to /var/lib instead of /var/lib/kubelet when kubeletDataVolumeName was set has been fixed. by @oliver-goetz [#9614]

🏃 Others

• [OPERATOR] The vpn-seed-server now has better minimum memory settings so that less auto-scaling should occur. by @ScheererJ [#9590]
• [OPERATOR] Resource utilization metrics for the kube-apiserver container are now federated in the runtime/prometheus. by @ialidzhikov [#9581]
• [OPERATOR] K8s dashboard tests are classified as beta. by @hendrikKahl [#9567]
• [OPERATOR] Update Istio to v1.21.1 by @axel7born [#9560]

Docker Images

• admission-controller: europe-docker.pkg.dev/gardener-project/releases/gardener/admission-controller:v1.93.0
• apiserver: europe-docker.pkg.dev/gardener-project/releases/gardener/apiserver:v1.93.0
• controller-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/controller-manager:v1.93.0
• gardenlet: europe-docker.pkg.dev/gardener-project/releases/gardener/gardenlet:v1.93.0
• node-agent: europe-docker.pkg.dev/gardener-project/releases/gardener/node-agent:v1.93.0
• operator: europe-docker.pkg.dev/gardener-project/releases/gardener/operator:v1.93.0
• resource-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/resource-manager:v1.93.0
• scheduler: europe-docker.pkg.dev/gardener-project/releases/gardener/scheduler:v1.93.0

[gardener/gardener]

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed which caused gardenlet to try deploying gardener-node-agent with a wrong image tag. by @rfranzke [#9582]

Docker Images

• admission-controller: europe-docker.pkg.dev/gardener-project/releases/gardener/admission-controller:v1.92.1
• apiserver: europe-docker.pkg.dev/gardener-project/releases/gardener/apiserver:v1.92.1
• controller-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/controller-manager:v1.92.1
• gardenlet: europe-docker.pkg.dev/gardener-project/releases/gardener/gardenlet:v1.92.1
• node-agent: europe-docker.pkg.dev/gardener-project/releases/gardener/node-agent:v1.92.1
• operator: europe-docker.pkg.dev/gardener-project/releases/gardener/operator:v1.92.1
• resource-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/resource-manager:v1.92.1
• scheduler: europe-docker.pkg.dev/gardener-project/releases/gardener/scheduler:v1.92.1

[gardener/gardener]

⚠️ Breaking Changes

• [OPERATOR] The deprecated .spec.runtimeCluster.ingress.domain field has been dropped from the Garden API. Make use of .spec.runtimeCluster.ingress.domains. by @ScheererJ [#9447]
• [OPERATOR] Graduated MachineControllerManagerDeployment feature gate was removed. by @ScheererJ [#9450]
• [OPERATOR] The graduated APIServerFastRollout feature gate has been dropped. by @rfranzke [#9478]
• [OPERATOR] The graduated UseGardenerNodeAgent feature gate has been dropped. Before upgrading to this Gardener version, make sure to have up-to-date OS extensions installed on your landscape: [email protected]+, [email protected]+, [email protected]+, [email protected]+, [email protected]+, [email protected]+. by @rfranzke [#9477]
• [USER] The specification of additional, non-primary DNS providers was deprecated and will be discontinued in a future release. If you need additional DNS providers for your shoot workload, please use the provider config for the respective DNS extension. by @timuthy [#9471]
• [DEVELOPER] The deprecated oscommon package has been removed. by @rfranzke [#9477]

📰 Noteworthy

• [DEVELOPER] Admission controllers of provider extensions should be adjusted to accommodate NamespacedCloudProfiles by @benedictweis [#9440]
• [DEVELOPER] The following fields in theextensions.gardener.cloud/v1alpha1.OperatingSystemConfig API have been deprecated and will be removed in a future version: .spec.reloadConfigFilePath, .status.command, .status.units, .status.files. by @rfranzke [#9477]
• [OPERATOR] Secret openvpn-diffie-hellman-key in the garden namespace containing the Diffie-Hellmann key can be deleted from landscapes as it is no longer needed. by @ary1992 [#9386]
• [OPERATOR] resource-manager's secret reconciler that removed resources.gardener.cloud/gardener-resource-manager finalizer has been removed. by @Kostov6 [#9490]

✨ New Features

• [DEVELOPER] A new extension lifecycle strategy reconcile: AfterWorker is now available for Extensions to use in their ControllerRegistration. by @maboehm [#9472]

🐛 Bug Fixes

• [DEVELOPER] provider-local: An issue causing the backup directory not to be deleted after successful control plane migration is now fixed. by @Kostov6 [#9182]
• [DEVELOPER] An issue was fixed that caused make verify to fail because of logcheck build issues. by @timuthy [#9458]
• [OPERATOR] Restarting systemd units by annotating the node now works without specifying the .service suffix in unit names. by @oliver-goetz [#9459]

🏃 Others

• [OPERATOR] Images for local development should now be properly rebuilt, if changes are made only to files in the main packages under ./cmd/... directories. by @plkokanov [#9496]
• [OPERATOR] During the restore phase of control plane migration of HA shoots, the shoot's kube-apiserver is deployed immediately after one replica is ready for each of the events and main etcds. The event and main etcds are scaled up to 3 replicas (the current default for HA shoots) after the kube-apiserver is deployed and ready. This should greatly reduce the downtime during control plane migration of HA shoots. by @plkokanov [#9462]
• [OPERATOR] A service is added for the shoot cluster's kube-system/node-problem-detector daemonset by @istvanballok [#9483]
• [OPERATOR] Secret vpn-seed-server-dh will no longer be created as gardener/vpn2 release v0.23.0 removed the need for supplying the vpn server with Diffie–Hellman parameters. by @ary1992 [#9386]
• [OPERATOR] The Data Transfer graph from the Istio Mesh Dashboard in the seed plutono now uses rate when displaying the istio_tcp_received_bytes_total and istio_tcp_sent_bytes_total metrics. by @plkokanov [#9495]
• [OPERATOR] Correct NodeNotHealthy filtering query to use v1beta1constants.TaintNodeCriticalComponentsNotReady by @adenitiu [#9470]
• [DEVELOPER] New consistOf and contain Gomega matchers for ManagedResources were added. Tests can concisely check for expected objects a ManagedResource is responsible for. by @timuthy [#9421]

📖 Documentation

• [DEVELOPER] Add yq as a local setup prerequisite. by @marc1404 [#9510]

Docker Images

• admission-controller: europe-docker.pkg.dev/gardener-project/releases/gardener/admission-controller:v1.92.0
• apiserver: europe-docker.pkg.dev/gardener-project/releases/gardener/apiserver:v1.92.0
• controller-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/controller-manager:v1.92.0
• gardenlet: europe-docker.pkg.dev/gardener-project/releases/gardener/gardenlet:v1.92.0
• node-agent: europe-docker.pkg.dev/gardener-project/releases/gardener/node-agent:v1.92.0
• operator: europe-docker.pkg.dev/gardener-project/releases/gardener/operator:v1.92.0
• resource-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/resource-manager:v1.92.0
• scheduler: europe-docker.pkg.dev/gardener-project/releases/gardener/scheduler:v1.92.0

[gardener/gardener]

🐛 Bug Fixes

• [USER] An issue causing the Shoot status.lastMaintenance.description to contain "Removed feature gates from" or "Removed admission plugins from" messages with zero entries is now fixed. by @shafeeqes [#9539]
• [OPERATOR] A bug has been fixed which caused PersistentVolumes without .spec.nodeAffinity to become unusable in case they still had the old, deprecated topology labels. by @rfranzke [#9541]

Docker Images

• admission-controller: europe-docker.pkg.dev/gardener-project/releases/gardener/admission-controller:v1.91.3
• apiserver: europe-docker.pkg.dev/gardener-project/releases/gardener/apiserver:v1.91.3
• controller-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/controller-manager:v1.91.3
• gardenlet: europe-docker.pkg.dev/gardener-project/releases/gardener/gardenlet:v1.91.3
• node-agent: europe-docker.pkg.dev/gardener-project/releases/gardener/node-agent:v1.91.3
• operator: europe-docker.pkg.dev/gardener-project/releases/gardener/operator:v1.91.3
• resource-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/resource-manager:v1.91.3
• scheduler: europe-docker.pkg.dev/gardener-project/releases/gardener/scheduler:v1.91.3

[gardener/gardener]

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed which caused PersistentVolumes without .spec.nodeAffinity to become unusable in case they still had the old, deprecated topology labels. by @rfranzke [#9540]

Docker Images

• admission-controller: europe-docker.pkg.dev/gardener-project/releases/gardener/admission-controller:v1.90.7
• apiserver: europe-docker.pkg.dev/gardener-project/releases/gardener/apiserver:v1.90.7
• controller-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/controller-manager:v1.90.7
• gardenlet: europe-docker.pkg.dev/gardener-project/releases/gardener/gardenlet:v1.90.7
• node-agent: europe-docker.pkg.dev/gardener-project/releases/gardener/node-agent:v1.90.7
• operator: europe-docker.pkg.dev/gardener-project/releases/gardener/operator:v1.90.7
• resource-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/resource-manager:v1.90.7
• scheduler: europe-docker.pkg.dev/gardener-project/releases/gardener/scheduler:v1.90.7

[gardener/gardener]

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed which caused PersistentVolumes without .spec.nodeAffinity to become unusable in case they still had the old, deprecated topology labels. by @rfranzke [#9544]

Docker Images

• admission-controller: europe-docker.pkg.dev/gardener-project/releases/gardener/admission-controller:v1.89.4
• apiserver: europe-docker.pkg.dev/gardener-project/releases/gardener/apiserver:v1.89.4
• controller-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/controller-manager:v1.89.4
• gardenlet: europe-docker.pkg.dev/gardener-project/releases/gardener/gardenlet:v1.89.4
• node-agent: europe-docker.pkg.dev/gardener-project/releases/gardener/node-agent:v1.89.4
• operator: europe-docker.pkg.dev/gardener-project/releases/gardener/operator:v1.89.4
• resource-manager: europe-docker.pkg.dev/gardener-project/releases/gardener/resource-manager:v1.89.4
• scheduler: europe-docker.pkg.dev/gardener-project/releases/gardener/scheduler:v1.89.4