Kubespray Versions Save

Deploy a Production Ready Kubernetes Cluster

v2.24.1

2 months ago

Changes by Kind

Feature

  • Make kubernetes v1.28.6 default (#10810, @mzaian)

Bug or Regression

  • Add configuration to create cilium CNI plugin file when cilium>=1.14.0 (#10945, @cleman95 )
  • Fix logical error when checking for boostrap-os (#10953, @VannTen)
  • Make containerd 1.7.13 default Make runc 1.1.12 default Patch GHSA-xr7r-f8xq-vfvv (#10877, @VannTen)

Other (Cleanup or Flake)

  • Bump galaxy version before release (#10890, @VannTen)

The release intend to address GHSA-xr7r-f8xq-vfvv

v2.22.2

2 months ago

Changes by Kind

Network

  • [calico] Use calico_pool_blocksize from cluster when existing (#10516, @VannTen)

API Change

  • Make kubernetes 1.26.11 default (#10704, @VannTen)

Feature

  • Add hashes for kubernetes version 1.26.6, 1.26.7, 1.26.8 & 1.26.9 (#10444, @bozzo)
  • Don't let find search filesystem mounts in docker build run step (#10131, @tomodachi)
  • Make kubernetes 1.26.13 the default version (#10823, @VannTen)

Failing Test

  • Bump vagrant version 2.3.7 (#10789, @yankay)

Bug or Regression

  • Fix hardcoded pod infra version (#10805, @ErikJiang)
  • Make containerd 1.7.13 default Make runc 1.1.12 default Patch GHSA-xr7r-f8xq-vfvv (#10878, @VannTen)
  • [Multus] Fix loop_control template error when item is None (#10347, @nicolas-goudry)

The release intend to address GHSA-xr7r-f8xq-vfvv

v2.23.3

2 months ago

Changes by Kind

Feature

  • Update kubernetes default version to 1.27.10 (#10876, @VannTen)

Bug or Regression

  • Fix hardcoded pod infra version (#10806, @ErikJiang)
  • Make containerd 1.7.13 default Make runc 1.1.12 default Make kubernetes 1.27.10 default Patch GHSA-xr7r-f8xq-vfvv (#10876, @VannTen)

Other (Cleanup or Flake)

  • Update KUBESPRAY_VERSION in galaxy.yml and Readme for v2.23.2 (#10801, @yankay)

The release intend to address GHSA-xr7r-f8xq-vfvv

v2.24.0

3 months ago

Deprecation / Removal

  • Migrate node-role.kubernetes.io/master to node-role.kubernetes.io/control-plane (#10464, @unai-ttxu)
  • Drop support for Kubernetes 1.25.x (move min version to 1.26.x) (#10420, @yankay)
  • Drop installation notes for Debian Jessie (#10642, @jelmer)

Feature / Major Changes

  • Make kubernetes v1.28.6 default (#10810, @mzaian)
  • Add kubernetes v1.28.0, v1.28.1, v1.28.2, v1.28.3, v1.28.4, v1.28.5 hash (#10435, #10541, #10739, @mzaian ; #10390, @tmurakam ; #10624, @tmurakam)
  • Add Retry for Applying PriorityClass (#10469, @hangscer8)
  • Add option crio_criu_support_enabled to enable container forensic analysis (#10479, @tu1h)
  • Add option kubectl_alias to set bash alias of kubectl (#10552, @tu1h)
  • Add variable to configure ipvs modules (kube_proxy_ipvs_modules) (#10580, @borgiacis)
  • Check nameserver only when dns is enable (#10561, @yckaolalala)
  • Correctly handle remove_default_searchdomains when value is undefined (#10533, @yckaolalala)
  • Kube-scheduler: remove/update deprecated component component config v1beta3. (#10484, @mzaian)
  • Terraform-aws: variable driven ami selection (ami_name_pattern/ami_virtualization_type/ami_owners) (#10520, @mertcancam)
  • Terraform-openstack: Added possibility to enable dhcp flag critical on one interface (#10446, @Xartos)
  • This will introduce a new variable kube_apiserver_admission_plugins_podnodeselector_default_node_selector that can be used with kube_apiserver_admission_plugins_needs_configuration: [PodNodeSelector] defined. So allows the users to configure PodNodeSelector plugin. (#10607, @titansmc)
  • UpCloud: Terraform provider updated to v2.12.0. Server groups with strict anti-affinity (move var from anti_affinity_policy to anti_affinity) (#10474, @robinAwallace)
  • Update dockerfile to follow best practices (#10708, @maxime1907)
  • Update to ansible 2.15 and set minimum version to 2.15.5 (#10481, @MrFreezeex)
  • [etcd] Update Default etcd version to 3.5.10 for kubernetes 1.28, 1.27 and 1.26 (#10798, @VannTen)
  • [etcd] update version to 3.5.9 for k8s 1.28 , 1.27 , 1.26 (#10482, @mzaian)
  • [etcd] add 3.5.10 hashes (#10566, @mzaian)
  • [vsphere_csi] Update to 3.1.0 supports Kubernetes Version 1.28 (#10451, @mzaian)
  • [cinder_csi] Cinder-CSI now use cluster_name variable instead of the default hardcoded "kubernetes" value (#10422, @floryut)

Applications

  • [argocd] update argocd to v2.8.4 (#10568, @mzaian)
  • [helm] upgrade to 3.13.1 (#10567, @mzaian)
  • [coredns] Added option coredns_additional_error_config to allow for configuration of the coredns error plugin. (#10501, @Elias-elastisys)
  • [coredns] Support CoreDNS use host network & config CoreDNS port (#10617, @liuxu623)
  • [coredns] Support disable dns autoscaler when use CoreDNS (#10608, @liuxu623)
  • [coredns] Add pdb to coredns (#10557, @lobiyedKarim1)
  • [cert-manager] upgrade to v1.13.2 (#10616, @liuxu623)
  • [cert-manager] Upgrade to v1.12.6 (#10582, @chansuke)
  • [cert-manager] Upgrade to v1.12.5 (#10500, @chansuke)

Network

  • [cilium] Fix invalid hubble yaml if cilium_hubble_tls_generate is enabled (#10430, @toonalbers)
  • [cilium] Use correct ports in cilium metrics services if metrics are enabled. (#10519, @bakito)
  • [cilium] Adds support for deploying clusters with cilium 1.14+ (#10684, @rl0nergan)
  • [calico] Separate calico-node and calico-cni-plugin service accounts and update default calico to v3.26.1 (#10416, @mzaian)
  • [calico] Use calico_pool_blocksize from cluster when existing (#10516, @VannTen)
  • [calico] Update default calico to v3.26.3 (#10526, @mzaian)
  • [calico] Update default calico to v3.26.4 (#10669, @mzaian)
  • [kube-router] Default kube-router version updated to v2.0.0 (#10503, @bozzo)
  • [kube-router] Default kube-router version updated to v1.6.0 (#10478, @bozzo)
  • [kube-router] Add kube_router_bgp_graceful_restart optional setting for disabling graceful BGP restarts (default to true) (#10489, @rosskusler)
  • [metallb] Add option to set avoidBuggyIPs in IPAddressPools and change the default back to false (#10458, @zeeZ)
  • [metallb] Metallb --lb-class cmd arg to support multiple LoadBalancer implementations (#10550, @Seal1998)
  • [custom_cni] Add helm support for custom_cni deployment (#10529, @kukacz)
  • [kube_vip] Add kube_vip_lb_fwdmethod option for kube-vip (#10762, @tu1h)

Container-Managers

  • [containerd] Fix invalid version check in containerd jinja-template config (#10620, @khanhngobackend)
  • [containerd] Make containerd 1.7.11 default (#10671, @mzaian)
  • [containerd] Add hashes for containerd versions 1.7.6 ~ 1.7.8 default (#10439, #10525, #10589, @mzaian)
  • [containerd] Specify the runc path when we use the containerd container engine and change the bin_dir path. (#10154, @qlijin)
  • [containerd] Refactor NRI activation for containerd and CRI-O (remove crio_enable_nri and containerd_nri_disable) now only one var nri_enabled default to false (#10470, @fmuyassarov)
  • [containerd] Add Boolean option enable_cdi to enable cdi (false by default) (#10603, @krembu)
  • [containerd] Add configuration option for NRI (disable by default) in crio & containerd (using new containerd_nri_disable and crio_enable_nri) (#10454, @fmuyassarov)
  • [containerd] add config support override_path (#10776, @yankay)
  • [runc] Upgrade to v1.1.10 (#10671, @mzaian)
  • [crio] Update to v1.28.1 (#10480, @qlijin)
  • [crio] Remove crio package configuration during cleanup (#10584, @yckaolalala)
  • [crio] Update docs for crio_registry_auth (#10785, @qlijin)
  • [docker] Ability to define GPG key path for Docker APT (using new variable docker_repo_key_keyring) (#10513, @emiran-orange)
  • [kata-containers] Freshens configuration-qemu to latest template compatible with kata-containers 3.1.3. (#10466, @Alphadelta14)
  • [nerdctl] Bump nerdctl version 1.7.1 (#10685, @yankay)
  • [nerdctl] Change nerdctl version from 1.5.0 to 1.6.0 (#10475, @MaGaroo)

Documentation

  • Add link to Cilium CNI documentation (#10431, @toonalbers)
  • Update docs for calico_iptables_backend in Redhat/Centos.md (#10417, @yankay)
  • Update metallb example configs (#10485, @caruccio)
  • Updated AWS ALB ingress controller version (#10680, @kundan2707)

Bug or Regression

  • Add a variable reset_restart_network_service_name in the reset role to be able to configure the name of the service which is restarted. (#10428, @RomainMou)
  • Add dnsPolicy: ClusterFirstWithHostNet to DaemonSets with hostNetwork: true (#10618, @Payback159)
  • Check for correct conntrack module presence, regardless of kernel versions (#10662, @VannTen)
  • Fallback_ips: ignore unreachable hosts (#10601, @poblahblahblah)
  • Fix 'kube-apiserver' tag inappropriately overwriting secrets at rest encryption token (#10460, @jwitko)
  • Fix assertion for task item verify-settings (#10699, @piwinkler)
  • Fix external-lb in kubelet.conf server address and kube-proxy api-server address (#10490, @ugur99)
  • Fix forgotten update of etcd-servers list in apiserver manifest when scaling (#8253, @liupeng0518)
  • Fix metallb example yaml (#10545, @caruccio)
  • Fix reset job for cri-o container engine (#10197, @turbosnail)
  • Fix restart network task cannot be skipped (ansible boolean conversion needed) (#10512, @ErikJiang)
  • Fix: add kubelet tag in task of Fetch facts to avoid kubelet config inconsistencies (#10423, @NierYYDS)
  • Fixes the path of the certificates use in the etcdctl.sh wrapper when the deployment type is not kubeadm. (#10467, @RomainMou)
  • Hubble relay will work when cilium_cluster_name is customised. (#10614, @eugene-eeo)
  • Disable podCIDR allocation from control-plane when using calico (#10639, @VannTen)
  • Kubespray-defaults: Check for boostrap-os FQDN (#10590, @VannTen)
  • Patch for modprobe_nf_conntrack for new Linux Kernel, when using ipvs (#10625, @abhishekkr)
  • Remove always tag applied on bootstrap (#10556, @yckaolalala)
  • Set remove_default_searchdomains to false by default (#10554, @hedayat)
  • Swap is now disabled using systemd (mask of swap.target) (#10587, @VannTen)
  • Fix undefined retries variable when copying etcdctl (#10634, @ErikJiang)
  • Move control plane certs renewal "spread out" into the systemd timer (#10596, @VannTen)
  • The dhcp configuration for dns nameservers are now the same than during installation (#10548, @smutel)
  • Use correct env var name for kube-vip per service leader election (#10433, @ThisIsQasim)
  • Don't fail on 304 Not Modified for an already downloaded file (#10452, @sathieu)
  • Fix download retry when get_url has no status_code (#10613, @RomainMou)
  • Fix ntp installation on SLES and openSUSE (#10786, @goldyfruit)
  • Set the maxUnavailable of the coredns rolling update strategy to 1 (#10748, @tu1h)
  • Fix crio_version version comparison (#10780, @ledroide)
  • Fix disable swap failed in Centos/RHEL 7 (#10751, @yankay)
  • Fix image pull fail with insecure-registry (#10775, @yankay)
  • Refactor check_galaxy + fix version (#10729, @VannTen)
  • Fix Helm installation on SLES and openSUSE (#10794, @goldyfruit)
  • Fix incorrect ciliumcli binary (#10575, @tu1h)
  • Fix ntp installation on SLES and openSUSE (#10786, @goldyfruit)
  • Fix the cluster installation on cluster using etcd clients nodes (cilium / calico / ...) (#10769, @VannTen)

Other (Cleanup or Flake)

  • Cleanup a deprecation warning (ipaddr filter) (#10518, @VannTen)
  • Decouple kubespray-defaults from download (#10626, @VannTen)
  • Etcd/backup: use native ansible modules instead of shell (#10540, @VannTen)
  • Etcd: use dynamic group for certs generation check (#10610, @VannTen)
  • Factorize some identical playbooks steps into their own sub-playbooks (#10633, @VannTen)
  • Pre-upgrade tasks cleanup (#10656, @VannTen)
  • Refactor "multi" handlers to use listen (#10542, @VannTen)
  • Remove unneeded workaround for removing kubeadm DNS (#10695, @VannTen)
  • Removed DEPRECATED --logtostderr from metrics-server (#10709, @michaelkebe)
  • Update KUBESPRAY_VERSION for v2.23.1 (#10600, @yankay)
  • Update several checksum for different modules & configuration (#10606, @mzaian)
  • Use non-deprecated stdout_callback in CI (#10647, @VannTen)
  • Validate systemd unit files when generating them (#10597, @VannTen)
  • Using ctr pull instead of nerdctl to workaround https://github.com/kubernetes-sigs/kubespray/issues/10670. (#10687, @yankay)
  • Jinja syntax pre-commit validation (#10667, @VannTen)
  • Bump vagrant version 2.3.7 (#10787, @yankay)
  • Update KUBESPRAY_VERSION for v2.23.2 (#10800, @yankay)

Supported Components

Known issues

N/A

Notes

  1. Swap is now disabled using systemd instead of changing /etc/fstab. #10587
  2. download.yml path changed. #10626
  3. UpCloud: Terraform provider updated to v2.12.0. Server groups with strict anti-affinity (move var from anti_affinity_policy to anti_affinity) #10474

v2.23.2

3 months ago

Container-Managers

  • [containerd] Fix invalid version check in containerd jinja-template config (#10620, @khanhngobackend)

API Change

  • Make kubernetes 1.27.9 the default version (#10797, @VannTen)

Feature

  • Don't fail on 304 Not Modified for an already downloaded file (#10452, @sathieu)
  • Update kubernetes default version to 1.27.9
  • Update etcd version for 1.27 and 1.26 to 3.5.10 (#10797, @VannTen)

Failing Test

  • Bump vagrant version 2.3.7 (#10788, @yankay)

Bug or Regression

  • Fix calico-node in etcd mode. (#10768, @VannTen)
  • Fix download retry when get_url has no status_code (#10613, @RomainMou) (#10791, @VannTen)
  • Kube-controller-manager will no longer assign pod CIDRs to cluster nodes when using calico (with its default IPAM, calico_ipam_host_local now has a default value of false) [⚠️ NOTE users using a non-true value for calico_ipam_host_local will need to change it to true] (#10639, @VannTen)

Other (Cleanup or Flake)

  • Kubespray collection will have the correct collection version. (#10728, @VannTen)

v2.23.1

5 months ago

Network

  • [Cilium] Fix invalid hubble yaml if cilium_hubble_tls_generate is enabled (#10476, @toonalbers)

Feature

  • Add hashes for kubernetes 1.27.6 & 1.26.9 (#10443, @bozzo)
  • Make kubernetes v1.27.7 default (#10543, @mzaian)
  • [etcd] Default version to 3.5.9 for k8s 1.25 , 1.26 , 1.27 (#10483, @mzaian)
  • Add crictl 1.26.1 for Kubernetes v1.26 (#10562, @mzaian)
  • Change default cri-o versions for Kubernetes 1.25, 1.26 (#10563, @mzaian)
  • [ingress-nginx] Fix nginx controller leader election RBAC permissions (#10569, @mzaian)
  • Refactor NRI activation for containerd and CRI-O (remove crio_enable_nri and containerd_nri_disable) now only one var nri_enabled default to false (#10496, @fmuyassarov)

Bug or Regression

  • Fix get currently configured nameservers error where there are inline comments in /etc/resolv.conf (#10415, @yankay)
  • Migrate node-role.kubernetes.io/master to node-role.kubernetes.io/control-plane (#10532, @unai-ttxu)
  • [download] Don't fail on 304 Not Modified (#10559, @RomainMou)

v2.23.0

7 months ago

Deprecation / Removal

  • Ubuntu 16 and 18 are no longer tested (#10107, @MrFreezeex)
  • Drop support for ansible-core 2.11 and update tests dependencies (#10034, @MrFreezeex)
  • Drop Kubernetes 1.24 support (#10234, @MrFreezeex)

Feature / Major Changes

  • Make kubernetes v1.27.5 default (#10392, @mzaian)
  • Add kubernetes v1.27.4 (#10359, @mzaian)
  • Add Kubernetes 1.27.2 (#9976, @mzaian)
  • Add hashes for 1.27.3 1.26.6, 1.25.11 (#10220, @mzaian)
  • Add hashes for 1.27.4 1.26.7, 1.25.12 (#10300, @mzaian)
  • Add CPU Management Policies on the Node (#10309, @yankay)
  • Add Debian 12(bookworm) support (#10221, @tu1h)
  • Add download.timeout to update download timeout value (#10149, @yjqg6666)
  • Add corresponding coredns versions to all the supported kubernetes releases. (#10233, @mzaian)
  • Add growpart azure enabled (#10241, @pedro-peter)
  • Add ingressClass resource for ingress_nginx by default (#10091, @peschmae)
  • Add kubelet topology manager policy on the node (kubelet_topology_manager_scope and kubelet_topoloy_manager_policy) (#10370, @tu1h)
  • Add labels to kube-vip static pods (#10139, @liupeng0518)
  • Add node_taints to aws_inventory script (#10170, @mstoetzer)
  • Add option to set SSL_CERT_FILE for offline installation using custom CA for https proxy (#10215, @HappyFX)
  • Add terraform support for NIFCLOUD (#10227, @ystkfujii)
  • Add the huawei cloud controller as external cloud controller (#10198, @dabeck)
  • Show detected ansible version when it isn't compatible with kubespray (#10109, @jcpunk)
  • Allow to override etcd listen-metrics-urls configuration (using etcd_listen_metrics_urls variable) (#10332, @forselli-stratio)
  • Don't let find search filesystem mounts in docker build run step (#10131, @tomodachi)
  • Permit custom names for API server lb/proxy containers (#10166, @jcpunk)
  • Permit skipping helm update (#10169, @jcpunk)
  • Split defaults main file into 2 files (checksums and version) (#10121, @electrocucaracha)
  • System upgrade for Debian-family nodes is available with system_upgrade=true (#10184, @sathieu)
  • Update download_hash.sh script (#10120, @electrocucaracha)
  • Use a uniform way to get the local path of the binaries (#10211, @ErikJiang)
  • Disable fapolicyd service (#10081, @epif4nio)
  • Upgrade the load balancer ( nginx and haproxy ) image version to Nginx 1.25, Haproxy 2.8. (#10409, @yankay)
  • [etcd] Default version to 3.5.7 for kubernetes 1.27 (#10410, @mzaian)

Applications

  • [argocd] update argocd to v2.7.4 (#10226, @mzaian)
  • [argocd] update argocd to v2.8.0 (#10364, @mzaian)
  • [argocd] Add argocd_install_url option to allow changing argocd url (#10176, @liupeng0518)
  • [helm] upgrade to 3.12.1 (#10225, @mzaian)
  • [helm] upgrade to 3.12.3 (#10365, @mzaian)
  • [helm] add python dependency check for helm-apps (#10192, @palmeXx)
  • [krew] add krew_no_upgrade_check (#10175, @liupeng0518)
  • [coredns] Bump coredns version to 1.10.1 (#10199, @eminaktas)
  • [coredns] Bump nodelocaldns version to 1.22.20 (#10200, @eminaktas)
  • [cert-manager] This introduces a new variable for the cert-manager implementation that will allow one to pass in extra arguments to the cert-manager controller.(#10049, @phunyguy)
  • Update Helm (v3.12.2) / Skopeo (v1.13.0) and yq (v4.34.2) (#10295, @tu1h)
  • Upgrade many tool versions (Helm, crun, kata, youki, gvisor, skopeo, Calico, Cilium etc...) (#9798, @electrocucaracha)
  • [local_path_provisioner] Fix invalid podhelper yaml (#10237, @MrFreezeex)
  • Update metrics server to v0.6.4 (#10400, @mzaian)

Container-Managers

  • [containerd] Make containerd 1.7.5 default (#10397, @mzaian)
  • [containerd] Support containerd v1.7.2 (#10219, @Dentrax)
  • [containerd] Support containerd 1.7.3 (#10368, @mzaian)
  • [containerd] containerd config_path enable mirrors config using new variable containerd_registries_mirrors (deprecate and remove containerd_insecure_registries for containrd and nerdctl_extra_flags and insecure_registry setting for nerdctl (#10196, @yckaolalala)
  • [crio] Add crio_insecure_registries option for specifying insecure_registries of crio (#10142, @qlijin)
  • [crio] runroot now needs to be setup in storage.conf instead of crio.conf (#10372, @floryut)
  • [crio] Fix etcdctl copy operation (#10242, @ErikJiang)
  • [Kata] Set/keep owner/group root/root when unarchiving kata-containers (#10338, @rybnico)
  • [youki] Fix youki binary download url (not requiring 'v' in version) (#10337, @ErikJiang)

Network

  • [calico] Use configmap to configure calico cni config (#10177, @cyclinder)
  • [calico] Update calico v3.25.2 (#10414, @mzaian)
  • [calico] Add calico version to v3.26.0 (#10224, @mzaian)
  • [calico] Add calico version to v3.26.1 (#10235, @mzaian)
  • [calico] Clean up calicoctl_alternate_download_url and calicoctl.mirrors (#10271, @yckaolalala)
  • [cilium] Add custom rules to clusterrole for cilium operator (#10267, @jeremythuon)
  • [cilium] Upgrade to version 1.13.4 (#10269, @yulng)
  • [Cilium] Do not mount tls when 'cilium_hubble_tls_generate' is false (#10357, @charlychiu)
  • [Cilium] Update cilium to 1.13.3 (#10158, @jcpunk)
  • [flannel] Only create /var/lib/calico when needed (#10156, @jcpunk)
  • [flannel] Bump flannel version to v0.22.0 and flannel-cni-plugin version to v1.1.2. Also, changes flannel repository from flannelcni to flannel (#10205, @eminaktas)
  • [flannel] Remove unused flannel_cni_download_url (#10188, @oomichi)
  • [kube-ovn]: update version v1.11.5 (#10125, @yankay)
  • [multus] Fix loop_control template error when item is None (#10347, @nicolas-goudry)

API Change

  • Unless the pod security standard versions are changed on intentionally, as default it will be the same major version with Kubernetes version. (#10210, @ugur99)
  • Upgrade ansible to 7.0 and ansible-core to 2.14.x (#10190, @MrFreezeex) ⚠️ (See Notes 2)

Documentation

  • Add github container registry (github_image_repo) to docs/offline-environment.md (#10265, @blackliner)
  • Update doc for ansible-core 2.14 support and clarify issues running older python versions (#10261, @MrFreezeex)
  • Update links for aws_alb_ingress_controller (#10264, @kundan2707)
  • Update links in ingress-controller and kuberentes-apps (#10239, @vaibhav2107)
  • Update Calico to lowercase and fix broken calico link in README (#10232, @Xieql)
  • Document containerd command to restart nginx-proxy container when adding control plane node (#10406, @nicolas-goudry)

Failing Test

  • Increase metallb wait timeout from 30sec to 2min (#10260, @MrFreezeex)
  • Update CentOS 7 image and test fedora 37 and 38 instead of fedora 35 and 36 (#10108, @MrFreezeex)

Bug or Regression

  • Fix Dockerfile for newest directory layout (#10128, @dabeck)
  • Fix Flatcar bootstrap issues (yaml module missing and ntp issue) (#10363, @tenni-paws)
  • Fix argocd install not working using the kubespray docker image (#10371, @cortex3)
  • Fix correctly mount ssl ca directories (#9794, @maxime1907)
  • Fix etcdctl copy operation (#10230, @ErikJiang)
  • Fix gce-pd-csi driver (#10208, @ashishsinghdev)
  • Fix grep command without -w option causing prefix matched while adding one etcd member (#10291, @yangsenzk)
  • Fix hcloud-cloud-controller-manager not working in certain setups (#10297, @cortex3)
  • Fix helm (kubelet-csr-approver) installation on redhat distro (#10204, @MrFreezeex)
  • Fix kubelet-csr-approver usage with upgrade-cluster.yml and missing package with helm role (#10165, @j4m3s-s)
  • Fix nginxingress-class template (missing newline) (#10174, @richard-fairthorne)
  • Fix problem migration problem with k8s 1.27 (#10136, @batazor)
  • Fix reset_confirmation not working when inputing correct value (#10288, @somewho)
  • Fix wrong path in manage-offline-files script (#9886, @Medosopher)
  • Fix an issue where using Rocky Linux 8 as OS for Vagrant for testing purposes causing etcd to fail on start. (#10252, @nltimv)
  • Fix ansible-lint galaxy rule (#10277, @MrFreezeex)
  • Fix ansible-lint key-order error (#10314, @MrFreezeex)
  • Fix outdated tag and experimental ansible-lint rules (#10254, @MrFreezeex)
  • Fix dockerfile build error (#10127, @yankay)
  • Fix metrics-server deployment to run with kubernetes 1.26+ (#10183, @mzaian)
  • Fix undefined reset_confirmation_prompt variable in reset play (#10303, @Mishavint)
  • Fix CIS Kubernetes V1.23 Benchmark item number 4.1.9 to enhance security (Change kubelet-config.yaml and kubelet.env file permissions from 640 to 600) (#10304, @satandyh)
  • Fix parsing of RHSM proxy configuration (#10228, @tmurakam)
  • Fix var-spacing ansible rule (#10266, @MrFreezeex)
  • Fix specify owner to kube_owner in task of copy cni plugins (#10407, @NierYYDS)
  • Fix typo kubelet_topoloy_manager_policy => kubelet_topology_manager_policy (#10384, @hangscer8)
  • Fix recover_control_plane playbook (also add debian 12 with cilium as a new nightly test) (#10411, @floryut)
  • Fix nameserver inline comments in /etc/resolv.conf (#10415, @yankay)
  • Added systemd_resolved_disable_stub_listener variable to disable systemd-resolved's stub listener, defaults to true on Flatcar. (#9875, @cosandr)
  • Remove auto_attach and syspurpose in RHEL subscription Organization ID/Activation Key registration. (#10258, @yckaolalala)
  • Replace "crio_packages" with "crio_bin_files" (#10182, @yckaolalala)
  • Update MetalLB deployment, wait for resource. (#9995, @Jeroen0494)
  • Upgrade ansible to 7.0 and ansible-core to 2.14.x in Dockerfile (#10259, @yckaolalala)
  • Fix typo kubelet_topoloy_manager_policy => kubelet_topology_manager_policy (#10384, @hangscer8) ⚠️ (See Notes 1)
  • Change maximal_ansible_version to 2.15(exclusive) (#10395, @yankay)
  • Install etcdutl file by default (#10385, @liupeng0518)

Other (Cleanup or Flake)

  • [CI] Add CI VM for debian12 (#10222, @yankay)
  • [CI] Removes Ansible reinstall from build pipeline (#10032, @luksi1)
  • [CI] cleanup stale packet namespace automatically (#10245, @MrFreezeex)
  • [CI] fix tf-elastx_cleanup fail (#10133, @yankay)
  • [CI] Sanitize branch name in testing before using it in kubernetes label for packet-ci (#10315, @MrFreezeex)
  • Add an exception for youki in download_hash script (#10346, @ErikJiang)
  • Drop support for Kubernetes 1.24.x (move min version to 1.25.x) (#10126, @yankay)
  • Ensure host entries from /etc/host are absent when populate_inventory_to_hosts_file is false (#10144, @rptaylor)
  • Exclude terraform.tfstate backups in .gitignore (#10216, @rptaylor)
  • Ping is no longer reported as a changed task (#10160, @jcpunk)
  • Reading mounted volumes no longer considered a changed task (#10161, @jcpunk)
  • Resolve ansible-lint name errors (#10253, @MrFreezeex)
  • Update KUBESPRAY_VERSION for v2.22.1 (#10201, @yankay)

Supported Components

Known issues

N/A

Notes

  1. Variable kubelet_topoloy_manager_policy change to kubelet_topology_manager_policy, please update your inventory
  2. Upgrade ansible to 7.0 and ansible-core to 2.14.x
  3. ⚠️ breaking change : containerd config_path enable mirrors config using new variable containerd_registries_mirrors (#10196, @yckaolalala)

v2.22.1

10 months ago

Bug or Regression

  • Don't let find search filesystem mounts in docker build run step (#10131, @tomodachi)
  • Fix Dockerfile for newest directory layout (#10128, @dabeck)
  • Fix dockerfile build error (#10181, @yankay)
  • Fix metrics-server deployment to run with kubernetes 1.26+ (#10183, @mzaian)
  • update README for v2.22.0 (#10180, @Payback159)
  • Fix Update MetalLB deployment, wait for resource. (#9995, @Jeroen0494)

v2.22.0

11 months ago

Deprecation / Removal

  • [Cilium] Delete the probe option of cilium_kube_proxy_replacement (#9929, @XiuguangHuang)
  • [Cilium] Remove use_localhost_as_kubeapi_loadbalancer and detect wether we can use localhost apiserver loadbalancer if cilium/calico replace kube-proxy (#9718, @MrFreezeex)
  • Drop crun_bin_dir unused variable, now using only bin_dir var (#9845, @electrocucaracha)
  • Drop the canal network_plugin support because the network_plugin is unmaintained. (#10100, @oomichi)
  • Remove the support of Debian 9 (#10097, @yankay)
  • Replaces storage.googleapis.com/kubernetes-release with dl.k8s.io (#10066, @KlwntSingh)

Feature / Major Changes

  • Add Kubernetes 1.26.x (#9570, @mzaian ; #9732, @yankay; #9829, @mzaian; #9900, @mzaian)
  • Make kubernetes v1.26.5 default (#9983, @mzaian)
  • "native" snapshotter of nerdctl config is replaced by new var nerdctl_snapshotter with default "overlayfs" value (#9979, @dmitrytretyakov)
  • Support multi-arch using the same image name (#9978, @ErikJiang)
  • Add DNS configuration for cert-manager (using new variables cert_manager_dns_policy|config) (#9673, @ErikJiang)
  • Add Retry for restart kube-controller-manager (#10013, @hangscer8)
  • Add coredns_additional_configuration variable to define extra Coredns configurations (#10025, @navidnabavi)
  • Add coredns_rewrite_block to perform internal message rewriting (#10045, @maxime1907)
  • Add a new simple network_plugins custom_cni to install user provided manifests (#9819, @MrFreezeex)
  • Add back openssh-client to docker image (#9835, @maxime1907)
  • Add download retries option download_retries (#9911, @tu1h)
  • Add support to install ContainerD on any Linux Distributions using new var allow_unsupported_distribution_setup (#9827, @XDRAGON2002)
  • Add the kube-profile config to the kubeadm's kube-scheduler config. (#9993, @yankay)
  • Add vim to kubespray docker image (#9805, @XDRAGON2002)
  • Adds support for Kubelet-CSR-approver to auto-approve kubelet CSR when kubelet_rotate_server_certificates. (#9877, @j4m3s-s)
  • Add dns_cpu_limit value to support large scaled coredns deployments (#10103, @mzaian)
  • Add provider meta module_name in Equinix Metal TF configs (#10044, @vasubabu)
  • Allow to configure image garbage collection (using kubelet_image_gc_high_threshold and kubelet_image_gc_low_threshold) (#9832, @zhan9san)
  • Apply kubeadm patches during upgrade as recommended by k8s (#9781, @mvandergiesen)
  • Cinder-csi: Allow VolumeSnapshotClass' deletionPolicy to be configurable (#9736, @huangkevin404)
  • Containerd add containerd_use_config_path config field. (#9770, @lengrongfu)
  • Enable control plane load balancing for kube-vip (#9785, @ErikJiang)
  • Feat(contrib/terraform): support custom ssh port (#9836, @maxime1907)
  • Fix kube-bench 1.2.20 to enhance security (Ensure that the --audit-log-maxbackup argument is set to 10) (#9939, @yankay)
  • Fix kube-bench 1.1.19 to enhance security (Change Kubernetes Cert directory and file ownership is set to root:root) (#9937, @yankay)
  • Fix kube-bench 4.1.1 to enhance security (Change kubelet systemd init file from 644 to 600) (#9934, @yankay)
  • Fix kubernetes-app/argocd: download related things with the download role (#9786, @pli01)
  • Kube.py now supports kubeconfig (#9982, @liupeng0518)
  • MetricsServer: Add extras nodeselector, affinity, tolerations (using metrics_server_nodeselector, metrics_server_extra_affinity ,metrics_server_extra_tolerations) (#9972, @pli01)
  • Refactor Hetzner terraform (fixing flatcar configs and remove deprecated provider) (#10002, @ThisIsQasim)
  • Support for MetalLB v0.13.9 with CRD (#9120, @Jeroen0494)
  • Throw an error when specifying unsupported os in Vagrant (#9965, @THUzxj)
  • Update CoreDNS manifests (remove deprecated annotations) (#9977, @mzaian)
  • Update dns-autoscaler configuration and remove deprecated annotations (#9996, @mzaian)
  • Update metrics server to v0.6.3 (#10026, @mzaian)
  • Upgrade argocd to v2.6.3 (#9848, @panguicai008)
  • Upgrades the following Python libraries to their latest available releases (cryptography / jinja2 / jmespath / MarkupSafe/ netaddr / pbr / ruamel.yaml / ruamel.yaml.clib) (#9938, @luksi1)
  • Add IPv6 listen directive to haproxy if enable_dual_stack_networks (#9674, @yankay)
  • Add support for Ansible collections in Kubespray (⚠️ See notes !) (#9582, @luksi1)
  • Support mTLS for Hubble and upgrade backend to v0.11.0 (#9959, @jeremythuon)
  • Update nodelocaldns to 1.22.18 (#9800, @sathieu)
  • Replace disable_swap variable with kubelet_fail_swap_on (#10036, @Manuelraa)
  • Replace nodelocaldns label to k8s-app: node-local-dns (#9745, @stelucz)
  • Upgrade rancher local-path-provisioner to v0.0.23 (#9855, @panguicai008)
  • Use kube_apiserver_address variable for advertiseAddress (#9967, @liupeng0518)
  • Use string for ipv6 forward conf value (#9992, @liupeng0518)
  • Update pause image version to v3.9 (#10112, @mzaian)
  • Upgrade cni version to v1.3.0 (#10058, @cyclinder)
  • [argocd] update argocd to v2.6.7 (#9953, @mzaian)
  • [helm] support to 3.11.1 (#9849, @mzaian)
  • [helm] support to 3.11.3 (#10022, @mzaian)
  • [helm] support to 3.11.2 (#9951, @mzaian)
  • [helm] upgrade to 3.12.0 (#10085, @mzaian)
  • [UpCloud] Add server group support for vms and target port for loadbalancers (#9831, @robinAwallace)
  • [argocd] update argocd to v2.5.10 (#9753, @yanggangtony)
  • [cert-manager] Upgrade to v1.11.1 (#9964, @rtsp)
  • [flannel] update to v0.21.4 (#10027, @mzaian)
  • [nerdctl] support version 1.3.1 (#10024, @mzaian)
  • [nerdctl] update to version 1.4.0 (#10119, @mzaian)

Applications

  • [kube-vip] Support to v0.5.8 (#9734, @hangscer8)
  • [kube-vip] Support kube-vip to v0.5.11 (#9852, @panguicai008)
  • [kube-vip] Update default kube-vip to v0.5.12 (#10005, @hangscer8)
  • [vSphere-csi] Add resources section to all containers releated to Vsphere CSI driver (#9687, @JRaver)
  • [argocd] update argocd to v2.7.2 (#10086, @mzaian)

Container-Managers

  • [containerd] Add hashes for containerd version 1.6.19 (#9838, @mzaian)
  • [containerd] Add hashes for containerd version 1.6.20 (#9954, @mzaian)
  • [containerd] Add hashes for containerd version 1.7.0 (#9892, @mzaian)
  • [containerd] Add hashes for containerd versions 1.7.1, 1.6.21 (#10061, @mzaian)
  • [containerd] Support version 1.6.16 (#9727, @yanggangtony)
  • [cri-o] Bump versions to 1.26.3, 1.25.3, 1.24.5 (#9999, @dkasanic)
  • [cri-o] Fix install order -> first runc then crictl (#9780, @mvandergiesen)
  • [cri-o] Fix missed double quotes in cri-o config (#10040, @turbosnail)
  • [cri-o] Fix CRI-O amd64 v1.26.0 wrong archive checksum (#9872, @panguicai008)
  • [cri-o] cri-o restart if config change (#10057, @MrFreezeex)
  • [cri-o] Remove deprecated crio_pids_limit (default is now unlimited) (#10056, @j4m3s-s)
  • [cri-o] Fix cri-o restart if config change (#10057, @MrFreezeex)
  • [runc] Upgrade to v1.1.7 (#10039, @pomland-94)

Network

  • [Calico] Add Retry and Ignore Error for Checking calico ready (#9883, @hangscer8)
  • [Calico] Add option calico_kubeconfig_wait_timeout (#9994, @tu1h)
  • [Calico] Improve version check command (#9861, @zhan9san)
  • [Calico] Optimize the detection of calico existence (#9873, @hangscer8)
  • [Calico] Support calico version v3.25.0 (#9860, @cyclinder)
  • [Calico] upgrade default calico version to v3.25.1 (#9950, @mzaian)
  • [Calico] Add missing ipamconfigs resource in RBAC (#9755, @chaunceyjiang)
  • [Calico] Fix installation while applying CRD (#10068, @hangscer8)
  • [Calico] Add calico version to v3.24.6 (#10113, @mzaian)
  • [Cilium] Add and support v1.13.0 (#9879, @utam0k)
  • [Cilium] Fix Hubble relay configuration (#9876, @prashantchitta)
  • [Cilium] Fix the configuration of TLS for hubble (#9880, @utam0k)
  • [Cilium] Remove duplicates in the configuration of tls for hubble (#9932, @CaMoPeZzz)
  • [Cilium] Support version above 1.13.x (#9914, @wbh1)
  • [Cilium] Updates hubble certgen arguments (wrong since v0.1.7) (#9856, @XDRAGON2002)
  • [Cilium] IPAM uses "Cluster Scope" mode by default. Also add the parameters required for this mode (#9443, @dcwbq)
  • [flannel] Update image repo from flannelcni to flannel (#10041, @ErikJiang)
  • [multus] fix multus include error (#10105, @darkobas2)

API Change

  • Openstack cloud controller manager bind address is now configurable using external_openstack_cloud_controller_bind_address (#9958, @dominykasn)

Documentation

  • Add a mention for custom_cni in CNI list (#9878, @j4m3s-s)
  • ArgoCD no longer uses the pod name as initial password (#9930, @peschmae)
  • Drop remaining part for supporting ansible 2.9 and 2.10 (#9842, @oomichi)
  • Fix sidebar documentation (#9988, @lijin-union)
  • Fixup link in docs/calico.md (#9940, @kundan2707)
  • Remove stale contents for cni documention (#9778, @tu1h)
  • Reword confusing etcd download url comment when etcd_deployment=host (#9686, @tjanson)
  • Suggest to run reset.yml playbook for first-time users (#9865, @kerryeon)
  • Update docker tag to v2.21.0 in README.md (#9802, @Payback159)
  • Update link for baremetel consideration (#9944, @kundan2707)
  • Add port requirements documentation (#9969, @yankay)

Failing Test

  • Update Terraform to 1.3.7 and Vagrant to 2.3.4 (#9699, @floryut)
  • [CI] Migrate CI_BUILD_ID to CI_JOB_ID and CI_BUILD_REF to CI_COMMIT_SHA following gitlab upgrade (#10063, @floryut)

Bug or Regression

  • Add PSS labels to metallb namespace (#9713, @manzsolutions-lpr)
  • Add jmespath back to Dockerfile image (#9697, @floryut)
  • Add missing krew_download_url to offline.yml (#9788, @jianse)
  • Add proxy_env variable to apt_key cleanup task (#9766, @SamuelBECK1)
  • Add rsync in Dockerfile (#9839, @zhan9san)
  • Add ruamel.yaml back to Dockerfile image (#9707, @floryut)
  • Cleanup MetalLB install following update (#10004, @eugene-marchanka)
  • Copy contrib/ to Dockerfile (#9774, @oomichi)
  • Downgrade the version of CoreDNS to 1.8.6 for compatibility with Kubernetes versions older than 1.25. (#9846, @JiffsMaverick)
  • Explicitly disable rhsm repo when rhel_enable_repos is false (#9973, @tu1h)
  • Fix cert_manager_trusted_internal_ca manifest failing when dns policy is set (#9922, @peschmae)
  • Fix containerd_insecure_registries => move with_item to with_dict (#9729, @lengrongfu)
  • Fix allow unsupported distribution (#9904, @ErikJiang)
  • Fix cilium's hubble ui configuration (#9735, @j4m3s-s)
  • Fix comma-separated-list splitting of kubelet_enforce_node_allocatable variable (#9694, @Tristan971)
  • Fix confusing instance sizing (etcd, kube_master) in Vagrantfile (#9966, @THUzxj)
  • Fix ingress url not found issue (#9789, @JaneLiuL)
  • Fix playbook names to support import via galaxy (#10021, @dkasanic)
  • Fix restart k8s components, checking yml files instead of manifest (#9962, @liupeng0518)
  • Fix uniontech OS installation failure (#9862, @ErikJiang)
  • Fixing default cgroups for kubelet and container_manager (#9834, @MrFreezeex)
  • Localhost task (validate mirror) don't need to ask for become (#9669, @chok)
  • Remove unneeded access_ip when not wanted in terraform scripts (#9869, @maxime1907)
  • Replace semicolons by commas in networkmanager dns configuration options (#9840, @lystor)
  • Retry other masters during upgrade and not only the first one (#9768, @maxime1907)
  • Skip steps of ensuring NTP and tzdata packages in the CoreOS and Flatcar (#9742, @ErthoAers)
  • Support extended settings for the Debian os family (#9943, @ErikJiang)
  • Fix calico rbac issue (#9806, @JaneLiuL)
  • Update nodes in etc hosts after cluster scale (#9837, @zhan9san)
  • Update rhsm repo trigger if no subscriptions is found (#10001, @tu1h)
  • Bootstrap ansible requirement in the facts playbook (#10069, @MrFreezeex)
  • Clear http scheme on containerd insecure-registry tls config (#10084, @tu1h)
  • Ignore errors in check mode performing "Disable swapOnZram for Fedora" (#10077, @gorozhin)
  • [etcd] fix make-ssl-etcd.sh.j2; move pem files only if any new certs exist (#9974, @2k0ri)
  • [vSphere-csi-driver] Fixes the run of the cluster.yml playbook when vsphere_csi_namespace is set to non-default (#9946, @eugene-marchanka)

Other (Cleanup or Flake)

  • Add checksum verification for kubectl binary in dockerfile (#9963, @alekseyolg)
  • Add generic pre-commit hook to the repository (#9750, @bbaassssiiee)
  • Cleanup of external-openstack-cloud-config to be in the same order/values as the documentation and not clutter config when defaults are used. (#9899, @jadams)
  • Cleanup v1.23.x references/conditions/hashes (#9698, @floryut)
  • Dockerfile update ubuntu version to 22.04 which has newer system packages with fewer (#10033, @alekseyolg)
  • Drop support for Kubernetes 1.23.x (move min version to 1.24.x) (#9691, @floryut)
  • Fix(contrib/terraform): do not set ansible_ssh_port to 22 (#9828, @maxime1907)
  • Move multus url to k8snetworkplumbingwg repository (#9850, @panguicai008)
  • New automated method to collect binaries checksums (#9782, @electrocucaracha)
  • Reducing the number of layers and commands for docker image (#9822, @alekseyolg)
  • Remove deprecated udpIdleTimeout field in KubeProxyConfiguration (#9925, @HirazawaUi)
  • Remove invalid character in crictl tasks file (#9970, @tu1h)
  • Replace bash for loop when checking API server SANs (#9060, @rptaylor)
  • Use var etcd_deployment_type instead of etcd_kubeadm_enabled (#9823, @liupeng0518)
  • Reducing the number of layers, increasing readability, reducing the size of the image (#9821, @alekseyolg)
  • Fix arithmetic outside of jinja (#10106, @MrFreezeex)
  • Fix CI broken by flannel-cni-plugin docker hub rate limit (#10083, @yankay)
  • [CI] Add CI for containerd insecure_registries (#9797, @yankay)
  • [CI] Updated version of ara included in CI job logs collection from 1.5.7 to 1.6.1 (#9737, @dmsimard)
  • [CI] Add checksum verification of kubectl binary in pipeline image (#9971, @alekseyolg)
  • [CI] Fix CentOS Extras repo url for Oracle Linux 7 aarch64 (#9791, @bin456789)
  • [CI] Use Docker buildkit + caching for builds to speed up the CI pipeline (#10008, @luksi1)
  • [CI] Add six module into openstack-cleanup/requirements.txt (#10099, @oomichi)
  • [CI] Fix tests for files lookup path for custom-cni (#10088, @j4m3s-s)

Supported Components

Known issues

N/A

Notes

  • Support for MetalLB v0.13.9 with CRD (⚠️ This release includes user facing changes for which there is action required. The way the inventory is setup for MetalLB deployment has changed significantly. Most prominently, we have switched from underscores to a dictionary for defining resources. Please follow the documentation for restructuring your MetalLB inventory variables.
  • Replace disable_swap variable with kubelet_fail_swap_on
  • Fix playbook names to support import via galaxy (⚠️ ADD NOTE : recover-control-panel => recover_control_plane, remove-node => remove_node, upgrade-cluster => upgrade_cluster)
  • [Cilium] IPAM uses "Cluster Scope" mode by default.
  • Add support for Ansible collections in Kubespray (This would cause a change to the repository's structure, meaning downstream users would either need to change their code to point to the playbooks directory or use the ansible.builtin.import_playbook module)

v2.21.0

1 year ago

Deprecation / Removal

  • Drop calico v3.21 support (#9515, @oomichi)

Feature / Major Changes

  • Add Check resolv.conf is empty to avoid CoreDNS crash (#9502, @yankay)
  • Add XDG related Helm paths to be removed from reset tasks (#9561, @emiran-orange)
  • Add a parameter (disable_host_nameservers) to disable host nameservers (#9357, @eminaktas)
  • Add an option (populate_loadbalancer_apiserver_to_hosts_file) to skip adding load balancer name in the hosts file (#9331, @JRaver)
  • Add custom options to coredns kubernets plugin (coredns_kubernetes_extra_opts ) (#9608, @mvandergiesen)
  • Add docker support for openEuler linux (#9498, @ErikJiang)
  • Add support for the OpenEuler Linux (#9494, @ErikJiang)
  • Add terraform script for Flatcar Linux on Hetzner (#9618, @florianow)
  • Add the ability to define options for DNS upstream servers (using new variable dns_upstream_forward_extra_opts) (#9311, @emiran-orange)
  • Add var (ingress_nginx_probe_initial_delay_seconds) for control initialDelaySeconds in ingress-nginx probes (#9405, @zvlb)
  • Add variable condition snapshot in vSphere CSI (vsphere_csi_block_volume_snapshot) (#9429, @yanggangtony)
  • Add variable in metrics_server deployment (metrics_server_replicas) to enable HA mode (#9539, @ugur99)
  • Change dns upstream condition for nodelocaldns when using host_resolvconf (#9378, @unai-ttxu)
  • Download coredns image to all hosts in k8s_cluster (#9316, @joes)
  • Enable check mode in DNS Cleanup tasks (#9472, @emiran-orange)
  • Etcd image has the same tag accross multiple archs (#9516, @hangscer8)
  • Fix a pre-upgrade node drain rescue task failure when kube_override_hostname is set (#9556, @chadswen)
  • Fix default value for kubelet_secure_addresses (#9355, @willtrnr)
  • Provides <kubeadm_init_timeout> to change the timeout of first control-plane initialization (#9617, @tu1h)
  • Remove PodSecurityPolicies in MetalLB for kubernetes 1.25 (#9442, @yanggangtony)
  • Support Python 3.11 - ruamel.yaml.clib need to be updated to 0.2.7 (#9426, @olivierlemasle)
  • Support customize the additional sysctl variables using additional_sysctl (#9351, @yankay)
  • Support patches field in kubeadm v1beta3 in both InitConfiguration and JoinConfiguration (using new variable kubeadm_patches) (#9326, @titaneric)
  • Switch helm install (from synchronize to copy) to support password authentication (#9343, @ghostloda)
  • Update api version for pdb and batch (deprecated in 1.25) (#9369, @yankay)
  • Update dashboard image repo to remove arch flag (#9530, @tu1h)
  • Update etcd log-level parameter name (new name: ETCD_LOG_LEVEL) (#9540, @ErikJiang)
  • Update local-volume-provisioner to 2.5.0 + add documentation (#9463, @olivierlemasle)
  • Update the number of nofile limits in containerd to 65535 (#9507, @ErikJiang)
  • Upgrade metrics server to v0.6.2 (#9554, @mzaian)
  • Upgrade the load balancer ( nginx and haproxy ) image version. (#9506, @yankay)
  • Use kube_apiserver_port variable instead of hard-coding 6443 (#9620, @huangkevin404)
  • [etcd] Default version to 3.5.5 for k8s 1.25.x (#9419, @mzaian)
  • Update CoreDNS version to v1.9.3 (#9503, @yankay)
  • Add the possibility to specify extra domains for the coredns kubernets plugin (using coredns_kubernetes_extra_domains) (#9635, @mvandergiesen)
  • Streamline ansible_default_ipv4 gathering loop (#9281, @rptaylor)
  • Update kubernetes dashboard to 2.7.0 (k8s 1.25 support) (#9425, @mzaian)
  • Skip retry operation with containerd when etcd installed on host VM (#9560, @JRaver)
  • Update pause image version to v3.8 (#9668, @mzaian)
  • Enable kubelet_authorization_mode_webhook back by default and remove extra role (#9662, @MrFreezeex)
  • Terraform gcp can now have extra ingress firewall rules, using new variable extra_ingress_firewalls (#9658, @sathieu)
  • kubeadm/etcd: use config to download certificate (#9609, @MrFreezeex)

Applications

  • [argocd] update argocd to v2.5.5 (#9604, @mzaian)
  • Upcloud: Reclaim policy for PV is now delete (#9574, @robinAwallace)
  • [Exoscale] Add missing zone input variable (#9495, @ayoubeddafali)
  • [MetalLB] Avoid MetalLB speaker image download when MetalLB speaker is disabled (#9248, @unai-ttxu)
  • [Openstack] Replace deprecated "template" Terraform provider with supported "cloudinit" Terraform provider (#9536, @inflatador)
  • [OpenStack] Updated openstack cloud controller to version v1.25.3 (#9500, @robinAwallace)
  • [Openstack] Add bastion_allowed_ports to allow custom security group rules on bastion node (#9336, @bl0m1)
  • [Openstack] Upgrade 1.22.0 to 1.23.4 (#9332, @QcFe) (See Notes 1)
  • [Openstack] Added override variable, additional server groups and cloudinit config (#9452, @Xartos)
  • [cinder-csi-nodeplugin] Remove the pods-cloud-data volume (delete upstream) (#9362, @huangkevin404)
  • [vsphere-csi] Add missing defaults for external_vsphere_* variables in the csi_driver/vsphere role (#9664, @rlacko58)
  • [hetzner] In config, rename ansible groups to use _ instead of - (#9569, @ym)
  • [kube-vip] Minor changes on Kube VIP configuration parameters (and fix wrong properties) (#9414, @woutergd)
  • [cert-manager] Upgrade to v1.10.1 (#9512, @rtsp) then v1.11.0 (#9661, @mzaian)
  • [helm] upgrade to 3.10.3 (#9605, @mzaian)
  • [ingress-nginx] upgrade to 1.5.1 (#9532, @mzaian)
  • [vSphere] Removing unneeded terraform dependencie & mark vsphere_password as sensitive (#9672, @sathieu)

Container-Managers

  • Optimize cgroups settings for node reserved (using new kube_reserved, see docs for more information) (#9209, @shelmingsong)
  • [Docker] Update docker package to 20.10.20 (partial fix for CVE-2022-39253) (#9410, @floryut)
  • [containerd] Add support for 1.6.11 (#9544, @yanggangtony)
  • [containerd] Added variables for unpriviledged ports and icmp (#9517, @Xartos)
  • [containerd] Allow containerd-common to execute multiple times per play (#9543, @chadswen)
  • [containerd] Newly started containers will be limited to 16384 open files. To change this number, set containerd_base_runtime_spec_rlimit_nofile, or remove base_runtime_spec from runc runtime to revert to previous behaviour. (#9319, @fungusakafungus)
  • [containerd] Support v1.6.13 and v1.6.14 (#9585, @yanggangtony)
  • [containerd] Add config_path var in config.toml.j2 file (#9566, @lengrongfu)
  • [containerd] Add hashes for containerd versions 1.5.14 , 1.5.15 , 1.5.16 (#9678, @yanggangtony)
  • [cri-o] Use cri-o from upstream instead of kubic/OBS (#9374, @cristicalin)
  • [nerdctl] upgrade to version 1.0.0 (#9424, @mzaian)

Network

  • Bump cni-plugins version to v1.2.0 (#9671, @cyclinder)
  • Fix remove Cilium CNI failed because the CNI bin dependency (#9563, @yankay)
  • [Calico] Add cni bin when installing (#9367, @ErikJiang)
  • [Calico] Add retry for start calico kube controller (#9450, @cleverhu)
  • [Calico] Adjust calico-kube-controller pod to non hostNetwork pod (#9465, @cyclinder)
  • [Calico] Adjust calico-kube-controller pod to use hostnetwork if using etcd (#9573, @JSpon)
  • [Calico] Disable 'Check that IP range is enough for the nodes' (#9491, @mzaian)
  • [Calico] Update the tag image to support multiple architectures with the same tag (#9529, @ErikJiang)
  • [Calico] remove deprecated PodSecurityPolicy (removed in Kubernetes in v1.25) (#9395, @yankay)
  • |Calico] Allow user to set env: FELIX_MTUIFACEPATTERN in calico-node.yml (using calico_felix_mtu_iface_pattern) (#9330, @shelmingsong)
  • [Calico] Replace node-role.kubernetes.io/master with control-plane (#9627, @my-git9)
  • [Calico] upgrade default calico version to v3.24.5 (#9580, @yankay)
  • [Calico] Add vxlan-v6.calico to the list of NetworkManager unmanaged interfaces (#9631, @cyclinder)
  • [Calico] Add retry to avoid 'unknown' state for calicoctl (#9633, @tu1h)
  • [Calico] Update Calico VXLAN offload docs because Calico changed the default value (#9639, @yankay)
  • [Calico] Add possibility to enable calico floatingIPs feature (using calico_felix_floating_ips) (#9680, @MatthieuFin)
  • [Cilium] Add download configuration for cilium hubble images (using cilium_enable_hubble variable) (#9376, @ErikJiang)
  • [Cilium] Add switch cilium_enable_bandwidth_manager (#9441, @dcwbq)
  • [Cilium] Cleanup cilium-init image from cilium template (#9508, @ErikJiang)
  • [Cilium] update cilium cli offline download url example (#9458, @cleverhu)
  • [Cilium] Install Cilium CLI alongside Cilium (#9436, @dcwbq)
  • [flannel] Initcontainer image now correctly support architecture suffix (#9461, @rollandf)
  • [flannel] Upgrade version to v0.20.1 (#9528, @ErikJiang)
  • [flannel] remove deprecated PodSecurityPolicy (removed in Kubernetes in v1.25) (#9365, @yankay)
  • [flannel] Add wireguard encryption backend as option (#9583, @janaurka)
  • [flannel] Support dual stack IPv4 & IPv6 networking (#9564, @styshoo)
  • [flannel] Allow setting the DirectRouting option on VXLAN (#9438, @willtrnr)
  • [flannel] update to v0.20.2 & make it default (#9675, @mzaian)
  • [kube-ovn] Update version to v1.10.7 (#9527, @liupeng0518)
  • [kube-ovn] Remove kube-ovn log directories when reseting (#9625, @JochenFriedrich)
  • [kube-ovn] Remove ovn.kubernetes.io/ovs_dp_type from nodeSelector (#9594, @JochenFriedrich)
  • [kube-ovn] Support OVN Interconnect (#9599, @JochenFriedrich)
  • [multus] added support for mixed type of container engine (#9224, @mr-yaky)

Bug or Regression

  • Change include to import_playbook in recover_control_plane playbook, to support ansible 2.12+ (#9576, @floryut)
  • Corrected vsphere directory in docs (#9534, @wojciehm)
  • Deleting worker nodes is now skipped if there is no kube_control_plane node. (#9430, @kerryeon)
  • Etcd arch can now support arm64 and amd64 (#9421, @yanggangtony)
  • Fix cert-manager deployment on hardening environments (#9404, @oomichi)
  • Fix checksum of ciliumcli v0.12.5 for arm64 (#9614, @oomichi)
  • Fix inconsistent handling of admission plugin list (kube_apiserver_enable_admission_plugins must be specified as a list of individual plugin names instead of a single item comma-separated list) (#9407, @willtrnr)
  • Fix kube token dir permissions (#9590, @C-Romeo)
  • Fix missing control plane taint in kubeadm (#9592, @yankay)
  • Fix regex for comments nameserver in resolv.conf (#9523, @yankay)
  • Fix reset for RedHat based distro with major version >=8 (#9537, @dougsland)
  • Fix wrong cri_socket path for containerd (#9401, @maxime1907)
  • Fix wrong rbac of the ClusterRole csi-snapshotter-role (#9610, @maxime1907)
  • Remove coredns_server from supersede_nameserver in dhclient.conf if nodelocaldns is enabled. (#9392, @JiffsMaverick)
  • Remove immutable flag from /var/lib/kubelet subdirs (#9597, @emiran-orange)
  • Skip the install of ping package in Fedora CoreOS & Flatcar (#9370, @yankay)
  • Fix OL9 setup - disable Centos Extras repo creation (#9483, @psvmcc)
  • Use hostname override in post-remove role, just as pre-remove role does (#9360, @JSpon)
  • [Calico] Install calico-kube-controller also when using kdd datastore (#9358, @wayfrro)
  • [Cilium] Fix the Hubble certificate being faulty because the cluster name has an hard coded value (#9340, @dcwbq)
  • [Cilium] Fix tls settings not being properly set (#9457, @charlychiu)
  • [Cilium] Remove trailing backslash and fix yaml indent (#9339, @reneluria)
  • [Openstack] Fix a race condition in terraform causing ports to not get an IP (#9345, @bl0m1)
  • [Openstack] Fix missing permissions for Openstack cloud-controller-manager (#9335, @bl0m1)
  • [gVisor] Allow installation on arm architecture systems (#9493, @ErikJiang)
  • [kube-ovn] Cluster support for ovn-central (#9596, @JochenFriedrich)
  • [upcloud] Fixed issue where DNS would be blocked while using allowlist (#9510, @Xartos)

Other (Cleanup or Flake)

  • Use the correct api version and resource type in secrets_encryption.yaml.j2 (#9575, @LukasNajman)
  • Minor cleanup of docs by rephrasing some unclear documentation (#9621, @anthonyeleven)
  • Add mirror doc to support mirror usage. (#9396, @yankay)
  • [CI] Add check_typo job (and fix a bunch of typos) (#9361, @oomichi)
  • [CI] Stop using python 'test' internal package (#9454, @olivierlemasle)
  • [CI] Update securityContext of netchecker (#9398, @oomichi)
  • [CI] Use agnhost instead of busybox for network test (#9390, @oomichi)
  • [CI] Add ubuntu20 hardening job (#9359, @oomichi)
  • [CI] Fix YAML format in hardening.md file (#9387, @oomichi)
  • [CI] Make vagrant-ubuntu20-flannel voting (by removing allow failure) (#9469, @oomichi)
  • [CI] Update sonobuoy version to a more recent one (#9485, @oomichi)
  • [CI] Increase the fedora memory at CI to fix the CI broken (#9640, @yankay)
  • [CI] Add CI for rockylinux9 and cilium (#9562, @yankay)

Component versions

  • Core
    • kubernetes v1.25.6
    • etcd v3.5.6
    • docker v20.10 (cri_dockerd: v0.3.0)
    • containerd v1.6.15
    • cri-o v1.24
  • Network Plugin
    • cni-plugins v1.2.0
    • calico v3.24.5
    • cilium v1.12.1
    • flannel v0.20.2
    • kube-ovn v1.10.7
    • kube-router v1.5.1
    • multus v3.8
    • weave v2.8.1
    • kube-vip v0.5.5
  • Application
    • cert-manager v1.11.0
    • coredns v1.9.3
    • ingress-nginx v1.5.1
    • krew v0.4.3
    • argocd v2.5.7
    • helm v3.10.3
    • metallb v0.12.1
    • registry v2.8.1
  • Storage Plugin
    • cephfs-provisioner v2.1.0-k8s1.11
    • rbd-provisioner v2.1.1-k8s1.11
    • aws-ebs-csi-plugin v0.5.0
    • azure-csi-plugin v1.10.0
    • cinder-csi-plugin v1.22.0
    • gcp-pd-csi-plugin v1.4.0
    • local-path-provisioner v0.0.22
    • local-volume-provisioner v2.5.0

Known issues

N/A

Notes

  1. As stated in cloud-provider-openstack:1.23.0: Load balancers don't relate to a dedicated Service anymore, any scripts relying on that relationship previously need to change to use the load balancer tags instead