Deploy a Production Ready Kubernetes Cluster
tar in common required package (#9184, @yankay)kubelet_systemd_hardening: [true|false] (#9194, @alegrey91)SeccompDefault admission plugin for kubelet (using new variable kubelet_seccomp_default) (#9074, @alegrey91)delete_node_retries,delete_node_delay_seconds) to tweak remove node process (#9096, @ydFu)metallb_avoid_buggy_ips for default IP address pool and avoid_buggy_ips for additional IP address pools defined in metallb_additional_address_pools) (#9166, @kerryeon) (see Notes 2)kube_owner/kube_cert_group/etcd_owner variables) (#8952, @alegrey91)vsphere_csi_namespace) (#9278, @MahdiAbbasi95)nat_outgoing) would not be disabled automatically when enabling peer_with_router. (#9255, @kerryeon)calcio_rr_id Is renamed to calico_rr_id for fixing a typo ⚠️ (#9327, @kerryeon)cilium_rolling_restart_wait_retries_{count, delay_seconds} (#9176, @Tristan971)cilium_ip_masq_agent_enable variable no longer exists. Use enable-ipv4-masquerade and enable-ipv4-masquerade to enable masquerade. (#9225, @necatican)kube_proxy_strict_arp is set to false in arp mode (#9223, @yankay)matallb_auto_assign variable to metallb_auto_assign (users disabling 'auto-assign' in metallb must update the variable name) (#8949, @orange-llajeanne)vsphere_csi_node_affinity variable (#9293, @dmitrytretyakov)containerd_limit_[proc_num/core/open_file_num/mem_lock) (#9269, @ErikJiang)containerd_default_runtime is now undifined by default (but default to runc) (#9026, @rptaylor)--supervisor-fss-namespace=kube-system flag to vcloud-csi installation (#9066, @yasintahaerol)scale.yml playbook when etcd installed by kubeadm (#9210, @LuckySB)auto_renew_certificates_systemd_calendar (#8979, @floryut)kube_pid_reserved must be a string (#9124, @liupeng0518)HW_OFFLOAD is now correctly handle (and will no longer always be false) (#9218, @floryut)max_concurrent=1000 in the CoreDNS config (#9307, @yankay)bin_dir value is changed to something other than /usr/local/bin, containerd configuration might need to be tweak, please check #9243Flatcar use-cases)cgroup_auto_mount: false. Moreover, you can enable or disable BPF with these variables cilium_enable_bpf_masquerade and cilium_enable_host_legacy_routingscale.yml playbook when etcd installed by kubeadm (#9210, @LuckySB)We are looking for maintainers, reach out in #5432.
etcd_deployment_type to replace the etcd_kubeadm_enabled variable (#8317, @necatican) (See Notes 3)
ip6 option host vars (#8542, @kakkotetsu)EventRateLimit plugin configuration (#8711, @alegrey91)kube_apiserver_service_account_lookup) (#8781, @alegrey91)eviction_hard/eviction_hard_control_plane) (#8421, @cyril-corbon)cert_manager_tolerations), nodeselector (cert_manager_nodeselector) and affinity (cert_manager_affinity) in cert-manager templates (#8389, @cyril-corbon)kubelet_streaming_connection_idle_timeout / kubelet_make_iptables_util_chains) (#8796, @alegrey91)upgrade_node_post_upgrade_confirm). (#8530, @mac-chaffee)snapshot_controller_namespace (#8305, @liupeng0518)kube_webhook_token_auth_url_skip_tls_verify / kube_webhook_token_auth) (#8777, @dlouks)container_manager variable for Etcd hosts (#8521, @vi7)generate_list.sh using ansible (#8538, @tmurakam)cache plugin configuration via the coredns_default_zone_cache_block variable (#8488, @Tristan971)calico_apiserver_enabled variable) (#8690, @liupeng0518)calico_ip6_auto_method (#8541, @kakkotetsu)calico_ipam_strictaffinity param) (#8581, @eyenx)v1.22.0 (#8629, @Xartos)for_each for openstack. Makes it easier to switch out master nodes via terraform. (#8709, @robinAwallace)var.vapp when a vapp is referenced (vsphere_hostname is also removed) (#8441, @ceesios)master_preemptible and worker_preemptible) (#8480, @sathieu)with_networks variable to external_hcloud_cloud in ansible playbook and network_zone variable to Hetzner Cloud Terraform. (#8702, @Anthony-Bible)0090-etchosts file when setting override_system_hostname=false (#7634, @liupeng0518)kube-dns service will no longer be deleted if not created by kubespray (#8565, @cyril-corbon)etcd_deployment_type: host (#8386, @rtsp)etcd_kubeadm_enabled: true (#7766, @forselli-stratio)first_kube_control_plane/joined_control_planes (#8412, @floryut)docker-ce.repo failed (#8856, @Thearas)local_path_provisioner_enabled/multi_networking (#8650, @liupeng0518)dns_early|dns_late) of cluster deployment
[systemd-resolved] Add upstream_dns_servers to FallbackDNS
[cluster-reset] Revert DNS configuration to early stage (for instance: only defined upstream nameservers) (#8561, @onock)check_mode: no from gen_certs_script.yml to prevent changing files (#8573, @fungusakafungus)n/a
identity_allocation_mode has been overridden locally, it needs to be changed to cilium_identity_allocation_mode.kube_encryption_algorithm flag, then you must set kube_encryption_algorithm to aescbc since the default value has changed to the more secure secretbox standard.etcd_kubeadm_enabled is deprecated. You can set etcd_deployment_type to kubeadm to get the same behaviour."contrib/terraform/openstack/modules/compute/templates/cloudinit.yaml before deployment. (Currently it uses one cloud init for all instances.)containerd_insecure_registries needs to be updated or won't work anymore!generate_list.sh using ansible (#8606, @tmurakam)etcd_deployment_type: host (#8404, @rtsp)containerd_insecure_registries needs to be updated or won't work anymoreWe are looking for maintainers, reach out in #5432.
argocd_enabled variable (#7895, @atorrescogollo)registry_service_type, registry_service_clusterIP, registry_service_loadBalancerIP, registry_service_annotations, registry_service_nodePort) (#8291, @zhengtianbao)registry_tls_secret, registry_htpasswd, registry_config) (#8229, @zhengtianbao)cert_manager_trusted_internal_ca to specify trusted internal ca of cert_manager. (#8135, @infra-monkey)metrics_server_resizer (default to false) to control the addon-resizer container deployment in metrics-server pod (#8018, @oomichi)--disable-eviction flag (#8094, @utkuozdemir)kubelet_fail_swap_on, default to true) (#8241, @cristicalin)kubeadm_join_phases_skip (#8067, @necatican)addon-resizer is used on a platform different than amd64 (#8144, @zhengtianbao)kube_feature_gates from kebelet args to kubelet config (#8048, @fungusakafungus)ruamel.yaml.clib need to be updated to 0.2.4 (#8034, @olivierlemasle)registry_storage_access_mode to changes access mode, registry_replica_count for replicas) (#8198, @zhengtianbao)registry_ingress_annotations, registry_ingress_host, registry_ingress_tls_secret) (#8311, @zhengtianbao)cinder_csi_rescan_on_resize to control rescan-on-resize option (#8057, @reneluria)cinder_tolerations that sets tolerations for cinder-csi-nodeplugin DaemonSet (no tolerations by default) (#8137, @Ajarmar)metallb_pool_name) (#8111, @damjanek)matallb_auto_assign) (#8193, @IKRozhkov)use_server_groups with the option to enable and set server group policy for each of the master, etcd, and node server groups respectively. (#8046, @OlleLarsson) (see Notes 2)
node_volume_type variable) (#8256, @robinAwallace)calico_allow_ip_forwarding (#8184, @zhengtianbao)calico_cni_config object allowing user to add nodes using both playbooks (#7717, @dlouks)calico_node_readinessprobe_timeout/calico_node_livenessprobe_timeout to tune them (#7981, @cristicalin)calico_min_version check relevant (#7939, @cristicalin)calico_pool_blocksize is defined in inventory, the assertion on blocksize equality wrongly fails because a string cast is missing (#8321, @emiran-orange)upgrade_post_cilium_wait_timeout to control that (By default 120 seconds) (#7978, @reneluria)enable-metrics key missing) (#8000, @L3o-pold)weave_npc_extra_args (#8140, @brainfair)etcd_deployment_type: host (#7532, @VannTen)containerd_manager==docker (default config) you will now need to use docker_containerd_version to change the containerd version instead of the established containerd_version (#8130, @cristicalin)containerd_runtimes is now containerd_additional_runtimes ) (#8123, @pasqualet)/usr/bin to bin_dir (defaults to /usr/local/bin) - Fixing install for FCOS (#8107, @mafn)docker_cli_versioned_pkg dict (#8019, @electrocucaracha)resolveconf_mode is set to docker_dns) (#8263, @toplordsaito)coredns-config.yml.j2 (#8224, @Ishizuka427)apiserver_loadbalancer_domain_name pointing to external LB instead of dbip (#8299, @singeleaf)~ [REVERTED]oci value (and removing deprecated ones) (#8164, @oomichi)loadbalancer_apiserver_localhost is true) (#8262, @Bledai)/usr/bin/ (#7992, @lazybetrayer)/os/) (#8208, @buker)kubectl delete node to k8s nodes and not etcd (#8101, @VannTen)check_mode (#8133, @Isakgicu)kubelet_shutdown_grace_period and kubelet_shutdown_grace_period_critical_pods (#7993, @cristicalin) (see Notes 1)
local-volume-provisioner image from quay to k8s.gcr (#8054, @foxdalas)kube_config_dir for kubeconfig instead of hard path in multiple plays (#7996, @oomichi)initial_delay_seconds to 10 seconds) (#8309, @zemkogabor)n/a
use_server_groups is no more, please use master_server_group_policy/node_server_group_policy and etcd_server_group_policy
metrics_server_resizer (default to false) to control the addon-resizer container deployment in metrics-server pod (#8018, @oomichi)--disable-eviction flag (#8102, @utkuozdemir)calico_cni_config object allowing user to add nodes using both playbooks (#7717, @dlouks)calico_node_readinessprobe_timeout/calico_node_livenessprobe_timeout to tune them (#7981, @cristicalin)enable-metrics key missing) (#8000, @L3o-pold)/usr/bin/ (#7992, @lazybetrayer)We are looking for maintainers, reach out in #5432.
kubeadm_upgrade_auto_cert_renewal to control certificates renewal during control plane upgrade (#7976)--allow-version-mismatch in calicoctl.sh to allow upgrades (#7873)enable_dual_stack_networks is true (#7944)tags: always to all included service playbook (#7906)--no-cache-dir flag to pip in dockerfiles to save space (#7898)roles/upgrade/pre-upgrade/defaults/main.yml:upgrade_node_always_cordon to true causes a node to be drained before an upgrade and uncordoned after an upgrade even if the node is not cordoned when the upgrade begins.kubernetes.io/ingress.class: nginx on managed ingressesnetworking.k8s.io/v1beta
--dynamic-config-dir has been deprecated, Feature DynamicKubeletConfig is deprecated in 1.22 and will not move to GA. It is planned to be removed from Kubernetes in the version 1.23. Please use alternative ways to update kubelet configuration.add command e.g. $ inventory.py add 10.0.1.8
We are looking for maintainers, reach out in #5432.
tolerations and nodeSelector for metallb components (controller and speaker) (#7334)io.containerd.runc.v2 and cgroup to systemd (#7398)docker_dns_servers_strict had different default values, the default is now the same everywhere: false (#7499)enablerepo: amzn2extra-docker to allow docker installation on Amazon linux (#7507)calico_bird_listen_port variable (#7419)calico_node_startup_loglevel to configure CALICO_STARTUP_LOGLEVEL (Default to error) (#7530)ingress_ambassador_multi_namespace setting, allows Ambassador operator to watch all namespaces for AmbassadorInstallation CRD resources (#7516)ping_access_ip variable to enable(default)/disable ping test during preinstall (#7020)upgrade_node_confirm, default false) and delay (upgrade_node_pause_seconds, default 0 seconds) (#7168)jinja2_native=True (#7612 / #7606)auto_renew_certificates, or manually use k8s-certs-renew.sh force_certificate_regeneration is removed as it was only renewing the api server certs and not all the other onesThis release includes the following changes (among other things):