Kismatic Versions Save

Kismatic Enterprise Toolkit: Fully-Automated, Production-Grade Kubernetes Operations

v1.12.0

5 years ago

Notable Changes

  • Add the ability to override apiserver port. (https://github.com/apprenda/kismatic/issues/750)
    • Field apiserver_cert_extra_sans to specify additional extra SANs for the API Server serving certificate.
    • Field load_balancer allows for setting both the IP or DNS and Port.
cluster:
  # Generated certs configuration.
  certificates:
...
    # Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate.
    # Can be both IP addresses and DNS names.
    apiserver_cert_extra_sans: ""
...

master:
  expected_count: 2

  # If you have set up load balancing for master nodes, enter the IP or DNS and Port.
  # Otherwise, use the IP address of a single master node and port '6443'.
  load_balancer: ""
...

Plan File Changes

Deprecated Fields

Field Notes
master.load_balanced_fqdn Replaced by master.load_balancer.
master.load_balanced_short_name Specify additional SANs in cluster.certificates.apiserver_cert_extra_sans.

New Fields

Field Notes
cluster.certificates.apiserver_cert_extra_sans Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate.
master.load_balancer If you have set up load balancing for master nodes, enter the IP or DNS and Port. Otherwise, use the IP address of a single master node and port '6443'.
add_ons.dashboard.options.node_port When using NodePort set the port to use. When left empty Kubernetes will allocate a random port.

Component Version Changes

  • Kubernetes v1.10.5

v1.11.1

6 years ago

Notable Changes

add_ons:
  cni:
    options:
      portmap:
        disable: false

Plan File Changes

New Fields

Field Notes
add_ons.cn.options.portmap.disable Set to true to disable the portmap plugin

Component Version Changes

  • Kubernetes v1.10.3
  • Calico v2.6.10
  • CoreDNS 1.1.3
  • Nginx Ingress 0.15.0
  • Helm v2.9.1

v1.11.0

6 years ago

Actions Required Before Upgrade

  • If your cluster contains any components using the unathenticated Kubelet port 10255 (heapster and metrics-server deployed by KET will be modified to use the correct port during the upgrade):
    • Modify those deployments to use port 10250 or
    • Set --read-only-port 
  kubelet:
    option_overrides:
      read-only-port: 10255

Notable Changes

  • Fix a bug where node labels were being overwritten if taints is also specified for that node. (#1213)
  • Properly set /etc/docker/daemon.json options restarting the docker daemon on change. (#1221)
  • Set --read-only-port=0, this disables the unathenticated Kubelet API access on port 10255. (#1135)
    • The instances of heapster and metrics-server deployed on the cluster were modified to use port 10250 and authorize via standard RBAC policies.
    • Kubelet option --authentication-token-webhook was enabled to support API bearer token authentication to the Kubelet’s HTTPS endpoint.
  • Enable the portmap CNI plugin for both Calico and Weave to suport hostPort in pods. (#1207)
  • By fixing #1207, this allowed to deploy the Nginx Ingress Controller in the cluster networking instead of having to set hostNetwork: true. (#1226)
    • A major benefit of not using hostNetworking is the ability to define networkPolicies for the Ingress Controller pods.
  • Validate Kubernetes components from each node to support adding workers when other nodes are unhealthy. (#1199)
    • Prior behavior was to compare the Desired pod count to Ready pod count on the DaemonSet resource.
  • Quote flags passed to Kubernetes components to support option_overrides with special characters. (#1217)
  • Use new API server option --enable-admission-plugins instead of the deprecated --admission-contro. (#1212)

Component Version Changes

  • Kubernetes v1.10.2
  • Calico v2.6.9
  • KubeDNS 1.14.10
  • CoreDNS 1.1.2
  • Nginx Ingress 0.14.0
  • Helm v2.9.0

v1.10.0

6 years ago

Actions Required Before Upgrade

  • A new flag --kubelet-certificate-authority on the kube-apiserver is now being set. (#1178)
    • With this flag set the api server will now verify a Kubelet's service certificate. Prior to this release the kubelet certificates SANs were not being set and new valid certificates are required.
    • Certificates will need to be removed from the generated/ directory prior to upgrading to allow the tool to regenerate them. (Or regenerated by the user manually)
    • Certificate SANs will be validated during install or upgrade to contain the node's hostname and internalip.
Configuring Certificates============================================================
Found valid certificate for etcd001 etcd server                                 [OK]
Found valid certificate for master001 API server                                [OK]
Found valid certificate for kubernetes controller manager                       [OK]
Found valid certificate for kubernetes scheduler                                [OK]
Found valid certificate for service account signing                             [OK]
Found certificate for master001 kubelet, but it is not valid                    [ERROR]
- Certificate "master001-kubelet.pem": SANs validation failed
    expected:
	[192.168.42.3 master001]
    instead got:
	[]
error generating certificates for the cluster: invalid certificate found for "master001 kubelet"

Notable Changes

  • Upgrade Kubernetes to v1.10.0. (#1188)
    • With this release of KET only Kubernetes v1.10.x versions will be permitted.
  • Expose taints option in the plan file for master, worker, etcd and storage nodes. (#1057)
worker:
  expected_count: 3
  nodes:
  - host: "worker1"
    ip: "1.2.3.1"
    labels: {}
    taints:
    - key: "foo"
      value: "bar"
      effect: "NoSchedule"

  - host: "worker2"
    ip: "1.2.3.2"
    labels: {}
    taints:
    - key: "xyz"
      value: "zyx"
      effect: "PreferNoSchedule"

  - host: "worker3"
    ip: "1.2.3.3"
    labels: {}
    taints: []
  • New reset command to rollback any changes made to the hosts by apply. (#955)
➜ ./kismatic reset --remove-assets
=> Are you sure you want to reset the cluster? All data will be lost [N/y]:
➜ ./kismatic reset --remove-assets --force

Resetting Nodes in the Cluster======================================================
Reset All Nodes                                                                 [OK]

Removing Assets Directory===========================================================
Remove "generated" directory                                                    [OK]
  • Improve upgrades, no longer fail checks for components that were deployed by the cluster. (#1104)
  • Improve preflights, validate package availability during an offline installation. (#949)
  • Improve the dashboard command with a new --token flag that will generate a kubeconfig file with kubernetes-dashboard-admin ServiceAccount token without running kismatic proxy. (#1195)
  • Fix kismatic volume delete command to properly work with volumes where PVs have been delete with kubectl. (#1185)
  • Deprecate nfs: storage options in the plan file. (#946)
    • Existing plan files with the nfs: tag will still function as before.
    • New plan files will not contain the nfs: tag.

Plan File Changes

Deprecated Fields

Field Notes
nfs Will be removed completely in a future release.

New Fields

Field Notes
dashboard.options.service_type Kubernetes service type of the Dashboard service. Options: ClusterIP, NodePort, LoadBalancer, ExternalName.
nodes.taints Kubernetes taints applied to the nodes during installation.

Component Version Changes

  • Kubernetes v1.10.0
  • Etcd v3.1.13
  • Weave v2.3.0
  • KubeDNS 1.14.9

v1.9.2

6 years ago

Notable Changes

additional_files:
- hosts:
  - "master001"
  source: /tmp/ca-key.pem
  destination: /etc/kubernetes/pki/ca-key.pem
  skip_validation: true
- hosts:
  - "all"
  source: /root/foo.txt
  destination: /tmp/foo.txt
- hosts:
  - "worker"
  source: /tmp/bar.txt
  destination: /tmp/bar.txt

Plan File Changes

New Fields

Field Notes
add_ons.cni.options.weave.password Used by Weave for network traffic encryption. Sets WEAVE_PASSWORD in the deployment.
add_ons.dns.options.replicas The number of cluster DNS pods to run.
additional_files Used to copy arbitrary files and directories to the cluster nodes.

Component Version Changes

  • Kubernetes v1.9.6
  • Helm v2.8.2
  • Weave v2.2.1
  • CoreDNS 1.1.1
  • Nginx Ingress Controller 0.12.0

v1.9.1

6 years ago

Notable Changes

Component Version Changes

  • Calico v2.6.8
  • Weave 2.2.0
  • Nginx Ingress Controller 0.11.0
  • CoreDNS 1.0.6
  • Kubernetes Dashboard v1.8.3
  • Heapster v1.5.1

v1.9.0

6 years ago

Actions Required Before Upgrade

Please read through the release notes as there are many Kubernetes component flag changes that might impact applications that communicate to either the kube-apiserver or the kublet API.

Notable Changes

Metrics Server

add_ons:
  # Metrics Server is a cluster-wide aggregator of resource usage data.
  metrics_server:
    disable: false

Cluster Security

NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
  • Set cadvisor-port to 0, this will disable access to the cAdvisor API on port 4194, if your cluster still depends on this API set:
cluster:
  kube_apiserver:
    option_overrides:
        cadvisor-port: 4194

Component Flag Changes

kube-apiserver:

--kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.pem
--kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client-key.pem
--profiling=false
--repair-malformed-updates=false

kube-controller-manager:

--profiling=false

kube-scheduler:

--profiling=false

kube-proxy:

--profiling=false

kubelet:

--authorization-mode=Webhook
--client-ca-file=/etc/kubernetes/pki/ca.pem
--cadvisor-port=0
--event-qps=0
--make-iptables-util-chains=true
--streaming-connection-idle-timeout=0

Plan File Changes

New Fields

Field Notes
add_ons.cni.options.calico.ip_autodetection_method Used to detect the IPv4 address of the host. Sets IP_AUTODETECTION_METHOD in the Calico deployment.
add_ons.metrics_server A cluster-wide aggregator of resource usage data.
add_ons.metrics_server.disable When set to true skips installation of the metrics_server

Component Version Changes

  • Kubernetes v1.9.3
  • Go v1.9.3
  • Calico v2.6.7
  • Helm v2.8.1

v1.8.1

6 years ago

Notable Changes

  • Fix a bug where preflight checks fail when adding a worker if preflight was originally skipped (#918)
    • This was fixed in v1.7.1 but an improper merge for v1.8.0 caused the regression
  • Fix a bug where installation does not start when a key in the plan file contains a number (#1099)

v1.8.0

6 years ago

Notable Changes

Kubernetes Version Decoupling

  • Allow for setting the Kubernetes patch version used during installation and upgrades - Documentation (https://github.com/apprenda/kismatic/pull/1048)
    • Leave blank if you wish to use the version of Kubernetes used in the integration-tests for this release.
    • A particular KET release will only support a single Kubernetes minor version, i.e. KET v1.8.0 will be able to install any v1.9.x version.
    • It's no longer necessary to upgrade KET just to install or upgrade to a new patch version of Kubernetes.
cluster:

  # Kubernetes cluster version (supported minor version "v1.9.x").
  version: v1.9.2

More Flexible Docker Support

  • Support installations on nodes with docker already installed and configured (https://github.com/apprenda/kismatic/issues/474)
  • Validate that an unsupported version of docker is not already installed (https://github.com/apprenda/kismatic/issues/672)
  • Support any docker storage driver (https://github.com/apprenda/kismatic/pull/1067)
    • Leave docker.storage.driver empty to have docker automatically select the best available option.
    • In previous versions the only additional driver supported was devicemapper in direct-lvm mode
    • The plan file docker.storage.opts now support providing docker storage options that will be set directly in docker's daemon.json file.
    • KET will still be able to create logical volumes used by docker in direct-lvm mode.

IMPORTANT: The storage driver option SHOULD NOT be modified for a running cluster. The result is undetermined and may result in a broken cluster. Only use when creating a brand new cluster.

# Docker daemon configuration of all cluster nodes.
docker:

  # Set to true if docker is already installed and configured.
  disable: false
  storage:

    # Leave empty to have docker automatically select the driver.
    driver: ""
    opts: {}

    # Used for setting up Device Mapper storage driver in direct-lvm mode.
    direct_lvm_block_device:

      # Absolute path to the block device that will be used for direct-lvm mode.
      # This device will be wiped and used exclusively by docker.
      path: ""
      thinpool_percent: "95"
      thinpool_metapercent: "1"
      thinpool_autoextend_threshold: "80"
      thinpool_autoextend_percent: "20"

To maintain backwards compatibility providing the docker.storage.direct_lvm field will still work like before but it is now considered deprecated and will be removed in a future version.

The new schema allows the user to override any of the previously hardcoded options.

Old config (Deprecated)

docker:
  disable: false
  storage:
    direct_lvm:
      enabled: true
      block_device: "/dev/xvdb"
      enable_deferred_deletion: false

New Equivalent Config

docker:
  storage:
    driver: devicemapper
    opts:
      dm.thinpooldev: /dev/mapper/docker-thinpool
      dm.use_deferred_deletion: "false"
      dm.use_deferred_removal: "true"
    direct_lvm_block_device:
      path: /dev/xvdb
      thinpool_percent: "95"
      thinpool_metapercent: "1"
      thinpool_autoextend_threshold: "80"
      thinpool_autoextend_percent: "20"

Helm Improvements

add_ons:
  package_manager:
    disable: false

    # Options: 'helm'.
    provider: helm
    options:
      helm:
        namespace: kube-system

Miscellaneous

./kismatic install add-node node001 192.168.42.1 --roles worker,ingress
./kismatic install add-node node002 192.168.42.2 --roles ingress
./kismatic install add-node node003 192.168.42.3 --roles storage
./kismatic install add-node node004 192.168.42.4 --roles worker,ingress,storage

The old CLI command (add-worker) has been aliased to the new add-node and will still work as before.

./kismatic install add-worker node005 192.168.42.5

add_ons:
  dns:
    disable: false

    # Options: 'kubedns','coredns'.
    provider: coredns

IMPORTANT: Switching providers during an upgrade is not supported at this point. If you choose to switch you may experience downtime.

Plan File Changes

New Fields

Field Notes
cluster.version Kubernetes version used during installation or upgrade
docker.disable Disables the installation of docker. Set to true if docker has already been installed and configured on the nodes
docker.storage.driver Equivalent to "storage-driver" in docker's daemon.json
docker.storage.opts Equivalent to "storage-opts" in docker's daemon.json
docker.storage. direct_lvm_block_device Block device configuration used with devicemapper storage driver. Replaces docker.storage.direct_lvm options.
add_ons.dns.provider A new in-cluster DNS add-on was added. Options: 'kubedns','coredns'
add_ons.package_manager.options.helm.namespace Configure the kubernetes namespace to deploy tiller

Deprecated Fields

Field Notes
docker.storage.direct_lvm Replaced by docker.storage. direct_lvm_block_device

CLI Changes

  • The add-worker command was renamed to add-node. Note add-worker is an alias and will still function as before.

Component Version Changes

  • Kubernetes v1.9.2
  • Go v1.9.2
  • Calico v2.6.6
  • KubeDNS 1.14.8
  • CoreDNS 1.0.5
  • Nginx Ingress Controller 0.10.2 - IMPORTANT release notes if you are using Prometheus to scrape the ingress pod
  • Kubernetes Dashboard v1.8.2

v1.7.1

6 years ago

Notable Changes

Component Version Changes

  • Kubernetes v1.9.2