Terraform Kubernetes Save

Table of Contents

• About the repo
• Quick start
• Repository structure
  • terraform-modules
  • my-cluster
  • accounts
• CI/CD example with Gitlab CI and Helm

About the repo

This repository contains an example of deploying and managing Kubernetes clusters to Google Cloud Platform (GCP) in a reliable and repeatable way.

Terraform is used to describe the desired state of the infrastructure, thus implementing Infrastructure as Code (IaC) approach.

Google Kubernetes Engine (GKE) service is used for cluster deployment. Since Google announced that they had eliminated the cluster management fees for GKE, it became the safest and cheapest way to run a Kubernetes cluster on GCP, because you only pay for the nodes (compute instances) running in your cluster and Google abstracts away and takes care of the master control plane.

Quick start

Prerequisite: make sure you're authenticated to GCP via gcloud command line tool using either default application credentials or service account with proper access.

Check terraform.tfvars.example file inside my-cluster folder to see what variables you need to define before you can use terraform to create a cluster.

You can run the following command in my-cluster to make your variables definitions available to terraform:

$ mv terraform.tfvars.example terraform.tfvars # variables defined in terraform.tfvars will be automatically picked up by terraform during the run

Once the required variables are defined, use the commands below to create a Kubernetes cluster:

$ terraform init
$ terraform apply

After the cluster is created, run a command from terraform output to configure access to the cluster via kubectl command line tool. The command from terraform output will be in the form of:

$ gcloud container clusters get-credentials my-cluster --zone europe-west1-b --project example-123456

Repository structure

├── accounts
│   └── service-accounts
├── my-cluster
│   ├── deploy-app-example
│   └── k8s-config
│       ├── charts
│       │   └── gitlab-omnibus
│       │       ├── charts
│       │       │   └── gitlab-runner
│       │       │       └── templates
│       │       └── templates
│       │           ├── fast-storage
│       │           ├── gitlab
│       │           ├── ingress
│       │           └── load-balancer
│       │               └── nginx
│       ├── env-namespaces
│       ├── kube-lego
│       └── storage-classes
└── terraform-modules
    ├── cluster
    ├── firewall
    │   └── ingress-allow
    ├── node-pool
    └── vpc

terraform-modules

The folder contains reusable pieces of terraform code which help us manage our configuration more efficiently by avoiding code repetition and reducing the volume of configuration.

The folder contains 4 modules at the moment of writing:

• cluster module allows to create new Kubernetes clusters.
• firewall/ingress-allow module allows to create firewall rules to filter incoming traffic.
• node-pool module is used to create Node Pools which is mechanism to add extra nodes of required configuration to a running Kubernetes cluster. Note that nodes which configuration is specified in the cluster module become the default node pool.
• vpc module is used to create new Virtual Private Cloud (VPC) networks.

my-cluster

Inside the my-cluster folder, I put terraform configuration for the creation and management of an example of Kubernetes cluster. Important files here:

• main.tf is the place where we define main configuration such as creation of a network for our cluster, creation of the cluster itself and node pools.

• firewall.tf is used to describe the firewall rules regarding our cluster.

• dns.tf is used to manage Google DNS service resources (again with regards to the services and applications which we will run in our cluster).

• static-ips.tf is used to manage static IP addresses for services and applications which will be running in the cluster.

• terraform.tfvars.example contains example terraform input variables which you need to define before you can start creating a cluster.

• outputs.tf contains output variables

• variables.tf contains input variables

• k8-confing folder contains Kubernetes configuration files (manifests) which are used to define configuration of the running Kubernetes cluster. It has 4 subdirectories inside:

  • env-namespaces contains manifests for creating namespaces, or virtual environments within the cluster, for running our services. In this example, raddit-namespaces.yml file is used to describe 3 namespaces: raddit-stage and raddit-prod for running example application (which is called raddit in this case) in different virtual environments, and infra namespace for running services vital to our infrastructure like CI/CD, monitoring, or logging software.
  • storage-classes folder is used to create storage classes that could be then used in dynamic volume provisioning for our applications.
  • kube-lego folder has the configuration required to run kube-lego service which is used for automatic SSL certificates requests for our services running inside the cluster.
  • charts contains Helm charts for deploying infra services. In this case it only has a chart for deploying Gitlab CI along with a Runner.

• deploy-app-example has an bunch of Kubernetes objects definitions which are used to deploy nginx to a Kubernetes cluster. You can use the command below to deploy nginx to the cluster once it is created:

   kubectl apply -f ./deploy-app-example/nginx-example.yml
  

accounts

This is another top level folder in this project. It has a separate set of terraform files which are used to manage access accounts to our clusters. For example, you may want to create a service account for your CI tool to allow it to deploy applications to the cluster.

CI/CD example with Gitlab CI and Helm

For an example of building a CI/CD pipeline with Kubernetes, Gitlab CI, and Helm see this blog post.