terraform-all-in-one

Get fine-grained Kubernetes + Infrastructure on AWS in 30 mins 🚀

• Prerequisite
• Usage
• Features
• Credits

Prerequisite

• ansible
• jq
• terraform
• kops
• awscli

$ brew install ansible jq terraform kops watch
$ pip install awscli

$ git clone [email protected]:1ambda/terraform-all-in-one.git
$ cd terraform-all-one

# Remove .gitigonre to index generated files
rm .gitignore

Usage

1. Export AWS Key Environment variables

The key should have AdministratorAccess permission.

$ export AWS_ACCESS_KEY_ID={VALUE} AWS_SECRET_ACCESS_KEY={VALUE}

2. Generate SSH Key Pair

# Modify values for `COMPANY`,`PROJECT`, and `EMAIL`
$ COMPANY=github PROJECT=1ambda [email protected] ./create-ssh-key.sh

3. Modify Terraform Variables to Customize

• variable.customize.tf
• variable.resource.tf

company and project variable should match with values used for the generated ssh key.

4. Applying Terraform Modules

• root-infra: Create VPC, Bastion, ECS, Stroages and build kops scripts

  cd root-infra;
  
  # build infra using terraform
  terraform init
  terraform apply -var 'rds_username={USERNAME}' -var 'rds_password={PASSWORD}'
  
  # provision non-managed stroages using ansible
  ../script-provision/generated.provision-zookeeper.sh
  

• root-kubernetes: Build Kubernetes Cluster and install add-ons

  cd root-kubernetes;
  
  # generate kops files
  $(cat generated.kops-env.sh);
  ./generated.kops-create.sh;
  
  # build kubernetes cluster
  terraform init
  terraform apply
  
  # wait for few minitues until Kube API ELB is ready (`api-kops-*`)
  # then validate the created cluster
  kops export kubecfg --name=$NAME
  ./generated.correct-kubectl-context.sh
  
  # wait for 3-5 mins until kubernetes cluster is ready
  kops validate cluster
  kubectl get pods
  

Features

• VPC
• Basiton Host
  • Install Storage Clients (e.g mysql, redis, ..) using user-data
  • Configurable SSH Whitelist
  • SSH Connection, Proxy Utilities
• ECS
  • ASG Scaling Policies
  • Cloudwatch Log Groups for ECS AWSLOGS
  • Cloudwatch Custom Metrics + Alerts for ECS: Logical Volume
  • Customized Container Volume Size (--storage-opt dm.basesize)
  • Report Inactive ECS Host Machine to Slack via Lambda + ECS Event
  • Dynamic Cloudwatch Alarm Registration for ASG Event
• Managed Storages: RDS (MariaDB), Elasticsearch, Elasticache (Redis)
  • Cloudwatch Default Metrics + Alerts
  • Cloudwatch Log Groups for RDS: Audit, Slow Index, Error, General
  • Cloudwatch Log Groups for ES: Slow Query, Slow Index
  • Configurable Clustering
  • RDS option group for utf8mb4 :see_no_evil:
• Barematal Storages: Zookeeper
  • Ansible Provisioning
  • Cloudwatch Log Groups for ZK AWSLOGS
  • Cloudwatch Custom Metrics + Alerts for EC2: Memory, Disk Space
  • ELB-backed Health Check (Cloudwatch Alarm
• Kubernetes Cluster (Single Master)
  • Terraform Intergrated Kubernetes Cluster Creation using kops
  • Cloudwatch Log Groups for Kube AWSLOGS
  • Dynamic Cloudwatch Alarm Registration for ASG Event
  • Cloudwatch Custom Metrics + Alerts for EC2: Memory, Disk Space
  • add-on: Nginx Ingerss Chart with AWS ACM
  • add-on: Kubernetes Dashboard
  • add-on: Elasticsearch, Kibana, Fluentd

Credits

• terraform-aws-vpc
• terraform-sns-slack