Terraform + Ansible + Docker Swarm boilerplate = DevOps on :fire::fire::fire: | Infrastructure as Code
A boilerplate to create a full Infrastructure as Code (IaC) repository, from provisioning to deployment with:
It handles different environments:
localhost
: a single node Docker Swarm cluster on your machine, useful for development (demo)vagrant
: a 3 nodes production-like cluster deployed with Vagrant on your machine, useful for testing (demo)production
: your production environment! It can be created by Terraform or you can use an existing bare metal/VMs infrastructure (demo)On top of that, it features:
./tads
), which is a wrapper around Terraform, Ansible and Vagrant commands. For example: ansible-playbook -i inventories/production -D --vault-id production@vault_keys/production deploy.yml
becomes ./tads ansible-playbook production deploy
. More convenient, don't you think? :smirk:/etc/hosts
management to access your local and Vagrant applications with .localhost
and .test
https URIsWith T.A.D.S., you will be able to onboard a new developer on your project in less than 3 minutes, with just 3 commands! Even if you have a complex microservices architecture. Forget about your outdated wikis or installation procedures, they are no longer needed! See the example user README to get a preview of what your new procedures could look like.
If you recognize yourself into some of these statements, this project is definitely for you:
On the contrary, this project might not be for you if:
... but don't be sad, I am thinking of creating a similar project for K8s ;) Tell me if you want to help!
Before going further, I assume that you already have the knowledge and practice with Docker Swarm mode, Ansible, Terraform, and Infrastructure as Code in general. If it is not the case, I urge you to study and practice before. You can use this material as a starter:
Have a look at Install the required dependencies for installation procedures.
OS X: It should not be that hard to make the project run on OS X. PRs are welcome! I am also thinking of creating a dockerized version of the project to improve compatibility.
Clone this repo, create your own and push the code to it.
git clone --single-branch https://github.com/Thomvaill/tads-boilerplate.git <YOUR_PROJECT_NAME>
cd <YOUR_PROJECT_NAME>
git remote set-url origin <YOUR_REPO_URL>
git push
This will install Ansible, Vagrant, Virtualbox and Terraform on your local machine:
./tads install-dependencies
You can also manually install the dependencies if your preferer.
ansible/group_vars/localhost_overrides.sample.yml
to ansible/group_vars/localhost_overrides.yml
ansible_user
variable with your user nameansible_user: <yourUserName>
./tads ansible-playbook localhost provision
./tads ansible-playbook localhost deploy
The first ./tads
command will:
yourcompany.localhost
to your /etc/hosts
fileAnd the second one will deploy traefik
and example_app
stacks.
If everything went well, you are now able to access it at this URL: https://yourcompany.localhost/
Now that the example stack is running on your machine, you can deploy your own services.
First, you probably need to change the domains
dict in ansible/group_vars/all.yml
.
This file contains all Ansible variables default values. These values can be overridden later in other group_vars files.
You are free to add your variables in it.
Then, you can write your own Docker Swarm Compose files, following this naming convention: ansible/stacks/<STACK_NAME>/<STACK_NAME>.yml.j2
These files are Jinja2 templates.
You are highly encouraged to use Ansible variables in them, so your template file can be used across all your environments.
Have a look at ansible/stacks/example_app/example_app.yml.j2
to see a good example.
Finally, do not forget to add your new stacks to ansible/deploy.yml
.
To help you with the ansible/group_vars
directory, here is a representation of Ansible groups:
all
├── dev
| ├── localhost
| └── vagrant
├── production
├── staging
└── any_other_remote_environment...
Each group has its _overrides
counterpart, which enables you to override some variables locally in a xxx_overrides.yml
file,
which is not versionned.
Have a look at .sample.yml
files to see some examples.
While developing, perform some ./tads ansible-playbook localhost deploy
to test your deployment.
Do not forget to run ./tads ansible-playbook localhost provision
again if you have changed domain names.
Tips:
./tads ansible-playbook localhost all
to provision and deploy in a single command./tads ansible-playbook localhost all --tags dev,stack-traefik
:latest
tag because you don't control what will be pushed to production. With specific tags, you will have idempotent deployments and you will be able to perform rollbacksNow that you are happy with your localhost environment, you should test the provisioning and the deployment on an environment which looks more like a production environment. For instance, on localhost, you can have just one node! And maybe you forgot some dependencies that are already installed on your computer. With Vagrant, you will be able to test your stacks on a fresh 3 nodes Swarm cluster.
vagrant/vagrant.sample.yml
to vagrant/vagrant.yml
and adjust its settings./tads vagrant up
./tads ansible-playbook vagrant all
Now, you will be able to test your stacks deployed on Vagrant. If you have kept the example app, you can test it on https://yourcompany.test/
Tips:
./tads vagrant destroy
./tads vagrant ssh vagrant-1
Before going further, you should edit your production group_vars files:
ansible/group_vars/production.yml
ansible/group_vars/production_encrypted.yml
When you are done, do not commit production_encrypted.yml
! You have to encrypt it first:
./tads ansible-vault production init-key
./tads ansible-vault production encrypt ansible/group_vars/production_encrypted.yml
The first command has generated a random key in ansible/vault_keys/production
.
You must not commit this file. You should keep it in a safe place, and share it with your authorized team members securely.
If you lose it, you won't be able to decrypt your files anymore. The second one has encrypted your file
with AES-256. You can now commit it.
You can still edit this file by running ./tads ansible-vault production edit ansible/group_vars/production_encrypted.yml
. Always check that you do not commit an unencrypted version of this file by mistake.
Now that everything is fine locally, it is time to create and deploy your production environment!
The terraform/environments/production
is an AWS example. PRs are welcome for other providers!
To make it work, you should:
pip3 install awscli --upgrade --user
aws configure
Terraform will use this default profile credentials.
Then, you can run ./tads terraform production init
and ./tads terraform production apply
.
This example will create:
The CLI will also create the corresponding Ansible inventory for you in ansible/inventories/production
from Terraform outputs. You should commit it.
You should also commit the Terraform state file, or better: use a remote state.
Then, you have to create an alias in Route53 to the ELB.
Finally, you can run ./tads ansible-playbook production all
and your website will be available!
Disclaimer:
./tads terraform production destroy
with caution :-)If you don't want to use a cloud provider, you can use classic Virtual Machines. For a production environment, you should have at least 3 manager nodes, so 3 VMs. They should be fresh installs. Ubuntu server 18.04 or Debian 9 is fine.
ansible/inventories/production.sample-baremetal
to ansible/inventories/production
./tads ansible-playbook production all
and your website will be available!You can add other remote environments, like production.
For Terraform, you just have to duplicate terraform/environments/production
to the directory of your choice, eg staging
.
After editing it, you can run ./tads terraform staging apply
, it will create the ansible/inventories/staging
inventory file.
For an existing bare metal infrastructure, you just have to create the ansible/inventories/staging
inventory file.
Then, in Ansible, you have to create these files:
ansible/group_vars/staging_encrypted.yml
ansible/group_vars/staging.yml
Then, create the ansible-vault key and encrypt the file:
./tads ansible-vault staging init-key
./tads ansible-vault staging encrypt ansible/group_vars/staging_encrypted.yml
Finally, provision and deploy! ./tads ansible-playbook staging all
It is one of this project's goals: DevOps is not a job, it is a mindset! Now that you have a beautiful IaC, it is time to onboard your team members.
README.md
by README.example.md
and customize it, so your team can use the project easilyThere is no documentation of the CLI since you will probably modify it, or add new commands!
To get some help, just run ./tads
. Do not hesitate also to have a look at the source into the scripts
directory. This CLI is just a wrapper of Terraform, Ansible and Vagrant commands.
Use Ansible tags! Example if you just want to deploy the traefik
and myapp
stack: ./tads ansible-playbook localhost deploy --tags stack-traefik,stack-myapp
.
I have not taken the time to develop this feature properly yet. But basically what you can do is:
tags.yml
file./tads ansible-playbook production deploy
, and in case of success commit and push the fileThis problematic is beyond the scope of this project and depends a lot on your infrastructure / cloud provider. I advise you to have a look at REX-Ray.
This might be a future feature to implement this plugin in the boilerplate.
Pull Requests are more than welcome! Please read CONTRIBUTING.md for more details.
Development:
./tads install-dependencies --dev
make lint
make test
This project is licensed under the MIT license, Copyright (c) 2019 Thomas Vaillant. For more information see LICENSE file.