Setup and run your own clusters on your own infrastructure in minutes (Eg. ECS or EKS). Roo is a zero config distributed ingress, edge-router & reverse-proxy (supporting multiple letsencrypt/https hosts) using Docker Swarm. No dependencies.
Roo is the opposite of Kubernetes. It's designed for sub-1 minute cluster setup times.
This basically lets you run your own encrypted and load balanced Amazon AWS clusters on your own hardware, and is a 5-minute replacement for Kubernetes, with no configuration (no additional setup for clustered kv stores, no janky config files, no defining providers, no dodgy second hand helm charts, no ssl setup, no manual ssl certification, no single point failures etc). You can setup a cluster, and publish a new domain in around 30 seconds (once you get the one time, 5 minute setup out of the way).
All you need to do is add a few lines to a docker-compose file and Roo does the rest see a full configuration example:
OriginHosts: example.com,www.example.com
OriginScheme: https
OriginPort: 443
DestinationHost: test_test
DestinationScheme: http
DestinationPort: 80
This aims to be a free replacement of Amazon's ECS (Elastic Compute Service), EKS (Kubernetes), CertificateManager, Load-Balancer and CloudWatch using your own Docker Swarm. It IS a complete replacement for nginx, traefik, haproxy, and a lot of kubernetes. The idea is to give developers back the power and take it back from ridiculous self-complicating dev-ops tools that get more complicated and less useful (for example Traefik 2 just removed support for clustered Letsencrypt from their open source version to spruik their enterprise version. Nginx and HAProxy do the same). I wasted a lot of time on their software before writing this. I truly hope it benefits others too.
If you are unfamiliar with swarm/kubernetes and are a developer and want a quick intro into how powerful and easy swarm can be, see how you can setup a thousand-machine cluster in just 20 lines (just copy + paste from there) or check out my command notes. In a day I was scaling clusters up and down on my own infrastructure with single commands.
The power Roo gives you is to add HTTPS://example1.com and HTTPS://example2.com to your clustered services with zero configuration. Let's encrypt allocates your service's certificates. It works across every machine, docker node, service, in your cluster.
Roo itself is clustered. Every machine it runs on shares the load to your services. It's distributed store shares certificates from Letsencrypt used across all your nodes. Now apple is denying certificates older than a year, I feel as a dev, that lets encrypt is almost mandatory as it creates a lot of admin.
docker run sfproductlabs/roo:latest
git mod download
make
# update the config if you need
sudo rood ./config.json
docker swarm init
docker network create -d overlay --attachable forenet --subnet 192.168.9.0/24
#the following label notifies that we should put a single instnce of roo on machines/swarm-nodes with the label "load_balancer"
docker node ls -q | xargs docker node update --label-add load_balancer=true
docker stack deploy -c roo-docker-compose.yml --resolve-image never roo
# Diagnostic Functions:
docker stack ls
docker stack services roo
docker service inspect roo_roo
docker stats --no-stream
docker node ls
docker node inspect --pretty <NODE_NAME>
# or for a single machine cluster
docker node ls -q | xargs docker node inspect --pretty
docker service ps roo_roo
docker service logs roo_roo -f
echo "docker service rm roo_roo # WARNING WILL REMOVE CLUSTER"
echo "docker stack rm roo # WARNING WILL REMOVE CLUSTER"
curl -X POST -i http://localhost:6299/roo/v1/kv/hop --data 'scotch'
This will set you up with a cluster on Hetzner Cloud (change the first 20 lines to suit your own cloud provider). I use this on my own production servers. I don't love Hetzner - the service isn't as good as I'd like - but it is improving and is CHEAP (~$3 per server).
# mac
brew install hcloud
brew install jq
#debian/ubuntu
#sudo apt install hcloud-cli jq
#create a project in hetzner called test (https://console.hetzner.cloud/projects)
#create a api key in the project you setup on hetzner
#hcloud context create test #connect the api key to the project
#test the connection
hcloud server-type list
#start with an empty project (check this is empty)
hcloud server list
hcloud ssh-key create --name andy --public-key-from-file ~/.ssh/id_rsa.pub
hcloud network create --ip-range=10.1.0.0/16 --name=aftnet
hcloud network add-subnet --ip-range=10.1.0.0/16 --type=server --network-zone=eu-central aftnet
#If you want a lot more machines see the horizontal web scraper project commands (https://github.com/sfproductlabs/scrp)
#for n in {1..30}; do (hcloud server create --name docker$RANDOM$RANDOM$RANDOM$RANDOM --type cx11 --image debian-9 --datacenter nbg1-dc3 --network aftnet --ssh-key andy 2>&1 >/dev/null &) ; done
#watch -n 5 "echo "Press Ctrl-c to exit when your server count meets the desired amount. You will need to copy and paste just the following instructions to proceed." && hcloud server list | grep 'running' | awk 'END {print NR}'"
hcloud server create --name docker1 --type cx11 --image debian-9 --datacenter nbg1-dc3 --network aftnet --ssh-key andy
hcloud server create --name docker2 --type cx11 --image debian-9 --datacenter nbg1-dc3 --network aftnet --ssh-key andy
hcloud server create --name docker3 --type cx11 --image debian-9 --datacenter nbg1-dc3 --network aftnet --ssh-key andy
rm *.txt
hcloud server list -o columns=name -o noheader > worker-names.txt
hcloud server list -o columns=ipv4 -o noheader > worker-ips.txt
cat worker-names.txt | xargs -I {} hcloud server describe -o json {} | jq -r '.private_net[0].ip' >> worker-vips.txt
hcloud server create --name manager1 --type cx11 --image debian-9 --datacenter nbg1-dc3 --network aftnet --ssh-key andy
hcloud server describe -o json manager1 | jq -r '.private_net[0].ip' > manager-vip.txt
scp -o StrictHostKeyChecking=no *.txt root@$(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}'):~/
scp -o StrictHostKeyChecking=no ansible/* root@$(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}'):~/
scp -o StrictHostKeyChecking=no *-docker-compose.yml root@$(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}'):~/
If it stuffs up run DANGEROUS it will delete all your servers for the project:
hcloud server list -o columns=name -o noheader | xargs -P 8 -I {} hcloud server delete {}
Get on the manager1 node:
#only required on a mac
eval `ssh-agent` && ssh-add ~/.ssh/id_rsa
#now login to manager1
ssh -l root -A $(hcloud server list -o columns=ipv4,name -o noheader | grep manager1 | awk '{print $1}')
Then run:
apt-get update && \
apt-get upgrade -y && \
apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common -y && \
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add - && \
apt-key fingerprint 0EBFCD88 && \
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" && \
apt-get update && \
apt-get install docker-ce docker-ce-cli containerd.io ansible -y && \
docker swarm init --advertise-addr=ens10 && \
docker swarm join-token worker | xargs | sed -r 's/^.*(docker.*).*$/\1/' > join.sh && \
chmod +x join.sh && \
printf "\n[defaults]\nhost_key_checking = False\n" >> /etc/ansible/ansible.cfg && \
printf "\n[managers]\n" >> /etc/ansible/hosts && \
cat manager-vip.txt >> /etc/ansible/hosts && \
printf "\n[dockers]\n" >> /etc/ansible/hosts && \
cat manager-vip.txt >> /etc/ansible/hosts && \
cat worker-vips.txt >> /etc/ansible/hosts && \
printf "\n[workers]\n" >> /etc/ansible/hosts && \
cat worker-vips.txt >> /etc/ansible/hosts && \
ansible dockers -a "uptime" && \
printf "\n $(cat join.sh | awk '{print $0}')" >> swarm-init.yml && \
ansible-playbook swarm-init.yml && \
ansible dockers -a "docker stats --no-stream" && \
docker node ls
#It's required to run roo on a manager node to get the automatic updates from the docker-compose files. You don't need to serve content from it though.
docker node update --label-add load_balancer=true manager1 && \
docker node update --label-add load_balancer=true docker1 && \
docker node update --label-add load_balancer=true docker2 && \
docker node update --label-add load_balancer=true docker3 && \
docker network create -d overlay --attachable forenet --subnet 192.168.9.0/24 && \
docker stack deploy -c roo-docker-compose.yml roo
You can watch roo boot & status using docker service logs roo_roo -f
otherwise wait about a minute for the services & raft-log to auto-start.
test.sfpl.io
with the test-domain, (make sure to set it up in your host records and use a load-balancer!).Then run:
docker stack deploy -c test-docker-compose.yml test
So long as you have your DNS setup, this will do:
Go to your domain. All Done
docker run -it --net=forenet alpine ash
apk add curl
curl -X GET http://tasks.roo_roo:6299/roo/v1/kvs
docker service logs roo_roo -f
docker service ps roo_roo
ansible workers -a "ip addr"
Run this all in the docker swarm manager look here to get started.
docker network create -d overlay --attachable forenet --subnet 192.168.9.0/24
docker node update --label-add load_balancer=true docker1-prod
ROO_ACME_STAGING=true
if you plan to muck about):# docker stack deploy -c roo-docker-compose.yml roo
version: "3.7"
services:
test:
image: test
deploy:
replicas: 1
networks:
- forenet
labels:
OriginHosts: test.sfpl.io
DestinationHost: test_test
DestinationScheme: http
DestinationPort: 80
networks:
forenet:
external: true
You need to tell roo what the incoming hostname etc is and where to route it to in the docker-compose file (if you want to go fully automoatic)
Please be extremely careful, letsencrypt/acme have weekly limits (which are super low) that will ban you from getting certificates for a week. Please use the staging setting in the roo-docker-compose-file.
https://letsencrypt.org/docs/rate-limits/
Roo comes with a clustered Distributed Key-Value (KV) Store (See the API below for access). You can use this to manually configure roo. To add the route and do the zeroconfig example above manually, do this instead:
docker stack deploy -c test-docker-compose.yml test
curl -X PUT roo_roo:6299/roo/v1/kv/com.roo.host:test.sfpl.io:https -d 'http://test_test:80'
make
in the root directory (not the roo directory).com.roo.host:<requesting_host<:port (optional)>:scheme> <destination_url>
curl -X PUT -d'test data' http://localhost:6299/roo/v1/kv/test
Put in a test route. For our example we are serving goole.com at our public endpoint, and routing to a swarm service with an endpoint inside our network on port 9001. The swarm stack is cool, the swarm service name is game. In docker swarm world this equates to a mesh network load balanced DNS record of tasks.cool_game. (you'll need to find your stack, service, and replace that and port 9001 with your details to get it working). The overall result to add this to our routing tables is:
curl -X PUT -d'http://tasks.cool_game:9001' http://localhost:6299/roo/v1/kv/com.roo.host:google.com:https
To test you can run something like this (this just makes your localhost pretend like its responding to a request to google.com):
curl -H "Host: google.com" https://localhost/
So to summarize, google.com:443 is the incoming route to roo from the internet, and tasks.cool_game:9001 is your service and port to your internal service (in this case its an internal intranet docker swarm service).
curl -X GET http://localhost:6299/roo/v1/kv/test
curl -X GET http://localhost:6299/roo/v1/kvs/te #Searches the prefix _te_
curl -X GET http://localhost:6299/roo/v1/kvs/tes #Searches the prefix _tes_
curl -X GET http://localhost:6299/roo/v1/kvs/test #Searches the prefix _test_
curl -X GET http://localhost:6299/roo/v1/kvs #Gets everything in the _entire_ kv store (filter is on nothing)
window.atob("dGVz......dCBkYXRh")
in javascript. Use json.Unmarshall or string([]byte) in Golang Go if you want a string... OR just use GET instead.curl -X PUT -i http://localhost:6299/roo/v1/perm --data '[{
"Entity" : { "Val" : "dioptre", "Rune" : 117},
"Context" : { "Val" : "/folder1", "Rune" : 102},
"Right" : [4],
"Delete" : false
},{
"Entity" : { "Val" : "marketing", "Rune" : 103},
"Context" : { "Val" : "/folder1/folder2", "Rune" : 102},
"Action" : { "Val" : "push_pr", "Rune" : 114},
"Delete" : false
},{
"Entity" : { "Val" : "admins", "Rune" : 103},
"Context" : { "Val" : "/folder1/folder2", "Rune" : 102},
"Right" : [4],
"Delete" : false
}]'
curl -X POST -i http://localhost:6299/roo/v1/perm --data '{
"User" : { "Val" : "dioptreX", "Rune" : 117},
"Entities" : [ { "Val" : "admins", "Rune" : 103}, { "Val" : "marketing", "Rune" : 103}],
"Resource" : { "Val" : "/folder1/folder2/wb1", "Rune" : 102},
"Right" : [4]
}
'
Another using Action:
curl -X POST -i http://localhost:6299/roo/v1/perm --data '{
"User" : { "Val" : "dioptreX", "Rune" : 117},
"Entities" : [ { "Val" : "marketing", "Rune" : 103}],
"Resource" : { "Val" : "/folder1/folder2/wb1", "Rune" : 102},
"Action" : { "Val" : "push_pr", "Rune": 114 }
}'
curl -X GET http://localhost:6299/roo/v1/kvs/
and finding com.roo.cache:your.domain.com
. If it exists it's not this.ansible docker-hosts -a "docker stats --no-stream"
ansible docker -a "bash -c 'ps waux'" | grep -E "(rood)"
This will get a realtime snapshot on all your machines in your swarm.
NOTE: This might be removed in a future release, but it's super useful now
Install golang on a computer in your network and get profiler insights (see more https://golang.org/pkg/runtime/pprof/):
On swarm run go tool pprof http://roo_roo_host_from_internal_forenet_network:6299/debug/pprof/heap
then type svg
(or output to something you may like better).
On my local development machine I do:
go tool pprof http://localhost:6299/debug/pprof/heap?debug=1
then type:
(pprof) svg
Or for intermittent results
Run from source directory (./roo):
mkdir pprof
cd pprof
curl http://localhost:6299/debug/pprof/heap > heap.0.pprof
sleep 30
curl http://localhost:6299/debug/pprof/heap > heap.1.pprof
sleep 30
curl http://localhost:6299/debug/pprof/heap > heap.2.pprof
sleep 30
curl http://localhost:6299/debug/pprof/heap > heap.3.pprof
Then:
go tool pprof pprof/heap.3.pprof
(pprof) top20
(pprof) list NewWriterSize
Or for manual results
Go to http://roo_roo_host_from_internal_forenet_network:6299/debug/pprof/
The docker image contains apt install valgrind
Run:
#valgrind --tool=memcheck --vgdb=yes --leak-check=yes --track-origins=yes --progress-interval=600 /app/roo/rood /app/roo/roo/config.json
Then run gdb (you may need to apt install gdb
on swarm node first):
#gdb /app/roo/rood
(gdb) target remote | vgdb
(gdb) set logging on
(gdb) set logging redirect on
(gdb) set logging overwrite on
(gdb) set logging debugredirect on
(gdb) show logging
(gdb) monitor leak_check full reachable any
Or alternatively pipe it to a gdb.txt logfile:
gdb -c /app/roo/rood -ex "target remote | vgdb" -ex "set logging on" -ex "set logging redirect on" -ex "set logging debugredirect on" -ex "set logging overwrite on" -ex "monitor leak_check full reachable any" -ex detach -ex quit 0 2>&1 >/dev/null
docker service logs roo_roo -f
docker run -it --net=forenet alpine ash
nslookup tasks.roo_roo.
curl -X GET http://<result_of_nslookup>:6299/roo/v1/kvs
ifconfig lo0 alias 127.0.0.2
Roo is in use commercially and in production. If you want support, email us at [email protected].