Scalable replication and change listening for CouchDB
Scalable replication and change listening for CouchDB
Spiegel was designed to provide scalable replication and change listening for Quizster, a photo-based feedback and submission system. Without Spiegel, a lot of complicated logic would need to exist in the Quizster application layer.
Like videos? Watch a 5 Minute Talk on Spiegel.
We recommend that you install Spiegel via Docker, especially Docker Swarm, as this will allow you to easily scale up or down as your needs change. Moreover, Docker will take care of automatically restarting the processes in the event of a permanent error. You can of course run Spiegel via npm, but then the scaling and auto restarting will be up to you to implement.
{
"yourapi.com": {
"apiuser": "apipassword"
}
}
{
"yourcouchdb.com": {
"user": "password"
}
}
$ docker run -it \
-e TYPE='install' \
-e URL='http://user:[email protected]:5984' \
redgeoff/spiegel
Note: the system database _global_changes
must exist. If it doesn't exist, create it.$ docker service create \
--name update-listener \
--detach=true \
--replicas 2 \
-e TYPE='update-listener' \
-e URL='http://user:[email protected]:5984' \
redgeoff/spiegel
$ docker service create \
--name change-listener \
--detach=true \
--replicas 2 \
-e TYPE='change-listener' \
-e URL='http://user:[email protected]:5984' \
--mount type=bind,source=change-listener-passwords.json,destination=/usr/src/app/passwords.json \
-e PASSWORDS_FILE=/usr/src/app/passwords.json \
redgeoff/spiegel
$ docker service create \
--name replicator \
--detach=true \
--replicas 2 \
-e TYPE='replicator' \
-e URL='http://user:[email protected]:5984' \
--mount type=bind,source=replicator-passwords.json,destination=/usr/src/app/passwords.json \
-e PASSWORDS_FILE=/usr/src/app/passwords.json \
redgeoff/spiegel
on_change
docs in the spiegel
DBreplicator
docs in the spiegel
DBNotes:
yourapi.com
and yourcouchdb.com
cannot be equal to localhost
as you will be running the Spiegel processes from within docker containers, which means that localhost will refer to the VM's localhost. If you don't have a domain name to use then you can use the IP address of your host machine. Alternatively, you can also use defined entries in your host's hosts file if you use the --network=host
when running docker service create
.$ docker service scale --detach=true update-listener=3
$ docker service scale --detach=true change-listener=3
$ docker service scale --detach=true replicator=3
$ docker pull redgeoff/spiegel
$ docker service update --detach=true --image redgeoff/spiegel update-listener
$ docker service update --detach=true --image redgeoff/spiegel change-listener
$ docker service update --detach=true --image redgeoff/spiegel replicator
$ npm install -g spiegel --unsafe
Note: --unsafe
is needed to properly install leveldown (a PouchDB dependency){
"yourapi.com": {
"apiuser": "apipassword"
}
}
{
"yourcouchdb.com": {
"user": "password"
}
}
$ spiegel \
--type='install' \
--url='http://user:[email protected]:5984'
Note: the system database _global_changes
must exist. If it doesn't exist, create it.$ spiegel \
--type='update-listener' \
--url='http://user:[email protected]:5984'
$ spiegel \
--type='change-listener' \
--url='http://user:[email protected]:5984' \
--passwords-file=change-listener-passwords.json
$ spiegel \
--type='replicator' \
--url='http://user:[email protected]:5984' \
--passwords-file=replicator-passwords.json
on_change
docs in the spiegel
DBreplicator
docs in the spiegel
DBUsage: spiegel --type=type --url=couchdb-url options
--type=type The type of the process: update-listener, change-listener, replicator,
install or uninstall. The install and uninstall processes run and then
exit, but the update-listener, change-listener and replicator will run
indefinitely. You can run as many update-listeners, change-listeners and
replicators as your DB setup can handle. In most cases you'll want to run
at least 2 of each of these processes for redundancy. In general, if you
need to listen to more changes or respond to these changes faster, add a
change-listener. Similarly, if you need to perform more replications or
replicate faster, add a replicator.
install Creates the spiegel DB and sieve
uninstall Destroys the spiegel DB and sieve
update-listener The update-listener listens for all DB updates and then
schedules on_changes and replications accordingly.
change-listener The change-listener runs on_change rules for all
matching changes
replicator The replicator performs replications
--url=couchdb-url The URL to the CouchDB instance
For a complete list of options see Usage
Notes:
docker run -it -e VERSION=true redgeoff/spiegel
Spiegel uses bunyan for logging and by default outputs entries in JSON, something that is useful when digesting log data programmatically, but not very readable for a human. Instead, you can pipe the output to bunyan for human-readable logs.
Example with Docker:
$ sudo npm install -g bunyan
$ docker ps # to view list of containers
$ docker logs -f <container> | bunyan
Example with NPM:
$ sudo npm install -g bunyan
$ spiegel --url=http://admin:admin@localhost:5984 --type=update-listener --log-level=debug | bunyan