An example with nix: How to automatically and reproducibly build and integration-test a database C++ app in 60 different build-configurations (library exchange/compiler exchange/static-nonstatic)
This project consists of 2 simple example applications that model a writer and a reader to/from a postgres database:
The "server" application (C++) listens on TCP socket 1300 and writes into the database whatever it receives. The "webserver" application (Python) is either a script that just prints the database content, or a webserver script (both included) that listens on port 8000 and lists the latest 10 database "messages".
Both applications are relatively simple and stupid as they are mere demonstration vehicles for showing:
nix
nix
nix
docker
images
(so minimal that they contain no base distro image) using nix
nix
At compile time, the applications need the following:
The applications can be compiled (the C++ app) and both be run without nix at
all.
But in that case you have to install/configure all dependencies yourself -
with nix
, everything will be handled automagically.
In order to build the server app, do the following:
$ cd $project/server
$ nix-build
# You may now run ./result/bin/messagedb-server
In order to package the python app into a wrapper script that can be called like any binary, do the following:
$ cd $project/python_client
$ nix-build
# You may now run ./result/bin/mdb-webserver
The python wrapper calls python with the dependencies in place, leaving no way
to accidentally call it with the wrong python interpreter.
(This way of packaging a python app is completely optional, nix
does not force
you to do so)
nix
can be obtained here.
There is always the quick and unsafe curl | sh
way to install, but there are
also instructions on the project website that include binary verification.
Please note that tab completion in the following examples works out of the box on NixOS, while users of other distros have to install it manually. https://github.com/hedning/nix-bash-completions
The server app can be compiled in different configurations:
boost
versions 1.6.3, 1.6.4, 1.6.5, 1.6.6, 1.6.7, 1.6.8To select one configuration, run:
$ cd $project
$ nix-build release.nix -A <tab-complete>
integrationtest-mdb-server-clang7-boost166 integrationtest-mdb-server-static-clang8-boost169 mdb-server-gcc8-boost166
integrationtest-mdb-server-clang7-boost167 integrationtest-mdb-server-static-gcc7-boost166 mdb-server-gcc8-boost167
integrationtest-mdb-server-clang7-boost168 integrationtest-mdb-server-static-gcc7-boost167 mdb-server-gcc8-boost168
integrationtest-mdb-server-clang7-boost169 integrationtest-mdb-server-static-gcc7-boost168 mdb-server-gcc8-boost169
integrationtest-mdb-server-clang8-boost166 integrationtest-mdb-server-static-gcc7-boost169 mdb-server-static-clang7-boost166
integrationtest-mdb-server-clang8-boost167 integrationtest-mdb-server-static-gcc8-boost166 mdb-server-static-clang7-boost167
integrationtest-mdb-server-clang8-boost168 integrationtest-mdb-server-static-gcc8-boost167 mdb-server-static-clang7-boost168
integrationtest-mdb-server-clang8-boost169 integrationtest-mdb-server-static-gcc8-boost168 mdb-server-static-clang7-boost169
integrationtest-mdb-server-gcc7-boost166 integrationtest-mdb-server-static-gcc8-boost169 mdb-server-static-clang8-boost166
integrationtest-mdb-server-gcc7-boost167 mdb-server-clang7-boost166 mdb-server-static-clang8-boost167
integrationtest-mdb-server-gcc7-boost168 mdb-server-clang7-boost167 mdb-server-static-clang8-boost168
integrationtest-mdb-server-gcc7-boost169 mdb-server-clang7-boost168 mdb-server-static-clang8-boost169
integrationtest-mdb-server-gcc8-boost166 mdb-server-clang7-boost169 mdb-server-static-gcc7-boost166
integrationtest-mdb-server-gcc8-boost167 mdb-server-clang8-boost166 mdb-server-static-gcc7-boost167
integrationtest-mdb-server-gcc8-boost168 mdb-server-clang8-boost167 mdb-server-static-gcc7-boost168
integrationtest-mdb-server-gcc8-boost169 mdb-server-clang8-boost168 mdb-server-static-gcc7-boost169
integrationtest-mdb-server-static-clang7-boost166 mdb-server-clang8-boost169 mdb-server-static-gcc8-boost166
integrationtest-mdb-server-static-clang7-boost167 mdb-server-docker mdb-server-static-gcc8-boost167
integrationtest-mdb-server-static-clang7-boost168 mdb-server-docker-static mdb-server-static-gcc8-boost168
integrationtest-mdb-server-static-clang7-boost169 mdb-server-gcc7-boost166 mdb-server-static-gcc8-boost169
integrationtest-mdb-server-static-clang8-boost166 mdb-server-gcc7-boost167 mdb-webservice
integrationtest-mdb-server-static-clang8-boost167 mdb-server-gcc7-boost168 mdb-webservice-docker
integrationtest-mdb-server-static-clang8-boost168 mdb-server-gcc7-boost169
$ nix-build release.nix -A mdb-server-gcc7-boost166
# You may now run ./result/bin/messagedb-server
docker
images for both appsCompiling and installing the docker images is simple.
The step of the docker image creation does not even require you to have docker
on your system.
server:
$ cd $project
$ nix-build release.nix -A mdb-server-docker
$ du -sh $(readlink -f result)
46M /nix/store/bnd9gzdgmrvdxd7cdrm7fxr39db8phfk-docker-image-mdb-server.tar.gz
The docker image with the static version of the library is a bit smaller:
$ nix-build release.nix -A mdb-server-docker-static
$ du -sh $(readlink -f result)
22M /nix/store/58lpaf0yh6skjdxdd0qfc4vrw9nj1j76-docker-image-mdb-server.tar.gz
Please note that this can of course be further reduced, but then the example would go out of scope and we have a nice starting point already.
The webserver docker image:
$ nix-build release.nix -A mdb-webservice-docker
$ du -sh $(readlink -f result)
60M /nix/store/l4ipb3p325yaflsdna8h7nijla61vwrz-docker-image-mdb-webservice.tar.gz
What's particularly interesting about these docker images compared to conventionally created images is, that they only contain the run-time dependencies they need. This means that for the statically linked binary, the corresponding docker image, in addition to bash and its run-time dependencies, only contains the binary itself. Likewise, for the dynamically linked binary, and the python webserver, the docker images contain exactly the needed dependencies, e.g. Python 3.xyz for the webserver. In contrast, building a Docker image traditionally, e.g., with a Docker file built upon a Ubuntu/Debian/Fedora base image, would quickly result in an image size around hundreds or MB.
There is only one integration test in this example. It does the following
Running the test does the following:
This integration test scenario can easily be run with all different build configurations of the server app. The NixOS distribution tests many of its packages with such integration tests. These are however much more complex by running full herds of VMs in virtual networks, to some extent even with the X desktop activated and OCR etc. - out of scope for this example.
To run the integration tests, do:
$ nix-build release.nix -A integrationtest-mdb-server-<tab-complete>
integrationtest-mdb-server-clang7-boost166 integrationtest-mdb-server-gcc7-boost169 integrationtest-mdb-server-static-clang8-boost168
integrationtest-mdb-server-clang7-boost167 integrationtest-mdb-server-gcc8-boost166 integrationtest-mdb-server-static-clang8-boost169
integrationtest-mdb-server-clang7-boost168 integrationtest-mdb-server-gcc8-boost167 integrationtest-mdb-server-static-gcc7-boost166
integrationtest-mdb-server-clang7-boost169 integrationtest-mdb-server-gcc8-boost168 integrationtest-mdb-server-static-gcc7-boost167
integrationtest-mdb-server-clang8-boost166 integrationtest-mdb-server-gcc8-boost169 integrationtest-mdb-server-static-gcc7-boost168
integrationtest-mdb-server-clang8-boost167 integrationtest-mdb-server-static-clang7-boost166 integrationtest-mdb-server-static-gcc7-boost169
integrationtest-mdb-server-clang8-boost168 integrationtest-mdb-server-static-clang7-boost167 integrationtest-mdb-server-static-gcc8-boost166
integrationtest-mdb-server-clang8-boost169 integrationtest-mdb-server-static-clang7-boost168 integrationtest-mdb-server-static-gcc8-boost167
integrationtest-mdb-server-gcc7-boost166 integrationtest-mdb-server-static-clang7-boost169 integrationtest-mdb-server-static-gcc8-boost168
integrationtest-mdb-server-gcc7-boost167 integrationtest-mdb-server-static-clang8-boost166 integrationtest-mdb-server-static-gcc8-boost169
integrationtest-mdb-server-gcc7-boost168 integrationtest-mdb-server-static-clang8-boost167
The result of such integration tests is a HTML/XML log document in the result
folder.
NixOS comes with a service for continuous integration, called
hydra
.
It does not look as polished as other CIs and its integration is very github
centric (because nix
and NixOS
source code and package lists are hosted on
github), but it is generally extensible with plugins.
A screenshot of a private hydra instance building this project: