Making CoreOS' Clair easily work in CI/CD pipelines
Clair, released by CoreOS in Nov '16, is a very effective tool for statically analyzing docker images to determine which known vulnerabilities exist in the images. Integrating Clair into a CI/CD pipeline:
This repo was created to address the above challenges.
The roots of this repo center around the following beliefs:
clair_cicd
to influence Docker image risk assessment decisionsTo start using clair-cicd
,
a service engineer inserts a single line of code into a service's CI pipeline.
The single line of code runs the shell script assess-image-risk.sh.
Part of the CI pipeline's responsibility is to build the docker image
and then push that docker image to a docker registry.
The single line of clair-cicd
code should appear after the docker image
is built and tested but before the docker image is pushed to a docker registry.
In this simple case, assess-image-risk.sh returns a zero exit status if the docker image contains no known vulnerabilities above a medium severity. If the docker image contains any known vulnerabilities with a severity higher than medium, assess-image-risk.sh returns a non-zero exit status and the build fails ie. the build should fail before the docker image is pushed to a docker registry.
The example illustrates what's described about for the alpine:3.4 docker image.
~> curl -s -L \
https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | \
bash -s -- alpine:3.4
~> echo $?
0
~>
To understand how assess-image-risk.sh
is making its risk assessment
decision try using the -v
flag.
~> curl -s -L \
https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | \
bash -s -- -v alpine:3.4
2020-01-12 16:43:35 pulling clair database image 'simonsdave/clair-cicd-database:latest'
2020-01-12 16:44:17 successfully pulled clair database image
2020-01-12 16:44:17 starting clair database container 'clair-db-c1dbb5f93ae98755'
2020-01-12 16:44:23 waiting for database server in container 'clair-db-c1dbb5f93ae98755' to start ...........................
2020-01-12 16:44:54 successfully started clair database container
2020-01-12 16:44:54 clair configuration in '/var/folders/7x/rr443kj575s8zz54jrbrp4jc0000gn/T/tmp.ElAlhGNl'
2020-01-12 16:44:59 pulling clair image 'simonsdave/clair-cicd-clair:latest'
2020-01-12 16:45:13 successfully pulled clair image 'simonsdave/clair-cicd-clair:latest'
2020-01-12 16:45:13 starting clair container 'clair-e9573ae537134fa0'
2020-01-12 16:45:15 successfully started clair container 'clair-e9573ae537134fa0'
2020-01-12 16:45:15 saving docker image 'alpine:3.4' to '/tmp/tmp.IaNHCH'
2020-01-12 16:45:16 successfully saved docker image 'alpine:3.4'
2020-01-12 16:45:16 starting to create clair layers
2020-01-12 16:45:16 creating clair layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:45:16 successfully created clair layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:45:16 done creating clair layers
2020-01-12 16:45:16 starting to get vulnerabilities for clair layers
2020-01-12 16:45:16 saving vulnerabilities to directory '/tmp/tmp.MDncHN'
2020-01-12 16:45:16 getting vulnerabilities for layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:45:16 successfully got vulnerabilities for layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:45:16 done getting vulnerabilities for clair layers
2020-01-12 21:45:17 INFO io:89 Looking for vulnerabilities in directory '/tmp/tmp.MDncHN'
2020-01-12 21:45:17 INFO io:95 Found 1 files with vulnerabilities in directory '/tmp/tmp.MDncHN'
2020-01-12 21:45:17 INFO io:104 Looking for vulnerabilities in '/tmp/tmp.MDncHN/378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d.json'
2020-01-12 21:45:17 INFO io:122 Found 0 vulnerabilities in '/tmp/tmp.MDncHN/378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d.json'
2020-01-12 21:45:17 INFO io:133 Found 0 vulnerabilities in 1 files in directory '/tmp/tmp.MDncHN'
2020-01-12 21:45:17 INFO assessor:19 Assessment starts
2020-01-12 21:45:17 INFO assessor:26 Assessment ends - pass
~> echo $?
0
~>
In the above examples a default vulnerability whitelist was used. When specified as a JSON doc, this whitelist would be:
{
"ignoreSevertiesAtOrBelow": "medium"
}
By default, assess-image-risk.sh returns a non-zero exit status if any vulnerabilities are identified in the image with a severity higher than medium. The medium is derived from the default vulnerability whitelist.
The example below illustrate how to specify a vulnerability whitelist
and with a severity other than medium. Note the use of the json://
prefix to indicate this is an inline whitelist.
~> curl -s -L \
https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | \
bash -s -- -v --whitelist 'json://{"ignoreSevertiesAtOrBelow": "negligible"}' ubuntu:18.04
2020-01-12 16:46:56 pulling clair database image 'simonsdave/clair-cicd-database:latest'
2020-01-12 16:46:58 successfully pulled clair database image
2020-01-12 16:46:58 starting clair database container 'clair-db-3b0811925f7e8bc2'
2020-01-12 16:46:59 waiting for database server in container 'clair-db-3b0811925f7e8bc2' to start .............................
2020-01-12 16:47:32 successfully started clair database container
2020-01-12 16:47:32 clair configuration in '/var/folders/7x/rr443kj575s8zz54jrbrp4jc0000gn/T/tmp.BXCs3Giy'
2020-01-12 16:47:34 pulling clair image 'simonsdave/clair-cicd-clair:latest'
2020-01-12 16:47:36 successfully pulled clair image 'simonsdave/clair-cicd-clair:latest'
2020-01-12 16:47:36 starting clair container 'clair-fc579c71e7daba57'
2020-01-12 16:47:38 successfully started clair container 'clair-fc579c71e7daba57'
2020-01-12 16:47:38 saving docker image 'ubuntu:18.04' to '/tmp/tmp.lPDhNd'
2020-01-12 16:47:43 successfully saved docker image 'ubuntu:18.04'
2020-01-12 16:47:43 starting to create clair layers
2020-01-12 16:47:43 creating clair layer 'cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd'
2020-01-12 16:47:43 successfully created clair layer 'cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd'
2020-01-12 16:47:43 creating clair layer '27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54'
2020-01-12 16:47:44 successfully created clair layer '27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54'
2020-01-12 16:47:44 creating clair layer 'd80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e'
2020-01-12 16:47:44 successfully created clair layer 'd80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e'
2020-01-12 16:47:44 creating clair layer '1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d'
2020-01-12 16:47:44 successfully created clair layer '1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d'
2020-01-12 16:47:44 done creating clair layers
2020-01-12 16:47:44 starting to get vulnerabilities for clair layers
2020-01-12 16:47:44 saving vulnerabilities to directory '/tmp/tmp.dkfgmI'
2020-01-12 16:47:44 getting vulnerabilities for layer 'cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd'
2020-01-12 16:47:44 successfully got vulnerabilities for layer 'cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd'
2020-01-12 16:47:44 getting vulnerabilities for layer '27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54'
2020-01-12 16:47:44 successfully got vulnerabilities for layer '27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54'
2020-01-12 16:47:44 getting vulnerabilities for layer 'd80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e'
2020-01-12 16:47:44 successfully got vulnerabilities for layer 'd80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e'
2020-01-12 16:47:44 getting vulnerabilities for layer '1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d'
2020-01-12 16:47:44 successfully got vulnerabilities for layer '1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d'
2020-01-12 16:47:44 done getting vulnerabilities for clair layers
2020-01-12 21:47:45 INFO io:89 Looking for vulnerabilities in directory '/tmp/tmp.dkfgmI'
2020-01-12 21:47:45 INFO io:95 Found 4 files with vulnerabilities in directory '/tmp/tmp.dkfgmI'
2020-01-12 21:47:45 INFO io:104 Looking for vulnerabilities in '/tmp/tmp.dkfgmI/27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54.json'
2020-01-12 21:47:45 INFO io:122 Found 33 vulnerabilities in '/tmp/tmp.dkfgmI/27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54.json'
2020-01-12 21:47:45 INFO io:104 Looking for vulnerabilities in '/tmp/tmp.dkfgmI/cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd.json'
2020-01-12 21:47:45 INFO io:122 Found 33 vulnerabilities in '/tmp/tmp.dkfgmI/cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd.json'
2020-01-12 21:47:45 INFO io:104 Looking for vulnerabilities in '/tmp/tmp.dkfgmI/1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d.json'
2020-01-12 21:47:45 INFO io:122 Found 33 vulnerabilities in '/tmp/tmp.dkfgmI/1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d.json'
2020-01-12 21:47:45 INFO io:104 Looking for vulnerabilities in '/tmp/tmp.dkfgmI/d80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e.json'
2020-01-12 21:47:45 INFO io:122 Found 33 vulnerabilities in '/tmp/tmp.dkfgmI/d80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e.json'
2020-01-12 21:47:45 INFO io:133 Found 33 vulnerabilities in 4 files in directory '/tmp/tmp.dkfgmI'
2020-01-12 21:47:45 INFO assessor:19 Assessment starts
2020-01-12 21:47:45 INFO assessor:34 Assessing vulnerability CVE-2018-11236 - start
2020-01-12 21:47:45 INFO assessor:52 Vulnerability CVE-2018-11236 @ severity medium greater than whitelist severity @ negligible - fail
2020-01-12 21:47:45 INFO assessor:36 Assessing vulnerability CVE-2018-11236 - finish
2020-01-12 21:47:45 INFO assessor:23 Assessment ends - fail
~> echo $?
1
~>
The above is an example of an inline whitelist. It's also possible
to specify a whitelist in a file.
The example below illustrates the usage.
Note use of the file://
prefix to indicate the whitelist is contained in a file.
~> cat whitelist.json
{
"ignoreSevertiesAtOrBelow": "medium"
}
~> curl -s -L \
https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | \
bash -s -- -v --whitelist file://whitelist.json alpine:3.4
2020-01-12 16:48:41 pulling clair database image 'simonsdave/clair-cicd-database:latest'
2020-01-12 16:48:42 successfully pulled clair database image
2020-01-12 16:48:42 starting clair database container 'clair-db-191152e37b864e4b'
2020-01-12 16:48:43 waiting for database server in container 'clair-db-191152e37b864e4b' to start .............................
2020-01-12 16:49:16 successfully started clair database container
2020-01-12 16:49:16 clair configuration in '/var/folders/7x/rr443kj575s8zz54jrbrp4jc0000gn/T/tmp.GdlBNmiG'
2020-01-12 16:49:19 pulling clair image 'simonsdave/clair-cicd-clair:latest'
2020-01-12 16:49:20 successfully pulled clair image 'simonsdave/clair-cicd-clair:latest'
2020-01-12 16:49:20 starting clair container 'clair-747d1c50606fba7e'
2020-01-12 16:49:21 successfully started clair container 'clair-747d1c50606fba7e'
2020-01-12 16:49:22 saving docker image 'alpine:3.4' to '/tmp/tmp.Eldkbe'
2020-01-12 16:49:23 successfully saved docker image 'alpine:3.4'
2020-01-12 16:49:23 starting to create clair layers
2020-01-12 16:49:23 creating clair layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:49:23 successfully created clair layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:49:23 done creating clair layers
2020-01-12 16:49:23 starting to get vulnerabilities for clair layers
2020-01-12 16:49:23 saving vulnerabilities to directory '/tmp/tmp.pCOhlL'
2020-01-12 16:49:23 getting vulnerabilities for layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:49:23 successfully got vulnerabilities for layer '378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'
2020-01-12 16:49:23 done getting vulnerabilities for clair layers
2020-01-12 21:49:23 INFO io:89 Looking for vulnerabilities in directory '/tmp/tmp.pCOhlL'
2020-01-12 21:49:23 INFO io:95 Found 1 files with vulnerabilities in directory '/tmp/tmp.pCOhlL'
2020-01-12 21:49:23 INFO io:104 Looking for vulnerabilities in '/tmp/tmp.pCOhlL/378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d.json'
2020-01-12 21:49:23 INFO io:122 Found 0 vulnerabilities in '/tmp/tmp.pCOhlL/378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d.json'
2020-01-12 21:49:23 INFO io:133 Found 0 vulnerabilities in 1 files in directory '/tmp/tmp.pCOhlL'
2020-01-12 21:49:23 INFO assessor:19 Assessment starts
2020-01-12 21:49:23 INFO assessor:26 Assessment ends - pass
~> echo $?
0
~>
Specific vulnerabilities can also be whitelisted.
The example below illustrates this capability.
If you add the -v
(verbose) flag to assess-image-risk.sh
you see exactly how whitelisted vulnerabilities impact the risk assessment
with statements like Vulnerability CVE-2019-13627 in whitelist - pass
~> curl -s -L \
https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | \
bash -s -- --whitelist 'json://{"ignoreSevertiesAtOrBelow":"low"}' ubuntu:18.04
~> echo $?
1
~> cat whitelist.json
{
"ignoreSevertiesAtOrBelow": "low",
"vulnerabilities": [
{ "cveId": "CVE-2018-20839", "rationale": "reason #1" },
{ "cveId": "CVE-2019-5188", "rationale": "reason #2" },
{ "cveId": "CVE-2018-11236", "rationale": "reason #3" },
{ "cveId": "CVE-2019-13627", "rationale": "reason #4" },
{ "cveId": "CVE-2019-13050", "rationale": "reason #5" },
{ "cveId": "CVE-2018-11237", "rationale": "reason #6" },
{ "cveId": "CVE-2018-19591", "rationale": "reason #7" }
]
}
~> curl -s -L \
https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh | \
bash -s -- --whitelist 'file://whitelist.json' ubuntu:18.04
~> echo $?
0
~>
ITO whitelists it's worth seeing issue #7 which says:
When specifying the whitelist for
assess-image-risk.sh
with the--whitelist
command line argument, should supporthttps://
scheme in addition to the existingjson://
,file://
schemes. Why is this important? Ideally whitelists should be maintained by a security analyst not a service engineer. This means that whitelists should be maintained in another repo with appropriate change management process. Thejson://
andfile://
schemes are fine for maintaining whitelists in the same repo is service code. However, it would be better to maintain whitelists in a repo that is readonly for service engineers and editable only by security analysts who could apply appropriate change management processes are used to make changes (code reviews, feature branches, etc).
There are 3 moving pieces:
From the samples at the start of this doc you'll see the approach of curl'ing the latest release of assess-image-risk.sh into a localy run bash shell. assess-image-risk.sh then spins up a container using the simonsdave/clair-database. Another container is then run using simonsdave/clair-cicd-clair with the simonsdave/clair-cicd-clair container being able to talk with the simonsdave/clair-database container. Once the simonsdave/clair-cicd-clair container is running, assess-image-risk.sh docker exec's this bash script which does the actual risk assessment.
Armed with the understanding of how clair-cicd
works you'll
appreciate that the ability to execute assess-image-risk.sh
is what defines the requirements for the execution
environment. assess-image-risk.sh is a bash script used to launch
the risk assessment process and as such it's this script which defines the bulk of
the assumptions/requirements for clair-cicd
- the script uses docker, sed and openssl
so all these need to be available in the environment running clair-cicd