Swarmstack Versions Save

Lots of work went into this release:

• Updates to many of the stack components to recent stable (and tested) versions.

• kthxbye added to the stack, facilitates automatic renewal of alert silences (preface any silence comment with the letters ACK)

• New 'up' script helps swarm redeploy only services within the stack that have changed "configs:" files. For example, changes to Alertmanager configuration, Caddyfile and certificates files, Docker Garbage Collection configuration, Grafana provisioned dashboards, Karma configuration, and Prometheus rules and configuration files will be detected and the relevant swarm configs will be renamed. This will cause swarm to redeploy only those services with updated files, without needed to take the entire stack down and then back up.

• node_exporter is now installed on nodes in addition to Netdata, and new Node and Containers dashboards for Grafana are included which use node_exporter metrics rather than Netdata ones. This facilitates running swarmstack on self-managed clusters where Netdata is not desired.

• New ancillary projects, such as VictoriaMetrics, have been added to https://github.com/swarmstack, and are referenced within the main swarmstack README.md.

• Thanks again to https://github.com/jm66 for further fixes and contributions.

Breaking Changes:

• Updated compose files to version 3.5, which requires a minimum version of Docker 17.12.0

1.9 includes new similar node_exporter+dockerd+cadvisor Grafana dashboards in addition to the existing Netdata+dockerd+cadvisor ones. This gives users a choice between their data sources for monitoring server nodes and containers within their clusters. This is in preparation for the swarmstack 2.0 release, which will include the installation of node_exporter via systemd onto nodes in addition to or in lieu of installing Netdata on each node under systemd. Recommendation is run both exporters (default now).

-- original commit comments -- preserve alertmanager silences into /alertmanager volume mount rev caddy, grafana, prometheus default https://github.com/clockworksoul/docker-gc-cron to not automa… … enable netdata service after reboot rev Portworx storage to 2.1.4 some bit-rot had set into this dashboard, switched to log() view on m… … rev Grafana to 6.3.4 due to CVE-2019-15043 move to use grafana:latest tag as this follows their release versions… … Updated Nodes Grafana dashboards, both for netdata+dockerd+cadvisor a… … move to alertmanager:latest, updated node and container dashboards new node_exporter + dockerd + cadvisor Nodes dashboard update node and container dashboards with netdata or node_exporter me… … rename a grafana panel dashboard cleanup / sync cleaning up dashboards remove rebooted restriction when waiting for portworx storage on node… … increase reboot timeout wait from 2 minutes to 3 before timing out deprecate storage.tsdb.retention directive to prometheus

A small workaround was implemented in Caddyfile to fix accessing container consoles via Portainer's UI (Caddy prometheus module disabled during connections to Portainer). Also rev versions for:

Grafana 6.3.2 karma 0.43 Portainer 1.22.0 Portainer Agent 1.4.0

This release replaces the previous installation method downloading netdata sources and compiling on each host in favor of using the pre-build netdata RPM packages hosted on packagecloud.io

You will need to un-install any previous netdata that was installed from compiled source (as was with swarmstack < 1.8) by executing the following on each of your hosts before running the 1.8+ swarmstack.yml playbook, otherwise netdata may not behave properly:

wget https://raw.githubusercontent.com/netdata/netdata/master/packaging/installer/netdata-uninstaller.sh chmod +x ./netdata-uninstaller.sh ./netdata-uninstaller.sh --yes

Going forward, netdata can then be updated at will by re-running the swarmstack.yml playbook, or simply invoking 'yum update netdata' on your hosts.

Also, karma has been updated to 0.42 and prometheus/pushgateway to 0.91 to pick up latest fixes.

Rev Karma to 0.41

Cleanup of all Grafana dashboards to use default datasource. This facilitates moving between Prometheus and Trickster as the datasource for the graphs. This helps offload Prometheus so that it can go about the business of collecting metrics when you have popular dashboards pulling many metrics repetitively.

This release adds node draining to the docker playbook to give services a better chance to cleanly migrate to another host before docker or kernel updates are applied.

also:

Alertmanager 0.18.0 and Prometheus v2.11.1 version updates README added a section referencing Trickster (Prometheus caching) in the SCALING section Changed Docker swarm service discovery container image to us https://github.com/bborysenko/prometheus-swarm-discovery instead of https://github.com/jmendiara/prometheus-swarm-discovery

This release moves to swarmstack/caddy:no-stats-1.0.1 container build. Also mitigation for CVE-2019-11477 was added within the docker playbook, which otherwise performs sysctl kernel tuning to EL7 hosts.

By default caddy will now log in the "combined" format (includes referer and user-agent) to the container stdout. This can aid in debugging failed service connections, and is additionally useful when ingesting container logs into Loki.

rev karma to 0.39 rev grafana to 6.2.5 add firewall rule for CVE-2019-11479 to drop TCP packets with MSS <= 500 add additional build tools that may be required for EL7 to install NetData via swarmstack.yml playbook change alertmanager storage.path=/home as containers run as user nobody

Documentation has been added to integrating the new Grafana Loki data source with Docker for container stdout/stderr aggregation and search features using Grafana! See https://github.com/swarmstack/loki or the new Logging section of README.md for more information.

This release also updates to the following container images:

Grafana 6.2.1 karma v0.38 Portainer 1.21.0 (compatible with the new GA version of Storidge) Portainer-agent 1.3.0 Prometheus v2.10.0