A python script that will parse the nginx access.log and send geolocation metrics and log metrics to InfluxDB
A python script that will parse the nginx access.log and send geolocation metrics and log metrics to InfluxDB
These are the default values for all envs. Add the ones that differ on your system.
Environment Variable | Example Value | Description |
---|---|---|
NGINX_LOG_PATH | /config/log/nginx/access.log | Container path for Nginx logfile , defaults to the example. |
INFLUX_HOST | localhost | Host running InfluxDB. |
INFLUX_HOST_PORT | 8086 | Optional, defaults to 8086. |
INFLUX_DATABASE | geoip2influx | Optional, defaults to geoip2influx. |
INFLUX_USER | root | Optional, defaults to root. |
INFLUX_PASS | root | Optional, defaults to root. |
GEO_MEASUREMENT | geoip2influx | InfluxDB measurement name for geohashes. Optional, defaults to the example. |
LOG_MEASUREMENT | nginx_access_logs | InfluxDB measurement name for nginx logs. Optional, defaults to the example. |
SEND_NGINX_LOGS | true | Set to false to disable nginx logs. Optional, defaults to true . |
GEOIP2INFLUX_LOG_LEVEL | info | Sets the log level in geoip2influx.log. Use debug for verbose logging Optional, defaults to info. |
GEOIP2INFLUX_LOG_PATH | /config/log/geoip2influx/geoip2influx.log | Optional. Defaults to example. |
INFLUX_RETENTION | 7d | Sets the retention for the database. Optional, defaults to example. |
INFLUX_SHARD | 1d | Set the shard for the database. Optional, defaults to example. |
MAXMINDDB_LICENSE_KEY | xxxxxxx | Add your Maxmind licence key |
Default download location is /config/geoip2db/GeoLite2-City.mmdb
Get your licence key here: https://www.maxmind.com/en/geolite2/signup
The InfluxDB database will be created automatically with the name you choose.
-e INFLUX_DATABASE=geoip2influx
docker create \
--name=geoip2influx \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Europe/Oslo \
-e INFLUX_HOST=<influxdb host> \
-e INFLUX_HOST_PORT=<influxdb port> \
-e MAXMINDDB_LICENSE_KEY=<license key>\
-v /path/to/appdata/geoip2influx:/config \
-v /path/to/nginx/accesslog/:/config/log/nginx/ \
--restart unless-stopped \
ghcr.io/gilbn/geoip2influx
version: "2.1"
services:
geoip2influx:
image: ghcr.io/gilbn/geoip2influx
container_name: geoip2influx
environment:
- PUID=1000
- PGID=1000
- TZ=Europe/Oslo
- INFLUX_HOST=<influxdb host>
- INFLUX_HOST_PORT=<influxdb port>
- MAXMINDDB_LICENSE_KEY=<license key>
volumes:
- /path/to/appdata/geoip2influx:/config
- /path/to/nginx/accesslog/:/config/log/nginx/
restart: unless-stopped
Needs the grafana-worldmap-panel
Nginx needs to be compiled with the geoip2 module: https://github.com/leev/ngx_http_geoip2_module
nginx.conf
file:geoip2 /config/geoip2db/GeoLite2-City.mmdb {
auto_reload 5m;
$geoip2_data_country_iso_code country iso_code;
$geoip2_data_city_name city names en;
}
log_format custom '$remote_addr - $remote_user [$time_local]'
'"$request" $status $body_bytes_sent'
'"$http_referer" $host "$http_user_agent"'
'"$request_time" "$upstream_connect_time"'
'"$geoip2_data_city_name" "$geoip2_data_country_iso_code"';
custom
log format.access_log /config/log/nginx/access.log custom;
If you separate your nginx log files but want this script to parse all of them you can do the following:
As nginx can have multiple access log
directives in a block, just add another one in the server block.
Example
access_log /config/log/nginx/technicalramblings/access.log custom;
access_log /config/log/nginx/access.log custom;
This will log the same lines to both files.
Then use the /config/log/nginx/access.log
file in the NGINX_LOG_PATH
variable.
21.06.20 - Added $host(domain) to the nginx log metrics. This will break your nginx logs parsing, as you need to update the custom log format.
06.06.20 - Added influx retention policy to try and mitigate max-values-per-tag limit exceeded errors.
-e INFLUX_RETENTION
Default 30d-e INFLUX_SHARD
Default 2d30.05.20 - Added logging. Use -e GEOIP2INFLUX_LOG_LEVEL
to set the log level.
15.05.20 - Removed GEOIP2_KEY
and GEOIP_DB_PATH
variables. With commit https://github.com/linuxserver/docker-letsencrypt/commit/75b9685fdb3ec6edda590300f289b0e75dd9efd0 the letsencrypt container now natively supports downloading and updating(weekly) the GeoLite2-City database!
Adapted source: https://github.com/ratibor78/geostat