The goal of this project is to implement a Spring Boot application, called movies-api, and use Filebeat & ELK Stack (Elasticsearch, Logstash and Kibana) to collect and visualize application's logs and Prometheus & Grafana to monitor application's metrics.
The goal of this project is to implement a Spring Boot
application, called movies-api
, and use Filebeat
& ELK Stack
(Elasticsearch
, Logstash
and Kibana
) to collect and visualize application's logs and Prometheus
& Grafana
to monitor application's metrics.
Note: In
kubernetes-minikube-environment
repository, it's shown how to deploy this project inKubernetes
(Minikube
)
On ivangfr.github.io, I have compiled my Proof-of-Concepts (PoCs) and articles. You can easily search for the technology you are interested in by using the filter. Who knows, perhaps I have already implemented a PoC or written an article about what you are looking for.
Spring Boot
Web Java application that exposes a REST API for managing movies. Its endpoints are displayed in the picture below.
Open a terminal and inside springboot-elk-prometheus-grafana
root folder run
docker compose up -d
Wait for Docker containers to be up and running. To check it, run
docker compose ps
Open a terminal and make sure you are inside springboot-elk-prometheus-grafana
folder
Run the following command
./mvnw clean spring-boot:run --projects movies-api
Note: If you want to change to "non-json-logs" (maybe during development it's useful), run
./mvnw clean spring-boot:run --projects movies-api -Dspring-boot.run.jvmArguments="-Dspring.profiles.active=non-json-logs"
springboot-elk-prometheus-grafana
root folder./docker-build.sh
./docker-build.sh native
Environment Variable | Description |
---|---|
MYSQL_HOST |
Specify host of the MySQL database to use (default localhost ) |
MYSQL_PORT |
Specify port of the MySQL database to use (default 3306 ) |
docker run --rm --name movies-api -p 8080:8080 \
-e MYSQL_HOST=mysql \
--network=springboot-elk-prometheus-grafana_default \
ivanfranchin/movies-api:1.0.0
Note: If you want to change to "non-json-logs", add
-e SPRING_PROFILES_ACTIVE=non-json-logs
to the command above
movies-api
movies-api
Swagger is http://localhost:8080/swagger-ui.html
MySQL
docker exec -it -e MYSQL_PWD=secret mysql mysql -uroot --database moviesdb
SELECT * FROM movies;
Type
exit
to get out ofMySQL monitor
Prometheus
Prometheus
can be accessed at http://localhost:9090
Grafana
Grafana
can be accessed at http://localhost:3000
admin
for both username
and password
General / Home
on the topmovies-api-dashboard
Kibana
Kibana
can be accessed at http://localhost:5601
Note: in order to see movies-api logs in Kibana, you must run the application as Docker container
Configuration
Kibana
websiteExplore on my own
Discover
Create index pattern
buttonCreate index pattern
form
filebeat-*
fot the Name
field@timestamp
for the Timestamp field
combo-boxCreate index pattern
buttonDiscover
to start performing searchesElasticsearch
Elasticsearch
URL is http://localhost:9200
Useful queries
# Check it's up and running
curl localhost:9200
# Check indexes
curl "localhost:9200/_cat/indices?v"
# Check filebeat index mapping
curl "localhost:9200/filebeat-*/_mapping"
# Simple search
curl "localhost:9200/filebeat-*/_search?pretty"
Ctrl+C
springboot-elk-prometheus-grafana
root folder, run the following command
docker compose down -v
To remove the Docker images created by this project, go to a terminal and, inside springboot-elk-prometheus-grafana
root folder, run the script below
./remove-docker-images.sh
https://medium.com/@sece.cosmin/docker-logs-with-elastic-stack-elk-filebeat-50e2b20a27c6