Kafka streaming with Spark and Flink example
Example project on how to use Apache Kafka and streaming consumers, namely:
mvn clean package
docker build -t kafka-spark-flink-example .
docker-compose up -d
Check producer logs
docker logs kafka-spark-flink-example_kafka-producer_1 -f
Output should be similar to:
[main] INFO org.davidcampos.kafka.producer.KafkaProducerExample - Sent (74b23319-084c-4309-80a7-c0d6f107a092, eight) to topic example @ 1525127107909.
Check consumer with Spark logs
docker logs kafka-spark-flink-example_kafka-consumer-spark_1 -f
Output should be similar to:
(two,3)
(one,3)
(nine,5)
(six,8)
(three,2)
(five,2)
(four,9)
(seven,3)
(eight,6)
(ten,6)
Check consumer with Flink logs
docker logs kafka-spark-flink-example_kafka-consumer-flink_1 -f
Output should be similar to:
1> (ten,85)
4> (nine,104)
1> (ten,86)
4> (five,91)
4> (one,94)
4> (six,90)
1> (three,89)
4> (six,91)
4> (five,92)
Kafka Manager Web UI available at http://localhost:9000.
Spark Web UI available at http://localhost:4040.
docker-compose down