Simple demo implementation of Lambda and Kappa architectures using Python, Docker, Kafka, Spark and Cassandra
This project shows how one could simply implement Lambda nad Kappa architectures for making product recommendations
for an imaginary e-commerce store. It is written in Python and employs Kafka, Spark (in Jupyter notebook), Cassandra and Falcon.
All the components are tied in with Docker and their relationships are captured in docker-compose
.
Here is the video and the slides from my presentation at Yow! Data Sydney. I've also used the same slides at Sydney Docker Meetup.
For the ease of deployment Docker Compose script is used. It still needs some manual steps, however.
docker-compose.yml
file and replace paths in volumes
to match your environmentdocker-compose up
data
folder and start a feeder script POSTing JSON messages to Falcon: ./user-simulator.py
docker exec -it bigdatariver_cassandra_1 bash
cqlsh -f bdr/init.sql
cqlsh
and start issuing CQL statements directly against Cassandrahttp://localhost:8888/
and choose Lambda - Stream - Users who bought X also bought
.
Choose from the top menu: Cell->Run AllLambda - Batch- Users who bought X also bought
Kappa - Users who bought X also bought
Kappa - Collaborative Filtering
http://127.0.0.1:8000/bdr?product-lambda=59
should return response like {"product":59, "recommendedProducts":[29,49,99,19,62]}
http://127.0.0.1:8000/bdr?product-kappa=41
should return response like {"product":41, "recommendedProducts":[21,5,95,83,37]}
http://127.0.0.1:8000/bdr?user=2105
with response like
{"user":2105, "recommendedProducts":[77,5,95,83,37]}
If required to connect to Kafka from local host (outside of Docker), add kafka
hostname to your /etc/hosts
like this:
more /etc/hosts
127.0.0.1 localhost kafka