Using Kafka to track cryptocurrency price trends
This repo polls Coinbase API for various cryptocurrency prices and uses Kafka to calculate moving average prices and stores the data in a timeseries database, QuestDB, for further analysis.
The project was inspired by "Using Kafka Streams to Analyze Live Trading Activity for Crypto Exchanges" talk by Ludvig Sandman and Bruce Zulu at Kafka Summit London 2019. Python code to poll Coinbase API was modified from fdallac/kafka-crypto-data-stream.
Start up the Kafka/QuestDB stack:
cd docker-compose
docker compose up
Wait until all the components are healthy (look at Kafka Connect container logs).
Post kafka-postgres-btc sink schema to Kafka Connect:
curl -X POST -H "Accept:application/json" -H "Content-Type:application/json" --data @questdb-sink-btc.json http://localhost:8083/connectors
Install the necessary packages:
pip install -r requirements.txt
Run the script to poll Coinbase API:
python getData.py
(Optional): calculate moving averages
python movingAverage.py