Bigdata Pipeline
Here is my project about real time stock price visualization by using bigdata pipeline. To work with big data, we request frameworks with high availability, high perfermance and high stability. The latest bigdata framework is composed by "SMACK", that is Spark, Mesos, Akka, Cassandra and Kafka. In my project, I connected Zookeeper, Kafka, Cassandra, redis and Nodejs in Docker machine, from the back end to the front end, to realize the stock price visualization. All of the code is in Python version.
Set the Docker Eve and run the follow command
Fetch multiple stock data by flask-data-producer.py
export ENV_CONFIG_FILE=`pwd`/config/dev.cfg
python flask-data-producer.py
spark-submit --jars spark-streaming-kafka-0-8-assembly_2.11-2.0.0.jar stream-processing.py stock-analyzer average-stock-price 192.168.99.100:9092
python redis-publisher.py average-stock-price 192.168.99.100:9092 average-stock-price 192.168.99.100 6379
node index.js --port=3000 --redis_host=192.168.99.102 --redis_port=6379 --subscribe_topic=average-stock-price
Add the stock brand you like to see the results
command line screen shots