A full big data pipeline (Lambda Architecture) with Spark, Kafka, HDFS and Cassandra.
Read about the project here
Watch the videos demonstrating the project here
Our Lambda project receives real-time IoT Data Events coming from Connected Vehicles, then ingested to Spark through Kafka. Using the Spark streaming API, we processed and analysed IoT data events and transformed them into vehicle information. While simultaneously the data is also stored into HDFS for Batch processing. We performed a series of stateless and stateful transformation using Spark streaming API on streams and persisted them to Cassandra database tables. In order to get accurate views, we also perform a batch processing and generating a batch view into Cassandra. We developed responsive web traffic monitoring dashboard using Spring Boot, SockJs and Bootstrap which get the views from the Cassandra database and push to the UI using web socket.
All component parts are dynamically managed using Docker, which means you don't need to worry about setting up your local environment, the only thing you need is to have Docker installed.
System stack:
The streaming part of the project was done from iot-traffic-project InfoQ
mvn package
docker-compose -p lambda up
./project-orchestrate.sh
docker exec spark-master /spark/bin/spark-submit --class com.apssouza.iot.streaming.StreamingProcessor --master spark://localhost:7077 /opt/spark-data/iot-spark-processor-1.0.0.jar
java -jar iot-kafka-producer/target/iot-kafka-producer-1.0.0.jar
java -jar iot-springboot-dashboard/target/iot-springboot-dashboard-1.0.0.jar
docker exec spark-master /spark/bin/spark-submit --class com.apssouza.iot.batch.BatchProcessor --master spark://localhost:7077 /opt/spark-data/iot-spark-processor-1.0.0.jar
docker exec spark-master /spark/bin/spark-submit --class com.apssouza.iot.ml.SpeedPrediction --master spark://localhost:7077 /opt/spark-data/iot-spark-processor-1.0.0.jar
mvn package
spark-submit --class com.apssouza.iot.streaming.StreamingProcessor --master spark://spark-master:7077 iot-spark-processor/target/iot-spark-processor-1.0.0.jar
Add spark-master
to /etc/hosts pointing to localhosthttp://localhost:8080 Master http://localhost:8081 Slave
Commands https://hortonworks.com/tutorial/manage-files-on-hdfs-via-cli-ambari-files-view/section/1/
Open a file - http://localhost:50070/webhdfs/v1/path/to/file/file.csv?op=open
Web file handle - https://hadoop.apache.org/docs/r1.0.4/webhdfs.html
hdfs dfs -mkdir /user
hdfs dfs -mkdir /user/lambda
hdfs dfs -put localhost.csv /user/lambda/
http://localhost:9870 http://localhost:50075
docker exec -it cassandra-iot cqlsh --username cassandra --password cassandra
use TrafficKeySpace;
SELECT * FROM TrafficKeySpace.Total_Traffic;