Data cleaning, pre-processing, and Analytics on a million movies using Spark and Scala.
Solving analytical questions on the semi-structured MovieLens dataset containing a million records using Spark and Scala. This features the use of Spark RDD, Spark SQL and Spark Dataframes executed on Spark-Shell (REPL) using Scala API. We aim to draw useful insights about users and movies by leveraging different forms of Spark APIs.
Simply clone the repository
git clone https://github.com/Thomas-George-T/Movies-Analytics-in-Spark-and-Scala.git
In the repo, Navigate to Spark RDD, Spark SQL or Spark Dataframe locations as needed.
Run the execute script to view results
sh execute.sh
The execute.sh
will pass the scala code through spark-shell and then display the findings in the terminal from the results folder.
Note: The results were collected and repartitioned into the same text file: This is not a recommended practice since performance is highly impacted but it is done here for the sake of readability.
This project was featured on Data Machina Issue #130 listed at number 3 under ScalaTOR. Thank you for the listing
This repository is licensed under Apache License 2.0 - see License for more details