Some class materials for a data processing course using PySpark
Materials for the Advanced Data Processing course of the Big Data Analytics Master at the Universitat Politècnica de València.
This course gives a 30 hours overview of many concepts, techniques and tools in data processing using Spark, including some key concepts from Apache Beam. We assume you're familiar with Python, but all the exercises can be easily followed in Java and Scala. We've included a Vagrant definition and docker images for both Spark and Beam.
If you find a bug or you want to contribute some comments, please fill an issue in this repository or simply write us. You're free to reuse course materials, please follow details in the license section.
Team work using Aronson's puzzle. We present a set of real case studies to solve and teams have to design and develop them using any technology available in the market today.
In the first phase, the teams will split with the goal of becoming experts into a particular area and dig into the proposed tools and framework specifics. In the second phase, they'll return to their peers to design a system that covers use case requirement. There's a 15 minute presentation per team to share the results.
To be added soon, stay tuned!
Final course assignments can be found in this document. They are in Spanish, they will be translated to English at some point.
I'm not publishing the solutions to avoid remaking the exercises every year. There's a test suite using py.test to help you validate the results. If you're really interested on them, please write me to [email protected].
Self-sufficiency is the state of not requiring any aid, support, or interaction, for survival; it is therefore a type of personal or collective autonomy - Wikipedia.
We follow a self-sufficiency principles for students to drive course goals. At the end of the course, students should have enough knowledge and tools to develop small data processing solutions their own.
We recommend the following papers to expand knowledge on Spark and other data processing techniques:
Some ideas we might add in forthcoming course editions:
Advanced Data Processing course materials. Copyright (C) 2016, Luis Belloch
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Luis Belloch, course materials for Advanced Data Processing, Spring 2016. Master on Big Data Analytics (http://bigdata.inf.upv.es), Universitat Politècnica de València. Downloaded on [DD Month YYYY].