Hadoop Framework Examples Save

An implementation of a real-world map-reduce workflow in each major framework.

Project README

Realistic Hadoop Data Processing Examples

This code is to accompany my blog post on map reduce frameworks

The point of the code in this repository is to provide an implementation for a business question (listed below) in each of the major Map Reduce frameworks.

Each implementation will get it's own subdirectory with it's own build and running instructions. Each framework will also get an accompanying test, and an in-depth walkthrough about implementation details.

The following implementations are complete:

The problem

The Data

We have two datasets: customers, and transactions.

Customer Fields:

Transaction Fields:

  • transaction-id (1)
  • product-id (1)
  • user-id (1)
  • purchase-amount (19.99)
  • product-description (a rubber chicken)

These two datasets are stored in tab-delimited files somewhere on HDFS.

The Question

For each product, we want to know the number of locations in which that product was purchased.

That's it!

In the real world, we might have other questions, like the number of purchases per location for each product.

Open Source Agenda is not affiliated with "Hadoop Framework Examples" Project. README Source: rathboma/hadoop-framework-examples
Stars
149
Open Issues
5
Last Commit
8 years ago
Tags

Open Source Agenda Badge

Open Source Agenda Rating