Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard.
Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard.
Infrastructure provisioning through Terraform, containerized through Docker and orchestrated through Airflow. Created dashboard through Metabase.
DAG Tasks:
make infra
: create AWS services. You will be asked to enter a password for your Redshift and RDS clusters.make config
: generate configuration with Terraform outputs and AWS credentials.make base-build
: build base airflow image with project requirements.make build
: build docker images for airflow.make up
: run the pipeline.