Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline
WORK-IN-PROGRESS
Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline (Medium.com)
This is an example end-to-end project that demonstrates the Debezium-Delta Lake combo pipeline
See medium post for more details
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8084/connectors/ -d @debezium/config.json
Import the notebook file in \voter-processing\voter-processing.html to a Databricks Community account and follow the instructions inside the notebook
https://community.cloud.databricks.com/
Make it a configurable generic tool that can be assembled on top of any supported database