Blockchain ETL Architecture
The nodes are deployed with Terraform and run in Kubernetes. Refer to these for more details:
The blockchain data is polled periodically from the nodes and pushed to Google Pub/Sub. Refer to these for more details:
Airflow DAGs export and load blockchain data to BigQuery daily. Refer to these for more details:
The blockchain data is pulled from Pub/Sub, transformed and streamed to BigQuery. Refer to these for more details:
Various applications of the public blockchain data: