Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift
Dataform core is an open source meta-language to create SQL tables and workflows. Dataform core extends SQL by providing a dependency management system, automated data quality testing, and data documentation.
Using Dataform core, data teams can build scalable SQL data transformation pipelines following software engineering best practices, like version control and testing.
Note: we have recently undergone a documentation transition from docs.dataform.co to cloud.google.com/dataform/docs. Content hosted on the old document site is published from the main_v1
branch.
You can install the Dataform CLI tool using the following command line. Follow the docs to get started.
npm i -g @dataform/cli
Dataform in Google Cloud Platform provides a fully managed experience to build scalable data transformations pipelines in BigQuery using SQL. It includes:
You can learn more on cloud.google.com/dataform
Check out our contributors guide to get started with setting up the repo.