Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift
Breaking changes:
defaultLocation
is required in dataform.json
Version 2.0.0
will be required for Dataform running on GCP.
Introduces a defaultLocation
field for setting default schema, job locations for Dataform on GCP.
requirePartitionFilter
and partitionExpirationDays
BigQuery options are now supported, see https://docs.dataform.co/warehouses/bigquery#configuration-options for more infoadditionalOptions
feature, e.g. bigquery: {
additionalOptions : {
partition_expiration_days : "1",
"require_partition_filter" : "true",
friendly_name : '"friendlyName"',
}
}
--include-dependents
flag is now supported in the CLI to request inclusion of dependent actions of all other selected actions in a runFixes Unexpected token (2:15)
error (https://github.com/dataform-co/dataform/issues/1305).
Support for materialized views has been added for BigQuery and Snowflake. Example:
config {
type: "view",
materialize: true,
...
}
...
In addition, performance improvements have been made to @dataform/core
that significantly improve Dataform compilation times, especially for larger projects. We strongly recommend upgrading to this version to receive these performance improvements.
This release removes support for run caching, and introduces a number of proto API cleanups in advance of Dataform's GCP integration.
Importantly, this removes name
and dependencies
fields from the CompiledGraph
proto. Please migrate to target.*
and dependencyTargets
instead.
(This release was previously rolled back in 1.20.2
).