dbt-core 1.8.0 - Julian Abele (May 09, 2024)
Breaking Changes
- Remove adapter.get_compiler interface (#9148)
- Move AdapterLogger to adapters folder (#9151)
- Rm --dry-run flag from 'dbt deps --add-package', in favor of just 'dbt deps --lock' (#9100)
- move event manager setup back to core, remove ref to global EVENT_MANAGER and clean up event manager functions (#9150)
- Remove dbt-tests-adapter and dbt-postgres packages from dbt-core (#9455)
- Update the default behaviour of require_explicit_package_overrides_for_builtin_materializations to True. (#10062)
Features
- Initial implementation of unit testing (#8287)
- Unit test manifest artifacts and selection (#8295)
- Support config with tags & meta for unit tests (#8294)
- Allow adapters to include package logs in dbt standard logging (#7859)
- Enable inline csv fixtures in unit tests (#8626)
- Add drop_schema_named macro (#8025)
- migrate utils to common and adapters folders (#8924)
- Move Agate helper client into common (#8926)
- remove usage of dbt.config.PartialProject from dbt/adapters (#8928)
- Add exports to SavedQuery spec (#8892)
- Support unit testing incremental models (#8422)
- Add support of csv file fixtures to unit testing (#8290)
- Remove legacy logger (#8027)
- Unit tests support --defer and state:modified (#8517)
- Support setting export configs hierarchically via saved query and project configs (#8956)
- Support source inputs in unit tests (#8507)
- Use daff to render diff displayed in stdout when unit test fails (#8558)
- Global config for --target and --profile CLI flags and DBT_TARGET and DBT_PROFILE environment variables. (#7798)
- Move unit testing to test command (#8979)
- Support --empty flag for schema-only dry runs (#8971)
- Support unit tests in non-root packages (#8285)
- Convert the
tests config to data_tests in both dbt_project.yml and schema files. in schema files. (#8699)
- Make fixture files full-fledged parts of the manifest and enable partial parsing (#9067)
- Adds support for parsing conversion metric related properties for the semantic layer. (#9203)
- Package selector syntax for the current package (#6891)
- In build command run unit tests before models (#9128)
- Move flags from UserConfig in profiles.yml to flags in dbt_project.yml (#9183)
- Added hook support for
dbt source freshness (#5609)
- Align with order of unit test output when
actual differs from expected (#9370)
- Added support for external nodes in unit test nodes (#8944)
- Enable unit testing versioned models (#9344)
- Enable list command for unit tests (#8508)
- Integration Test Optimizations (#9498)
- Accelerate integration tests with caching. (#9498)
- Cache environment variables (#9489)
- Support meta at the config level for Metric nodes (#9441)
- Add cache to SavedQuery config (#9540)
- Support scrubbing secret vars (#7247)
- Allow excluding resource types for build, list, and clone commands, and provide env vars (#9237)
- SourceDefinition.meta represents source-level and table-level meta properties, instead of only table-level (#9766)
- Allow metrics in semantic layer filters. (#9804)
- Add wildcard support to the group selector method (#9811)
- source freshness precomputes metadata-based freshness in batch, if possible (#8705)
- Better error message when trying to select a disabled model (#9747)
- Support SQL in unit testing fixtures (#9405)
- Add require_explicit_package_overrides_for_builtin_materializations to dbt_project.yml flags, which can be used to opt-out of overriding built-in materializations from packages (#10007)
- add --empty flag to dbt build command (#10026)
- Ability to
silence warnings via warn_error_options (#9644)
- Allow aliases
error for include and warn for exclude in warn_error_options (#9644)
- Add unit_test: selection method (#10053)
Fixes
- For packages installed with tarball method, fetch metadata to resolve nested dependencies (#8621)
- Fix partial parsing not working for semantic model change (#8859)
- Handle unknown
type_code for model contracts (#8877, #8353)
- Rework get_catalog implementation to retain previous adapter interface semantics (#8846)
- Add back contract enforcement for temporary tables on postgres (#8857)
- Add version to fqn when version==0 (#8836)
- Fix cased comparison in catalog-retrieval function. (#8939)
- Catalog queries now assign the correct type to materialized views (#8864)
- Fix compilation exception running empty seed file and support new Integer agate data_type (#8895)
- Make relation filtering None-tolerant for maximal flexibility across adapters. (#8974)
- Update run_results.json from previous versions of dbt to support deferral and rerun from failure (#9010)
- Use MANIFEST.in to recursively include all jinja templates; fixes issue where some templates were not included in the distribution (#9016)
- Fix git repository with subdirectory for Deps (#9000)
- Use seed file from disk for unit testing if rows not specified in YAML config (#8652)
- Fix formatting of tarball information in packages-lock.yml (#9062)
- deps: Lock git packages to commit SHA during resolution (#9050)
- deps: Use PackageRenderer to read package-lock.json (#9127)
- Ensure we produce valid jsonschema schemas for manifest, catalog, run-results, and sources (#8991)
- Get sources working again in dbt docs generate (#9119)
- Fix parsing f-strings in python models (#6976)
- Preserve the value of vars and the --full-refresh flags when using retry. (#9112)
- fix lock-file bad indentation (#9319)
- fix configuration of turning test warnings into failures with WARN_ERROR_OPTIONS (#7761)
- Support reasonably long unit test names (#9015)
- Fix back-compat parsing for model-level 'tests', source table-level 'tests', and 'tests' defined on model versions (#9411)
- Fix retry command run from CLI (#9444)
- Fix seed and source selection in
dbt docs generate (#9161)
- Add TestGenerateCatalogWithExternalNodes, include empty nodes in node selection during docs generate (#9456)
- Fix node type plurals in FoundStats log message (#9464)
- Run manifest upgrade preprocessing on any older manifest version, including v11 (#9487)
- Update 'compiled_code' context member logic to route based on command ('clone' or not). Reimplement 'sql' context member as wrapper of 'compiled_code'. (#9502)
- Fix bug where Semantic Layer filter strings are parsed into lists. (#9507)
- Initialize invocation context before test fixtures are built. (##9489)
- Fix conflict with newer versions of Snowplow tracker (#8719)
- When patching versioned models, set constraints after config (#9364)
- only include unmodified semantic mdodels in state:modified selection (#9548)
- Set query headers when manifest is passed in to dbtRunner (#9546)
- Store node_info in node associated logging events (#9557)
- Fix Semantic Model Compare node relations (#9548)
- Tighten exception handling to avoid worker thread hangs. (#9583)
- Clearer no-op logging in stubbed SavedQueryRunner (#9533)
- Fix node_info contextvar handling so incorrect node_info doesn't persist (#8866)
- Add target-path to retry (#8948)
- Do not add duplicate input_measures (#9360)
- Throw a ParsingError if a primary key constraint is defined on multiple columns or at both the column and model level. (#9581)
- Bug fix: don't parse Jinja in filters for input metrics or measures. (#9582)
- Fix traceback parsing for exceptions raised due to csv fixtures moved into or out of fixture/subfolders. (#9570)
- Fix partial parsing
KeyError on deleted schema files (#8860)
- Support saved queries in
dbt list (#9532)
- include sources in catalog.json when over 100 relations selected for catalog generation (#9755)
- Support overriding macros in packages in unit testing (#9624)
- Handle exceptions for failing on-run-* hooks in source freshness (#9511)
- Validation of unit test parsing for incremental models (#9593)
- Fix use of retry command on command using defer (#9770)
- Make
args variable to be un-modified by dbt.invoke(args) (#8938, #9787)
- Only create the packages-install-path / dbt_packages folder during dbt deps (#6985, #9584)
- Unit test path outputs (#9608)
- Fix assorted source freshness edgecases so check is run or actionable information is given (#9078)
- "Fix Docker release process to account for both historical and current versions of
dbt-postgres (#9827)
- Exclude password-like fields for considering reparse (#9795)
- Fixed query comments test (#9860)
- Begin warning people about spaces in model names (#9397)
- Add NodeRelation to SavedQuery Export (#9534)
- Disambiguiate FreshnessConfigProblem error message (#9891)
- Use consistent secret scrubbing with the log function. (#9987)
- Validate against empty strings in package definitions (#9985)
- Fix default value for indirect selection in selector cannot overwritten by CLI flag and env var (#9976, #7673)
- Simplify error message if test severity isn't 'warn' or 'error' (#9715)
- Support overriding source level loaded_at_field with a null table level definition (#9320)
- Undo conditional agate import to prevent UnresolvedTypeReferenceError during RunResult serialization (#10098)
- Restore previous behavior for --favor-state: only favor defer_relation if not selected in current command" (#10107)
- Unit test fixture (csv) returns null for empty value (#9881)
Docs
Under the Hood
- Added more type annotations. (#8537)
- Add unit testing functional tests (#8512)
- Remove usage of dbt.include.global_project in dbt/adapters (#8925)
- Add a no-op runner for Saved Qeury (#8893)
- remove dbt.flags.MP_CONTEXT usage in dbt/adapters (#8967)
- Remove usage of dbt.flags.LOG_CACHE_EVENTS in dbt/adapters (#8969)
- Move CatalogRelationTypes test case to the shared test suite to be reused by adapter maintainers (#8952)
- Treat SystemExit as an interrupt if raised during node execution. (#n/a)
- Removing unused 'documentable' (#8871)
- Remove use of dbt/core exceptions in dbt/adapter (#8920)
- Cache dbt plugin modules to improve integration test performance (#9029)
- Consolidate deferral methods & flags (#7965, #8715)
- Fix test_current_timestamp_matches_utc test; allow for MacOS runner system clock variance (#9057)
- Remove usage of dbt.deprecations in dbt/adapters, enable core & adapter-specific event types and protos (#8927, #8918)
- Clean up unused adaptor folders (#9123)
- Move column constraints into common/contracts, removing another dependency of adapters on core. (#9024)
- Move dbt.semver to dbt.common.semver and update references. (#9039)
- Move lowercase utils method to common (#9180)
- Remove usages of dbt.clients.jinja in dbt/adapters (#9205)
- Remove usage of dbt.contracts in dbt/adapters (#9208)
- Remove usage of dbt.contracts.graph.nodes.ResultNode in dbt/adapters (#9214)
- Introduce RelationConfig Protocol, consolidate Relation.create_from (#9215)
- remove manifest from adapter.set_relations_cache signature (#9217)
- remove manifest from adapter catalog method signatures (#9218)
- Move BaseConfig, Metadata and various other contract classes from model_config to common/contracts/config (#8919)
- Add MacroResolverProtocol, remove lazy loading of manifest in adapter.execute_macro (#9244)
- pass query header context to MacroQueryStringSetter (#9249, #9250)
- add macro_context_generator on adapter (#9247)
- pass mp_context to adapter factory as argument instead of import (#9025)
- have dbt-postgres use RelationConfig protocol for materialized views' (#9292)
- move system.py to common as dbt-bigquery relies on it to call gcloud (#9293)
- Reorganizing event definitions to define core events in dbt/events rather than dbt/common (#9152)
- move exceptions used only in dbt/common to dbt/common/exceptions (#9332)
- Remove usage of dbt.adapters.factory in dbt/common (#9334)
- Accept valid_error_names in WarnErrorOptions constructor, remove global usage of event modules (#9337)
- Move result objects to dbt.artifacts (#9193)
- dbt Labs OSS standardization of docs and templates. (#9252)
- Add dbt-common as a dependency and remove dbt/common (#9357)
- move cache exceptions to dbt/adapters (#9362)
- Clean up macro contexts. (#9422)
- Add the @requires.manifest decorator to the retry command. (#9426)
- Move WritableManifest + Documentation to dbt/artifacts (#9378, #9379)
- Define Macro and Group resources in dbt/artifacts (#9381, #9382)
- Move
SavedQuery data definition to dbt/artifacts (#9386)
- Migrate data parts of
Metric node to dbt/artifacts (#9383)
- Move data portion of
SemanticModel to dbt/artifacts (#9387)
- Move data parts of
Exposure class to dbt/artifacts (#9380)
- Split up deferral across parsing (adding 'defer_relation' from state manifest) and runtime ref resolution" (#9199)
- Start using
Mergeable from dbt-common (#9505)
- Move manifest nodes to artifacts (#9388)
- Move data parts of
SourceDefinition class to dbt/artifacts (#9384)
- Remove uses of Replaceable class (#7802)
- Make dbt-core compatible with Python 3.12 (#9007)
- Restrict protobuf to major version 4. (#9566)
- Remove references to dbt.tracking and dbt.flags from dbt/artifacts (#9390)
- Remove unused key
wildcard from MethodName enum (#9641)
- Implement primary key inference for model nodes (#9652)
- Define UnitTestDefinition resource in dbt/artifacts/resources (#9667)
- Use Manifest instead of WritableManifest in PreviousState and _get_deferred_manifest (#9567)
- Improve dbt CLI speed (#4627)
- Include node_info in various Result events (#9619)
- Remove non dbt.artifacts dbt.* imports from dbt/artifacts (#9926)
- Migrate to using
error_tag provided by dbt-common (#9914)
- Add a test for semantic manifest and move test fixtures needed for it (#9665)
- Raise deprecation warning if installed package overrides built-in materialization (#9971)
- Use the SECRET_ENV_PREFIX from dbt_common instead of duplicating it in dbt-core (#10018)
- Enable use of record mode via environment variable (#10045)
- Consistent naming + deprecation warnings for "legacy behavior" flags (#10062)
- Enable use of context in serialization (#10093)
Dependencies
- Bump actions/checkout from 3 to 4 (#8781)
- Begin using DSI 0.4.x (#8892)
- Update typing-extensions version to >=4.4 (#9012)
- Bump ddtrace from 2.1.7 to 2.3.0 (#9132)
- Bump freezegun from 0.3.12 to 1.3.0 (#9197)
- Bump actions/setup-python from 4 to 5 (#9267)
- Bump actions/download-artifact from 3 to 4 (#9374)
- Relax pathspec upper bound version restriction (#9373)
- remove dbt/adapters and add dependency on dbt-adapters (#9430)
- Bump actions/upload-artifact from 3 to 4 (#9470)
- Bump actions/cache from 3 to 4 (#9471)
- Bump peter-evans/create-pull-request from 5 to 6 (#9552)
- Restrict protobuf to 4.* versions (#9566)
- Bump codecov/codecov-action from 3 to 4 (#9659)
- Cap dbt-semantic-interfaces version range to <0.6 (#9671)
- Bump python from 3.10.7-slim-nullseye to 3.11.2-slim-bullseye in /docker (#9687)
- bump dbt-common to accept major version 1 (#9690)
- Remove duplicate dependency of protobuf in dev-requirements (#9830)
- Bump black from 23.3.0 to >=24.3.0,<25.0 (#8074)
- Update the agate pin to "agate>=1.7.0,<1.10" (#9934)
Security
- Update Jinja2 to >= 3.1.3 to address CVE-2024-22195 (#9638)
- Bump sqlparse to >=0.5.0, <0.6.0 to address GHSA-2m57-hf25-phgg (#9951)
Contributors
Julian Abele (1881-1950)
Thanks to @graciegoheen for writing this biography of Julian Abele—a famous Philadelphian, and the namesake of this dbt Core release.
Julian Abele (1881-1950) was a native Philadelphian architect who overcame deep racial prejudice to become the most accomplished Black architect of the 20th century. As the first Black student to attend the University of Pennsylvania's Department of Architecture, he was not allowed to live in the school's dorms or eat in any of the cafeterias. On projects assigned to pairs of students, he worked almost entirely with Louis Magaziner, the only Jewish student in the department, which began a lifelong friendship between the two architects.
Despite many challenges, Abele won several student prizes and secured a job at the prestigious Trumbauer firm shortly after graduation. Within three years, he became the chief designer and the second-highest paid employee of the firm. While working at Trumbauer, Abele led some of its largest projects, including the campus of Duke University, Harvard's Widener Library, and the Philadelphia Museum of Art. In particular, Abele was entirely responsible for the design of the exterior terracing of the Philadelphia Museum of Art, which was featured prominently in the 1976 film Rocky -- a fitting image for an architect whose entire life was a story of beating the odds.