[v0.9.0] - 2022-09-13

New Platform Support

Databricks

:sparkles: Staging :sparkles: Hubs :sparkles: Links :sparkles: Satellites

Postgres**

:sparkles: Staging :sparkles: Hubs :sparkles: Links :sparkles: Satellites

**There are currently limitations on Postgres. Click here for more details

New Features

Introducing:

:star2: Payload column exclusion: Satellite's payload can now be configured to select all columns, except a user-defined list using an exclude configuration (#128) :star2: Null Business Key Handling: Now users can provide a configuration in their staging tables to handle NULL keys elegantly, according to business needs (#133) :star2: Extra Columns Parameter: All table macros now provide a src_extra_columns parameter which allows users to add extra columns outside of the standard template for business needs. :star2: More logging: Hubs and Links now provide additional logging about the number of sources they are loading from. Minor but helpful! This is our first step towards giving our users more information.

Fixes

:white_check_mark: Fixed an edge case for vault_insert_by_period when the staging table and the target table were in different databases (#121) :white_check_mark: Removed the uppercase conversion in the staging macro (#122, #134) :white_check_mark: Fixed an issue where duplicate records (same hashdiff) would sometimes be loaded into a Satellite (#126) :white_check_mark: Disabled automatic column name escaping in derived columns when using the stage() macro. Escaping can now be configured on a case-by-case basis for each column to escape when they are reserved words etc. (#114, #141)

Breaking changes

• Read our 0.8.3 to 0.9.0 migration guide

Behind the scenes

• Re-factor PIT and Bridge macros to ensure better maintainability and readability

Docs

• Moved stage configuration details to a new page
• Updated packages behind the scenes for security and bug fixes
• Created landing pages for sections, which should make navigation easier, e.g. getting started is now the home page when clicking 'Tutorials' in the menu, instead of having to click twice.

With thanks to:

Thank you to the following people/teams for making this release possible and working closely with the dbtvault team over the last few months to give the community an initial set of macros for new platform support! Fantastic work. We are proud to have such a great community behind us.

@kk-ais - Thank you to KK and the AIS team for the initial Databricks support. @johnoscott - Thank you for the hard work on initial Postgres support. @eamontaaffe - Migrate flags.FULL_REFRESH to should_full_refresh() macro #154

Honourable Mentions

Thank you to @NikkaZ and their team for their eagerness for Databricks support and their support, advice and patience anticipating its release.

[v0.8.3] - 2022-05-10

All existing macros are now supported by all platforms!

New

General

• dbt 1.1.x support*

*NOTE Currently not supported in MS SQL Server Read More for details.

Google BigQuery and MS SQL Server

• PITs (pit macro)
• Bridges (bridge macro)

Fixed

Effectivity Satellites

• Fixed an issue affecting auto-end-dating in flip-flop situations eff_sat (#115)

Staging

• Fixed an issue where hashed columns with lower-case columns provided to an exclude_columns config, behaved incorrectly (#110)

[v0.8.2] - 2022-03-14

More Google BigQuery and MS SQL Server support, plus fixes!

New

Google BigQuery and MS SQL Server

• T-Links (t_link macro)
• Effectivity Satellites (eff_sat macro)
• Multi-Active Satellites (ma_sat macro)
• Extended Tracking Satellites (xts macro)

See our Platform support matrix for more details.

Fixed

• Fixed a bug where vault_insert_by_period would give an error during incremental loads (#108)
• Fixed vault_insert_by_x issues for MS SQL Server
• Fixed (increased) datetime precision in max_datetime for Google BigQuery

[v0.8.1] - 2022-02-22

HOTFIX RELEASE

Fixed

• Fixed a bug where vault_insert_by_rank unintentionally used logic from vault_insert_by_period when in full-refresh mode or replacing an existing view.

[v0.8.0] - 2022-02-21

This is a big release for dbtvault. It's the first time we are releasing support for new platforms! Please welcome to the dbtvault family, Google BigQuery and MS SQL Server!

This is just the start, and we're excited to bring even more platforms (and further support for existing platforms) to you in the future!

New

Google BigQuery and MS SQL Server

• Hubs (hub macro)
• Links (link macro)
• Satellites (sat macro)

Take a look at our new Platform support matrix for a visual representation.

All platforms

• Column Escaping (#28, #23) - Docs: dbtvault now automatically surrounds all column names with quotes. This is to allow for columns with reserved words, spaces, and other oddities. The type of quotes is configurable, please refer to the docs linked above.

Thanks

With thanks to @Andrew-Wichmann for bringing to our attention and providing code for improving our custom materialisaions with dbt's new should_full_refresh() macro.

[v0.7.9] - 2021-12-13

Dependencies

Breaking

• dbt v1.0.0 Support
  • Support for earlier dbt versions (dbt <=0.21.0) removed. This is as a result of the upgrade to dbt v1.0.0. How do I upgrade my project?
• Updated to dbt_utils v0.8.0 (for dbt 1.0.0 compatibility)

New

Table structures

• Point in Time tables Tutorial - Macro docs
• Bridge tables Tutorial - Macro docs
• Extended Tracking Satellites (XTS) Tutorial - Macro docs

Materialisations

• Custom materialisation for PITs Docs
• Custom materialisation for Bridges Docs

Behind the Scenes

• More test coverage for incremental loading.
• Improved consistency and support for Composite PKs.
• Significantly simplified Multi-Active Satellite (MAS) logic.

Bug Fixes

• Multi-Active Satellite record duplication under some circumstances #50

[v0.7.8] - 2021-10-25

Dependencies

• dbt 0.21.0 support
• dbt utils package dependency now has a version range

Fixes

• Effectivity Satellites with auto-end-dating off now handle the use case where records may already be end-dated in the staging layer, as a result of loading data 'manually' end-dated by business rules.

Features

Rank column configurations in stage macro (ranked_columns):

• Provide ASC or DESC for an order_by column Read More
• Configure the ranking to use DENSE_RANK() or RANK() Read More

Configuration for hash strings

Read More

• Concatenation string can now be user defined
• Null placeholder string can now be user defined

[v0.7.7] - 2021-08-24

• Re-release of v0.7.6.1 to ensure deployment to dbt Hub

[v0.7.6.1] - 2021-07-14

• Hotfix for 0.7.6 to remove unintentionally added macros from the beta branch. #36

Installing

Note: This version cannot be installed via dbt hub, please install as follows:

packages:
  - git: "https://github.com/Datavault-UK/dbtvault.git"
    revision: v0.7.6.1

[v0.7.6] - 2021-07-13

• Updated to dbt 0.20.0 and incorporated adapter.dispatch changes (#32)

THIS VERSION HAS A MAJOR BUG (#36) Please install v0.7.6.1, which contains a hotfix, as follows:

packages:
  - git: "https://github.com/Datavault-UK/dbtvault.git"
    revision: v0.7.6.1