Dlt Versions Save

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

0.4.9a2

3 weeks ago

A pre-release that allows to try out the following features and includes the following bugfixes:

SCD2 support by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1168 We are still working on BigQuery support) https://dlthub.com/devel/general-usage/incremental-loading#scd2-strategy
A fully configurable layout for filesystem files by @sultaniman in https://github.com/dlt-hub/dlt/pull/1182 https://dlthub.com/devel/dlt-ecosystem/destinations/filesystem#files-layout
picks file format matching item format by @rudolfix in https://github.com/dlt-hub/dlt/pull/1222
fix athena iceberg's trailing location by @romanperesypkin in https://github.com/dlt-hub/dlt/pull/1230
Pass options to parse iso like strings by @VioletM in https://github.com/dlt-hub/dlt/pull/1219
filesystem state sync by @sh-rp in https://github.com/dlt-hub/dlt/pull/1184 - https://dlthub.com/devel/dlt-ecosystem/destinations/filesystem#syncing-of-dlt-state
Remove staging-optimized replace strategy for synapse by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1231
fixes bug, where configs where not injected for async functions by @sh-rp in https://github.com/dlt-hub/dlt/pull/1241
adds options to write csv headers, change delimiter by @rudolfix in https://github.com/dlt-hub/dlt/pull/1239

Final release is scheduled for next week

0.4.8

1 month ago

Core Library

Add Dremio as a destination by @maxfirman in https://github.com/dlt-hub/dlt/pull/1026
adds a fast loading of arrow tables/pandas to postgres via COPY csv by @rudolfix in https://github.com/dlt-hub/dlt/pull/1185
adds a csv writer for filesystem and postgres by @rudolfix in https://github.com/dlt-hub/dlt/pull/1185
saves parquet with all logical types, spark flavor is not a default any longer by @rudolfix in https://github.com/dlt-hub/dlt/pull/1185 https://github.com/dlt-hub/dlt/pull/1185
feat(bigquery): add streaming inserts support by @IlyaFaer in https://github.com/dlt-hub/dlt/pull/1123
Feat: parameterize pipeline class in the primary factory method by @z3z1ma in https://github.com/dlt-hub/dlt/pull/1176
Fix: check for typeddict before class or subclass checks which fail by @z3z1ma in https://github.com/dlt-hub/dlt/pull/1160
fixes column order and add hints table variants by @rudolfix in https://github.com/dlt-hub/dlt/pull/1127
fixes schema versioning by @rudolfix in https://github.com/dlt-hub/dlt/pull/1140
regular initializers for credentials / config specs are type checked like dataclasses by @rudolfix in https://github.com/dlt-hub/dlt/pull/1142
fix streamlit app state display: Add yaml representer for pendulum datetime by @sultaniman in https://github.com/dlt-hub/dlt/pull/1192
synapse and mssql bugfixes and improvements (INSERT VALUES UNION) by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/1174
various improvements to arrow table normalization by @rudolfix in https://github.com/dlt-hub/dlt/pull/1185
arrow tables without rows create tables in destination by @rudolfix in https://github.com/dlt-hub/dlt/pull/1185
fixes Motherduck configuration to use my_db default database and makes password / token mandatory by @rudolfix in

Docs

docs: add typechecking to embedded snippets by @sh-rp in https://github.com/dlt-hub/dlt/pull/1130
Fix typo with switched column names in schema evolution docs page by @b-per in https://github.com/dlt-hub/dlt/pull/1132
Docs: deploy with Kestra by @dat-a-man in https://github.com/dlt-hub/dlt/pull/1087
Docs: Deploy dlt on dagster by @dat-a-man in https://github.com/dlt-hub/dlt/pull/1086
Update example connection string by @MiConnell in https://github.com/dlt-hub/dlt/pull/1188
Changed directory of all the blog images to google cloud storage. by @dat-a-man in https://github.com/dlt-hub/dlt/pull/1156

Verified Sources

postgres replication / CDC by @jorritsandbrink https://github.com/dlt-hub/verified-sources/pull/392

New Contributors

@b-per made their first contribution in https://github.com/dlt-hub/dlt/pull/1132
@MiConnell made their first contribution in https://github.com/dlt-hub/dlt/pull/1188
@maxfirman made their first contribution in https://github.com/dlt-hub/dlt/pull/1026

Full Changelog: https://github.com/dlt-hub/dlt/compare/0.4.7...0.4.8

0.4.7

1 month ago

Core Library

Custom destinations with @dlt.destination decorator by @sh-rp in https://github.com/dlt-hub/dlt/pull/1065
A BigQuery custom destination supporting STRUCT data types by @sh-rp in https://github.com/dlt-hub/dlt/pull/1107
Built-in Streamlit rewrite, UI improvements, dark theme a by @sultaniman in https://github.com/dlt-hub/dlt/pull/1060
fixes various edge cases with Incremental data deduplication, for ordered and unordered results #971 by @rudolfix in https://github.com/dlt-hub/dlt/pull/1062
Adds new dlt.mark marker to materialize table schemas without data by @rudolfix in https://github.com/dlt-hub/dlt/pull/1122
validates class instances in typed dict by @rudolfix in https://github.com/dlt-hub/dlt/pull/1082
feat(airflow): allow re-using sources in airflow wrapper by @IlyaFaer in https://github.com/dlt-hub/dlt/pull/1080
feat(core): drop default value for write disposition by @IlyaFaer in https://github.com/dlt-hub/dlt/pull/1057
splits pandas and arrow imports to fix pyarrow.compute missing by @rudolfix in https://github.com/dlt-hub/dlt/pull/1112
improve no schema upgrade path exception by @sh-rp in https://github.com/dlt-hub/dlt/pull/1125

Docs

docs(airflow): add description of new decompose methods by @IlyaFaer in https://github.com/dlt-hub/dlt/pull/1072
check embedded code blocks by @sh-rp in https://github.com/dlt-hub/dlt/pull/1093
docs(kafka): describe the possible sync issues by @IlyaFaer in https://github.com/dlt-hub/dlt/pull/1100
Docs: schema evolution by @dat-a-man in https://github.com/dlt-hub/dlt/pull/1078
Add example link to the custom destination page by @VioletM in https://github.com/dlt-hub/dlt/pull/1120

Full Changelog: https://github.com/dlt-hub/dlt/compare/0.4.6...0.4.7

0.4.6

2 months ago

Core Library

feat(airflow): expose the Airflow runner method to create custom DAGs by @IlyaFaer in https://github.com/dlt-hub/dlt/pull/1014
removes sql alchemy dependency and port parts of URL class by @rudolfix in https://github.com/dlt-hub/dlt/pull/1028
Parallelize decorator - run many regular generators in parallel by @steinitzu in https://github.com/dlt-hub/dlt/pull/965
Add main entry point to support calling dlt as python module by @sultaniman in https://github.com/dlt-hub/dlt/pull/1023

Library Bugfixes

fixes naive datetime bug in incremental by @rudolfix in https://github.com/dlt-hub/dlt/pull/1020
Import missing pyarrow compute for transforms on arrowitems by @sh-rp in https://github.com/dlt-hub/dlt/pull/1010
delete normalized package in case it already existed by @sh-rp in https://github.com/dlt-hub/dlt/pull/1012
fix(core): validation error with TTableHintTemplate by @IlyaFaer in https://github.com/dlt-hub/dlt/pull/1039
adds test case where payload data contains PUA unicode characters by @willi-mueller in https://github.com/dlt-hub/dlt/pull/1053
fix add_limit behavior in edge cases by @sh-rp in https://github.com/dlt-hub/dlt/pull/1052
adds row_order to Incremental - automatically stop taking data when out of range by @rudolfix in https://github.com/dlt-hub/dlt/pull/1041
Fix to serialize load metrics as list instead of a dictionary by @sultaniman in https://github.com/dlt-hub/dlt/pull/1051
fix import schema workflow by @sh-rp in https://github.com/dlt-hub/dlt/pull/1013
rollback all changes to live schemas when extraction fails by @sh-rp in https://github.com/dlt-hub/dlt/pull/1013

Docs

Fix zendesk example test by @VioletM in https://github.com/dlt-hub/dlt/pull/1027
Edit arrow-pandas.md and fix a typo by @Bl3f in https://github.com/dlt-hub/dlt/pull/1001
Added info about file compression to filesystem docs by @dat-a-man in https://github.com/dlt-hub/dlt/pull/975
Update "create destination" docs with new file layouts by @steinitzu in https://github.com/dlt-hub/dlt/pull/1032
Docs update on how to set query limits. by @dat-a-man in https://github.com/dlt-hub/dlt/pull/973
Docs/Updated for slack alerts. by @dat-a-man in https://github.com/dlt-hub/dlt/pull/1042

Verified Sources

scrape web sites with spiders and Scrapy and send data to dlt @sultaniman https://github.com/dlt-hub/verified-sources/pull/332
sql_database recoginizes end_value and row_order to return rows in range and optionally ordered. backfill and proper Airflow intervals support @rudolfix https://github.com/dlt-hub/verified-sources/pull/388/

New Contributors

@Bl3f made their first contribution in https://github.com/dlt-hub/dlt/pull/1001

Full Changelog: https://github.com/dlt-hub/dlt/compare/0.4.5...0.4.6

0.4.5

2 months ago

Core Library

enables google drive filesystem for sources and destinations (second one experimental, google drive listings are only eventually consistent!) by @IlyaFaer in https://github.com/dlt-hub/dlt/pull/932
creates parallel Airflow DAGs in airflow helper to allow many resources to be executed at once @IlyaFaer in https://github.com/dlt-hub/dlt/pull/966
855 create bigquery adapter for dlt resources: easily configure partitions, clustering, data retention etc. by @Pipboyguy in https://github.com/dlt-hub/dlt/pull/952 and https://dlthub.com/docs/dlt-ecosystem/destinations/bigquery#bigquery-adapter
Use BIGNUMERIC for large decimals in bigquery by @steinitzu in https://github.com/dlt-hub/dlt/pull/984
Normalize keys for Google secrets config provider by @sultaniman in https://github.com/dlt-hub/dlt/pull/963
does not lowercase postgres and redshift database names by @rudolfix in https://github.com/dlt-hub/dlt/pull/990
Introduce hard_delete and dedup_sort columns hint for merge by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/960 and https://dlthub.com/docs/general-usage/incremental-loading#delete-records
adjustment of pua start in typed json encoding, pass through on decoding errors by @rudolfix in https://github.com/dlt-hub/dlt/pull/974
creates isolated parallel Airflow DAGs in airflow helper to execute resources parallel in isolated pipelines @IlyaFaer in https://github.com/dlt-hub/dlt/pull/979
Fix annotation processing and rebuilding, mark dataclass as complex by @sultaniman in https://github.com/dlt-hub/dlt/pull/980
allows async functions to be decorated with dlt.source by @rudolfix in https://github.com/dlt-hub/dlt/pull/985
allows right pipe operator to feed simple lists into a transformer @rudolfix in https://github.com/dlt-hub/dlt/pull/985
allows pendulum datetime as incremental cursor when loading arrow tables @rudolfix in https://github.com/dlt-hub/dlt/pull/985
enables Python 3.12 (mind that not all extras have python 3.12 libraries!) @rudolfix in https://github.com/dlt-hub/dlt/pull/985

Docs

docs(filesystem): include Google Drive into filesystem tutorial by @IlyaFaer in https://github.com/dlt-hub/dlt/pull/962
Fix typos/grammar in tutorial docs by @taljaards in https://github.com/dlt-hub/dlt/pull/972
add blog post observability by @adrianbr in https://github.com/dlt-hub/dlt/pull/989
Update arrow-pandas.md by @snehangsude in https://github.com/dlt-hub/dlt/pull/992
Clarify info about GoodData in modelling tools article by @mhauzirek in https://github.com/dlt-hub/dlt/pull/956
Fix small typings in contributing guide by @VioletM in https://github.com/dlt-hub/dlt/pull/993
Docs/google sheets update by @dat-a-man in https://github.com/dlt-hub/dlt/pull/976
Added "Incremental Configuration" section to SQL Databases documentat… by @dat-a-man in https://github.com/dlt-hub/dlt/pull/977

Verified Sources

Bing Webmaster source by @willi-mueller

New Contributors

@taljaards made their first contribution in https://github.com/dlt-hub/dlt/pull/972
@mhauzirek made their first contribution in https://github.com/dlt-hub/dlt/pull/956
@snehangsude made their first contribution in https://github.com/dlt-hub/dlt/pull/992
@VioletM made their first contribution in https://github.com/dlt-hub/dlt/pull/993

Full Changelog: https://github.com/dlt-hub/dlt/compare/0.4.4...0.4.5

0.4.4

3 months ago

Core Library

passes incremental from apply hints to resource function by @rudolfix in https://github.com/dlt-hub/dlt/pull/953
Handle UnionType when checking is_union_type and is_optional_type by @sultaniman in https://github.com/dlt-hub/dlt/pull/951
yanks orjson to <=0.3.10 by @rudolfix in https://github.com/dlt-hub/dlt/pull/958

Docs

Databricks workspace setup docs by @steinitzu in https://github.com/dlt-hub/dlt/pull/949

Verified Source

allows for table reflection at runtime, column selection and buffer control in sql_database @rudolfix (https://github.com/dlt-hub/verified-sources/pull/351)

Full Changelog: https://github.com/dlt-hub/dlt/compare/0.4.3...0.4.4

0.4.3

3 months ago

Core Library

Databricks destination by @steinitzu and @phillem15 in https://github.com/dlt-hub/dlt/pull/892
Synapse destination by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/900
BigQuery Partitioning Improvements by @Pipboyguy in https://github.com/dlt-hub/dlt/pull/887
enable async generators as resources by @sh-rp in https://github.com/dlt-hub/dlt/pull/905
fix: use truthy value in ternary since 0 cause div by zero by @z3z1ma in https://github.com/dlt-hub/dlt/pull/902
feat(filesystem): add compression flag if the read file is GZ by @IlyaFaer in https://github.com/dlt-hub/dlt/pull/912
Enhancements in Filesystem Configuration by @Pipboyguy in https://github.com/dlt-hub/dlt/pull/869
add mark function to emit resource hints from decorated function by @rudolfix in https://github.com/dlt-hub/dlt/pull/938
handles nested Pydantic models when generating dlt schema by @sultaniman in https://github.com/dlt-hub/dlt/pull/901

Docs

Restructure intro, getting started and tutorial by @burnash in https://github.com/dlt-hub/dlt/pull/702
Update the release instructions in CONTRIBUTING.md by @burnash in https://github.com/dlt-hub/dlt/pull/867
Add explicit sub section about streamlit under getting started by @sultaniman in https://github.com/dlt-hub/dlt/pull/884
Examples: google sheets by @AstrakhantsevaAA in https://github.com/dlt-hub/dlt/pull/846
Added URL-parser documentation by @dat-a-man in https://github.com/dlt-hub/dlt/pull/909

Verified Sources

feat(filesystem): implement a csv reader with duckdb engine @IlyaFaer https://github.com/dlt-hub/verified-sources/pull/319
fix(notion): define payload within the while-loop @glebzhidkov (https://github.com/dlt-hub/verified-sources/pull/338)
sql alchemy + connector x example @rudolfix (https://github.com/dlt-hub/verified-sources/pull/334)
Shopify: Standalone resource for partner API queries @steinitzu (https://github.com/dlt-hub/verified-sources/pull/329)
sql-database: detect precision and scale of supported column types @steinitzu (https://github.com/dlt-hub/verified-sources/pull/324)
feat(sources.kafka): implement Kafka source @IlyaFaer (https://github.com/dlt-hub/verified-sources/pull/306)

New Contributors

@Pipboyguy made their first contribution in https://github.com/dlt-hub/dlt/pull/869
@sultaniman made their first contribution in https://github.com/dlt-hub/dlt/pull/883

Full Changelog: https://github.com/dlt-hub/dlt/compare/0.4.2...0.4.3

0.4.2

4 months ago

Core Library

Fix the data type used in the from_db_type() method from MsSqlTypeMapper by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/863
Use Secret Manager in CI by @AstrakhantsevaAA in https://github.com/dlt-hub/dlt/pull/859
Move destination adapters to dlt.destination.adapters by @rudolfix in https://github.com/dlt-hub/dlt/pull/854

Docs

Improve HubSpot source docs by @IlyaFaer in https://github.com/dlt-hub/dlt/pull/864
Add new topic to docs: Destination; improve Configuration docs by @rudolfix in https://github.com/dlt-hub/dlt/pull/861

Full Changelog: https://github.com/dlt-hub/dlt/compare/0.4.1...0.4.2

0.4.1

4 months ago

Major release

This is a major dlt release (as per our semantic versioninghttps://github.com/dlt-hub/dlt?tab=readme-ov-file#adding-as-dependency). It brings several interesting new features like: schema evolution control, data contracts, deeper Pydantic integration, parametrized destinations, improvements to parallelism and data lineage + many more

There are no significant breaking changes, but minor ones exist, please refer to #763 for details

Core Library

Parametrized destinations - import destinations from dlt.destinations module and instantiate them: by @steinitzu in https://github.com/dlt-hub/dlt/pull/746
schema and data contracts by @sh-rp in https://github.com/dlt-hub/dlt/pull/594
load package id in extract step by @rudolfix in https://github.com/dlt-hub/dlt/pull/790
named destinations: configure many destinations with different names by @sh-rp in https://github.com/dlt-hub/dlt/pull/783
rich tracing information from pipeline steps (extract, normalize, load) by @rudolfix in https://github.com/dlt-hub/dlt/pull/801
adds exception stack to pipeline trace by @rudolfix in https://github.com/dlt-hub/dlt/pull/806
fixed attribute check: getuid -> geteuid by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/823
allows to run parallel pipelines in separate threads by @rudolfix in https://github.com/dlt-hub/dlt/pull/813
791 test mssql credentialspy is odbc driver 18 dependent by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/834
adds extract and normalize traces by @rudolfix in https://github.com/dlt-hub/dlt/pull/839

Plus some tooling changes

introduce black formatting by @sh-rp in https://github.com/dlt-hub/dlt/pull/583
Fix: ensure accessor typing does not make static type checker error by @z3z1ma in https://github.com/dlt-hub/dlt/pull/785
Hot fix: add skipifgithubfork to nested_data example by @AstrakhantsevaAA in https://github.com/dlt-hub/dlt/pull/802
Fix Windows lint issue and implement CI lint matrix strategy by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/827

Docs

documents schema and data contract by @rudolfix in https://github.com/dlt-hub/dlt/pull/782
Added Kinesis documentation. by @dat-a-man in https://github.com/dlt-hub/dlt/pull/804
788 clarify docs intro by @deanja in https://github.com/dlt-hub/dlt/pull/797
Fix links to source code by @AstrakhantsevaAA in https://github.com/dlt-hub/dlt/pull/805
Clarify docs dev process by @deanja in https://github.com/dlt-hub/dlt/pull/809
Qdrant ingestion pipeline example eg by @hibajamal in https://github.com/dlt-hub/dlt/pull/775
Personio doc: added more endpoints by @AstrakhantsevaAA in https://github.com/dlt-hub/dlt/pull/829

New Contributors

@deanja made their first contribution in https://github.com/dlt-hub/dlt/pull/797
@IlyaFaer made their first contribution in https://github.com/dlt-hub/dlt/pull/820
@jorritsandbrink made their first contribution in https://github.com/dlt-hub/dlt/pull/823

Full Changelog: https://github.com/dlt-hub/dlt/compare/0.3.25...0.4.1

0.4.1a2

4 months ago

🧪 pre-release of 0.4.x (do not use in production)

fixed attribute check: getuid -> geteuid by @jorritsandbrink in https://github.com/dlt-hub/dlt/pull/823
allows to run parallel pipelines in separate threads by @rudolfix in https://github.com/dlt-hub/dlt/pull/813
parallel pipelines docs update: https://dlthub.com/devel/reference/performance#running-several-pipelines-in-parallel-in-single-process

New Contributors

@IlyaFaer made their first contribution in https://github.com/dlt-hub/dlt/pull/820
@jorritsandbrink made their first contribution in https://github.com/dlt-hub/dlt/pull/823

Full Changelog: https://github.com/dlt-hub/dlt/compare/0.4.1a1...0.4.1a2