data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
Be it a Google Colab notebook, AWS Lambda function, an Airflow DAG, your local laptop,
or a GPT-4 assisted development playground—dlt can be dropped in anywhere.
🚀 Join our thriving community of likeminded developers and build the future together!
dlt supports Python 3.8+.
pip install dlt
Load chess game data from chess.com API and save it in DuckDB:
import dlt
from dlt.sources.helpers import requests
# Create a dlt pipeline that will load
# chess player data to the DuckDB destination
pipeline = dlt.pipeline(
pipeline_name='chess_pipeline',
destination='duckdb',
dataset_name='player_data'
)
# Grab some player data from Chess.com API
data = []
for player in ['magnuscarlsen', 'rpragchess']:
response = requests.get(f'https://api.chess.com/pub/player/{player}')
response.raise_for_status()
data.append(response.json())
# Extract, normalize, and load the data
pipeline.run(data, table_name='player')
Try it out in our Colab Demo
Explore ready to use sources (e.g. Google Sheets) in the Verified Sources docs and supported destinations (e.g. DuckDB) in the Destinations docs.
For detailed usage and configuration, please refer to the official documentation.
You can find examples for various use cases in the examples folder.
dlt
follows the semantic versioning with the MAJOR.MINOR.PATCH
pattern. Currently, we are using pre-release versioning with the major version being 0.
minor
version change means breaking changespatch
version change means new features that should be backward compatiblepost10
-> post11
, is considered a patchWe suggest that you allow only patch
level updates automatically:
The dlt project is quickly growing, and we're excited to have you join our community! Here's how you can get involved:
dlt
is released under the Apache 2.0 License.