DataFusion is an extensible query planning, optimization, and execution framework, written in
Rust, that uses Apache Arrow as its
- SQL query planner with support for multiple SQL dialects
- DataFrame API
- Parquet, CSV, JSON, and Avro file formats are supported natively. Custom
file formats can be supported by implementing a
- Supports popular object stores, including AWS S3, Azure Blob
Storage, and Google Cloud Storage. There are extension points for implementing
custom object stores.
DataFusion is modular in design with many extension points and can be
used without modification as an embedded query engine and can also provide
a foundation for building new systems. Here are some example use cases:
- DataFusion can be used as a SQL query planner and query optimizer, providing
optimized logical plans that can then be mapped to other execution engines.
- DataFusion is used to create modern, fast and efficient data
pipelines, ETL processes, and database systems, which need the
performance of Rust and Apache Arrow and want to provide their users
the convenience of an SQL interface or a DataFrame API.
High Performance: Leveraging Rust and Arrow's memory model, DataFusion achieves very high performance
Easy to Connect: Being part of the Apache Arrow ecosystem (Arrow, Parquet and Flight), DataFusion works well with the rest of the big data ecosystem
Easy to Embed: Allowing extension at almost any point in its design, DataFusion can be tailored for your specific use case
High Quality: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can be used as the foundation for production systems.
There are a number of community projects that extend DataFusion or provide integrations with other systems.
Here are some of the projects known to use DataFusion:
(if you know of another project, please submit a PR to add a link!)
Please see example usage to find how to use DataFusion.
Please see Roadmap for information of where the project is headed.
There is no formal document describing DataFusion's architecture yet, but the following presentations offer a good overview of its different components and how they interact together.
- (July 2022): DataFusion and Arrow: Supercharge Your Data Analytical Tool with a Rusty Query Engine: recording and slides
- (March 2021): The DataFusion architecture is described in Query Engine Design and the Rust-Based DataFusion in Apache Arrow: recording (DataFusion content starts ~ 15 minutes in) and slides
- (February 2021): How DataFusion is used within the Ballista Project is described in *Ballista: Distributed Compute with Rust and Apache Arrow: recording
Please see User Guide for more information about DataFusion.
Please see Contributor Guide for information about contributing to DataFusion.