Apache DataFusion#
DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory format.
The documentation on this site is for the core DataFusion project, which contains libraries and binaries for developers building fast and feature rich database and analytic systems, customized to particular workloads. See use cases for examples.
The following related subprojects target end users and have separate documentation.
DataFusion Python offers a Python interface for SQL and DataFrame queries.
DataFusion Java offers a Java interface for SQL and DataFrame queries.
DataFusion Comet is an accelerator for Apache Spark based on DataFusion.
DataFusion Ballista is distributed processing extension for DataFusion.
“Out of the box,” DataFusion offers SQL and Dataframe APIs, excellent performance, built-in support for CSV, Parquet, JSON, and Avro, extensive customization, and a great community. Python Bindings are also available. Ballista is Apache DataFusion extension enabling the parallelized execution of workloads across multiple nodes in a distributed environment.
DataFusion features a full query planner, a columnar, streaming, multi-threaded, vectorized execution engine, and partitioned data sources. You can customize DataFusion at almost all points including additional data sources, query languages, functions, custom operators and more. See the Architecture section for more details.
To get started, see
The example usage section of the user guide and the datafusion-examples directory.
The library user guide for examples of using DataFusion’s extension APIs
The developer’s guide for contributing and communication for getting in touch with us.
ASF Links
User Guide
- Introduction
- Example Usage
- Features
- Concepts, Readings, Events
- 🎥 Community Showcase
- 🌎 Community Events
- Crate Configuration
- DataFusion CLI
- DataFrame API
- Gentle Arrow Introduction
- Expression API
- SQL Reference
- Configuration Settings
- Runtime Configuration Settings
- Tuning Guide
- Join Algorithm Optimizer Configurations
- Reading Explain Plans
- Parquet Content-Defined Chunking
- Metrics
- Frequently Asked Questions
- How does DataFusion Compare with
XYZ?
Library User Guide
- Introduction
- Securing DataFusion
- Upgrade Guides
- Extensions List
- Using the SQL API
- Extending SQL Syntax
- Working with
Exprs - Using the DataFrame API
- Write DataFrame to Files
- Building Logical Plans
- Catalogs, Schemas, and Tables
- Functions
- Custom Table Provider
- Table Constraint Enforcement
- Extending Operators
- Profiling Cookbook
- Query Optimizer
Contributor Guide
- Introduction
- Developer’s guide
- AI Policy
- Reviewing Pull Requests
- Community Communication
- Development Environment
- Architecture
- Workspace Dependency Graph
- Testing
- API health policy
- HOWTOs
- Release Management
- Roadmap and Improvement Proposals
- Governance
- Inviting New Committers and PMC Members
- Specifications
- Google Summer of Code (GSOC)
DataFusion Subprojects