Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

python rust streaming real-time kafka etl machine-learning-algorithms stream-processing data-analytics dataflow data-processing data-pipelines batch-processing pathway iot-analytics etl-framework time-series-analysis

• Updated Nov 12, 2026
• Python

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

python workflow data-science machine-learning airflow automation etl workflow-engine scheduler apache orchestration data-engineering data-integration elt data-pipelines dag apache-airflow mlops workflow-orchestration data-orchestrator

• Updated Nov 12, 2026
• Python

An orchestration platform for the development, production, and observation of data assets.

python metadata workflow data-science etl analytics scheduler orchestration data-engineering data-integration data-pipelines workflow-automation mlops dagster data-orchestrator

• Updated Nov 10, 2026
• Python

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

nlp pdf machine-learning natural-language-processing information-retrieval ocr deep-learning ml docx preprocessing pdf-to-text data-pipelines donut document-image-processing document-parser pdf-to-json document-image-analysis llm document-parsing langchain

• Updated Nov 6, 2026
• HTML

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

workflow airflow job-scheduler orchestration cloud-native task-scheduler data-pipelines azkaban workflow-orchestration workflow-schedule powerful-data-pipelines

• Updated Nov 12, 2026
• Java

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

python machine-learning data-pipelines hacktoberfest machine-learning-engineering mlops kedro agentic-workflow agentic-ai

• Updated Nov 10, 2026
• Python

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

• Updated Oct 23, 2026
• Python

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

data-science data operators data-processing data-pipelines data-cleaning data-synthesis gradio-interface llms data-agent vllm-backend sglang-bankend quick-data-processing

• Updated Nov 11, 2026
• Python

🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

rust distributed-systems streaming real-time serverless webassembly data-flow stream-processing data-analytics data-integration cloud-native data-pipelines stateful streaming-data stream-processing-engine event-driven-architecture streaming-analytics streaming-data-processing streaming-data-pipelines

• Updated Nov 5, 2026
• Rust

Preswald is a WASM packager for Python-based interactive data apps: bundle full complex data workflows, particularly visualizations, into single files, runnable completely in-browser, using Pyodide, DuckDB, Pandas, and Plotly, Matplotlib, etc. Build dashboards, reports, and notebooks that run offline, load fast, and share like a document.

python open-source data ai analytics vscode data-visualization gpt copilot data-pipelines schema-management data-infrastructure analytics-engineering data-sdk llm data-applications

• Updated Jun 11, 2026
• Python

A system for agentic LLM-powered data processing and ETL

python workflow data etl semantic-data elt data-pipelines agents document-analysis document-processing unstructured-data unstructured-data-analysis llm

• Updated Nov 9, 2026
• Python

Build data pipelines, the easy way 🛠️

python docker kubernetes data-science machine-learning airflow cloud deployment jupyter etl ide pipelines self-hosted jupyterlab notebooks data-pipelines dag etl-pipeline orchest

• Updated Jun 6, 2023
• TypeScript

Maestro: Netflix’s Workflow Orchestrator

java workflow data-science machine-learning automation etl analytics workflow-engine scheduler orchestration data-engineering elt data-pipelines batch-processing dag mlops data-ops workflow-orchestration data-orchestrator agentic-workflow

• Updated Nov 6, 2026
• Java

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

open-source tap data opensource integration pipelines data-engineering target dataops loaders elt extract-data data-pipelines singer connectors targets taps meltano dataops-platform meltano-sdk

• Updated Nov 12, 2026
• Python

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

bigquery snowflake data-warehouse dataops data-analysis redshift dbt data-pipelines data-pipeline lineage data-governance data-lineage analytics-engineer dbt-packages data-observability data-reliability dbt-artifacts

• Updated Nov 5, 2026
• HTML

The Feldera Incremental Computation Engine

rust streaming real-time sql database data-analytics data-pipelines materialized-views incremental-computation ivm incremental-view-maintenance

• Updated Nov 12, 2026
• Rust

The best place to learn data engineering. Built and maintained by the data engineering community.

data sql database etl data-engineering data-pipelines data-modeling data-engineer

• Updated Sep 15, 2026
• CSS

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

python bigquery sql analytics data-transformation snowflake data-platform data-analysis data-pipelines data-modeling data-ingestion

• Updated Nov 12, 2026
• Go

Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.

nlp rust pdf machine-learning natural-language-processing ocr etl tika extraction docx data-pipelines pdf-parser unstructured unstructured-data rag etl-pipelines llm

• Updated Dec 21, 2024
• Rust

MLeap: Deploy ML Pipelines to Production

python scala spark tensorflow scikit-learn transformers data-pipelines

• Updated Nov 9, 2026
• Scala