Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

python rust streaming real-time kafka etl machine-learning-algorithms stream-processing data-analytics dataflow data-processing data-pipelines batch-processing pathway iot-analytics etl-framework time-series-analysis

• Updated Nov 12, 2026
• Python

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

python workflow data-science machine-learning airflow automation etl workflow-engine scheduler apache orchestration data-engineering data-integration elt data-pipelines dag apache-airflow mlops workflow-orchestration data-orchestrator

• Updated Nov 12, 2026
• Python

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

java workflow devops automation pipeline etl scheduler orchestration data-engineering infrastructure-as-code high-availability control-plane hacktoberfest low-code ai-agents pipeline-as-code infra-ops infra-automation data-orchestration data-orchestrator

• Updated Nov 12, 2026
• Java

A high-performance observability data pipeline.

agent events monitoring etl metrics high-performance telemetry logs data-transformation pipelines datadog stream-processing rust-lang cloud-native forwarder hacktoberfest observability traces

• Updated Nov 12, 2026
• Rust

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

mysql python java bigquery data pipeline etl postgresql s3 snowflake self-hosted data-engineering data-analysis mssql data-integration data-collection redshift elt data-pipeline change-data-capture

• Updated Nov 12, 2026
• Python

An orchestration platform for the development, production, and observation of data assets.

python metadata workflow data-science etl analytics scheduler orchestration data-engineering data-integration data-pipelines workflow-automation mlops dagster data-orchestrator

• Updated Nov 10, 2026
• Python

Incremental engine for long horizon agents 🌟 Star if you like it!

python rust real-time ai etl indexing data-engineering knowledge-graph help-wanted data-processing semantic-search change-data-capture ai-agents data-indexing rag llm context-engineering codebase-intelligence agentic-data-framework long-horizon-agent

• Updated Nov 11, 2026
• Rust

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

• Updated Oct 23, 2026
• Python

Fancy stream processing made operationally mundane

go golang kafka cqrs etl rabbitmq amqp logs message-bus event-sourcing nats stream-processing message-queue data-engineering streaming-data stream-processor data-ops

• Updated Nov 12, 2026
• Go

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

kubernetes golang aws security devops cis cloud etl azure sqlite terraform postgresql gcp hacktoberfest devsecops cspm steampipe postgresql-fdw cnapp zero-etl

• Updated Nov 12, 2026
• Go

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

go kubernetes github-api bigquery aws data google sql etl azure gcp data-engineering data-analysis data-integration data-collection elt etl-framework cspm airbyte attack-surface-management

• Updated Nov 11, 2026
• Go

Flink CDC is a streaming data integration tool

mysql real-time kafka etl postgresql distributed batch data-integration schema-evolution elt flink cdc data-pipeline change-data-capture paimon

• Updated Nov 12, 2026
• Java

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

python rust distributed-systems machine-learning big-data etl arrow distributed-computing distributed embeddings artificial-intelligence data-engineering parquet ray iceberg multimodal ai-pipeline huggingface ai-engineering

• Updated Nov 9, 2026
• Rust

Privacy and Security focused Segment-alternative, in Golang and React

bigquery privacy etl snowflake data-warehouse data-engineering data-integration redshift elt cdp warehouse-management data-synchronization data-pipeline customer-data event-streaming customer-data-platform segment-alternative customer-data-pipeline customer-data-lake warehouse-native

• Updated Nov 12, 2026
• Go

A system for agentic LLM-powered data processing and ETL

python workflow data etl semantic-data elt data-pipelines agents document-analysis document-processing unstructured-data unstructured-data-analysis llm

• Updated Nov 9, 2026
• Python

Build data pipelines, the easy way 🛠️

python docker kubernetes data-science machine-learning airflow cloud deployment jupyter etl ide pipelines self-hosted jupyterlab notebooks data-pipelines dag etl-pipeline orchest

• Updated Jun 6, 2023
• TypeScript

Open Source Data Security Platform for Developers to Monitor and Detect PII, Anonymize Production Data and Sync it across environments.

mysql testing docker kubernetes golang open-source typescript etl reactjs nextjs postgresql faker self-hosted orchestration test-data-generator synthetic-data fine-tuning benthos synthetic-data-generation

• Updated Aug 30, 2025
• Go

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

mysql python emr aws data-science lambda aws-lambda athena etl pandas data-engineering redshift ray apache-parquet amazon-athena apache-arrow aws-glue modin glue-catalog amazon-sagemaker-notebook

• Updated Nov 8, 2026
• Python

Maestro: Netflix’s Workflow Orchestrator

java workflow data-science machine-learning automation etl analytics workflow-engine scheduler orchestration data-engineering elt data-pipelines batch-processing dag mlops data-ops workflow-orchestration data-orchestrator agentic-workflow

• Updated Nov 6, 2026
• Java

A curated list with resources about node-based UIs

etl awesome-list visual-programming workflow-editor node-based-ui

• Updated Jun 29, 2025