Apache Superset is a Data Visualization and Data Exploration Platform

react python flask data-science bi analytics superset apache data-visualization data-engineering business-intelligence data-viz data-analytics data-analysis sql-editor asf business-analytics apache-superset

• Updated Nov 12, 2026
• Python

Learn how to develop, deploy and iterate on production-grade ML applications.

python data-science machine-learning natural-language-processing deep-learning pytorch data-engineering ray data-quality distributed-training mlops distributed-ml llms llmops

• Updated Mar 4, 2026
• Jupyter Notebook

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

python workflow data-science machine-learning airflow automation etl workflow-engine scheduler apache orchestration data-engineering data-integration elt data-pipelines dag apache-airflow mlops workflow-orchestration data-orchestrator

• Updated Nov 12, 2026
• Python

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼

docker course kafka spark data-engineering free dbt kestra

• Updated Oct 15, 2026
• Jupyter Notebook

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

search data-science machine-learning natural-language-processing reinforcement-learning computer-vision deep-learning production data-engineering data-discovery recsys data-quality applied-data-science applied-machine-learning

• Updated Jul 18, 2024

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

java workflow devops automation pipeline etl scheduler orchestration data-engineering infrastructure-as-code high-availability control-plane hacktoberfest low-code ai-agents pipeline-as-code infra-ops infra-automation data-orchestration data-orchestrator

• Updated Nov 12, 2026
• Java

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

python infrastructure workflow data-science data automation pipeline workflow-engine orchestration data-engineering observability prefect data-ops ml-ops

• Updated Nov 12, 2026
• Python

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

mysql python java bigquery data pipeline etl postgresql s3 snowflake self-hosted data-engineering data-analysis mssql data-integration data-collection redshift elt data-pipeline change-data-capture

• Updated Nov 12, 2026
• Python

Turns Data and AI algorithms into production-ready web applications in no time.

python workflow automation pipeline job-scheduler pipelines data-visualization orchestration datascience data-engineering developer-tools data-integration scenario hacktoberfest mlops data-ops scenario-analysis hacktoberfest2023 taipy-core taipy-gui

• Updated Aug 10, 2026
• Python

Workflow Engine for Kubernetes

kubernetes workflow machine-learning airflow workflow-engine cncf pipelines data-engineering argo k8s cloud-native batch-processing hacktoberfest dag gitops mlops knative argo-workflows

• Updated Nov 12, 2026
• Go

An orchestration platform for the development, production, and observation of data assets.

python metadata workflow data-science etl analytics scheduler orchestration data-engineering data-integration data-pipelines workflow-automation mlops dagster data-orchestrator

• Updated Nov 10, 2026
• Python

The Data Engineering Cookbook

big-data best-practices cookbook data-engineering data-engineer

• Updated Oct 16, 2026
• Python

Graph-Native Infrastructure for Context and Accountable AI Systems

python ai ontology provenance artificial-intelligence data-engineering knowledge-graph developer-tools semantic-search reasoning explainable-ai decision-intelligence llm ai-governance generative-ai graph-rag ai-infrastructure agent-memory context-engineering context-graphs

• Updated Nov 12, 2026
• Python

Roadmap to becoming a data engineer in 2021

roadmap cloud data-engineering data-engineer-roadmap

• Updated Jan 25, 2022

Always know what to expect from your data.

data-science pipeline exploratory-data-analysis eda data-engineering data-quality data-profiling datacleaner exploratory-analysis cleandata dataquality datacleaning mlops pipeline-tests pipeline-testing dataunittest data-unit-tests exploratorydataanalysis pipeline-debt data-profilers

• Updated Nov 12, 2026
• Python

Incremental engine for long horizon agents 🌟 Star if you like it!

python rust real-time ai etl indexing data-engineering knowledge-graph help-wanted data-processing semantic-search change-data-capture ai-agents data-indexing rag llm context-engineering codebase-intelligence agentic-data-framework long-horizon-agent

• Updated Nov 11, 2026
• Rust

🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.

raspberry-pi devops nix iterm2 nixos hyper data-engineering vscode-extension data-scientists kitty xonsh security-automation devsecops claude python-automation cmux wezterm anthropic claude-code

• Updated Nov 9, 2026
• Python

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

rust database kafka webhook postgresql stream-processing data-engineering materialized-view etl-pipeline event-streaming apache-iceberg

• Updated Nov 12, 2026
• Rust

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

• Updated Oct 23, 2026
• Python

Fancy stream processing made operationally mundane

go golang kafka cqrs etl rabbitmq amqp logs message-bus event-sourcing nats stream-processing message-queue data-engineering streaming-data stream-processor data-ops

• Updated Nov 12, 2026
• Go