This is a repo with links to everything you'd ever want to learn about data engineering

data awesome sql bigdata dataengineering apachespark

• Updated Aug 3, 2026
• Jupyter Notebook

RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.

rust filesystem bigdata s3 minio cloud-native objectstorage object-storage multi-cloud amazon-s3 ai-storage ai-native

• Updated Nov 12, 2026
• Rust

High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios

iot sql database monitoring time-series metrics cluster bigdata scalability distributed cloud-native financial-analysis tsdb time-series-database connected-vehicles industrial-iot tdengine

• Updated Nov 7, 2026
• C

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

mysql sql database bigdata postgresql shard distributed-database encrypt data-pipeline data-encryption database-cluster distributed-transaction read-write-splitting database-middleware distributed-sql-database database-gateway

• Updated Nov 12, 2026
• Java

大数据入门指南 ⭐

phoenix scala kafka big-data spark yarn hive hadoop storm bigdata hbase zookeeper hdfs mapreduce flume azkaban sqoop

• Updated Jan 5, 2024
• Java

A curated list of awesome big data frameworks, ressources and other awesomeness.

data-science data awesome database data-stream bigdata series-database data-visualization data-warehouse stream-processing data-analytics awesome-list distributed-database visualize-data streaming-data

• Updated Nov 6, 2026

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

go redis golang distributed-systems storage filesystem bigdata s3 posix hdfs cloud-native object-storage

• Updated Nov 12, 2026
• Go

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

kafka spark hive hadoop bigdata hbase zookeeper hdfs flume flink azkaban

• Updated Aug 7, 2023

Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.

rust elasticsearch sql database ai serverless geospatial bigdata snowflake olap cloud-native vector-search vector-database lakehouse

• Updated Nov 12, 2026
• Rust

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

visualization python data-science machine-learning bigdata tabular-data hdf5 machinelearning dataframe memory-mapped-file pyarrow

• Updated Nov 6, 2026
• Python

Upserts, Deletes And Incremental Processing on Big Data.

bigdata stream-processing data-integration datalake apachespark hudi apachehudi incremental-processing apacheflink

• Updated Nov 12, 2026
• Java

A Cloud Native Batch System (Project under CNCF)

training kubernetes golang machine-learning gene ai hpc bigdata serving batch-systems

• Updated Nov 12, 2026
• Go

100+套大数据可视化炫酷大屏Html5模板；包含行业：社区、物业、政务、交通、金融银行等，全网最新、最多，最全、最酷、最炫大数据可视化模板。陆续更新中

bigdata html-template viewmodel echarts bigdataviewer

• Updated Oct 24, 2025
• JavaScript

A data integration framework

java framework bigdata data-integration flink

• Updated Oct 9, 2026
• Java

🔨 用 JSON 来生成结构化的 SQL 语句，基于 Vue3 + TypeScript + Vite + Ant Design + MonacoEditor 实现，项目简单（重逻辑轻页面）、适合练手~

javascript mysql json typescript sql spark vue hive bigdata monaco-editor ant-design vue3 vite

• Updated Jan 17, 2024
• Vue

Apache Avro is a data serialization system.

ruby python c java php csharp cplusplus avro dotnet perl bigdata

• Updated Nov 12, 2026
• Java

大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料

scala kafka spark hive hadoop bigdata hbase zookeeper flume flink javase

• Updated Jan 20, 2026

Python clone of Spark, a MapReduce alike framework in Python

python spark bigdata stream-processing mapreduce dpark

• Updated Dec 25, 2020
• Python

GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.

fast iot sql database timeseries time-series nosql bigdata newsql griddb

• Updated Mar 19, 2026
• C++

Aix-DB 基于 LangChain/LangGraph 框架，结合 MCP Skills 多智能体协作架构，实现自然语言到数据洞察的端到端转换。

python cli neo4j mcp bigdata sanic hermes echarts dify vue3 text2sql langchain qwen lamaindex langgraph-python deepseek-r1 skils openclaw

• Updated Nov 6, 2026
• JavaScript