Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

python rust streaming real-time kafka etl machine-learning-algorithms stream-processing data-analytics dataflow data-processing data-pipelines batch-processing pathway iot-analytics etl-framework time-series-analysis

• Updated Nov 12, 2026
• Python

High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.

python machine-learning typescript sdk ai cpp mcp vscode-extension data-processing data-pipeline

• Updated Nov 12, 2026
• Python

Incremental engine for long horizon agents 🌟 Star if you like it!

python rust real-time ai etl indexing data-engineering knowledge-graph help-wanted data-processing semantic-search change-data-capture ai-agents data-indexing rag llm context-engineering codebase-intelligence agentic-data-framework long-horizon-agent

• Updated Nov 11, 2026
• Rust

A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.

linux shell bash terminal system hardware grep data-processing variables xargs xwindow one-liners linux-administration oneliner-commands shell-oneliner

• Updated Sep 22, 2026

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

tsv devops json statistics csv command-line json-data tabular-data data-reduction unix-toolkit statistical-analysis csv-format devops-tools data-regression data-processing command-line-tools data-cleaning streaming-algorithms streaming-data miller

• Updated Nov 10, 2026
• Go

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

data-science data operators data-processing data-pipelines data-cleaning data-synthesis gradio-interface llms data-agent vllm-backend sglang-bankend quick-data-processing

• Updated Nov 11, 2026
• Python

Unified querying, transformation, and modification of JSON, TOML, YAML, XML, INI, HCL, KDL and CSV.

config go cli golang yaml toml parser json query xml configuration update selector data-structures data-wrangling devops-tools data-processing yaml-processor json-processing hcl2

• Updated Nov 2, 2026
• Go

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

data-science data data-visualization data-analysis data-processing multi-modal data-pipeline synthetic-data pre-training foundation-models large-language-models llm llms instruction-tuning

• Updated Nov 12, 2026
• Python

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

python machine-learning deep-learning neural-network mxnet gpu image-processing pytorch gpu-tensorflow data-processing data-augmentation audio-processing paddle image-augmentation fast-data-pipeline

• Updated Nov 12, 2026
• C++

A lightweight data processing framework built on DuckDB and 3FS.

data-processing duckdb

• Updated Mar 5, 2025
• Python

A light-weight, flexible, and expressive statistical data testing library

testing schema validation data-validation pandas-dataframe assertions pandas testing-tools data-processing dataframes data-cleaning hypothesis-testing data-verification pandas-validation data-check data-assertions dataframe-schema pandas-validator

• Updated Nov 11, 2026
• Python

Kubernetes-native platform to run massively parallel data/streaming jobs

kubernetes pipeline stream-processing map-reduce k8s data-processing hacktoberfest

• Updated Nov 10, 2026
• Rust

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

knowledge-base data-processing codex ai-agents unstructured-data multimodal mlops pydantic claude-code harness-engineering data-context-layer

• Updated Nov 12, 2026
• Python

Concurrent and multi-stage data ingestion and data processing with Elixir

elixir broadway concurrent data-processing genstage data-ingestion

• Updated Nov 6, 2026
• Elixir

Large-scale pretraining for dialogue

machine-learning dialogue text-generation pytorch transformer data-processing text-data gpt-2 dialogpt

• Updated Oct 17, 2022
• Python

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

python machine-learning natural-language-processing deep-learning tensorflow machine-translation text-generation data-processing bert text-data dialog-systems gpt-2 texar xlnet casl-project

• Updated Jul 21, 2026
• Python

Python Stream Processing

python rust data-science machine-learning stream-processing data-engineering dataflow data-processing streaming-data

• Updated Nov 8, 2026
• Python

Scalable data pre processing and curation toolkit for LLMs

python data data-processing data-preparation deduplication data-quality data-curation data-prep fine-tuning fast-data-processing data-processing-pipelines datacuration large-language-models llm llmapps large-scale-data-processing datarecipes semantic-deduplication llm-data-quality

• Updated Nov 11, 2026
• Python

Extract Transform Load for Python 3.5+

automation parallelization python3 data-processing bonobo extract-transform-load

• Updated May 12, 2023
• Python

Data and tools for generating and inspecting OLMo pre-training data.

nlp data-processing machile-learning large-language-models llm

• Updated Aug 24, 2026
• Python