Scalable data pre processing and curation toolkit for LLMs

python data data-processing data-preparation deduplication data-quality data-curation data-prep fine-tuning fast-data-processing data-processing-pipelines datacuration large-language-models llm llmapps large-scale-data-processing datarecipes semantic-deduplication llm-data-quality

• Updated Nov 11, 2026
• Python

Open source project for data preparation for GenAI applications

python data spark malware code-quality data-preprocessing ray data-preparation deduplication data-prep finetuning data-preprocessing-pipelines datacuration large-language-models llm llmapps large-scale-data-processing datarecipes

• Updated Sep 29, 2026
• HTML

EN : Water Pipe Leakage Risk Prediction System v0-3 | Industrial-grade GPR system achieving R²=0.9894 accuracy with 1,907x parallel processing efficiency | Complete scalability proven from N=44 to N=6000 datasets. JP : 水道管漏水リスク予測GPRシステム v0-3 | 産業レベル予測精度R²=0.9894、1,907倍並列処理効率化を実現したガウス過程回帰による高精度漏水リスク分析システム | N=44→N=6000完全スケーラビリティ実証済み

infrastructure-monitoring regression-models gpr predictive-maintenance ground-penetrating-radar cbm industrial-ai risk-prediction large-scale-data-processing subsurface-imaging sensor-data-analysis civil-engineering-ai decision-support-ai water-pipe-leakage pipeline-risk-prediction utility-maintenance

• Updated Sep 21, 2025
• Python