Package and access data on Amazon S3 in a squash manner

s3 archive squash cost-optimization small-files

• Updated Aug 6, 2023
• Rust

A file uploader specialized for uploading many small files onto HDFS

uploader hdfs small-files

• Updated Apr 18, 2018
• Java

PySpark script to aggregate small parquet files in a prefix into larger files. Designed to be run on AWS Glue

aws s3 glue pyspark small-files

• Updated May 13, 2026
• Python

Production-grade Zstd dictionary compression microservice for small JSON APIs. Solves frame overhead & schema drift with trained dictionaries, versioned registries, and dual-signal drift monitoring. 75%+ savings on disjoint holdouts.

python compression microservice json-api data-compression zstd zstandard small-files dictionary-compression drift-detection api-optimization schemadrift

• Updated Oct 20, 2026
• Python

Command Line Hangman/Word Guessing game.

game python c command-line hangman guessing-game small-files

• Updated Sep 4, 2016
• C

Read-only "small-file tax" analyzer for Delta Lake. Reads only the _delta_log to reconstruct the per-partition file-size distribution, then prices the wasted compute and S3 requests in dollars using real Spark internals (openCostInBytes, maxPartitionBytes) and emits a compaction plan with ROI and payback plus the OPTIMIZE SQL. Never runs OPTIMIZE.

python cli spark s3 optimize data-engineering zero-dependency roi cost-optimization small-files finops compaction delta-lake lakehouse data-observability delta-transaction-log

• Updated Jul 8, 2026
• Python

Read-only Delta Lake table health scanner with safe fix plans for Databricks, Python, and PySpark.

python spark optimize pyspark data-engineering vacuum databricks small-files delta-lake table-maintenance

• Updated May 12, 2026
• Python

antfs: a distributed blob storage system in Rust, purpose-built for massive small files. Inspired by Ambry / SeaweedFS / CubeFS — append-only log + index for O(1) small-file I/O, built-in Raft control plane, REST API, admin dashboard.

rust distributed-systems storage-engine raft distributed-storage object-storage blob-storage seaweedfs small-files ambry cubefs

• Updated Nov 2, 2026
• Rust

File merge action to merge files in HDFS or local filesystem

hdfs cdap cdap-plugin file-merge cask-marketplace small-files

• Updated Apr 12, 2022
• Java

小文件生成器，用于清理硬盘分区MFT

ntfs file-generator mft small-files

• Updated Aug 20, 2022
• C++

Fast multithreaded archiver for massive numbers of small files, optimized for cloud object storage

compression cpp tar cloud-storage multithreading archive small-files multimodal-data ai-infrastructure

• Updated Nov 13, 2026
• C++