Problem:

Have you faced issues inserting data into databases? Constant schema mismatch errors, incorrect data types, manual checks, and even silent data corruption? If you work with large ETL pipelines and databases, you know how painful it can be.

Solution:

Insert Tools is a robust and flexible tool designed for safe and fast data insertion into databases — starting with ClickHouse. It validates schema by column names (not by order), supports automatic type casting, and lets you dry-run your inserts before touching real data. Perfect for ETL pipelines where target table schemas evolve frequently.

• ✅ Data safety: Validates column names and types before insert.
• ⚙️ Auto type casting: Converts mismatched types when enabled.
• 🚧 Dry-run mode: Test inserts without touching data.
• 🐳 Docker-ready: Comes with ready-to-use Docker integration.
• 🔧 Configurable: Fully controllable insert pipeline.
• 🔥 Time saver: Automates validation and error prevention.

• 🖥️ Simple CLI and Python API.
• 🛡️ Strict mode to block extra columns.
• 📌 Detailed logging and diagnostics.
• 🔄 Easy CI/CD integration.

pip install insert-tools

To install for development:

pip install -e .[dev]

Link to the project on PyPI

from insert_tools.runner import InsertConfig, run_insert

config = InsertConfig(
    host="localhost",
    database="default",
    target_table="my_table",
    select_sql="SELECT * FROM source_table",
    user="default",
    password="admin123",
    allow_type_cast=True,
    strict_column_match=True
)

run_insert(config)

insert-tools \
  --host localhost \
  --port 8123 \
  --user default \
  --password admin123 \
  --database default \
  --target_table my_table \
  --select_sql "SELECT * FROM source_table" \
  --allow_type_cast \
  --strict \
  --dry-run \
  --verbose

pytest -v --cov=insert_tools tests/

Integration tests are supported via Docker (docker-compose.yml).

Planned and upcoming features:

• ClickHouse support (stable)
• Manual insert_columns mapping
• Logging configuration (file, level, formatting)
• Dry-run + exit codes
• Strict schema validator with preview

• MySQL — no name-based insert, requires exact column order
• PostgreSQL — order and column count must match
• SQLite — insert depends on column order
• Oracle — insert requires explicit column mapping
• SQL Server — insert must follow column order

• Error handling strategies (fail, warn, skip)
• Config file validation (optional)
• Secure secrets handling (.env / vault)
• Optional CAST rules config

• Full documentation site (mkdocs)
• Schema + config reference
• Auto-generated help from CLI
• GitHub Discussions / Community page

1. Fetches target table schema from ClickHouse.
2. Extracts column names and types from SELECT query.
3. Applies optional CAST(...) if types mismatch.
4. Validates column alignment and inserts data.

Ideas, bug reports, and pull requests are welcome! Join the community and help make Insert Tools better.

This project uses a dual-license model:

• 🆓 Non-commercial license — free to use for personal, educational, and internal non-commercial purposes. See LICENSE_NONCOMMERCIAL.md
• 💼 Commercial license — required for any commercial use. See LICENSE_COMMERCIAL.md or contact [email protected] for licensing terms.

Insert Tools makes data insertion simple, fast, and safe. Save your time and nerves today!