Independent benchmark and testing suite for OCR, extraction, retrieval, evidence attribution, graph generation, permissions, synchronization, and offline consistency.

docsgraph-eval is the quality-assurance layer for docsgraph, a local-first document/contract management platform built from docsgraph (the client) and docsgraph-server (the self-hostable backend). This repo independently benchmarks whether those two actually work correctly.

This project uses uv for packaging, ruff for linting and formatting, mypy (strict mode) for typing, and pytest for tests.

uv sync --all-extras --locked
uv run ruff check .
uv run ruff format --check .
uv run mypy src tests
uv run pytest

Install the git hooks with pre-commit so these checks run automatically before each commit:

uv run pre-commit install

uv run docsgraph-eval --help
uv run docsgraph-eval run ocr
uv run docsgraph-eval run all

• ocr — OCR accuracy against ground-truth transcriptions (character/word error rate).
• extraction — structured field/clause extraction quality.
• retrieval — search/retrieval quality (precision/recall, ranking metrics).
• evidence_attribution — whether answers correctly cite the source passage they came from.
• graph_generation — quality of the generated knowledge graph (entity/relation correctness).
• permissions — whether permission/access-control decisions match expected outcomes.
• sync — whether the sync protocol converges correctly across simulated clients.
• offline_consistency — whether data stays consistent across offline/online transitions and reconnects.

Each area lives in src/docsgraph_eval/<area>/ as a skeleton: a BenchmarkCase type (shared, defined in src/docsgraph_eval/core.py) and a run() stub that will grow into the real benchmark implementation. Sample documents and golden answers will live in fixtures/<area>/.

Early scaffold. Benchmark logic is not implemented yet — run() in each area currently raises NotImplementedError.

MIT — see LICENSE.