- 🤖 LLM usage: $1.1163 (23 commits)
- 👤 Human dev: ~$1136 (11.4h @ $100/h, 30min dedup)
Generated on 2026-07-06 using openrouter/deep/deep-v4-pro
Correct code is not the same as mergeable code. eslint + code review, but for AI. Add to your pipeline in 2 minutes.
pip install qualbench qualbench quickstart
No config, no API keys. QualBench evaluates your current diff and prints a Quality Score.
# .github/workflows/qualbench.yml name: QualBench on: [pull_request] jobs: quality-check: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: semcod/qualbench-action@v1 with: tool: prollama fail_on_score: 70
Every AI-generated PR gets a quality review comment. Set fail_on_score and the pipeline fails if quality is below your threshold.
🧠 QualBench Review
Quality Score: 78/100
❌ Complexity increased (+12%)
⚠ Security: 1 new medium-severity finding
✔ Tests pass, no regressions
Verdict: needs_review
# .github/workflows/qualbench.yml name: QualBench on: [pull_request] jobs: quality-check: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: semcod/qualbench-action@v1 with: tool: prollama fail_on_score: 70
# .gitlab-ci.yml qualbench: stage: test image: python:3.12-slim before_script: - pip install qualbench script: - qualbench run --tool prollama --json --fail-on-score 70 only: - merge_requests
# azure-pipelines.yml steps: - task: UsePythonVersion@0 inputs: versionSpec: '3.12' - script: | pip install qualbench qualbench run --tool prollama --json --fail-on-score 70 displayName: 'QualBench Quality Check'
// Jenkinsfile stage('Quality Check') { steps { sh ''' pip install qualbench qualbench run --tool prollama --fail-on-score 70 ''' } }
# .circleci/config.yml version: 2.1 jobs: quality: docker: - image: python:3.12-slim steps: - checkout - run: pip install qualbench - run: qualbench run --tool prollama --fail-on-score 70 workflows: quality-check: jobs: - quality
AI coding tools resolve 70–80% of benchmark tasks. But most AI-generated PRs are not mergeable without human fixes. Every existing benchmark asks "do tests pass?" — nobody asks "would a senior developer approve this PR?"
| Dimension | What it measures | Weight |
|---|---|---|
| Correctness | All tests pass, no regressions | 25% |
| Mergeability | Would a senior dev merge this? (1–5) | 25% |
| Security | New vulnerabilities introduced | 15% |
| Code quality | Complexity delta, dead code | 15% |
| Iterations | Attempts to reach acceptable output | 10% |
| Cost efficiency | USD per successful patch | 10% |
Verdicts: ready_to_merge (≥85), needs_review (65–84), not_merge_ready (<65).
qualbench run --tool prollama # score current diff qualbench run --tool prollama --json # portable JSON output qualbench run --mode cheap # lowest-cost models qualbench quickstart # first score in 60 seconds qualbench compare my_tool # vs leaderboard qualbench info # dataset summary qualbench doctor # check dependencies
CLI, API, GitHub Action — same JSON schema. See docs/schema.md.
cp runners/template.py runners/my_tool.py # Implement run() → return portable schema qualbench run --tool my_tool # Submit PR with results
Licensed under Apache-2.0.
Last updated by taskill at 2026-04-25 13:46 UTC
| Metric | Value |
|---|---|
| HEAD | c199cf4 |
| Coverage | — |
| Failing tests | — |
| Commits in last cycle | 26 |
Repository received a mix of fixes, refactors and configuration updates: tests were hardened (mocks added), a JSON test fix was applied, many vallm/style issues and magic numbers were addressed, and release-related features (v0.3.0, Supervisor AI, new runners) were added. PyQual configuration thresholds and gates were adjusted and documentation/TODOs were updated after a PyQual run.