Benchmark dataset evaluating code review and security analysis tools on the OpenSSF CVE Benchmark.

Last updated: April 12, 2026

• DeepSource
• Claude Code
• Codex
• CodeRabbit
• Cursor
• Devin
• GitLab Duo
• Greptile
• Semgrep

Final evaluation results in JSONL format with fields:

• cve_id: CVE identifier
• variant: fixed or unfixed
• detected_issues: Issues found by the tool
• TP, FP, TN, FN: Classification metrics
• judge_reasoning: Explanation of the judgment

Intermediate formatted results from each tool, normalized for comparison.

Original tool outputs per CVE, preserving the exact response from each tool.

The archive/ directory contains prompts and data from earlier benchmark runs:

• OpenSSF CVE Benchmark