[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.

nlp benchmark natural-language-processing ai toolkit evaluation dataset ai-agents hallucination pypi-package responsible-ai trustworthy-machine-learning trustworthy-ai large-language-models llm llm-evaluation llm-safety openai-compatible llm-benchmark

• Updated Oct 4, 2026
• Python

Can AI agents rebuild websites through browser exploration?

benchmarking benchmark web-development ai reverse-engineering full-stack browser-automation harbor ai-agents web-agents llm llm-benchmarking browser-use agent-evaluation swe-bench llm-benchmark llm-benchmarks harbor-framework website-reconstruction visual-evaluation

• Updated Nov 10, 2026
• Python

How many bugs can LMs find & fix in large codebases?

benchmarking benchmark ai harbor ai-agents llm llm-benchmarking swe-bench llm-benchmark llm-benchmarks harbor-framework

• Updated Nov 5, 2026
• Python

HAEnv — synthetic longitudinal patients with code-derived gold, for evaluating health agents over time

benchmark health longitudinal-data wearables synthetic-data clinical-decision-support medical-ai healthcare-ai synthetic-patients medical-llm llm-evaluation agent-evals evaluation-harness llm-benchmark

• Updated Nov 1, 2026
• Python

A curated list on AI token economics: what tokens cost, where they get wasted, and how to cut the bill. Tools, benchmarks, papers, and copy-paste configs for the token economy of LLMs and coding agents.

awesome ai awesome-list ai-agents cost-optimization finops token-economy tokenomics llm prompt-caching token-usage cost-tracking llm-pricing llm-observability llm-cost token-economics claude-code token-optimization agentic-coding llm-benchmark

• Updated Nov 9, 2026
• Python

Bug Hunt Bench: 105 real bugs in two production repos, frontier coding models (GPT-6, Claude, Grok, Gemini, DeepSeek...) find and fix them in their own CLI, graded blind. Live leaderboard + every receipt.

benchmark leaderboard bug-fixing llm-evaluation coding-agents ai-coding claude-code codex-cli llm-benchmark gpt-6 software-engineering-agents

• Updated Nov 11, 2026
• JavaScript

A reproducible local LLM benchmarking platform with versioned datasets, deterministic scoring, and auditable reports.

docker benchmarking typescript model-evaluation ai-evaluation local-llm llm-evaluation llm-benchmark

• Updated Oct 28, 2026
• TypeScript

AI Stupid Level — independent, continuous LLM benchmarks (coding, reasoning, tool use), drift detection and a benchmark-driven Smart Router. Web app source; methodology public, test tasks private.

typescript nextjs drift-detection model-drift llm-evaluation ai-gateway llm-router llm-leaderboard ai-benchmark openai-compatible llm-benchmark

• Updated Nov 10, 2026
• TypeScript

A dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable in a browser. The game inside is called Spotless.

game benchmark typescript simulation mcp evaluation physics-simulation webgpu ai-agents llm llm-benchmark

• Updated Oct 17, 2026
• TypeScript

An Open Harness and Benchmark for AI in Cybersecurity Operations.

cybersecurity threat-hunting detection-engineering soc-analyst llm-evaluation llm-agents agentic-ai reasoning-models llm-benchmark

• Updated Nov 9, 2026
• Python

The AI-native wire format for structured data. 100% comprehension on every frontier model. 50-92% fewer tokens than JSON. 43B+ lossless round-trips across 17 formats. Spec v3.5.1 Stable.

mcp gemini tokens openai data-serialization structured-data gcf json-alternative ai-agents wire-format agent-communication llm anthropic model-context-protocol mcp-tools token-optimization token-efficiency llm-benchmark toon-alternative graph-compact-format

• Updated Nov 12, 2026

Открытый бенчмарк LLM: какая нейросеть лучше пишет код 1С:Предприятие (BSL). Объективная оценка LLM по методике SMOP с реальным исполнением в 1С — Claude, GPT, Gemini, DeepSeek, YandexGPT, GigaChat.

benchmark ai evaluation code-generation code-quality onescript bsl 1c 1c-enterprise llm ai-code-generation agentic function-calling llm-evaluation smop llm-benchmark

• Updated Oct 29, 2026
• Python

Community benchmark database for running LLMs on Apple Silicon Macs

benchmark inference apple-silicon llm llama-cpp local-llm llm-benchmark tokens-per-second

• Updated Apr 22, 2026
• Shell

HalachaBench early results: AI models on real semicha exam questions (provisional)

hebrew halacha llm-benchmark

• Updated Nov 10, 2026

Open-source AI tutor that teaches on a whiteboard. Benchmark of 6 LLMs (Claude, Gemini, DeepSeek): 288 lessons drawn stroke by stroke, blind-reviewed. 白板 AI 导师 / AI 老师讲课评测

education whiteboard edtech gemini claude llm ai-whiteboard ai-tutoring deepseek ai-tutor ai-teacher llm-benchmark math-tutor

• Updated Nov 12, 2026
• HTML

[ICCV 2025] AdsQA: Towards Advertisement Video Understanding Arxiv: https://arxiv.org/abs/2509.08621

benchmark video-understanding video-question-answering advertisement-dataset video-llms llm-benchmark

• Updated Oct 30, 2025
• Python

Benchmark coding agents and agent harnesses on 226 deterministic file-editing tasks with byte-exact verification and a public leaderboard.

benchmark evaluation developer-tools software-engineering ai-agents llm coding-agents code-editing llm-benchmark agent-harness coding-agent-benchmark

• Updated Nov 9, 2026
• JavaScript

ICRTL Benchmark: Industrial-level RTL design challenges for evaluating PPA optimization, code generation, and LLM applications in EDA.

benchmark evolution natural-language ppa eda rtl verilog system-verilog yosys ic-design vlsi-design rtl-design hardware-accelerator ai-agent verilogeval rtllm llm-benchmark spec-to-rtl

• Updated Jul 3, 2026
• SystemVerilog

Daily LLM value rankings - compare 300+ models by intelligence, speed and price. OpenRouter + Artificial Analysis. 大模型性价比排行榜

github-pages benchmark machine-learning static-site artificial-intelligence gemini gpt price-comparison claude model-comparison ai-tools large-language-models llm generative-ai openrouter deepseek llm-benchmark value-for-money ai-ranking token-pricing

• Updated Nov 11, 2026
• CSS

Open-source benchmark for evaluating LLMs on 220 real professional tasks across 9 sectors and 44 occupations. Reproducible experiments, artifact validation, grading, and a live evidence dashboard.

dashboard mlops github-actions huggingface azure-openai llm-evaluation benchmark-automation llm-benchmark gdpval real-world-tasks professional-tasks artifact-validation

• Updated Nov 12, 2026
• Python