Issues

Issues

• On BigCodeBench-Hard (n=148), ~55 of the leaderboard's models are statistically tied with #1 — consider showing significance tiers

  bigcode-project/bigcodebench#121 · opened on Jul 18, 2026

• Test Qwen2.5-Coder-32b-instruct pass@10

  bigcode-project/bigcodebench#120 · opened on Mar 19, 2026

• [Feat] Give estimates for exact cost/token count for hard

  bigcode-project/bigcodebench#118 · opened on Feb 22, 2026

• Seeding / Reproducibility

  bigcode-project/bigcodebench#115 · opened on Sep 25, 2025

• 🤗 [REQUEST] - Qwen3-Next series

  bigcode-project/bigcodebench#114 · opened on Sep 12, 2025

• 🐛 [TaskRemoval/TaskRepair] - <120> <Unexpected test cases>

  bigcode-project/bigcodebench#111 · opened on Aug 8, 2025

• 🤗 [REQUEST] - GPT-OSS 20B and 120B

  bigcode-project/bigcodebench#110 · opened on Aug 6, 2025

• Question: Why do sequential generation on OpenAIChatDecoder

  bigcode-project/bigcodebench#109 · opened on Aug 5, 2025
  1

• where is the file: audio.wav?

  bigcode-project/bigcodebench#108 · opened on Jul 17, 2025

• Is it normal for the ground truth accuracy not to be 100%?

  bigcode-project/bigcodebench#107 · opened on Jul 14, 2025

• Gradio does not work properly

  bigcode-project/bigcodebench#106 · opened on Jul 14, 2025
  1

• 🤗 [REQUEST] - Kimi K2 + Kimi-Dev

  bigcode-project/bigcodebench#105 · opened on Jul 14, 2025

• TypeError: pass_k int not iterable in evaluate.py

  bigcode-project/bigcodebench#103 · opened on Jun 23, 2025
  2

• 🤗 [REQUEST] - Arsh-llm-0.7b

  bigcode-project/bigcodebench#102 · opened on Jun 12, 2025
  2

• 🤗 [REQUEST] - RedNote Dots.LLM1

  bigcode-project/bigcodebench#101 · opened on Jun 8, 2025

• 🤗 [REQUEST] - DeepSeek 3.1 + DeepSeek-R1-0528 update

  bigcode-project/bigcodebench#100 · opened on May 28, 2025

• 🤗 [REQUEST] - Devstral

  bigcode-project/bigcodebench#99 · opened on May 28, 2025

• Add Qwen3

  bigcode-project/bigcodebench#98 · opened on May 25, 2025
  1

• 🤗 [REQUEST] - Arsh-llm

  bigcode-project/bigcodebench#97 · opened on May 22, 2025
  1

• Are there any standard instructions for the prompts?

  bigcode-project/bigcodebench#96 · opened on May 16, 2025
  3

• What is the prompt format?

  bigcode-project/bigcodebench#91 · opened on Apr 1, 2025
  9

• how to add new model?

  bigcode-project/bigcodebench#89 · opened on Mar 24, 2025
  1

• 🤗 [REQUEST] - Qwen2.5-Coder-7b

  bigcode-project/bigcodebench#82 · opened on Feb 14, 2025
  1

• [docs] Clarify risks of running evals locally.

  bigcode-project/bigcodebench#41 · opened on Sep 3, 2024

• [docs] Add working end-to-end example for a local model

  bigcode-project/bigcodebench#40 · opened on Sep 3, 2024
  1