Issues

Issues

• humaneval: a completion whose return value's __eq__ always returns True gets pass@1 = 1.0

  EleutherAI/lm-evaluation-harness#4585 · opened on Nov 11, 2026

• bbh web_of_lies / sports_understanding: flexible-extract ignores "X lies" and "implausible", so the score follows an earlier mention

  EleutherAI/lm-evaluation-harness#4533 · opened on Nov 5, 2026

• IFEval nth-paragraph checker reads the wrong paragraph when a blank separator is present

  EleutherAI/lm-evaluation-harness#4530 · opened on Nov 4, 2026

• Megatron tensor-parallel generation never broadcasts selected tokens

  EleutherAI/lm-evaluation-harness#4518 · opened on Nov 3, 2026

• Megatron silently ignores the documented CLI batch size

  EleutherAI/lm-evaluation-harness#4517 · opened on Nov 3, 2026

• IFEval highlight checker counts one ***bold-italic*** span as two highlighted sections

  EleutherAI/lm-evaluation-harness#4510 · opened on Nov 2, 2026

• MultiChoiceRegexFilter maps choices that normalise to the same text ("5"/"-5", "15"/"1.5", "Na"/"NA") to the last one's letter

  EleutherAI/lm-evaluation-harness#4509 · opened on Nov 2, 2026
  2

• IFEval section checker counts prose references like "see Section 2" as sections

  EleutherAI/lm-evaluation-harness#4508 · opened on Nov 2, 2026
  2

• IFEval bullet-list checker counts markdown rules (---), negative numbers and italic lines as bullet points

  EleutherAI/lm-evaluation-harness#4507 · opened on Nov 2, 2026
  1

• IFEval keyword checks match substrings, so "cat" is found in "category" and "art" is counted in "start"

  EleutherAI/lm-evaluation-harness#4506 · opened on Nov 2, 2026
  2

• Multi-GPU: per-document metric keys are dropped or crash when ranks see different keys

  EleutherAI/lm-evaluation-harness#4504 · opened on Nov 1, 2026

• HRM8K parses reference answers as generated text, rejecting correct expressions and crediting their last number

  EleutherAI/lm-evaluation-harness#4502 · opened on Nov 1, 2026

• API token-limit aliases can truncate the wrong prompt or exceed the context window

  EleutherAI/lm-evaluation-harness#4496 · opened on Oct 31, 2026

• HRM8K exact-match grading accepts distinct large integers after lossy float normalization

  EleutherAI/lm-evaluation-harness#4493 · opened on Oct 30, 2026

• Megatron-LM ignores generation token-limit aliases

  EleutherAI/lm-evaluation-harness#4488 · opened on Oct 30, 2026

• NeMo ignores generation token-limit aliases and reserves the wrong prompt space

  EleutherAI/lm-evaluation-harness#4487 · opened on Oct 30, 2026

• afrobench: groups aggregate acc but their tasks report bleu/chrf/rougeL, so the group scores come out empty

  EleutherAI/lm-evaluation-harness#4480 · opened on Oct 28, 2026

• gguf backend (teacher forcing): silently assumes deterministic natural logprobs from llama-server — fails quiet on nondeterministic-logprob servers

  EleutherAI/lm-evaluation-harness#4476 · opened on Oct 26, 2026
  7

• ONNX GenAI and WinML ignore generation token-limit aliases

  EleutherAI/lm-evaluation-harness#4466 · opened on Oct 26, 2026

• IPEX and Neuron backends fail registration due to incorrect lazy targets

  EleutherAI/lm-evaluation-harness#4465 · opened on Oct 26, 2026

• Docs example: point local-chat-completions at a cloud OpenAI-compatible endpoint

  EleutherAI/lm-evaluation-harness#4460 · opened on Oct 25, 2026

• Bug: postprocess_generated_text strips leading whitespace even when think_end_token is absent from the generation, which breaks code-continuation tasks (HumanEval pass@1 = 0)

  EleutherAI/lm-evaluation-harness#4458 · opened on Oct 25, 2026

• Azure chat backend is missing from lazy model discovery

  EleutherAI/lm-evaluation-harness#4454 · opened on Oct 24, 2026

• LiteLLM ignores the model-level base_url in completion calls

  EleutherAI/lm-evaluation-harness#4453 · opened on Oct 24, 2026

• LiteLLM async generation returns None for null chat content

  EleutherAI/lm-evaluation-harness#4437 · opened on Oct 22, 2026