How We Found and Fixed a Scoring Bug in Our LLM Benchmark

When we set out to see whether large language models could draft a realistic project risk register, we built a fairly simple pipeline: feed the model the project’s planning documents, ask it to output a JSON‑encoded list of risks, then compare that list to the human‑written version using semantic similarity.

Everything seemed straightforward until, late July 2026, a single cell in our results matrix stared back at us with a perfect zero. All 42 runs for the Llama‑3.3‑70B model on the “structured” prompt failed to parse, giving a recall of 0.0. At first glance it looked like a catastrophic model collapse, but we knew better – perfect 0% or 100% usually points to the test harness.

We stopped staring at the summary table and opened the raw logs. The old parser complained in three ways: it expected the model to repeat a project_id field we already knew, it threw away everything after the first JSON object when the model returned a list of objects, and it demanded a literal “FINAL JSON:” marker at the very end of the response. In reality the model was often doing the right thing; it just wasn’t obeying our overly‑strict formatting rules.

Changing a scorer after you’ve seen the numbers is a research no‑no, so we wrote down exactly what we were allowed to modify before touching any code. First, we vowed never to edit the raw model outputs – they were saved verbatim on disk and we would only re‑run the parsing step on that saved text. Second, we limited ourselves to fixing format, not content. That meant automatically inserting the missing project_id from the metadata, collecting every JSON object in the reply (instead of just the first one), and accepting a bare array or even a friendly sign‑off sentence after the JSON block.

To achieve that we replaced the fragile regular‑expression extractor with a loop that uses json.JSONDecoder().raw_decode. The decoder walks through the string, skips commas and whitespace, and pulls out each complete JSON value until it runs out of parsable text. We also relaxed a few schema checks: risk_register is now treated as a synonym for risks, and case‑insensitive matches for the eight approved risk categories are allowed.

Importantly, we left the actual schema untouched. If a model invented a new category like “legal” or “social”, the parser still flags it as an error. To guard against future slip‑ups we added a regression test called test_invalid_category_still_rejected_not_papered_over that will fail if someone later decides to widen the taxonomy.

After re‑running the parsing step on the saved outputs, the numbers changed dramatically. The “structured” prompt now shows a respectable recall, and the previously mysterious 100% failure rate disappears. The lesson? A tiny piece of glue code can masquerade as a model flaw, and fixing it requires surgical precision – never touch the raw answers, only the wrapper.

We also discovered a second, unrelated issue: hidden “reasoning” tokens that some APIs emit at the start of a response ate into the token budget on free‑tier plans, truncating the answer and making it look worse than it really was. Both problems are easy to make and easy to overlook, which is why we’ve published the parsing utilities and tests as open‑source helpers for anyone running LLM evaluations.