When the Exam Is Wrong: What Do AI Scores Measure?

A rope-tension question in PHYBench produced a model answer that looked different from the reference. Ordinary denominator rationalization turns them into the same expression:

T = P / (3 * sqrt(6))
  = P * sqrt(6) / (3 * 6)
  = sqrt(6) * P / 18

The Expression Edit Distance (EED) evaluator nevertheless awarded zero credit and a failed binary judgment. Figure 3 of a physics benchmark audit submitted on September 11, 2026 records the error: the model supplied an ordinary algebraic equivalent and the grading program counted it as a failure.

Researchers at Yale and other institutions conducted the study, How Good Are Frontier Models at Physics?, across six benchmarks: UGPhysics, PHYBench, PRISM-Physics, HLE-Physics, CMT-Benchmark, and CritPt. Its scope is text-only, closed-ended physics problems with verifiable final answers. Some scores rose sharply after repair: GPT-5.6-Sol's mean@4 increased from 61.00% to 87.24% on CMT and from 47.28% to 78.66% on HLE-Physics.

This exposes a weak point in evaluation. We spend enormous effort tracking a model's score while treating the questions and grading program as an inherently sound ruler. A faulty ruler makes even a precise percentage misleading. Separating grading errors, repaired questions, and repeated attempts shows what actually changed here.

The denominator behind 95.2% is rejected answers

Experts classified cases as model errors, grader errors, or benchmark errors. In the first category, the problem and reference are sound and the model is wrong. In the second, a correct response is rejected. The third covers incorrect references, missing conditions, and ambiguous questions. When benchmark materials are defective, the paper gives that classification precedence rather than grading the response as though the task were valid.

Across four pooled audits, 250 rejected cases ended with 143 benchmark errors, 95 grader errors, and 12 model errors. Those 250 rejections are the denominator behind the widely cited 95.2%: most audited failures were traced to the measurement process. CMT and CritPt used different audit coverage and are counted separately.

The reviewed sample was selected to investigate where those failures came from. Following rejected answers naturally exposes false rejections. The immediate result is a different explanation of these lost points. Estimating defects across the whole question pool calls for a separate sample drawn across that pool.

Benchmark errors are not always careless arithmetic, either. In Figure 4, a CritPt question about the Kitaev honeycomb model asks for ground-state energy without explicitly specifying a Hamiltonian using Pauli matrices or spin operators. For spin 1/2, using S = σ/2 changes the two-body product by a factor of four. The repaired question supplies the Hamiltonian and convention. An individual researcher's customary convention cannot substitute for a condition absent from the question.

Missing conditions are harder to fix than a grader that fails to recognize an equation. Even a stronger LLM judge has to choose a physical convention first. Leaving the solver and grader to guess what the question should have specified undermines a common basis for judgment from the start.

From 61 to 87, the exam changed too

CMT originally contained 50 questions. Experts found defects in 30, repaired 29, and excluded one, leaving 49. For HLE-Physics, 202 text-only questions remained after multimodal exclusions; removing 86 benchmark-error questions left 116. The figures below come from Table 1 and refer to GPT-5.6-Sol High's mean@4 under the corresponding tool conditions.

CMT's increase primarily comes from repairing benchmark materials. Its native evaluator was unavailable, so the study used the same HLE-adapted evaluator before and after repair, as Section 3.2.2 explains. Locating the error in a specific part of the process is more useful than a general claim that the old grader underestimated the model.

Keeping the original problem and response fixed while changing the grade tests grading error. Correcting a reference changes the judgment basis. Adding conditions and generating a new response changes the task. Excluding questions changes the denominator and difficulty composition. Each operation can be justified, but they are not the same measurement.

HLE's revisions particularly need a record. Auditing begins with rejected answers and then excludes invalid questions, linking deletion decisions to earlier failure records. Publish the exclusions, reasons, and retained set together so readers can inspect the new exam's composition alongside the higher score.

Repair the bad questions and publish the repair process as part of the experiment. Report fixed-output re-grading separately from fresh attempts on repaired tasks. Keeping the earlier records lets us distinguish better measurement from better models. A polished new percentage cannot carry that history by itself.

Behind 97.96% are four attempts

On repaired CMT, GPT-5.6-Sol High achieves 87.24% mean@4 and 97.96% pass@4. The former averages correctness over four attempts per question. The latter asks only whether at least one of those four attempts succeeds.

pass@4 measures whether at least one of four attempts succeeds. If one answer is right and three are wrong, the user still has to pick the right one. That selection problem is difficult in a product and easy to overlook behind a figure near 98%. Single-attempt performance and an answer-selection mechanism are needed to connect the result to use.

With 49 questions, each contributes about 2.04 percentage points to pass@4. Rankings separated by one or two questions need their per-question differences shown. Four attempts at the same problem still share that problem; uncertainty should be organized by question. Near the ceiling, these details determine how much a ranking means.

I therefore distinguish repeatability, validity for the target capability, and comparability across versions. Consistently rejecting an equivalent expression is a repeatable mistake, not valid measurement. Even excellent questions cannot guarantee comparable historical scores if tool access, reasoning budgets, or evaluator versions keep changing.

Bowman and Dahl discussed task design, annotation reliability, and scale in their 2021 evaluation-methodology paper. As models approach the ceiling, a few defective tasks and grading errors gain more influence over rankings. At that point, the exam itself needs an upgrade.

Expert grading needs auditing too

Expert review contains real disagreements. Of 196 double-reviewed cases, 140 initially agreed and 56 went to a third review; another 54 cases received one review. The 71.43% figure records initial agreement among double-reviewed cases. Publishing how disagreements were resolved gives expert judgment a route to scrutiny.

The distinction between a domain convention and a missing condition can rest on a reviewer's judgment. Reviewers also saw model responses and preliminary AI analysis, information that can influence their reading. Checking question validity first and retaining independent opinions and adjudication reasons reduces reliance on any one expert's intuition.

I would try to assess question validity before showing the model response and preserve reasons for disagreements. Accepted responses also need sampling. Checking only failures more readily uncovers false negatives, where correct answers are rejected. False positives, where wrong answers are accepted, also distort scores, in the opposite direction.

In its August 2026 review of the original HLE, Epoch AI randomly selected six questions from each of eight categories, 48 in total. It confirmed substantive defects in 22: 12 were unanswerable as written, and 10 could reject correct answers, with five of those also potentially accepting wrong answers. This equally stratified sample exposes errors in both directions.

Audit both directions when repairing a grader. A wider acceptance rule can recover lost credit and admit wrong answers. Regression tests should contain both valid transformations and plausible but incorrect expressions, exercising the fix from both sides.

The tests required a function name the task did not

Coding evaluation has the same mismatch. OpenAI's SWE-bench Verified review focused on 138 difficult tasks that o3 repeatedly failed to solve reliably. It found substantive testing or specification problems in 59.4% of that subset. So many failures tied to the test contract warrant a closer reading of “the model failed the tests.”

One Pylint task is especially concrete: tests directly imported a new function, get_annotation, although the problem statement did not require that name. A patch meeting the requirement through another internal implementation could still be rejected during test collection because the import failed. The tests had turned a detail of the reference patch into an unstated requirement.

A physics question asks for a correct result; a software requirement specifies behavior. If an internal helper's name is an acceptance condition, put it in the public task contract. When tests silently make the reference patch's implementation the only accepted solution, the score also measures guessing how the author wrote it.

Public tasks also face data contamination. Fixes and discussions remain visible and can enter training. OpenAI observed frontier models reproducing reference patches or question details in some tasks, a reason to investigate exposure and memory's contribution to scores.

The more often a test guides training feedback, model selection, and public rankings, the more development adapts to it. Evaluation maintenance needs both better grading and tracking of task exposure. Familiarity with the measuring tool can otherwise look like broader ability.

New questions make old scores harder to compare

LiveBench continually introduces new material and emphasizes objectively verifiable answers. LiveCodeBench collects contest problems over time and organizes evaluation by date ranges. These approaches reduce risks from prolonged exposure, but "new" does not automatically mean "equally difficult."

If model A scores 80 on one set this year and model B scores 85 on another next year, the five-point increase alone does not prove B is stronger. Subfield coverage, difficulty, and settings may have changed. Comparing new and old models should at least involve rerunning them on one frozen release under the same budget and tool conditions, rather than stitching annual leaderboard figures into a capability curve.

I would retain two sets for different purposes. Stable, validated questions help track regressions and configuration differences. New questions with controlled exposure help evaluate performance on new tasks. Saturated old questions can still detect degradation; they just become poor instruments for finely ranking the strongest models.

Cross-version comparisons need their calibration process on the record. Shared questions lose discriminating power once widely memorized or nearly saturated. Report task coverage, tool configuration, exposure time, and anchor performance so readers can see how scores from different years connect.

Maintain the evaluator as software

In engineering terms, I want a benchmark to preserve the process producing a score, not just a dataset and leaderboard. Questions, references, graders, prompts, model versions, reasoning budgets, and tool permissions must be traceable. Even adding one condition should record whether the valid-answer set changed, rather than quietly overwriting the task.

Response generation and scoring should be separated. Preserve raw outputs and run settings so evaluator repairs can re-score the same responses. A changed question starts another experiment. That separation makes it possible to distinguish model progress from measurement changes and explain a ranking reversal.

Graders need their own regression cases. For formulas, include positive examples of algebraic equivalence and negative examples that look similar but differ, have wrong units, or omit conditions. Layer symbolic computation, numerical checks, and expert review. Do not just relax rules around previously observed false rejections.

Numerical substitution is easy to overuse. Equality at a few points does not establish an identity. Domains, singularities, and square-root branches can change equivalence. For example, sqrt(x*x) = x requires additional conditions; for general real numbers the result is abs(x). Symbolic simplification also needs assumptions. A plausible-looking simplification cannot substitute for physical constraints.

A report can retain parallel results: original tasks and grader, corrected scores for fixed outputs, and new evaluations of repaired tasks. Each should identify question counts, model and configuration, audit coverage, and uncertainty organized by question. It is less concise than one overall accuracy, but readers can identify which experiment they are comparing.

From solving physics problems to doing research

The audit's strongest finding is concrete: older procedures underestimated frontier models on a set of text-only physics problems with explicit conditions and verifiable final answers. Credit lost to defective questions and graders should be restored. Protecting an intuition that AI cannot do physics is a poor reason to keep a faulty ruler.

But a problem set normally supplies the subject, conditions, and quantity to find. Open research also requires deciding what is worth asking, choosing representations, distinguishing explanations, and revising assumptions in response to evidence. Reading closed-ended scores as independent research ability skips that difference.

In the subsequent Is Physics Dead?, author John Sous describes preliminary agent experiments outside the paper. None of the open theoretical-physics problems had yet been solved completely and autonomously. Open research needs its own evaluation of problem selection, derivation, verification, and revision to capture more of what research involves.

Harder tasks are not automatically more valid. An open task scored only by another model's vague claim of "insight" can also lose its basis for review. Derivations, validation steps, testable predictions, and human-review reasons need to be preserved to discuss research progress rather than merely change the scoring vocabulary.

I want leaderboards to place the questions, grading program, and experimental setup beside the score. Repair defective tasks, correct misjudgments, and retain historical versions. Models deserve a fair exam, and readers deserve to know how the score was produced.