Agent frameworks & orchestration
Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes
A benchmark pass can be unearned, so integrity depends on auditing trajectories and retesting both exploit access and legitimate solvability after repairs.
Why it matters: Because benchmarks guide model selection and training, a pass may give a misleading signal if it comes from an exploit rather than the intended capability. Repairs need to test both whether the exploit route is closed and whether legitimate solving still works.