Research & academia
When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark
An audit finds that benchmark scoring can alter the state it measures.
Why it matters: A flawed reward function can change a benchmark result before grading.