Loading the edition…
· Past edition · 15 stories
Retrospective edition reconstructed from later captures. Paper versions may include later updates; coverage labels describe the evidence used.
Agent frameworks & orchestration
On AppWorld, GPT-4.1 succeeded on 77.4% of runs, but on all five runs for only 53.0% of tasks—a 24.4-point consistency gap.
Why it matters: For teams shipping agents, average accuracy alone can conceal inconsistent behavior on repeated requests. Reporting Pass^k alongside Mean@k—and diagnosing flip-prone steps—offers a way to assess and improve repeatability without a live task replay or ground-truth grader.
Agent frameworks & orchestration
Open each story in the reader. Summaries are shown when available; some sources provide an abstract only. With an account you can keep your own notes.
A passing result can hide an unrecovered mistake or harmful actions. This abstract maps 6,967 mistakes across 2,518 agent trajectories and reports a verifier designed to locate failures.
Why it matters: A passing score alone may conceal a run that deleted data, corrupted systems, or fabricated success. Locating failures offers a way to assess what happened during a run, not just its final result.
Agent frameworks & orchestration
ERPBench tests whether screenshot-only agents leave enterprise records correct—not merely whether they reach a form or appear to save it.
Why it matters: A visible save is not proof of a correct business update: some agents save in up to 85% of runs but write the correct value in as few as 3%. Evaluations and deployment checks therefore need to verify stored state, not just apparent task completion.