For MCP agents, a claim can be true in the evidence pool and still be attributed to the wrong source. ProvenanceGuard checks both support and source attribution.
Why it matters: When agents combine records, research, and other tools, checking that a fact appears somewhere is not enough. Claim-to-source verdicts let reviewers inspect whether the answer relies on the right source, while the tested conservative policy favors review over letting unsupported claims through.
LoLBench tests whether coding agents can turn long enhancement proposals into large repository changes—not just implement edits from a specification someone else has already written.
Why it matters: Incomplete code localization was a major reported bottleneck. Reference-derived file trees alongside API specifications improved resolved rates by 16–22 percentage points (2.4–17×), reaching at most 34%, suggesting repository context matters alongside implementation capability.
E2E-SWE tests whether an agent can turn only a natural-language specification and an empty workspace into a complete, installable repository—not merely patch existing code.
Why it matters: The benchmark targets a capability beyond localized code changes: generating a complete working repository. Its reported score range differentiates models, while the authors say considerable headroom remains for future progress.
OpenCollab argues that evaluating collaboration requires checking whether agents actually follow the declared organization—not merely comparing outcomes.
Why it matters: For a fair comparison of agent teams, check that the intended collaboration occurred and control the runtime. Otherwise, observed gains may not be attributable to organization design.
AgentBug-Smith turns real-world agent harness failures into reproducible benchmark cases and reusable repair knowledge.
Why it matters: A live collection of reproducible failures can support ongoing evaluation, while its bug-fix knowledge base offers a route to improving repair rates. The abstract reports both uses, but does not establish how well the results generalize beyond the evaluated setting.