E-Commerce Bench tests merchant agents over a simulated year, where delayed revenue, limited time, inventory risk and shifting demand make steady operation harder than completing a short task.
Why it matters: A strong ending balance can hide weaknesses in fraud avoidance, cash flow or efficiency. Separate scores help identify those failure modes, while deterministic economic decisions make comparisons more reproducible.
Optimize the completed task, not the token count of one tool call: missing context can trigger recovery work that raises end-to-end cost and duration.
Why it matters: A locally shorter response can make a task slower and more expensive if the agent must recover what was omitted. Measure the complete workflow—including recovery behavior—and test each change in the product surface where it will run.
CAPTURE asks whether a conflicting memory signals a real preference change, a temporary context shift, ambiguity, or adversarial poisoning—and models that uncertainty explicitly.
Why it matters: The results frame memory safety as a balance, not a simple block-or-allow rule: resisting fixed-policy poisoning while accepting genuine updates matters, and stronger attacker access raises attack success.