Agentic memory helps only at the right dose: in an eight-model AppWorld evaluation, strong, weaker, and saturated models responded differently to full guidelines and curated retrieval.
Why it matters: Treat memory as a model-specific deployment choice, not a feature to enable uniformly. Compare full guidelines and curated retrieval against a baseline, and account for token overhead; prompt caching may reduce effective cost, while the role of context-window size remains untested.
How well do repository-repair agents cope when code changes but behavior does not? This study tests semantics-preserving transformations and finds robustness depends on the scaffold, model, and dataset—not just the model.
Why it matters: An agent’s reliability on a codebase may shift after behavior-preserving edits, and a model that is robust with one scaffold may be brittle with another. Deployment assessments should therefore account for the scaffold and code variation, not rely on a single model ranking.