Across 11 Ironclad research tasks, GPT‑6 Astra averaged 55.0% against GPT‑5.6 Sol’s 41.6%, with lower estimated time per attempt—but the timing is simulated, not measured customer savings.
Why it matters: The evaluation focuses on whether an agent preserves business rules across a whole workflow, not just whether it completes individual actions. That is a useful test for complex software tasks, but the limited task set and simulated timing do not establish broad performance or real customer time savings; the article also says human oversight remains important.
ParanoiaEval tests whether coding agents choose risk treatments warranted by repository evidence—not merely whether they can complete a task.
Why it matters: Evaluating coding agents requires checking whether their defensive actions match the evidence, not just whether they perform the task. The reported findings treat risk treatment as an independent capability dimension.