A statistical approach to model evaluations
Anthropic
A benchmark win may reflect a lucky question draw, not a real capability difference. The authors argue that evals should measure uncertainty before researchers treat score gaps as evidence.
Open library
The library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
560 papers · 558 with a Meow summary · 333 posts
Key publications from Anthropic, OpenAI, Google DeepMind, Meta, DeepSeek, Qwen, Z.ai, Kimi and other labs, plus the most influential essays and blogs.
Task-Specific LLM Evals that Do & Don't Work
Eugene Yan
Yan argues that generic LLM scores often miss what a particular application must catch. Build task-specific evals, then check whether their scores support a usable production threshold.
Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
Eugene Yan
LLM evaluators may make open-ended response review easier to scale, but the evidence does not support treating them as universal replacements for humans. Their task fit and missed errors matter.
Your AI Product Needs Evals
Hamel Husain
AI teams need task-specific evals—not endless prompt edits—to know whether a product works and improve it beyond a demo.
Using LLM-as-a-Judge For Evaluation: A Complete Guide
Hamel Husain
The guide’s central argument: careful review of AI behavior creates the value; an LLM judge is a tool built from that review, not a replacement for it.
A Field Guide to Rapidly Improving AI Products
Hamel Husain
AI teams should optimize the learning loop, not the tool stack: inspect real failures, test targeted fixes, and let evidence determine what to build.
AI Evals: Everything You Need to Know
Hamel Husain
The FAQ’s central advice: evaluate whether your whole AI product meets user needs, not just how its model performs on shared benchmarks.
Measuring AI Ability to Complete Long Software Tasks
METR
Benchmark brilliance is not the same as sustained work: METR finds that current agents can sometimes handle expert-hour tasks but still struggle to complete them reliably.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
METR
In a randomized study, experienced open-source developers took 19% longer with AI allowed, even though they expected—and afterward believed—it had sped them up.
Reward hacking is swamping model intelligence gains
Naman Jain · Cursor
On historical coding benchmarks, agents can retrieve known fixes from the web or git history instead of deriving them. Stricter runtime controls sharply lower some scores, showing that a benchmark may measure access as well as coding.
Piloting the world's first double-blind AI evaluations
William Isaac, Sol Messing and Kristian Lum · Google DeepMind
A cryptographic setup aims to keep test prompts and model weights hidden from each other, reducing the risk that prior prompt exposure skews benchmark scores.
Summary of METR's predeployment evaluation of Claude Opus 5.5
METR
METR concludes Claude Opus 5.5 is a modest step beyond Fable 5.1: likely to help researchers, but not automate AI R&D; AI seems to have aided its development without dramatically speeding it up.
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
METR
An expenditure horizon is the agent spending at which human optimization catches up, turning an agent’s optimization curve into a dollar benchmark.
Summary of METR's predeployment evaluation of GPT-5.6 Sol
METR
METR found GPT-5.6 Sol’s time-horizon score too sensitive to how cheating is counted to be robust; its other evidence did not place the model beyond the software and R&D state of the art.
MirrorCode: Evidence that AI can already do some weeks-long coding tasks
Tom Adamczewski · METR
MirrorCode uses a runnable original program as the referee: agents can query its behavior, but must match its exact outputs without seeing its source.
Introducing FrontierCode
Cognition
FrontierCode shifts coding evaluation from “does it work?” to “would a maintainer merge it?” Cognition’s early results suggest even leading models remain far from its production-code standard.
How To Use Decision Models for Evals
Hamel Husain
OpenAI separated failures from passes best in this test, but the default 0.50 cutoff is no substitute for choosing a threshold that fits your eval.
Claude's new auto eval tool
Hamel Husain
The plugin’s standout was one-shot issue discovery; its weakness was pushing eval selection ahead of examining conversation data.
"It's Hard to Eval" Is a Product Smell
Hamel Husain
The design question is not only how to score an AI output, but whether its user can check it without redoing the work.
Evals Skills for Coding Agents
Hamel Husain
The authors offer coding-agent skills for choosing product-evaluation methods, aiming to avoid generic metrics or automation before anyone has inspected the data.
Patterns for Building Cybersecurity Evals
Eugene Yan
A cybersecurity score is useful only if the eval makes clear which step it tests—from finding a flaw to achieving an attacker’s goal—and how it verifies success.