Open library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
560 papers · 558 with a Meow summary · 333 posts
The selected papers and blogs: the essential papers of generative AI and agents, and the key lab publications and essays, each with Meow and Standard summaries.
Multimodal Large Language Models (MLLMs) have advanced VQA and now support Vision-DeepResearch systems that use search engines for complex visual-textual fact-finding. However, evaluating these visual and textual search abilities is still difficult, and existing benchmarks have two major limitations. First, **existing…
Yinger Zhang, Shutong Jiang, Renhao Li et al.
While agent evaluation has shifted toward long-horizon tasks, most benchmarks still emphasize local, step-level reasoning rather than the global constrained optimization (e.g., time and financial budgets) that demands genuine planning ability. Meanwhile, existing LLM planning benchmarks underrepresent the active infor…
Yibo Wang, Lei Wang, Yue Deng et al.
Deep research systems are widely used for multi-step web research, analysis, and cross-source synthesis, yet their evaluation remains challenging. Existing benchmarks often require annotation-intensive task construction, rely on static evaluation dimensions, or fail to reliably verify facts when citations are missing.…
Anthropic
A benchmark win may reflect a lucky question draw, not a real capability difference. The authors argue that evals should measure uncertainty before researchers treat score gaps as evidence.
Eugene Yan
Yan argues that generic LLM scores often miss what a particular application must catch. Build task-specific evals, then check whether their scores support a usable production threshold.
Eugene Yan
LLM evaluators may make open-ended response review easier to scale, but the evidence does not support treating them as universal replacements for humans. Their task fit and missed errors matter.
Hamel Husain
AI teams need task-specific evals—not endless prompt edits—to know whether a product works and improve it beyond a demo.
Hamel Husain
The guide’s central argument: careful review of AI behavior creates the value; an LLM judge is a tool built from that review, not a replacement for it.
Hamel Husain
AI teams should optimize the learning loop, not the tool stack: inspect real failures, test targeted fixes, and let evidence determine what to build.
Hamel Husain
The FAQ’s central advice: evaluate whether your whole AI product meets user needs, not just how its model performs on shared benchmarks.
METR
Benchmark brilliance is not the same as sustained work: METR finds that current agents can sometimes handle expert-hour tasks but still struggle to complete them reliably.
METR
In a randomized study, experienced open-source developers took 19% longer with AI allowed, even though they expected—and afterward believed—it had sped them up.
Naman Jain · Cursor
On historical coding benchmarks, agents can retrieve known fixes from the web or git history instead of deriving them. Stricter runtime controls sharply lower some scores, showing that a benchmark may measure access as well as coding.
William Isaac, Sol Messing and Kristian Lum · Google DeepMind
A cryptographic setup aims to keep test prompts and model weights hidden from each other, reducing the risk that prior prompt exposure skews benchmark scores.
METR
METR concludes Claude Opus 5.5 is a modest step beyond Fable 5.1: likely to help researchers, but not automate AI R&D; AI seems to have aided its development without dramatically speeding it up.
METR
An expenditure horizon is the agent spending at which human optimization catches up, turning an agent’s optimization curve into a dollar benchmark.
METR
METR found GPT-5.6 Sol’s time-horizon score too sensitive to how cheating is counted to be robust; its other evidence did not place the model beyond the software and R&D state of the art.
Tom Adamczewski · METR
MirrorCode uses a runnable original program as the referee: agents can query its behavior, but must match its exact outputs without seeing its source.
Cognition
FrontierCode shifts coding evaluation from “does it work?” to “would a maintainer merge it?” Cognition’s early results suggest even leading models remain far from its production-code standard.
Hamel Husain
OpenAI separated failures from passes best in this test, but the default 0.50 cutoff is no substitute for choosing a threshold that fits your eval.
Hamel Husain
The plugin’s standout was one-shot issue discovery; its weakness was pushing eval selection ahead of examining conversation data.
Hamel Husain
The design question is not only how to score an AI output, but whether its user can check it without redoing the work.
Hamel Husain
The authors offer coding-agent skills for choosing product-evaluation methods, aiming to avoid generic metrics or automation before anyone has inspected the data.
Eugene Yan
A cybersecurity score is useful only if the eval makes clear which step it tests—from finding a flaw to achieving an attacker’s goal—and how it verifies success.