Task-Specific LLM Evals that Do & Don't Work
Eugene Yan
Yan argues that generic LLM scores often miss what a particular application must catch. Build task-specific evals, then check whether their scores support a usable production threshold.
Open library
The library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
563 papers · 561 with a Meow summary · 333 posts
The selected papers and blogs: the essential papers of generative AI and agents, and the key lab publications and essays, each with Meow and Standard summaries.
Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
Eugene Yan
LLM evaluators may make open-ended response review easier to scale, but the evidence does not support treating them as universal replacements for humans. Their task fit and missed errors matter.
What We’ve Learned From A Year of Building with LLMs
Eugene Yan · Applied LLMs
The authors’ year of building yields a practical thesis: getting an LLM demo to work is easier than making a dependable product; reliability comes from engineering the system around the model.
Your AI Product Needs Evals
Hamel Husain
AI teams need task-specific evals—not endless prompt edits—to know whether a product works and improve it beyond a demo.
Using LLM-as-a-Judge For Evaluation: A Complete Guide
Hamel Husain
The guide’s central argument: careful review of AI behavior creates the value; an LLM judge is a tool built from that review, not a replacement for it.
A Field Guide to Rapidly Improving AI Products
Hamel Husain
AI teams should optimize the learning loop, not the tool stack: inspect real failures, test targeted fixes, and let evidence determine what to build.
AI Evals: Everything You Need to Know
Hamel Husain
The FAQ’s central advice: evaluate whether your whole AI product meets user needs, not just how its model performs on shared benchmarks.
Understanding Reasoning LLMs
Sebastian Raschka, PhD · Sebastian Raschka
Reasoning models are a specialization for multi-step problems—not a universal upgrade—and their gains come with compute, cost, and overthinking trade-offs.
The State of Reinforcement Learning for LLM Reasoning
Sebastian Raschka, PhD · Sebastian Raschka
The article argues that reasoning-focused reinforcement learning can improve hard-task performance beyond scaling alone—but reward design, response length, and evaluation quality determine whether the gains are useful.
From GPT-2 to gpt-oss: Analyzing the Architectural Advances
Sebastian Raschka, PhD · Sebastian Raschka
gpt-oss keeps the decoder-only Transformer but combines sparse expert routing, attention changes, and quantization to make large models more practical to run. The author’s benchmark comparisons are promising, but real-world reliability remains uncertain.
Understanding Multimodal LLMs
Sebastian Raschka, PhD · Sebastian Raschka
The article’s central point: multimodal LLMs can join image and text either by concatenating their embeddings or through cross-attention. Neither design wins every trade-off.
New LLM Pre-training and Post-training Paradigms
Sebastian Raschka, PhD · Sebastian Raschka
Across these reports, the clearest trend is deliberate data quality and staged training—not one winning recipe—with teams extending context and customizing alignment.
DeepSeek R1's recipe to replicate o1 and the future of reasoning LMs
Nathan Lambert
R1 makes reasoning training more reproducible: the author argues its new behavior comes from large-scale reinforcement learning, not an explicit inference-time search procedure.
What comes next with reinforcement learning
Nathan Lambert
The author expects more frequent model updates from scaling current RL, but argues that this does not show models can learn from very long tasks or continually adapt from experience.
RLHF Book: Reinforcement Learning from Human Feedback and LLM Post-Training
Nathan Lambert
This page maps web edits against changes carried into the October 2026 first reprint, helping readers tell whether the print and web versions match.
Asymmetry of verification and verifier’s rule
Jason Wei
The useful leverage is not just generating candidates; it is building a quick, trustworthy way to tell which ones work.
Six intuitions about large language models
Jason Wei
Wei connects broad learning to data, compute and context, while warning that smooth overall trends can hide abrupt changes in particular abilities.
137 emergent abilities of large language models
Jason Wei
The author catalogs more than 100 empirical examples, suggesting emergent abilities are broader than a rare exception in scaled language models.
The Second Half
Shunyu Yao
The author calls the gap between AI’s benchmark achievements and its limited change to the world the “utility problem”—and argues that evaluation must address it.
The Scaling Hypothesis
Gwern
The scaling hypothesis asks whether simple neural networks can develop more general behavior by training on enough varied data and compute.
I. From GPT-4 to AGI: Counting the OOMs
Leopold Aschenbrenner
An OOM-based forecast, with the data wall as its biggest warning label.
Dario Amodei — Machines of Loving Grace
Dario Amodei
Amodei’s upside case is conditional: powerful AI could compress major advances into years, but physical bottlenecks and human decisions—not intelligence alone—shape the pace and who benefits.
Dario Amodei — The Urgency of Interpretability
Dario Amodei
Amodei argues that interpretability could help steer AI’s deployment, but its diagnostic tools must advance before models become more capable.
Dario Amodei — On DeepSeek and Export Controls
Dario Amodei
Amodei argues DeepSeek’s efficiency gains do not weaken chip controls: lower costs can fuel larger training runs, making access to millions of chips—not one model’s reported training bill—the key question for 2026–2027.
Dario Amodei — The Adolescence of Technology
Dario Amodei
The essay’s central question is whether human institutions can mature quickly enough to govern powerful AI without surrendering its benefits or its safeguards.
The Gentle Singularity
Sam Altman
Altman argues AI progress is becoming self-reinforcing, but the promised abundance depends on aligning systems and governing access as the gains and risks compound.
Three Observations
Sam Altman
Altman argues that AI’s economics could sustain rising investment and spread agents across knowledge work. But he says broad benefits and individual control will depend on choices about safety, policy, and distribution.
Introducing smolagents: simple agents that write actions in code.
Aymeric Roucher · Hugging Face
smolagents launches a compact library for language models to control tool-using workflows, with agents that write actions in code at its center.
What is an AI agent?
LangChain Accounts · LangChain
An agent is not a yes-or-no label: the article ties autonomy to how much application control the LLM owns—and to the engineering support that level demands.
Context Engineering
LangChain Accounts · LangChain
An agent’s context window is working memory, not a transcript that should grow without limit.
Deep Agents
LangChain Accounts · LangChain
A plain LLM tool loop can be “shallow”—unable to plan across longer tasks. The proposed fix keeps the loop and adds scaffolding for planning, subagents, and context.
How to think about agent frameworks
LangChain Accounts · LangChain
Agent frameworks are easy to compare badly: this article separates workflow-versus-agent behavior from orchestration style and high-level abstractions, with context control as the reliability test.
Introducing ambient agents
LangChain Accounts · LangChain
Ambient agents move beyond one-at-a-time, user-started chat: they listen for events, can act across several at once, and still bring people in when needed.
Don’t Build Multi-Agents
Walden Yan · Cognition
The failure isn’t simply parallelism: it’s agents making decisions without seeing the context and choices that shaped the rest of the work.
SWE-bench technical report
The Cognition Team · Cognition
The report argues that coding agents should be evaluated in full repositories, where they must locate files and use test feedback—not only solve isolated coding problems.
Context Engineering for AI Agents: Lessons from Building Manus
Manus
Manus chose context engineering as a way to improve its agent in hours instead of weeks, without tying the product to the underlying models.
Composer: Building a fast frontier model with RL
Cursor Team · Cursor
Cursor says Composer achieves frontier coding results with generation speed four times faster than similar models.
Measuring AI Ability to Complete Long Software Tasks
METR
Benchmark brilliance is not the same as sustained work: METR finds that current agents can sometimes handle expert-hour tasks but still struggle to complete them reliably.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
METR
In a randomized study, experienced open-source developers took 19% longer with AI allowed, even though they expected—and afterward believed—it had sped them up.
Recent Frontier Models Are Reward Hacking
METR
METR reports frontier models sometimes change the grader or task setup to earn high scores without doing the requested work.