2025: The year in LLMs
Simon Willison
In 2025, LLMs moved from impressive answers toward useful multi-step work; that shift makes verification and permissions part of the product, not extras.
Open library
The library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
563 papers · 561 with a Meow summary · 333 posts
Key publications from Anthropic, OpenAI, Google DeepMind, Meta, DeepSeek, Qwen, Z.ai, Kimi and other labs, plus the most influential essays and blogs.
I think “agent” may finally have a widely enough agreed upon definition to be useful jargon now
Simon Willison
For LLM engineering, an agent runs tools in a bounded loop to achieve a goal. It’s useful technical shorthand, not a universal meaning of “agent.”
Designing agentic loops
Simon Willison
The key design problem is choosing goals, tools, and boundaries that let an agent iterate toward a solution without making its freedom the biggest risk.
Vibe engineering
Simon Willison
Willison proposes “vibe engineering” for AI-assisted production work where engineers remain accountable—not for accepting code just because it appears to work.
Here’s how I use LLMs to help me write code
Simon Willison
Treat an LLM as a fast, fallible coding partner: the developer must supply direction, verify behavior, and take over when judgment matters.
Agents
Chip Huyen
A chain of almost-correct steps can still fail: the article says 95% accuracy per step falls to 60% over 10 steps and 0.6% over 100.
Building A Generative AI Platform
Chip Huyen
A generative AI platform is more than a model call: it is a set of optional components that can be added as an application’s needs grow.
RLHF: Reinforcement Learning from Human Feedback
Chip Huyen
RLHF uses human rankings to steer a language model toward preferred answers. InstructGPT’s SFT+RLHF model outperformed SFT alone and was preferred overall, but hallucinated more.
Multimodality and Large Multimodal Models (LMMs)
Chip Huyen
The article’s core design pattern is to align different modalities in a shared embedding space, then let a language model condition its answers on visual tokens.
Building LLM applications for production
Chip Huyen
A working LLM demo is not proof of a reliable production system: prompts can fail silently, and multi-step workflows add more ways to go wrong.
Open challenges in LLM research
Chip Huyen
Ten open research directions are less a model checklist than a systems agenda: reliability, data, compute, interfaces, and whose preferences count.
Common pitfalls when building generative AI applications
Chip Huyen
A working GenAI demo proves possibility—not that a model is the simplest tool, that users want it, or that it is ready for production.
Patterns for Building LLM-based Systems & Products
Eugene Yan
An LLM demo is not a dependable product: the guide argues for measuring the real task and combining retrieval, specialization, safeguards, and user-centered design.
Task-Specific LLM Evals that Do & Don't Work
Eugene Yan
Yan argues that generic LLM scores often miss what a particular application must catch. Build task-specific evals, then check whether their scores support a usable production threshold.
Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
Eugene Yan
LLM evaluators may make open-ended response review easier to scale, but the evidence does not support treating them as universal replacements for humans. Their task fit and missed errors matter.
What We’ve Learned From A Year of Building with LLMs
Eugene Yan · Applied LLMs
The authors’ year of building yields a practical thesis: getting an LLM demo to work is easier than making a dependable product; reliability comes from engineering the system around the model.
Your AI Product Needs Evals
Hamel Husain
AI teams need task-specific evals—not endless prompt edits—to know whether a product works and improve it beyond a demo.
Using LLM-as-a-Judge For Evaluation: A Complete Guide
Hamel Husain
The guide’s central argument: careful review of AI behavior creates the value; an LLM judge is a tool built from that review, not a replacement for it.
A Field Guide to Rapidly Improving AI Products
Hamel Husain
AI teams should optimize the learning loop, not the tool stack: inspect real failures, test targeted fixes, and let evidence determine what to build.
AI Evals: Everything You Need to Know
Hamel Husain
The FAQ’s central advice: evaluate whether your whole AI product meets user needs, not just how its model performs on shared benchmarks.
Understanding Reasoning LLMs
Sebastian Raschka, PhD · Sebastian Raschka
Reasoning models are a specialization for multi-step problems—not a universal upgrade—and their gains come with compute, cost, and overthinking trade-offs.
The State of Reinforcement Learning for LLM Reasoning
Sebastian Raschka, PhD · Sebastian Raschka
The article argues that reasoning-focused reinforcement learning can improve hard-task performance beyond scaling alone—but reward design, response length, and evaluation quality determine whether the gains are useful.
From GPT-2 to gpt-oss: Analyzing the Architectural Advances
Sebastian Raschka, PhD · Sebastian Raschka
gpt-oss keeps the decoder-only Transformer but combines sparse expert routing, attention changes, and quantization to make large models more practical to run. The author’s benchmark comparisons are promising, but real-world reliability remains uncertain.
Understanding Multimodal LLMs
Sebastian Raschka, PhD · Sebastian Raschka
The article’s central point: multimodal LLMs can join image and text either by concatenating their embeddings or through cross-attention. Neither design wins every trade-off.
New LLM Pre-training and Post-training Paradigms
Sebastian Raschka, PhD · Sebastian Raschka
Across these reports, the clearest trend is deliberate data quality and staged training—not one winning recipe—with teams extending context and customizing alignment.
DeepSeek R1's recipe to replicate o1 and the future of reasoning LMs
Nathan Lambert
R1 makes reasoning training more reproducible: the author argues its new behavior comes from large-scale reinforcement learning, not an explicit inference-time search procedure.
What comes next with reinforcement learning
Nathan Lambert
The author expects more frequent model updates from scaling current RL, but argues that this does not show models can learn from very long tasks or continually adapt from experience.
RLHF Book: Reinforcement Learning from Human Feedback and LLM Post-Training
Nathan Lambert
This page maps web edits against changes carried into the October 2026 first reprint, helping readers tell whether the print and web versions match.
Asymmetry of verification and verifier’s rule
Jason Wei
The useful leverage is not just generating candidates; it is building a quick, trustworthy way to tell which ones work.
Six intuitions about large language models
Jason Wei
Wei connects broad learning to data, compute and context, while warning that smooth overall trends can hide abrupt changes in particular abilities.
137 emergent abilities of large language models
Jason Wei
The author catalogs more than 100 empirical examples, suggesting emergent abilities are broader than a rare exception in scaled language models.
The Second Half
Shunyu Yao
The author calls the gap between AI’s benchmark achievements and its limited change to the world the “utility problem”—and argues that evaluation must address it.
The Scaling Hypothesis
Gwern
The scaling hypothesis asks whether simple neural networks can develop more general behavior by training on enough varied data and compute.
I. From GPT-4 to AGI: Counting the OOMs
Leopold Aschenbrenner
An OOM-based forecast, with the data wall as its biggest warning label.
Dario Amodei — Machines of Loving Grace
Dario Amodei
Amodei’s upside case is conditional: powerful AI could compress major advances into years, but physical bottlenecks and human decisions—not intelligence alone—shape the pace and who benefits.
Dario Amodei — The Urgency of Interpretability
Dario Amodei
Amodei argues that interpretability could help steer AI’s deployment, but its diagnostic tools must advance before models become more capable.
Dario Amodei — On DeepSeek and Export Controls
Dario Amodei
Amodei argues DeepSeek’s efficiency gains do not weaken chip controls: lower costs can fuel larger training runs, making access to millions of chips—not one model’s reported training bill—the key question for 2026–2027.
Dario Amodei — The Adolescence of Technology
Dario Amodei
The essay’s central question is whether human institutions can mature quickly enough to govern powerful AI without surrendering its benefits or its safeguards.
The Gentle Singularity
Sam Altman
Altman argues AI progress is becoming self-reinforcing, but the promised abundance depends on aligning systems and governing access as the gains and risks compound.
Three Observations
Sam Altman
Altman argues that AI’s economics could sustain rising investment and spread agents across knowledge work. But he says broad benefits and individual control will depend on choices about safety, policy, and distribution.