Scaling Laws, Carefully
Lilian Weng
Scaling laws make small training runs useful for forecasting larger ones, but their compute recommendations can shift with the experiment range, data regime, and fitting choices.
Open library
The library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
560 papers · 558 with a Meow summary · 333 posts
Key publications from Anthropic, OpenAI, Google DeepMind, Meta, DeepSeek, Qwen, Z.ai, Kimi and other labs, plus the most influential essays and blogs.
How to Train Really Large Models on Many GPUs?
Lilian Weng
Training giant networks is a memory-and-scheduling problem: divide data or computation across GPUs, then manage synchronization, idle stages, and extra state so the split actually helps.
Six intuitions about large language models
Jason Wei
Wei connects broad learning to data, compute and context, while warning that smooth overall trends can hide abrupt changes in particular abilities.
137 emergent abilities of large language models
Jason Wei
The author catalogs more than 100 empirical examples, suggesting emergent abilities are broader than a rare exception in scaled language models.
The Scaling Hypothesis
Gwern
The scaling hypothesis asks whether simple neural networks can develop more general behavior by training on enough varied data and compute.
Pretraining progress is mostly coming from data
Dwarkesh Patel
The measured data advantage is a pretraining compute-efficiency result—not evidence that model research mattered less, since the authors argue model advances also make larger runs feasible.
Why compute might get 10x+ more expensive in coming years
Dwarkesh Patel
If AI can do human-level software work, Patel argues an H100-equivalent could rent for over $250k a year—15× today’s spot price.
Empty shelves or lost keys? Recall is the bottleneck for parametric factuality
Google Research
A wrong answer may mean the fact is missing—or stored but unreachable. The authors’ framework tests those as different failures.
Can AI automate AI R&D yet?
David Owen · Epoch AI
End-to-end AI research requires more than running experiments: InnovationEval tested whether agents could independently invent, implement, and validate a new technique.
Is a compute crunch coming?
Luke Emberson · Epoch AI
The authors think a compute crunch is likely near: demand may be growing faster than inference capacity, putting access to frontier models under price pressure.