Open library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
560 papers · 558 with a Meow summary · 333 posts
The selected papers and blogs: the essential papers of generative AI and agents, and the key lab publications and essays, each with Meow and Standard summaries.
Gengsheng Li, Tianyu Yang, Junfeng Fang et al.
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models. While Group Relative Policy Optimization (GRPO) is widely adopted, its coarse credit assignment uniformly penalizes failed rollouts, lacking the token-level focus needed to efficiently address…
Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, Yuqing Yang Microsoft Research KAIST SK hynix Seoul National University blog post beanie00/self-distillation-analysis
Self-distillation has emerged as an effective post-training paradigm for LLMs, often improving performance while shortening reasoning traces. However, in mathematical reasoning, we find that it can reduce response length while degrading performance. We trace this degradation to the suppression of **epistemic verbaliza…
Qwen Pilot Team, Alibaba Group
We present **Future-KL Influenced Policy Optimization (FIPO)**, a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across ev…
Tianzhu Ye Li Dong
The prevailing paradigm for improving large language models relies on offline training with human annotations or simulated environments, leaving the rich experience accumulated during real-world deployment entirely unexploited. We propose **Online Experiential Learning** (OEL), a framework that enables language models…
Bingxiang He, Yuxin Zuo, Zeyuan Liu et al.
Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear.…
Chen Yueh-Han NYU, MATS Robert McCarthy UCL, MATS Bruce W. Lee UPenn, MATS He He NYU
Chain-of-thought (CoT) monitoring is a promising tool for detecting misbehaviors and understanding the motivations of modern reasoning models. However, if models can control what they verbalize in their CoT, it could undermine CoT monitorability. To measure this undesirable capability — CoT controllability — we introd…
Wenkai Yang, Weijie Liu, Ruobing Xie et al.
On-policy distillation (OPD), which aligns the student with the teacher’s logit distribution on student-generated trajectories, has demonstrated strong empirical gains in improving student performance and often outperforms off-policy distillation and reinforcement learning (RL) paradigms. In this work, we first theore…
Penghui Qi, Xiangxin Zhou, Zichen Liu et al.
Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm. Despite its ubiquity, we argue that the core ratio clipping mechanism in PPO is structurally ill-suited for the large vocabularies inher…
Jonas Hübotter Frederike Lübeck Lejs Behric Anton Baumann, Marco Bagatella Daniel Marta Ido Hakimi Idan Shenfeld, Thomas Kleine Buening Carlos Guestrin Andreas Krause
Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiabl…
Yanqi Dai, Yuxiang Ji, Xiao Zhang et al.
Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mechanism for enhancing mathematical reasoning in large models. However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining u…
Fengkai Yang, Zherui Chen, Xiaohan Wang et al.
Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet it…
Sunzhu Li, Jiale Zhao, Miteto Wei et al.
Reinforcement Learning with Verifiable Rewards (RLVR) has driven substantial progress in reasoning-intensive domains like mathematics. However, optimizing open-ended generation remains challenging due to the lack of ground truth. While rubric-based evaluation offers a structured proxy for verification, existing method…
Jingcheng Hu, Yinmin Zhang, Shijie Shang, Xiaobo Yang, Yue Peng, Zhewei Huang, Hebin Zhou, Xin Wu, Jie Cheng, Fanqi Wan, Xiangwen Kong, Chengyuan Yao, Kaiwen Yan, Ailin Huang, Hongyu Zhou, Qi Han, Zheng Ge, Daxin Jiang, Xiangyu Zhang, Heung-Yeung Shum
We introduce Parallel Coordinated Reasoning (PaCoRe), a training-and-inference framework designed to overcome a central limitation of contemporary language models: their inability to scale test-time compute (TTC) far beyond sequential reasoning under a fixed context window. PaCoRe departs from the traditional sequenti…
Shih-Yang Liu¹, Xin Dong, Ximing Lu et al.
As language models become increasingly capable, users expect them to provide not only accurate responses but also behaviors aligned with diverse human preferences across a variety of scenarios. To achieve this, Reinforcement learning (RL) pipelines have begun incorporating multiple rewards, each capturing a distinct p…
Muxi Diao, Lele Yang, Wuxuan Gong, Yutong Zhang,
Supervised Fine-Tuning (SFT) is the standard paradigm for domain adaptation, yet it frequently incurs the cost of catastrophic forgetting. In sharp contrast, on-policy Reinforcement Learning (RL) effectively preserves general capabilities. We investigate this discrepancy and identify a fundamental distributional gap:…
Trieu Trinh and Thang Luong · Google DeepMind
AlphaGeometry turns learned suggestions for added geometric constructions into formal, checkable proofs.
Thang Luong and Edward Lockhart · Google DeepMind
Gemini Deep Think reached gold-medal-level performance by producing rigorous proofs directly from natural-language problems, within the IMO’s 4.5-hour limit.
Thang Luong and Vahab Mirrokni · Google DeepMind
Aletheia’s key move is to make a proposed proof pass through a verifier that can revise it, restart it, or admit it is unsolved.
Google Research
A few worked examples can prompt a model to unpack multi-step problems, but the reported gains emerge only at sufficient scale—around 100B parameters.
Qwen Team · Qwen
Qwen reports that 32-billion-parameter QwQ-32B performs comparably to DeepSeek-R1 (671 billion parameters, 37 billion activated), arguing that reinforcement learning can help a strong pretrained model improve its performance.
Lilian Weng
More computation at inference can improve reasoning, but only when the extra work is spent well: longer traces and self-revision are not automatically more reliable.
Sebastian Raschka, PhD · Sebastian Raschka
Reasoning models are a specialization for multi-step problems—not a universal upgrade—and their gains come with compute, cost, and overthinking trade-offs.
Sebastian Raschka, PhD · Sebastian Raschka
The article argues that reasoning-focused reinforcement learning can improve hard-task performance beyond scaling alone—but reward design, response length, and evaluation quality determine whether the gains are useful.
Nathan Lambert
R1 makes reasoning training more reproducible: the author argues its new behavior comes from large-scale reinforcement learning, not an explicit inference-time search procedure.
Nathan Lambert
The author expects more frequent model updates from scaling current RL, but argues that this does not show models can learn from very long tasks or continually adapt from experience.
Jason Wei
The useful leverage is not just generating candidates; it is building a quick, trustworthy way to tell which ones work.
Mistral
Mistral presents Magistral as a reasoning model for traceable, multilingual work. Its benchmark and speed claims are useful signals, but the announcement gives limited evaluation detail.
The Deep Think team · Google DeepMind
Google upgraded Gemini 3 Deep Think for difficult science, research and engineering work, pairing benchmark claims with early examples from research and engineering.
Simon Willison
Willison argues that a rumor of an unpublished solution can trigger an expensive LLM race—and that the episode leaves the meaning of “used to improve model performance” unclear.
Sebastian Raschka, PhD · Sebastian Raschka
Looped transformers reuse the same blocks for extra depth, but that still costs compute. Raschka finds no strong evidence that looping makes chain-of-thought deceptive, and Astra's rumored architecture is unconfirmed.
Sebastian Raschka, PhD · Sebastian Raschka
Reasoning effort is a trained control over how much inference-time work an LLM spends—not a magic prompt word—and it trades answer quality against tokens, cost, and latency.
Ankit Maloo · Latent Space
The essay argues that expert work depends not just on producing polished output, but on anticipating how self-interested people will react, adapt, and exploit it.
Google Research
Reasoning can expose parametric memory—knowledge encoded in a model’s weights—even on simple fact questions, through extra computation and related-fact priming.