Open library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
563 papers · 561 with a Meow summary · 333 posts
Every paper prepared on Reading + AI, newest first.
LimiX Team
We introduce LimiX-2, a new model in the LimiX family, developed through model and data scaling guided by our previously established scaling laws. LimiX-2 adopts the Contextual Mechanism Networks (CMNs) paradigm and is pretrained with Context-Conditional Masked Modeling (CCMM). CMNs shifts the organizing principle of…
Muxi Diao, Lele Yang, Wuxuan Gong, Yutong Zhang,
Supervised Fine-Tuning (SFT) is the standard paradigm for domain adaptation, yet it frequently incurs the cost of catastrophic forgetting. In sharp contrast, on-policy Reinforcement Learning (RL) effectively preserves general capabilities. We investigate this discrepancy and identify a fundamental distributional gap:…
Tong Zheng, Xidong Wu, Zheng Zhang et al.
Recursive self-improvement is becoming essential for autonomous AI agents, whose progress depends on discovering high-value solutions across complex domains. Effective exploration drives this process, yet managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilem…
Peng Xia, Rujun Han, Zifeng Wang et al.
An LLM agent’s capability is largely magnified by its harness, namely the prompts, control flow, tooling, memory, and context management surrounding the frozen backbone model. Recent methods increasingly automate this process by iteratively proposing and selecting component-wise edits of an agent harness, practically…
Shih-Yang Liu¹, Xin Dong, Ximing Lu et al.
As language models become increasingly capable, users expect them to provide not only accurate responses but also behaviors aligned with diverse human preferences across a variety of scenarios. To achieve this, Reinforcement learning (RL) pipelines have begun incorporating multiple rewards, each capturing a distinct p…
Fengkai Yang, Zherui Chen, Xiaohan Wang et al.
Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet it…
Sunzhu Li, Jiale Zhao, Miteto Wei et al.
Reinforcement Learning with Verifiable Rewards (RLVR) has driven substantial progress in reasoning-intensive domains like mathematics. However, optimizing open-ended generation remains challenging due to the lack of ground truth. While rubric-based evaluation offers a structured proxy for verification, existing method…
Jingcheng Hu, Yinmin Zhang, Shijie Shang, Xiaobo Yang, Yue Peng, Zhewei Huang, Hebin Zhou, Xin Wu, Jie Cheng, Fanqi Wan, Xiangwen Kong, Chengyuan Yao, Kaiwen Yan, Ailin Huang, Hongyu Zhou, Qi Han, Zheng Ge, Daxin Jiang, Xiangyu Zhang, Heung-Yeung Shum
We introduce Parallel Coordinated Reasoning (PaCoRe), a training-and-inference framework designed to overcome a central limitation of contemporary language models: their inability to scale test-time compute (TTC) far beyond sequential reasoning under a fixed context window. PaCoRe departs from the traditional sequenti…
Jonas Hübotter Frederike Lübeck Lejs Behric Anton Baumann, Marco Bagatella Daniel Marta Ido Hakimi Idan Shenfeld, Thomas Kleine Buening Carlos Guestrin Andreas Krause
Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiabl…
Yanqi Dai, Yuxiang Ji, Xiao Zhang et al.
Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mechanism for enhancing mathematical reasoning in large models. However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining u…
Bingxiang He, Yuxin Zuo, Zeyuan Liu et al.
Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear.…
Penghui Qi, Xiangxin Zhou, Zichen Liu et al.
Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm. Despite its ubiquity, we argue that the core ratio clipping mechanism in PPO is structurally ill-suited for the large vocabularies inher…
Chen Yueh-Han NYU, MATS Robert McCarthy UCL, MATS Bruce W. Lee UPenn, MATS He He NYU
Chain-of-thought (CoT) monitoring is a promising tool for detecting misbehaviors and understanding the motivations of modern reasoning models. However, if models can control what they verbalize in their CoT, it could undermine CoT monitorability. To measure this undesirable capability — CoT controllability — we introd…
Alexander Panfilov*, David Schmotz*, Ilia Shumailov* et al.
Leading large language model providers now conceal their models’ step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage. Rather than storing these traces server-side, providers return them to the client as blocks of encrypted text, which the client passes back with…
Wenkai Yang, Weijie Liu, Ruobing Xie et al.
On-policy distillation (OPD), which aligns the student with the teacher’s logit distribution on student-generated trajectories, has demonstrated strong empirical gains in improving student performance and often outperforms off-policy distillation and reinforcement learning (RL) paradigms. In this work, we first theore…
Qihan Ren, Peng Wang, Ruikun Cai et al.
A prevailing narrative in LLM post-training holds that supervised fine-tuning (SFT) memorizes while reinforcement learning (RL) generalizes. We revisit this claim for reasoning SFT with long chain-of-thought (CoT) supervision and find that cross-domain generalization is not absent but conditional, jointly shaped by op…
Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, Yuqing Yang Microsoft Research KAIST SK hynix Seoul National University blog post beanie00/self-distillation-analysis
Self-distillation has emerged as an effective post-training paradigm for LLMs, often improving performance while shortening reasoning traces. However, in mathematical reasoning, we find that it can reduce response length while degrading performance. We trace this degradation to the suppression of **epistemic verbaliza…
Qwen Pilot Team, Alibaba Group
We present **Future-KL Influenced Policy Optimization (FIPO)**, a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across ev…
Tianzhu Ye Li Dong
The prevailing paradigm for improving large language models relies on offline training with human annotations or simulated environments, leaving the rich experience accumulated during real-world deployment entirely unexploited. We propose **Online Experiential Learning** (OEL), a framework that enables language models…
Gengsheng Li, Tianyu Yang, Junfeng Fang et al.
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models. While Group Relative Policy Optimization (GRPO) is widely adopted, its coarse credit assignment uniformly penalizes failed rollouts, lacking the token-level focus needed to efficiently address…
Zhennan Shen, Yanshu Li, Qingyu Yin, Chak Tou Leong, Zhilin Wang Yanxu Chen, Rongduo Han, Sunbowen Lee, Yi R. Fung HKUST Brown University Zhejiang University Hong Kong PolyU USTC BUPT Nankai University BIT
On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A…
Yaxuan Li, Yuxin Zuo, Bingxiang He et al.
On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the…
Chenxu Yang, Chuanyu Qin, Qingyi Si et al.
On-policy distillation (OPD) has become a popular training paradigm in the LLM community. This paradigm selects a larger model as the teacher to provide dense, fine-grained signals for each sampled trajectory, in contrast to reinforcement learning with verifiable rewards (RLVR), which only obtains sparse signals from…
Binbin Zheng Xing Ma Yiheng Liang Jingqing Ruan Xiaoliang Fu Kepeng Lin Benchang Zhu Ke Zeng Xunliang Cai University of Science and Technology of China Meituan LongCat Interaction Team Nanjing University Fudan University Huazhong University of Science and Technology https://github.com/machine981/SCOPE
On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult. On-Policy Distillation (OPD) alleviates this by introducing dense, token-level KL supervision from a teache…
Yuanyi Wang, Su Lu, Yanggan Gu et al.
On-policy distillation (OPD) trains a student on its own rollouts with token-level teacher supervision. Recent selective OPD methods exploit the non-uniformity of OPD signals by prioritizing high-entropy or high-disagreement tokens. We revisit this principle and ask: which token-level teacher signals are actually lear…
Shanghai Artificial Intelligence Laboratory, https://github.com/AI45Lab/AgentDoG, https://huggingface.co/collections/AI45Research/agentdog
The rise of AI agents introduces complex safety and security challenges arising from autonomous tool use and environmental interactions. Current guardrail models lack agentic risk awareness and transparency in risk diagnosis. To introduce an agentic guardrail that covers complex and numerous risky behaviors, we use a…
Yutao Mou, Zhangchi Xue, Lijun Li et al.
While LLM-based agents can interact with environments via invoking external tools, their expanded capabilities also amplify security risks. Monitoring step-level tool invocation behaviors in real time and proactively intervening before unsafe execution is critical for agent deployment, yet remains underexplored. In th…
Xingrun Xing, Haoqing Wang, Boyan Gao et al.
On-Policy Distillation (OPD) is a fundamental technique for efficient post-training of large language models (LLMs), with broad applications in agent learning, multi-task enhancement, and model compression. However, OPD training becomes unstable when the teacher and student distributions differ substantially, as teach…
Wenhan Ma, Jianyu Wei, Liang Zhao et al.
Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose **Mult…
Natalie Shapira Chris Wendler Avery Yen, Gabriele Sarti Koyena Pal Olivia Floody Adam Belfki, Alex Loftus Aditya Ratan Jannali Nikhil Prakash Jasmine Cui et al.
We report an exploratory red-teaming study of autonomous language-model–powered agents deployed in a live laboratory environment with persistent memory, email accounts, Discord access, file systems, and shell execution. Over a two-week period, twenty AI researchers interacted with the agents under benign and adversari…
Yijiang Li, Bingyang Wang, Yijun Liang et al.
On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine “self”-…
Zixuan Fu, Bingxiang He, Yuxin Zuo et al.
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps impr…
Wenbo Pan Shujie Liu Chin-Yew Lin Jingying Zeng Xianfeng Tang, Xiangyang Zhou Yan Lu Xiaohua Jia
AI agents rely on a harness of skills, tools, and workflows to solve complex problems. Continually improving this harness is essential for adapting to new tasks. However, existing optimization methods typically require ground-truth validation sets, yet such labeled data is difficult to acquire in practical deployment…
Haowei Zhang Shudong Yang Jinlan Fu See-Kiong Ng Xipeng Qiu
Zhen Fang Wenxuan Huang Yu Zeng Yiming Zhao Shuang Chen Kaituo Feng, Yunlong Lin Lin Chen Zehui Chen Shaosheng Cao Feng Zhao
Existing Flow Matching (FM) text-to-image models suffer from two critical bottlenecks under multi-task alignment: the reward sparsity induced by scalar-valued rewards, and the gradient interference arising from jointly optimizing heterogeneous objectives, which together give rise to a “seesaw effect" of competing metr…
Yuchao Gu, Guian Fang, Yuxin Jiang, Weijia Mao, Song Han,, Han Cai, Mike Zheng Shou
Few-step video generation has been significantly advanced by consistency distillation. However, the performance of consistency-distilled models often degrades as more sampling steps are allocated at test time, limiting their effectiveness for any-step video diffusion. We argue that this limitation arises because consi…
Yuran Wang, Siqiao Huang, Mingleyang Li et al.
World–Action Models inherit world knowledge from video-generative priors, and channel it into executable control signals through embodied experience. Existing systems, however, are monolithic: the generative backbone, visual representation, architecture, information flow, inference procedure, and training data are tig…
Jian Chen, Yesheng Liang, Zhijian Liu
Autoregressive large language models (LLMs) deliver strong performance but require inherently sequential decoding, leading to high inference latency and poor GPU utilization. Speculative decoding mitigates this bottleneck by using a fast draft model whose outputs are verified in parallel by the target LLM. However, ex…
Weian Mao, Xi Lin, Wei Huang et al.
Extended reasoning in large language models (LLMs) creates severe KV cache memory bottlenecks. Leading KV cache compression methods estimate KV importance using attention scores from recent post-RoPE queries. However, queries rotate with position during RoPE, making representative queries very few, leading to poor top…