Open library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
563 papers · 561 with a Meow summary · 333 posts
The selected papers and blogs: the essential papers of generative AI and agents, and the key lab publications and essays, each with Meow and Standard summaries.
We study subliminal learning, a surprising phenomenon where language models transmit behavioral traits via semantically unrelated data. In our main experiments, a “teacher” model with some trait T (such as liking owls or being misaligned) generates a dataset consisting solely of number sequences. Remarkably, a “studen…
Runjin Chen, Andy Arditi, Henry Sleight et al.
Large language models interact with users through a simulated “Assistant” persona. While the Assistant is typically trained to be helpful, harmless, and honest, it sometimes deviates from these ideals. In this paper, we identify directions in the model’s activation space—persona vectors—underlying several traits, such…
Monte MacDiarmid, Benjamin Wright, Jonathan Uesato et al.
We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic prod…
⋆\star Adam Fourney, Gagan Bansal, Hussein Mozannar et al.
Modern AI agents, driven by advances in large foundation models, promise to enhance our productivity and transform our lives by augmenting our knowledge and capabilities. To achieve this vision, AI agents must effectively plan, perform multi-step reasoning and actions, respond to novel observations, and recover from e…
Mingchen Zhuge, Changsheng Zhao, Dylan R. Ashley et al.
Contemporary evaluation techniques are inadequate for agentic systems. These approaches either focus exclusively on final outcomes—ignoring the step-by-step nature of agentic systems, or require excessive manual labour. To address this, we introduce the **Agent-as-a-Judge** framework, wherein agentic systems are used…
Edoardo Debenedetti, Jie Zhang, Mislav Balunovic et al.
AI agents aim to solve complex tasks by combining text-based reasoning with external tool calls. Unfortunately, AI agents are vulnerable to prompt injection attacks where data returned by external tools hijacks the agent to execute malicious tasks. To measure the adversarial robustness of AI agents, we introduce Agent…
Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan et al.
Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an untrusted environment. However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data. In this paper we propose CaMeL, a robust defense that creates a protective system layer around the LLM, sec…
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and…
Xuying Ning, Katherine Tieu, Dongqi Fu et al.
**Abstract:** Recent large language models (LLMs) have demonstrated strong capabilities in understanding and generating code, from competitive programming to repository-level software engineering. In emerging agentic systems, code is no longer only a target output. It increasingly serves as an operational substrate fo…
Jiazhan Feng, Shijue Huang, Xingwei Qu et al.
While reasoning models (e.g., DeepSeek R1) trained with reinforcement learning (RL), excel in textual reasoning, they struggle in scenarios requiring structured problem-solving, such as geometric reasoning, concise computation, or complex equation solving—areas where computational tools like code interpreters (CI) dem…
Kuan Li, Zhongwang Zhang, Huifeng Yin et al.
Transcending human cognitive limitations represents a critical frontier in LLM training. Proprietary agentic systems like DeepResearch have demonstrated superhuman capabilities on extremely complex information-seeking benchmarks such as BrowseComp, a feat previously unattainable. We posit that their success hinges on…
Yuran Wang, Siqiao Huang, Mingleyang Li et al.
World–Action Models inherit world knowledge from video-generative priors, and channel it into executable control signals through embodied experience. Existing systems, however, are monolithic: the generative backbone, visual representation, architecture, information flow, inference procedure, and training data are tig…
Shaobo Wang Xuan Ouyang Tianyi Xu Yuzheng Hu Jialin Liu, Guo Chen Tianyu Zhang Junhao Zheng Kexin Yang Xingzhang Ren, Dayiheng Liu Linfeng Zhang
As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall ([Villalobos et al., 2022](#bib.bib21)), pre-training is shifting from more tokens to better tokens. However, existing methods either rely on heuristic static filters that ignore training dynamics, or use dynamic yet optimizer-agnos…
Idan Shenfeld, Mehul Damani, Jonas Hübotter et al.
Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models. While on-policy reinforcement learning can reduce forgetting, it requires explicit reward functions that are often unavailable. Learning from expert de…
Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV c…
Jianuo Huang Yaojie Zhang Qituan Zhang Hao Lin Hanlin Xu Linfeng Zhang
Speculative decoding accelerates LLM inference by drafting multiple tokens and verifying them in parallel with the target model. However, its practical speedup is constrained by the trade-off between draft quality and drafting cost: autoregressive drafters model causal dependencies among draft tokens but incur sequent…
Weian Mao, Xi Lin, Wei Huang et al.
Extended reasoning in large language models (LLMs) creates severe KV cache memory bottlenecks. Leading KV cache compression methods estimate KV importance using attention scores from recent post-RoPE queries. However, queries rotate with position during RoPE, making representative queries very few, leading to poor top…
Yushi Bai, Qian Dong, Ting Jiang et al.
Long-context agentic workflows have emerged as a defining use case for large language models, making attention efficiency critical for both inference speed and serving cost. Sparse attention addresses this challenge effectively, and DeepSeek Sparse Attention (DSA) is a representative production-grade solution: a light…
Jian Chen, Yesheng Liang, Zhijian Liu
Autoregressive large language models (LLMs) deliver strong performance but require inherently sequential decoding, leading to high inference latency and poor GPU utilization. Speculative decoding mitigates this bottleneck by using a fast draft model whose outputs are verified in parallel by the target LLM. However, ex…
Haowei Zhang Shudong Yang Jinlan Fu See-Kiong Ng Xipeng Qiu
Yuchao Gu, Guian Fang, Yuxin Jiang, Weijia Mao, Song Han,, Han Cai, Mike Zheng Shou
Few-step video generation has been significantly advanced by consistency distillation. However, the performance of consistency-distilled models often degrades as more sampling steps are allocated at test time, limiting their effectiveness for any-step video diffusion. We argue that this limitation arises because consi…
Zhen Fang Wenxuan Huang Yu Zeng Yiming Zhao Shuang Chen Kaituo Feng, Yunlong Lin Lin Chen Zehui Chen Shaosheng Cao Feng Zhao
Existing Flow Matching (FM) text-to-image models suffer from two critical bottlenecks under multi-task alignment: the reward sparsity induced by scalar-valued rewards, and the gradient interference arising from jointly optimizing heterogeneous objectives, which together give rise to a “seesaw effect" of competing metr…
Wenbo Pan Shujie Liu Chin-Yew Lin Jingying Zeng Xianfeng Tang, Xiangyang Zhou Yan Lu Xiaohua Jia
AI agents rely on a harness of skills, tools, and workflows to solve complex problems. Continually improving this harness is essential for adapting to new tasks. However, existing optimization methods typically require ground-truth validation sets, yet such labeled data is difficult to acquire in practical deployment…
Alexander Panfilov*, David Schmotz*, Ilia Shumailov* et al.
Leading large language model providers now conceal their models’ step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage. Rather than storing these traces server-side, providers return them to the client as blocks of encrypted text, which the client passes back with…
Natalie Shapira Chris Wendler Avery Yen, Gabriele Sarti Koyena Pal Olivia Floody Adam Belfki, Alex Loftus Aditya Ratan Jannali Nikhil Prakash Jasmine Cui et al.
We report an exploratory red-teaming study of autonomous language-model–powered agents deployed in a live laboratory environment with persistent memory, email accounts, Discord access, file systems, and shell execution. Over a two-week period, twenty AI researchers interacted with the agents under benign and adversari…
Shanghai Artificial Intelligence Laboratory, https://github.com/AI45Lab/AgentDoG, https://huggingface.co/collections/AI45Research/agentdog
The rise of AI agents introduces complex safety and security challenges arising from autonomous tool use and environmental interactions. Current guardrail models lack agentic risk awareness and transparency in risk diagnosis. To introduce an agentic guardrail that covers complex and numerous risky behaviors, we use a…
Yutao Mou, Zhangchi Xue, Lijun Li et al.
While LLM-based agents can interact with environments via invoking external tools, their expanded capabilities also amplify security risks. Monitoring step-level tool invocation behaviors in real time and proactively intervening before unsafe execution is critical for agent deployment, yet remains underexplored. In th…
Zixuan Fu, Bingxiang He, Yuxin Zuo et al.
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps impr…
Yijiang Li, Bingyang Wang, Yijun Liang et al.
On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine “self”-…
Wenhan Ma, Jianyu Wei, Liang Zhao et al.
Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose **Mult…
Zhennan Shen, Yanshu Li, Qingyu Yin, Chak Tou Leong, Zhilin Wang Yanxu Chen, Rongduo Han, Sunbowen Lee, Yi R. Fung HKUST Brown University Zhejiang University Hong Kong PolyU USTC BUPT Nankai University BIT
On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A…
Xingrun Xing, Haoqing Wang, Boyan Gao et al.
On-Policy Distillation (OPD) is a fundamental technique for efficient post-training of large language models (LLMs), with broad applications in agent learning, multi-task enhancement, and model compression. However, OPD training becomes unstable when the teacher and student distributions differ substantially, as teach…
Yuanyi Wang, Su Lu, Yanggan Gu et al.
On-policy distillation (OPD) trains a student on its own rollouts with token-level teacher supervision. Recent selective OPD methods exploit the non-uniformity of OPD signals by prioritizing high-entropy or high-disagreement tokens. We revisit this principle and ask: which token-level teacher signals are actually lear…
Yaxuan Li, Yuxin Zuo, Bingxiang He et al.
On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the…
Binbin Zheng Xing Ma Yiheng Liang Jingqing Ruan Xiaoliang Fu Kepeng Lin Benchang Zhu Ke Zeng Xunliang Cai University of Science and Technology of China Meituan LongCat Interaction Team Nanjing University Fudan University Huazhong University of Science and Technology https://github.com/machine981/SCOPE
On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult. On-Policy Distillation (OPD) alleviates this by introducing dense, token-level KL supervision from a teache…
Qihan Ren, Peng Wang, Ruikun Cai et al.
A prevailing narrative in LLM post-training holds that supervised fine-tuning (SFT) memorizes while reinforcement learning (RL) generalizes. We revisit this claim for reasoning SFT with long chain-of-thought (CoT) supervision and find that cross-domain generalization is not absent but conditional, jointly shaped by op…
Chenxu Yang, Chuanyu Qin, Qingyi Si et al.
On-policy distillation (OPD) has become a popular training paradigm in the LLM community. This paradigm selects a larger model as the teacher to provide dense, fine-grained signals for each sampled trajectory, in contrast to reinforcement learning with verifiable rewards (RLVR), which only obtains sparse signals from…
Gengsheng Li, Tianyu Yang, Junfeng Fang et al.
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models. While Group Relative Policy Optimization (GRPO) is widely adopted, its coarse credit assignment uniformly penalizes failed rollouts, lacking the token-level focus needed to efficiently address…
Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, Yuqing Yang Microsoft Research KAIST SK hynix Seoul National University blog post beanie00/self-distillation-analysis
Self-distillation has emerged as an effective post-training paradigm for LLMs, often improving performance while shortening reasoning traces. However, in mathematical reasoning, we find that it can reduce response length while degrading performance. We trace this degradation to the suppression of **epistemic verbaliza…