Open library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
560 papers · 558 with a Meow summary · 333 posts
The papers that shaped today’s generative AI, agent research and agent engineering, from word vectors to agentic RL.
Deep Ganguli, Liane Lovitt, Jackson Kernion et al.
We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model typ…
Ethan Perez Saffron Huang Francis Song Trevor Cai Roman Ring, John Aslanides Amelia Glaese Nat McAleese Geoffrey Irving
Language Models (LMs) often cannot be deployed because of their potential to harm users in hard-to-predict ways. Prior work identifies harmful behaviors before deployment by using human annotators to hand-write test cases. However, human annotation is expensive, limiting the number and diversity of test cases. In this…
Ryan Greenblatt, Carson Denison, Benjamin Wright et al.
We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training. First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, wh…
Dario Amodei, Chris Olah, Jacob Steinhardt et al.
Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emer…
Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit et al.
Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broad…
Geoffrey Irving, Paul Christiano, Dario Amodei
To make AI systems broadly useful for challenging real-world tasks, we need them to learn complex human goals and preferences. One approach to specifying complex goals asks humans to judge during training which agent behaviors are safe and useful, but this approach can fail if the task is too complicated for a human t…
Andy Zou, Zifan Wang, Nicholas Carlini et al.
Because “out-of-the-box” large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures—so-called “jailbreaks” against LLMs—these a…
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra et al.
Large Language Models (LLMs) are increasingly being integrated into various applications. The functionalities of recent LLMs can be flexibly modulated via natural language prompts. This renders them susceptible to targeted adversarial prompting, e.g., Prompt Injection (PI) attacks enable attackers to override original…
Mrinank Sharma,∗\overset{*}{\text{,}} Meg Tong,∗\overset{*}{\text{,}} Tomasz Korbak, David Duvenaud, Amanda Askell et al.
Human feedback is commonly utilized to finetune AI assistants. But human feedback can encourage model responses that match user beliefs over truthful ones, a behavior known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning used human feedback, and the potential role of human prefere…
Bowen Baker Joost Huizinga
Mitigating reward hacking—where AI systems misbehave due to flaws or misspecifications in their learning objectives—remains a key challenge in constructing capable and aligned models. We show that we can monitor a frontier reasoning model, such as OpenAI o3-mini, for reward hacking in agentic coding environments by us…
Mrinank Sharma Meg Tong Jesse Mu Jerry Wei Jorrit Kruthoff Scott Goodfriend Euan Ong Alwin Peng Raj Agarwal Cem Anil Amanda Askell Nathan Bailey Joe Benton Emma Bluemke Samuel R. Bowman Eric Christiansen Hoagy Cunningham Andy Dau Anjali Gopal Rob Gilson Logan Graham Logan Howard Nimit Kalra Taesung Lee Kevin Lin Peter Lofgren Francesco Mosconi Clare O’Hara Catherine Olsson Linda Petrini Samir Rajani Nikhil Saxena Alex Silverstein Tanya Singh Theodore Sumers Leonard Tang Kevin K. Troy Constantin Weisser Ruiqi Zhong Giulio Zhou Jan Leike Jared Kaplan Ethan Perez Safeguards Research Team, Anthropic
Large language models (LLMs) are vulnerable to universal jailbreaks—prompting strategies that systematically bypass model safeguards and enable users to carry out harmful processes that require many model interactions, like manufacturing illegal substances at scale. To defend against these attacks, we introduce Consti…
Milad Nasr, Nicholas Carlini, Jonathan Hayase et al.
This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset. We show an adversary can extract gigabytes of training data from open-source language models like Pythia or GPT-Neo, semi-open model…
Aengus Lynch, Benjamin Wright, Caleb Larson et al.
We stress-tested 16 leading models from multiple developers in hypothetical corporate environments to identify potentially risky agentic behaviors before they cause real harm. In the scenarios, we allowed models to autonomously send emails and access sensitive information. They were assigned only harmless business goa…
Tomek Korbak UK AI Security Institute Mikita Balesni Apollo Research Elizabeth Barnes METR Yoshua Bengio University of Montreal & Mila Joe Benton Anthropic Joseph Bloom UK AI Security Institute Mark Chen OpenAI Alan Cooney UK AI Security Institute Allan Dafoe Google DeepMind Anca Dragan Google DeepMind Scott Emmons Google DeepMind Owain Evans Truthful AI & UC Berkeley David Farhi OpenAI Ryan Greenblatt Redwood Research Dan Hendrycks Center for AI Safety Marius Hobbhahn Apollo Research Evan Hubinger Anthropic Geoffrey Irving UK AI Security Institute Erik Jenner Google DeepMind Daniel Kokotajlo AI Futures Project Victoria Krakovna Google DeepMind Shane Legg Google DeepMind David Lindner Google DeepMind David Luan Amazon Aleksander Mądry OpenAI Julian Michael Scale AI Neel Nanda Google DeepMind Dave Orr Google DeepMind Jakub Pachocki OpenAI Ethan Perez Anthropic Mary Phuong Google DeepMind Fabien Roger Anthropic Joshua Saxe Meta Buck Shlegeris Redwood Research Martín Soto UK AI Security Institute Eric Steinberger Magic Jasmine Wang UK AI Security Institute Wojciech Zaremba OpenAI Bowen Baker OpenAI Rohin Shah Google DeepMind Vlad Mikulik Anthropic
AI systems that “think” in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recomme…
Samuel Marks, Johannes Treutlein, Trenton Bricken, Jack Lindsey, Jonathan Marcus,
We study the feasibility of conducting **alignment audits**: investigations into whether models have undesired objectives. As a testbed, we train a language model with a hidden objective. Our training pipeline first teaches the model about exploitable errors in RLHF reward models (RMs), then trains the model to exploi…
We study subliminal learning, a surprising phenomenon where language models transmit behavioral traits via semantically unrelated data. In our main experiments, a “teacher” model with some trait T (such as liking owls or being misaligned) generates a dataset consisting solely of number sequences. Remarkably, a “studen…
Monte MacDiarmid, Benjamin Wright, Jonathan Uesato et al.
We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic prod…
Edoardo Debenedetti, Jie Zhang, Mislav Balunovic et al.
AI agents aim to solve complex tasks by combining text-based reasoning with external tool calls. Unfortunately, AI agents are vulnerable to prompt injection attacks where data returned by external tools hijacks the agent to execute malicious tasks. To measure the adversarial robustness of AI agents, we introduce Agent…
Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan et al.
Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an untrusted environment. However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data. In this paper we propose CaMeL, a robust defense that creates a protective system layer around the LLM, sec…
Alexander Panfilov*, David Schmotz*, Ilia Shumailov* et al.
Leading large language model providers now conceal their models’ step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage. Rather than storing these traces server-side, providers return them to the client as blocks of encrypted text, which the client passes back with…
Natalie Shapira Chris Wendler Avery Yen, Gabriele Sarti Koyena Pal Olivia Floody Adam Belfki, Alex Loftus Aditya Ratan Jannali Nikhil Prakash Jasmine Cui et al.
We report an exploratory red-teaming study of autonomous language-model–powered agents deployed in a live laboratory environment with persistent memory, email accounts, Discord access, file systems, and shell execution. Over a two-week period, twenty AI researchers interacted with the agents under benign and adversari…
Shanghai Artificial Intelligence Laboratory, https://github.com/AI45Lab/AgentDoG, https://huggingface.co/collections/AI45Research/agentdog
The rise of AI agents introduces complex safety and security challenges arising from autonomous tool use and environmental interactions. Current guardrail models lack agentic risk awareness and transparency in risk diagnosis. To introduce an agentic guardrail that covers complex and numerous risky behaviors, we use a…
Yutao Mou, Zhangchi Xue, Lijun Li et al.
While LLM-based agents can interact with environments via invoking external tools, their expanded capabilities also amplify security risks. Monitoring step-level tool invocation behaviors in real time and proactively intervening before unsafe execution is critical for agent deployment, yet remains underexplored. In th…