Open library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
560 papers · 558 with a Meow summary · 333 posts
The selected papers and blogs: the essential papers of generative AI and agents, and the key lab publications and essays, each with Meow and Standard summaries.
Deep Ganguli, Liane Lovitt, Jackson Kernion et al.
We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model typ…
Ethan Perez Saffron Huang Francis Song Trevor Cai Roman Ring, John Aslanides Amelia Glaese Nat McAleese Geoffrey Irving
Language Models (LMs) often cannot be deployed because of their potential to harm users in hard-to-predict ways. Prior work identifies harmful behaviors before deployment by using human annotators to hand-write test cases. However, human annotation is expensive, limiting the number and diversity of test cases. In this…
Ryan Greenblatt, Carson Denison, Benjamin Wright et al.
We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training. First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, wh…
Dario Amodei, Chris Olah, Jacob Steinhardt et al.
Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emer…
Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit et al.
Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broad…
Geoffrey Irving, Paul Christiano, Dario Amodei
To make AI systems broadly useful for challenging real-world tasks, we need them to learn complex human goals and preferences. One approach to specifying complex goals asks humans to judge during training which agent behaviors are safe and useful, but this approach can fail if the task is too complicated for a human t…
Andy Zou, Zifan Wang, Nicholas Carlini et al.
Because “out-of-the-box” large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures—so-called “jailbreaks” against LLMs—these a…
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra et al.
Large Language Models (LLMs) are increasingly being integrated into various applications. The functionalities of recent LLMs can be flexibly modulated via natural language prompts. This renders them susceptible to targeted adversarial prompting, e.g., Prompt Injection (PI) attacks enable attackers to override original…
Mrinank Sharma,∗\overset{*}{\text{,}} Meg Tong,∗\overset{*}{\text{,}} Tomasz Korbak, David Duvenaud, Amanda Askell et al.
Human feedback is commonly utilized to finetune AI assistants. But human feedback can encourage model responses that match user beliefs over truthful ones, a behavior known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning used human feedback, and the potential role of human prefere…
Bowen Baker Joost Huizinga
Mitigating reward hacking—where AI systems misbehave due to flaws or misspecifications in their learning objectives—remains a key challenge in constructing capable and aligned models. We show that we can monitor a frontier reasoning model, such as OpenAI o3-mini, for reward hacking in agentic coding environments by us…
Mrinank Sharma Meg Tong Jesse Mu Jerry Wei Jorrit Kruthoff Scott Goodfriend Euan Ong Alwin Peng Raj Agarwal Cem Anil Amanda Askell Nathan Bailey Joe Benton Emma Bluemke Samuel R. Bowman Eric Christiansen Hoagy Cunningham Andy Dau Anjali Gopal Rob Gilson Logan Graham Logan Howard Nimit Kalra Taesung Lee Kevin Lin Peter Lofgren Francesco Mosconi Clare O’Hara Catherine Olsson Linda Petrini Samir Rajani Nikhil Saxena Alex Silverstein Tanya Singh Theodore Sumers Leonard Tang Kevin K. Troy Constantin Weisser Ruiqi Zhong Giulio Zhou Jan Leike Jared Kaplan Ethan Perez Safeguards Research Team, Anthropic
Large language models (LLMs) are vulnerable to universal jailbreaks—prompting strategies that systematically bypass model safeguards and enable users to carry out harmful processes that require many model interactions, like manufacturing illegal substances at scale. To defend against these attacks, we introduce Consti…
Milad Nasr, Nicholas Carlini, Jonathan Hayase et al.
This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset. We show an adversary can extract gigabytes of training data from open-source language models like Pythia or GPT-Neo, semi-open model…
Aengus Lynch, Benjamin Wright, Caleb Larson et al.
We stress-tested 16 leading models from multiple developers in hypothetical corporate environments to identify potentially risky agentic behaviors before they cause real harm. In the scenarios, we allowed models to autonomously send emails and access sensitive information. They were assigned only harmless business goa…
Tomek Korbak UK AI Security Institute Mikita Balesni Apollo Research Elizabeth Barnes METR Yoshua Bengio University of Montreal & Mila Joe Benton Anthropic Joseph Bloom UK AI Security Institute Mark Chen OpenAI Alan Cooney UK AI Security Institute Allan Dafoe Google DeepMind Anca Dragan Google DeepMind Scott Emmons Google DeepMind Owain Evans Truthful AI & UC Berkeley David Farhi OpenAI Ryan Greenblatt Redwood Research Dan Hendrycks Center for AI Safety Marius Hobbhahn Apollo Research Evan Hubinger Anthropic Geoffrey Irving UK AI Security Institute Erik Jenner Google DeepMind Daniel Kokotajlo AI Futures Project Victoria Krakovna Google DeepMind Shane Legg Google DeepMind David Lindner Google DeepMind David Luan Amazon Aleksander Mądry OpenAI Julian Michael Scale AI Neel Nanda Google DeepMind Dave Orr Google DeepMind Jakub Pachocki OpenAI Ethan Perez Anthropic Mary Phuong Google DeepMind Fabien Roger Anthropic Joshua Saxe Meta Buck Shlegeris Redwood Research Martín Soto UK AI Security Institute Eric Steinberger Magic Jasmine Wang UK AI Security Institute Wojciech Zaremba OpenAI Bowen Baker OpenAI Rohin Shah Google DeepMind Vlad Mikulik Anthropic
AI systems that “think” in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recomme…
Samuel Marks, Johannes Treutlein, Trenton Bricken, Jack Lindsey, Jonathan Marcus,
We study the feasibility of conducting **alignment audits**: investigations into whether models have undesired objectives. As a testbed, we train a language model with a hidden objective. Our training pipeline first teaches the model about exploitable errors in RLHF reward models (RMs), then trains the model to exploi…
We study subliminal learning, a surprising phenomenon where language models transmit behavioral traits via semantically unrelated data. In our main experiments, a “teacher” model with some trait T (such as liking owls or being misaligned) generates a dataset consisting solely of number sequences. Remarkably, a “studen…
Monte MacDiarmid, Benjamin Wright, Jonathan Uesato et al.
We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic prod…
Edoardo Debenedetti, Jie Zhang, Mislav Balunovic et al.
AI agents aim to solve complex tasks by combining text-based reasoning with external tool calls. Unfortunately, AI agents are vulnerable to prompt injection attacks where data returned by external tools hijacks the agent to execute malicious tasks. To measure the adversarial robustness of AI agents, we introduce Agent…
Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan et al.
Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an untrusted environment. However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data. In this paper we propose CaMeL, a robust defense that creates a protective system layer around the LLM, sec…
Alexander Panfilov*, David Schmotz*, Ilia Shumailov* et al.
Leading large language model providers now conceal their models’ step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage. Rather than storing these traces server-side, providers return them to the client as blocks of encrypted text, which the client passes back with…
Natalie Shapira Chris Wendler Avery Yen, Gabriele Sarti Koyena Pal Olivia Floody Adam Belfki, Alex Loftus Aditya Ratan Jannali Nikhil Prakash Jasmine Cui et al.
We report an exploratory red-teaming study of autonomous language-model–powered agents deployed in a live laboratory environment with persistent memory, email accounts, Discord access, file systems, and shell execution. Over a two-week period, twenty AI researchers interacted with the agents under benign and adversari…
Shanghai Artificial Intelligence Laboratory, https://github.com/AI45Lab/AgentDoG, https://huggingface.co/collections/AI45Research/agentdog
The rise of AI agents introduces complex safety and security challenges arising from autonomous tool use and environmental interactions. Current guardrail models lack agentic risk awareness and transparency in risk diagnosis. To introduce an agentic guardrail that covers complex and numerous risky behaviors, we use a…
Yutao Mou, Zhangchi Xue, Lijun Li et al.
While LLM-based agents can interact with environments via invoking external tools, their expanded capabilities also amplify security risks. Monitoring step-level tool invocation behaviors in real time and proactively intervening before unsafe execution is critical for agent deployment, yet remains underexplored. In th…
Anthropic
Many-shot jailbreaking turns long-context example learning into a safety weakness: enough faux dialogues showing harmful compliance can make a model answer a target query it was trained to refuse.
Anthropic
The authors report a first known demonstration that specification gaming can sometimes generalize into reward tampering.
Anthropic
The paper introduces four evaluations for covert sabotage—misleading people, planting code bugs, hiding capabilities, and weakening oversight—so developers can probe risks that may grow with model capability.
Anthropic
SHADE-Arena finds that current agents rarely complete a complex sabotage task undetected, while the best tested monitor still needs a high false-alarm rate to catch most attempts.
Anthropic
Petri turns risky-behavior evaluation into parallel agent-run tests, but its authors frame the scores as provisional signals for triage, not definitive safety judgments.
Anthropic
Poisoning success tracked a near-constant document count, not a percentage of training data, challenging the assumption that larger models need proportionally more poisoned data.
Anthropic
The study shows Claude can improve a tightly measurable alignment-research task, but this does not make frontier models general-purpose alignment scientists.
Anthropic
Generic linear probes separated defection-inducing prompts from safe ones in intentionally backdoored sleeper agents, with AUROC above 99%; whether the method works for naturally deceptive models remains open.
Anthropic
“Character” here means learned conversational dispositions, not personhood or a claim that Claude is objective.
Anthropic
Anthropic argues that AI evaluations are difficult to build and interpret, even though effective AI governance depends on meaningful evaluations.
Anca Dragan, Rohin Shah, Four Flynn and Shane Legg · Google DeepMind
Google DeepMind presents AGI safety as a readiness effort, not a finished guarantee: assess four risk areas, focus deeply on misuse and misalignment, and build technical and institutional safeguards while open problems remain.
Anca Dragan, Helen King and Allan Dafoe · Google DeepMind
The framework turns concern about future AI harms into a process: define capability thresholds, test for approaching them, then apply safeguards.
Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, Shane Legg · Google DeepMind
Specification gaming means satisfying an objective’s literal specification without achieving the intended outcome.
Toby Shevlane · Google DeepMind
Shevlane proposes evaluating general-purpose models for extreme-risk capabilities and alignment, then using the results to guide safety and governance decisions.
Rohin Shah and Four Flynn · Google DeepMind
The roadmap’s key bet: internal agent safety must hold even when alignment is imperfect.
Lilian Weng
Reward hacking means a high score can mark a loophole, not task success; in RLHF, optimization can also make incorrect answers more persuasive to human evaluators.
Lilian Weng
Extrinsic hallucination is fabricated content that lacks grounding in context or world knowledge. The post argues that reliable models must be factual and acknowledge when they do not know.