Open library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
560 papers · 558 with a Meow summary · 333 posts
The selected papers and blogs: the essential papers of generative AI and agents, and the key lab publications and essays, each with Meow and Standard summaries.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu et al.
As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and s…
Rafael Rafailov, Archit Sharma, Eric Mitchell et al.
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality o…
Jason Wei , Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu
This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning—finetuning language models on a collection of datasets described via instructions—substantially improves zero-shot performance on unseen tasks. We take a 137B parameter pretrained lang…
Victor Sanh
Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks ([Brown et al., 2020](#bib.bib12)). It has been hypothesized that this is a consequence of implicit multitask learning in language models’ pretraining ([Radford et al., 2019](#bib.bib70)). Can zero-sh…
Hyung Won Chung, Le Hou, Shayne Longpre et al.
Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on ch…
Yizhong Wang Yeganeh Kordi Swaroop Mishra Alisa Liu, Noah A. Smith Daniel Khashabi Hannaneh Hajishirzi
Large “instruction-tuned” language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is often limited in quantity, diversity, and creativity, therefore hindering the g…
Chunting Zhou Pengfei Liu Puxin Xu Srini Iyer Jiao Sun, Yuning Mao Xuezhe Ma, Avia Efrat Ping Yu Lili Yu Susan Zhang et al.
Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training…
Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang, Zhanghao Wu Yonghao Zhuang Zi Lin Zhuohan Li Dacheng Li, Eric P. Xing Hao Zhang et al.
Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and li…
Harrison Lee, Samrat Phatale, Hassan Mansoor et al.
Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in [Bai et al. (2022b)](#bib.bib3), offers a promising alternative that trains the r…
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho et al.
We posit that to achieve superhuman agents, future models require superhuman feedback in order to provide an adequate training signal. Current approaches commonly train reward models from human preferences, which may then be bottlenecked by human performance level, and secondly these separate frozen reward models cann…
Leo Gao, John Schulman, Jacob Hilton
In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart’s law. This effect has been frequently obse…
Stephen Casper, MIT CSAIL, scasper@mit.edu et al.
Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state-of-the-art large language models (LLMs). Despite this popularity, there has been relatively little public work systematizing its flaws. In th…
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff et al.
Kahneman & Tversky’s prospect theory tells us that humans perceive random variables in a biased but well-defined manner ([1992](#bib.bib50)); for example, humans are famously loss-averse. We show that objectives for aligning LLMs with human feedback implicitly incorporate many of these biases—the success of these obje…
Yu Meng Mengzhou Xia Danqi Chen
Direct Preference Optimization (DPO) is a widely used offline preference optimization algorithm that reparameterizes reward functions in reinforcement learning from human feedback (RLHF) to enhance simplicity and training stability. In this work, we propose SimPO, a simpler yet more effective approach. The effectivene…
Language model post-training is applied to refine behaviors and unlock new skills across a wide range of language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and recipes for post-training are simultaneously the most important pieces of the puzzle and…
Amanda Askell, Yuntao Bai, Anna Chen et al.
Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as pr…
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner et al.
Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior—for example, to evaluate whether a model faithfully followed instructions or generated safe outputs. However, future superhuman models will behave in complex ways too d…
Deep Ganguli, Liane Lovitt, Jackson Kernion et al.
We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model typ…
Ethan Perez Saffron Huang Francis Song Trevor Cai Roman Ring, John Aslanides Amelia Glaese Nat McAleese Geoffrey Irving
Language Models (LMs) often cannot be deployed because of their potential to harm users in hard-to-predict ways. Prior work identifies harmful behaviors before deployment by using human annotators to hand-write test cases. However, human annotation is expensive, limiting the number and diversity of test cases. In this…
Ryan Greenblatt, Carson Denison, Benjamin Wright et al.
We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training. First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, wh…
Dario Amodei, Chris Olah, Jacob Steinhardt et al.
Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emer…
Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit et al.
Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broad…
Geoffrey Irving, Paul Christiano, Dario Amodei
To make AI systems broadly useful for challenging real-world tasks, we need them to learn complex human goals and preferences. One approach to specifying complex goals asks humans to judge during training which agent behaviors are safe and useful, but this approach can fail if the task is too complicated for a human t…
Andy Zou, Zifan Wang, Nicholas Carlini et al.
Because “out-of-the-box” large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures—so-called “jailbreaks” against LLMs—these a…
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra et al.
Large Language Models (LLMs) are increasingly being integrated into various applications. The functionalities of recent LLMs can be flexibly modulated via natural language prompts. This renders them susceptible to targeted adversarial prompting, e.g., Prompt Injection (PI) attacks enable attackers to override original…
Mrinank Sharma,∗\overset{*}{\text{,}} Meg Tong,∗\overset{*}{\text{,}} Tomasz Korbak, David Duvenaud, Amanda Askell et al.
Human feedback is commonly utilized to finetune AI assistants. But human feedback can encourage model responses that match user beliefs over truthful ones, a behavior known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning used human feedback, and the potential role of human prefere…
Bowen Baker Joost Huizinga
Mitigating reward hacking—where AI systems misbehave due to flaws or misspecifications in their learning objectives—remains a key challenge in constructing capable and aligned models. We show that we can monitor a frontier reasoning model, such as OpenAI o3-mini, for reward hacking in agentic coding environments by us…
Mrinank Sharma Meg Tong Jesse Mu Jerry Wei Jorrit Kruthoff Scott Goodfriend Euan Ong Alwin Peng Raj Agarwal Cem Anil Amanda Askell Nathan Bailey Joe Benton Emma Bluemke Samuel R. Bowman Eric Christiansen Hoagy Cunningham Andy Dau Anjali Gopal Rob Gilson Logan Graham Logan Howard Nimit Kalra Taesung Lee Kevin Lin Peter Lofgren Francesco Mosconi Clare O’Hara Catherine Olsson Linda Petrini Samir Rajani Nikhil Saxena Alex Silverstein Tanya Singh Theodore Sumers Leonard Tang Kevin K. Troy Constantin Weisser Ruiqi Zhong Giulio Zhou Jan Leike Jared Kaplan Ethan Perez Safeguards Research Team, Anthropic
Large language models (LLMs) are vulnerable to universal jailbreaks—prompting strategies that systematically bypass model safeguards and enable users to carry out harmful processes that require many model interactions, like manufacturing illegal substances at scale. To defend against these attacks, we introduce Consti…
Milad Nasr, Nicholas Carlini, Jonathan Hayase et al.
This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset. We show an adversary can extract gigabytes of training data from open-source language models like Pythia or GPT-Neo, semi-open model…
Jason Wei, Xuezhi Wang, Dale Schuurmans et al.
We explore how generating a chain of thought—a series of intermediate reasoning steps—significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain-of-…
Takeshi Kojima, Shixiang Shane Gu, Machel Reid et al.
Pretrained large language models (LLMs) are widely used in many sub-fields of natural language processing (NLP) and generally known as excellent few-shot learners with task-specific exemplars. Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-by-step…
Xuezhi Wang, Jason Wei, Dale Schuurmans et al.
Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting. It first samples a diverse set of reasoni…
Shunyu Yao, Dian Yu, Jeffrey Zhao et al.
Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions pl…
Maciej Besta, Nils Blach, Ales Kubicek et al.
We introduce Graph of Thoughts (GoT): a framework that advances prompting capabilities in large language models (LLMs) beyond those offered by paradigms such as Chain-of-Thought or Tree of Thoughts (ToT). The key idea and primary advantage of GoT is the ability to model the information generated by an LLM as an arbitr…
Karl Cobbe Vineet Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun Łukasz Kaiser Matthias Plappert Jerry Tworek Jacob Hilton Reiichiro Nakano Christopher Hesse John Schulman
State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word pr…
Hunter Lightman Vineet Kosaraju Yura Burda Harri Edwards
In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or…
Eric Zelikman, Yuhuai Wu, Jesse Mu et al.
Generating step-by-step "chain-of-thought" rationales improves language model performance on complex reasoning tasks like mathematics or commonsense question-answering. However, inducing language model rationale generation currently requires either constructing massive rationale datasets or sacrificing accuracy by usi…
Eric Zelikman, Georges Harik, Yijia Shao et al.
When writing and talking, people sometimes pause to think. Although reasoning-focused works have often framed reasoning as a method of answering questions or completing agentic tasks, reasoning is implicit in almost all written text. For example, this applies to the steps not stated between the lines of a proof or to…
Charlie Snell, Jaehoon Lee, Kelvin Xu et al.
Enabling LLMs to improve their outputs by using more test-time computation is a critical step towards building generally self-improving agents that can operate on open-ended natural language. In this paper, we study the scaling of inference-time computation in LLMs, with a focus on answering the question: if an LLM is…
Bradley Brown, Jordan Juravsky, Ryan Ehrlich et al.
Scaling the amount of compute used to train language models has dramatically improved their capabilities. However, when it comes to inference, we often limit models to making only one attempt at a problem. Here, we explore inference compute as another axis for scaling, using the simple technique of repeatedly sampling…