Open library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
560 papers · 558 with a Meow summary · 333 posts
The selected papers and blogs: the essential papers of generative AI and agents, and the key lab publications and essays, each with Meow and Standard summaries.
Tomas Mikolov, Kai Chen, Greg Corrado et al.
We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks.…
Dzmitry Bahdanau, KyungHyun Cho Yoshua Bengio
Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural m…
Ilya Sutskever, Oriol Vinyals, Quoc V. Le
Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learni…
Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton
Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to comput…
Ashish Vaswani, Noam Shazeer, Niki Parmar et al.
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on…
Matthew E. Peters, Mark Neumann, Mohit Iyyer et al.
We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirection…
Colin Raffel craffel@gmail.com, Noam Shazeer noam@google.com, Adam Roberts adarob@google.com et al.
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper,…
Dan Hendrycks, Kevin Gimpel
We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $x\Phi(x)$, where $\Phi(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in R…
Noam Shazeer
Gated Linear Units [[Dauphin et al., 2016](#bib.bib1)] consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-f…
Jianlin Su, Yu Lu, Shengfeng Pan et al.
Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transform…
Biao Zhang, Rico Sennrich
Layer normalization (LayerNorm) has been successfully applied to various deep neural networks to help stabilize training and boost model convergence because of its capability in handling re-centering and re-scaling of both inputs and weight matrix. However, the computational overhead introduced by LayerNorm makes thes…
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz et al.
The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practic…
Geoffrey Hinton, Oriol Vinyals, Jeff Dean
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions [[3](#bib.bib3)]. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to a…
Diederik P. Kingma, Max Welling
How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets? We introduce a stochastic variational inference and learning algorithm that scales to large datasets and, under some mild di…
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza et al.
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model $G$ that captures the data distribution, and a discriminative model $D$ that estimates the probability that a sample came from the training data rather than $G$. The t…
Volodymyr Mnih, Koray Kavukcuoglu, David Silver et al.
We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future…
David Silver, Thomas Hubert, Julian Schrittwieser et al.
The game of chess is the most widely-studied domain in the history of artificial intelligence. The strongest programs are based on a combination of sophisticated search techniques, domain-specific adaptations, and handcrafted evaluation functions that have been refined by human experts over several decades. In contras…
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert et al.
Constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Tree-based planning methods have enjoyed huge success in challenging domains, such as chess and Go, where a perfect simulator is available. However, in real-world problems the dynamics gov…
John Schulman, Philipp Moritz, Sergey Levine et al.
Policy gradient methods are an appealing approach in reinforcement learning because they directly optimize the cumulative reward and can straightforwardly be used with nonlinear function approximators such as neural networks. The two main challenges are the large number of samples typically required, and the difficult…
John Schulman, Sergey Levine, Philipp Moritz et al.
We describe an iterative procedure for optimizing policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified procedure, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is similar to natural policy gradient methods an…
Lili Chen, Kevin Lu, Aravind Rajeswaran et al.
We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an archite…
Rishi Bommasani*, Drew A. Hudson, Ehsan Adeli et al.
AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) trained on broad data (generally using self-supervision at scale) that can be adapted to a wide range of downstream tasks. We call these models foundation models to underscore their critically central yet incomplete character. This r…
Google Research
The Transformer’s central change is direct self-attention: words can connect across a sentence without recurrent word-by-word processing.
Lilian Weng
Contrastive representation learning places related samples close together in an embedding space and separates unrelated ones. Its effectiveness depends on how it constructs positive pairs and negatives.
Lilian Weng
Attention replaces one all-purpose summary with a different weighted view of the input for each prediction.
Lilian Weng
The key is to inspect what each policy-gradient method estimates, reuses, and constrains—not just its optimizer name.
Andrej Karpathy
An RNN reuses a learned update and carries state across steps, so it can handle sequences of varying length. Character-level LSTMs show both the structure this can learn and the errors that remain.
Andrej Karpathy
Neural-net training can fail silently: a runnable model may still be logically misconfigured, so verify each step before adding complexity.
Chris Olah
Long Short Term Memory networks (LSTMs) add a gated cell state that lets useful information travel through a recurrent network more easily than in a standard RNN.
Chris Olah · Distill
The article argues that attention can extend recurrent networks beyond sequence processing by letting them use memory, focus on inputs, vary computation, and build programs.
Jay Alammar
A word can use clues from elsewhere in its sentence: the Transformer learns how much to attend to each position.
Jay Alammar
GPT-2’s text generation came from scaling a familiar decoder-only Transformer: causal masking lets it predict the next token without seeing future tokens, while self-attention builds context.
Jay Alammar
BERT paired reusable pretraining with a Transformer encoder that can use left and right context, then adapt to labeled NLP tasks.