The library

Key publications from Anthropic, OpenAI, Google DeepMind, Meta, DeepSeek, Qwen, Z.ai, Kimi and other labs, plus the most influential essays and blogs.

1. Transformer: A Novel Neural Network Architecture for Language Understanding

   Google Research

   The Transformer’s central change is direct self-attention: words can connect across a sentence without recurrent word-by-word processing.

   research.googleSummariesFoundations

2. Contrastive Representation Learning

   Lilian Weng

   Contrastive representation learning places related samples close together in an embedding space and separates unrelated ones. Its effectiveness depends on how it constructs positive pairs and negatives.

   lilianweng.github.ioSummariesFoundationsMay 2021

3. Attention? Attention!

   Lilian Weng

   Attention replaces one all-purpose summary with a different weighted view of the input for each prediction.

   lilianweng.github.ioSummariesFoundationsJun 2018

4. Policy Gradient Algorithms

   Lilian Weng

   The key is to inspect what each policy-gradient method estimates, reuses, and constrains—not just its optimizer name.

   lilianweng.github.ioSummariesFoundationsApr 2018

5. The Unreasonable Effectiveness of Recurrent Neural Networks

   Andrej Karpathy

   An RNN reuses a learned update and carries state across steps, so it can handle sequences of varying length. Character-level LSTMs show both the structure this can learn and the errors that remain.

   karpathy.github.ioSummariesFoundations

6. A Recipe for Training Neural Networks

   Andrej Karpathy

   Neural-net training can fail silently: a runnable model may still be logically misconfigured, so verify each step before adding complexity.

   karpathy.github.ioSummariesFoundations

7. Understanding LSTM Networks -- colah's blog

   Chris Olah

   Long Short Term Memory networks (LSTMs) add a gated cell state that lets useful information travel through a recurrent network more easily than in a standard RNN.

   colah.github.ioSummariesFoundations

8. Attention and Augmented Recurrent Neural Networks

   Chris Olah · Distill

   The article argues that attention can extend recurrent networks beyond sequence processing by letting them use memory, focus on inputs, vary computation, and build programs.

   distill.pubSummariesFoundationsSep 2016

9. The Illustrated Transformer

   Jay Alammar

   A word can use clues from elsewhere in its sentence: the Transformer learns how much to attend to each position.

   jalammar.github.ioSummariesFoundations

10. The Illustrated GPT-2 (Visualizing Transformer Language Models)

    Jay Alammar

    GPT-2’s text generation came from scaling a familiar decoder-only Transformer: causal masking lets it predict the next token without seeing future tokens, while self-attention builds context.

    jalammar.github.ioSummariesFoundations

11. The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)

    Jay Alammar

    BERT paired reusable pretraining with a Transformer encoder that can use left and right context, then adapt to labeled NLP tasks.

    jalammar.github.ioSummariesFoundations