Open library
The essential reading of generative AI and agents, open in a clean reading view, with Meow and Standard summaries when available. No account needed to read.
560 papers · 558 with a Meow summary · 333 posts
The selected papers and blogs: the essential papers of generative AI and agents, and the key lab publications and essays, each with Meow and Standard summaries.
Tri Dao
Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock new applications in code, audio, and video generation. The attention layer is the main bottleneck in sc…
Jay Shah, Ganesh Bikshandi, Ying Zhang et al.
Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for large language models and long-context applications. FlashAttention elaborated an approach to speed up attention on GPUs through minimizing memory reads/writes. However, it has yet to take advantage of new capabilities present…
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving et al.
We present speculative sampling, an algorithm for accelerating transformer decoding by enabling the generation of multiple tokens from each transformer call. Our algorithm relies on the observation that the latency of parallel scoring of short continuations, generated by a faster but less powerful draft model, is comp…
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman et al.
We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters (LoRA).…
Tim Dettmers, Mike Lewis, Younes Belkada et al.
Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our…
Elias Frantar, Saleh Ashkboos, Torsten Hoefler et al.
Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models…
Guangxuan Xiao, Ji Lin, Mickael Seznec et al.
Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, existing methods cannot maintain accuracy and hardware efficiency at the same time. We propose SmoothQuant, a training-free, accuracy-preserving, and general-…
Noam Shazeer
Multi-head attention layers, as used in the Transformer neural sequence model, are a powerful alternative to RNNs for moving information across and between sequences. While training these layers is generally fast and simple, due to parallelizability across the length of the sequence, incremental inference (where such…
Joshua Ainslie, James Lee-Thorp, Michiel de Jong et al.
Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster inference. We (1) propose a recipe for uptraining existing multi-head language mode…
Jingyuan Liu, Jianlin Su, Xingcheng Yao et al.
Recently, the Muon optimizer ([Jordan et al., 2024](#bib.bibx16)) based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2)…
Jianuo Huang Yaojie Zhang Qituan Zhang Hao Lin Hanlin Xu Linfeng Zhang
Speculative decoding accelerates LLM inference by drafting multiple tokens and verifying them in parallel with the target model. However, its practical speedup is constrained by the trade-off between draft quality and drafting cost: autoregressive drafters model causal dependencies among draft tokens but incur sequent…
Weian Mao, Xi Lin, Wei Huang et al.
Extended reasoning in large language models (LLMs) creates severe KV cache memory bottlenecks. Leading KV cache compression methods estimate KV importance using attention scores from recent post-RoPE queries. However, queries rotate with position during RoPE, making representative queries very few, leading to poor top…
Yushi Bai, Qian Dong, Ting Jiang et al.
Long-context agentic workflows have emerged as a defining use case for large language models, making attention efficiency critical for both inference speed and serving cost. Sparse attention addresses this challenge effectively, and DeepSeek Sparse Attention (DSA) is a representative production-grade solution: a light…
Jian Chen, Yesheng Liang, Zhijian Liu
Autoregressive large language models (LLMs) deliver strong performance but require inherently sequential decoding, leading to high inference latency and poor GPU utilization. Speculative decoding mitigates this bottleneck by using a fast draft model whose outputs are verified in parallel by the target LLM. However, ex…
Haowei Zhang Shudong Yang Jinlan Fu See-Kiong Ng Xipeng Qiu
Google Research
Speculative decoding speeds autoregressive LLM inference by computing several likely tokens in parallel while preserving the target model’s exact output distribution. Its key is probabilistic checking, not simply accepting or rejecting a draft.
Lilian Weng
Transformer inference gets cheaper only when the fix targets the actual bottleneck—memory, computation, or latency—and fits the hardware that must run it.
Stuart Sul · Cursor
Cursor has open-sourced MoK, a deterministic NVL72 MoE training kernel that fuses communication and computation into one kernel.
Less Wright · Cursor
For small-batch MoE decode on Blackwell, warp decode assigns work by output rather than expert; its gain comes from removing data-layout bookkeeping, not merely fusing the old path.
Brendan O'Donoghue · Google DeepMind
DiffusionGemma generates text in parallel blocks, trading some output quality for up to 4x faster inference on dedicated GPUs.
Olivier Lacombe · Google DeepMind
A small Gemma 4 drafter proposes several tokens while the larger target model checks them in parallel.
Mistral
Mistral Small 4 combines instruct, reasoning, agentic coding and multimodal input in one model rather than asking users to choose among specialized models.
Mistral
UCX’s memory hook was meant to speed InfiniBand transfers, but in this vLLM setup it helped anonymous memory accumulate outside the heap. Disabling the hook fixed the leak without a performance hit in this use case.
Z.ai
The Infra Agent’s contribution was not just writing code: the authors credit a feedback loop that made its systems work testable. They describe this as an early step, not recursive self-improvement itself.
ben burtenshaw · DeepSeek
DeepSeek-V4 aims to make a 1M-token window practical for agents by combining cheaper long-context attention with tool-aware reasoning and execution.
Gergely Orosz · The Pragmatic Engineer
Inference engineering turns open-model access into control over serving—but its payoff depends on balancing latency, throughput, quality, reliability, and cost for a real workload.
Vicki Boykis
In the author’s experience, local models have moved from development lookups to agentic coding; the results are promising, but not benchmark proof.