MiniMax-01: Scaling Foundation Models with Lightning Attention

MiniMax

arXiv:2501.08313v1 · This workspace stays on this version. Check arXiv for newer versions

Affiliations and author notes

Please send correspondence to model@minimaxi.com.

Some source details need review 23 to check

Original formulas, table values and figures are preserved. Comparison can miss differences; use the original PDF to check details.

Reading view checked: 246 math expressions, 0 rendering failures; 51 of 51 images available.

Compared 187 of 290 source passages against 35 of 68 PDF pages.

The HTML source contains failed formatting commands; these were omitted from the reading view. The original sources are retained. (1)

Some source graphics could not be placed in the reading view. (45)
Some graphics are laid out differently in the HTML source than in the paper (text or formulas outside their frame); check the original PDF. (8)
S2_F3: the arXiv HTML draws text or formulas of this graphic outside its frame and it could not be corrected safely (too-small); the image shows the HTML layout, which differs from the PDF.
S3_F9: the arXiv HTML draws text or formulas of this graphic outside its frame and it could not be corrected safely (too-small); the image shows the HTML layout, which differs from the PDF.
S3_F10: the arXiv HTML draws text or formulas of this graphic outside its frame and it could not be corrected safely (too-small); the image shows the HTML layout, which differs from the PDF.
S3_F12: the arXiv HTML draws text or formulas of this graphic outside its frame and it could not be corrected safely (too-small); the image shows the HTML layout, which differs from the PDF.
A2_SS1_p2_pic1: the arXiv HTML draws 1 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 97% or more) without changing any text or formula. Text may look smaller than in the PDF.
A2_SS2_p2_pic1: the arXiv HTML draws text or formulas of this graphic outside its frame and it could not be corrected safely (too-small); the image shows the HTML layout, which differs from the PDF.
A2_SS3_p2_pic1: the arXiv HTML draws text or formulas of this graphic outside its frame and it could not be corrected safely (too-small, verification-failed); the image shows the HTML layout, which differs from the PDF.
A2_SS4_p1_pic1: the arXiv HTML draws text or formulas of this graphic outside its frame and it could not be corrected safely (too-small); the image shows the HTML layout, which differs from the PDF.
A2_SS4_p2_pic1: the arXiv HTML draws 1 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 69% or more) without changing any text or formula. Text may look smaller than in the PDF.
A2_SS5_p1_pic1: the arXiv HTML draws 2 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 83% or more) without changing any text or formula. Text may look smaller than in the PDF.
A2_SS6_p1_pic1: the arXiv HTML draws 1 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 82% or more) without changing any text or formula. Text may look smaller than in the PDF.
A2_SS7_p1_pic1: the arXiv HTML draws text or formulas of this graphic outside its frame and it could not be corrected safely (verification-failed); the image shows the HTML layout, which differs from the PDF.
A2_SS8_p1_pic1: the arXiv HTML draws 1 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 93% or more) without changing any text or formula. Text may look smaller than in the PDF.
A2_SS9_p2_pic1: the arXiv HTML draws 1 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 91% or more) without changing any text or formula. Text may look smaller than in the PDF.
A3_SS1_p1_pic1: the arXiv HTML draws 3 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 54% or more) without changing any text or formula. Text may look smaller than in the PDF.
A3_SS2_p1_pic1: the arXiv HTML draws 3 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 51% or more) without changing any text or formula. Text may look smaller than in the PDF.
A3_SS2_p2_pic1: the arXiv HTML draws 2 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 99% or more) without changing any text or formula. Text may look smaller than in the PDF.
A3_SS3_p1_pic1: the arXiv HTML draws 3 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 51% or more) without changing any text or formula. Text may look smaller than in the PDF.
A3_SS3_p2_pic1: the arXiv HTML draws 3 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 51% or more) without changing any text or formula. Text may look smaller than in the PDF.
A3_SS4_p1_pic1: the arXiv HTML draws 3 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 51% or more) without changing any text or formula. Text may look smaller than in the PDF.
This long paper was only partially compared; the original sources remain available.

51 of 51 figure, table and graphic images captured. Open original PDF

Abstract

We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer contexts. The core lies in lightning attention and its efficient scaling. To maximize computational capacity, we integrate it with Mixture of Experts (MoE), creating a model with 32 experts and 456 billion total parameters, of which 45.9 billion are activated for each token. We develop an optimized parallel strategy and highly efficient computation-communication overlap techniques for MoE and lightning attention. This approach enables us to conduct efficient training and inference on models with hundreds of billions of parameters across contexts spanning millions of tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost. Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens. Experiments on both standard and in-house benchmarks show that our models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering a 20-32 times longer context window. We publicly release MiniMax-01 at https://github.com/MiniMax-AI.

Figure 1: Benchmark performance. (a) MiniMax-Text-01 on core text benchmarks. (b) MiniMax-VL-01 on core multimodal benchmarks. (c) MiniMax-Text-01 on the long-context RULER (Hsieh et al., 2024) benchmark. The performance of leading commercial and open-source models is presented for reference.

Large Language Models (LLMs) (Hurst et al., 2024; Team et al., 2024a; Anthropic, 2024; Dubey et al., 2024) and Vision Language Models (VLMs) (Hurst et al., 2024; Team et al., 2024a; Anthropic, 2024; Dubey et al., 2024) have made rapid progress in recent years, excelling at tasks like knowledge Q&A, complex reasoning, mathematics, coding, and vision-language understanding. The context window for most models currently ranges from 32K to 256K tokens. However, these lengths often fall short of practical needs—whether using a professional book as context, assisting with an entire programming project, or maximizing the potential of in-context learning through many-shot examples.

Context window expansion in the past two years has primarily resulted from more powerful GPUs and better I/O-aware softmax attention implementation (Dao et al., 2022; Liu et al., 2024a). However, extending these windows further has proven challenging. This limitation arises from the inherent quadratic computational complexity of the transformer (Vaswani et al., 2017) architecture—further length extension causes computational demands to grow much faster than hardware capabilities can match. To address this challenge, researchers have proposed various methods for reducing the attention mechanism’s computational complexity: sparse attention (Beltagy et al., 2020; Zaheer et al., 2020), linear attention (Qin et al., 2022b; Qin et al., 2022a; Qin et al., 2024c), long convolutions (Qin et al., 2023a), state space models (the Mamba series) (Gu and Dao, 2024; Dao and Gu, 2024; Ren et al., 2024; Glorioso et al., 2024; Team et al., 2024b), and linear RNNs (Qin et al., 2023b; Qin et al., 2024d). Despite their theoretical promise, these innovations have seen limited adoption in commercial-scale models.

In this report, we aim to build a model that matches the performance of leading commercial models while providing a context window longer by an order of magnitude. This ambitious objective requires carefully balancing multiple factors: network architecture, data, and computation.

Our approach begins with selecting the most promising architecture, succeeded by the optimization of the underlying training and inference framework to ensure its support. For the network architecture, we required linear attention—not just theoretically sound but highly efficient in practice, especially with long contexts. After extensive experimentation, we settled on a hybrid architecture mainly using lightning attention (Qin et al., 2024b), an I/O-aware implementation of a linear attention variant (Qin et al., 2022a). In the architecture, one transformer block with softmax attention follows every seven transnormer blocks (Qin et al., 2022a) with lightning attention.

We determined the model’s total parameters based on a practical constraint: the ability to process more than 1 million tokens on a single machine with up to 8 GPUs and 640GB memory using 8-bit quantization. To maximize parameter and computation capacity, we implemented a Mixture of Experts (MoE) (Lepikhin et al., 2021; Fedus et al., 2022). We comprehensively consider training resources, inference resources, and the final model performance, aiming to find a better balance among the three. Extensive experiments guided us toward the final model specifications: 456 billion parameters, 45.9 billion activations, and 32 experts.

Existing distributed training and inference frameworks are primarily optimized for softmax attention. However, our novel architecture, which integrates lightning attention, softmax attention, and MoE, necessitates a complete redesign of both our training and inference frameworks. Furthermore, the framework must possess the capability to support the training and inference of models with hundreds of billions of parameters and context windows extending over millions of tokens. To this end, we implement the all-to-all communication in MoE using expert parallel (EP) and expert tensor parallel (ETP). It aims to minimize the overhead associated with inter-GPU communication. To facilitate context windows with unlimited expansion, we design varlen ring attention to reduce the redundancy in computation and the improved version of Linear Attention Sequence Parallelism (LASP) (Sun et al., 2024) to fully utilize the device’s parallel capabilities. Additionally, we have implemented a comprehensive set of CUDA kernels tailored for lightning attention inference, achieving over 75% Model Flops Utilization (MFU) (Chowdhery et al., 2023) end-to-end on the Nvidia H20.

Building upon the architecture design and computation optimizations, we train our foundational language model, MiniMax-Text-01. Our pre-training process began with curating a diverse and high-quality corpus through rigorous data cleaning, reward-based quality enhancement, and better

Figure 2: Prefilling latency of different models. The MiniMax-Text-01 and Llama3-70B models are tested on H800 GPUs with tensor parallelism set to 8, utilizing a custom inference framework with 8-bit weight-only quantization (W8A16). Other models are tested through their official APIs. Within the maximum length supported by each model, a sufficient number of uniformly distributed points were selected for testing. After removing outliers, the data is fitted with a quadratic function.

data mixture balancing, validated through systematic repetition-aware testing. To fully utilize the architecture’s long-context capability, we introduce in-depth analysis of the hyperparameters and propose a three-stage training procedure, successfully extending the context window to one million tokens. During the alignment phase, we incentivize the model’s various capabilities through precisely tuned reward dimensions and multi-stage training methodology, especially in the areas of long-context and real-world scenarios. Subsequently, we augment our language model with visual capabilities by integrating a lightweight Vision Transformer (ViT) (Dosovitskiy et al., 2021) module, thereby creating our vision-language model, MiniMax-VL-01. MiniMax-VL-01 undergoes additional training with 512 billion vision-language tokens, utilizing a four-stage training process. The final stage of this training is specifically designed to optimize the user experience.

Comprehensive evaluations on core academic benchmarks demonstrate that both models attain performance levels comparable to those of closed-source top-tier models in both text and vision-language tasks, as illustrated in Figure 1 (a,b). For contexts longer than 200k, our model performs significantly better, as shown in Figure 1 (c). In addition to academic benchmarks, we also assess the models’ performance using in-house benchmarks derived from real-world usage and show that our model is top-tier in those scenarios. In addition to its performance, our model exhibits significant advantages in prefilling latency, attributed to its novel architecture, as illustrated in Figure 2.

We summarize our contributions as follows:

1.
We build a model that rivals the top-tier closed-source models on standard academic benchmarks. Furthermore, this model supports context inputs of up to 4 million tokens, showcasing outstanding performance in long-context evaluations.
2.
We demonstrate the first successful large-scale implementation of linear attention. While linear attention has been studied before, it has never been deployed at this scale. We provide comprehensive details on our algorithm design and engineering optimizations.
3.
We outline a practical approach and experimental methodology for the exploration of various models, datasets, evaluations, and algorithms, which may serve as a valuable reference.
4.
We publicly release the weights and offer a cost-effective API, aiming to help others develop models that push beyond current limitations.

In this section, we present the design of our network architecture. To achieve optimal performance within constrained resources and better handle longer sequences, we adopt MoE approach and employ linear attention as much as possible instead of the traditional softmax attention used in standard transformers.

To facilitate a more intuitive understanding, we illustrate the main architecture in Figure 3. Our design follows the Transformer-style block, with each comprises a channel mixer (an attention block) and a feature mixer (an MLP block). We employ two types of channel mixers: lightning attention and softmax attention. The feature mixer is an MoE that incorporates multiple feed-forward networks (FFNs). To ensure load balancing in the MoE blocks, we propose a novel load balancing strategy inspired by GShard (Lepikhin et al., 2021), which we refer to the global router. This strategy is

Figure 3: **The architecture of MiniMax-Text-01.**

designed to maintain training stability. Additionally, DeepNorm (Wang et al., 2024a) is integrated to enhance overall performance.

The final MiniMax-Text-01 architecture integrates both linear attention and softmax attention mechanisms in a structured pattern. Specifically, a transformber block with softmax attention is positioned after every 7 transnormer blocks (Qin et al., 2022a) of linear attention, leading to a total of 80 layers. Each attention module is composed of 64 heads, each with a head dimension of 128. The softmax attention layers employ Group Query Attention (GQA) (Ainslie et al., 2023) with a group size of 8. Rotary Position Embedding (RoPE) (Su et al., 2024) is applied to half of the attention head dimension, with a base frequency set to 10,000. The model’s hidden size is configured to 6144, and each layer incorporates 32 experts with a top-2 routing strategy. The feed-forward network within each expert has a hidden dimension of 9216. In total, MiniMax-Text-01 compromises 456 billion parameters, of which 45.9 billion are activated for each processed token.

In the subsequent sections, we will delve into our considerations regarding the model architecture, i.e., the integration of different attention mechanisms, the synergy between MoE and linear attention, the rationale behind hyperparameter selection, and the methodology for determining the model’s size based on scaling laws.

MoE provides a pathway to enhance both scalability and efficiency compared to the dense version. Typically, MoE is a substitute for the feed forward networks (FFN) in feature-mixer layers (Lepikhin et al., 2021; Fedus et al., 2022), which consists of multiple FFN experts, where each token is routed to one or more of these experts. Specifically, for an input token $\mathbf{x}_{t}$ , its corresponding output hidden state $\mathbf{h}_{t}$ is calculated as:

\mathbf{h}_{t}=\sum_{i=1}^{E}\text{Softmax}_{i}\left(\text{TopK}(\mathbf{x}_{t}\cdot\mathbf{W}_{g})\right)\cdot\text{FFN}_{i}(\mathbf{x}_{t}),

where $E$ represents the total number of experts, $\mathbf{W}_{g}$ is the weight of the gate, $\text{FFN}_{i}$ stands for the $i$ -th expert, and $\text{TopK}(\cdot)$ denotes the operation that preserves the top $k$ scores among all $E$ experts while setting the remaining scores to $-\infty$ .

The training of MoE based LLMs can be categorized into token-drop and dropless. We adopt the token-drop strategy to improve training efficiency. With this approach, each expert is assigned a capacity limit specifying the maximum number of tokens it can handle. Once this capacity is reached, any additional token routed to that expert is discarded.

To assess the effectiveness of the MoE architecture, we conduct a comparative study between a dense model with 7 billion parameters and an MoE model with 2 billion activation parameters out of a total of 24 billion parameters. The results, as illustrated in Figure 4, demonstrate that the MoE model significantly outperforms the dense model under the same computational budget on various benchmarks, including HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), Natural Questions(Kwiatkowski et al., 2019), PIQA(Bisk et al., 2020) and TriviaQA(Joshi et al., 2017).

Figure 4: Isoflop Comparison: MoE vs. Dense on various benchmarks. Both models are trained on 1 trillion tokens. The gray dashed lines indicate the difference in the computation required for the two models to achieve the same performance.

When scaling up to larger models, we encounter the challenge of routing collapse, which arises due to the concentrated distribution of tokens designated for allocation. To mitigate this issue, we incorporate a simple global routing strategy to the GShard (Lepikhin et al., 2021) auxiliary loss for better load balancing.

Auxiliary Loss. To ensure differentiability, the auxiliary loss is defined as $L_{\text{aux}}=\alpha_{\text{aux}}\cdot\frac{1}{E}\sum_{i=1}^{E}f_{i}\cdot m_{i}$ , where $\alpha_{\text{aux}}$ represents the coefficient of the auxiliary loss, $f_{i}$ denotes the fraction of tokens assigned to the $i$ -th expert, and $m_{i}$ is the average routing probability of expert $i$ .

Global Router. The GPU memory size constrains the micro batch size in LLM training, leading to substantial fluctuations in the token distribution within individual Expert Parallel (EP) groups. Moreover, token distributions vary across different EP groups, potentially resulting in load imbalances where experts in one EP group may be overloaded while those in another are underutilized. To address this, we implement a global token dispatching strategy across EP groups. Specifically, we introduce an additional allgather communication step to synchronize the number of tokens awaiting processing by each expert before dispatching tokens across different EP groups. Under the same capacity constraints, this global routing mechanism can effectively reduce the overall token drop rate, thereby ensuring training stability.

Figure 5: Illustration of the computations for **softmax attention** (left) and **linear attention** (right). The input length is $N$ and feature dimension is $d$ , with $d\ll N$ . Tensors in the same box are associated with computation. The linearized formulation allows $O(N)$ time and space complexity.

Linear attention utilizes the “right product kernel trick” to transform quadratic computational complexity into linear complexity, as illustrated in Figure 5. By taking TransNormer (Qin et al., 2022a) as an example, the NormAttention mechanism can be written as:

\mathbf{O}=\mathrm{Norm}((\mathbf{Q}\mathbf{K}^{\top})\mathbf{V}),

where $\mathbf{Q}$ , $\mathbf{K}$ , and $\mathbf{V}\in\mathbb{R}^{n\times d}$ are the query, key, and value matrices, respectively, with $n$ for sequence length and $d$ for feature dimension. The equation can be transformed into its linear variant using right matrix multiplication:

\mathbf{O}=\mathrm{Norm}(\mathbf{Q}(\mathbf{K}^{\top}\mathbf{V})),

The linear formulation facilitates efficient recurrent prediction with a training complexity of $O(nd^{2})$ . Furthermore, linear attention ensures a constant computational complexity of $O(d^{2})$ , irrespective of the sequence length. This is accomplished by recurrently updating the term $\mathbf{K}^{\top}\mathbf{V}$ , thereby obviating the need for repetitive computation of the entire attention matrix. In contrast, softmax attention incurs a complexity of $O(nd^{2})$ during inference.

When addressing causal language modeling tasks, the efficacy of the right product is compromised, necessitating the computation of cumsum (Hua et al., 2022). This limitation impedes the realization of highly efficient parallel computation, which likely explains why, despite being proposed by Brébisson et al. (de Brébisson and Vincent, 2016) nine years ago, none of the current leading open-source LLMs—including LLaMA3 (Dubey et al., 2024), Qwen2.5 (Yang et al., 2024), DeepSeekV3 (DeepSeek-AI, 2024), and Mistral (Jiang et al., 2023)—have adopted this linear attention mechanism.

Lightning attention (Qin et al., 2024c; Qin et al., 2024b) represents an I/O-aware, optimized implementation of TransNormer (Qin et al., 2022a). This approach identifies the primary bottleneck in the computational efficiency of existing linear attention mechanisms: the slow cumsum operation inherent in causal language modeling. To alleviate this problem, Lightning Attention proposes a novel tiling technique that effectively circumvents the cumsum operation. The key innovation lies in the strategic division of the attention calculation into two distinct components: intra-block and inter-block computations. The left product attention calculation is employed for intra-block operations, while the right product is utilized for inter-block operations. This division is crucial because the intra-blocks can be significantly reduced in size, thereby ensuring that the overall computational complexity remains linear.

Note that the lightning attention was originally proposed by our team members in Qin et al. (2024c), we recall some of the core processes to elucidate why it can achieve theoretical linear complexity in practice for the sake of completeness. In the interest of analytical tractability, we deliberately omit the consideration of normalization, sigmoid linear unit (SiLU) activation, and gating mechanisms in the following derivation.

Let us start with the forward pass in lightning attention. The left product in causal attention calculation is defined as:

\mathbf{O}=[(\mathbf{Q}\mathbf{K}^{\top})\odot\mathbf{M}]\mathbf{V}

where $\mathbf{M}_{ts}=1$ if $t\geq s$ , otherwise 0. The right product operation can be computed in a recursive formula as:

\mathbf{kv}_{0}=\mathbf{0},\mathbf{kv}_{t}=\mathbf{kv}_{t-1}+\mathbf{k}_{t}\mathbf{v}_{t}^{\top},\mathbf{o}_{t}^{\top}=\mathbf{q}_{t}^{\top}\mathbf{kv_{t}}.

It is important to note that while Eq. 5 exhibits linear computational complexity, it is inherently unparallelizable.

The fundamental concept underlying the implementation of lightning attention involves the utilization of a tiling technique to compute attention scores. Specifically, the matrices $\mathbf{Q},\mathbf{K},\mathbf{V}$ are partitioned into two distinct blocks along the row dimension:

\displaystyle\mathbf{X}=\left[\begin{matrix}\mathbf{X}_{1}\\ \mathbf{X}_{2}\end{matrix}\right],\mathbf{X}_{1}\in\mathbb{R}^{m\times d},\mathbf{X}_{2}\in\mathbb{R}^{(n-m)\times d},\mathbf{X}\in\{\mathbf{Q},\mathbf{K},\mathbf{V}\}.

By unfolding Eq. 4, we obtain the following expression (noting that $\mathbf{kv}_{0}=0$ ):

\mathbf{kv}_{s}=\mathbf{kv}_{0}+\sum_{j=1}^{s}\mathbf{k}_{j}\mathbf{v}_{j}^{\top},s=1,\ldots,m.\quad\mathbf{o}_{s}^{\top}=\mathbf{q}_{s}^{\top}\mathbf{kv}_{s}=\mathbf{q}_{s}^{\top}\mathbf{kv}_{0}+\mathbf{q}_{s}^{\top}\sum_{j=1}^{s}\mathbf{k}_{j}\mathbf{v}_{j}^{\top}.

Rewrite it in block form, we have:

\mathbf{O}_{1}=\mathbf{Q}_{1}\mathbf{kv}_{0}+[(\mathbf{Q}_{1}\mathbf{K}_{1}^{\top})\odot\mathbf{M}]\mathbf{V}_{1}\triangleq\mathbf{Q}_{1}\mathbf{KV}_{0}+[(\mathbf{Q}_{1}\mathbf{K}_{1}^{\top})\odot\mathbf{M}]\mathbf{V}_{1}.

As shown, the intra-block $[(\mathbf{Q}_{1}\mathbf{K}_{1}^{\top})\odot\mathbf{M}]\mathbf{V}_{1}$ can use the left product and the inter-block $\mathbf{Q}_{1}\mathbf{KV}_{0}$ can use the right product. Note that the intra-block can be further divided using the same strategy:

\begin{aligned}\displaystyle\mathbf{kv}_{m+t} & \displaystyle=\mathbf{kv}_{m}+\sum_{j=m+1}^{m+t}\mathbf{k}_{j}\mathbf{v}_{j}^{\top},t=1,\ldots,n-m,\ \mathbf{o}_{m+t}^{\top}=\mathbf{q}_{m+t}^{\top}\mathbf{kv}_{m+t}, \\ \displaystyle\mathbf{O}_{2} & \displaystyle=\mathbf{Q}_{2}\mathbf{kv}_{m}+[(\mathbf{Q}_{2}\mathbf{K}_{2}^{\top})\odot\mathbf{M}]\mathbf{V}_{2}\triangleq\mathbf{Q}_{2}\mathbf{KV}_{1}+[(\mathbf{Q}_{2}\mathbf{K}_{2}^{\top})\odot\mathbf{M}]\mathbf{V}_{2}.\end{aligned}

To compute the second block, we use $\mathbf{KV}_{1}=\mathbf{kv}_{m}$ , which can be computed by:

\mathbf{KV}_{1}=\mathbf{KV}_{0}+\sum_{j=1}^{m}\mathbf{k}_{m}\mathbf{v}_{m}^{\top}=\mathbf{KV}_{0}+\mathbf{K}_{1}^{\top}\mathbf{V}_{1}.

where $\mathbf{KV}_{0}=\mathbf{kv}_{0}$ . By recursively applying the aforementioned strategy of partitioning the matrix into multiple blocks, the practical computational complexity can be reduced to linear. The final time complexity of lightning attention is $O(nd^{2}+nBd)$ , where $B$ is the block size. Algorithm 1 illustrates the IO-aware implementation of lightning attention forward pass.

Algorithm 1 Lightning Attention Forward Pass

Input:

\mathbf{Q},\mathbf{K},\mathbf{V}\in\mathbb{R}^{n\times d}

, block sizes

B

Divide

\mathbf{X}

into

T=\frac{n}{B}

blocks

\mathbf{X}_{1},\mathbf{X}_{2},...\mathbf{X}_{T}

of size

B\times d

each, where

\mathbf{X}\in\{\mathbf{Q},\mathbf{K},\mathbf{V},\mathbf{O}\}

Initialize mask

\mathbf{M}\in\mathbb{R}^{B\times B}

, where

\mathbf{M}_{ts}=1

, if

t\geq s

, else 0.

Initialize

\mathbf{KV}=0\in\mathbb{R}^{d\times d}

for

t=1,\ldots,T

Load

\mathbf{Q}_{t},\mathbf{K}_{t},\mathbf{V}_{t}\in\mathbb{R}^{B\times d}

from HBM to on-chip SRAM.

On chip, compute

\mathbf{O}_{\mathrm{intra}}=[(\mathbf{Q}_{t}\mathbf{K}_{t}^{\top})\odot\mathbf{M}]\mathbf{V}_{t}

On chip, compute

\mathbf{O}_{\mathrm{inter}}=\mathbf{Q}_{t}(\mathbf{KV})

On chip, compute

\mathbf{KV}=\mathbf{KV}+\mathbf{K}_{t}^{\top}\mathbf{V}_{t}

Write

\mathbf{O}_{t}=\mathbf{O}_{\mathrm{intra}}+\mathbf{O}_{\mathrm{inter}}

to HBM as the

t

-th block of

\mathbf{O}

end for

Return

\mathbf{O}

Although lightning attention demonstrates promise and competitive performance in small-scale experiments, its scaling behavior and capability in the downstream tasks under large-scale settings remain unexplored. To mitigate the gap, we conduct a series of scaling experiments to evaluate the scalability of the lightning attention mechanism in comparison to softmax attention, meanwhile verifying the performance on the extensive downstream tasks. It is noteworthy that during our experiments, we observed that lightning attention demonstrates limited retrieval capabilities. This finding inspired us to explore a hybrid approach (Hybrid-lightning) that takes the advantages of both lightning and softmax attention to enhance retrieval performance by substituting lightning attention with softmax attention at intervals of every eight layers.

We adhere to the FLOPs calculation methodology established by Kaplan et al. (2020). For the purpose of our analysis, we define the following variables: $l$ (number of layers), $d$ (model dimension), $h$ (number of attention heads), $b$ (batch size) and $n$ (sequence length). The checklist of model parameters and FLOPs is presented in Table 1.

Original Table 1 — Table 1: **Model Parameters and FLOPs Comparisons Across Architectures.** For scaling law calculations, embedding parameters and other subleading terms are excluded to improve alignment with fitted results.

Table 1
Architecture	Parameter count	FLOPs count
Softmax Attention	$12ld^{2}$	$72bnld^{2}(1+\frac{n}{6d}+\frac{5}{18d})$
Lightning Attention	$12ld^{2}+2ld^{2}/h$	$72bnld^{2}(1+\frac{1}{2h}+\frac{5}{18d})$
Hybrid-lightning	$12ld^{2}+7ld^{2}/4h$	$72bnld^{2}(1+\frac{n}{48d}+\frac{7}{16h}+\frac{5}{18d})$

We conducted training on softmax (equipped with FlashAttention-2 (Dao, 2024)), lightning attention, and hybrid-lightning attention models across various scales: 70 million, 160 million, 410 million, 1 billion, 3 billion, and 7 billion parameters. Each model was trained on a dataset consisting of up to 300 billion tokens, with a context length of 8192. Our training methodology follows the approach proposed by Chinchilla (Hoffmann et al., 2022), where the training loss serves as a direct indicator of test performance. For each model architecture and training sequence length, we maintained a uniform global batch size of 4 million tokens. The Adam optimizer was employed, configured with a learning rate of 3e-4 and a weight decay of 0.1. A fixed learning rate scheduler was applied across all experiments due to constrained computational resources.

We employ a diverse set of evaluation benchmarks, including BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC (both easy and challenge variants) (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), Needle in A Haystack (NIAH) (Shen et al., 2024), and SCROLLS (Shaham et al., 2022). Each benchmark assesses distinct capabilities of the models.

Original Table 2 — Table 2: **Summary of Scaling Laws:** It shows the relationships between loss ( $L$ ), optimal model size ( $N_{opt}$ ), and optimal dataset size ( $D_{opt}$ ) as functions of computational budget ( $C$ ). It reveals that, given the same budget, the hybrid model uses more parameters and tokens but achieves lower loss.

Table 2
Arch	$L(C)$	$N_{opt}(C)$	$D_{opt}(C)$
Softmax Attention	$3.7087C^{-0.0798}$	$(1.82\times 10^{8})C^{0.7118}$	$(2.56\times 10^{10})C^{0.5102}$
Lightning Attention	$3.5391C^{-0.0768}$	$(2.74\times 10^{8})C^{0.6470}$	$(4.43\times 10^{10})C^{0.4684}$
Hybrid-lightning	$3.4797C^{-0.0763}$	$(2.57\times 10^{8})C^{0.6670}$	$(3.70\times 10^{10})C^{0.4707}$

Figure 6: Summary of Scaling Laws. Training curves (left) span models from 70M to 7B parameters. Optimal model size (center) and training tokens (right) are derived based on a specified compute budget estimation.

We fit the scaling curves based on our experiments over the above mentioned settings, where we alter the model size ( $N$ ) and dataset size ( $D$ ) for different computational budget ( $C$ ) and observe the corresponding training loss ( $L$ ) that serving as an estimator of test loss. We begin by establishing power-law relationships between $L$ and $C$ , following Chinchilla’s methodology (Hoffmann et al., 2022). Using the fitted curve, we derive coefficients for optimal model size $N_{opt}\propto C^{a}$ and optimal dataset size $D_{opt}\propto C^{b}$ . The original scaling laws (Kaplan et al., 2020) use $L(X)=(X_{0}/X)^{\alpha_{X}}$ , while subsequent studies (Henighan et al., 2020; Clark et al., 2022; Hoffmann et al., 2022; Gao et al., 2024) employ $L(X)=\epsilon+(X_{0}/X)^{\alpha_{X}}$ for better fitting, where $\epsilon$ denotes the irreducible loss. For simplicity, we unify these forms into $L(X)=\beta_{X}X^{\alpha_{X}}$ , facilitating a direct comparison of scaling capabilities based on $\alpha_{X}$ and $\beta_{X}$ . The summary of scaling laws is shown in Table 2 and Figure 6. It can be intuitively understood that given the same computational budget, models with lightning attention tend to utilize more parameters and tokens, yet they achieve a lower loss compared to models with pure softmax attention.

Figure 7: Larger models and hybrid-lightning attention achieve the best performance across benchmarks. Performance is evaluated on CSR (Common Sense Reasoning), NIAH (Needle in a Haystack), and SCROLLS benchmarks using three attention mechanism models from 410M to 7B parameters.

We present the benchmark results of downstream tasks in Figure 7. Lightning attention demonstrates comparable performance across most downstream tasks, with the exception of NIAH. This indicates that linear attention exhibits similar language modeling capabilities to Transformer models but falls short in retrieval tasks, rendering it unsuitable for LLMs. However, the hybrid-lightning attention not only matches but surpasses the retrieval and extrapolation capabilities of softmax attention, making it

Figure 8: The training speed of various attention mechanisms, including softmax, lightning, hybrid-lightning, HGRN2, and Mamba2, was benchmarked across sequence lengths ranging from 1,024 to 65,536. Performance was measured in terms of training speed, reported as tokens processed per GPU per second (TGS).

well-suited for in-context learning in LLMs.

We assess the end-to-end training speed of softmax attention, lightning attention, and hybrid-lightning models with 3 billion parameters by measuring the tokens processed per GPU per second (TGS). For completeness, we also included popular linear models such as HGRN2 and Mamba2 in our evaluation. For the speed benchmark, the training context length was gradually increased until reaching the out-of-memory limit on a single-node H800 GPUs. As illustrated in Fig. 8, lightning attention achieves a constant training speed irrespective of the sequence length and is the sole linear model that outperforms FlashAttention2.

Our preliminary experiments with the hybrid architecture have yielded promising results, motivating us to delve deeper into its potential through two variants: hybrid-cosformer2 and hybrid-hgrn2. In the hybrid-cosformer2 model, we replace the linear attention layers in the cosformer2 architecture with softmax attention layers at intervals of every eight layers. This substitution strategy is similarly applied in the hybrid-hgrn2 model. We conduct experiments using consistent setups to evaluate the downstream performance of these alternatives. Our findings, as summarized in Table 3, indicate that the hybrid-lightning model achieves the best performance.

Original Table 3 — Table 3: **Benchmarking various hybrid-linear models with 1 Billion Parameters.** We present the average CSR score, weighted average accuracy for NIAH, and the average SCROLLS score. Higher scores indicate better performance across all tasks. Abbreviations: TGS (token per gpu per second), HS (HellaSwag), WG (WinoGrande), OBQA (OpenBookQA), NIAH, and SCR (SCROLLS).

Table 3
Hybrid-linear Arch.	TGS $\uparrow$	PIQA $\uparrow$	HS $\uparrow$	WG $\uparrow$	ARC-E $\uparrow$	ARC-C $\uparrow$	OBQA $\uparrow$	CSR $\uparrow$	NIAH $\uparrow$	SCR $\uparrow$
Hybrid-cosformer2	23.3K	70.29	45.63	51.46	55.77	26.11	30.60	46.64	43.6	10.9
Hybrid-hgrn2	29.5K	70.89	51.23	56.51	59.68	28.50	32.40	49.87	91.8	10.8
Hybrid-lightning	33.4K	70.73	50.41	55.80	59.93	27.65	32.80	49.55	95.7	13.3

In addition to linear models, sliding window attention can also achieve linear computational complexity by appropriately adjusting the window size. As it is grounded in softmax attention, it serves as a robust baseline for evaluating linear architectures. Therefore, we incorporated the hybrid-window approach by replacing the sliding window attention with full softmax attention every eight layers. We evaluated various window sizes of SWA ranging from 256 to 1024. Our results indicate that larger window sizes lead to slower training speeds compared to the hybrid-lightning model. To compare these models under equivalent speed conditions, we did not consider window sizes larger than 1024. As shown in Table 4, the hybrid-lightning model outperforms all other models across all metrics, particularly excelling in the NIAH benchmark.

Original Table 4 — Table 4: **Benchmark comparison of hybrid-lightning and hybrid-window Models.** Metrics include average CSR score, weighted NIAH accuracy, and average SCROLLS score. Higher scores indicate better performance across all tasks. Abbreviations: PS (parameter size, billion), W.S. (window size of SWA), HS (HellaSwag), WG (WinoGrande), OBQA (OpenBookQA), NIAH, SCR (SCROLLS), TGS (token per gpu per second).

Table 4
P.S	Arch.	W.S.	TGS $\uparrow$	PIQA $\uparrow$	HS $\uparrow$	WG $\uparrow$	ARC-E $\uparrow$	ARC-C $\uparrow$	OBQA $\uparrow$	CSR $\uparrow$	NIAH $\uparrow$	SCR $\uparrow$
1B	Hybrid- window	256	35.6K	70.29	48.68	53.35	57.95	28.75	32.60	48.61	46.8	10.6
512	35.1K	70.95	48.19	52.33	57.53	27.22	30.00	47.70	25.7	11.9
1024	33.6K	69.75	47.80	53.12	57.53	28.33	31.60	48.02	53.9	10.6
Hybrid-lightning	33.4K	70.73	50.41	55.80	59.93	27.65	32.80	49.55	95.7	13.3
3B	Hybrid- window	256	16.1K	73.83	59.70	59.59	64.10	33.62	35.00	54.31	40.9	14.2
512	15.8K	73.29	60.00	59.04	62.96	32.51	36.00	53.97	57.9	14.2
1024	15.4K	74.27	59.02	57.85	64.56	31.91	33.00	53.44	41.6	13.3
Hybrid-lightning	15.1K	74.21	61.06	59.51	65.49	34.90	35.80	55.16	98.0	14.7

Based on our analysis of scaling law experiment, downstream performance and speed comparison, we conclude that while pure linear attention models are computationally efficient, they are not suitable for LLMs. This is due to their inherent inability to perform retrieval, a capability that is essential for in-context learning. In contrast, our hybrid model not only matches but also surpasses softmax attention in both retrieval and extrapolation tasks. This outcome is somewhat counterintuitive. To understand this phenomenon, consider the following explanation of softmax attention:

\mathbf{O}=\mathrm{Softmax}(\mathbf{Q}\mathbf{K}^{\top}/\sqrt{d})\mathbf{V}.

It can be rewritten into a linear recurrent form as:

s_{t}^{0}=0,\quad s_{t}^{j}=s_{t}^{j-1}+\exp(\mathbf{q}_{t}\mathbf{k}_{j}^{T}/\sqrt{d}),\quad\mathbf{o}_{t}^{j}=(s_{t}^{j-1}/s_{t}^{j})\mathbf{o}_{t}^{j-1}+(1-s_{t}^{j-1}/s_{t}^{j})\mathbf{v}_{j},\quad\mathbf{o}_{t}=\mathbf{o}_{t}^{t},j=1,\ldots,t.

Note that the linear recurrence form of lightning attention is as follows:

\mathbf{kv}_{0}=0,\quad\mathbf{kv}_{j}=\mathbf{kv}_{j-1}+\mathbf{k}_{j}\mathbf{v}_{j}^{\top}\quad\mathbf{o}_{j}=\mathbf{kv}_{j}^{\top}\mathbf{q}_{j},j=1,\ldots,t.

The softmax attention mechanism can be interpreted as a linear RNN (Qin et al., 2024a). At each time step $t$ , the hidden state is recalculated starting from the initial time $t_{0}=1$ , a process often described as "Going Through a Book." This method enables the model to accurately retain input information by systematically revisiting previous data. In contrast, linear models lack this recomputation process, which hinders their ability to effectively retain input data.

Let us define the capacity of an RNN as the size of its recurrent state. Upon closer examination of Eq. 11, we can deduce that the capacity of softmax attention is $O(d)$ . In contrast, as illustrated in Eq. 12, the capacity of lightning attention is $O(d^{2}/h)$ . Given that $d>h$ , it follows that lightning attention possesses a larger capacity than softmax attention. Consequently, the hybrid-lightning model exhibits superior retrieval and extrapolation capabilities compared to models relying solely on softmax attention.

Based on the conclusions from previous sections, we conduct two additional sets of ablation experiments to validate module choices within the MoE architecture on a larger scale: (1) Hybrid-lightning attention versus softmax attention: To verify the advantages of the hybrid lightning attention in the MoE. (2) Pre-Layer Normalization versus Post-Layer Normalization: In our hybrid architecture, the effective depth of the model plays a significant role. Thus, we expect to find a better normalization algorithm for the deep model.

Hybrid-lightning Attention versus Softmax Attention. We perform a small-scale comparative analysis between softmax attention and hybrid-lightning attention within the MoE architecture. Specifically, we use a 28 billion parameter MoE with 5 billion activation parameters that utilize softmax attention as the base model. For every 8 consecutive layers in the base model, we systematically replace softmax attention with lightning attention in the first 7 layers. Both the base model and the modified model are trained on 1 trillion tokens. As shown in Table 5, the results reveal that substituting certain softmax attention layers with lightning attention improves accuracy across most benchmarks.

Pre Layer Normalization versus Post Layer Normalization. Pre Layer Normalization(Baevski and Auli, 2018; Child et al., 2019; Wang et al., 2019) (PreNorm), which applies normalization layers before residual connections and attention mechanisms, has demonstrated enhanced stability and performance in LLMs. Since PreNorm allows gradients to flow more directly from the output to the input through residual connections, bypassing the sub-layers to a certain extent, it reduces the effective depth of the model. In contrast, Post Layer Normalization(Wang et al., 2019) (PostNorm) applies normalization after the residual connection and attention mechanisms, thereby preserving the model’s effective depth. However, PostNorm can be prone to vanishing and exploding gradients, presenting significant challenges in training LLMs. Most existing LLMs predominantly use PreNorm, as the performance differences between wider and deeper networks in the conventional Transformer architecture are often negligible, and training stability is prioritized.

The experiments are performed on models with 9.3 billion activation parameters and a total of 60 billion parameters, each consisting of 48 layers that employ different normalization methods. Both models are trained on 500 billion tokens. For PostNorm, we utilize DeepNorm (Wang et al., 2024a) to ensure more stable training. As illustrated in Table 5, PostNorm consistently outperforms PreNorm across all evaluated metrics.

Original Table 5 — Table 5: **Module Ablations.** Abbreviations: BBH (BIG-Bench Hard), DROP (Discrete Reasoning Over Paragraphs), MMLU (Massive Multitask Language Understanding), CMMLU (Massive Multitask Language Understanding in Chinese), GSM8k (Grade School Math 8K), ARC-C (Arc-Challenge), WG (WinoGrande).

Table 5
Arch.	BBH $\uparrow$	DROP $\uparrow$	MMLU $\uparrow$	CMMLU $\uparrow$	MATH $\uparrow$	GSM8k $\uparrow$	ARC-C $\uparrow$	WG $\uparrow$
Softmax	28.2	27.4	49.3	47.3	4.6	18.8	46.4	65.6
Hybrid-lightning	32.2	29.0	49.5	46.0	6.8	18.5	47.4	67.5
Pre Layer Norm.	29.9	26.8	43.9	41.8	4.8	12.2	43.5	65.5
Post Layer Norm.	32.6	27.6	50.2	49.2	5.7	16.8	46.2	65.4

Upon finalizing the architecture of the model’s modules, the subsequent step entails scaling up the model, which necessitates a meticulous design of the model’s hyperparameters across various dimensions. Our primary goal is to strike a balance between performance and inference efficiency. Single-device inference offers superior efficiency compared to multi-device implementations by eliminating cross-machine communication overhead. Consequently, we constrain the model’s total parameters to 500B, ensuring compatibility with single-node inference on an $8\times 80$ G configuration for sequences up to 1M tokens under 8-bit quantization. Given our limited training budget, we formulate the following optimization problem to determine optimal parameter allocations:

\min_{P_{\text{all}},P_{\text{act}}}L(P_{\text{all}},P_{\text{act}},T)\quad\text{subject to}\quad C_{\text{compute}}(P_{\text{all}},P_{\text{act}},T)<C\quad\text{and}\quad P_{\text{all}}<500B,

where $L$ denotes the loss, $P_{\text{all}}$ and $P_{\text{act}}$ represent the total and activation parameter counts respectively, $T$ is the number of training tokens, $C_{\text{compute}}$ denotes the computational costs (dependent on parameter counts and data consumption), and $C$ signifies the budget constraint.

Through comparative experiments on small-scale models, we first establish optimal ranges for several key variables: (1) the mixing ratio between softmax and linear attention mechanisms; (2) the depth-to-width ratio of the model architecture; (3) the ratio of linear attention memory size to hidden size; (4) the ratio of activated FFN to attention; (5) the proportion of dimensions utilizing RoPE for softmax attention.

Our experiments reveal that the hybrid architecture demonstrates particular sensitivity to layer depth, with deeper models consistently outperforming shallower counterparts. Notably, shallow models require substantially more softmax attention layers to achieve comparable performance, underlining the efficiency advantages of deeper architectures. We also observe that increasing linear attention memory size significantly enhances model performance, and implementing RoPE on half of the softmax attention dimensions enables length extrapolation without performance degradation.

Based on these optimized architectural variables, we employ established scaling laws (Hoffmann et al., 2022; Clark et al., 2022) to determine the optimal model size. We train models with activation parameters ranging from 44 million to 1.2 billion across 500 billion tokens, utilizing 16, 32, and 64 experts. However, we find the predictions from these methods become less reliable when extrapolating to a larger model with 9.3 billion parameters. To address this limitation and achieve more accurate predictions, we propose the following formula:

L(P_{\text{act}},T|E)=d+aP_{\text{act}}^{\alpha}+bT^{\beta}+c(P_{\text{act}}T)^{\gamma},

where $L(P_{\text{act}},T|E)$ represents the loss conditioned on the number of experts, while $a$ , $b$ , $c$ , $d$ , $\alpha$ , $\beta$ , and $\gamma$ are parameters to be fitted in relation to the number of experts. Based on the predictions of Eq. 13 and Eq. 14, we have identified a candidate model with 45.9 billion activation parameters and 456 billion total parameters as the optimal configuration.

In this section, we present our computation part, including the training and inference. In this project, we have a dynamically changing GPU cluster, where the number of H800 GPUs ranges from 1500 to 2500. An efficient architecture necessitates robust implementation optimization to fully harness its computational benefits at scale. To scale our novel architecture to the requisite size, we present three key optimization strategies that primarily address the following three challenges:

1.
Mitigating the all-to-all (a2a) communication overhead during the training of a Mixture of Experts (MoE) architecture is a persistent challenge. The configuration we choose for our experts, specifically opting for large models, imposes substantial demands on GPU memory. Therefore, the primary challenge lies in achieving an optimal equilibrium between memory utilization, computational efficiency, and the overhead associated with all-to-all communication.
2.
As we endeavor to support at least 1 million token context window in both training and inference, the accurate distribution of tokens within such an extensive context window across different GPUs becomes imperative for this colossal model. This necessity, however, inevitably introduces additional communication overhead. As a result, devising strategies to minimize this overhead, particularly in the context of our hybrid architecture, presents a significant challenge.
3.
The current implementation of the lightning attention mechanism is specifically optimized for training processes. However, in the inference scenario, the challenge arises in effectively managing real-world batched inputs, which may encompass variable sequence lengths and specific inputs that incorporate prefix caching.

It is noteworthy that the existing open-source frameworks in the industry currently lack the necessary mature technical support to adequately address these challenges. Thus, we independently and comprehensively reinvent our distributed training and inference framework, thereby successfully addressing these challenges with the desired level of efficiency.

The primary objective in optimizing the MoE architecture is to minimize communication overhead, particularly for MoE models that utilize all-to-all (a2a) communication. To address this, We implement a token-grouping-based overlap scheme, as illustrated in Figure 9. In this scheme, the a2a communication is performed within the expert parallel (EP) communication group, and it overlaps with the processing of tokens from different expert groups. To ensure the correctness of the communication results, we restrict each ProcessGroup to execute communication operators sequentially. As a result, a2a communications across different groups cannot overlap, leading to the emergence of idle time.

Figure 9: **Expert Parallel (EP) Overlap Illustration. Chunk tokens into 2 groups thus computation can overlap with communication between different groups.**

This approach leads to significant performance improvements. However, upon more detailed analysis, we identified a critical trade-off specific to the expert configuration of the MiniMax-Text-01 model. When Tensor Parallelism (TP) is employed to partition the expert parameters, the computational intensity becomes excessively low, thereby hindering the efficiency of the computation. However, opting not to use TP leads to an excessively large parameter count, which necessitates the activation of a larger Pipeline Parallelism (PP) configuration. The challenge emerges because PP does not reduce the memory footprint required for storing activations. This limitation is particularly detrimental for training models with long contexts, as the increase in memory consumption does not provide proportional benefits in terms of computational efficiency or training speed. Consequently, it is imperative to develop a new parameter partitioning strategy that adeptly balances memory usage and computational intensity to optimize the training process for our specific model and task.

To achieve enhanced efficiency, we first introduce a novel ProcessGroup, termed ETP (Expert Tensor Parallel), which is specifically designed to manage the weight partitioning of experts. Concurrently, we propose another distinct ProcessGroup, named EDP (Expert Data Parallel), to encapsulate the data parallelism of identical experts. In our system, we define the total number of GPUs involved in training as $world\_size$ . The system must satisfy two key conditions:

world\_size=size_{PP}\times size_{DP}\times size_{CP}\times size_{TP}

and

world\_size=size_{PP}\times size_{EDP}\times size_{ETP}\times size_{EP}

This configuration empowers the MoE component with the flexibility to define the distribution of experts, manage the weight partitioning of experts, and independently configure the ZeRO (Zero Redundancy Optimizer) algorithm (Rajbhandari et al., 2020). Based on this implementation, we are able to completely decouple the parallel strategies of the MoE components from those of the non-MoE components.

Building upon this modification, we can flexibly configure the ETP to achieve an optimal balance between memory usage and computational intensity. Furthermore, to mitigate communication overhead, we design an EP-ETP overlap strategy. This strategy aims to maximize the utilization of both network resources and computational resources, as illustrated in Figure 10 (a).

Since communications within the same process group must be executed sequentially, extended periods of computation not only facilitate overlap with a greater number of communications but also create additional opportunities for communications across different process groups to overlap, leading to enhanced overall performance as illustrated in Figure 10 (b).

Figure 10: EP-ETP Overlap Illustration. (a) EP-ETP overlap with the lower computation portion. (b) EP-ETP overlap with the higher computation portion. (c) EP-ETP overlap with fewer groups. Compared with (a) and (b), it shows that if the compute time cost is longer, the efficiency will be better. Comparing with (b) and (c), it shows that fewer groups will lead to insufficient overlap.

When determining the number of groups, several trade-offs must be considered. Theoretically, only by dividing the workload into a sufficiently large number of groups can we achieve ample overlap between communication and computation, as illustrated in Figure 10 (c). However, in practice, an excessive number of groups can significantly increase the complexity of scheduling and introduce the risk of becoming CPU-bound. Given that the proportion of ETP (Expert Tensor Parallel) in the overall MoE (Mixture of Experts) architecture is not substantial, it is crucial to make adjustments based on the specific context and requirements.

Through the aforementioned optimization strategies, we achieve a balanced configuration of storage and computational intensity for the specific expert specifications in the MoE (Mixture of Experts) structure of the MiniMax-Text-01 model. Furthermore, based on these optimizations, we reduce the pure communication overhead of the MoE component by 50% compared to the pre-optimization state, resulting in a significant improvement in training efficiency.

A significant challenge in long context training is that real training samples are difficult to standardize into a uniform length. The conventional approach of using padding to make samples the same length leads to substantial computational waste. In the context of training at the 1M sequence length scale, this waste becomes particularly significant. To address this issue, we adopt a data formatting technique during training where different samples are concatenated end-to-end along the sequence dimension. We refer to this technique as "data-packing". This format minimizes computational waste during the computation process, thereby conserving computational resources.

For Softmax Attention, the ring attention algorithm (Liu et al., 2024a) offers an effective method to partition data, thereby enabling unlimited scalability. However, the existing implementations are not optimized to efficiently handle the ring attention mechanism for the data-packing format. In the case of FlashAttention (Dao, 2024), while it provides a varlen (variable length) interface to accommodate the data-packing format, there is no corresponding ring attention implementation available. Regarding TransformerEngine (NVIDIA, 2023), the implementation incorporates a Context Parallel (CP) ProcessGroup to support the ring attention algorithm. However, this approach poses a risk of computational resource waste when dealing with the data-packing format. This is because the algorithm divides each sequence into $2\times size_{CP}$ segments and applies the ring attention mechanism to each segment. Consequently, this approach restricts each sequence to a length that must be an integer multiple of $2\times size_{CP}$ . In scenarios where the sample distribution is unknown and the CP size is set to a large value, this can lead to significant padding, resulting in the waste of computational resources.

Motivated by the principle of not making assumptions about the sample distribution, we redesign the algorithm and name it Varlen Ring Attention. This approach avoids the excessive padding and subsequent computational waste associated with traditional methods by applying the ring attention algorithm directly to the entire sequence after data-packing. Specifically, the implementation involves distinguishing the offset of the attention mask corresponding to each sequence within the ring attention computation. The key modification is to transform the original causal computations into varlen causal computations and similarly convert the non-causal computations into varlen non-causal computations, shown in Figure 11.

Figure 11: **Ring Attention v.s. Varlen Ring Attention. (a) No data packing in ring attention. (b) Pack 3 samples with different lengths in varlen ring attention.**

For lightning attention, the LASP (Linear Attention Sequence Parallelism) algorithm (Sun et al., 2024) leverages the communication group of CP to facilitate the expansion of long sequences. As illustrated in Figure 12 (a), the LASP algorithm mandates that all CP ranks engage in send-recv operations to exchange intermediate key-value ( $KV$ ) block results. This requirement imposes a sequential dependency among the CP ranks, thereby compelling the computation to be performed in a serial manner. Consequently, this sequential dependency significantly impedes the overall efficiency of the training process, as the inherent parallelism of the system is not fully exploited.

Figure 12: **Difference of LASP Algorithm and LASP+ Algorithm.** (a) LASP Algorithm. 1. Initialization Phase: initializing KV to zero and the diagonal decay matrix. 2. Data Partitioning and Padding: partitioning the Q, K, and V matrices along the sequence dimension into CP size (4 segments illustrated in the figure) blocks, dividing each block into smaller blocks based on the BlockSize $B$ and padding the remaining part (e.g. Q7, K7, V7) that cannot be divided evenly by $B$ . 3. Intra-block Computation: performing intra-block of each CP rank computations in parallel. 4. Inter-block Computation and Communication: starting from CP rank 0, computing the inter-block portion of the current $Q_{i}$ with all previous KV blocks and the prefix sum $K_{i}V_{i}$ . Different CP ranks communicate data through send-recv operations. (b) LASP+ Algorithm. Building upon figure (a), each CP rank computes the local prefix sum $KV_{L}$ and performs AllGather operation to synchronize, then selects the local prefix sum $KV_{L}$ to compute the global prefix sum $KV_{G}$ . The remaining computational components are same as (a).

To fully harness the parallel computing capabilities of GPU devices, we propose an optimized approach that refines the computational and communication workflow to eliminate dependencies during the computation process. This optimization effectively transforms serial computation into a parallelized one. The enhanced approach, termed LASP+ (Figure 12 (b)), operates as follows:

1.
Local Prefix Sum Calculation: Each computing node i.e., the CP rank, initiates the process by independently calculating its local prefix sum, denoted as $KV_{L}$ .
2.
Global Synchronization via AllGather: Following the local calculations, an AllGather operation is performed to synchronize the information from all nodes globally. This step ensures that each node has access to the necessary data from all other nodes.
3.
Prefix Sum Computation: Each node selects the specific CP rank’s $KV_{L}$ on which to perform the prefix sums, a decision based on its assigned computation order.

By implementing these steps, the LASP+ approach effectively removes the original dependencies between the computation nodes. This elimination of dependencies facilitates a fully parallelized computation process, thereby significantly enhancing the overall efficiency and throughput of the system. The transformation from serial to parallel computation not only leverages the full potential of GPU devices but also ensures that the training process can be executed more rapidly and with greater scalability.

The proposed modifications, while incurring additional costs in terms of increased total communication volume and temporary memory usage, are unequivocally justified by the substantial performance benefits they confer. These enhancements significantly outweigh the associated overhead in communication and memory consumption.

Through comprehensive testing and verification, it is empirically demonstrated that the computation speed in the LASP+ approach can attain up to $1/N_{pcn}$ of the original LASP algorithm, where $N_{pcn}$ denotes the number of parallel computing nodes. Furthermore, the overhead introduced by the AllGather operation is minimal, which is consistent with our anticipations and underscores the efficacy of the optimization.

Building upon the LASP+ framework, we further introduce support for the varlen feature to effectively manage the data-packing data structure. This enhancement is particularly beneficial for handling batched samples that comprise inputs with unequal token lengths. The process involves the following steps: 1). Padding to Block Size: Each input within the batch is padded to ensure that its length is a multiple of the predefined block size, which is set to 256. This padding step is crucial for aligning the data structure with the computational requirements of the kernel. 2). Sequential Concatenation: After padding, the inputs are sequentially concatenated. This concatenation facilitates the use of a single kernel to perform parallel computations across multiple batches. By organizing the data in this manner, we can efficiently leverage the parallel processing capabilities of the GPU, thereby optimizing computational performance.

The integration of the varlen feature with the LASP+ framework ensures that the system can handle diverse input lengths without compromising on efficiency. This approach not only simplifies the computational workflow but also maximizes resource utilization by enabling the processing of multiple batches concurrently.

The initial implementation of the lightning attention mechanism is primarily research-oriented and not yet suitable for practical applications, especially for inference. However, the optimization of inference processes is of paramount importance in real-world scenarios, as the long-term cost of deploying a trained model is predominantly determined by the efficiency of its inference. To this end, we implement four optimization strategies for lightning attention: batched kernel fusion, separated prefill and decoding execution, multi-level padding, and strided batched matmul extension.

We fuse multiple memory-bound kernels and extend support to accommodate all batch inputs. In the prefill phase, we perform a kernel fusion for processing the $Q,K,$ and $V$ tensors, including padding in the sequence dimension, partitioning into blocks, adjusting the internal layout, and computing the decay values. In the decoding phase, we perform a kernel fusion for the computation of $KV$ and the updating of the prefix $KV$ cache. These kernel fusions reduce intermediate result storage and memory access operations, thereby significantly improving memory access efficiency and reducing end-to-end latency by 10% in the decoding phase and short-text input scenarios. By the way, these optimizations can bring very noticeable benefits on H20 compared to H800.

The implementation of the lightning attention mechanism for long sequence computations primarily revolves around the differentiation between intra-block and inter-block computations. However, this approach is not optimal for inference tasks, particularly in the decoding phase, where the token length is consistently equal to 1.

Given that the computational kernel for tokens of length 1 is predominantly memory-bound and necessitates only a limited number of GPU Streaming Multiprocessors (SMs), we propose a strategy that segregates the processing of tokens with a length of 1 from those with a length greater than 1. This is achieved by employing two distinct kernels. Subsequently, we utilize two separate CUDA streams to schedule these kernels in parallel, thereby enhancing computational efficiency and ensuring balanced GPU utilization, especially in scenarios involving mixed inputs.

For instance, in a batch size of 20, where all inputs contain a prefix key-value (KV) cache, and the scenario includes one or two inputs with a token length of 50 while the remaining inputs have a token length of 1, this approach can significantly reduce latency. Specifically, the latency can be approximately equivalent to that of processing only the longer inputs, demonstrating a reduction from 100 milliseconds to 50 milliseconds.

By applying padding to the $Q,K,V$ tensors along the sequence dimension, the intra-block and inter-block components can be effectively decomposed into multiple identical matrix multiplications. This decomposition is particularly advantageous as it aligns seamlessly with the StrideBatchedMatmul interface, thereby facilitating the maximization of parallel processing capabilities.

Initially, the block size for padding was set to 256, a configuration that was consistent with the training parameters. However, upon the implementation of the prefix cache technique, it is observed that the token lengths within a batch typically fall below 256. This discrepancy led to redundant computations within each matrix multiplication operation. To address this inefficiency and minimize unnecessary computations, we propose the introduction of additional segmentation options, specifically 32, 64, and 128.

This multi-level padding approach enables the dynamic selection of the computational scale that incurs the minimal padding overhead, based on the current input sequence length. By adopting this approach, the utilization of computational resources is optimized, ensuring that the system operates with increased efficiency and reduced redundancy. This strategic adjustment not only conserves computational resources but also contributes to the overall performance enhancement of the system.

We utilize the optimized function cublasGemmStridedBatchedEx from the NVIDIA cuBLAS Library to manage StridedBatchedMatmul operations, thereby ensuring both high performance and versatility across diverse hardware architectures. Concurrently, we are in the process of implementing a more extensive kernel fusion strategy, with the objective of substantially improving the computational efficiency of Hopper GPUs.

Given that our sequence partitioning block size is configured to 256, the associated General Matrix-Matrix Multiplication (GEMM) operations, which involve matrices of dimensions 256x256, can leverage warpgroup-wide WGMMA instructions for computation. To further enhance memory access efficiency, we integrate the asynchronous operations of the Tensor Memory Accelerator (TMA) and delegate certain preprocessing and postprocessing computational tasks to be executed asynchronously on the CUDA Cores.

Ultimately, our goal is to dynamically regulate the number of pipeline stages to adaptively attain optimal performance across both H20 and H800 GPU architectures. This adaptive control mechanism will ensure that the system can efficiently handle varying workloads and hardware configurations, thus maximizing overall computational throughput and resource utilization.

By implementing the aforementioned optimizations, we achieve a Model Flops Utilization (MFU) exceeding 75% on the H20 GPU for end-to-end inference tasks (Chowdhery et al., 2023). Specifically, in our MiniMax-Text-01 and MiniMax-VL-01 inference, when considering the latency ratio between the attention operation and the Feed-Forward Network (FFN) operation within the MoE structure, the softmax attention constitutes 95% of the latency at a sequence length of 1,024,000 tokens. In contrast, the lightning attention implementation contributes to less than 12% of the latency under the same conditions.

Our lightning attention implementation exhibits remarkable efficiency in managing heterogeneous batch inputs, which are characterized by diverse sequence lengths. This efficiency is particularly evident in scenarios where some inputs incorporate the prefix caching strategy while others do not. The reduction in latency not only enhances the overall speed of the inference process but also ensures that the system can handle a wide range of input types with minimal performance degradation. This adaptability underscores the robustness and versatility of our lightning attention approach in real-world applications.

In this section, we provide an overview of the pre-training methodology for MiniMax-Text-01. First, we detail the meticulous construction of our pre-training corpus, with particular emphasis on data quality, standardized formatting, and mixing strategies to maximize model performance. Subsequently, we outline our innovative data experimentation framework, which enables rapid and resource-efficient evaluation of data effectiveness while minimizing computational costs. Lastly, we present an in-depth analysis of the model’s training hyper-parameters and present a hierarchical training approach, which enables context length scaling up to 4 million tokens.

The pre-training corpus for MiniMax-Text-01 encompasses a comprehensive and meticulously curated dataset, incorporating diverse sources including academic literature, books, web content, and programming code. We enhance corpus quality through several strategic dimensions:

•
Data Quality Enhancement. Superior data quality is fundamental for Large Language Models. We implement a sophisticated filtering pipeline, combining rule-based cleaning and deduplication procedures aligned with established practices (Rae et al., 2021; Penedo et al., 2023; Penedo et al., 2024). To assess document quality at a granular level, we utilize our previous-generation model as the reward labeler (a MoE model with 5B activations and 60B total parameters). Initially, we evaluate multiple quality dimensions including coherence, conciseness, educational value, helpfulness, knowledge richness, and categorical relevance. Through comprehensive analysis, we identify significant correlations among these metrics and ultimately focus on three key dimensions: knowledge depth, practical helpfulness, and categorical distribution, while maintaining other metrics as secondary validation indicators.
•
Data Formatting Optimization. The content from websites and books, once appropriately extracted and cleaned, can naturally be used as high-quality textbooks (Gunasekar et al., 2023) without further formatting. For dialogue and question-answering data, the sequential nature of text inherently captures conversational logic and question-answer relationships. Although humans benefit from additional formatting (e.g., Markdown) for readability and comprehension, we find that heavy formatting can actually diminish data diversity and quality by introducing fixed patterns that constrain the natural variation present in human conversations. Ultimately, to maintain format generalization capabilities and accommodate human preferences in alignment, we implement a nested document format with versatile templates for dialogue and QA data, carefully balancing natural comprehension with structural consistency across various interaction patterns.
•
Data Mixture Investigation. We develop a sophisticated approach to tuning the data distribution, leveraging our three primary quality metrics. Based on the experiment paradigm detailed in the subsequent section, we discover that while high-scoring content on knowledge depth and helpfulness generally yielded superior performance in capability assessments, completely eliminating lower-scoring content can adversely affect downstream task performance. Therefore, we implement a balanced sampling strategy, beginning with a uniform distribution across the base corpus, and then adjusting sampling weights to favor high-quality content while maintaining sufficient representation of diverse categories.

For tokenization, we employ byte-level Byte Pair Encoding (BPE) (Shibata et al., 1999; Brown et al., 2020), incorporating the pre-tokenizer methodology. We strategically up-sample multilingual content, to enhance the corresponding compression efficiency. The resulting vocabulary size is set to 200K tokens.

To systematically evaluate our design choices regarding pre-training data quality, format, and composition, we conduct extensive ablation experiments. These experiments involve training multiple small-scale MoE models using comparable token quantities but varying data characteristics. This approach enables us to isolate and measure the impact of individual data attributes while maintaining computational efficiency.

Formulation. We conduct Data Experiments to systematically compare the performance of different model variants. Specifically, we formulate experiments as statistical hypothesis tests that compare evaluation metric distributions between a baseline model and models trained with different data configurations. When testing the effectiveness of a new data corpus $\mathcal{D}$ , we formulate our alternative hypothesis as $H_{1}:\mu_{T_{\mathcal{D}}}>\mu_{T_{\text{baseline}}}$ , where $\mu$ represents the weighted average performance metric and $T$ denotes the distribution of evaluation values across test samples.

Evaluation. We carefully design our evaluation norms to ensure meaningful insights. We look at a wide range of multiple-choice benchmarks, discarding choice indices in query formulation and look at the likelihoods of completion. We observe the distributions of sample-wise log-normalized accuracy $\log\text{acc}_{\text{norm}^{2}}$ , defined as

\log\text{acc}_{\text{norm}^{2}}(x)=\log\text{softmax}_{p^{\prime}(c\in C_{x})}\Big\{(p^{\prime}(c^{*}))\Big\},

where $p_{i}^{\prime}(c)=\frac{p_{i}(c)}{\text{bytes(c)}}$ is the byte-normalized probability of choice $c$ for sample $i$ . We choose byte-wise normalization to exclude the effect of tokenizer, while alleviating the disfavor towards longer choices. We conduct extensive experiments to ensure that this metric is stable across training, while maintaining the discriminative power of the metric, which is quantified by the ratio $\Delta_{\text{obvious}}/\sigma_{\text{seed}}$ , where $\Delta_{\text{obvious}}$ represents the obvious difference in performance between models and $\sigma_{\text{seed}}$ denotes the standard deviation across different random seeds.

Experiment Efficiency & Setup. With such statistical setup, we are able to conduct a power analysis to decide minimal test sample size while maintaining the MDE (Minimal Detectable Effect) at a similar level as our training variance, and guaranteeing 95% confidence level and 80% power for decision making. With the confidence methodologies set, we conduct simple scaling experiments on token amount and the model size, and eventually land at an experiment step of training MoEs of 1B activation and 8B total parameters with 40B tokens of data, where data mixture comprises 20B web documents and 20B data of hypothesis.

The incorporation of repeated data has been empirically demonstrated to introduce several detrimental effects on the model’s performance and generalization capabilities (Hernandez et al., 2022). Consequently, implementing deduplication strategies is essential for optimizing LLM performance. Recent studies (Abdin et al., 2024; Penedo et al., 2024) suggest that repeatedly training high-quality documents can lead to enhanced downstream performance, with certain high-quality domains being trained up to 50 times, where the repetition is measured by MinHash similarity(Broder, 1997; Lee et al., 2022). However, our empirical analysis reveals that their experimental paradigm is inadequate for assessing the impact of repetition, as data efficiency is not consistent throughout the training process.

To achieve better alignment with the results of the full training, we introduce a novel repetition-aware experimental framework. Specifically, we first perform global deduplication on the dataset to remove redundant entries. Then, we down-sample the documents to align the repetition frequency with the requirements of the final training schedule while adhering to the budget constraints of our ablation experiments, different from the previous experimental setups which directly adopted data distributions identical or similar to those used in the final training stage. Our findings indicate that low-quality data suffer a substantial decrease in performance after training for more than two epochs, while high-quality data can be effectively trained for up to four epochs, similar to previous observations (Muennighoff et al., 2023). Notably, the solution derived from the proposed framework yields better alignment with the results obtained using considerably more computational resources. By carefully controlling the repetition and quality of the training data, we achieve a more efficient and effective data mixture, ultimately leading to better model performance.

Initial Pre-training. We initialize all model parameters using the Xavier initialization method (Glorot and Bengio, 2010), the scaling factors of DeepNorm (Wang et al., 2024a) are set to $\alpha=(2N)^{0.25}$ and $\beta=(8N)^{-0.25}$ , where $N$ denotes the number of layers. We employ the AdamW optimizer (Loshchilov and Hutter, 2019) with $\beta_{1}=0.9$ , $\beta_{2}=0.95$ , and the weight decay is set to 0.1. The training sequence length is 8192, and the batch size is progressively scaled from an initial size of 16M to 32M at 69B tokens, to 64M at 790B tokens, and finally to 128M at 4.7T tokens, where it remains until the end of training. The schedule is designed based on the correlation between training loss and the critical batch size (McCandlish et al., 2018). It is argued that training at the critical batch size yields a near-optimal balance between training time and data efficiency (Kaplan et al., 2020). Following this, we fit a power-law relationship between the loss and the critical batch size on data from smaller models, as shown in Figure 13. The batch size is doubled when the corresponding loss is reached.

Figure 13: The power-law fit for the training loss and the critical batch size, utilizing data from models ranging from 50M to 600M in activated parameters counts. We mark the points where the batch size is doubled with dashed gray lines.

The learning rate schedule begins with a linear warm-up over 500 iterations to a peak value of $2\times 10^{-4}$ , followed by training with a constant learning rate for 7.2T tokens. In the latter stages of training, we notice anomalous gradient norm values. This issue is attributed to an excessively high learning rate and we adjusted lr to $1.3\times 10^{-4}$ for the remaining 3.2T tokens. During the fast decay phase, we train 1T tokens and exponentially decrease the learning rate to $3\times 10^{-5}$ . Additionally, the MoE auxiliary loss coefficient is set to 0.01.

Long-Context Extension. We incrementally expand the model’s training context length to 1M tokens. Due to our architecture’s effective length extrapolation capabilities, the model successfully demonstrates its ability to process sequences up to 4M tokens in the vanilla Needle-In-A-Haystack retrieval task (NIAH) test $^{1}$ , despite only being trained on contexts up to 1M tokens, as illustrated in Figure 14.

Figure 14: **4 Million vanilla Needle-In-A-Haystack retrieval task pressure test on MiniMax-Text-01. The token interval is 32K when it is less than 1M, and the token interval is 0.5M when it is greater than 1M.**

Specifically, we employ a three-stage training procedure to systematically upsample long-context data across diverse length ranges, while preserving the distributional characteristics of critical domains to preserve short-context evaluation performances steady. The details of the training data mixture, RoPE base frequency, and training length are shown in Table 6. We also mix in 10% of high-quality long-context question-answering data with similar length distribution as long-context pre-training data during the last 20% of training cycles in each stage(Parmar et al., 2024). To mitigate potential instabilities resulting from distributional shifts, we utilize linear interpolation of source-specific weights throughout the transitional phase. This method facilitates a gradual and controlled evolution of the data distribution towards the desired target distribution, thereby ensuring training stability and preserving convergence properties.

Additionally, our findings indicate that NIAH is inadequate for effectively monitoring the model’s performance throughout the training process. This is primarily because NIAH metric performance reaches its peak score early on, specifically within the initial 128K training steps. To tackle this limitation, we evaluate the model’s intermediate checkpoints using more demanding tasks, which are designed to increase in complexity as training progresses. Notably, despite the escalating difficulty of these tasks, we consistently observe a steady improvement in the model’s performance metrics. This sustained upward trajectory clearly demonstrates the critical importance and necessity of implementing long-context continual pretraining. More details are given in Section 5.7.2.

Original Table 6 — Table 6: Long-Context Extension Recipe. For clarity, we categorize the data as follows: data with fewer than 32K tokens are labeled as “Short”; data ranging from 32K to 128K tokens are labeled as “Medium”; and data exceeding 128K tokens are categorized as “Long”.

Table 6
Training Length	RoPE Frequency	# Tokens	Short (%)	Medium (%)	Long (%)
128K	5M	300B	30	70	0
512K	10M	32B	35	35	30
1M	10M	26B	30	30	40

In this section, we present a thorough post-training framework designed to enhance the model’s general performance, long-context capability, and real-world applicability. Our approach begins with the creation of a diverse, high-quality prompt dataset, accompanied by a hierarchical reward system that evaluates responses across multiple dimensions: correctness, truthfulness, helpfulness, and harmlessness. The training process consists of Supervised Fine-Tuning (SFT), Offline and Online Reinforcement Learning (RL). Through these phases, we systematically align the model with our defined objectives. Model safety is ensured through exhaustive data mining techniques and a specialized harmless reward model. We introduce a novel multi-stage training methodology that significantly enhances the model’s capacity to process extended contexts while maintaining optimal performance on shorter sequences. This approach results in a robust system capable of handling complex, real-world scenarios. Extensive evaluations conducted across both academic and in-house benchmarks demonstrate that our model achieves top performance across all tasks, while establishing new standards of extremely long-context processing.

Our extensive prompt collection encompasses millions of diverse, high-quality queries from various sources. We develop a tagging system that categorizes each prompt based on task type, knowledge domain, and difficulty level. The collection process incorporates sophisticated filtering mechanisms to eliminate redundant prompts while maintaining an optimal difficulty distribution. The prompt set spans various domains including long-context, programming, math, logical reasoning, creative writing, function calling, general-knowledge, and safety-related scenarios.

Our reward model framework evaluates responses across four critical dimensions to ensure alignment with our core principles:

•
Correctness. We implement a rigorous evaluation system for responses that can be strictly validated. For mathematical and reasoning tasks, we utilize early-version MiniMax-Text-01 to generate binary reward signals based on answer consistency. Programming solutions undergo comprehensive testing in a secured sandbox environment, with performance metrics derived from test case success rates.
•
Truthfulness. We employ a verification pipeline to assess the factual accuracy of the response. The process involves systematic response sampling, statement decomposition and clustering, crowd-sourced verification, and automated comparison using advanced language models to generate truthfulness scores.
•
Helpfulness. Our evaluation framework assesses compliance with user instructions through both deterministic and probabilistic approaches. We implement automated rule-based constraint verification systems complemented by human evaluation of key metrics including coherence, depth, contextual relevance, and stylistic appropriateness. The final helpfulness score combines multiple evaluation signals through a weighted scoring system.
•
Harmlessness. Building upon Constitutional AI principles (Bai et al., 2022b), we develop evaluation criteria encompassing safety protocols, content appropriateness, and legal compliance. Our assessment system leverages carefully calibrated prompts validated against human annotations, with early-version MiniMax-Text-01 providing standardized safety evaluations.

Our SFT dataset construction involves a multi-stage process utilizing domain-specific expert models trained through iterative SFT and RL cycles. We implement rejection sampling (Bai et al., 2022a; Dubey et al., 2024) to generate high-quality responses by the experts, sampling multiple variations per prompt across different temperature settings to select optimal demonstrations measured by the reward hierarchy. The response selection process further incorporates both n-gram and semantic similarity filters to ensure maximum diversity and quality in the training data.

We incorporate the offline RL phase, i.e., Direct Preference Optimization (DPO) (Rafailov et al., 2023), to optimize the model’s performance across diverse prompt distributions, owing to its simplicity and ease of data construction for long-context scenarios. We specifically focus on prompts that maintain distributional consistency with those utilized in the SFT stage. To evaluate the impact of prompt selection, we conduct comparative experiments using two prompt categories: SFT-trained prompts and SFT-untrained but homologous prompts. Empirical results demonstrate negligible performance variations between SFT-trained prompts and their untrained counterparts. Thus, we adopt the SFT-trained ones for the offline RL phase. The experimental protocol involves generating responses with varying temperature parameters for each prompt, followed by systematic evaluation using the reward models described in Section 5.2. We then identify the best and the worst responses to construct preference pairs for DPO training.

Online learning demonstrates superior sample efficiency and cross-domain generalization capabilities compared to offline learning methodologies. Therefore, we implement online RL to improve model performance, particularly in mathematical reasoning tasks. Our approach emphasizes prompt diversity and prioritizes prompts with moderate success rates to maximize information gain during policy updates. Notably, we employ SFT-untrained prompts during online RL, as our empirical observations indicate that reusing prompts from previous phases resulted in model saturation, characterized by diminished response perplexity. We propose a modified Group Relative Policy Optimization (GRPO) (Shao et al., 2024) approach incorporating the following key innovations:

•
Importance Sampling Weight Clipping. The conventional PPO/GRPO implementation employs one-sided clipping (Schulman et al., 2017; Shao et al., 2024), sometimes leading to gradient instability when processing tokens with a large policy ratio and negative advantage. To address this issue, we implement additional clipping that abandoned this case in the loss function, which effectively regulates the importance sampling magnitude and mitigates noise propagation.
•
KL Divergence Optimization. Due to the similar gradient instability issue, we reformulate the KL divergence term through theoretical analysis of the variance-bias trade-off to further stabilize gradient behavior, resulting in $\mathbb{D}_{KL}(\theta)=\mathbb{E}_{t}[\text{SG}(\pi_{\theta}(a_{t}|s_{t})-\pi_{\text{ref}}(a_{t}|s_{t}))\log\pi_{\theta}(a_{t}|s_{t})]$ , where $\text{SG}(\cdot)$ denotes the stop-gradient operator. This formulation maintains policy consistency while reducing gradient variance.
•
Balanced Advantage Estimation. We also ensure equitable reward contributions between positive and negative examples, which proves particularly effective in scenarios with skewed distributions. This approach maintains stable training dynamics by regulating the absolute magnitude of rewards across different example groups.

The safety alignment of our model is meticulously addressed throughout both the SFT and RL stages. To strike an optimal balance between the model’s harmlessness and helpfulness, we employ an approach that encompasses the following key components.

We construct high-quality alignment training data with a focus on ensuring data diversity and accuracy. This involves the implementation of several data collection methodologies designed to cover a broad spectrum of safety scenarios:

•
Safety-Category Specific Prompts. Leveraging established safety classification standards and insights from safety and domain experts, we generate tailored prompts for specific safety categories. This ensures that the model is exposed to a comprehensive set of safety-related scenarios.
•
Real-World User Data Collection. We collect real-world user questions from various web documents to incorporate authentic and diverse safety-related queries into our training data.
•
Prompt Augmentation. We instruct early-version MiniMax-Text-01 to generate additional related prompts based on the collected typical red team attack prompts. This approach aims to expand the diversity of safety scenarios and enhance the robustness of the model’s safety mechanisms.

To generate safe and appropriate responses, we employ a harmless reward model (Bai et al., 2022b) that is developed based on a set of detailed safety rules. To prevent the model from producing unreasonable refusals, we carefully integrate principles of helpfulness into the safety rules. This integration plays a crucial role in achieving a balanced output capability, enabling the model to provide safer responses without compromising its utility to the user. The resulting safety-aligned system demonstrates robust protection against potential misuse while maintaining high performance across intended use cases.

We propose a systematic multi-stage training methodology to enhance the model’s capacity for processing extended contexts, as shown in Tab. 7. This approach is methodically designed to optimize long-sequence handling while maintaining performance efficacy on conventional shorter sequences. The RoPE base frequency is maintained at 10 million throughout the post-training phase to ensure consistency in positional encoding.

Stage I: Initial Short-Context Training. The first stage implements SFT with sequences constrained to 8,192 tokens. This foundational phase establishes baseline competency in processing standard-length queries and responses, which constitute the majority of practical applications. We remove the long-context prompts that are longer than 8,192 tokens in this stage.

Stage II: Extended Context Training. The second stage implements a significant extension of the sequence length to 1,032,192 tokens. This phase incorporates training samples across diverse sequence lengths with 50% long-context prompts, facilitating comprehensive model adaptation to extensive contextual processing. The strategic expansion of the sequence length is fundamental to achieving robust long-context capabilities.

Stage III: Short-Context Preference Optimization. In this phase, we revert to 8,192 tokens for sequence length and implement Direct Preference Optimization (DPO). This calibration ensures optimal performance on conventional context sizes while maintaining the previously acquired capabilities.

Stage IV: Long-Context Preference Optimization. The fourth stage focuses on reinforcing long-context processing capabilities through DPO with sequences of 1,032,192 tokens. This phase employs training protocols analogous to Stage III with entirely long-context data, adapted for extended sequence lengths.

Stage V: Online Reinforcement Learning. The final stage implements short-context Online Reinforcement Learning with a sequence length of 8,192 tokens. More details have been outlined in Section 5.4.2.

Original Table 7 — Table 7: **Training Recipe for Post-training Alignment.**

Table 7
	Stage I	Stage II	Stag III	Stage IV	Stage V
Sequence Length	8192	1032192	8192	1032192	8192
Epoch	2	2	1	1	1
Batch Size	128	80	64	64	512
Max LR	1e-5	3e-6	5e-7	5e-7	1e-6
Min LR	1e-6	3e-6	5e-8	5e-7	1e-7
LR Decay	Cosine	Constant	Cosine	Constant	Cosine

We observe and report open-source short- and long-context benchmarks that highlight our model’s capabilities across various aspects. Along with the user-oriented evaluations we will discuss in Section 5.8, we show that MiniMax-Text-01 is a leading open-source model that achieves top performance in long-context retrieval, understanding, long in-context learning and knowledge-based requests, while performing well in math, reasoning, and code tasks and demonstrating strong usefulness in real-user assistant scenarios.

MMLU (Hendrycks et al., 2021a) and MMLU-Pro (Wang et al., 2024b) are widely adopted datasets that assess the extent of a model’s knowledge across a broad range of domains. We further observe SimpleQA (Wei et al., 2024), a factuality benchmark that challenges the model’s knowledge boundary, and C-SimpleQA (He et al., 2024b) which is an adapted version of SimpleQA under the Chinese culture. For the observation of reasoning capabilities, we evaluate on GPQA (Rein et al., 2024) for graduate-level knowledge reasoning, and DROP (Dua et al., 2019) for reading comprehension reasoning. We test our model’s performance on math problem-solving with grade-school-level task GSM8k (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) that spans from AMC-8 to AIME-level across 7 subjects. We monitor our model’s coding capability by observing the Pass $@$ 1 rate on HumanEval (Chen et al., 2021) and MBPP Plus (Austin et al., 2021; Liu et al., 2023) datasets. To test the models’ ability to interpret and execute detailed and nuanced instructions, we evaluate the IFEval (Zhou et al., 2023) benchmark. Furthermore, we observe Arena-Hard-Auto (Li et al., 2024b) that reflects the alignment to human preferences.

We adopt greedy decoding and a zero-shot chain-of-thought strategy (Wei et al., 2022) in evaluating our instruction-tuned model. We compare with other leading and open-source LLMs, which we evaluate under the same setting, if not reported. We present the performance of MiniMax-Text-01 in Table 8. As shown, MiniMax-Text-01 exhibits remarkable performance across most dimensions. It surpasses all models on C-SimpleQA with its more extensive knowledge boundary under Chinese culture. MiniMax-Text-01 also achieves top-3 performance across MMLU, IFEval, and Arena-Hard, showing its exceptional capability of applying its comprehensive knowledge within given constraints to well satisfy user queries and align with human preferences. Meanwhile, it achieves a better MATH pass $@$ 1 rate than GPT-4o, Claude-3.5-Sonnet, and Llama-3.1-405B, and exhibits comparable performance with instructed Qwen2.5-72B on HumanEval. Moreover, MiniMax-Text-01 achieves 54.4 on GPQA Diamond, which exceeds most open-source instruction-tuned LLMs and the latest version of GPT-4o.

Original Table 8 — Table 8: **Performance of MiniMax-Text-01 on core academic benchmarks.**

Table 8
Tasks	GPT-4o (11-20)	Claude-3.5- Sonnet (10-22)	Gemini-1.5- Pro (002)	Gemini-2.0- Flash (exp)	Qwen2.5- 72B-Inst.	DeepSeek- V3	Llama-3.1- 405B-Inst.	MiniMax- Text-01
General
MMLU∗	85.7	88.3	86.8	86.5	86.1	88.5	88.6	88.5
MMLU-Pro∗	74.4	78.0	75.8	76.4	71.1	75.9	73.3	75.7
SimpleQA	39.0	28.1	23.4	26.6	10.3	24.9	23.2	23.7
C-SimpleQA	64.6	56.8	59.4	63.3	52.2	64.8	54.7	67.4
IFEval (avg)	84.1	90.1	89.4	88.4	87.2	87.3	86.4	89.1
Arena-Hard	92.4	87.6	85.3	72.7	81.2	91.4	63.5	89.1
Reasoning
GPQA∗ (diamond)	46.0	65.0	59.1	62.1	49.0	59.1	50.7	54.4
DROP∗ (F1)	89.2	88.8	89.2	89.3	85.0	91.0	92.5	87.8
Mathematics
GSM8k∗	95.6	96.9	95.2	95.4	95.8	96.7	96.7	94.8
MATH∗	76.6	74.1	84.6	83.9	81.8	84.6	73.8	77.4
Coding
MBPP +	76.2	75.1	75.4	75.9	77.0	78.8	73.0	71.7
HumanEval	90.2	93.7	86.6	89.6	86.6	92.1	89.0	86.9
∗ Evaluated following a 0-shot CoT setting.

As previously discussed in the long-context extension part of section 4.2, the NIAH task is kind of simplistic for our model, rendering it insufficient for observing the model’s optimization progress. Consequently, we shift our evaluation to more challenging tasks. Our current long-context evaluation framework focuses on three primary dimensions: (1) Long-Context Retrieval, (2) Long-Context Understanding, and (3) Long In-Context Learning.

This dimension assesses the model’s memory capabilities, which serve as the foundation for almost all long-context tasks. In addition to vanilla k-M NIAH (Kamradt, 2023), we construct a more challenging variation to assess our Long-Context Retrieval performance, namely Multi-Round Needles-In-A-Haystack (MR-NIAH), serving as a crucial back up for retrieval tasks in long multi-turn dialogue contexts, revealing the fundamental capabilities for building lifelong companion AI assistants. Similar to Multi-round co-reference resolution (MRCR) (Vodrahalli et al., 2024) which is not open-source, we construct haystacks of MR-NIAH as history dialogues, where user queries are synthetic but explicit requests of event descriptions and creative writing. In the last round, the query requests the model to repeat the response of one of the history requests. The haystacks span from 2K to 1M tokens (up to around 2000 interactions), and each needle request is injected at 25%, 50%, and 75% of the conversation, respectively. Each ground truth response contains three core components, and we look at an adjusted recall $\frac{\text{corr. comp.}}{3}$ . We show a case illustration in Appendix B.2.

Figure 15 illustrates comparison results of MR-NIAH. Our model (“MiniMax-Text-01”, red line) shows strong performance across a wide range of sequence lengths in both English and Chinese evaluations. Compared to competing baselines (e.g., GPT, Claude, and Gemini variants), our model also shows less performance degradation at large input lengths, underscoring its robustness for long-context retrieval tasks.

Figure 15: **MR-NIAH in English and Chinese.**

This dimension measures the model’s long-context understanding ability which contains logical reasoning skills based on long-context inputs. We utilize two comprehensive long-context QA datasets, Ruler (Hsieh et al., 2024) and LongBench-V2 (Bai et al., 2024) to evaluate this aspect. Ruler includes 13 different tasks and notably introduces multi-hop tracing and aggregation tasks to evaluate the complex reasoning abilities of models. We test Ruler up to a sequence length of 1M tokens. LongBench-V2 encompasses question-answering tasks of varying difficulty levels across multiple context types, including single and multi-document, multi-turn dialogue, code repositories, and long structured data, among others. Following LongBench-V2 (Bai et al., 2024), we consider two test modes: w/o CoT and w/ CoT, and the text lengths are categorized as follows: Short, ranging from 0 to 32K words; Medium, spanning from 32K to 128K words; and Long, covering 128K to 2M words.

As Table 9 illustrates, our model exhibits notable strengths in processing Ruler’s long-context reasoning tasks. While performance at the 64k input level remains competitive with leading models (including GPT-4o and Claude-3.5-Sonnet) with minimal variation, MiniMax-Text-01 establishes a distinct advantage beginning at 128k, achieving impressive scores and surpassing all benchmark models. This superiority becomes particularly pronounced in ultra-long-context scenarios (such as 1M), where MiniMax-Text-01 maintains its commanding lead. Moreover, as evident in Table 10 $^{2}$ , MiniMax-Text-01 exhibits outstanding capabilities in LongBench-V2’s long-context reasoning tasks. The model achieves state-of-the-art results among all evaluated systems in the w/ CoT setting, while also displaying remarkable effectiveness in scenarios w/o CoT.

Overall, MiniMax-Text-01 demonstrates exceptional capability in long-context understanding especially reasoning tasks, both with and without CoT reasoning, particularly excelling in scenarios requiring complex reasoning. The exceptional robustness and stability of the model in processing long-context understanding tasks can be attributed to the hybrid architecture with half RoPE and carefully tuned training recipes for both pre-training and alignment, which enhance the model’s ability to handle long sequences effectively.

Original Table 9 — Table 9: **Performance comparison of MiniMax-Text-01 on Ruler.**

Table 9
Model	4k	8k	16k	32k	64k	128k	256k	512k	1M
GPT-4o (11-20)	0.970	0.921	0.890	0.888	0.884	-	-	-	-
Claude-3.5-Sonnet (10-22)	0.965	0.960	0.957	0.950	0.952	0.938	-	-	-
Gemini-1.5-Pro (002)	0.962	0.960	0.960	0.958	0.938	0.917	0.916	0.861	0.850
Gemini-2.0-Flash (exp)	0.960	0.960	0.951	0.957	0.937	0.860	0.797	0.709	-
MiniMax-Text-01	0.963	0.961	0.953	0.954	0.943	0.947	0.945	0.928	0.910

Original Table 10 — Table 10: **Performance comparison of MiniMax-Text-01 on LongBench v2.**

Table 10
Model	overall	easy	hard	short	medium	long
Human	53.7	100.0	25.1	47.2	59.1	53.7
w/ CoT
GPT-4o (11-20)	51.4	54.2	49.7	59.6	48.6	43.5
Claude-3.5-Sonnet (10-22)	46.7	55.2	41.5	53.9	41.9	44.4
Deepseek-V3	-	-	-	-	-	-
Qwen2.5-72B-Inst.	43.5	47.9	40.8	48.9	40.9	39.8
MiniMax-Text-01	56.5	66.1	50.5	61.7	56.7	47.2
w/o CoT
GPT-4o (11-20)	50.1	57.4	45.6	53.3	52.4	40.2
Claude-3.5-Sonnet (10-22)	41.0	46.9	37.3	46.1	38.6	37.0
Deepseek-V3	48.7	-	-	-	-	-
Qwen2.5-72B-Inst.	42.1	42.7	41.8	45.6	38.1	44.4
MiniMax-Text-01	52.9	60.9	47.9	58.9	52.6	43.5

This dimension evaluates the model’s ability to learn from context, a core area of research in lifelong learning. We benchmark our Long In-Context Learning capability with the MTOB (Machine Translation

Figure 16: Changes of eng $\rightarrow$ kalam (ChrF) during the whole long-context extension training process.

from One Book) (Tanzer et al., 2024) dataset. The task requires a model to translate between English and Kalamang, a language that is very limited in open data and thus within the training corpus, and the LLM is expected to learn the language only from parts of a grammar book and 375 translation examples, all given in the context for each translation query (Appendix B.1). The context length is $\sim 81$ K tokens under a half-book setting and $\sim 133$ K tokens under a total-book setting. We present our results in Table 11.

We carefully examined the pre-training data and found that only a very small amount of data contains Kalamang-related content. As a result, the eng $\rightarrow$ kalam (ChrF) score of our model is the lowest in the no-context scenario, while other models we compared with likely have had their pre-train or post-train data enhanced with relevant Kalamang data. As well as the delta half and full book metrics, our model surpasses all models in terms of the eng $\rightarrow$ kalam (ChrF) metric. And our model also has comparable performance with other models on kalam $\rightarrow$ eng (BLEURT) metric.

In the course of long-context extension, as described in section 4.2, we observed a gradual enhancement in In-Context Learning ability, as indicated by MTOB, illustrated in Figure 16. While we have explored some remarkable works(Agarwal et al., 2024; Dong et al., 2024) specifically aimed at improving In-Context Learning capabilities, we believe that such ability should merely be one aspect of the reasoning capabilities of long-context models. Therefore, we plan to conduct in-depth research on long-context data quality and scale from a more fundamental perspective to further enhance the long-context reasoning capabilities of our model.

Original Table 11 — Table 11: **Performance comparison of MiniMax-Text-01 on MTOB.**

Table 11
Context Type	no context	half book	full book	$\Delta$ half book	$\Delta$ full book
eng $\rightarrow$ kalam (ChrF)
GPT-4o (11-20)	9.90	54.30	-	44.40	-
Claude-3.5-Sonnet (10-22)	20.22	53.62	55.65	33.39	35.42
Gemini-1.5-Pro (002)	16.79	53.68	57.90	36.89	41.11
Gemini-2.0-Flash (exp)	12.20	49.50	53.30	37.30	41.10
Qwen-Long	16.55	48.48	45.94	31.92	29.39
MiniMax-Text-01	6.0	51.74	51.60	45.7	45.6
kalam $\rightarrow$ eng (BLEURT)
GPT-4o (11-20)	33.20	58.30	-	25.10	-
Claude-3.5-Sonnet (10-22)	31.42	59.70	62.30	28.28	30.88
Gemini-1.5-Pro (002)	32.02	61.52	63.09	29.50	31.07
Gemini-2.0-Flash (exp)	33.80	57.50	57.00	23.70	23.20
Qwen-Long	30.13	53.14	32.15	23.01	2.02
MiniMax-Text-01	33.65	57.10	58.00	23.45	24.35

While achieving top performance on the core open-source benchmarks, we realize that academic evaluations lack an understanding of real-world user interactions. Hence, we also focus on monitoring and improving user experience through our Hailuo AI $^{3}$ by incorporating user-in-the-loop evaluations based on real-world cases and adapting tools for better usability and performance in practical applications.

We maintain a series of in-house evaluations that include: (1) automatic assessments of General Assistant capabilities, Knowledge Q&A, Creative Writing, Hard Capability, Instruction Following, Coding, Safety, and Long Context, and (2) expert human evaluations. It’s worth noting that since our test queries are primarily derived from Hailuo AI user interactions, a significant portion of our in-house samples are in Mandarin and deeply rooted in Chinese cultural contexts.

Our results indicate a notable discrepancy between performance on academic benchmarks and actual user experience, where leading open-source and commercial models can underperform when used as interactive assistants. We show in Table 12 $^{4}$ that, through our dedicated efforts, MiniMax-Text-01 is able to handle these situations quite well. In general, our model outperforms other models in common Assistant scenarios, particularly when compared to open-source counterparts. This superiority is most evident in our Creative Writing (Appendix B.5, B.7, B.6) and Knowledge Q&A collections, where it aligns more closely with user intentions than other models, delivering accurate and detailed responses to a wide range of queries. In productivity scenarios that require Long Context (Appendix B.3), such as document translation, summarization, and analysis, our model demonstrates high proficiency and reliability. Moreover, we prioritize the safety of our model, as it achieves top-tier performance on our established in-house Safety benchmarks.

Meanwhile, we are agile in gathering and updating complex productivity scenarios with multilevel instruction following requests at which our model fails and current LLMs cannot master, constructing our Harder Capability and Instruction Following in-house evaluations. While leading LLMs tend to underperform in these sets, these requests reflect our model’s limitations when given multi-level instructions, which stems primarily from insufficient training data for specific instruction types. Moving forward, we are committed to substantially expanding our training dataset with high-quality, targeted content to address these gaps and improve model capabilities.

Original Table 12 — Table 12: **Performance comparison of MiniMax-Text-01 on in-house benchmarks.**

Table 12
	General Assistant	Hard Capability	Creative Writing	Knowledge Q&A	Instruction Following	Coding	Safety	Long Context
GPT-4o (11-20)	70.9	73.5	70.3	69.2	50.4	94.0	85.4	86.2
GPT-4o (08-06)	63.5	62.0	66	68.0	49.1	93.6	79.7	58.3
GPT-4o (05-13)	67.7	63.3	58.3	69.6	49.6	93.2	79.7	77.2
Claude-3.5-Sonnet (10-22)	66.8	68.3	54.3	52.0	61.5	94.4	92.9	47.1
Claude-3.5-Sonnet (06-20)	60.5	67.4	51.0	51.8	64.4	93.6	95.0	47.1
Gemini-2.0-Flash (exp)	70.1	61.8	70.0	75.1	39.9	86.5	66.2	81.9
Qwen2.5-72B-Inst.	66.4	66.1	61.7	68.9	34.1	93.9	-	81.5
DeepSeek-V3	66.8	68.7	64.6	77.0	51.8	94.0	74.9	77.8
Llama-3.1-405B-Inst.	53.3	-	63.6	46.0	50.3	87.6	70.7	60.3
MiniMax-Text-01	73.9	64.8	81.3	78.6	46.3	90.2	90.9	93.8

During user interaction case studies, we find a model’s capability to utilize search tools can compensate for the limited knowledge boundary by accessing real-time, extensive, and precise information from the web. To maximize the model’s benefits from search while minimizing additional performance degradation, we first carefully pre-define the scope of search scenarios, which cover approximately $30\sim 40$ % of user queries, including but not limited to precision-demanding, domain-specific, and time-sensitive requests. Meanwhile, to ensure a seamless conversation experience, we define the system as invoking tools directly through special tokens, which avoid the complexity of multi-step planning (Chen et al., 2024b) or chain-of-thought reasoning $^{5}$ that might disrupt the natural flow of the interactions. We create SFT datasets comprising search and non-search decisions across diverse domains, while carefully controlling for other interaction features unrelated to search decisions, such as conversation length, to maintain uniform data distribution across each dimension and prevent overfitting. Importantly, we employ the corresponding reward model of each sample to ensure response quality, failing at which would introduce suboptimal samples into the training data, potentially affecting the model’s fundamental capabilities. The search decision boundary was calibrated to align with the model’s knowledge boundaries, discarding samples that our model already masters from the search corpus, such as general Chinese knowledge Q&A. After careful assessments by human evaluation experts, we conclude that our model’s use of the search tool extensively improved user experience, landing at a performance leap from $58\%$ to $71.5\%$ on our out-of-domain Hailuo AI end-to-end evaluation (Appendix B.9). Since we are unsure whether other LLM-based assistants include similar search tools, we refrain from making unfair performance comparisons.

By integrating an image encoder and an image adapter into our MiniMax-Text-01 model, we develop MiniMax-VL-01, which extends the capabilities of the model to visual understanding tasks. To ensure robust visual understanding, we design a proprietary dataset and implement a multi-stage training strategy, where the newly introduced image encoder and adapter first undergo large-scale visual pre-training, followed by comprehensive fine-tuning of the entire pipeline.

In the following section, we begin with a comprehensive description of the dataset used for training our image encoder and vision-language model. Subsequently, we provide an in-depth overview of the model architecture, followed by an exposition of our four-stage training regimen. We conclude the section by presenting our benchmark results.

To pre-train the vision encoder, we curate a substantial image-caption dataset by aggregating and filtering data from internet sources. Our Vision Transformer (ViT) is trained using 694 million unique image-caption pairs. To enhance data quality, we acquire refined captions for 180 million images within these pairs. During the training process, we employ an augmentation strategy by randomly sampling raw and refined captions with equal probability ( $p=0.5$ ).

In existing vision-language models, the utility of descriptive imagery for model training has been well-documented (Schuhmann et al., 2021; Li et al., 2022; Li et al., 2023; Li et al., 2024a). To further explore this avenue, we have compiled a dataset consisting of 100 million images sourced from open resources such as Common Crawl. Each image in this dataset is paired with a fine-grained description, which is initially synthesized by a caption model and subsequently refined through humans. On average, these descriptions comprise approximately 300 text tokens per image. Description data serves as a robust resource for modal alignment and enhancing understanding in further training.

To train MiniMax-VL-01, we construct a comprehensive and diverse instruction-based dataset by synthesizing an extensive range of question-answer (QA) pairs involving visual inputs. These QA pairs are meticulously designed to cover a wide array of image-related tasks, such as text extraction, object localization, and geometry problem solving. The dataset generation process prioritizes both diversity and realism, ensuring that the instructions capture varying degrees of complexity and linguistic styles. During training, we apply an augmentation strategy by randomly sampling different types of QA prompts with balanced probabilities, thereby enabling the model to generalize effectively across diverse instructional formats and interaction patterns.

Figure 17: Visualization of top tags of sampled instruction data. The category and percentage for each group of clustered tags are displayed in the inner layer, only top-10 tags of each group are displayed for clarity.

To demonstrate the diversity of our VLM data, we uniformly sample 1 million image-instruction pairs from the instruction data and use another VLM to assign a concise tag (e.g., object localization) that represents the primary capability required for each pair. This analysis yielded around 50,000 unique tags, and the top 2,817 tags appeared more than 10 times. The distribution of these prominent tags is visualized in Figure 17, where we further group these top tags into 14 major categories.

Our MiniMax-VL-01 architecture adheres to the “ViT-MLP-LLM” paradigm, which has been widely embraced in numerous multimodal large language models (MLLMs). The architecture consists of three main components: a Vision Transformer (ViT) with 303 million parameters for visual encoding, a two-layer MLP projector initialized randomly for image adaptation, and the MiniMax-Text-01 model serving as the foundational large language model (LLM).

We implement a dynamic resolution strategy by resizing the input image according to a predefined grid configuration list, ranging from $336\times 336$ to $2016\times 2016$ , while maintaining a standard thumbnail at a resolution of $336\times 336$ . The resized images are subsequently partitioned into non-overlapping patches, each measuring $336\times 336$ . Both the image patches and the thumbnail are independently encoded, and their encoded features are concatenated to construct a comprehensive image feature representation.

In contrast to traditional approaches that rely on pooling or other downsampling techniques to compress feature representations, our model leverages its powerful capacity for processing long sequences, allowing for the direct utilization of raw high-dimensional features during training. This strategy mitigates potential information loss and substantially improves the model’s adaptability to multi-scale inputs. Moreover, by projecting both image patches and thumbnails into a unified feature space, our method significantly enhances the model’s robustness and representational expressiveness when handling diverse and complex visual inputs.

We employ a lightweight ViT-L/14 (Dosovitskiy et al., 2021) as the foundational structure for our vision encoder and train it from scratch. Following a standard pipeline, the input image tensor is initially processed through a convolutional layer to extract discrete patches, to which absolute positional embeddings are subsequently appended. The resulting tensors are then passed through a series of multi-head residual attention blocks. This architecture is particularly effective in capturing intricate visual details and the complex interrelationships within images.

We utilize contrastive learning to enhance the alignment between corresponding image-caption pairs while diminishing the alignment between non-corresponding pairs. Specifically, we follow the approach introduced in CoCa (Yu et al., 2022), which augments image-text contrastive learning with an additional decoder and image-text cross-attention mechanisms. The network is jointly optimized using a combination of contrastive loss and cross-entropy loss.

Our ViT-L/14 model is initially trained at a resolution of $224\times 224$ for 37 billion image-caption pairs and subsequently fine-tuned at $336\times 336$ for 1.2 billion pairs. For both resolutions, the captions are truncated to 76 tokens. Our ViT-L/14 encoder achieves a zero-shot classification accuracy of 80.55% at $336\times 336$ resolution on the ImageNet-1K dataset.

We employ a four-stage training strategy to enable the model to progressively develop comprehensive multimodal understanding capabilities while retaining its language understanding skills. Additionally, the model’s question-answering and instruction-following abilities, as well as its alignment with human preferences, are methodically refined throughout these stages.

Stage I: Modality alignment. In this stage, our primary objective is to achieve alignment between visual and text tokens by enabling the model to accurately generate appropriate captions for given images. To this end, we update the weights of both the image adapter and the vision encoder to optimize their performance in this multimodal task. During this phase, we utilize a total of 80 billion tokens sampled from our image description dataset. Empirically, we have found that increasing the image resolution does not yield improvements in downstream task accuracy. Therefore, all images are processed at a fixed resolution of $336\times 336$ to reduce computational costs.

Stage II: Enhancement of Vision Understanding. This stage can be regarded as a standard instruction tuning phase, during which all model parameters are open to updates. The primary goal is to align the model’s output with human instructions and enhance its ability to perform a diverse range of vision understanding tasks. To achieve this, the model is trained using 420 billion multimodal tokens sampled from our instruction datasets, combined with MiniMax-Text-01 post-training data in a ratio of 20:1. This approach ensures that the language modeling capability is maintained while the model acquires new multimodal capabilities.

Stage III: Enhancement of User Experience. This stage is designed to further enhance the model’s capabilities in real-world scenarios and when handling challenging user inputs. We curate sophisticated multimodal data using images sourced from applications that people commonly interact with. Conversations are meticulously labeled to emulate authentic user input and to ensure the provision of accurate, helpful, and diverse responses across multiple conversational turns. The data construction for this stage is guided by an independent human-labeled test set that prioritizes not only accuracy but also the overall quality in terms of user experience. The resulting dataset comprises 44.8 billion multimodal tokens and is trained for one epoch.

Stage IV: Enhancement of Preference. In the final stage, we utilize Direct Preference Optimization (DPO) to further enhance model performance and user experience. We construct a training dataset consisting of 40,000 image-text pairs through the following process:

•
Prompt Selection. Prompts are curated from both instruction data and real user interaction data. These prompts are selected to cover a wide range of general scenarios and to specifically address persistent issues identified after Stage III, such as occasional repetitive outputs in complex OCR scenarios.
•
Response Generation. We employ diverse strategies, including: generating multiple candidate responses by varying sampling temperature parameters; creating response variants through image weakening in specific scenarios; and using MiniMax-Text-01 to deliberately introduce hallucinations or errors into high-quality responses to generate contrastive samples in specific scenarios.
•
Reward Assignment. Large language models, particularly MiniMax-Text-01, are utilized as evaluators in this stage. Multi-dimensional evaluation criteria are designed to enable a systematic and comprehensive assessment of the relationships among prompts, ground truth answers, and generated responses.
•
Pair Construction. Based on the evaluation results, we select the highest-scoring responses as positive samples and the lowest-scoring ones as negative samples, while discarding pairs with insignificant score differences.

In addition to incorporating image-text pairs, we also include a significant proportion of pure text pairs, as elaborated in Section 5.4.1. It is noteworthy that when Direct Preference Optimization (DPO) is applied to highly capable foundation models, there is a propensity for overfitting. To counteract this issue, we adopt an early stopping strategy, which involves terminating the training process prior to the completion of a full epoch. This approach is designed to preserve the model’s generalization capabilities.

By following this multi-stage training strategy, we ensure that our model not only demonstrates proficiency in understanding and generating high-quality text but also aligns with human values and safety standards. This comprehensive approach to training allows us to strike a balance between model performance and ethical considerations, thereby producing a model that is both effective and responsible.

To assess the performance of our vision-language model, we maintain a diverse set of benchmarks, including MMMU (Yue et al., 2024a), MMMU-Pro (Yue et al., 2024b), ChartQA (Masry et al., 2022), DocVQA (Mathew et al., 2021), OCRBench (Liu et al., 2024b), AI2D (Kembhavi et al., 2016), MathVista (Lu et al., 2023), OlympiadBench (He et al., 2024a), MMLongBench-Doc (Ma et al., 2024), MEGA-Bench (Chen et al., 2024a) and an in-house benchmark. These benchmarks help evaluate the model’s abilities in various areas, including knowledge, visual reasoning, mathematics, science, long context handling, and user experience. We detail our evaluation configuration for each benchmark in Appendix D. As shown in Table 13, MiniMax-VL-01 achieves competitive performance across various vision-language tasks, demonstrating the following key strengths and limitations:

Original Table 13 — Table 13: **Performance of MiniMax-VL-01 on academic and in-house benchmarks.**

Table 13
Tasks	GPT-4o (11-20)	Claude-3.5- Sonnet (10-22)	Gemini-1.5- Pro (002)	Gemini-2.0- Flash (exp)	Qwen2-VL- 72B-Inst.	InternVL 2.5-78B	LLama- 3.2-90B	MiniMax- VL-01
Knowledge
MMMU ${}^{*}_{\emph{val+dev}}$	63.5	72.0	68.4	70.6	64.5	66.5	62.1	68.5
MMMU-Pro ${}^{*}_{\emph{full}}$	54.5	54.7	50.9	57.0	43.2	47.3	36.0	52.7
Visual Q&A
ChartQA ${}^{*}_{\emph{relaxed}}$	88.1	90.8	88.7	88.3	91.2	91.5	85.5	91.7
DocVQA∗	91.1	94.2	91.5	92.9	97.1	96.1	90.1	96.4
OCRBench	806	790	800	846	856	847	805	865
Mathematics & Sciences
AI2D∗	83.1	82.0	80.9	85.1	84.4	86.8	78.9	83.3
MathVista ${}^{*}_{\text{testmini}}$	62.1	65.4	70.6	73.1	69.6	68.4	57.3	68.6
$\text{OlympiadBench}_{\emph{full}}$	25.2	28.4	32.1	46.1	21.9	25.1	19.3	24.2
Long Context
M-LongDoc ${}_{\emph{acc}}$	41.4	31.4	26.2	31.4	11.6	19.7	13.9	32.5
Comprehensive
MEGA-Bench ${}_{\emph{macro}}$	49.4	51.4	45.9	53.9	46.8	45.3	19.9	47.4
User Experience
In-house Benchmark	62.3	47.0	49.2	72.1	40.6	34.8	13.6	56.6
∗ Evaluated following a 0-shot CoT setting.

Common Downstream Tasks. In standard vision-language downstream tasks, MiniMax-VL-01 exhibits performance on par with GPT-4o, particularly excelling in visual question answering. This strong performance is attributed to its extensive multi-stage training process, enabling the model to effectively understand and reason across visual and textual inputs. However, MiniMax-VL-01 still struggles with advanced mathematical reasoning tasks, as assessed by OlympiadBench (He et al., 2024a).

Long Context. We assess MiniMax-VL-01’s capability for long-context comprehension and retrieval using MMLongBench-Doc (Ma et al., 2024). The results show that our model outperforms most counterparts, except GPT-4o-11-20. Despite its strong performance overall, MiniMax-VL-01 demonstrates a noticeable gap in both single-page (acc: 47.3%) and cross-page (acc: 28.4%) subsets.

Comprehensive Benchmark. On the recently introduced MEGA-Bench (Chen et al., 2024a), a realistic and comprehensive evaluation suite, MiniMax-VL-01 shows competitive overall capabilities, surpassing existing open-source vision LLMs. While it excels in diverse sub-tasks such as knowledge and coding, the model faces challenges in more complex tasks, including planning and metric assessments.

In-house User Experience Benchmark. While academic benchmarks often focus on problem-solving, they frequently fail to capture the nuances of real-world user interactions with models. To bridge this gap, we develop an in-house benchmark comprising 90 diverse image-related tasks, each designed with tailored and challenging instructions. The images and instructions in the benchmark are strictly deduplicated to not overlap with the training set at any stage. Task relevance is manually verified, with a detailed checklist annotated for each sample to ensure precise evaluation. The final test set consists of 524 meticulously annotated samples in both Chinese and English, but Chinese is primarily used. We illustrate some samples in Appendix C. In a win-rate comparison against a top-leading vision-language model, our model outperforms all open-source models and approaches the performance of GPT-4o-11-20 with a narrow margin.

In this report, we present MiniMax-Text-01 and MiniMax-VL-01, two novel models developed entirely from the ground up. These models demonstrate top-tier performance across standard benchmarks, particularly excelling in long-context processing with the ability to handle context windows of up to 4 million tokens. Our research findings challenge the prevailing assumption that state-of-the-art language models must be built upon traditional attention mechanisms. By strategically integrating linear attention with optimized hardware utilization and carefully designing training recipes, we have successfully expanded the context window by an order of magnitude. This breakthrough not only enhances the efficiency and scalability of LLMs but also paves the way for future models to support even longer context windows and facilitate the development of more sophisticated AI agents. To promote collaboration and advancement in the field, we have made our model publicly available at https://github.com/MiniMax-AI. For general use and evaluation, we provide a Chatbot with online search capabilities (https://www.hailuo.ai/) and the online API (https://intl.minimaxi.com). We are committed to keeping this series open source and will release updates as we develop improved models.

While MiniMax-Text-01 and MiniMax-VL-01 show strong performance in general language and vision-language tasks, we acknowledge several limitations that necessitate further exploration:

1.
Long-Context Evaluation: Current evaluation datasets for long-context retrieval tasks are primarily designed for artificial or simplified scenarios, and the assessment of long-text reasoning capabilities remains limited in practical applications such as document analysis. We plan to enhance long-context retrieval in more realistic settings and expand the evaluation of long-context reasoning across a wider array of tasks.
2.
Model Architecture: The model currently retains a 1/8 component with vanilla softmax attention. We are investigating more efficient architectures that can eliminate softmax attention entirely, potentially enabling unlimited context windows without computational overhead.
3.
Complex Programming Tasks: The model’s performance on advanced programming tasks is to be improved, as the coding dataset in our pre-training stage is still limited at the moment. We are continuously improving training data selection and refining continue training procedures to address these limitations in the next model version.

References

Abdin et al. (2024) Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024.
Agarwal et al. (2024) Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018, 2024.
Ainslie et al. (2023) Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, 2023.
Anthropic (2024) Anthropic. Introducing claude 3.5 sonnet, 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet.Source
Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
Baevski and Auli (2018) Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853, 2018.
Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a.
Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022b.
Bai et al. (2024) Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204, 2024.
Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020.
Broder (1997) Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21–29. IEEE, 1997.
Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
Chen et al. (2024a) Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang, Kai Wang, Yubo Wang, Yuansheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, et al. MEGA-Bench: Scaling multimodal evaluation to over 500 real-world tasks. arXiv preprint arXiv:2410.10563, 2024a.
Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Chen et al. (2024b) Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. MindSearch: Mimicking human minds elicits deep ai searcher. arXiv preprint arXiv:2407.20183, 2024b.
Child et al. (2019) Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023.
Clark et al. (2022) Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference on Machine Learning (ICML), pages 4057–4086. PMLR, 2022.
Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, 2019.
Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Dao (2024) Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=mZn2Xyh9Ec.Source
Dao and Gu (2024) Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024.
Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022.
de Brébisson and Vincent (2016) Alexandre de Brébisson and Pascal Vincent. A cheap linear attention mechanism with fast lookups and fixed-size representations. arXiv preprint arXiv:1609.05866, 2016.
DeepSeek-AI (2024) DeepSeek-AI. DeepSeek-V3 technical report, 2024. URL https://arxiv.org/abs/2412.19437.Source
Dong et al. (2024) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107–1128, 2024.
Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.Source
Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368–2378, 2019.
Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022.
Gao et al. (2024) Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024.
Glorioso et al. (2024) Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b SSM hybrid model. arXiv preprint arXiv:2405.16712, 2024.
Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
Gu and Dao (2024) Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=tEYskw1VY2.Source
Gunasekar et al. (2023) Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023.
He et al. (2024a) Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. OlympiadBench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008, 2024a.
He et al. (2024b) Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, et al. Chinese simpleQA: A chinese factuality evaluation for large language models. arXiv preprint arXiv:2411.07140, 2024b.
Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=d7KBjmI3GmQ.Source
Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b. URL https://openreview.net/forum?id=7Bywt2mQsCe.Source
Henighan et al. (2020) Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020.
Hernandez et al. (2022) Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.10487, 2022.
Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 30016–30030, 2022.
Hsieh et al. (2024) Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024.
Hua et al. (2022) Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In International conference on machine learning, pages 9099–9117. PMLR, 2022.
Hurst et al. (2024) Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.
Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, 2017.
Kamradt (2023) G. Kamradt. Llmtest_needleinahaystack, 2023. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack.Source
Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Kembhavi et al. (2016) Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. Springer, 2016.
Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019.
Lee et al. (2022) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424–8445, 2022.
Lepikhin et al. (2021) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb.Source
Li et al. (2024a) Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2024a.
Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022.
Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023.
Li et al. (2024b) Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-Hard and BenchBuilder pipeline. arXiv preprint arXiv:2406.11939, 2024b.
Liu et al. (2024a) Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=WsRHpHH4s0.Source
Liu et al. (2023) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 21558–21572, 2023.
Liu et al. (2024b) Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. OCRBench: On the hidden mystery of OCR in large multimodal models. Science China Information Sciences, 67(12):220102, 2024b.
Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.Source
Lu et al. (2023) Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Ma et al. (2024) Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. MMLongBench-Doc: Benchmarking long-context document understanding with visualizations. arXiv preprint arXiv:2407.01523, 2024.
Masry et al. (2022) Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022.
Mathew et al. (2021) Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. DocVQA: A dataset for VQA on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021.
McCandlish et al. (2018) Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018.
Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, 2018.
Muennighoff et al. (2023) Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358–50376, 2023.
NVIDIA (2023) NVIDIA. Transformer engine, 2023. URL https://github.com/NVIDIA/TransformerEngine.Source
Parmar et al. (2024) Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Reuse, don’t retrain: A recipe for continued pretraining of language models. arXiv preprint arXiv:2407.07263, 2024.
Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data only. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=kM5eGcdCzq.Source
Penedo et al. (2024) Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=n6SCkn2QaG.Source
Qin et al. (2022a) Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7025–7041, 2022a.
Qin et al. (2022b) Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosFormer: Rethinking softmax in attention. In International Conference on Learning Representations, 2022b. URL https://openreview.net/forum?id=Bl8CQrx2Up4.Source
Qin et al. (2023a) Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Toeplitz neural network for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=IxmWsm4xrua.Source
Qin et al. (2023b) Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 33202–33221, 2023b.
Qin et al. (2024a) Zhen Qin, Yuxin Mao, Xuyang Shen, Dong Li, Jing Zhang, Yuchao Dai, and Yiran Zhong. You only scan once: Efficient multi-dimension sequential modeling with lightnet. arXiv preprint arXiv:2405.21022, 2024a.
Qin et al. (2024b) Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models. arXiv preprint arXiv:2401.04658, 2024b.
Qin et al. (2024c) Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Various lengths, constant speed: Efficient language modeling with lightning attention. In International conference on machine learning, pages 41517–41535. PMLR, 2024c.
Qin et al. (2024d) Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. HGRN2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904, 2024d.
Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446, 2021.
Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 53728–53741, 2023.
Rajbhandari et al. (2020) Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020.
Rein et al. (2024) David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98.Source
Ren et al. (2024) Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint arXiv:2406.07522, 2024.
Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021.
Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473, 2019.
Schuhmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Shaham et al. (2022) Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. SCROLLS: Standardized comparison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12007–12021, 2022.
Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Shen et al. (2024) Xuyang Shen, Dong Li, Ruitao Leng, Zhen Qin, Weigao Sun, and Yiran Zhong. Scaling laws for linear complexity language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16377–16426, 2024.
Shibata et al. (1999) Yusuxke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, Takeshi Shinohara, and Setsuo Arikawa. Byte pair encoding: A text compression scheme that accelerates pattern matching. Technical Report DOI-TR-161, Department of Informatics, Kyushu University, 1999.
Su et al. (2024) Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Sun et al. (2024) Weigao Sun, Zhen Qin, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong. Linear attention sequence parallelism. arXiv preprint arXiv:2404.02882, 2024.
Tanzer et al. (2024) Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. A benchmark for learning to translate a new language from one grammar book. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=tbVWug9f2h.Source
Team et al. (2024a) Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024a.
Team et al. (2024b) Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. Jamba-1.5: Hybrid Transformer-Mamba models at scale. arXiv preprint arXiv:2408.12570, 2024b.
Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Vodrahalli et al. (2024) Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640, 2024.
Wang et al. (2024a) Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. DeepNet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Wang et al. (2019) Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810–1822, 2019.
Wang et al. (2024b) Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024b. URL https://openreview.net/forum?id=y10DM6R2r3.Source
Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.
Wei et al. (2024) Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024.
Yang et al. (2024) An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Yu et al. (2022) Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=Ee277P3AYC.Source
Yue et al. (2024a) Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2024a.
Yue et al. (2024b) Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024b.
Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big Bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297, 2020.
Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019.
Zhou et al. (2023) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.

The contributors to the report are listed in alphabetical order as follows:

Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, Zijia Wu

We show our model’s performance under real-world user interactions. To protect the privacy of our users, all user requests shown below are written by our human evaluators, imitating the way users interact with the model, if not from open-source benchmarks.

Our prompt for applying MTOB follows that of Gemini-1.5 (Team et al., 2024a), detailed as follows.

Graphic from the paper; its text follows

As shown, MiniMax-Text-01 can learn from the given grammar book, word list and parallel sentences, where a under-performing model’s response has semantic errors.

We present an example of how our MR-NIAH samples are designed, where the model is asked to precisley retrieve one of the history interactions with the user from up to 1889 history interactions for the English benchmark and 2053 history interactions for the Chinese benchmark. These history queries were created by Mandarin-speaking human evaluators with help from a previous version of our model, where each query is translated into English, and we obtain the assistants’ responses from this model. We combine these interactions as history dialogues, and insert each needle conversation into corresponding positions detailed in Section 5.7.2.1.

We will release the data and code of this benchmark to the community.

When dumped a PDF paper and instructed to add figures from the paper, Hailuo AI is able to summarize or analyze the given long text and respond with key figures cited.

We can see from the following case that, with search tools, Hailuo AI is able to identify the error in the user request, and claim that NaVi did not beat FaZe, being less likely to be misled by user queries.

For evaluating models with context-length and image-number limitations that exceed requirements of MMLongBench-Doc, we adopt an image concatenation approach suggested by the original repository $^{6}$ , resulting in the concatenation of all images extracted from a single PDF input into 5 images for the open-source models evaluated and 10 for Claude-3.5-Sonnet-1022. For evaluating other commercial models and MiniMax-Text-01, we use the default configuration which sets the maximum number of image pages to 120 and resolution to 144.

MEGA-Bench is a comprehensive multimodal benchmark that spans 7 input formats, 6 output formats, 10 different types of skills, and varying forms of visual inputs, including images and videos. Each request may consider multiple images, consisting of visual task description, request-response demonstration and videos. For video inputs, the benchmark slices each video into multiple frames. The number of frames and the resulting number of total input images are limited to the model’s context length and image constraints. We follow the general principles of the original repository $^{7}$ when deciding our evaluation configurations, as detailed in Table 14.

Original Table 14 — Table 14: **Configuration of different models for MEGA-Bench.**

Table 14
Model/Configuration.	MAX_NUM_IMAGE	TOTAL_DEMO_VIDEO_FRAMES
GPT-4o-2024-1120	64	8
Claude-3.5-Sonnet-1022	64	8
Gemini-1.5-Pro-002	128	16
Gemini-2.0-Flash-exp	128	16
InternVL2.5-78B	24	2
Qwen2-VL-72B-Instruct	10	1
LLama-3.2-90B	10	1
MiniMax-VL-01	128	16

We note that rule-based methods may misjudge cases where the correct answer has mulitple forms (e.g. U.S. vs. United States), we adopt GPT-4o (specifically GPT-4o-2024-05-13) as the judge model if the rule-based method fails for MMMU and DocVQA evaluation.

MiniMax-01: Scaling Foundation Models with Lightning Attention

Abstract

References

Meow summary

The bottleneck

The trick: lightning attention

Why it scales

What it proves—and doesn’t

Make notes of your own

Abstract

1 Introduction

2 Model Architecture

2.1 Mixture of Experts

2.2 Linear Attention

2.2.1 Lightning Attention

2.2.2 Effectiveness of Lightning Attention

2.2.2.1 Experimental Setup

2.2.2.2 Scaling Laws

2.2.2.3 Performance on Downstream Task.

2.2.2.4 Speed.

2.2.3 Hybrid Architecture

2.2.4 Discussion

2.3 Module Ablations in MoE

2.4 Model Spec

3 Computation Optimization

3.1 MoE Optimization

3.2 Long Context Optimization

3.2.1 Varlen Ring Attention

3.2.2 Improved Linear Attention Sequence Parallelism

3.3 Lightning Attention Inference Optimization

3.3.1 Batched Kernel Fusion

3.3.2 Separated Prefill and Decoding Execution

3.3.3 Multi-level Padding

3.3.4 StridedBatchedMatmul Extension

4 Pre-Training

4.1 Data

4.1.1 Pre-training Corpus

4.1.2 Tokenization

4.1.3 Data Experiment

4.1.3.1 Paradigm

4.1.3.2 Effect of Repetition

4.2 Training Strategy

5 Post-training

5.1 Prompt Collection

5.2 Reward Model

5.3 Supervised Fine-Tuning

5.4 Reinforcement Learning

5.4.1 Offline Reinforcement Learning

5.4.2 Online Reinforcement Learning

5.5 Safety Alignment

5.5.1 Training Data Construction

5.5.2 Response Generation with Harmless Reward Model

5.6 Training Methodology with Long-Context Adaptation

5.7 Academic Benchmarks

5.7.1 Core Benchmarks

5.7.2 Long Benchmarks

5.7.2.1 Long-Context Retrieval

5.7.2.2 Long-Context Understanding

5.7.2.3 Long In-Context Learning

5.8 User-in-the-loop

5.8.1 In-House Evaluations

5.8.2 Search in Hailuo AI

6 Vision-language Model

6.1 Multimodal Data

6.1.1 Caption Data

6.1.2 Description Data

6.1.3 Instruction Data

6.1.4 Data Distribution

6.2 Architecture

6.2.1 Overall Architecture

6.2.2 Vision Encoder

6.3 Training Recipes

6.4 Benchmarks

7 Conclusion and Future work

References

Appendix A Contributors

Appendix B MiniMax-Text-01 Case Demonstrations

B.1 Learning A ‘New’ Language From Long Context

B.2 Memorizing Long History Dialogues

B.3 Summarizing Long Papers with Figures

B.4 Translating Long Papers

B.5 Lyrics Writing

B.6 Story Writing

B.7 Poetry Composition

B.8 Safety

B.9 Search

Appendix C Vision In-house User Experience Benchmark Examples

C.1 Nature

C.2 Daily Life