Reading + AI
LibrarySign in
Open the original paper on arXiv (new tab)

LLaDA2.1: Speeding Up Text Diffusion via Token Editing

Tiwei Bie, Maosong Cao, Xiang Cao, Bingsen Chen, Fuyuan Chen, Kun Chen, Lun Du, Haibo Feng
and 36 more authorsMingliang Gong, Zhuocheng Gong, Yanmei Gu, Jian Guan, Hongliang He, Zenan Huang, Juyong Jiang, Zhonghui Jiang, Zhenzhong Lan, Jianguo Li, Zehuan Li, Huabin Liu, Lin Liu, Guoshan Lu, Yuan Lu, Yuxin Ma, Zhenxuan Pan, Kaida Qiu, Yuji Ren, Jianfeng Tan, Yiding Tian, Zian Wang, Tao Wu, Yipeng Xing, Wentao Ye, Liangyu Zha, Tianze Zhang, Xiaolu Zhang, Da Zheng, Hao Zhong, Wanli Zhong, Jun Zhou, Junlin Zhou, Liwang Zhu, Muzhi Zhu, Yihong Zhuang

arXiv:2602.08676v3 · This workspace stays on this version. Check arXiv for newer versions

Affiliations and author notes
  • Daozhuo Feng
  • Kaiyuan Guan
  • Chengxi Li
  • Xingyu Mou
  • Lanning Wei
  • Junbo Zhao
  • Ant Group, Zhejiang University, Westlake University, Southern University of Science and Technology
Some source details need review 1 to check

Original formulas, table values and figures are preserved. Comparison can miss differences; use the original PDF to check details.

Reading view checked: 242 math expressions, 0 rendering failures; 7 of 7 images available.

Compared 50 of 50 source passages against 11 of 11 PDF pages.

PDF page 4 reads “ASystem (Ling Team et al., 2025),” while HTML block S4_SS1_SSS0_Px2_p1_1 reads “ASystem ([Ling Team and others, 2025](#bib.bib49))”; the original PDF contradicts the HTML. Check PDF page 4

7 of 7 figure, table and graphic images captured. Open original PDF

Abstract

While LLaDA2.0 showcased the scaling potential of 100B-level block-diffusion models and their inherent parallelization, the delicate equilibrium between decoding speed and generation quality has remained an elusive frontier. Today, we unveil LLaDA2.1, a paradigm shift designed to transcend this trade-off. By seamlessly weaving Token-to-Token (T2T) editing into the conventional Mask-to-Token (M2T) scheme, we introduce a joint, configurable threshold-decoding scheme. This structural innovation gives rise to two distinct personas: the Speedy Mode (S Mode), which audaciously lowers the M2T threshold to bypass traditional constraints while relying on T2T to refine the output; and the Quality Mode (Q Mode), which leans into conservative thresholds to secure superior benchmark performances with manageable efficiency degrade. Furthering this evolution, underpinned by an expansive context window, we implement the first large-scale Reinforcement Learning (RL) framework specifically tailored for dLLMs, anchored by specialized techniques for stable gradient estimation. This alignment not only sharpens reasoning precision but also elevates instruction-following fidelity, bridging the chasm between diffusion dynamics and complex human intent. We culminate this work by releasing LLaDA2.1-Mini (16B) and LLaDA2.1-Flash (100B). Across 33 rigorous benchmarks, LLaDA2.1 delivers strong task performance and lightning-fast decoding speed. Despite its 100B volume, on coding tasks it attains an astounding 892 TPS on HumanEval+, 801 TPS on BigCodeBench, and 663 TPS on LiveCodeBench.

Figure 1
Figure 1: Aggressive parallel drafting, backed by retroactive correction, accelerates inference.

Discrete diffusion Large Language Models (dLLMs) have emerged as a compelling alternative to autoregressive generation, offering the potential for non-monotonic reasoning and parallel decoding. However, the standard absorbing-state framework—which enforces a rigid, monotonic transition from [MASK] to fixed tokens—faces inherent limitations in fidelity. As highlighted by Kang et al. (2025), the independent nature of parallel decoding often amplifies token-level inconsistencies. While recent studies have attempted to mitigate this via confidence-based remasking (Wang et al., 2025b) or by employing external guide models (Lee et al., 2025). To bridge the gap between efficient parallel generation and high-fidelity reasoning, we align with the direction of generalizing discrete diffusion beyond absorbing states (Rütte et al., 2025) and propose a comprehensive framework for Editable State Evolution.

Unlike prior work such as Song et al. (2025), we first design a novel Error-Correcting Editable decoding strategy, which introduces a dynamic paradigm controlled by dual probability thresholds. This paradigm encompasses two types of operations: direct decoding from mask to token, and editing from one token to another. This strategy enables the model to directly refine its own outputs during the generation process, thereby effectively addressing the local inconsistencies commonly encountered in parallel decoding. To cultivate this editing capability, our CPT and SFT phases expose the model to both masked positions and stochastic noise, incentivizing it to not only generate new content but also identify and rectify existing errors.

Crucially, this architecture transforms the rigid trade-off between latency and fidelity into a flexible, user-configurable continuum. By allowing the model to retroactively correct errors, we can aggressively lower the confidence threshold for the initial Mask-to-Token (M2T) phase without collapsing the generation quality. This insight gives rise to two distinct operating personas: a Speedy Mode (S Mode), which prioritizes high-throughput generation by accepting lower-confidence tokens and relying on subsequent Token-to-Token (T2T) passes for rectification; and a Quality Mode (Q Mode), which adheres to conservative thresholds to maximize reasoning rigor. This duality demonstrates that editability is not merely a mechanism for error repair, but a fundamental lever for accelerating parallel decoding.

To further elevate the model’s capabilities, we integrate a Reinforcement Learning (RL) stage. While recent works such as SPG (Wang et al., 2025a), TraceRL (Wang et al., 2025c) and ESPO (Ou et al., 2025) have demonstrated the potential of RL in improving dLLMs, applying policy gradients to block-autoregressive models remains challenging due to the intractability of sequence log-likelihoods. We circumvent this by adopting an ELBO-based Block-level Policy Optimization (EBPO) framework tailored for our editable setting.

Notice that LLaDA2.1 extends its previous version (LLaDA2.0) by prioritizing decoding versatility over mere parameter scaling or benchmark peaking. By keeping the model size constant and minimal change of training data, we prove that our novel editing scheme enables lightning-fast execution with minimal overhead. This work serves as a proof-of-concept for a new dLLM paradigm that balances high-quality generation with extreme operational efficiency.

During LLM decoding, Exposure Bias—where errors compound as the model conditions on its own imperfect predictions—is inevitable. This phenomenon is particularly severe in dLLMs due to their parallel generation nature. We observe that once such decoding errors occur, dLLMs tend to become increasingly conservative in subsequent steps, significantly slowing down the generation process. In contrast, autoregressive models exhibit lower exposure bias and can self-correct through extended chain-of-thought reasoning. To address this challenge, we introduce an editing operation into the decoding process, enabling the model to retrospectively correct errors introduced during parallel generation, thereby achieving a much better balance between generation speed and quality.

Specifically, we extend standard discrete diffusion to support it. Unlike conventional absorbing-state models that enforce a rigid monotonic transition from [MASK] to fixed tokens, our framework introduces a dynamic “Draft-and-Edit” paradigm controlled by dual probability thresholds. We formalize the state evolution by defining two active update sets at timestep tt: the Unmasking Set Γt\Gamma_{t} and the Editing Set Δt\Delta_{t}.

We formalize the state evolution by defining two active update sets at timestep tt: the Unmasking Set Γt\Gamma_{t} and the Editing Set Δt\Delta_{t}. Let vti=arg⁡max⁡vpθ(v∣xt)v_{t}^{i}=\arg\max_{v}p_{\theta}(v|\bm{x}_{t}) be the top-candidate. The update indices are identified as:

Γt={i∣xti=[MASK] and pθ(vti∣xt)>τmask},\begin{aligned}\displaystyle\Gamma_{t} & \displaystyle=\left\{i\mid x_{t}^{i}=\text{[MASK]}\text{ and }p_{\theta}(v_{t}^{i}|\bm{x}_{t})>\tau_{\text{mask}}\right\},\end{aligned}
(1)
Δt={i∣xti≠vti and pθ(vti∣xt)>τedit},\begin{aligned}\displaystyle\Delta_{t} & \displaystyle=\left\{i\mid x_{t}^{i}\neq v_{t}^{i}\text{ and }p_{\theta}(v_{t}^{i}|\bm{x}_{t})>\tau_{\text{edit}}\right\},\end{aligned}
(2)

with τmask,τedit∈[0,1]\tau_{\text{mask}},\tau_{\text{edit}}\in[0,1] being the confidence thresholds configuring the decoding dynamics. The transition operator then applies the updates strictly on the union of these sets:

xt−1i={vtiif i∈Γt∪Δt,xtiotherwise.x_{t-1}^{i}=\begin{cases}v_{t}^{i}&\text{if }i\in\Gamma_{t}\cup\Delta_{t},\\ x_{t}^{i}&\text{otherwise}.\end{cases}
(3)

Figure 2
Figure 2: Overview of training & inference framework of LLaDA2.1

To align the model with the “Draft-and-Edit” inference paradigm and mitigate the Exposure Bias inherent in standard mask-based training, we employ a unified Mixture of M2T and T2T objective. This objective is applied throughout both the Continual Pre-Training (CPT) and Supervised Finetuning (SFT) stages.

This dual-stream training objective enables the model to develop two complementary capabilities fundamental to our framework:

  • •

    Drafting Stream (Mask-to-Token): The model learns to predict the correct token at each masked position to generate initial content, establishing the foundational drafting capability.

  • •

    Editing Stream (Token-to-Token): The model learns to recover original tokens from random noise perturbations (rectifying errors), equipping it with the ability to identify and rewrite artifacts.

By consistently applying this dual-stream supervision from CPT through SFT, we ensure that LLaDA2.1 is fundamentally conditioned to function as both a fast drafter and a precise editor within a single parameter space. Additionally, we employ a Multi-turn Forward (MTF) data augmentation technique, by exposing the model to a wider variety of editing scenarios, enhance the model’s editing capabilities.

The application of policy gradient methods to diffusion models faces a fundamental hurdle: the intractability of the sequence-level log-likelihood, log⁡πθ(x)\log\pi_{\theta}(\bm{x}), which is essential for computing policy updates. While prior works have explored various approximations, they have historically struggled with high variance and prohibitive computational costs, limiting RL to small-scale experiments (Wang et al., 2025c, Ou et al., 2025, Wang et al., 2025a). We overcome this bottleneck by synthesizing ELBO-based Block-level Policy Optimization (EBPO) with robust infrastructure optimizations. By utilizing the Evidence Lower Bound (ELBO) as a principled proxy for exact likelihood and implementing Vectorized Likelihood Estimation (Arriola et al., 2025) to parallelize bound computation, we achieve orders-of-magnitude acceleration. This integration allows us to scale dLLMs RL to unprecedented context lengths and training magnitudes, establishing a stable and efficient pipeline for post-training.

Formally, we maximize a clipped surrogate objective, where the advantage is weighted by the probability ratio ρ\rho:

JEBPO(θ)=Ex,y∼πθold[min⁡(ρ(y∣x)A^,clip(ρ(y∣x),1−ϵlow,1+ϵhigh)A^)],\mathcal{J}_{\text{EBPO}}(\theta)=\mathbb{E}_{\bm{x},\bm{y}\sim\pi_{\theta_{\text{old}}}}\left[\min\left(\rho(\bm{y}|\bm{x})\hat{A},\text{clip}(\rho(\bm{y}|\bm{x}),1-\epsilon_{\text{low}},1+\epsilon_{\text{high}})\hat{A}\right)\right],
(4)

where A^\hat{A} is an estimator of the advantage function at timestep tt, quantifying the relative improvement of the chosen action over the average expectation under the current policy. For a set of discretized timesteps {tn}n=1N\{t_{n}\}_{n=1}^{N} and weights {wn}\{w_{n}\}, we construct a composite input zn=ytn⊕y0\bm{z}_{n}=\bm{y}_{t_{n}}\oplus\bm{y}_{0} to compute all block-conditional probabilities in parallel:

log⁡ρ(y∣x)≈∑n=1Nwn∑b=1B(log⁡pθ(yb∣zn,x;M)−log⁡pθold(yb∣zn,x;M)).\log\rho(\bm{y}|\bm{x})\approx\sum_{n=1}^{N}w_{n}\sum_{b=1}^{B}\left(\log p_{\theta}(\bm{y}^{b}\mid\bm{z}_{n},\bm{x};\mathcal{M})-\log p_{\theta_{\text{old}}}(\bm{y}^{b}\mid\bm{z}_{n},\bm{x};\mathcal{M})\right).
(5)

Here, M\mathcal{M} denotes a Block-Causal Mask ensuring the bb-th block attends only to valid history. By aggregating block-level contributions (∑b=1B\sum_{b=1}^{B}) within a single forward pass per timestep nn, we establish a computationally tractable pipeline for scaling reinforcement learning to long-context diffusion generation.

For both continued pre-training (CPT) and supervised fine-tuning (SFT), we adopt the same training infrastructure as LLaDA2.0 (Bie et al., 2025), leveraging dFactory (InclusionAI, 2025), which provides efficient training recipes specifically designed for dLLMs, except that we introduce a dedicated optimized implementation for the multi-turn forward (MTF) stage.

To enable effective policy optimization for dLLMs, we extend the AReaL framework (Fu et al., 2025, Mei et al., 2025) by developing specialized likelihood estimation and advantage estimation protocols that leverage diffusion sampling, explicitly supporting both T2T and M2T modes. This workflow is powered by ASystem (Ling Team and others, 2025) for distributed orchestration and utilizes a customized version of SGLang (Ant Group Team and SGLang Team,) as the dedicated rollout engine.

We use a customized version of SGLang (Ant Group Team and SGLang Team,) for inference. To further accelerate the inference speed, we integrate Alpha-MoE (Aleph-Alpha,), a MoE megakernel that combines the two FusedMoE computations into one kernel, and adopt per-block FP8 quantization to balance the inference speed and model accuracy. To accelerate inference on long-context sequences, we adopt block-wise causal masked attention, allowing the KV cache for the entire long context to be computed in a single forward pass. We further enable radix caching and batching support for block diffusion LLMs in SGLang.

In the inference stage, we adopt a decoding algorithm that combines Threshold Decoding (Ma et al., 2025) with an explicit editing mechanism. In the basic setting, decoding and editing are performed within a single block: tokens are generated under a threshold-based constraint, and local edits are applied to revise intermediate outputs before the block is finalized.

Beyond single-block editing, we further introduce a Multiple Block Editing (MBE) mechanism. MBE allows the model to revisit and revise previously generated blocks based on the content of newly decoded blocks.

To comprehensively evaluate the quality of instruction-tuned models, we employ a diverse suite of benchmarks categorized into five dimensions:

We report the comparative scores and TPF (tokens per forward) of LLaDA2.1-flash and LLaDA2.1-mini against other models in Tables 1 and 2, respectively. From the results, we observe that LLaDA2.1’s scores under S Mode decrease compared to LLaDA2.0, but a substantial improvement in TPF is achieved. While under Q Mode, LLaDA2.1 surpasses the results of LLaDA2.0 on both mini and flash model.

In Table 3, we focus on showcasing the speed performance of LLaDA2.1 in S Mode. It can be observed that LLaDA2.1 exhibits significant speed variations across different domains, being highest in the code domain and lowest in instruction following. Specifically, after quantization, LLaDA2.1-flash achieves a peak TPS of 891.74 on HumanEval+, while LLaDA2.1-mini reaches 1586.93 in peak TPS, demonstrating significant speed advantages.

Figure 3
Figure 3: Throughput (TPS) comparison on nine benchmarks, consistent with the evaluation settings in Table 3, for LLaDA2.1 variants against LLaDA2.0, Ling, and Qwen3 across the mini (left) and flash (right) series.

As shown in Table 4, under the same S Mode setting, Multi-Block Editing (MBE) yields consistent performance improvements across benchmarks for both Flash and Mini variants, at the cost of a modest reduction in throughput. The gains are particularly evident on reasoning and coding tasks, indicating that iterative cross-block refinement effectively corrects local errors and improves global consistency without substantially compromising decoding efficiency.

Figure 3 further illustrates the throughput (in terms of token per sec) comparison of LLaDA 2.1 variants against LLaDA 2.0, Ling, and Qwen-3 across 5 different benchmark domains as shown in Table 3. This comparison spotlights LLaDA-2.1 (S Mode)’s striking speed advantage: it achieves dramatically faster inference while sacrificing only a negligible sliver of output quality.

Table 1
BenchmarkQwen3-30B- A3B-Inst-2507 (Score)Ling-flash-2.0 (Score)LLaDA2.0-flash (Score ∣\mid TPF)LLaDA2.1-flash (S Mode) (Score ∣\mid TPF)LLaDA2.1-flash (Q Mode) (Score ∣\mid TPF)
Average73.0971.5272.43 ∣\mid 3.0872.34 ∣\mid 5.9373.54 ∣\mid 3.64
Knowledge
GPQA54.1469.1662.31 ∣\mid 3.2966.67 ∣\mid 3.9567.30 ∣\mid 2.37
MMLU-Pro74.2177.5574.79 ∣\mid 2.3675.31 ∣\mid 4.4376.59 ∣\mid 2.62
C-EVAL88.1287.5485.21 ∣\mid 1.9086.93 ∣\mid 2.7186.71 ∣\mid 1.75
PHYBench29.8427.6730.06 ∣\mid 2.7026.04 ∣\mid 4.1028.23 ∣\mid 2.66
TriviaQA65.6169.7666.88 ∣\mid 1.9472.55 ∣\mid 4.3072.93 ∣\mid 2.92
Reasoning
BIG-Bench Hard85.5489.3686.75 ∣\mid 2.6687.82 ∣\mid 5.6188.69 ∣\mid 3.28
BIG-Bench Extra Hard37.8023.2427.86 ∣\mid 4.6033.51 ∣\mid 5.0435.77 ∣\mid 3.17
bbh-zh86.1875.0987.52 ∣\mid 3.2182.55 ∣\mid 5.7886.23 ∣\mid 3.77
MuSR79.1582.7280.48 ∣\mid 1.7080.10 ∣\mid 2.9079.84 ∣\mid 1.85
ZebraLogic90.9787.6082.30 ∣\mid 2.7484.20 ∣\mid 5.8088.90 ∣\mid 3.26
PrOntoQA97.1297.8896.50 ∣\mid 2.6495.00 ∣\mid 9.2397.00 ∣\mid 5.73
PIQA91.5791.9592.76 ∣\mid 1.4392.44 ∣\mid 2.3892.17 ∣\mid 1.44
OCNLI71.5965.3671.63 ∣\mid 1.0972.17 ∣\mid 1.8372.75 ∣\mid 1.32
HellaSwag86.3181.5984.97 ∣\mid 1.2685.60 ∣\mid 2.3185.31 ∣\mid 1.51
KOR-Bench69.2069.4463.04 ∣\mid 3.4462.80 ∣\mid 4.9765.12 ∣\mid 2.77
DROP87.5788.3287.90 ∣\mid 2.2687.55 ∣\mid 5.4087.86 ∣\mid 2.53
SQuAD 2.089.5181.3290.00 ∣\mid 3.1090.65 ∣\mid 5.0190.80 ∣\mid 3.90
Coding
LiveCodeBench46.4252.4842.51 ∣\mid 4.2344.05 ∣\mid 6.4845.37 ∣\mid 3.80
CRUXEval-O86.7582.7585.12 ∣\mid 3.2185.25 ∣\mid 6.5487.50 ∣\mid 3.80
MBPP+78.2180.8979.37 ∣\mid 4.0276.72 ∣\mid 10.4377.25 ∣\mid 5.96
HumanEval+87.8887.5888.41 ∣\mid 6.4589.63 ∣\mid 13.8189.63 ∣\mid 9.18
MultiPL-E70.6765.7674.87 ∣\mid 3.1470.89 ∣\mid 7.7773.34 ∣\mid 4.33
BigCodeBench-Full41.4940.7041.58 ∣\mid 3.3337.11 ∣\mid 8.5139.21 ∣\mid 4.70
BIRD-SQL47.7547.4945.76 ∣\mid 2.1642.18 ∣\mid 5.0944.04 ∣\mid 2.95
Spider81.7980.5882.49 ∣\mid 4.4279.18 ∣\mid 8.7481.04 ∣\mid 5.70
Math
AIME 202561.8855.8960.00 ∣\mid 4.5763.33 ∣\mid 5.3663.33 ∣\mid 3.46
OlympiadBench77.5976.1974.07 ∣\mid 3.7075.85 ∣\mid 6.4676.59 ∣\mid 3.81
GSM-Plus89.4189.7189.74 ∣\mid 2.6889.23 ∣\mid 7.1489.69 ∣\mid 3.83
CMATH96.5896.5296.90 ∣\mid 2.1796.54 ∣\mid 4.8496.63 ∣\mid 2.65
Omni-MATH54.0053.0050.30 ∣\mid 3.3952.30 ∣\mid 6.0154.10 ∣\mid 3.50
Agent & Alignment
IFEval-strict-prompt83.7381.1582.62 ∣\mid 1.4783.36 ∣\mid 2.2483.55 ∣\mid 1.41
BFCL v373.4167.6974.94 ∣\mid 4.8774.86 ∣\mid 9.2475.61 ∣\mid 6.76
Nexus FC49.9336.2550.45 ∣\mid 5.5344.83 ∣\mid 11.2947.65 ∣\mid 7.38
Table 1: Benchmark Performance of LLaDA2.1-flash, comparing with several baseline models. For diffusion language model, we report its scores across each benchmark along with its TPF (tokens per forward); for AR model, we report its scores only, as its TPF is inherently equal to 1.
View original tableOriginal Table 1
Table 2
BenchmarkQwen3-8B (no_think) (Score)Ling-mini-2.0 (Score)LLaDA2.0-mini (Score ∣\mid TPF)LLaDA2.1-mini (S Mode) (Score ∣\mid TPF)LLaDA2.1-mini (Q Mode) (Score ∣\mid TPF)
Average61.5964.7263.39 ∣\mid 2.6062.07 ∣\mid 5.3463.90 ∣\mid 3.12
Knowledge
GPQA48.0159.4147.76 ∣\mid 2.7348.36 ∣\mid 3.6253.28 ∣\mid 2.12
MMLU-Pro65.8367.1864.27 ∣\mid 2.1563.42 ∣\mid 4.2264.84 ∣\mid 2.41
C-EVAL80.6082.1781.80 ∣\mid 1.7878.40 ∣\mid 3.3978.59 ∣\mid 1.91
PHYBench9.7614.5911.70 ∣\mid 2.4812.75 ∣\mid 4.4113.05 ∣\mid 2.52
TriviaQA52.5155.6351.33 ∣\mid 1.5453.33 ∣\mid 3.2154.24 ∣\mid 2.02
Reasoning
BIG-Bench Hard79.4883.7078.21 ∣\mid 2.3678.42 ∣\mid 5.0280.58 ∣\mid 2.86
BIG-Bench Extra Hard18.2714.8116.47 ∣\mid 2.0315.30 ∣\mid 3.1915.78 ∣\mid 1.66
bbh-zh80.0966.1175.75 ∣\mid 2.7767.65 ∣\mid 3.8970.40 ∣\mid 2.35
MuSR70.0271.3671.48 ∣\mid 1.4570.43 ∣\mid 2.4871.89 ∣\mid 1.56
ZebraLogic37.4879.8564.20 ∣\mid 2.3068.50 ∣\mid 5.3877.10 ∣\mid 2.93
PrOntoQA93.1296.0686.00 ∣\mid 2.3687.50 ∣\mid 4.8684.50 ∣\mid 2.73
PIQA88.3087.5486.51 ∣\mid 1.4584.87 ∣\mid 2.5986.89 ∣\mid 1.45
OCNLI61.4960.1764.51 ∣\mid 4.0661.02 ∣\mid 1.7861.59 ∣\mid 1.23
HellaSwag79.5669.0279.01 ∣\mid 1.5075.71 ∣\mid 2.3976.19 ∣\mid 1.49
KOR-Bench54.9663.2049.92 ∣\mid 2.4546.64 ∣\mid 4.2848.00 ∣\mid 2.35
DROP84.5678.8081.91 ∣\mid 2.0281.55 ∣\mid 5.8482.37 ∣\mid 2.87
SQuAD 2.085.2175.5686.50 ∣\mid 2.4784.51 ∣\mid 4.3385.13 ∣\mid 3.09
Coding
LiveCodeBench26.7642.2931.83 ∣\mid 3.3428.85 ∣\mid 6.4230.40 ∣\mid 3.63
CRUXEval-O74.0676.1271.62 ∣\mid 2.7870.62 ∣\mid 5.8573.75 ∣\mid 3.35
MBPP+72.6977.2578.24 ∣\mid 3.4373.28 ∣\mid 10.5974.07 ∣\mid 6.30
HumanEval+79.5080.0381.40 ∣\mid 5.1680.49 ∣\mid 12.3282.93 ∣\mid 7.77
MultiPL-E61.7067.0967.46 ∣\mid 2.7864.16 ∣\mid 7.2367.17 ∣\mid 4.01
BigCodeBench-Full36.0535.0032.89 ∣\mid 2.8730.18 ∣\mid 7.3334.39 ∣\mid 4.09
BIRD-SQL36.1139.6739.34 ∣\mid 1.9637.32 ∣\mid 4.4838.40 ∣\mid 2.42
Spider72.8076.4376.76 ∣\mid 3.9375.78 ∣\mid 7.9877.55 ∣\mid 5.48
Math
AIME 202522.0847.6636.67 ∣\mid 2.4136.67 ∣\mid 6.3443.33 ∣\mid 3.29
OlympiadBench55.3372.3067.70 ∣\mid 2.6364.30 ∣\mid 7.0866.67 ∣\mid 3.99
GSM-Plus85.5687.1886.50 ∣\mid 2.4185.88 ∣\mid 6.8286.55 ∣\mid 3.69
CMATH95.4296.4095.72 ∣\mid 1.9895.63 ∣\mid 4.9494.99 ∣\mid 2.56
Omni-MATH33.2048.8041.70 ∣\mid 2.5741.70 ∣\mid 6.4143.60 ∣\mid 3.56
Agent & Alignment
IFEval-strict-prompt84.2976.1680.78 ∣\mid 1.2481.33 ∣\mid 1.8383.18 ∣\mid 1.25
BFCL v370.1253.7570.72 ∣\mid 4.2672.06 ∣\mid 7.3973.61 ∣\mid 5.14
Nexus FC37.7134.3835.18 ∣\mid 4.0631.59 ∣\mid 8.2733.69 ∣\mid 4.91
Table 2: Benchmark Performance of LLaDA2.0-mini, comparing with several baseline models. For diffusion language model, we report its scores across each benchmark along with its TPF (tokens per forward); for AR model, we report its scores only, as its TPF is inherently equal to 1.
View original tableOriginal Table 2
Table 3
CategoryBenchmarkLLaDA2.1-flashLLaDA2.1-mini
w/o Quant TPSw/ Quant TPS ∣\mid Δ\DeltaScorew/o Quant TPSw/ Quant TPS ∣\mid Δ\DeltaScore
CodingHumanEval+746.66891.74 -3.041496.671586.93 -0.61
MBPP+639.47761.38 -1.851286.961303.96 +1.85
CRUXEval-O550.09645.72 -0.24980.821063.94 -1.00
BigCodeBench-Full691.14801.48 +1.061220.401307.45 -0.09
LiveCodeBench571.60663.39 -1.761015.821102.92 +1.98
MathGSM-Plus574.65667.07 -0.031080.511186.18 -0.30
KnowledgeGPQA-Diamond416.92477.79 -0.64724.30784.62 -1.64
Instruction FollowingIFEval219.37248.25 +1.48338.58365.52 -1.29
ReasoningPrOntoQA770.88912.16 -1.00880.19938.93 -1.50
Table 3: Throughput (TPS) and relative score changes of Flash and Mini variants across benchmarks. For each model family, the w/o Quant setting serves as the baseline. Cells under w/ Quant are vertically split into TPS ∣\mid Δ\DeltaScore.
View original tableOriginal Table 3
Table 4
CategoryBenchmarkLLaDA2.1-flashLLaDA2.1-mini
w/o MBE Score TPFw/ MBE Score TPFw/o MBE Score TPFw/ MBE Score TPF
KnowledgeMMLU-Pro75.31 4.4375.90 3.8863.42 4.2263.10 3.66
TriviaQA72.55 4.3072.45 4.2853.33 3.2153.41 3.14
Reasoningbbh-zh82.55 5.7883.21 4.8567.65 3.8967.94 3.41
ZebraLogic84.20 5.8088.20 5.0368.50 5.3870.00 4.62
CodingLiveCodeBench44.05 6.4846.48 5.6228.85 6.4229.74 5.44
CRUXEval-O85.25 6.5487.00 5.6270.62 5.8570.62 5.02
BigCodeBench-Full37.11 8.5139.30 7.0030.18 7.3330.70 6.05
Spider79.18 8.7480.58 8.3375.78 7.9876.67 7.59
MathAIME 202563.33 5.3670.00 4.7136.67 6.3436.67 5.25
Agent & AlignmentIFEval-strict-prompt83.36 2.2483.55 2.1181.33 1.8383.55 1.70
Average–70.69 5.8272.67 5.1457.63 5.2558.24 4.59
Table 4: Performance comparison of LLaDA2.1-flash and Mini variants with and without Multi-Block Editing (MBE) across benchmarks. Each cell reports Score ∣\mid TPF.
View original tableOriginal Table 4

While LLaDA2.1 significantly improves inference speed, a clear speed-accuracy tradeoff persists, particularly with noticeable performance differences across various domains. It is necessary to adjust threshold parameters for different domains to balance speed and accuracy. In structured-data fields such as code and math, setting S Mode achieves high speed with little accuracy loss. However, in some general chat cases, these settings can cause undesirable output. In such cases, we recommend adjusting the parameters to Q Mode. Our conjecture is that this pattern may be related to the model’s inherent preference for structured data or the distributional characteristics of training dataset. Further validation will be conducted in our future research.

Although dLLMs inherently support high parallelism, theoretically offering speed advantages over AR models, our experimental observations show that this high parallelism also introduces a higher error rate compared to AR models. These hidden errors can reduce the model’s confidence in subsequent reasoning, ultimately slowing down the overall process. Therefore, timely editing to correct errors is essential. In our case analysis of LLaDA2.1, we observed that prompt editing corrected decoding errors, helping to maintain higher inference speeds. However, research on the editing capabilities of dLLMs is still in its early stages. We anticipate that future work, such as integrating editing into reinforcement learning, will further enhance the performance of editable dLLMs.

Empirical observations show that aggressively lowering the masking threshold τmask\tau_{\text{mask}} can quickly generate “rough drafts”. Although the model’s self-correction can partially alleviate the “stuttering” artifacts (such as n-gram repetitions) caused by independent parallel sampling, balancing drafting speed with the quality of the initial structure remains a key operational frontier. Overall, by unifying dynamic inference, hybrid training, and principled reinforcement learning, our work establishes a solid foundation for self-correcting discrete diffusion language models.

Overall, LLaDA2.1 introduces an editing feature, which, through cumulative error correction, significantly lowered the decoding threshold of the dLLM and yielded considerable inference speed benefits. However, this model still faces many unresolved issues, and we anticipate that more powerful editable dLLMs will deliver even more unexpected and impressive results.

References

  1. AIME (2025) AIME AIME Problems and Solutions. External Links: Link Cited by: 4th item.Source
  2. [2] Aleph-Alpha Alpha-MoE: a megakernel for faster tensor parallel inference. External Links: Link Cited by: §4.2.Source
  3. [3] Ant Group Team and SGLang Team Power Up Diffusion LLMs: Day-0 Support for LLaDA 2.0 | LMSYS Org. External Links: Link Cited by: §4.1, §4.2.Source
  4. Arriola et al. (2025) M. Arriola, A. Gokaslan, J. T. Chiu, Z. Yang, Z. Qi, J. Han, S. S. Sahoo, and V. Kuleshov Block diffusion: interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573. Cited by: §3.2.
  5. Bie et al. (2025) T. Bie, M. Cao, K. Chen, L. Du, M. Gong, Z. Gong, Y. Gu, J. Hu, Z. Huang, Z. Lan, C. Li, C. Li, J. Li, Z. Li, H. Liu, L. Liu, G. Lu, X. Lu, Y. Ma, J. Tan, L. Wei, J. Wen, Y. Xing, X. Zhang, J. Zhao, D. Zheng, J. Zhou, J. Zhou, Z. Zhou, L. Zhu, and Y. Zhuang LLaDA2.0: Scaling Up Diffusion Language Models to 100B. arXiv. External Links: 2512.15745, Document Cited by: §4.1.Source
  6. Bisk et al. (2020) Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. Piqa: reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, pp. 7432–7439. Cited by: 2nd item.
  7. Cassano et al. (2023) F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, et al. MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation. IEEE Transactions on Software Engineering 49 (7), pp. 3675–3691. Cited by: 3rd item.
  8. Dua et al. (2019) D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning over Paragraphs. arXiv preprint arXiv:1903.00161. Cited by: 2nd item.
  9. Fu et al. (2025) W. Fu, J. Gao, X. Shen, C. Zhu, Z. Mei, C. He, S. Xu, G. Wei, J. Mei, J. Wang, T. Yang, B. Yuan, and Y. Wu AReaL: a large-scale asynchronous reinforcement learning system for language reasoning. External Links: 2505.24298, Link Cited by: §4.1.Source
  10. Gao et al. (2024) B. Gao, F. Song, Z. Yang, Z. Cai, Y. Miao, Q. Dong, L. Li, C. Ma, L. Chen, R. Xu, et al. Omni-math: a universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985. Cited by: 4th item.
  11. Gu et al. (2024) A. Gu, B. Rozière, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang CruxEval: A Benchmark for Code Reasoning, Understanding and Execution. arXiv preprint arXiv:2401.03065. Cited by: 3rd item.
  12. He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems. arXiv preprint arXiv:2402.14008. Cited by: 4th item.
  13. Hu et al. (2020) H. Hu, K. Richardson, L. Xu, L. Li, S. Kübler, and L. S. Moss Ocnli: original chinese natural language inference. arXiv preprint arXiv:2010.05444. Cited by: 2nd item.
  14. Huang et al. (2023) Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, Y. Fu, et al. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. Advances in Neural Information Processing Systems 36, pp. 62991–63010. Cited by: 1st item.
  15. InclusionAI (2025) InclusionAI dFactory: Easy and Efficient dLLM Fine-Tuning. External Links: Link Cited by: §4.1.Source
  16. Jain et al. (2024) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica Livecodebench: holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974. Cited by: 3rd item.
  17. Joshi et al. (2017) M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer Triviaqa: a large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551. Cited by: 1st item.
  18. Kang et al. (2025) W. Kang, K. Galim, S. Oh, M. Lee, Y. Zeng, S. Zhang, C. Hooper, Y. Hu, H. I. Koo, N. I. Cho, and K. Lee ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs. arXiv. Note: arXiv:2510.04767 [cs] External Links: Link, Document Cited by: §1.SourceSource
  19. Kazemi et al. (2025) M. Kazemi, B. Fatemi, H. Bansal, J. Palowitch, C. Anastasiou, S. V. Mehta, L. K. Jain, V. Aglietti, D. Jindal, Y. P. Chen, et al. Big-bench extra hard. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 26473–26501. Cited by: 2nd item.
  20. Lee et al. (2025) S. Lee, S. Kim, S. Kim, J. Park, and D. Park Effective Test-Time Scaling of Discrete Diffusion through Iterative Refinement. arXiv. Note: arXiv:2511.05562 [cs] External Links: Link, Document Cited by: §1.SourceSource
  21. Li et al. (2023) J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems 36, pp. 42330–42357. Cited by: 3rd item.
  22. Li et al. (2024) Q. Li, L. Cui, X. Zhao, L. Kong, and W. Bi Gsm-plus: a comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. arXiv preprint arXiv:2402.19255. Cited by: 4th item.
  23. Lin et al. (2025) B. Y. Lin, R. L. Bras, K. Richardson, A. Sabharwal, R. Poovendran, P. Clark, and Y. Choi Zebralogic: on the scaling limits of llms for logical reasoning. arXiv preprint arXiv:2502.01100. Cited by: 2nd item.
  24. Ling Team et al. (2025) Ling Team et al. Every step evolves: scaling reinforcement learning for trillion-scale thinking model. External Links: 2510.18855, Link Cited by: §4.1.Source
  25. Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36, pp. 21558–21572. Cited by: 3rd item.
  26. Ma et al. (2024) K. Ma, X. Du, Y. Wang, H. Zhang, Z. Wen, X. Qu, J. Yang, J. Liu, M. Liu, X. Yue, et al. Kor-bench: benchmarking language models on knowledge-orthogonal reasoning tasks. arXiv preprint arXiv:2410.06526. Cited by: 2nd item.
  27. Ma et al. (2025) Y. Ma, L. Du, L. Wei, K. Chen, Q. Xu, K. Wang, G. Feng, G. Lu, L. Liu, X. Qi, et al. Dinfer: an efficient inference framework for diffusion language models. arXiv preprint arXiv:2510.08666. Cited by: §4.3.
  28. Mei et al. (2025) Z. Mei, W. Fu, K. Li, G. Wang, H. Zhang, and Y. Wu ReaL: efficient rlhf training of large language models with parameter reallocation. In Proceedings of the Eighth Conference on Machine Learning and Systems, MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025, Cited by: §4.1.
  29. Nexusflow.ai Team (2023) Nexusflow.ai Team NexusRaven-v2: surpassing gpt-4 for zero-shot function calling. External Links: Link Cited by: 5th item.Source
  30. Opencompass Team (2023) Opencompass Team Open-compass/opencompass. External Links: Link Cited by: 2nd item.Source
  31. Ou et al. (2025) J. Ou, J. Han, M. Xu, S. Xu, J. Xie, S. Ermon, Y. Wu, and C. Li Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective. arXiv. Note: arXiv:2512.03759 [cs] External Links: Link, Document Cited by: §1, §3.2.SourceSource
  32. Patil et al. (2025) S. G. Patil, H. Mao, C. Cheng-Jie Ji, F. Yan, V. Suresh, I. Stoica, and J. E. Gonzalez The berkeley function calling leaderboard (bfcl): from tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, Cited by: 5th item.
  33. Qiu et al. (2025) S. Qiu, S. Guo, Z. Song, Y. Sun, Z. Cai, J. Wei, T. Luo, Y. Yin, H. Zhang, Y. Hu, et al. Phybench: holistic evaluation of physical perception and reasoning in large language models. arXiv preprint arXiv:2504.16074. Cited by: 1st item.
  34. Rajpurkar et al. (2018) P. Rajpurkar, R. Jia, and P. Liang Know what you don’t know: unanswerable questions for squad. arXiv preprint arXiv:1806.03822. Cited by: 2nd item.
  35. Rein et al. (2024) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman GPQA: A Graduate-Level Google-Proof Q&Q Benchmark. In First Conference on Language Modeling, Cited by: 1st item.
  36. Rütte et al. (2025) D. v. Rütte, J. Fluri, Y. Ding, A. Orvieto, B. Schölkopf, et al. Generalized Interpolating Discrete Diffusion. (en). External Links: Link Cited by: §1.Source
  37. Saparov and He (2022) A. Saparov and H. He Language models are greedy reasoners: a systematic formal analysis of chain-of-thought. arXiv preprint arXiv:2210.01240. Cited by: 2nd item.
  38. Song et al. (2025) Y. Song, Z. Zhang, C. Luo, P. Gao, F. Xia, H. Luo, Z. Li, Y. Yang, et al. Seed diffusion: a large-scale diffusion language model with high-speed inference. External Links: 2508.02193 Cited by: §1.
  39. Sprague et al. (2023) Z. Sprague, X. Ye, K. Bostrom, S. Chaudhuri, and G. Durrett Musr: testing the limits of chain-of-thought with multistep soft reasoning. arXiv preprint arXiv:2310.16049. Cited by: 2nd item.
  40. Suzgun et al. (2023) M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003–13051. Cited by: 2nd item.
  41. Wang et al. (2025a) C. Wang, P. Rashidinejad, D. Su, S. Jiang, S. Wang, S. Zhao, C. Zhou, S. Z. Shen, F. Chen, T. Jaakkola, Y. Tian, and B. Liu SPG: sandwiched policy gradient for masked diffusion language models. arXiv preprint arXiv:2510.09541. Cited by: §1, §3.2.
  42. Wang et al. (2025b) G. Wang, Y. Schiff, S. S. Sahoo, and V. Kuleshov Remasking Discrete Diffusion Models with Inference-Time Scaling. (en). External Links: Link Cited by: §1.Source
  43. Wang et al. (2025c) Y. Wang, L. Yang, B. Li, Y. Tian, K. Shen, and M. Wang Revolutionizing reinforcement learning framework for diffusion large language models. arXiv preprint arXiv:2509.06949. Cited by: §1, §3.2.
  44. Wang et al. (2024) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, Cited by: 1st item.
  45. Wei et al. (2023) T. Wei, J. Luan, W. Liu, S. Dong, and B. Wang Cmath: can your language model pass chinese elementary school math test?. arXiv preprint arXiv:2306.16636. Cited by: 4th item.
  46. Yu et al. (2018) T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al. Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887. Cited by: 3rd item.
  47. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi Hellaswag: can a machine really finish your sentence?. arXiv preprint arXiv:1905.07830. Cited by: 2nd item.
  48. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, et al. Instruction-Following Evaluation for Large Language Models. arXiv preprint arXiv:2311.07911. Cited by: 5th item.
  49. Zhuo et al. (2024) T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, et al. Bigcodebench: benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877. Cited by: 3rd item.