dLLM: Simple Diffusion Language Modeling

Zhanhui Zhou, Lingjie Chen, Hanghang Tong, Dawn Song

arXiv:2602.22661v1 · This workspace stays on this version. Check arXiv for newer versions

Affiliations and author notes

UC Berkeley
UIUC
Equal contribution. Correspondence to zhanhui@berkeley.edu.

Some source details need review 13 to check

Original formulas, table values and figures are preserved. Comparison can miss differences; use the original PDF to check details.

Reading view checked: 462 math expressions, 0 rendering failures; 17 of 24 images available.

Compared 71 of 71 source passages against 18 of 18 PDF pages.

Some source graphics could not be placed in the reading view. (1)
Some figure, table or graphic images are unavailable; the original paper is still accessible. (7)
Some graphics are laid out differently in the HTML source than in the paper (text or formulas outside their frame); check the original PDF. (2)
abstract1_g1: blank, oversized or incomplete source rendering; capture unavailable.
abstract1_g2: blank, oversized or incomplete source rendering; capture unavailable.
S3_F1: the arXiv HTML draws text or formulas of this graphic outside its frame and it could not be corrected safely (too-small); the image shows the HTML layout, which differs from the PDF.
S3_F2: the arXiv HTML draws text or formulas of this graphic outside its frame and it could not be corrected safely (too-small); the image shows the HTML layout, which differs from the PDF.
S4_p1_g1: blank, oversized or incomplete source rendering; capture unavailable.
S4_SS2_SSS1_p1_g1: blank, oversized or incomplete source rendering; capture unavailable.
S4_SS2_SSS1_Px1_p1_g1: blank, oversized or incomplete source rendering; capture unavailable.
S4_SS2_SSS2_p1_g1: blank, oversized or incomplete source rendering; capture unavailable.
S4_SS2_SSS2_Px1_p1_g1: blank, oversized or incomplete source rendering; capture unavailable.

HTML block S3_SS2_SSS0_Px3_p1_1 says “Tables [6(b)](#A2.T6.st2) and [7(b)](#A2.T7.st2) in Appendix [B](#A2)”; PDF page 5 says “Tables 6 and 7 in Appendix B,” without subtable labels. Check PDF page 5

17 of 24 figure, table and graphic images captured. Open original PDF

Abstract

Although diffusion language models (DLMs) are evolving quickly, many recent models converge on a set of shared components. These components, however, are distributed across ad-hoc research codebases or lack transparent implementations, making them difficult to reproduce or extend. As the field accelerates, there is a clear need for a unified framework that standardizes these common components while remaining flexible enough to support new methods and architectures.

To address this gap, we introduce dLLM, an open-source framework that unifies the core components of diffusion language modeling—training, inference, and evaluation—and makes them easy to customize for new designs. With dLLM, users can reproduce, finetune, deploy, and evaluate open-source large DLMs such as LLaDA and Dream through a standardized pipeline. The framework also provides minimal, reproducible recipes for building small DLMs from scratch with accessible compute—including converting any BERT-style encoder or autoregressive LM into a DLM. We also release the checkpoints of these small DLMs to make DLMs more accessible and accelerate future research.

dLLM: https://github.com/ZHZisZZ/dllm

This graphic couldn’t be loaded.

dllm-hub: https://huggingface.co/dllm-hub

This graphic couldn’t be loaded.

Diffusion language models (DLMs) have emerged as a promising alternative to standard autoregressive language modeling (Austin et al., 2021a; Lou et al., 2024; Sahoo et al., 2024; Shi et al., 2024; Arriola et al., 2025), enabling iterative refinement (Wang et al., 2025; Havasi et al., 2025), flexible steering (Li et al., 2022; Schiff et al., 2025) and efficient decoding (Wu et al., 2026b; Wu et al., 2026a; Ma et al., 2025; Ben-Hamu et al., 2025). Alongside this rapid progress, a growing number of open-weight DLMs have appeared (Nie et al., 2025a; Nie et al., 2025b; Ye et al., 2025; Chandrasegaran et al., 2025; Bie et al., 2025), and many of them share similar design choices. However, these common components are frequently distributed across ad-hoc research codebases, or lack transparent implementations, making them difficult to reproduce, compare, or extend.

To address this critical gap, we introduce dLLM, an open-source framework that standardizes the end-to-end development pipeline for diffusion language modeling around three core components: training, inference, and evaluation. (1) For training, dLLM provides unified trainer modules that cover the most common objectives in DLMs, including Masked Diffusion (Sahoo et al., 2024) and Block Diffusion (Arriola et al., 2025), while keeping diffusion modeling logic decoupled from model architectures so that new objectives and variants can be added with minimal refactoring. In practice, this enables users to reproduce and finetune existing DLMs (e.g., LLaDA (Nie et al., 2025b) and Dream (Ye et al., 2025)) and develop new models from scratch. (2) For inference, dLLM introduces a lightweight abstraction that enables plug-and-play inference algorithms (including optimized efficient decoding algorithms (Wu et al., 2026b)) without modifying existing model implementations. (3) For evaluation, dLLM provides a unified evaluation interface for reproducing official results across models.

Beyond unifying existing DLM development pipelines, dLLM provides minimal, reproducible recipes for building small DLMs with accessible compute. These recipes include transparent end-to-end pipelines for converting existing LMs (e.g., BERT-style encoders (Devlin et al., 2019) and autoregressive language models (Gong et al., 2025)) into DLMs. We release checkpoints for these small models to support future research.

The key contributions of this work are:

•
We introduce dLLM, an open-source framework that unifies the core components of diffusion language modeling—training, inference, and evaluation—in a standardized, modular and extensible workflow, enabling transparent development and faster iteration across new designs.
•
We release minimal, end-to-end recipes and checkpoints for training small DLMs from scratch (e.g., converting BERT-style encoders and autoregressive LMs into DLMs), providing accessible starting points and baselines for future research.

We denote a sequence of discrete tokens as $x=(x^{1},\dots,x^{L})\in\mathcal{V}^{L}$ , where $\mathcal{V}$ is a finite vocabulary. We introduce a continuous time variable $t\in[0,1]$ and a special mask token $m\notin\mathcal{V}$ . The clean data is denoted $x_{0}$ , and $x_{t}$ represents the corrupted sequence at time $t$ .

Discrete diffusion models (Austin et al., 2021a; Sahoo et al., 2024; Lou et al., 2024) generate data by reversing a forward process that progressively destroys information. The forward process $q(x_{t}|x_{0})$ adds noise (e.g., random masking) over time $t:0\to 1$ , transforming the data into an uninformative state $x_{1}$ . The generative reverse process $p_{\theta}(x_{s}|x_{t})$ (where $s<t$ ) learns to denoise $x_{t}$ to recover $x_{0}$ . Unlike continuous diffusion, the state space remains discrete.

Masked Diffusion (MDLM) (Sahoo et al., 2024; Shi et al., 2024) simplifies the forward process as an absorbing-state masking process. In the forward process, each token $x_{0}^{i}$ is independently masked with probability $t$ , assuming linear schedule:

q(x_{t}^{i}|x_{0}^{i})=(1-t)\mathbb{I}(x_{t}^{i}=x_{0}^{i})+t\mathbb{I}(x_{t}^{i}=m),

where $\mathbb{I}$ is the indicator function. The model $p_{\theta}(x_{0}|x_{t})$ is trained to predict the unmasked tokens at indices $\mathcal{M}_{t}$ where $x_{t}^{i}=m$ . The training objective minimizes the negative log-likelihood of the clean tokens given the masked input, with a time-dependent reweighting (e.g., 1/t assuming linear schedule) to balance contributions across noise levels:

\mathcal{L}_{\text{MDLM}}=\mathbb{E}_{t\sim\mathcal{U}(0,1),x_{0}}\left[\frac{1}{t}\sum_{i\in\mathcal{M}_{t}}-\log p_{\theta}(x_{0}^{i}|x_{t})\right].

Block Diffusion (BD3LM) (Arriola et al., 2025) combines autoregression with diffusion. The sequence $x$ is partitioned into $K$ non-overlapping blocks $B_{1},\dots,B_{K}$ . We write $x^{B_{k}}$ to denote the tokens in block $B_{k}$ , and $x_{t}^{B_{k}}$ for its corrupted version at time $t$ . The model generates blocks autoregressively, but each block is generated via a diffusion process conditioned on the clean history of previous blocks $x_{<B_{k}}$ . The joint probability factorizes as $p_{\theta}(x)=\prod_{k=1}^{K}p_{\theta}(x^{B_{k}}\mid x^{<B_{k}})$ . For a specific block $B_{k}$ , the objective is the diffusion loss averaged over time, strictly applied to the current block tokens while freezing the history. We define $\text{mask}(B_{k},t)$ as the set of masked indices within block $B_{k}$ at time $t$ , i.e., $\text{mask}(B_{k},t):=\mathcal{M}_{t}\cap B_{k}$ :

\mathcal{L}_{\text{BD3LM}}=\sum_{k=1}^{K}\mathbb{E}_{t\sim\mathcal{U}(0,1),x_{0}}\left[\frac{1}{t}\sum_{i\in\text{mask}(B_{k},t)}-\log p_{\theta}(x_{0}^{i}\mid x_{t}^{B_{k}},x^{<B_{k}})\right].

This factorization allows the model to leverage cached key-values from previous blocks while generating the current block in parallel.

In this section, we provide an overview of the three core components of dLLM: Trainer (Section 3.1), Sampler (Section 3.2) and Evaluation (Section 3.3).

Figure 1: A unified trainer interface supports a variety of purposes via modular trainers and configuration changes. Figure 1(a) shows the MDLM pretraining setup. Figure 1(b) shows the single-line trainer swap from MDLMTrainer to BD3LMTrainer. Figure 1(c) shows the minimal changes to use MDLMTrainer for SFT: NoAttentionMaskWrapper keeps padding EOS visible, and label_pad_token_id=eos_token_id trains the model to generate EOS from extra mask tokens in inputs. Figure 1(d) shows the minimal changes to adapt an autoregressive LM to MDLM: right_shift_logits reuses next-token prediction, and PrependBOSWrapper prepends BOS to provide the predictions for the first mask token.

Most open-weight DLMs to date are trained with Masked Diffusion (MDLM) (Sahoo et al., 2024; Nie et al., 2025b; Ye et al., 2025) or Block Diffusion (BD3LM) (Arriola et al., 2025). Accordingly, the current version of dLLM focuses on unified MDLMTrainer and BD3LMTrainer as core training modules (dllm/core/trainers) that support both pretraining and finetuning most DLMs. At the same time, the framework’s modular design can be naturally extended to new diffusion objectives. For example, dLLM also includes a reference implementation of an EditFlow (Havasi et al., 2025; Nguyen et al., 2025) trainer for text diffusion with parallel insertion, substitution, and deletion operations.

Our training pipeline follows a modular design that allows core components to be reused and extended with minimal changes, improving both flexibility and readability. Figure 1 illustrates this modularity in practice: switching between MDLM/BD3LM pretraining, MDLM SFT, and AR-to-MDLM adaptation requires only localized changes (e.g., swapping the trainer, toggling a small set of arguments, or wrapping the data collator), without altering the overall pipeline.

Our training pipeline builds directly on the HuggingFace ecosystem. We use accelerate to support diverse training configurations (e.g., FSDP (Zhao et al., 2023) and DeepSpeed (Rajbhandari et al., 2020) for distributed training), and peft for parameter-efficient finetuning. Our custom trainers (e.g., MDLMTrainer) are lightweight wrappers around the transformers Trainer (Wolf et al., 2020). By using these components as building blocks, the framework stays easy to learn, which allows users to focus on DLM-specific logic and meanwhile remains scalable enough to support large-model pretraining and research experimentation.

Figure 2: **Inference pipeline: sampler swap from vanilla to FastdLLM MDLM sampler.**

Figure 3: **Terminal Visualizer showing transition from masked to decoded tokens.**

Different DLMs and inference algorithms expose inconsistent inference APIs, making it hard to reuse and compare inference algorithms across models. To address this issue without modifying existing model implementations, we introduce a lightweight inference abstraction: Sampler(model).sample(). This wrapper decouples models from inference algorithms, allowing different samplers to be swapped in a plug-and-play manner while keeping the underlying model unchanged. Figure 2 illustrates the unified inference pipeline enabled by this interface.

Unlike autoregressive LMs, which decode tokens strictly left-to-right, DLMs decode tokens in any order. As a result, the decoding order, beyond the final decoded output, is an important feature of DLMs and is valuable for analysis. To support debugging and interpretability, we provide a terminal visualizer that reveals the token decoding order and the evolution of the sample over decoding steps (Figure 3).

DLM inference speed is a practical bottleneck (Wu et al., 2026b; Wu et al., 2026a; Ma et al., 2025; Ben-Hamu et al., 2025). Building on the unified inference interface, dLLM includes an implementation of Fast-dLLM (Wu et al., 2026b) for accelerated MDLM decoding: MDLMFastdLLMSampler, which can be used as a drop-in replacement for the standard MDLMSampler (Figure 2). We report benchmarking results consistent with official Fast-dLLM implementations, demonstrating substantial inference speedups (see Figure 4 for visualization and Tables 6(b) and 7(b) in Appendix B for detailed results).

Figure 4: Fast-dLLM evaluation results with max new tokens @ $256$ and $512$ . Model selection follows the original Fast-dLLM evaluation for consistency and fair comparison. Cache uses block-wise approximate KV caching within each decoding block; Parallel uses confidence-based parallel token updates; Cache & Parallel combines both. Note that max new tokens determines the number of pre-allocated padding tokens in the bidirectional context window, therefore affecting compute and measured performance.

Open-weight DLMs (Nie et al., 2025b; Ye et al., 2025) rely on different evaluation tools, making unified evaluation difficult. This is further complicated by the fact that DLMs are especially sensitive to inference hyperparameters, as prior work often relies on task-specific hyperparameter tuning and postprocessing to achieve the best performance. For example, even a single change in an inference parameter can significantly alter performance (Figure 5).

Figure 5: Sensitivity to decoding hyperparameters. We vary individual sampling hyperparameters at inference time and observe that performance can degrade sharply from the optimal configuration. Baseline denotes the best-performing setting; Suppress does not suppress <eos> from the beginning of generation; CFG sets cfg=0.5; Parallel @ $4$ generates four tokens per step; and Temp @ $0$ sets temperature=0.0.

A unified evaluation pipeline must therefore be flexible enough to support customization while faithfully reproducing the evaluation configurations used in prior work. To achieve this, we extend the lm-evaluation-harness (Gao et al., 2024) framework and carefully match the preprocessing, decoding settings, and post-processing used for each model–task pair with its corresponding official pipeline. These details vary across models and tasks and require manual verification, but this enables our framework to reproduce the reported, model-specific scores while supporting consistent comparisons across models. Tables 4(b) and 5(b) (Appendix B) compare our reproduced results against the originally reported results, showing that our evaluation framework closely matches the official results.

Graphic from the paper; its text follows

Building on dLLM, we provide a set of fully reproducible recipes for training DLMs. These recipes cover (1) finetuning open-weight DLMs to reason (Section 4.1), and (2) training small DLMs from scratch with minimal compute (e.g., Section 4.2.1, Section 4.2). We make all of these model checkpoints available in dllm-hub along with their evaluation results.

This graphic couldn’t be loaded.

Training autoregressive LMs to reason before providing final answer has proven effective in solving complex tasks. Recent efforts such as d1 (Zhao et al., 2025) have begun exploring similar reasoning capabilities in DLMs. Using the unified trainer in dLLM, finetuning large DLMs is straightforward. We demonstrate that MDLM-style SFT can elicit reasoning capabilities in existing open-weight DLMs and improve their downstream performance.

We finetune both the Base and Instruct variants of LLaDA (Nie et al., 2025b) and Dream (Ye et al., 2025) using MDLM SFT with LoRA on the s1K dataset (Muennighoff et al., 2025). Loss is computed only on response tokens. We use maximum sequence length of $4096$ , $20$ epochs, learning rate $10^{-5}$ , global batch size $32$ with gradient accumulation steps of $4$ . We apply LoRA adaptation with $r=128$ , $\alpha=256$ , and weight decay $0.1$ . We adopt a cosine learning-rate schedule with $10\%$ warmup. Training is conducted on $8\times$ A100 GPUs using DeepSpeed ZeRO-2. See Figure 6 for training curves.

We evaluate models SFTed on reasoning data by prepending a <reasoning> token at inference to force reasoning (Table 1) using evaluation pipelines from Zhao et al. (2025). For Instruct models, reasoning SFT yields consistent gains across math, planning, and coding benchmarks. Base models show improvements on in-distribution math tasks (e.g., GSM8K (Cobbe et al., 2021), MATH500 (Hendrycks et al., 2021b)) but regress on out-of-distribution benchmarks. Overall, the results indicate that SFT is an effective starting point for reasoning in DLMs; all of these are achieved with the unified trainer interface (Figure 1) with little changes.

Original Table 1 — Table 1: **MDLM SFT evaluation results**. Instruct models show consistent gains, Base models gain on in-distribution math but may regress on out-of-distribution planning and coding.

Table 1
Model	GSM8K	MATH500	Countdown	Sudoku	HumanEval	MBPP
LLaDA-Instruct	$79.91$	$34.80$	$19.92$	$11.62$	$35.37$	$42.02$
+ SFT (MDLM)	$80.59$	$35.40$	$26.95$	$14.36$	$36.59$	$43.97$
LLaDA-Base	$64.67$	$10.20$	$10.16$	$0.34$	$25.00$	$40.08$
+ SFT (MDLM)	$73.62$	$17.40$	$8.98$	$0.00$	$18.90$	$28.79$
Dream-Instruct	$64.44$	$28.20$	$22.27$	$6.93$	$35.98$	$44.36$
+ SFT (MDLM)	$70.43$	$32.80$	$20.31$	$19.68$	$37.20$	$45.53$
Dream-Base	$49.05$	$20.40$	$10.55$	$1.07$	$15.85$	$22.96$
+ SFT (MDLM)	$63.00$	$23.40$	$8.59$	$1.03$	$29.88$	$23.35$

In addition to finetuning open-weight large DLMs, dLLM includes recipes and released checkpoints for training small DLMs from scratch (starting from backbones that are not DLMs). We cover two applications: (1) converting discriminative BERT models (Devlin et al., 2019) into DLMs and (2) converting autoregressive LMs into DLMs (Gong et al., 2025).

Despite their traditional use in discriminative tasks, BERT-style models (Devlin et al., 2019) offer bidirectional representations well-suited for diffusive generation (Sahoo et al., 2024). We show that an off-the-shelf BERT-style model can be turned into a diffusion chatbot, without architectural changes, by finetuning only on instruction-following data. We build on top of the ModernBERT series (Warner et al., 2025) as the backbone, as they are among the strongest-performing BERT variants, and release two checkpoints, ModernBERT-base-chat-v0.1 and ModernBERT-large-chat-v0.1.

This graphic couldn’t be loaded.

We finetune ModernBERT-base and ModernBERT-large via MDLM SFT (no continual pretraining) on a mixture of instruction-tuning datasets: Tulu 3 SFT (Lambert et al., 2024) and SmolTalk (Ben Allal et al., 2025). The loss is computed only on response tokens. We use maximum sequence length $1024$ , $10$ epochs, learning rate $10^{-4}$ , global batch size $384$ , bf16 precision, and a cosine learning-rate schedule with $10\%$ warmup. Training runs on $8\times$ A100 GPUs with DeepSpeed ZeRO-2 (Rajbhandari et al., 2020). See Figure 7 for training curves. We release the scripts to reproduce the models at dllm/examples/bert.

This graphic couldn’t be loaded.

We evaluate BERT-Chats using dLLM’s unified evaluation pipeline (Table 2). A gap remains compared to decoder-only ARLMs of a similar size (e.g., Qwen1.5-0.5B and Qwen1.5-0.5B-Chat (Bai et al., 2023) on MMLU (Hendrycks et al., 2021a) and HellaSwag (Zellers et al., 2019)), yet the results are still noteworthy: ModernBERT-large-chat surpasses both GPT-2 (Radford et al., 2019) variants on most benchmarks and outperforms Qwen1.5-0.5B-Chat on BBH (Suzgun et al., 2023) and MATH (Hendrycks et al., 2021b), despite being an encoder-only model with no architectural modification for generation. This suggests that BERT-style backbones are a viable, if under-explored, starting point for DLMs.

Autoregressive language models (ARLMs) dominate open-ended text generation, but DLMs offer complementary benefits such as parallel decoding and iterative refinement. Prior work has explored AR-to-diffusion conversion to bootstrap ARLM training artifacts into DLMs (e.g., RND1 (Chandrasegaran et al., 2025) and DiffuLLaMA (Gong et al., 2025)). We show that an off-the-shelf ARLM can be converted into a diffusion chatbot with minimal changes: we take Qwen3-0.6B (Yang et al., 2025) as the backbone and tune it under two diffusion objectives, MDLM (masked diffusion) (Sahoo et al., 2024) and BD3LM (block diffusion) (Arriola et al., 2025), on instruction-following data. We release two checkpoints, Qwen3-0.6B-diffusion-mdlm-v0.1 and Qwen3-0.6B-diffusion-bd3lm-v0.1.

This graphic couldn’t be loaded.

We train both variants with only SFT (no continual pretraining) on the mixture of Tulu 3 SFT (Lambert et al., 2024), SmolTalk (Ben Allal et al., 2025), and opc-sft-stage1&2 (Huang et al., 2024). Loss is computed only on response tokens and we do not apply the logits right shifting tricks as in prior work (Gong et al., 2025; Chandrasegaran et al., 2025), because in our experiments this leads to performance degradation. For the MDLM variant we use maximum sequence length $1024$ ; for the BD3LM variant we use length $512$ and block size $32$ . Both use $10$ epochs, learning rate $10^{-4}$ , global batch size $2048$ , bf16 precision, and a cosine learning-rate schedule. Training is run on $64\times$ A100 GPUs with DeepSpeed ZeRO-2 (Rajbhandari et al., 2020). See Figure 8 for training curves. We also release the scripts to reproduce the models at dllm/examples/a2d.

This graphic couldn’t be loaded.

We evaluate both converted models using dLLM’s unified evaluation pipeline (Table 3). The BD3LM variant shows particular strength on code generation, with HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021b) scores that surpass the original Qwen3-0.6B-Base (Yang et al., 2025) despite being trained with SFT alone. Overall, both DLM variants still trail their AR counterparts on most knowledge and reasoning benchmarks (e.g., MMLU (Hendrycks et al., 2021a), BBH (Suzgun et al., 2023)), reflecting the expected gap at this scale. Nonetheless, the fact that a competitive DLM can be obtained from an off-the-shelf ARLM with only SFT and no continual pretraining demonstrates that AR-to-diffusion conversion is a practical and compute-efficient path to building DLMs.

Original Table 2 — Table 2: **ModernBERT-Chat evaluation results.** ModernBERT-Chat (Warner et al., 2025) and GPT-2 (Radford et al., 2019) models are evaluated with **dLLM**’s pipeline; Qwen1.5 (Bai et al., 2023) numbers are reported by original sources. See Figure 7 for training curves.

Table 2
Model	GSM8K	BBH	MATH	MMLU	HellaSwag	LAMBADA	WinoGrande
ModernBERT-base-chat-v0.1	$3.6$	$21.1$	$3.1$	$26.2$	$34.5$	$49.3$	$48.8$
ModernBERT-large-chat-v0.1	$9.3$	$25.6$	$3.6$	$29.6$	$40.9$	$46.3$	$49.0$
Qwen1.5-0.5B	$22.0$	$18.3$	$3.1$	$39.2$	$48.2$	$48.6$	$55.0$
Qwen1.5-0.5B-Chat	$11.3$	$18.2$	$2.1$	$35.0$	$36.9$	$41.2$	$52.0$
GPT-2	$0.7$	$6.9$	$1.8$	$22.9$	$31.1$	$46.0$	$51.6$
GPT-2-medium	$2.1$	$17.8$	$1.4$	$22.9$	$39.4$	$55.5$	$53.1$

Original Table 3 — Table 3: **Qwen-A2D evaluation results.** MDLM and BD3LM models are evaluated with **dLLM**’s pipeline; Autoregressive Qwen2.5/Qwen3 baselines are reported by original sources (Qwen Team et al., 2024; Yang et al., 2025). See Figure 8 for training curves.

Table 3
Model	GSM8K	BBH	MATH	MMLU	MMLU-Pro	HellaSwag	HumanEval	MBPP
Qwen3-0.6B-mdlm-v0.1	$29.3$	$26.7$	$8.7$	$40.0$	$17.3$	$42.1$	$30.5$	$29.2$
Qwen3-0.6B-bd3lm-v0.1	$46.3$	$26.6$	$12.9$	$39.1$	$13.8$	$39.3$	$46.3$	$38.2$
Qwen2.5-0.5B	$41.6$	$20.3$	$19.5$	$47.5$	$15.7$	$52.1$	$30.5$	$39.3$
Qwen3-0.6B-Base	$59.6$	$41.5$	$32.4$	$52.8$	$24.7$	$47.4$	$32.3$	$36.6$

Diffusion models, originally developed for continuous domains (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021), are extended to discrete text via absorbing-state (D3PM (Austin et al., 2021a)) and uniform-state (multinomial diffusion (Hoogeboom et al., 2021)) formulations. Continuous-time extensions (Campbell et al., 2022), score-based (Sun et al., 2023; Meng et al., 2022), and ratio-based (Lou et al., 2024) objectives further unify the theory. Masked diffusion language models (MDLMs) simplify the forward process to independent token masking, with recent work clarifying equivalences and simplifying training (Sahoo et al., 2024; Shi et al., 2024; Ou et al., 2025; Zheng et al., 2025). Alternative directions include continuous diffusion in embedding space (Li et al., 2022; Gong et al., 2023; Dieleman et al., 2022; Lin et al., 2023), flow matching and edit-based methods (Gat et al., 2024; Havasi et al., 2025; Nguyen et al., 2025), block diffusion (Arriola et al., 2025), which interpolates between AR and diffusion decoding for KV-cache reuse, and hybrid AR–diffusion architectures that use diffusion for speculative drafting (Christopher et al., 2025), planned outline-then-diffuse generation (Israel et al., 2026), or unified draft-and-verify passes (Liu et al., 2025).

Scaling DLMs has progressed rapidly. Nie et al. first scaled masked diffusion to 1.1B parameters. Converting pretrained autoregressive models into DLMs has proven effective: DiffuGPT/DiffuLLaMA adapt GPT-2 and LLaMA (127M–7B) (Gong et al., 2025); RND1 (Chandrasegaran et al., 2025) extends this to 30B; and LLaDA2.0 (Bie et al., 2025) scales to 100B with a 3-phase block-level scheme. At the 7–8B scale, Dream (Ye et al., 2025) adapts Qwen-2.5 with context-adaptive noise rescheduling, and LLaDA (Nie et al., 2025b) trains an 8B MDLM from scratch, achieving performance competitive with LLaMA3-8B (Grattafiori et al., 2024). Commercial systems such as Mercury (Khanna et al., 2025) further demonstrate DLM viability in production.

Prior open-weight DLMs (Nie et al., 2025b; Ye et al., 2025; Gong et al., 2025; Chandrasegaran et al., 2025; Bie et al., 2025) often lack unified development pipelines, making reproduction and comparison difficult. Open efficient inference tools for DLMs such as Fast-dLLM (Wu et al., 2026b) and Fast-dLLM v2 (Wu et al., 2026a) accelerate decoding but are developed independently of training and evaluation. Evaluation pipelines also vary across papers (e.g., task sets, inference hyperparameters), and interfaces remain inconsistent. A framework unifying training, inference, and evaluation has been lacking. dLLM fills this gap with modular trainers, a plug-and-play sampler abstraction, and a reproducible evaluation pipeline aligned with official benchmarks (Section 3, Section 4).

We present dLLM, an open-source framework that unifies the training, inference, and evaluation of DLMs in a modular, extensible pipeline. By standardizing the common components shared across recent DLMs, dLLM lowers the barrier to reproducing, finetuning, and fairly comparing existing models while making it straightforward to integrate new designs. Alongside the framework, we provide minimal recipes and checkpoints showing that existing pretrained models, both BERT-style encoders and autoregressive LMs, can be converted into competitive DLMs with lightweight finetuning alone, making DLM development increasingly accessible with minimal compute. We hope dLLM accelerates research and lowers the entry barrier for the broader community.

We plan to continue expanding dLLM by incorporating new methods as the field evolves, e.g., integrating RL algorithms once widely adopted approaches for DLMs emerge, and supporting additional open-weight models as they are released.

Zhanhui Zhou gratefully acknowledges support from the Berkeley Fellowship.

References

Arriola et al. (2025) M. Arriola, A. Gokaslan, J. T. Chiu, Z. Yang, Z. Qi, J. Han, S. S. Sahoo, and V. Kuleshov Block diffusion: interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations, Cited by: §1, §1, §2, §3.1, §4.2.2, §5.
Austin et al. (2021a) J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. van den Berg Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, Cited by: §1, §2, §5.
Austin et al. (2021b) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: §4.2.2.
Bai et al. (2023) J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B. Yu, H. Yuan, Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu Qwen technical report. arXiv preprint arXiv:2309.16609. Cited by: §4.2.1, Table 2.
Ben Allal et al. (2025) L. Ben Allal, A. Lozhkov, G. Penedo, L. Tunstall, C. Raffel, L. von Werra, T. Wolf, et al. SmolLM2: when Smol goes big – data-centric training of a small language model. arXiv preprint arXiv:2502.02737. Cited by: §4.2.1, §4.2.2.
Ben-Hamu et al. (2025) H. Ben-Hamu, I. Gat, D. Severo, N. Nolte, and B. Karrer Accelerated sampling from masked diffusion models via entropy bounded unmasking. In Advances in Neural Information Processing Systems, Cited by: §1, §3.2.
Bie et al. (2025) T. Bie, M. Cao, K. Chen, L. Du, M. Gong, Z. Gong, Y. Gu, J. Hu, Z. Huang, Z. Lan, C. Li, C. Li, J. Li, Z. Li, H. Liu, L. Liu, G. Lu, X. Lu, Y. Ma, J. Tan, L. Wei, J. Wen, Y. Xing, X. Zhang, J. Zhao, D. Zheng, J. Zhou, J. Zhou, Z. Zhou, L. Zhu, and Y. Zhuang LLaDA2.0: scaling up diffusion language models to 100B. arXiv preprint arXiv:2512.15745. Cited by: §1, §5, §5.
Campbell et al. (2022) A. Campbell, J. Benton, V. De Bortoli, T. Rainforth, G. Deligiannidis, and A. Doucet A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, Cited by: §5.
Chandrasegaran et al. (2025) K. Chandrasegaran, A. W. Thomas, J. Ku, F. Berto, J. M. Kim, G. Brixi, E. Nguyen, S. Massaroli, and M. Poli RND1: simple, scalable AR-to-Diffusion conversion. Note: Radical Numerics Technical Report External Links: Link Cited by: §1, §4.2.2, §4.2.2, §5, §5.Source
Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §4.2.2.
Christopher et al. (2025) J. K. Christopher, B. R. Bartoldson, T. Ben-Nun, M. Cardei, B. Kailkhura, and F. Fioretto Speculative diffusion decoding: accelerating language generation through diffusion. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Cited by: §5.
Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §4.1.
Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Cited by: §1, §4.2.1, §4.2.
Dieleman et al. (2022) S. Dieleman, L. Sartran, A. Roshannai, N. Savinov, Y. Ganin, P. H. Richemond, A. Doucet, R. Strudel, C. Dyer, C. Durkan, et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089. Cited by: §5.
Fu et al. (2025) Y. Fu, L. Whalen, Z. Ye, X. Dong, S. Diao, J. Liu, C. Wu, H. Zhang, E. Xie, S. Han, M. Khadkevich, J. Kautz, Y. C. Lin, and P. Molchanov Efficient-DLM: from autoregressive to diffusion language models, and beyond in speed. arXiv preprint arXiv:2512.14067. Cited by: §4.2.2.
Gao et al. (2024) L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, et al. EleutherAI/lm-evaluation-harness: v0.4.3. Note: Zenodo External Links: Document, Link Cited by: §3.3.Source Source
Gat et al. (2024) I. Gat, T. Remez, N. Shaul, F. Kreuk, R. T. Q. Chen, G. Synnaeve, Y. Adi, and Y. Lipman Discrete flow matching. In Advances in Neural Information Processing Systems, Cited by: §5.
Gong et al. (2025) S. Gong, S. Agarwal, Y. Zhang, J. Ye, L. Zheng, M. Li, C. An, P. Zhao, W. Bi, J. Han, H. Peng, and L. Kong Scaling diffusion language models via adaptation from autoregressive models. In International Conference on Learning Representations, Cited by: §1, §4.2.2, §4.2.2, §4.2, §5, §5.
Gong et al. (2023) S. Gong, M. Li, J. Feng, Z. Wu, and L. Kong DiffuSeq: sequence to sequence text generation with diffusion models. In International Conference on Learning Representations, Cited by: §5.
Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §5.
Havasi et al. (2025) M. Havasi, B. Karrer, I. Gat, and R. T. Q. Chen Edit flows: variable length discrete flow matching with sequence-level edit operations. In Advances in Neural Information Processing Systems, Cited by: §1, §3.1, §5.
Hendrycks et al. (2021a) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In International Conference on Learning Representations, Cited by: §4.2.1, §4.2.2.
Hendrycks et al. (2021b) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the MATH dataset. In NeurIPS Datasets and Benchmarks, Cited by: §4.1, §4.2.1.
Ho et al. (2020) J. Ho, A. Jain, and P. Abbeel Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, Cited by: §5.
Hoogeboom et al. (2021) E. Hoogeboom, D. Nielsen, P. Jaini, P. Forré, and M. Welling Argmax flows and multinomial diffusion: learning categorical distributions. In Advances in Neural Information Processing Systems, Cited by: §5.
Huang et al. (2024) S. Huang, T. Cheng, J. K. Liu, J. Hao, L. Song, Y. Xu, J. Yang, J. Liu, C. Zhang, L. Chai, R. Yuan, Z. Zhang, J. Fu, Q. Liu, G. Zhang, Z. Wang, Y. Qi, Y. Xu, and W. Chu OpenCoder: the open cookbook for top-tier code large language models. arXiv preprint arXiv:2411.04905. Cited by: §4.2.2.
Israel et al. (2026) D. Israel, T. Jin, E. Cheng, G. Van den Broeck, A. Grover, S. Subramanian, and M. Carbin Planned diffusion. In International Conference on Learning Representations, Cited by: §5.
Khanna et al. (2025) S. Khanna, S. Kharbanda, S. Li, H. Varma, E. Wang, S. Birnbaum, Z. Luo, Y. Miraoui, A. Palrecha, S. Ermon, A. Grover, and V. Kuleshov Mercury: ultra-fast language models based on diffusion. arXiv preprint arXiv:2506.17298. Cited by: §5.
Lambert et al. (2024) N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang, R. Le Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi Tulu 3: pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124. Cited by: §4.2.1, §4.2.2.
Li et al. (2022) X. L. Li, J. Thickstun, I. Gulrajani, P. Liang, and T. B. Hashimoto Diffusion-LM improves controllable text generation. In Advances in Neural Information Processing Systems, Cited by: §1, §5.
Lin et al. (2023) Z. Lin, Y. Gong, Y. Shen, T. Wu, Z. Fan, C. Lin, N. Duan, and W. Chen Text generation with diffusion language models: a pre-training approach with continuous paragraph denoise. In Proceedings of the 40th International Conference on Machine Learning, Cited by: §5.
Liu et al. (2025) J. Liu, X. Dong, Z. Ye, R. Mehta, Y. Fu, V. Singh, J. Kautz, C. Zhang, and P. Molchanov TiDAR: think in diffusion, talk in autoregression. arXiv preprint arXiv:2511.08923. Cited by: §5.
Lou et al. (2024) A. Lou, C. Meng, and S. Ermon Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning, Cited by: §1, §2, §5.
Ma et al. (2025) X. Ma, R. Yu, G. Fang, and X. Wang dKV-Cache: the cache for diffusion language models. In Advances in Neural Information Processing Systems, Cited by: §1, §3.2.
Meng et al. (2022) C. Meng, K. Choi, J. Song, and S. Ermon Concrete score matching: generalized score matching for discrete data. In Advances in Neural Information Processing Systems, Cited by: §5.
Muennighoff et al. (2025) N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. B. Hashimoto s1: simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Cited by: §4.1.
Nguyen et al. (2025) J. Nguyen, M. Havasi, T. Berrada, L. Zettlemoyer, and R. T. Q. Chen OneFlow: concurrent mixed-modal and interleaved generation with edit flows. arXiv preprint arXiv:2510.03506. Cited by: §3.1, §5.
Nie et al. (2025a) S. Nie, F. Zhu, C. Du, T. Pang, Q. Liu, G. Zeng, M. Lin, and C. Li Scaling up masked diffusion models on text. In International Conference on Learning Representations, Cited by: §1, §5.
Nie et al. (2025b) S. Nie, F. Zhu, Z. You, X. Zhang, J. Ou, J. Hu, J. Zhou, Y. Lin, J. Wen, and C. Li Large language diffusion models. In Advances in Neural Information Processing Systems, Cited by: Appendix B, §1, §1, §3.1, §3.3, §4.1, §5, §5.
Ou et al. (2025) J. Ou, S. Nie, K. Xue, F. Zhu, J. Sun, Z. Li, and C. Li Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In International Conference on Learning Representations, Cited by: §5.
Qwen Team et al. (2024) Qwen Team, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: Table 3.
Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog. Cited by: §4.2.1, Table 2.
Rajbhandari et al. (2020) S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He ZeRO: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, Cited by: §3.1, §4.2.1, §4.2.2.
Sahoo et al. (2024) S. S. Sahoo, M. Arriola, Y. Schiff, A. Gokaslan, E. Marroquin, J. T. Chiu, A. Rush, and V. Kuleshov Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, Cited by: §1, §1, §2, §2, §3.1, §4.2.1, §4.2.2, §5.
Schiff et al. (2025) Y. Schiff, S. S. Sahoo, H. Phung, G. Wang, S. Boshar, H. Dalla-torre, B. P. de Almeida, A. M. Rush, T. Pierrot, and V. Kuleshov Simple guidance mechanisms for discrete diffusion models. In International Conference on Learning Representations, Cited by: §1.
Shi et al. (2024) J. Shi, K. Han, Z. Wang, A. Doucet, and M. K. Titsias Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, Cited by: §1, §2, §5.
Sohl-Dickstein et al. (2015) J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, Cited by: §5.
Song et al. (2021) Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, Cited by: §5.
Sun et al. (2023) H. Sun, L. Yu, B. Dai, D. Schuurmans, and H. Dai Score-based continuous-time discrete diffusion models. In International Conference on Learning Representations, Cited by: §5.
Suzgun et al. (2023) M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, Cited by: §4.2.1, §4.2.2.
Wang et al. (2025) G. Wang, Y. Schiff, S. S. Sahoo, and V. Kuleshov Remasking discrete diffusion models with inference-time scaling. In Advances in Neural Information Processing Systems, Cited by: §1.
Warner et al. (2025) B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, G. T. Adams, J. Howard, and I. Poli Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Cited by: §4.2.1, Table 2.
Wolf et al. (2020) T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush Transformers: state-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Cited by: §3.1.
Wu et al. (2026a) C. Wu, H. Zhang, S. Xue, S. Diao, Y. Fu, Z. Liu, P. Molchanov, P. Luo, S. Han, and E. Xie Fast-dLLM v2: efficient block-diffusion LLM. In International Conference on Learning Representations, Cited by: §1, §3.2, §5.
Wu et al. (2026b) C. Wu, H. Zhang, S. Xue, Z. Liu, S. Diao, L. Zhu, P. Luo, S. Han, and E. Xie Fast-dLLM: training-free acceleration of diffusion LLM by enabling KV cache and parallel decoding. In International Conference on Learning Representations, Cited by: §1, §1, §3.2, §5.
Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §4.2.2, §4.2.2, Table 3.
Ye et al. (2025) J. Ye, Z. Xie, L. Zheng, J. Gao, Z. Wu, X. Jiang, Z. Li, and L. Kong Dream 7B: diffusion large language models. arXiv preprint arXiv:2508.15487. Cited by: Appendix B, §1, §1, §3.1, §3.3, §4.1, §5, §5.
Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: can a machine really finish your sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Cited by: §4.2.1.
Zhao et al. (2025) S. Zhao, D. Gupta, Q. Zheng, and A. Grover d1: scaling reasoning in diffusion large language models via reinforcement learning. In Advances in Neural Information Processing Systems, Cited by: §4.1, §4.1.
Zhao et al. (2023) Y. Zhao, A. Gu, R. Varma, L. Luo, C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y. Hao, A. Mathews, and S. Li PyTorch FSDP: experiences on scaling fully sharded data parallel. Proc. VLDB Endow. 16 (12), pp. 3848–3860. Cited by: §3.1.
Zheng et al. (2025) K. Zheng, Y. Chen, H. Mao, M. Liu, J. Zhu, and Q. Zhang Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. In International Conference on Learning Representations, Cited by: §5.

Figure 6: Training loss for finetuning open-weight DLMs to reason (Section 4.1).

Figure 7: Training loss for finetuning BERT to chat (Section 4.2.1).

Figure 8: Training loss for finetuning autoregressive LMs to be DLMs (Section 4.2.2).

In this section, we report evaluation results comparing the official implementation (as reported in the original paper) with our unified dLLM reimplementation under the same configurations. Overall, our framework reproduces the official results closely across benchmarks, indicating that our evaluation pipeline and implementation are consistent with the official setup (with only minor necessary adjustments).

Tables 4(b) and 5(b) report our reproduced evaluation results for LLaDA (Nie et al., 2025b) and Dream (Ye et al., 2025) with dLLM. Tables 6(b) and 7(b) report Fast-dLLM results, showing that our dLLM reimplementation achieves similar accuracy to the official numbers while substantially improving generation throughput.

Original Table 4 — Table 4: **LLaDA evaluation results.** “Official” denotes results from the original paper; “**dLLM**” denotes results from our **dLLM** reimplementation. “Hella.” stands for HellaSwag, “HEval” stands for HumanEval and “WinoG.” stands for WinoGrande.

Original Table 5 — Table 5: **Dream evaluation results.** “Official” denotes results from the original paper; “**dLLM**” denotes results from our **dLLM** reimplementation. “Hella.” stands for HellaSwag, “HEval” stands for HumanEval and “WinoG.” stands for WinoGrande.

Original Table 6 — Table 6: **Fast-dLLM LLaDA-Instruct evaluation results with max new tokens @ $256$ (a) and $512$ (b).** “Official” denotes results from the Fast-dLLM paper; “**dLLM**” denotes results from our **dLLM** reimplementation.

Original Table 7 — Table 7: **Fast-dLLM Dream-Base evaluation results with max new tokens @ $256$ (a) and $512$ (b).** “Official” denotes results from the Fast-dLLM paper; “**dLLM**” denotes results from our **dLLM** reimplementation.

	MMLU	BBH	ARC-C	Hella.	WinoG.	PIQA	GSM8K	MATH	GPQA	HEval	MBPP
Official	65.9	49.7	45.9	70.5	74.8	73.6	70.3	31.4	25.2	35.4	40.0
dLLM	65.9	47.2	44.1	69.2	70.4	70.7	70.7	32.4	31.9	32.9	38.8

	MMLU	MMLU-Pro	ARC-C	Hella.	GSM8K	Math	GPQA	HEval	MBPP
Official	65.5	37.0	88.5	74.6	69.4	31.9	33.3	49.4	41.0
dLLM	69.8	36.2	86.4	76.7	74.7	31.9	30.6	47.0	40.0

	MMLU	BBH	ARC-C	Hella.	WinoG.	PIQA	GSM8K	MATH	GPQA	HEval	MBPP
Official	69.5	57.9	59.9	73.3	74.8	75.8	77.2	39.6	36.6	57.9	56.2
dLLM	70.0	63.7	59.0	73.5	72.5	76.4	77.0	42.4	34.6	56.7	56.0

Benchmark	Source	Baseline		+Cache		+Parallel		+Both
		Acc	Tok/s ( $\times$ )	Acc	( $\times$ )	Acc	( $\times$ )	Acc	( $\times$ )
GSM8K	Official	$79.3$	$6.7\ (1.0\times)$	$79.5$	$3.2\times$	$79.2$	$2.5\times$	$78.5$	$8.1\times$
	dLLM	$78.0$	$8.1\ (1.0\times)$	$78.2$	$3.2\times$	$78.9$	$2.3\times$	$78.0$	$6.5\times$
MATH	Official	$33.5$	$9.1\ (1.0\times)$	$33.3$	$2.6\times$	$33.4$	$2.7\times$	$33.2$	$5.7\times$
	dLLM	$38.3$	$9.7\ (1.0\times)$	$37.6$	$2.7\times$	$38.6$	$2.0\times$	$37.5$	$5.0\times$
HumanEval	Official	$41.5$	$30.5\ (1.0\times)$	$42.7$	$1.3\times$	$43.9$	$3.3\times$	$43.3$	$3.7\times$
	dLLM	$38.4$	$18.8\ (1.0\times)$	$36.0$	$1.5\times$	$39.6$	$2.8\times$	$36.0$	$3.6\times$
MBPP	Official	$29.4$	$6.0\ (1.0\times)$	$29.6$	$2.8\times$	$28.4$	$4.1\times$	$28.2$	$7.5\times$
	dLLM	$36.4$	$9.3\ (1.0\times)$	$38.0$	$2.8\times$	$29.0$	$1.9\times$	$37.8$	$4.8\times$

Benchmark	Source	Baseline		+Cache		+Parallel		+Both
		Acc	Tok/s ( $\times$ )	Acc	( $\times$ )	Acc	( $\times$ )	Acc	( $\times$ )
GSM8K	Official	$77.5$	$3.2\ (1.0\times)$	$77.0$	$3.3\times$	$77.6$	$5.8\times$	$77.2$	$11.0\times$
	dLLM	$81.1$	$6.7\ (1.0\times)$	$76.0$	$3.0\times$	$77.6$	$3.3\times$	$76.6$	$7.8\times$
MATH	Official	$37.2$	$8.0\ (1.0\times)$	$36.2$	$2.5\times$	$36.8$	$3.0\times$	$36.0$	$5.9\times$
	dLLM	$42.4$	$7.4\ (1.0\times)$	$41.9$	$2.9\times$	$42.5$	$2.7\times$	$41.8$	$6.0\times$
HumanEval	Official	$43.9$	$18.4\ (1.0\times)$	$45.7$	$1.6\times$	$43.3$	$3.1\times$	$44.5$	$4.0\times$
	dLLM	$48.2$	$13.0\ (1.0\times)$	$41.5$	$1.8\times$	$50.6$	$2.8\times$	$41.5$	$4.3\times$
MBPP	Official	$14.8$	$4.3\ (1.0\times)$	$13.4$	$2.3\times$	$15.0$	$5.1\times$	$13.8$	$9.2\times$
	dLLM	$32.2$	$7.7\ (1.0\times)$	$22.0$	$2.7\times$	$7.6$	$2.7\times$	$21.4$	$5.7\times$

dLLM: Simple Diffusion Language Modeling

Abstract

References

Meow summary

The problem dLLM tackles

How diffusion repairs text

What dLLM makes reusable

What the results show—and don’t

Make notes of your own

	MMLU	MMLU-Pro	ARC-C	Hella.	GSM8K	MATH	GPQA	HEval	MBPP
Official	67.0	43.3	—	—	81.0	39.2	33.0	55.5	58.8
dLLM	69.8	45.5	61.4	71.8	82.0	48.6	31.5	57.9	58.2

Abstract

1 Introduction

2 Preliminaries

Discrete Diffusion.

Masked Diffusion (MDLM).

Block Diffusion (BD3LM).

3 dLLM Overview

3.1 Trainer

Unified training interface with Trainer (Figure 1).

Modular design enables easy customization (Figure 1).

Simple yet scalable training powered by HF infrastructure.

3.2 Sampler

Unified inference interface with Sampler (Figure 2).

Terminal visualizer (Figure 3).

Efficient DLM inference (Figures 2 & 4).

3.3 Evaluation

4 Open DLMs with Open Recipes

4.1 Finetuning Open-Weight Large DLMs

Training details.

Evaluation results.

4.2 Training Small DLMs from Scratch

4.2.1 BERT-Chat: Converting BERTs to DLMs

Training details.

Evaluation results.

4.2.2 Tiny-A2D: Converting ARLMs to DLMs

Training details.

Evaluation results.

5 Related Work

Discrete diffusion for text.

Open-weight DLMs.

Open tools for DLMs.

6 Conclusion

Future work.

Acknowledgements

References

Appendix A Training Curves

Appendix B Evaluation Reproduction

Meow summary

The problem dLLM tackles

How diffusion repairs text

What dLLM makes reusable

What the results show—and don’t

Make notes of your own