Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

Let $x\in\mathcal{X}$ denote an input and $y=(y_{1},\dots,y_{T})$ a sequence generated by a language model $\pi_{\theta}$. The model defines an autoregressive distribution $\pi_{\theta}(y|x)=\prod_{t=1}^{T}\pi_{\theta}(y_{t}\mid x,y_{<t}).$ In self-distillation, the same model acts as both a student and a teacher under different conditioning contexts. The student first generates a sequence $y\sim\pi_{\theta}(\cdot\mid x)$. The teacher policy is obtained by conditioning the model on a richer context $c$ that provides additional information about the input (e.g., solutions, environment feedback, or other auxiliary signals). Training minimizes the divergence between the student and teacher next-token distributions:

This objective encourages the student to match the teacher’s predictions under the richer context, enabling the model to improve by distilling privileged information available at training time without requiring an external teacher.

In LLMs, math reasoning can be viewed as self-Bayesian reasoning, where each step is generated conditioned only on the prompt $x$ and previously generated tokens $y=(y_{1},\dots,y_{T})$, allowing the model to iteratively update its belief over intermediate hypotheses (Kim et al., 2026). Such reasoning spans diverse tasks such as arithmetic, algebra, geometry, word problems, and logical pattern recognition, so evaluation benchmarks are frequently OOD relative to training data due to compositional and reasoning-depth shifts.

Within this process, verbalized uncertainty toward correct response $y^{*}$, referred to as epistemic verbalization (Kim et al., 2026), can serve as an informative signal rather than stylistic redundancy. As shown in Figure 2(2a), reasoning without such signals may prematurely commit to incorrect hypotheses with little room for recovery, whereas epistemic verbalization preserves alternative hypotheses and supports gradual uncertainty reduction.

The prompts used for unguided and solution-guided settings are as follows. For regeneration, we used the same prompts as in Hübotter et al. (2026). A comparative analysis with other regeneration prompts is provided in Appendix H.

Following Kim et al. (2026), we define a set of 10 epistemic markers $\mathcal{T}=\{\textit{wait},\,\textit{hmm},\,\textit{perhaps},\,\textit{maybe},\,\textit{actually},\,\textit{alternatively},\,\textit{seems},\,\textit{might},\,\textit{likely},\,\textit{check}\}$ as practical indicators of regions where the externalization of uncertainty is likely to occur. These tokens are intuitively recognizable as uncertainty expressions, with their selection further supported by a cross-family analysis of 9,600 reasoning traces (Appendix B). We measure the epistemic token count of a response $y$ as $E(y)=\sum_{t\in\mathcal{T}}\mathrm{count}(t,\,y)$.

We analyze how different forms of conditioning information toward the correct answer affect the model’s reasoning behavior by comparing the average response length $\mathbb{E}[L(y)]$, score, and the epistemic token count $\mathbb{E}[E(y)]$ across the four settings. As shown in Table 1, both quantities decrease monotonically as $I(y^{*};\,c\mid x)$ increases:

and analogously for $\mathbb{E}[E(y)]$, confirming that richer conditioning information leads to more concise and confident reasoning.

Specifically, unguided generation ($c=\emptyset$) produces substantially longer responses with the highest epistemic token counts. When the full solution $s$ is provided in (2), the model follows the given reasoning trace with high confidence, and its concise output can be viewed as a compressed representation of the reasoning in $s$. In (3), removing the <think> portion retains only $s_{\setminus\text{think}}$ (640 out of 13,054 response tokens), and both $\mathbb{E}[L(y)]$ and $\mathbb{E}[E(y)]$ increase again toward the unguided level, reflecting the substantial information loss. Setting (4), conditioning on the regenerated response $y_{r}$, yields intermediate values (lower than (3) but higher than (2)), indicating that $y_{r}$ preserves much of the informative structure of the full solution. Detailed per-token breakdowns are reported in Appendix E.1.

DeepSeek-R1-Distill-Qwen-7B is a representative high-reasoning model that generates rich epistemic verbalizations within <think> tags, averaging about 180 epistemic tokens per response, as shown in Section 3, leading to lengthy yet high-performing reasoning outputs.

Qwen3-8B is a hybrid reasoning model that can operate in either a thinking mode for deliberate reasoning or a non-thinking mode for direct responses. With thinking mode enabled, Qwen3-8B produces longer responses than DeepSeek-R1-Distill-Qwen-7B, with a higher number of epistemic tokens (see Appendix E.2 for a comparison of epistemic tokens across models).

When Qwen3-8B is used without thinking mode, the <think> tag is absent, so we consider only $c=s$. The model initially produces much shorter responses and exhibits significantly lower performance. GRPO rapidly increases $\mathbb{E}[L(y)]$ by promoting epistemic verbalization (as shown in Appendix J.1), quickly achieving a high training score. In contrast, SDPO reduces $\mathbb{E}[L(y)]$ and improves much more slowly. Even when the training score slightly increases, as shown in Figure 5b, performance on AIME24 slightly declines ($0.25\to 0.23$), further illustrating the cost of epistemic suppression under self-distillation.

To further examine whether the suppression of epistemic verbalization is a causal contributor to the observed performance degradation, we explicitly penalized epistemic tokens during GRPO training and tested whether this intervention induces a similar degradation pattern.

Following the epistemic marker set $\mathcal{T}$ defined in Section 3, we override the standard GRPO advantage at every position $t$ whose generated token realizes a marker in $\mathcal{T}$, setting $\tilde{A}_{t}=\min(A_{t},-\kappa)$ if $y_{t}\in\mathcal{T}$ and $\tilde{A}_{t}=A_{t}$ otherwise, with suppression strength $\kappa=0.5$. Since $\tilde{A}_{t}\leq-\kappa$ holds even for correct rollouts, the policy gradient consistently reduces $\mathbb{E}[E(y)]$ throughout training.

As shown in Figure 6, explicitly penalizing epistemic tokens during GRPO produces shorter responses and lower performance, qualitatively reproducing the degradation pattern observed with SDPO. The degradation is less pronounced than under SDPO, possibly because the predefined epistemic token set does not capture all linguistic realizations of uncertainty or epistemic reasoning, as discussed further in Appendix E.5. Nevertheless, this result provides causal evidence that suppressing uncertainty expression contributes to the performance decline under SDPO.

In naive on-policy self-distillation, the teacher and student share a continuously updated policy, making the teacher a moving target that can introduce training instability (Zhao et al., 2026; Ye et al., 2026). To mitigate this, SDPO uses an EMA-smoothed teacher (EMA rate: 0.05). However, we find that setting the EMA rate to 0.0 (i.e., fixing the teacher to the initial policy) yields better performance, and thus Section 5 follows this setting.

Figure 7a shows additional comparison results when the teacher is updated during training. As shown, even slow updates (e.g., rate 0.05) lead to a sharper reduction in response length, resulting in larger performance degradation. This can be interpreted as a feedback loop in self-distillation. The model is trained to produce increasingly confident outputs, and when this is used as the teacher, it generates even more confident responses, amplifying the effect over iterations. Further ablations on learning rate, top-$k$ logits, and the effect of limiting the maximum response length to 4,096 tokens are provided in Appendix D.

To test this hypothesis, we compare the dataset characteristics of the settings where SDPO outperformed GRPO against our experimental setup. As shown in Table 3, the Chemistry dataset, despite its large size, draws from only six main problem types that differ primarily in surface details rather than underlying structure. LiveCodeBench v6 contains diverse problems but only 131 in total, leading to repeated exposure during training with identical train/eval splits. In contrast, DAPO-Math-17k exposes the model to  14,000 distinct problems (78% of the 25,600 samples drawn over 100 steps, due to repeated sampling), spanning a broad, non-overlapping range of problem types, and evaluation is performed on unseen problem types.

To further investigate the interplay between task coverage and generalization, we vary the number of training questions $|\mathcal{D}|\in\{1,8,64,128,512\}$ from DAPO-Math-17k and train with both GRPO and SDPO. All experiments use Qwen3-8B (Thinking Mode OFF).

Self-distillation, where a single model acts as both teacher and student under asymmetric conditioning (Snell et al., 2022), has recently been widely adopted for LLM post-training. SDPO (Hübotter et al., 2026) conditions the self-teacher on its previously generated correct trajectories or rich environment feedback to provide dense credit assignment, while OPSD (Zhao et al., 2026) leverages ground-truth solutions as privileged information, achieving strong efficiency gains in both cases. Our analysis builds upon the SDPO codebase, and a detailed comparison with OPSD is provided in Appendix I. Liu et al. (2026); Penaloza et al. (2026); Ye et al. (2026) further utilize this paradigm in settings that incorporate additional signals, such as privileged information, self-generated hints derived from environmental interactions, or contextual guidance, with the goal of internalizing this external knowledge into the model’s parameters.

While these works demonstrate the effectiveness of self-distillation, they do not investigate when or why self-distillation may degrade reasoning performance, especially in scenarios where a model must solve problems entirely on its own, without interaction with the external environment, which is the central question of our work.

Kim et al. (2026) introduce an information-theoretic framework that identifies epistemic verbalization, the explicit externalization of uncertainty, as informationally necessary for robust reasoning. Our work answers a question this framework cannot readily explain: Under what conditions is this behavior encouraged or suppressed? And if epistemic verbalization is necessary, why does suppressing it lead to better performance in some domains, such as Chemistry, while degrading performance in others? We trace this to two factors: conditioning context richness (Section 3) and task coverage (Section 6), supported by a systematic analysis of recent on-policy self-distillation methods across various models with varying pretrained capabilities.

Several recent methods aim to shorten reasoning traces while preserving accuracy. GFPO (Shrivastava et al., 2026) filters training rollouts by length and token efficiency within enlarged sample groups, reducing GRPO’s length inflation by 46–85% on STEM and coding benchmarks. OPSDC (Sang et al., 2026) uses a conciseness instruction as privileged context, achieving 57–59% token reduction on MATH-500. ConPress (Deng et al., 2026) exploits multi-question contextual pressure to elicit self-compressed traces, yielding 30–60% reductions via SFT. Accordion-Thinking (Yang et al., 2026) trains models to produce periodic step summaries that can be discarded, achieving 3$\times$ throughput with a gap-vanishing phenomenon during RL training. CEEH (Luo et al., 2026) proposes difficulty-aware entropy regularization to prevent entropy collapse under length constraints.

Our analysis complements these efforts by showing that compression which suppresses uncertainty expression, even when producing correct answers, can harm OOD generalization, and that this effect is modulated by task coverage. Therefore, the key challenge is to filter out non-informative content while retaining epistemic expressions that enable iterative belief refinement, rather than blindly compressing the reasoning process.

For GRPO and SDPO training, we built upon the SDPO implementation (https://github.com/lasgroup/SDPO) and additionally incorporated the DAPO-Math-17k dataset. The original DAPO-Math-17k dataset uses the following prompt format:

Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.\n\n{question}\nRemember to put your answer on its own line after "Answer:".

We replaced this with a simpler format:

{question}\nPlease reason step by step, and put your final answer within \boxed{}.

as we observed that this format consistently yielded higher evaluation performance. For reward verification, we used the scoring function from the verl framework, which extracts the answer from the \boxed{} expression and verifies correctness via exact match followed by mathematical equivalence checking using math-verify, adapted from EleutherAI’s lm-evaluation-harness (Gao et al., 2024).

The GRPO and SDPO training hyperparameters are listed in Tables 5, 6, and 7. For the experiments on the relationship between task coverage and learning performance in Figure 9, we reduced the question batch size to 64 due to the use of smaller training questions.

We evaluated all models using the hyperparameters listed in Table 8, following the recommended settings from each model’s official documentation.$^{1}$ $^{,}$ $^{2}$

Table 9 summarizes the chat template formats used by several open-weight model families, each using distinct special tokens to delineate user and assistant turns. The placeholder {prompt} denotes the user message.

In our main experiments, as detailed in Table 5, we set the maximum training response length to 20,480 tokens to avoid prematurely truncating the long reasoning trajectories. Under a budget-constrained setting, we additionally examine how the training behavior of GRPO and SDPO changes when the maximum response length is reduced to 4,096 tokens.

As shown in Figure 10, under the 4,096-token constraint, SDPO achieves relatively high training scores early in training, whereas GRPO improves more gradually and eventually reaches comparable training performance. More importantly, GRPO consistently achieves higher performance on AIME24 and AMC23, while SDPO plateaus at lower performance. Under a strict response budget, SDPO’s tendency to produce shorter reasoning traces allows more responses to be completed within the available token budget, which likely contributes to its rapid initial gains. However, this early advantage does not translate into continued improvement in reasoning performance.

To examine the effect of various training hyperparameters on self-distillation behavior, we conduct additional experiments by varying the top-$k$ distillation parameter and the learning rate. As shown in Figure 11a, increasing top-$k$ from 100 to 256 yields no significant difference in training dynamics or final performance. Meanwhile, as illustrated in Figure 11b, reducing the learning rate from $1\mathrm{e}{-5}$ to $1\mathrm{e}{-6}$ slows the rate of degradation, but the model ultimately converges to the same reasoning behavior.

In Table 1 of Section 3, we compared the average number of ten epistemic tokens per response. Figure 12 further extends this analysis by examining how the average per-response count of each individual token changes under varying levels of conditioning information. When examining the per-token counts, all tokens exhibit a consistent trend:

where tokens such as wait, maybe, and perhaps are particularly prominent.

Following the analysis of DeepSeek-R1-Distill-Qwen-7B (DeepSeek-Distill-7B) in Section 3, we further compare epistemic token usage across three settings: DeepSeek-Distill-7B, Qwen3-8B (Thinking Mode: ON), and Qwen3-8B (Thinking Mode: OFF).

As shown in Figure 13, both DeepSeek-Distill-7B and Qwen3-8B (Thinking Mode: ON) produce substantially more epistemic tokens than Qwen3-8B (Thinking Mode: OFF). While the two thinking-enabled models share a similar tendency to express uncertainty, they differ in their preferred epistemic tokens. For instance, DeepSeek-Distill-7B frequently uses wait and employs perhaps and maybe at comparable rates, whereas Qwen3-8B uses perhaps relatively less and favors maybe. Qwen3-8B also uses alternatively and check far more than DeepSeek-Distill-7B, and overall embeds a greater amount of uncertainty within its reasoning.

Extending the discussion in Section 3, we also observe that Qwen3-8B generates far fewer epistemic tokens under solution-guided generation than under unguided generation. Across all three settings, Qwen3-8B (Thinking Mode: ON) produces the most epistemic tokens, followed by DeepSeek-Distill-7B, and then Qwen3-8B (Thinking Mode: OFF).

To provide a more in-depth analysis of the results in Figure 9 (Section 6.2), we compare the change in epistemic token counts relative to the base model on AIME24 across six training configurations: GRPO and SDPO, each with $|D|\in\{1,64,512\}$. Figure 14 shows that GRPO tends to increase epistemic token usage relative to the base model as $|D|$ grows, whereas SDPO reduces epistemic token usage, with smaller $|D|$ leading to a greater reduction.

In addition to analyzing epistemic tokens, we examine how the overall word distribution in model responses shifts across the base model, GRPO-trained model, and SDPO-trained model. We first extract English words from each response using regular expressions and construct a reference vocabulary $\mathcal{V}$ from all unique words appearing in the base model responses. For each word $w\in\mathcal{V}$ and model $m\in\{\text{base},\text{GRPO},\text{SDPO}\}$, we compute the average per-response frequency:

where $f(w,r)$ denotes the number of occurrences of word $w$ in response $r$. We then define the frequency shift $\Delta_{m}(w)=\mu_{m}(w)-\mu_{\text{base}}(w)$ to capture how much each word’s usage changes relative to the base model.

Using these frequency shifts, we compare the behavior of the predefined set of epistemic tokens $T$ (as defined in Section 3) against the rest of the vocabulary. Specifically, we compare $\{\Delta_{m}(w)\mid w\in T\}$ against $\{\Delta_{m}(w)\mid w\in\mathcal{V}\setminus T\}$ to determine whether the changes in epistemic token usage are disproportionately large relative to the overall shift in word distribution. This allows us to distinguish whether the training procedure specifically amplifies epistemic expressions or merely induces a uniform change across the vocabulary.

As shown in Figure 15, the disparity between the two panels confirms that the observed epistemic shifts are not byproducts of general vocabulary drift. Across all model-method pairs, the average per-word shift over the full vocabulary remains near zero ($|\Delta|<1$), whereas epistemic tokens exhibit shifts up to $30\times$–$40\times$ larger in magnitude (e.g., $-11.9$ and $-12.2$ for SDPO). This indicates that both training objectives selectively target epistemic expressions rather than uniformly perturbing the output distribution.

We employed epistemic markers as indicators of uncertainty expressions, as they serve as practical and interpretable cues for regions where uncertainty externalization is likely to occur. To more comprehensively capture diverse forms of uncertainty expression, we additionally leveraged GPT-5.4 (Singh et al., 2025) as an automated annotator. Specifically, GPT-5.4 was prompted to identify and count uncertainty expressions in each model-generated response. The prompt used for this annotation is shown below.

Table 10 presents representative examples of uncertainty expressions identified by the annotator. The majority of identified expressions coincide with the epistemic tokens examined in the previous analysis (e.g., maybe, perhaps, wait), while others manifest as multi-token phrases such as “I’m not entirely sure” that cannot be captured by single-token frequency alone. This confirms that epistemic markers serve as a reliable proxy for uncertainty externalization, while the LLM-as-Judge approach provides complementary coverage over more diverse expression forms.

Table 11 reports the average number of such expressions per response for each method. When evaluated through LLM-as-Judge annotation, SDPO again substantially reduces uncertainty expressions compared to both the baseline and GRPO, consistent with the token-level analysis. Figure 16 complements this with a box plot comparison across methods: notably, GRPO exhibits considerably greater variance than the baseline, suggesting that RLVR can amplify inconsistency in hedging behavior. In contrast, SDPO maintains consistently low counts with a narrow spread, confirming that on-policy self-distillation suppresses unnecessary hedging and self-doubt during reasoning.

As an extension of Figure 5 in Section 5.3, we additionally compare AMC23 evaluation scores and the change in epistemic token usage on AIME24. As shown in Figure 20, GRPO significantly increases response length and yields substantial performance gains on both benchmarks. In contrast, SDPO exhibits divergent trends: on AMC23, acc@16 increases from 0.67 to 0.73 while reducing response length by roughly half, whereas on AIME24, acc@16 slightly decreases from 0.25 to 0.23, with pass@16 dropping more substantially. Notably on AMC23, SDPO achieves approximately a 6-point improvement with shorter responses, whereas GRPO obtains a much larger improvement of around 36 points at the cost of considerably longer responses. Achieving large performance gains while maintaining a reasonable response length remains an open challenge.

To investigate whether our findings generalize to smaller models, we conducted the same on-policy self-distillation experiment on Qwen3-1.7B. As shown in Figure 21, our finding that self-distillation reduces uncertainty and makes outputs more concise, while simultaneously degrading mathematical reasoning performance, remains consistent across model sizes. Notably, the performance degradation is more severe for the 1.7B model (45.9%) compared to Qwen3-8B with Thinking Mode ON (12.1%) and DeepSeek-R1-Distill-Qwen-7B (40.7%). This suggests that the externalization of uncertainty may be particularly important for robust reasoning in smaller models, which possess more limited inherent capabilities.

In addition to DeepSeek-R1-Distill-Qwen-7B and Qwen3-8B, we further evaluate on-policy self-distillation on OLMo-3-7B-Instruct from a different model family. As shown in Figure 22, consistent with our previous analyses, SDPO also degrades reasoning performance on this model, with OOD evaluation scores falling below those of the base model. This confirms that our findings are not model-dependent but reflect robust characteristics of reasoning behavior across diverse model families.

In addition to the acc@16 scores for DeepSeek-Distill-7B and Qwen3-8B (Thinking Mode: ON) presented in Figures 3b, 3c, 4b, and 4c, we report pass@16 scores in Figure 23. The gap between GRPO and SDPO in pass@16 is larger for DeepSeek-Distill-7B than for Qwen3-8B, and more pronounced on the harder benchmark (AIME24) compared to AMC23.

We analyze examples of $\mathcal{D}_{\text{ug}}$ (Unguided responses) and $\mathcal{D}_{\text{sg}}$ (Solution-guided responses) introduced in Section 4. Below, we present the two responses generated for the same problem. As illustrated, the unguided response exhibits substantially more uncertainty and exploratory reasoning, frequently revisiting assumptions, considering alternative approaches, and self-correcting along the way. In contrast, the solution-guided response proceeds in a more concise and direct manner, arriving at each intermediate result without detours.

Problem. How many ways can $8$ mutually non-attacking rooks be placed on a $9\times 9$ chessboard so that all $8$ rooks are on squares of the same color? (Two rooks are said to be attacking each other if they are placed in the same row or column of the board.) Please reason step by step, and put your final answer within $\boxed{\phantom{0}}$.