Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving multimodal reasoning by optimizing policies against automatically checkable outcomes (Guo et al., 2025; Yang et al., 2025). In GRPO (Shao et al., 2024), a verifier scores each rollout and group-relative normalization turns these rewards into an outcome advantage $A_{\mathrm{ORM}}$ shared by every token. Such supervision is reliable at the trajectory level, but coarse at the token level: the same advantage is assigned to every token, regardless of which visual decision determines the answer. Consequently, RLVR can distinguish successful trajectories from unsuccessful ones without localizing the visual reasoning errors within them.

On-policy distillation (OPD) (Agarwal et al., 2024; Gu et al., 2023; Song and Zheng, 2026) addresses this limitation by evaluating student-sampled trajectories with a teacher and supplying dense token-level guidance. On-policy self-distillation (OPSD) (Zhao et al., 2026; Hübotter et al., 2026; Yang et al., 2026) further avoids dependence on an external teacher by using the policy itself under privileged information as the teacher, which for multimodal reasoning can be a zoomed-in view that exposes task-relevant evidence (Yuan et al., 2026). The resulting teacher–student log-probability gap $\delta_{t}$ provides a dense, token-level correction, yet it is not guaranteed to be attributable to the privileged visual evidence alone: it may also be contaminated by non-visual shifts induced by linguistic priors, response style, or the privileged view itself.

We formalize this contamination as privileged-correction contamination: the teacher–student correction mixes evidence-grounded guidance with residual effects that are not attributable to the visual evidence. Applying such a correction directly may reinforce a token even when the visual evidence argues against it, or suppress a token that the evidence supports.

To make this contamination concrete, Fig. 1 illustrates a single token-level diagnostic. For each generated token, we compare the teacher correction $\delta_{t}$ with the visual contrast $v_{t}$ between evidence-centered and random-masked teacher views, a probe of visual dependence introduced by OPD-V (Aniri et al., 2026). Fig. 1 (a) shows a conflicting case, in which the teacher suppresses a token ($\delta_{t}=-0.612$) that the visual contrast supports ($v_{t}=+0.421$). Such a correction should be discarded rather than applied, since the student already assigns the token a higher probability than the teacher does ($0.98$ vs. $0.53$): the teacher is lower here because the privileged view shifts its probabilities downward, not because the visual evidence refutes the token. Fig. 1 (b) summarizes the comparison over tokens by the signs of the two signals: agreeing signs indicate a correction consistent with the evidence, whereas opposite signs expose the two conflict cases, in which the teacher suppresses a visually supported token or promotes a visually refuted one. Measured with Qwen3.5-4B on our training data (§4.1), these conflicts affect $39.1\%$ of the generated tokens. Exposing these conflicts, however, does not remove them. What the diagnostic calls for is visual attribution in a strict sense: an explicit mechanism that checks whether the direction of each correction agrees with the visual evidence.

Motivated by this diagnosis, we design Visual-Attribution Gating (VAG). Rather than distilling every privileged correction, VAG validates the direction of each one against the visual contrast: the correction $\delta_{t}$ is admitted only when it agrees in sign with $v_{t}$, that is $v_{t}\delta_{t}>0$, which keeps both supporting and refuting corrections and removes only the conflicting residuals exposed by the diagnostic. Furthermore, VAG operates on top of RLVR rather than in place of it: the verifier-derived advantage remains the optimization anchor, and self-distillation contributes only the corrections that pass the gate. Taken together, we propose reinforcement learning with visually attributable on-policy self-distillation, a new evidence-consistent learning paradigm.

Fig. 2 compares our paradigm with representative on-policy learning paradigms for multimodal credit assignment along three properties: trajectory correctness, dense token correction, and visual attribution. RLVR (Fig. 2 (a)) supplies only the outcome advantage $A_{\mathrm{ORM}}$, which anchors optimization to trajectory correctness but assigns the same scalar to every token. RLVR with distillation (Fig. 2 (b)) keeps this anchor and modulates each token by $\delta_{t}$ without verifying where that correction comes from, so the privileged signal it distills may be contaminated (Lin et al., 2026; Yang et al., 2026). Visual contrastive distillation (Fig. 2 (c)) instead uses visual contrast $v_{t}$ to select only visually sensitive tokens (Aniri et al., 2026). Yet this selection does not verify whether an update follows the visual evidence, so the corrections applied to the selected tokens remain contaminated, while the verifier-grounded anchor is lost as well.

VAG (Fig. 2 (d)) is the only paradigm that attains all three properties at once: it keeps $A_{\mathrm{ORM}}$ as the anchor, preserves the dense correction $\delta_{t}$, and admits that correction only when $v_{t}$ supports its direction. This coverage comes at almost no cost. VAG needs no learned contamination estimator and no auxiliary distillation objective, and relative to a dual-view OPSD baseline that already evaluates both teacher views, it adds only an elementwise sign test and no additional teacher forward pass. Across six visual reasoning benchmarks, VAG consistently improves over matched GRPO and OPSD baselines and surpasses much larger open-source models (e.g., Qwen3.5-397B), leading closed-source models (e.g., GPT-5.4), and agentic “Thinking-with-Images” methods on tasks that require dense local evidence. On holdout tasks beyond the training distribution, it retains general visual understanding and reasoning ability, indicating that the gains do not come at the cost of forgetting, while controlled ablations isolate the contribution of the sign-agreement gate.

Our contributions are summarized as follows:

We introduce reinforcement learning with visually attributable on-policy self-distillation to mitigate privileged-correction contamination. Our Visual-Attribution Gating anchors optimization to verifier-derived trajectory advantages and accepts token-level corrections only when consistent with the contrast between evidence-present and evidence-degraded teacher views, preserving supportive and refuting visual evidence while suppressing inconsistent residuals. Across six public benchmarks, VAG consistently outperforms state-of-the-art methods, demonstrating that visual grounding verification unifies outcome verification, token-level self-distillation, and visual attribution for multimodal policy optimization.

To ensure that every method is compared under identical conditions, we adopt a single evaluation protocol and apply it uniformly to VAG and to all reproduced baselines. Table 7 lists the number of evaluation samples and the scoring rule used for each benchmark.

Following OPD-V (Aniri et al., 2026), scoring proceeds in two stages. We first apply deterministic answer matching, which resolves the majority of responses without any model in the loop. For the multiple-choice benchmarks, a response counts as correct if it matches either the answer text or the option letter, after normalizing surface variations such as capitalization, surrounding punctuation, and common answer prefixes. Only the responses that remain unresolved are passed to the second stage, where a Qwen3.5-series (Qwen Team, 2026) model acts as a binary Yes/No judge and decides whether the response is consistent with the verified answer.

Both stages are necessary for the benchmarks we evaluate. Responses on fine-grained perception tasks frequently state the correct answer inside a longer justification, which deterministic matching alone would count as wrong; delegating every response to the judge instead would introduce scoring variance unrelated to the method under evaluation. Calling the judge only on the unresolved responses therefore keeps most decisions exact while still absorbing benign paraphrasing. Since the same protocol, judge model, and prompts apply to every entry in Table 1, the reported differences reflect the trained policies rather than the evaluation.

The evaluation in our main table (Table 1) strictly follows the legacy evaluation protocol of OPD-V Aniri et al. (2026), in which the trained model itself serves as the judge. For a fair comparison, we further re-evaluate several released models under the evaluation protocol of Vision-OPD (Yuan et al., 2026), using GPT-OSS-120B (OpenAI, 2025b) as the judge model, and report the results in Table 6 below. As shown, our model achieves the best accuracy at both the 4B and 9B scales under this setting.

Fig. 6 presents one complete training instance: the original image observed by the student, the two teacher images derived from it, and the textual prompt used in each pass.

The three image conditions differ only in the visual evidence they expose. The student receives the original, unmodified image and must therefore locate the relevant detail on its own. Both teacher images are constructed from this same image. The positive teacher receives the evidence-centered crop defined by the annotated bounding box, so the decisive detail occupies a far larger fraction of the visual field. The negative teacher receives the same crop after random masking, which weakens the local evidence while preserving the surrounding spatial context.

The textual prompt is assembled from a fixed set of fields: an image placeholder, the question, a visual focus instruction that directs the model to the annotated region, the answer choices, and an output-format instruction that requests only the option letter. The ground-truth answer is consumed by the verifier and is never shown to the model. Constraining the output format yields two additional benefits. First, it reduces the fraction of responses that require the Yes/No judge at evaluation time. Second, it leaves little room for stylistic variation between the teacher and student branches, so the gap $\delta_{t}$ is less likely to reflect differences in wording instead of differences in visual evidence.

Because the question, answer choices, output-format instruction, and the student-generated prefix are identical across the three passes, the only controlled difference between the two teacher branches is the availability of the target evidence. This is what allows the visual contrast $v_{t}$ in Eq. 7 to be interpreted as a statement about visual evidence rather than about prompt or sampling differences. Both teacher views are used only during training; at inference time the deployed policy observes the original image alone and requires no privileged input.

Algorithm 1 states the full VAG training loop and makes explicit the order in which the outcome signal and the token-level signal are combined.

Each iteration begins with rollout collection. For each input $x=(q,I)$, the student samples $G$ on-policy responses $\hat{\mathbf{y}}^{(i)}$. The verifier assigns each response a reward $R^{(i)}$, which is normalized within the group to obtain $A_{\mathrm{ORM}}^{(i)}$ as in Eq. 2. This advantage is shared by all tokens of the response. The two teacher branches then replay each sampled response using the same student-generated prefix $\hat{\mathbf{y}}^{(i)}_{<t}$. The sampled-token probabilities $p_{S}^{(i)}(\hat{y}_{t})$, $p_{T+}^{(i)}(\hat{y}_{t})$, and $p_{T-}^{(i)}(\hat{y}_{t})$ are defined in Eqs. 3, 4, and 6, respectively.

From these probabilities, the algorithm computes the dense teacher correction $\delta_{t}^{(i)}$ and visual evidence contrast $v_{t}^{(i)}$ using the stop-gradient log-ratios in Eqs. 5 and 7. The attribution gate $g_{t}^{(i)}$ retains corrections only when $\delta_{t}^{(i)}v_{t}^{(i)}>0$ (Eq. 8). The gate checks whether the correction and visual contrast have the same sign, rather than comparing their magnitudes. Thus, only corrections whose direction agrees with the visual evidence are retained. Admitted corrections are bounded by $\tanh(\delta_{t}^{(i)}/\tau)$ and weighted by $\eta\,\widetilde{v}_{t}^{(i)}$, then added to $A_{\mathrm{ORM}}^{(i)}$ to obtain $\widetilde{A}_{t}^{(i)}$ as in Eq. 10.

Two properties of the loop are worth noting. First, all teacher-derived quantities are detached, so gradients reach the student only through $\rho_{t}^{(i)}(\theta)$ in $\mathcal{L}_{\mathrm{VAG}}(\theta)$; the teacher acts as a signal source rather than as a differentiable path. Second, when $g_{t}^{(i)}=0$, the modulated advantage satisfies $\widetilde{A}_{t}^{(i)}=A_{\mathrm{ORM}}^{(i)}$. Thus, rejecting a correction preserves the token’s verifier-derived supervision. After each student update, the teacher parameters are updated by EMA to track the student policy without separate teacher training.

Fig. 7 reports two statistics throughout training: the mean verifier reward within each rollout group (left) and the mean response length (right). The reward curve shows a clear upward trend, increasing from approximately $0.4$–$0.5$ over the first twenty steps to $0.65$–$0.75$ in the second half of training, before gradually stabilizing. This trend suggests that our training strategy consistently improves the policy while maintaining stable optimization in the later stages of training.

The response-length curve exhibits a different pattern. It oscillates within a relatively stable range throughout training and ends at a level comparable to its initial value, with no sustained growth. This is important because response-length inflation is a common confound in reinforcement learning with verifiable rewards: a policy may increase its pass rate by producing longer responses that enumerate more candidate answers. In such cases, an increasing reward curve may reflect a change in response strategy rather than genuine improvement in visual perception.

Taken together, these two curves support the interpretation of our quantitative results. Gating corrections by visual attribution sharpens token-level credit assignment without destabilizing the policy’s generation behavior, consistent with retaining $A_{\mathrm{ORM}}$ as the optimization anchor rather than replacing it with dense teacher supervision.

Figs. 8, 9, and 10 present qualitative comparisons between VAG and OPD-V (Aniri et al., 2026) on the test set. Fig. 8 covers fine-grained attribute recognition and spatial localization. In the attribute case, the question asks for the color of a helmet that occupies a small portion of the frame. VAG describes the region as orange-red and maps it to the closest available option, red, matching the ground truth, whereas OPD-V reports the helmet as predominantly yellow. The two answers differ in what the model reads off the region rather than in how it reasons afterwards, which is precisely the failure mode that dense but unattributed corrections are least able to repair. In the localization case, the question asks where a green region with two oval patterns appears, and VAG places it in the bottom-left corner in agreement with the annotation.

Fig. 9 shows an object-part recognition case in which the model must name the car part visible on the right side of the image. VAG enumerates the options against the visible content, rules out the headlight and the side mirror as absent from the frame, and selects the taillight, which matches the ground truth. Fig. 10 is the most demanding case, because the visual reading must survive several arithmetic steps. The question asks for the average of two bars in a thirty-bar chart whose x-axis is labeled only every fifth item. VAG initially misaligns the labels with the bars and obtains a value that is absent from the answer choices; it then re-examines the axis, re-derives the bar-to-item mapping, reads the two bars as $1$ and $11$, and computes their average as $6$, which is correct. OPD-V instead commits to unsupported bar values and reports $9$. The example shows that the benefit of attribution is not confined to perception: an incorrectly read value propagates into every subsequent step, so reading the chart correctly is a precondition for the arithmetic to be meaningful at all.

Across these cases the pattern is consistent with the aggregate results in Table 1. The largest gains appear where the answer hinges on a localized detail that must be read correctly before any further reasoning can help.