Flow-OPD: On-Policy Distillation for Flow Matching Models

The success of RL-based alignment in large language models has recently inspired reinforcement learning for text-to-image (T2I) generation. Early methods such as DDPO Black et al. (2024), DPOK Fan et al. (2023), and ImageReward/ReFL Xu et al. (2023) formulate diffusion generation as policy optimization with rewards for aesthetics, human preference, or text-image alignment, while Diffusion-DPO Wallace et al. (2023) aligns diffusion models using preference pairs. More recent GRPO-style methods extend RL to modern visual generators, including those for flow models Liu et al. (2025b); Xue et al. (2025), and AR paradigms Yuan et al. (2025); Zhang et al. (2025b); Ma et al. (2025); Zhang et al. (2025a); Ma et al. (2026). However, T2I generation requires multiple rewards to cover aesthetics, alignment, fidelity, and compositional correctness. Existing solutions remain hard to control: DanceGRPO Xue et al. (2025) directly mixes rewards such as HPS and CLIP, often trading off one metric against another; Flow-GRPO Liu et al. (2025b) uses staged reward/dataset curricula, making results sensitive to ordering and stage design; and GDPO Liu et al. (2026) shows that GRPO Guo et al. (2025a) may suffer from reward-normalization collapse under multi-reward settings. This motivates a more controllable multi-reward coordination mechanism.

Traditional offline distillation relies on fixed datasets and fails to adapt to the student’s evolving trajectory. In contrast, On-Policy Distillation (OPD) dynamically couples the teacher’s supervisory signal with the student’s exploration space. In the LLM domain, OPD has seen rapid development: GKD Agarwal et al. (2024) established the canonical framework to mitigate exposure bias; MiniLLM Gu et al. (2024) and DistiLLM Ko et al. (2025) introduced Reverse and Skewed Kullback-Leibler (KL) to refine mode-seeking and optimization stability; G-OPD Yang et al. (2026) unified OPD under KL-constrained RL theory; Entropy-Aware OPD Jin et al. (2026) preserves diversity through adaptive divergence functions; Fast OPD Zhang et al. (2026) significantly accelerates computation via prefix truncation; and PACED Xu et al. (2026) implements a competence-aware curriculum based on gradient signal-to-noise analysis. Despite these LLM advancements, OPD remains underexplored in visual Flow Matching models, which require dense supervision within high-dimensional velocity fields. We propose Flow-OPD, the first systematic migration of on-policy distillation to Flow Matching, utilizing multi-teacher dense supervision to overcome the reward sparsity bottleneck.

Flow Matching (FM) maps a noise distribution $p_{0}$ to data $p_{\text{data}}$ via an ODE $\text{d}\mathbf{x}_{t}=v_{t}(\mathbf{x}_{t},t)\text{d}t$. Under the Optimal Transport (OT) formulation Lipman et al. (2022), the path is $\mathbf{x}_{t}=(1-t)\mathbf{x}_{0}+t\mathbf{x}_{1}$, and the model $v_{\theta}$ learns the constant velocity $(\mathbf{x}_{1}-\mathbf{x}_{0})$ via:

Following Flow-GRPO Liu et al. (2025a), we conceptualize the discretized ODE integration as a sequential Markovian denoising process. By formulating each transition $\mathbf{x}_{t}\to\mathbf{x}_{t+\Delta t}$ as a Markovian state step, this perspective bridges continuous generative dynamics with reinforcement learning, defining a formal trajectory for step-wise policy optimization.

Knowledge distillation aims to compress teacher capabilities into a student model by minimizing their output divergence. To mitigate distribution shift, on-policy distillation (OPD) Lu and Lab (2025) requires the student $f_{\theta}$ to generate trajectories $\tau\sim p_{\theta}(\tau)$ under the guidance of real-time teacher supervision. For Autoregressive (AR) models, this optimization is formulated as minimizing the Reverse Kullback-Leibler (KL) divergence between the student and teacher distributions:

By aligning the model on its own generated distribution, OPD effectively suppresses exposure bias and ensures robust generalization in interactive or iterative generation tasks.

Standard FM relies on offline reconstruction, fundamentally limiting performance to static dataset quality and failing to optimize non-differentiable preferences. GRPO Guo et al. (2025a); Liu et al. (2025a); Xue et al. (2025) overcomes this via online exploration. By actively sampling $G$ outputs from its current policy $\pi_{\theta}$, it evaluates self-generated states using a Group Relative Advantage, $A(\mathbf{x}_{1}^{(i)})=(r(\mathbf{x}_{1}^{(i)})-\mu)/\sigma$. The policy gradient is then explicitly driven by these online experiences:

This continuous exploration of its own dynamic distribution enables the model to discover novel, high-reward trajectories, successfully breaking the performance ceiling of offline SFT.

Despite its target-specific efficacy, single-reward GRPO incurs severe degradation in orthogonal capabilities (Fig. 2). This catastrophic forgetting stems from unconstrained gradient interference driven by sparse scalar rewards within a shared parameter space $\theta$.

For a parameter update $\Delta\theta$ driven by a target task $\mathcal{T}_{1}$ with advantage $A_{1}$, the collateral impact on an unmonitored capability $\mathcal{T}_{k}$ ($k\neq 1$) can be approximated via first-order Taylor expansion:

In high-dimensional spaces, divergent task gradients frequently conflict ($\langle\nabla_{\theta}\mathcal{J}_{k},\nabla_{\theta}\mathcal{J}_{1}\rangle<0$). Lacking supervisory signals for $\mathcal{T}_{k}$, the optimizer aggressively exploits these unmonitored degrees of freedom to maximize $A_{1}$, dismantling pre-trained synergies and leading to manifold collapse. This prompts a natural question: Can we resolve this degradation by simply mixing multiple datasets and rewards for joint optimization?

To explore the feasibility of mix training approach, we conduct a controlled empirical experiment on Stable Diffusion 3.5 Medium (SD-3.5-M) Esser et al. (2024). Following Flow-GRPO, we progressively stack four distinct reward functions: GenEval, OCR, PickScore, and DeQA. As demonstrated in Table 4.3, mixing scalar rewards fails to construct a stable cognitive foundation.

While the initial reward (+GenEval) succeeds, subsequent additions trigger catastrophic forgetting (e.g., +OCR degrades GenEval by 5%). This corroborates our hypothesis of Gradient Interference ($\langle\nabla_{\theta}\mathcal{J}_{i},\nabla_{\theta}\mathcal{J}_{j}\rangle<0$). Compressing multi-dimensional conflicts into a scalar advantage forces a zero-sum game; for instance, accommodating aesthetic stylization (PickScore) aggressively overwrites precise geometric representations. Consequently, scalar reward mixing is fundamentally unscalable due to this sparse Information Bottleneck. To avoid parameter cannibalization, we require a supervisory signal that is simultaneously on-policy (maintaining exploration) and densely uncoupled (preventing interference). Inspired by Multi-Teacher On-Policy Distillation (OPD) in LLMs, we propose Flow-OPD. This framework seamlessly introduces the multi-teacher paradigm into continuous Foundation Models, achieving active on-policy exploration guided by dense supervision.

To ensure a stable initialization $\theta_{0}$ and prevent trajectory divergence during early rollout, we explore two cold-start strategies: SFT-based and model-merging initialization. Our SFT protocol follows Flow-GRPO but utilizes trajectories sampled from specialized teachers, ensuring the student inherits expert-level knowledge distributions from the outset. Alternatively, model merging superposes the anisotropic priors of divergent teachers into a unified parameter state. This "merging-as-initialization" approach positions the student in a high-competence region of the loss landscape, where multi-task synergies are already nascent, providing a robust foundation for subsequent distillation.

Following Flow-GRPO Liu et al. (2025a), we evaluate our method on four tasks: GenEval Ghosh et al. (2023), OCR Chen et al. (2023), PickScore Kirstain et al. (2023), and DeQA You et al. (2025). We adopt the official checkpoints as expert teachers for the first three tasks. The DeQA teacher is specifically trained across the three datasets by blending DeQA and PickScore rewards at a 4:6 ratio. All training and test data strictly follow the Flow-GRPO splits. Training is executed on 4 nodes ($8\times\text{H800}$ GPUs each), while evaluation is conducted on a single $8\times\text{H800}$ node.

We primarily evaluate Flow-OPD against two categories of baselines: (1) Monolithic-Reward GRPO, denoted as GRPO-[reward name], where the model is fine-tuned using Flow-GRPO on a single reward objective; (2) Hybrid-Reward GRPO, denoted as GRPO-Mix, which employs a weighted reward combination with a fixed ratio of GenEval : OCR : PickScore = 3 : 1 : 1. These baselines serve to highlight the limitations of conventional scalar-based alignment when scaling to multi-dimensional expert capabilities.

The quantitative results in Table 2 demonstrate that Flow-OPD consistently matches or surpasses the specialized teacher models across all benchmarks, particularly in text rendering and DeQA image quality. Crucially, it resolves the severe cross-domain interference inherent to specialization (e.g., the PickScore teacher’s GenEval score dropping to 0.51) and overcomes the optimization bottlenecks of sparse-reward multi-task GRPO. By leveraging dense multi-expert supervision, Flow-OPD seamlessly consolidates diverse expertise without capability degradation.

Qualitative results in Fig. 3) show that Flow-OPD achieves an optimal multi-task trade-off, balancing high prompt fidelity with superior visual aesthetics. Remarkably, Flow-OPD succeeds in certain edge cases where all individual teachers fail, a phenomenon we term Teacher-Surpassing. We hypothesize this emergent superiority stems from knowledge cross-pollination within the latent flow manifold. While individual teachers are constrained by domain-specific biases, simultaneous dense guidance forces the student to learn a more holistic, smoothed representation. This collective supervision bridges epistemic gaps, enabling the student to synthesize novel trajectories that ultimately surpass any single supervisor.

Except for $\beta$, GRPO hyperparameters are fixed across tasks. We use a sampling timestep $T=10$ and an evaluation timestep $T=40$. Other settings include a group size $G=24$, an noise level $a=0.7$ and an image resolution of 512. The MAR KL ratio $\beta$ is set to 0.02. We use Lora with $\alpha=64$ and $r=32$.

Regarding Qwenvl Score, we adapt the prompt used in Flow-GRPO [16]. The prompt is shown in Fig. 6. We use Qwen3-30B-A3B-Instruct-2507.

More qualitative results are shown in Fig. 7,  8 and  9. Our approach not only ensures precise content generation but also delivers superior image quality and coherent structural layouts. By achieving stronger alignment with human preferences, Flow-OPD demonstrates significant novelty in bridging functional accuracy with aesthetic excellence.

DiffusionNFT [49] introduces an online reinforcement learning framework that directly integrates reward feedback into the forward diffusion process, enabling effective policy optimization during the noise-injection phase. Despite achieving competitive benchmark scores, DiffusionNFT exhibits several critical limitations. First, it is fundamentally incompatible with Classifier-Free Guidance (CFG), which severely bottlenecks its performance upper bound. Second, it suffers from pronounced reward hacking. As illustrated in Fig. 10 , while the model correctly generates the targeted text and ’sunset’ elements, it simultaneously hallucinates malformed hands and extraneous objects (e.g., oranges), accompanied by severe over-smoothed, plastic-like textural artifacts. Current standard benchmarks largely overlook these localized structural and aesthetic failures. To address this evaluation blind spot, we employ the Qwenvl Score for a more comprehensive assessment. By leveraging continuous-time dense multi-expert supervision and task-style decoupling, Flow-OPD effectively circumvents these reward hacking behaviors, achieving significantly higher Qwen-VL scores than DiffusionNFT. These findings also underscore a pressing need within the community to develop more robust, fine-grained evaluation paradigms for text-to-image generation.

Despite the superior performance of Flow-OPD across both subjective and objective benchmarks, certain limitations persist. A primary constraint is the performance ceiling imposed by teacher models. As illustrated in Fig. 11, when specialized teachers fail to synthesize semantically correct images, these inaccuracies are propagated through the dense supervisory signals. Such erroneous guidance introduces noise into the distillation objective, ultimately hindering the student’s ability to transcend the inherent limitations of the teacher ensemble. Another inherent limitation is the requirement for architectural homogeneity between the teacher and student models to facilitate fine-grained, step-wise supervision. Looking forward, we aim to explore the broader potential of Flow-OPD through several promising directions, including: (1) Co-evolutionary Distillation, where teachers and students iteratively refine each other; (2) Self-Distillation mechanisms to boost performance without external teachers; and (3) Cross-Vocabulary Distillation to bridge the gap between heterogeneous model architectures.

The primary contribution of this work is the enhancement of functional reliability in AI-generated content. By improving layout coherence and OCR accuracy, Flow-OPD can significantly benefit professional fields such as automated graphic design, educational content creation, and assistive technologies for the visually impaired. Furthermore, our Multi-Teacher paradigm promotes a more balanced optimization objective, which mitigates the "winner-takes-all" bias inherent in single-reward reinforcement learning, potentially leading to more diverse and representative generative systems.

Despite these benefits, the increased proficiency in generating high-quality, instruction-following images could be misused for the creation of sophisticated disinformation or deceptive visual content. Although our model inherits the safety filters of its foundation model, the improved structural realism might be exploited to generate more convincing fake documents or misleading social media assets. To mitigate this, we advocate for the integration of digital watermarking and provenance tracking (e.g., C2PA) in downstream applications. Additionally, like all large-scale generative models, there is a risk that the specialized teachers may harbor latent biases present in their training data, which could be inadvertently distilled into the student model. We encourage the community to employ bias-detection benchmarks alongside our framework to ensure equitable performance across all demographics.