Ling and Ring 2.6 Technical Report Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale

Basic Configuration. We set the number of Transformer layers to 80 and the hidden dimension $d$ to 8,192. Each layer employs 64 attention heads with a head dimension of 128. The model uses a vocabulary size of 157,184 and supports a maximum context length of 262,144 tokens via RoPE (Su et al., 2024) positional encoding with $\theta=6{,}000{,}000$. We employ SiLU as the activation function and RMSNorm with $\epsilon=10^{-6}$ for layer normalization. The rotary dimension is set to 64, i.e., Partial RoPE is applied to a subset of head dimensions.

Mixture-of-Experts. Following Ling-2.0, Ling-2.6-1T-base employs a fine-grained MoE architecture for Feed-Forward Networks (FFNs), which sets routed experts and shared experts. Each MoE layer consists of 1 shared expert and 256 routed experts, where the intermediate hidden dimension of each routed expert is 2,048. Among the routed experts, 8 experts are activated for each token. We employ the auxiliary-loss-free load balancing strategy with expert bias enabled, where the router operates in FP32 precision with sigmoid scoring. We adopt a grouped routing strategy with $n_{\text{group}}=8$ groups and top-4 group selection, and the routed output is scaled by a factor of 2.5 with normalized top-$k$ probabilities. For the first 4 Transformer blocks, we use dense FFN layers with an intermediate size of 18,432 rather than MoE layers.

We investigate the optimal ratio of Lightning Attention to MLA layers through scaling law experiments under equal FLOPs constraints. We define the layer group size $M$ such that each group contains 1 Full Attention (MLA) layer and $M-1$ Linear Attention layers, and compare configurations with $M\in\{2,4,8,16\}$. Figure 3 presents the scaling law curves, and Table 2 summarizes the results.

The $M=8$ configuration (7:1 Linear-to-Full ratio) achieves the best scaling performance while providing substantial inference cost savings, representing the optimal balance between model quality and efficiency. We also observe that as the total FLOPs budget scales larger, the proportion of Linear Attention layers can be moderately increased without degradation. Based on these results, we select the 7:1 ratio for Ling-2.6-1T-base.

As the context length increases, the attention mechanism emerges as the dominant computational bottleneck. We observe that at context windows exceeding 32K tokens, the computational cost shifts decisively from MoE layers to attention layers, and at 256K+ tokens, Full Attention (GQA) becomes the absolute performance bottleneck. To address this, we design a hybrid attention architecture that replaces a subset of GQA layers with Lightning Attention, while converting the remaining Full Attention layers from GQA to MLA for KV cache compression. This subsection describes the two attention components and the structural compatibility solutions required for MLA conversion.

Lightning Attention Conversion. We replace a portion of GQA layers with linear Lightning Attention layers, following the Ring-flash-linear-2.0 (Team et al., 2025a). During conversion, the GQA dimensions are expanded to standard Multi-Head Attention (MHA) by augmenting the $W_{qkv}$ projection along the head dimension. The newly introduced parameters are randomly initialized, and additional gating parameters $W_{\text{gate}}$ and gating normalization $\gamma_{\text{gate}}$ are introduced for the linear attention mechanism. During this initial conversion stage, we retain QK Norm and Partial RoPE from the original architecture to stabilize training, improve FP8 training compatibility, and enhance robustness at long-context windows.

MLA Conversion. MLA achieves extreme KV cache compression compared to GQA by projecting key-value pairs into a low-rank latent space. We validated this conversion through ablation experiments at two scales: a mini configuration (16B total, 1.3B activated) and a flash configuration (100B total, 5B activated). In both settings, replacing all GQA layers with MLA and training on 700B tokens followed by 600B mid-training tokens yielded performance that ultimately surpassed the original GQA baseline. This result held consistently across both the pure MLA and the hybrid Linear+MLA architectures.

However, directly converting Ling-2.0’s attention to MLA raises two structural incompatibilities that must be resolved.

QK Norm Incompatibility. QK Norm is a nonlinear operation that prevents the KV weight matrix absorption required for efficient MLA inference. We resolve this by removing QK Norm prior to the MLA conversion. Leveraging the mathematical properties of RMSNorm, we approximately fuse the QK Norm parameters into the query and key projection weights. Specifically, for a dataset of $N$ calibration samples, we compute the per-dimension statistics of the query and key outputs $q_{ij}$ and $k_{ij}$ (for sample $i$ and dimension $j$ within head dimension $d$), and derive corrected projection weights $\hat{W}_{q}$ and $\hat{W}_{k}$ that absorb the effect of the QK Norm parameters $\gamma_{Q}$ and $\gamma_{K}$:

This calibration-based fusion effectively removes the nonlinear QK Norm while preserving its normalizing effect, enabling subsequent MLA weight absorption.

Positional Encoding Incompatibility. TransMLA (Meng et al., 2025) natively supports Full RoPE, whereas Ling-2.0-1T-base employs Partial RoPE, in which only a subset of head dimensions receive rotary positional encoding. We address this by decoupling the RoPE module: the dimensions affected by RoPE are separated from unaffected dimensions, PCA-based weight rotation is applied exclusively to the RoPE-affected dimensions, and the results are concatenated to reconstruct the full representation. This decoupled treatment enables correct TransMLA conversion while preserving the Partial RoPE structure of Ling-2.0.

Agentic Corpus. Agentic corpus represents a paradigm shift in pre-training data, moving beyond static text toward data that captures the complete information flow, decision pathways, and environmental dynamics an agent experiences during live deployment (Zeng et al., 2026b). The corpus features an exceptionally high diversity of tasks, spanning two primary domains: agentic tool use and agentic coding. Specifically, we construct a broad spectrum of tool-use tasks by leveraging over 500 real-world MCP environments encompassing more than 3,000 distinct tools; alongside this, we synthesize large-scale, diverse coding tasks integrated with bash commands, web-based QA, and relevant software repositories. Subsequently, we employ model-based quality filtering to ensure the relevance, logical coherence, and appropriate difficulty of the tasks. This is followed by the generation of agentic trajectories using diverse teacher models and various interaction scaffolds, incorporating rigorous rule-based and model-based verification to guarantee high-quality trajectories.

Long-Context Corpus. To extend the context window to 256K and mitigate the scarcity of high-quality ultra-long corpora in naturally occurring data distributions, we jointly emphasized targeted retrieval and data synthesis, thereby constructing an ultra-long corpus covering multiple domains, including mathematics, complex web parsing, long-document summarization, retrieval-augmented generation (RAG) fusion, and multi-hop reasoning.

In parallel, we further enhanced the quality assurance pipeline. Specifically, by introducing a deep detection framework that integrates rule-based and model-based methods, we were able to effectively identify and remove common defects in ultra-long texts, including excessive self-repetition, structural collapse, and long-range semantic hallucinations. This pipeline was also applied to re-clean and refine a subset of the existing long-context data.

Web Corpus. To improve the general knowledge of model during pre-training, we built an efficient general feature engineering pipeline on top of a wide-table infrastructure and fastText (Joulin et al., 2016), enabling hourly model iterations and daily feature updates. From a large-scale web index, we perform targeted STEM data recall, QA-oriented retrieval, and closed-loop retrieval via an internal search engine, and further rewrite the retrieved web content into textbook-style materials with reduced noise and stronger logical structure, effectively alleviating the knowledge coverage limitations of Common Crawl. To further improve the model factual question-answering capability, we observed that dispersed facts embedded in long-form text are difficult for models to learn effectively. To address this issue, we developed an atomic fact construction pipeline that converts Wikipedia articles into standalone propositions and structured triplets, together with a multi-strategy QA synthesis pipeline for fine-grained knowledge augmentation. Experimental results show that this approach substantially reduces the difficulty of factual memorization and effectively enhances the model factual question-answering ability.

Math & Code Corpus. Mathematical and programming corpora remain the core components of our pre-training data. For the current release, we have systematically augmented and refined these specific corpora via rephrasing corpus, and mining a broader spectrum of high-quality web content, comprehensive literature, and source code repositories.

Multilingual Corpus. To mitigate performance disparities across commonly-used languages in state-of-the-art multilingual models, we prioritize the expansion of monolingual corpora coverage for 21 languages, with focused additions in Arabic, Japanese, and Hindi. Regarding data composition, we incorporate 1.1T tokens from the open-source corpora Fineweb2 (Penedo et al., 2025) and Fineweb2-hq (Messmer et al., 2025) (covering 8 key languages) and introduce approximately 70B tokens of synthetic web code data to enhance domain-specific representation capabilities.

During the migration pre-training and continue pre-training, multilingual data receive a 4% allocation of all datasets, internally stratified as 70% web corpora—spanning Romance, Germanic, Slavic, and Southeast Asian language families—and 30% capability-oriented mixes: mathematics , code, exams, synthetic web, and parallel corpora. And for mid-training, we strip away general web datasets, keeping only the high-value capability categories—mathematical reasoning, code, exams, synthetic data, and parallel corpora—to curate a low-noise, competency-focused distribution.

Ling-2.6-1T-base inherits the MoE backbone from Ling-2.0-1T-base and further introduces a hybrid linear attention architecture. Key architectural parameters scale with model size; see Table 1 for details. For training hyper-parameters, we first re-establish the Ling scaling laws on the hybrid linear attention architecture to determine the appropriate learning rate and batch size given the total training budget. We adopt the auxiliary-loss-free load-balancing strategy, setting the bias-update rate $\gamma{=}0.001$ during continue pre-training, which is then reduced to 0.0001 for mid-training. The MTP loss weight is set to 0.1. All other hyper-parameters remain consistent with Ling-2.0. We continue to use the WSM scheduler from Ling-2.0: a linear warmup to a peak learning rate, followed by a constant phase until training concludes; the final annealing effect is achieved through checkpoint merging.

As illustrated in Figure 4, the pre-training processes approximately 9.6T tokens across three stages: Migration Pre-Training, Continue Pre-Training, and Mid-Training.

Migration Pre-Training. Starting from the Ling-2.0 base checkpoint, this stage smoothly transitions the architecture from GQA-based Softmax Attention to the hybrid linear attention with MLA, while minimizing performance degradation. It proceeds in four steps over approximately 400B tokens. First, we perform Lightning Attention Conversion: a subset of GQA layers is converted to Lightning Attention by expanding $W_{qkv}$ parameters along the head dimension and initializing new gating parameters $W_{\text{gate}}$ and $\gamma_{\text{gate}}$, while retaining QK Norm and Partial RoPE for training stability. Next, a Linear Warmup phase freezes all parameters except $W_{qkv}$ and QK Norm weights. A learning rate warmup with a small data budget brings the loss back to pre-conversion levels, allowing the randomly initialized linear attention parameters to align with the pre-trained representations. The third step is MLA Conversion, which involves three sequential operations: (1) QK Norm removal via the calibration-based fusion described in Section 2.1.3. At mini scale, this increases test perplexity from 6.65 to 11.13, a controlled and recoverable degradation; (2) a brief partial-parameter training phase (unfreezing only $W_{qkv}$, QK Norm, $W_{\text{gate}}$, and $\gamma_{\text{gate}}$) with learning rate warmup to mitigate the QK Norm removal impact; and (3) structural conversion via Partial RoPE adaptation and TransMLA (Meng et al., 2025) conversion. Finally, an MLA Warmup freezes the parameters that were not structurally modified and applies a learning rate warmup to restore the loss to pre-conversion levels. At the end of Migration Pre-Training, the model has fully adopted the target hybrid linear attention architecture and is ready for large-scale continued training.

Continue Pre-Training. After migration pre-training, all parameters are unfrozen for full continued training on 8T tokens with a 4K context window. The data mixture allocates approximately 46% to reasoning-intensive domains (e.g., mathematics and code), 50% to general corpora (e.g., web text), and 4% to multilingual data. In preliminary experiments, we compare two data switching strategies: (1) a conservative approach that first trains on 2T tokens of the original pre-training data to recover model capabilities before introducing new, higher-quality data; and (2) an aggressive approach that introduces the new data from the outset of this stage. The aggressive strategy proves superior: early exposure to higher-quality data accelerates capability recovery, reduces the total tokens needed, and yields a higher final performance ceiling. We therefore adopt the aggressive strategy for Ling-2.6’s Continue Pre-Training.

Mid-Training. In the mid-training stage, we train on high-quality data with long-context activation to refine capabilities and extend context windows. This stage comprises approximately 1.2T tokens and is divided into three phases. The first phase trains on 250B tokens, sampling 20% of sequences at 32K length while maintaining a data mixture similar to the previous stage, which expands the model’s effective context window from 4K to 32K. The learning rate is kept at the peak value inherited from Continue Pre-Training. The second phase spans 425B tokens at 32K context length, significantly increasing the proportion of high-quality data. The third phase covers 525B tokens and extends the context window to 256K, while maintaining the high-quality data mixture.

To provide a stable initialization for subsequent specialization in RL, the Ling-2.6 SFT corpus carefully balances reasoning, long-context window, and agentic tool use scenarios. This foundation is engineered to support token-efficient instant responses, deep long-context reasoning, and robust agentic behaviors.

Reasoning. We compile a diverse reasoning dataset encompassing mathematics, STEM, coding, and logic. To maintain a balanced difficulty distribution, all samples are verified via model-based pass-rate annotations. We significantly expand the mathematical split with high-difficulty competition problems, broaden scientific coverage in chemistry and biology, and introduce over 50,000 synthesized logic queries across propositional and visual domains. Furthermore, we incorporate agent-centric coding tasks—such as repository exploration, debugging, and test generation—across multiple programming languages. Together, these verifiable, difficulty-controlled datasets provide optimal targets for reasoning-oriented RL.

Agent. For agentic behavior and tool utilization, we construct RL data atop scalable, executable environments covering over 200 real-world and synthetic toolkits (e.g., search tools, Model Context Protocol servers). These environments offer more than 2,500 callable functions spanning search, e-commerce, finance, and scientific computation. Based on this infrastructure, we generate verifiable tool-use trajectories encompassing irrelevant tool rejection, long-horizon single-turn execution, and interactive multi-turn tasks (He et al., 2025a). By employing both seed-centric and critical-tool-chain-centric synthesis, the resulting corpus trains the model for accurate, concise, and highly efficient agentic interactions.

Long Context. We extend the post-training context window to 256K tokens through a dedicated corpus comprising books, academic papers, code repositories, financial reports, and web data. To explicitly enhance long-context reasoning, we focus on number-dense corpora. We synthesize extended contexts by strategically merging multi-company and cross-year financial reports to create high-density, structured summaries. Instead of simple retrieval, we generate multi-hop calculation questions that force the model to reason across these extended contexts. To guarantee data quality, we employ a rigorous verification pipeline: an automated sandbox evaluates executable code for numerical accuracy, followed by human-in-the-loop validation, ensuring only high-quality, perfectly aligned reasoning traces are retained for training.

Following the cold-start SFT, the pipeline shifts to a specialist training stage to develop individual expert models. This stage combines expert-level SFT with targeted Reinforcement Learning (RL). The core objective is to maximize instant-response quality under strict token-efficiency constraints, preparing these specialized capabilities for the final distillation into Ling-2.6.

Reasoning. To instill adaptive reasoning capabilities while curbing verbosity, we implement a cohesive strategy spanning data refinement and reward design. We first refine the SFT reasoning data by utilizing proprietary expert models to generate responses, strictly retaining the shortest accurate candidate. To further eliminate "over-reflective" patterns—where redundant secondary reflection occurs after the correct answer has been found—we employ an LLM judge to prune these segments. This data-level intervention reduces the average output length by 200 to 300 tokens.

During the RL phase, we build upon the Evolutionary Chain of Thought (Evo-CoT) framework from Ling-2.0 (Team et al., 2025b). Starting from the refined SFT checkpoint, we train the model using a composite reward system that explicitly penalizes redundancy:

Agentic. For agentic tasks, we employ Group Sequence Policy Optimization (GSPO) (Zheng et al., 2025) tailored specifically to maximize token-efficient tool use. To achieve this, we introduce two dedicated reward signals. First, a process reward measures the alignment between the model’s predicted trajectory and the optimal ground-truth tool-call sequence (Qian et al., 2026), penalizing unnecessary or exploratory invocations to encourage concise execution. Second, we apply a compression-based repetition penalty utilizing the zlib Compression Ratio. Since highly repetitive and degenerate outputs are highly compressible, they receive correspondingly harsher penalties, naturally fostering concise and coherent responses.

Beyond reward design, we improve training efficiency via a novel sample selection strategy called Dynamic Pass Rating (DPR). Instead of relying on static historical pass rates, DPR assesses task difficulty dynamically based on training behavior. From a temporal perspective, tasks solved early in training are deemed easy; from a stability perspective, tasks with consistently high pass rates are similarly classified. Conversely, tasks that remain unsolved or unstable are prioritized as hard. This creates an adaptive curriculum that constantly focuses training resources on informative samples near the model’s current capability frontier.

As the final stage of efficiency-oriented post-training for Ling-2.6, we introduce a bidirectional focus reward mechanism. This aligns the model with fine-grained human preferences while actively preserving concise, information-dense responses. Unlike conventional unidirectional models, our design integrates positive incentives and negative penalties into a single reward model. This broadens the scoring range and provides preference gradients with a significantly higher signal-to-noise ratio. To prevent the model from hacking the reward by merely increasing output length, we design a focus reward mechanism (Huang et al., 2026). By monitoring on-policy saturation levels across different rubric dimensions, training weights are dynamically shifted away from saturated metrics toward dimensions requiring improvement.

For complex tasks, we complement this general mechanism with specialized evaluators. For long-form writing, we propose ReportLogic framework (Zhao et al., 2026), which evaluates macro-structure and discourse organization, shifting the optimization objective from superficial fluency to deep textual logic. For multifaceted instruction following, an independent verification agent decomposes requests into explicit itemized rules for granular validation. While these task-specific rewards operate in their respective domains, the bidirectional focus reward remains the fundamental driver for general queries, ensuring that Ling-2.6 consistently delivers high-quality, token-efficient instant responses.

The tool-use data supports sustained agentic task execution under realistic constraints, emphasizing three complementary capabilities: repository-level coding, information seeking over mobile and web environments, and general-purpose workflows requiring planning and recovery from intermediate failures. Across all settings, we prioritize verifiability, environmental realism, and interaction diversity.

Coding Agent Tasks

We mine PR-Issue pairs from GitHub at scale to train coding agents on real software engineering workflows. Starting from the full GH-Archive (Dec 2015–2023, $\sim$1B records), we retain only repositories with $>$100 stars, require merged PRs linked to closed issues, and mandate that each PR include a test patch for verifiability. Repositories overlapping with existing SWE benchmarks are excluded to prevent contamination. An LLM links PRs to their corresponding issues, yielding approximately 300K raw pairs. Combined with the reproducible execution environments described in Appendix A, these instances provide rigorous tasks for agentic reinforcement learning.

Search Agent Tasks

With the task mixture and execution environments in place, we train Ring-2.6 through an agentic reinforcement learning pipeline designed for verifiable long-horizon behavior. We develop a lightweight agent framework built on function-calling capabilities and equip the agent with three core tools: (1) execute_bash for general bash command execution, (2) search_replace for precise file editing, and (3) task_done for signaling task completion. The maximum conversation length is set to 200 turns during training and 500 turns during evaluation.

We conduct reinforcement learning from a cold-start model in sandbox environments supported by AEnvironment $^{4}$. SWE tasks are inherently long-horizon and typically require 30 to 200 solution steps. To improve training efficiency, we apply several filtering stages to remove redundant and low-quality instances:

The resulting training set comprises approximately 2,500 instances drawn from 1,550 repositories spanning more than 30 programming languages, including Python, Java, C, Rust, and JavaScript.

To prevent the model from exploiting reward signals through information leakage, we adopt two safeguard mechanisms. First, under Restricted Git History Access, the ‘–all‘ flag is removed from ‘git log‘ commands, limiting the model’s visibility to the commit history of the current branch. Second, under Real-time Monitoring, continuous monitoring is deployed during training to detect reward hacking behaviors. Our analysis reveals that approximately 0.2% of trajectories exhibit cheating patterns, which remains negligible in practice.

In the release of Ring-1T (Ling Team, 2025), we proposed IcePop, which leverages double-sided masking to improve the training stability for reinforcement learning on MoE models. However, we observe that a uniform constant-ratio constraint implicitly assumes disproportional mismatch across tokens, which fails to reflect the heterogeneous training-inference discrepancy induced by different token probabilities. We therefore introduce KPop (Guo et al., 2026), which replaces the uniform fixed-ratio constraint with binary KL divergence to better capture heterogeneous token-level mismatch across high- and low-probability regions. For the technical details, please refer to our blog$^{5}$.

Recall IcePop’s formulation,

IcePop adopts a uniform constant-ratio constraint on the policy probability ratio within a fixed global range $[\alpha,\beta]$, with additional double-sided masking, where a constant-ratio threshold treats all tokens the same way, regardless of their probability. But in reality, the noise in the ratio is not uniform across tokens, as the ratio divergence depends on token probability (Guo et al., 2026). Thus, IcePop tends to over-mask the low-probability tokens.

KPop replaces IcePop’s constant ratio bound with a symmetric binary KL criterion. For each output token $y_{t},$ we compute the binary KL divergence between \pi_{{\color[rgb]{0,0,1}\text{train}}}(y_{t}) and \pi_{{\color[rgb]{1,0,0}\text{infer}}}(y_{t}), which views the full vocabulary as a two-event partition, the current sampled token compared to everything else.

An asymmetric variant checks only one direction, which is either forward or reverse. The symmetric KPop mask requires the binary KL to be small in both directions.

The entire mechanism is controlled by a single hyperparameter $\phi$.

Empirical Results. Thanks to the help of Kpop, as shown in Figure 7, the reward curve exhibits a steady upward trajectory throughout training, rising from an initial value of 0.54 to approximately 0.68, which indicates consistent policy improvement. Figure 8 presents the evaluation results obtained during RL training. Notably, KPop enables effective agentic RL scaling on the lightweight agent, improving the solve rate on SWE-bench Verified from 70.8% to 76.28%, demonstrating the potential of KPop for large-scale agentic RL. All reported evaluation results are averaged over three independent runs.

We present a comprehensive evaluation of the Ling-2.6 series models, specifically the high-performance Ling-2.6-1T and its lightweight counterpart, Ling-2.6-flash. Their capabilities are assessed across a diverse array of benchmarks spanning five key domains: knowledge, reasoning, agentic capabilities, instruction following, and long-context understanding. We compare their performance against a wide range of state-of-the-art systems, including leading proprietary models in their non-reasoning or instant modes (e.g., Kimi-K2.5, GPT-5.4, GLM-5, Nemotron-3-Super, DeepSeek-V3.2).

Benchmarks

To comprehensively assess the Ling-2.6 series models, we conduct evaluations across a wide range of benchmarks, primarily covering 5 domains: knowledge, reasoning, agentic capabilities, instruction following, and long-context understanding.

We evaluate our models, Ling-2.6-1T and Ling-2.6-flash, against a diverse set of leading and open-source models. The comparison includes various non-reasoning or instant-response configurations of proprietary models, such as Kimi-K2.5, DeepSeek-V3.2, GPT-5.4, GPT-5.4-mini, GLM-5, GLM-4.5-Air, Nemotron-3-Super-120B-A12B and GPT-OSS-120B.

Results

The evaluation results, presented in Table 4 and Table 5, demonstrate the strong and versatile capabilities of the Ling-2.6 series models across multiple domains.

In this section, we present a comprehensive evaluation of Ring-2.6-1T across a wide range of benchmarks spanning reasoning, OpenClaw, agentic coding, agentic search, and function calling. We compare its performance against leading frontier models, including both open-weights and proprietary models, to contextualize its capabilities and identify areas of strength and improvement.

Benchmarks

To comprehensively assess Ring-2.6-1T, we conduct evaluations across a wide range of benchmarks, primarily covering 5 domains: reasoning, OpenClaw, agentic coding, agentic search, and function calling.

We evaluate Ring-2.6-1T under two inference configurations: Ring-2.6-1T (xhigh), which uses an extended thinking budget for maximum reasoning depth, and Ring-2.6-1T (high), optimized for efficiency with reduced reasoning overhead. We compare against leading models including Kimi-K2.6-Thinking, DeepSeek-V4-Pro$^{14}$, ChatGPT-5.4 (Singh and others, 2025), Gemini-3.1-Pro$^{15}$, GLM-5.1-Thinking (Zhipu-AI, 2026), Claude-Opus-4.7-Thinking$^{16}$, and Claude-Opus-4.6-Thinking. All evaluations use controlled experimental conditions with standardized configurations.

Results

Table 6 provides a comprehensive comparison of Ring-2.6-1T against leading frontier models. The following sections provide a detailed analysis of its performance across different aspects.

Linear Attention reduces complexity from $O(N^{2})$ to $O(N)$ via an RNN-style recurrence, making it a key primitive for long-context training. However, its sequential dependency along the sequence dimension precludes a direct port of conventional Context Parallel (CP) schemes designed for Softmax Attention: Ring Attention relies on online-softmax correction, which is inapplicable to recurrent state accumulation, while DeepSpeed Ulysses requires cp_size to divide head_num, limiting scalability under extreme context lengths.

AllGather-based CP. We introduce an AllGather CP design for Linear Attention, built on a local-recurrence-then-global-correction principle, as shown in Figure 10:

This design is free of head-divisibility constraints and scales linearly with cp_size, providing strictly better scalability than All2All-style alternatives for ultra-long-context regimes.

In long-context training with a large cp_size, varlen inputs frequently contain a large number of short sub-sequences. Under such workloads, a naive implementation incurs frequent kernel launches that translate into substantial CPU-side overhead and markedly degraded GPU utilization, with the effect particularly pronounced in the backward pass. We mitigate this by analytically unrolling the state-correction recurrence into a vectorized matrix form and consolidating the per-sub-sequence output correction into a single Triton-fused kernel, jointly delivering an approximately 68% end-to-end speedup at 256K context length.

Long-context training of MoE models creates severe imbalances in computational load and memory use across GPUs, affecting performance and stability. Key issues include router load-balancing fluctuations and uneven workloads from block-diagonal causal masking, which worsen with context length.

Our throughput-focused strategy co-designs parallelism methods (expert, pipeline, and context parallelism) with selective activation recomputation to maximize memory use and minimize overhead. However, operating near GPU memory limits risks OOM failures during sudden router imbalance. To manage this, we use different memory policies per training phase:

We also found a critical scaling issue: some existing MoE kernel implementations assume token counts fit in 32-bit integers. Expert token counts may exceed this limit in long-context training, causing overflow or crashes. We address this by auditing and modifying critical pathways to use 64-bit integers (int64) for indexing and counting, ensuring robustness at scale.

In summary, we balance high throughput and stability through phase-specific memory policies and foundational fixes, enabling reliable distributed MoE training on ultra-long context.

Ring-2.6 further extends ASystem with asynchronous RL execution to handle long-tailed reasoning rollouts and environment-bound agentic workloads. Reasoning tasks such as math, coding, logic, and STEM can produce chains of thought whose tail generations are much longer than the average. Agentic tasks such as coding, search, and tool use introduce additional latency from code sandboxes, terminal sessions, retrieval services, and tool/MCP endpoints. A synchronous lock-step RL pipeline would leave the trainer waiting for the slowest rollout. ASystem therefore focuses on continuously feeding the trainer while bounding the policy staleness introduced by asynchronous execution.

Partial-Rollout Pipeline. Rather than fully decoupling rollout and training, Ring-2.6 RL runs in a partial-rollout regime in which each optimization step is gated by a global token budget on rollout generation, in the spirit of C3PO++ (Ling Team, 2025). A rollout controller continuously dispatches prompts to the inference engine through an overlap batch dispatcher, which submits new work while previous trajectories are still being collected so that the inference pool is never drained at a batch boundary. As the trajectories arrive, the controller tracks the cumulative number of generated tokens against the per-iteration budget $\Phi$. Completed trajectories are routed through ASandbox for verifier or tool-driven reward computation; trajectories that are still in flight when the budget is reached are paused, persisted into a cross-version rollout buffer together with their KV-cache fingerprint, and resumed by the next policy version in a subsequent iteration. The trainer then performs one optimization step on the harvested batch and triggers an update of weights on the inference engines. By limiting each step based on token counts instead of the slowest-running trajectory, this approach avoids the long-tail overhead that a fully synchronous rollout would incur, while still keeping the boundary between rollout and training deterministic and easy to checkpoint.

Because trajectories may now be assembled from segments generated by different policy versions, we manage the resulting version skew through a dedicated staleness manager. Every rollout segment is tagged with the inference engine’s policy version at the time of generation; the manager admits new rollouts into the pool only up to a configurable bound of $\texttt{max\_staleness}\times\texttt{consumer\_batch\_size}$, and discards or retires segments whose version gap to the trainer exceeds this bound. The result is a bounded-staleness regime in which the RL algorithm can treat the rollouts as approximately on-policy, while the system extracts the throughput of partial-rollout execution. Together with ASystem’s sub-second weight broadcasts, the effective staleness during the reinforcement learning of Ring-2.6 is limited to only a few optimization steps, and we observe no loss in stability or final reward compared to the synchronous baseline.

Asynchronous Agentic Rollouts. The partial-rollout pipeline is what allows us to run agentic RL in a tractable way at our scale. For coding tasks, the trajectories operate within a containerized repository, invoking tools for building, testing, and shell access; for search tasks, they perform retrieval, browsing, and reasoning via calls to external services; for tool-use tasks, they interact with a multi-turn customer-service simulator whose state is tracked outside of the inference engine. In each case, the rollout is a sequence of interactions (model generate, environment step, model generate, $\dots$) whose per-step latency is highly variable and dominated by the environment rather than by the GPU.

Three properties of the stack make these workloads efficient. First, the inference engine and the environments are scaled independently: thousands of ASandbox replicas handle varied environments, and the rollout controller models each tool invocation as an awaitable, ensuring that a GPU is never held up waiting on a pending HTTP/MCP request. Second, the overlap batch dispatcher continuously refills the inference engine as soon as any in-flight trajectory yields a partial result, while the token-budget cutoff bounds the cost of pathologically long trajectories by placing them in the rollout buffer for the next policy version; together, these mechanisms ensure that the long tail of a code patch or a browsing session no longer leaves the GPU idle. Third, the unified verifier interface covering diverse reasoning tasks, the coding harness, the tool-use simulator, and the browsers, all presented through a single HTTP/MCP contract under ASandbox that allows the trainer to combine reasoning and agentic datasets within a single run, without needing task-specific scheduling.