From Context to Skills: Can Language Models Learn from Context Skillfully?

Current language models (LMs) have achieved impressive performance on problems whose relevant knowledge was present during large-scale pre-training [5, 28, 39], such as competition-level mathematical problems [14] and competitive programming challenges [17]. However, many real-world tasks require LMs to learn from complex contexts and leverage new knowledge rather than parametric knowledge to reason and solve them effectively [26, 36, 47]. Previous works refer to this capability as context learning [12]. Effective context learning enables models to reason beyond their pre-trained knowledge and solve complex, domain-specific tasks by learning directly from rich contextual information, much as humans do. For example, it allows LMs to rapidly make use of previously unseen product documentation to generate step-by-step operational procedures or troubleshoot issues.

Despite the central role of context learning in solving real-world tasks, it has been largely overlooked in current research [16]. A key obstacle is the diversity of real-world contexts: contextual knowledge can be conveyed through books, experimental data, and search results, among others, making context learning fundamentally difficult. An intuitive paradigm is inference-time skill augmentation [3], i.e., extracting rules and procedures from context into natural-language skills that encode reusable procedural knowledge for LMs. This approach has proven effective in many agent tasks [42, 49, 37], e.g., coding agents [24]. However, this paradigm faces two fundamental challenges in context learning scenarios. Specifically, (1) Prohibitive Cost for Manual Skill Annotation: Existing agent skill libraries are predominantly constructed through human annotation [18, 20, 41]. However, this paradigm breaks down in context learning scenarios, where contexts are long, technically dense, and domain-specific. Curating high-quality skills requires annotators to fully internalize complex, multi-section documents, a process that is both cognitively demanding and economically infeasible, rendering human-curated skill construction impractical at scale. (2) Lack of External Feedback for Automated Skill Construction: Unlike verifiable tasks such as coding or mathematical reasoning [7, 31], where skill quality can be assessed via execution feedback or ground-truth comparison, context learning tasks lack automatic feedback to evaluate whether extracted skills faithfully and completely capture context-specific knowledge. Given only the context, there is no external feedback signal to tell whether a generated skill is useful or whether critical knowledge has been omitted. This feedback-free nature renders automated skill construction pipelines inapplicable [1, 44, 46]. Together, these challenges highlight the need for an automated framework that discovers skills from the context without relying on either human annotation or external feedback signals.

In this paper, we propose Ctx2Skill shown in Figure 1, which is designed to autonomously discover, refine, and select skills directly from complex contexts via skill-optimized self-play, requiring neither human annotation nor external feedback. At the core of Ctx2Skill is a multi-agent self-play loop [10, 22, 48, 21] comprising two competing but co-evolving roles: a Challenger agent, which generates a batch of tasks and associated rubrics [15] based on context and its skill set, aiming to probe deep contextual understanding, and a Reasoner agent, which reads the context and attempts to solve these tasks guided by its current skill set. The core idea of Ctx2Skill is that the two agents co-evolve by iteratively updating their respective skill sets rather than model parameters. Specifically, a neutral Judge agent evaluates the Reasoner’s responses against each rubric, producing feedback on whether it passes. Crucially, both the Challenger and Reasoner evolve through accumulated natural-language skills: failed cases are routed to dedicated Proposer and Generator agents on the Reasoner side to diagnose missing contextual knowledge and update skills accordingly, while easily solved cases are routed to the Challenger side to strengthen its task and rubric generation strategies, ensuring sustained adversarial pressure. The updated skills are carried forward into the next iteration, allowing both sides to continuously co-evolve through failure-driven textual feedback without any parameter updates. Furthermore, we identify a key risk in this multi-agent self-play loop, namely adversarial collapse phenomenon: as iterations progress, the Challenger may generate increasingly extreme tasks while the Reasoner’s skills over-specialize to these pathological cases, resulting in redundant skill accumulation that degrades generalization. To address this, we further propose a Cross-time Replay mechanism that selects the skill set that best balances performance across representative cases, ensuring robust and generalizable skill evolution. At inference time, the resulting context-specific Reasoner skills can be plugged into any LM to enhance its context learning capability on arbitrary tasks over the same context, including those unseen during the self-play loop.

We evaluate our Ctx2Skill on four context learning tasks from CL-Bench [12], including Domain Knowledge Reasoning, Rule System Application, Procedural Task Execution, and Empirical Discovery & Simulation. Ctx2Skill consistently improves solving rate across all four categories and multiple backbone models: it raises GPT-4.1 [27] from $11.1\%$ to $16.5\%$, GPT-5.1 [28] from $21.1\%$ to $25.8\%$, and GPT-5.2 [28] from $18.2\%$ to $21.4\%$, demonstrating that autonomously discovered context-specific skills provide substantial and generalizable gains.

Context Learning. Current language models excel at tasks whose relevant knowledge was present during pre-training [9, 28, 33, 34]. However, many real-world tasks require models to learn from complex contexts and leverage new knowledge to reason effectively, a capability termed context learning [12]. This context learning capability is essential in practice: a physician consults newly released clinical guidelines to adjust treatment, an engineer reads documentation to execute procedures, and a researcher induces patterns from experimental data to form hypotheses. However, prior work finds that LMs have yet to master this capability [12]. The core difficulty is that contextual knowledge spans diverse formats, from books to experimental datasets, and the relevant rules and procedures are often embedded implicitly, requiring deep comprehension rather than surface-level retrieval. Ctx2Skill directly targets this challenge by autonomously discovering reusable, context-specific skills from the contexts, equipping any LM to better learn from and reason over complex contexts.

Skills for LMs. Skills are natural-language modules that encode reusable procedural knowledge to augment LMs at inference time [2, 3], and skill augmentation has been widely validated across agent tasks [32], e.g., coding [24] and web navigation [42]. Early skill libraries are predominantly constructed through human annotation [18, 19, 20, 41], which is effective but does not scale, especially for context learning scenarios where contexts are long, technically dense, and domain-specific. Recent work has shifted toward automated skill construction. AutoSkill [44] extracts reusable behaviors from interaction traces as lifelong versioned artifacts, and AutoRefine [30] converts agent trajectories into reusable expertise. CoEvoSkills [45] generates multi-file skill packages and refines them through generator–verifier co-evolution, while EvoSkill [1] performs failure-driven refinement into structured skill folders. SkillX [40] distills the agent trajectories into a hierarchical skill knowledge base and iteratively refines them through execution feedback. However, these methods rely on external feedback signals [25], such as execution feedback, ground-truth comparison, or task-completion rewards, to evaluate and improve skill quality, which are not available in context learning scenarios where there is no automatic feedback. A separate line of work tries to internalize skills into model parameters. SKILL0 [23] uses in-context reinforcement learning (RL) to absorb skills, and SkillRL [43] builds a hierarchical skill bank via RL-guided distillation from teacher trajectories. These approaches require parameter access, making them inapplicable to closed-source models, and sacrificing the interpretability of natural-language skill documents. In this paper, we propose Ctx2Skill, a self-evolving framework designed to directly discover and evolve skills from complex contexts alone, which requires no human annotation, no external feedback, and no parameter updates.

In this section, we conduct experiments and analyses to show the advantages of our Ctx2Skill.

We presented Ctx2Skill, a self-evolving framework that autonomously discovers, refines, and selects context-specific skills from complex contexts without human annotation or external feedback. Through a skill-optimized self-play loop, a Challenger and a Reasoner co-evolve their skill sets via failure-driven textual edits, while a Cross-Time Replay mechanism prevents adversarial collapse by selecting the most generalizable skill set across iterations. Experiments on CL-bench demonstrate that Ctx2Skill consistently and substantially improves context learning performance across multiple backbone models and task categories, and that the resulting skills are transferable across models. We hope Ctx2Skill provides a practical and scalable paradigm for equipping language models with the ability to learn skillfully from complex, previously unseen contexts.

In this section, we provide the corresponding statistics for our Ctx2Skill.

CL-bench. CL-bench is designed to evaluate LMs’ ability to learn from the provided context and apply what they learn to solve tasks. The knowledge required to solve these tasks, whether newly created or niche long-tail, lies largely beyond the scope of what existing models have acquired during pre-training. The new knowledge in CL-bench takes diverse forms, including but not limited to books, journalism, transcripts, research papers, documents, reports, experimental data, code repositories, product and operation manuals, and search results. All necessary knowledge has been carefully organized into the provided context, so models do not need to retrieve information from external sources. Each context in CL-bench involves solving multiple tasks. 51.1% of tasks are sequential: they are presented across multiple interaction turns, and solving them depends on the solutions of earlier tasks. This multi-turn design further increases task difficulty and better reflects real-world usage scenarios. The statistics of CL-bench are shown in Table 4.

Task-level Self-Play Dynamics. Table 5 reports the average number of solved and failed tasks (out of $M{=}5$) per iteration during self-play, averaged over all $500$ contexts. On GPT-4.1, the solved rate gradually increases from $18.2\%$ (Iter-1) to $23.3\%$ (Iter-5), with the average number of solved tasks rising from $0.91$ to $1.17$. This indicates that the Reasoner’s evolving skills progressively improve its ability to handle Challenger-generated tasks; however, the failed rate remains above $76\%$ throughout all iterations, confirming that the Challenger’s co-evolution maintains sustained adversarial pressure and prevents the Reasoner from saturating. On GPT-5.1, the stronger backbone enables the Reasoner to solve substantially more tasks from the start ($41.7\%$ at Iter-1), and the solved rate gradually rises to $48.8\%$ by Iter-4 before slightly decreasing to $48.2\%$ at Iter-5. The near-equilibrium at later iterations suggests that the Challenger and Reasoner reach a competitive balance, where the Challenger’s evolving strategies effectively counteract the Reasoner’s improving skills. GPT-5.2 exhibits a distinct pattern: the solved rate decreases from $36.1\%$ at Iter-1 to $23.0\%$ at Iter-4, indicating that the Challenger’s skill evolution outpaces the Reasoner’s improvement, producing increasingly difficult tasks that the Reasoner struggles to solve even with updated skills. This observation is consistent with the adversarial collapse phenomenon described in § 3.4, and further motivates the need for the Cross-Time Replay mechanism to select the most generalizable skill set rather than defaulting to the last iteration. Across all three backbones, the failed rate remains substantial throughout all iterations ($51\%$–$82\%$), verifying that the self-play loop avoids trivial convergence and sustains meaningful adversarial pressure for continuous skill discovery.

Rubric-level Self-Play Dynamics. Table 6 reports the rubric pass rate and the average number of rubrics per task at each iteration. Compared with the task-level solved rate (Table 5), the rubric pass rate is substantially higher across all backbones ($79\%$–$91\%$ vs. $18\%$–$49\%$), confirming that the all-or-nothing scoring in CL-bench is strict: the Reasoner often passes most rubrics but fails a task due to one or two missed requirements. On GPT-4.1, the rubric pass rate steadily improves from $79.2\%$ to $81.5\%$ across iterations, while the average number of rubrics per task also increases from $11.7$ to $12.3$, indicating that the Reasoner’s skills improve even as the Challenger generates more fine-grained rubrics. On GPT-5.1, the rubric pass rate remains consistently high ($90.0\%$–$91.3\%$) with stable rubric counts ($11.2$–$11.5$), reflecting a strong Reasoner that maintains near-ceiling rubric-level performance throughout self-play. On GPT-5.2, the rubric pass rate decreases from $87.5\%$ to $83.6\%$ while the average rubric count increases from $11.2$ to $12.0$, revealing that the Challenger simultaneously generates more rubrics and harder ones, compounding the difficulty and contributing to the declining task-level solved rate observed in Table 5. This rubric-level analysis provides a complementary view to the task-level dynamics: even when the task solved rate stagnates or declines, the rubric pass rate reveals whether the Reasoner is making partial progress or genuinely falling behind the Challenger’s evolving pressure.

Rubric Count Distribution. Table 7 reports the distribution of rubric counts per Challenger-generated task, excluding a small number of tasks ($<1\%$) with zero rubrics due to occasional formatting failures. Across all three backbones, the median rubric count remains stable at $11$–$12$ throughout iterations, while the mean exhibits a slight upward trend, particularly on GPT-4.1 ($11.7\to 12.3$) and GPT-5.2 ($11.2\to 12.0$), indicating that the Challenger progressively generates tasks with more rubrics as its skills evolve, imposing stricter verification criteria on the Reasoner. The minimum rubric count on GPT-4.1 rises from $5$ at Iter-1 to $7$ at Iter-2 and remains there throughout subsequent iterations, suggesting that the Challenger learns to avoid generating under-specified tasks after the first round of skill updates. A similar stabilization is observed on GPT-5.1 (min rises to $7$ by Iter-3) and GPT-5.2 (min stabilizes at $4$), though the floor is lower for GPT-5.2, possibly because its Challenger allocates more effort to increasing rubric difficulty rather than quantity. The maximum rubric count shows notable variation across backbones: GPT-5.1 reaches up to $33$ at Iter-5, while GPT-4.1 and GPT-5.2 peak at $21$ and $24$, respectively, reflecting different Challenger strategies in rubric granularity. The fact that GPT-5.1’s maximum is substantially higher yet its rubric pass rate remains above $90\%$ (Table 6) further highlights the strength of GPT-5.1 as a Reasoner backbone. Combined with Table 6, the increasing mean rubric count on GPT-4.1 and GPT-5.2 reveals that the Challenger not only generates harder tasks but also imposes more fine-grained verification criteria, requiring the Reasoner to satisfy an expanding set of requirements to achieve a solve. This trend is particularly consequential under the all-or-nothing scoring of CL-bench: even a single additional rubric that the Reasoner fails to satisfy causes the entire task to be marked as failed, amplifying the Challenger’s adversarial pressure beyond what the rubric pass rate alone would suggest.

Challenger Task Length. Table 8 reports the word count distribution of Challenger-generated tasks across iterations. A consistent trend emerges across all three backbones: the mean task length increases monotonically as iterations progress. On GPT-4.1, the mean rises from $46.3$ words at Iter-1 to $59.4$ at Iter-5 ($+28\%$), reflecting a moderate increase in task specificity. GPT-5.2 shows a more dramatic growth, with the mean nearly doubling from $69.1$ to $139.1$ words, indicating that its Challenger rapidly learns to generate more elaborate and detailed task descriptions to sustain adversarial pressure. GPT-5.1 produces the longest tasks overall (mean $172.0$ words), starting at $142.1$ and rising to $185.1$, consistent with GPT-5.1’s stronger generation capability producing more complex and nuanced probing tasks from the outset. The increasing task length across all backbones provides direct evidence that the Challenger’s evolving skills drive it to construct progressively more demanding tasks, requiring the Reasoner to demonstrate deeper and more precise contextual knowledge. Notably, the growth rate differs substantially across backbones: GPT-4.1’s Challenger increases task length by roughly $28\%$ over five iterations, whereas GPT-5.2’s Challenger increases it by over $100\%$, which is consistent with GPT-5.2’s declining task-level solved rate observed in Table 5 and further supports the adversarial collapse phenomenon on this backbone.

Reasoner Output Length. Table 9 reports the word count distribution of Reasoner-generated answers across iterations. Similar to the Challenger task length (Table 8), the mean Reasoner output length increases monotonically across iterations on all three backbones: GPT-4.1 rises from $209.4$ to $322.2$ words ($+54\%$), GPT-5.2 from $216.9$ to $312.3$ ($+44\%$), and GPT-5.1 from $337.6$ to $399.6$ ($+18\%$). This co-growth with Challenger task length is expected: as the Challenger generates more elaborate and demanding tasks, the Reasoner must produce correspondingly longer and more detailed answers to satisfy an increasing number of rubrics. Notably, GPT-5.1 starts with the longest outputs ($337.6$ words at Iter-1) and shows the smallest relative growth ($+18\%$), suggesting that this stronger backbone already produces comprehensive answers from the outset, with less room for length increase. In contrast, GPT-4.1 and GPT-5.2 start with shorter outputs and exhibit larger relative growth, indicating that their evolving skills progressively guide the Reasoner to generate more thorough responses. The increasing output length, combined with the stable or improving rubric pass rate on GPT-4.1 and GPT-5.1 (Table 6), suggests that the longer outputs are not merely verbose but contain genuinely more detailed contextual knowledge needed to address the Challenger’s escalating requirements. On GPT-5.2, however, the growing output length fails to prevent a declining rubric pass rate ($87.5\%\to 83.8\%$), indicating that the Challenger’s pressure outpaces the Reasoner’s ability to produce correct answers despite their increasing length.

Skill Set Word Count. Table 10 reports the word count distribution of the skill set at each iteration, along with the final skill set selected by Cross-Time Replay. Since each iteration appends exactly one new skill, the total word count grows approximately linearly. GPT-5.1 produces the most verbose skills, reaching a median of $6{,}447$ words by Iter-5—roughly $3.8\times$ that of GPT-4.1 ($1{,}703$) and $1.8\times$ that of GPT-5.2 ($3{,}582$). The per-skill granularity also differs considerably: GPT-4.1 averages ${\sim}340$ words per skill, while GPT-5.1 averages ${\sim}1{,}260$ and GPT-5.2 ${\sim}720$, suggesting that stronger backbones produce more detailed and elaborated skill descriptions. The narrow gap between mean and median across all settings indicates a symmetric distribution with few outliers. Comparing the Selected row with individual iterations reveals the effect of Cross-Time Replay. On GPT-4.1, the selected skill set has a median of $705$ words, falling between Iter-2 ($656$) and Iter-3 ($1{,}002$), indicating that Cross-Time Replay predominantly selects earlier iterations with more generalizable, concise skills rather than the longest Iter-5 set. A similar pattern holds for GPT-5.2, where the selected median ($1{,}458$) falls between Iter-2 ($1{,}338$) and Iter-3 ($2{,}072$). In contrast, GPT-5.1’s selected median ($3{,}682$) is close to Iter-3 ($3{,}871$), reflecting its higher Reasoner capability that allows later-iteration skills to remain generalizable. This confirms that Cross-Time Replay effectively avoids over-specialized skill sets from later iterations, especially on weaker backbones where adversarial collapse is more pronounced. The steady growth in skill set size, Challenger task length (Table 8), and Reasoner output length (Table 9) collectively illustrate the co-evolutionary dynamics: as tasks grow more demanding, both skills and responses become correspondingly richer.

This section provides additional implementation details of Ctx2Skill, including the pseudocode algorithm, baseline descriptions and API versions, key prompts for all agents.

Baselines and API Versions. We compare Ctx2Skill with two baselines. (1) Prompting directly prompts the LM to read the context $C$ and generate a skill set in a single pass; the resulting skills are then pre-pended to LMs during inference. (2) AutoSkill4Doc$^{1}$ is a variant of AutoSkill [44] designed for document-level contexts: it partitions the context $C$ into different windows, identifies the independent skills within each window, and recombines skills across windows to produce a skill set. All experiments use fixed model snapshots to ensure reproducibility. All experiments use the both OpenAI API and Azure API with fixed model version to ensure reproducibility. For the GPT-4.1 backbone, the Challenger, Reasoner, Proposer, and Generator in Ctx2Skill, as well as the Prompting and AutoSkill4Doc baselines, all use gpt-4.1-2025-04-14. For the GPT-5.1 backbone, these agents and baselines use gpt-5.1-2025-11-13. For the GPT-5.2 backbone, these agents and baselines use gpt-5.2-2025-12-11. The Judge uses gpt-5.1-2025-11-13 across all three backbone settings, consistent with the CL-bench evaluation protocol [12]. The Skill Quality Evaluator (Table 2) uses gpt-4.1-2025-04-14. Meanwhile, we use claude-opus-4-5-20251101 for Claude Opus 4.5, kimi-k2.5 for Kimi K2.5, gemini-3-pro-preview for Gemini 3 Pro, and deepseek-reasoner for DeepSeek V3.2. Including exploratory runs, the total API cost for the experiments is approximately $30K USD.

Key Prompts. The Challenger (Figure 6) generates $M$ evaluation tasks with binary rubrics from the conversation context. The Judge (Figure 11) evaluates Reasoner responses via a three-step grading process (requirement analysis, per-rubric verification, and self-reflection) under strict all-or-nothing scoring; it does not participate in skill evolution. The Proposer (Figures 7 and 9) diagnoses failure patterns from passed or failed tasks and proposes skill improvements through structured analysis. The Generator (Figures 8 and 10) implements each proposal into a concrete SKILL.md. We also include the prompts for the Skill Quality Evaluator (Figure 12) used in Table 2 and the Prompting baseline (Figure 13). Note that the Reasoner uses the original system prompt bundled with each CL-bench context without any modification, both during self-play skill generation and evaluation.

Algorithm. Algorithm 1 summarizes the complete Ctx2Skill pipeline. Given a context $C$, the framework runs $N$ self-play iterations (lines 2–13). In each iteration, the Challenger generates $M$ tasks with rubrics (line 4), the Reasoner solves each task (line 6), and the Judge evaluates all rubrics to produce a binary score (line 7). The outcomes are then routed: failed tasks are sent to the Reasoner Proposer–Generator pair, and passed tasks to the Challenger Proposer–Generator pair, yielding updated skill sets $\mathcal{S}^{\mathrm{R}}_{i}$ and $\mathcal{S}^{\mathrm{C}}_{i}$ (lines 8–9). Meanwhile, the hardest failure and easiest success from each iteration are accumulated into the probe sets $\mathcal{Q}^{\mathrm{h}}$ and $\mathcal{Q}^{\mathrm{e}}$ (lines 11–12). After all iterations, Cross-Time Replay re-evaluates every $\mathcal{S}^{\mathrm{R}}_{i}$ on both probe sets and selects the iteration $i^{*}$ that maximizes $\rho^{\mathrm{h}}(i)\cdot\rho^{\mathrm{e}}(i)$ (lines 15–19), returning $\mathcal{S}^{\mathrm{R}}_{i^{*}}$ as the final skill set (line 20).

Limitations and Future Work. Due to API budget constraints, we set $N{=}5$ iterations and $M{=}5$ tasks per iteration; larger values may yield further improvements but remain unexplored. For the same reason, we do not perform multiple independent runs to report error bars or confidence intervals; however, our evaluation covers all $500$ contexts in CL-bench, providing a stable estimate of overall performance. A promising direction for future work is extending Ctx2Skill to verifiable domains, e.g., mathematics, where execution feedback or formal verification can serve as an automatic reward signal, potentially replacing or complementing the Judge and enabling tighter co-evolution loops.

In this section, we present a case study from CL-bench comparing GPT-5.1 responses generated with Ctx2Skill skills against those without skills (Figure 5), where the context is a heterogeneous collection of over $172$K characters in which the governing rulebook occupies roughly $33$–$48\%$ and the remaining documents act as distractors. Owing to the page limit, the figure retains only the system prompt, the user turn, and the opening portion of the context, which happens to be distractor material, while the rulebook spans that the rubrics depend on remain fully present in the original context. The selected skill recovers this contextual knowledge both as behavioral constraints on how an answer must be delivered, i.e., the alternating persona cycle and the exact headers, openings, signatures, and Strategic Principle: line, and as procedural routines that force the Reasoner to re-derive the governing rules before answering. Without skills the response opens in the wrong persona and misreports the tie and the $8$–$5$ split, satisfying only $5$ of $15$ rubrics, whereas with skills it satisfies all $15$, indicating that the gains of Ctx2Skill stem from distilled context knowledge rather than generic answering strategies.

We discuss three potential concerns regarding the validity of Ctx2Skill’s improvements: (1) whether the gains stem from reward hacking, i.e., the skills exploiting the preferences of the specific Judge model rather than encoding genuine contextual knowledge; (2) whether the Challenger-generated tasks overlap with the CL-bench test tasks, causing test set contamination; and (3) whether the discovered skills are truly context-specific rather than generic answering strategies. Our analyses show that none of these concerns holds, confirming that the improvements of Ctx2Skill come from genuinely learned, context-specific knowledge.

Analysis of Potential Reward Hacking. In Ctx2Skill, the Judge that provides binary feedback during the self-play loop is GPT-5.1, which is also the LLM-as-a-judge verifier of the CL-bench evaluation protocol [12]. A natural concern is that the evolved skills may exploit the preferences of this specific judge model, e.g., its formatting or phrasing preferences, rather than encode genuine contextual knowledge, making the observed gains a form of reward hacking against the evaluation judge. To examine this, we decouple the two judges: on the GPT-5.2-based methods, we replace the Judge in the self-play loop with GPT-5.2 while keeping GPT-5.1 as the evaluation judge, and rerun Ctx2Skill on all $500$ contexts. As shown in Table 11, the resulting skills achieve a $21.2\%$ overall solving rate, nearly identical to the $21.4\%$ obtained with the default GPT-5.1 loop judge, with all per-category differences within $0.4\%$. The gains of Ctx2Skill therefore do not depend on the loop judge matching the evaluation judge, indicating that the discovered skills encode judge-agnostic contextual knowledge rather than exploiting the preferences of a specific judge model.

Test Set Contamination Analysis. Since the Challenger generates probing tasks from the same context on which the CL-bench test tasks are built, a potential concern is that the generated tasks may closely replicate the test tasks, so that the Reasoner’s skills would be implicitly optimized on the test set. To examine this, we measure the word-level $n$-gram overlap between the Challenger-generated tasks and the CL-bench test tasks within the same context. We analyze the GPT-5.1-based setting, whose Challenger generates the longest tasks (Table 8) and thus poses the highest a priori risk of overlap. Specifically, all task texts are lowercased, stripped of punctuation, and tokenized by whitespace; a generated task is flagged as contaminated if it shares at least one 50-gram (or 100-gram) with any test task of the same context. As shown in Table 12, none of the approximately $12{,}500$ generated tasks shares any 50-gram or 100-gram with the test tasks, even though both the generated tasks (mean $172.0$ words) and the test tasks (mean $654.0$ words) are long enough to contain such $n$-grams. These results indicate that the improvements of Ctx2Skill do not stem from test set leakage through task generation.

Context-Specificity of the Discovered Skills. Ctx2Skill claims that the discovered skills encode context-specific knowledge; an alternative explanation of the gains, however, is that the skills merely offer generic answering strategies, e.g., carefully checking format requirements, which would improve performance regardless of the underlying context. To distinguish the two, we conduct a skill-swap experiment on GPT-4.1: when evaluating the tasks over a context, we replace its own final skill set $\mathcal{S}^{\mathrm{R}}_{\star}$ with the final skill set of a randomly sampled context from a different sub-category, keeping all other inference settings unchanged. As shown in Table 13, the mismatched skills yield only a marginal improvement over the base model ($12.5\%$ vs. $11.1\%$), far below the $16.5\%$ achieved by the matched skills, i.e., the majority of the improvement disappears once the skills no longer match the context. Notably, the mismatched performance is close to that of the single-pass Prompting baseline ($12.3\%$, Table 1), suggesting that the small residual gain reflects generic answering strategies, whereas the advantage of Ctx2Skill comes from the context-specific knowledge encoded in the skills.