SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Raw trajectories $\tau$ collected from environment interactions are verbose, containing exploratory actions, backtracking, and redundant steps that obscure the critical decisions leading to success or failure. To transform these experiences into actionable knowledge, we employ a teacher model $\mathcal{M}_{T}$ to distill trajectories into compact, reusable skills.

Specifically, we first deploy a base LLM agent $\pi_{\text{base}}$ in the target environment $\mathcal{E}$ to collect diverse trajectories. Unlike prior approaches that retain only successful episodes, we deliberately preserve both successful trajectories $\mathcal{T}^{+}=\{\tau_{i}:r(\tau_{i})=1\}$ and failed trajectories $\mathcal{T}^{-}=\{\tau_{i}:r(\tau_{i})=0\}$, where $r(\tau)$ denotes the binary task success indicator. Failed trajectories reveal failure modes and boundary conditions, i.e., information difficult to infer from successes alone.

We apply differential processing based on trajectory outcomes. For successful trajectories $\tau^{+}\in\mathcal{T}^{+}$, we extract the strategic patterns that led to task completion:

The teacher model identifies critical decision points, the reasoning behind correct actions, and generalizable patterns that transfer beyond the specific task instance.

For failed trajectories $\tau^{-}\in\mathcal{T}^{-}$, direct inclusion in context is infeasible due to their length and noise. Instead, we synthesize concise failure lessons:

The analysis identifies: (1) the point of failure, (2) the flawed reasoning or action, (3) what should have been done, and (4) general principles to prevent similar failures. This transforms verbose failed episodes into counterfactuals.

Following the design principles of Agent Skills [2], we organize the distilled knowledge into a hierarchical skill library SkillBank that enables efficient retrieval of relevant expertise during decision-making.

A static skill library cannot anticipate all scenarios the agent will encounter. As the policy improves and explores new state regions, it faces situations where existing skills provide insufficient guidance. We introduce recursive skill evolution during reinforcement learning to address this limitation, enabling the skill library and agent policy to co-evolve.

Cold-Start Initialization. Before RL training, we address a critical challenge: the base agent has not learned how to effectively utilize skills. Simply providing skills to an unchanged model yields limited benefit [12]. We therefore perform a cold-start supervised fine-tuning (SFT) stage [28], where the teacher model $\mathcal{M}_{T}$ generates $N$ skill-augmented reasoning traces $\mathcal{D}_{\text{SFT}}=\{(d_{i},\mathcal{S}_{i},\tau_{i}^{*})\}_{i=1}^{N}$ demonstrating how to retrieve, interpret, and apply skills during decision-making. The base model is then fine-tuned on these demonstrations:

where $\mathcal{L}_{\text{CE}}$ denotes the cross-entropy loss. The resulting model $\pi_{\theta_{\text{sft}}}$ serves as both the starting point for RL training and the reference policy $\pi_{\text{ref}}$ for KL regularization.

Recursive Skill Evolution. A static skill library cannot anticipate all scenarios the agent will encounter. As the policy improves and explores new state regions, it faces situations where existing skills provide insufficient guidance. We introduce recursive skill evolution to address this limitation. The process begins with an initial skill library containing baseline task-action principles.

After each validation epoch, we monitor the success rate $Acc(C)$ for each task category $C$. To ensure targeted growth, the evolution is triggered only for categories where $Acc(C)<\delta$. We then collect failed trajectories $\mathcal{T}_{\text{val}}^{-}=\{\tau_{j}:r(\tau_{j})=0\}_{j=1}^{M}$ using a diversity-aware stratified sampling strategy: trajectories are grouped by category, prioritized by the severity of failure (negative rewards), and selected via round-robin sampling to maintain categorical entropy. Then we will analyze these samples to identify gaps:

The teacher model is prompted to: (1) identify failure patterns not addressed by current skills, (2) propose new skills to cover these gaps, and (3) suggest refinements to existing skills that proved ineffective. The library is then updated: $\textsc{SkillBank}{}\leftarrow\textsc{SkillBank}{}\cup\mathcal{S}_{\text{new}}$.

This creates a virtuous cycle: as the agent improves, it encounters new challenges, which drive skill library expansion, which enables further improvement.

RL-based Policy Optimization. We optimize the skill-augmented policy using GRPO. For each task with description $d$, the agent first retrieves relevant skills and then samples $G$ complete trajectories $\{\tau^{(1)},\ldots,\tau^{(G)}\}$ from the current policy $\pi_{\theta}$. Each trajectory $\tau^{(i)}$ receives a binary reward $R_{i}=r(\tau^{(i)})\in\{0,1\}$ indicating task successfulness. The normalized advantage for each trajectory is computed as:

The policy is updated according to:

where $\rho_{i}=\frac{\pi_{\theta}(\tau^{(i)}|d,\mathcal{S}_{g},\mathcal{S}_{\text{ret}})}{\pi_{\text{old}}(\tau^{(i)}|d,\mathcal{S}_{g},\mathcal{S}_{\text{ret}})}$ is the importance ratio computed over the skill-augmented context. The KL penalty anchored to $\pi_{\text{ref}}=\pi_{\theta_{\text{sft}}}$ ensures that RL optimization preserves the learned skill utilization capabilities while improving task performance. The complete training procedure is summarized in Algorithm 1.

Environments. ALFWorld [35] is a text-based game aligned with the ALFRED embodied AI benchmark. Agents must complete household tasks by navigating and interacting with objects through text commands. WebShop [53] simulates web shopping. Agents navigate a realistic web interface to find and purchase products matching user specifications. In addition, we also evaluate the performance of SkillRL on search-augmented QA tasks, including single-hop QA datasets (NQ [18], TriviaQA [17], and PopQA [23]) and multi-hop QA datasets (HotpotQA [52], 2Wiki [13], MuSiQue [41], and Bamboogle [31]).

Baselines. We compare SkillRL against four categories of competitive methods. First, we include closed-source LLMs, specifically GPT-4o [24] and Gemini-2.5-Pro [5], which represent the state-of-the-art in general-purpose reasoning and instruction following. Second, we evaluate prompt-based agentic or memory-based methods, including ReAct [54] and Reflexion [34], which rely on in-context prompting for multi-step reasoning, as well as Mem0 [4], ExpeL [58], and MemP [7], which utilize external memory or experience pools to guide behavior without parameter updates. Third, we consider RL-based methods, including group-based online RL algorithms such as RLOO [1] and GRPO [33] that optimize policies via advantage estimation over trajectory groups. Finally, we compare against memory-augmented RL-based methods, such as EvolveR [49], MemRL [57], and the combination of Mem0+GRPO and SimpleMem [21]+GRPO, which integrate persistent memory mechanisms directly into the reinforcement learning optimization process to handle long-term dependencies. For search-augmented QA, we compare SkillRL with R1-Instruct, Search-o1 [20], Search-R1 [16], ZeroSearch [36], and StepSearch [59].

Implementation Details. We use Qwen2.5-7B-Instruct [3] as our base model and OpenAI o3 [25] as the teacher model for skill distillation and SFT data generation. For RL training, we use GRPO with learning rate $1\times 10^{-6}$, batch size 16, group size 8, and 4 gradient accumulation steps. We set $K=6$ for task-specific skill retrieval and $\delta=0.4$ for the collection of failed trajectories. For more detailed information on training hyperparameters, please see Appendix D.1.

Comparison with Baselines. We compare SkillRL with baselines across two benchmarks as shown in Table 1. Our method consistently outperforms all baselines, with key observations as follows:

1) Significant Gains over Prompt-based Methods. SkillRL achieves a 89.9% success rate on ALFWorld and 72.7% on WebShop, outperforming the best prompt-based baselines by a large margin. This gap suggests that while in-context learning can leverage past experiences, it often fails to distill actionable knowledge from verbose trajectories or fundamentally adapt the agent’s policy.

2) Superiority over Vanilla RL. RL training brings substantial gains, yet SkillRL consistently surpasses standard RL baselines. Compared to PPO, RLOO, and GRPO, SkillRL achieves the best overall performance. Notably, since SkillRL utilizes GRPO as its base optimizer, the 12.3% absolute improvement over GRPO on ALFWorld (from 77.6% to 89.9%) is directly attributable to our skill-augmentation mechanism rather than algorithmic variance. In complex subtasks like Cool and Pick2, SkillRL outperforms GRPO by 23.0% and 22.8% respectively, proving that structured skill priors effectively accelerate and enhance policy learning in sparse-reward environments.

3) Advantage over Memory-Augmented RL. SkillRL substantially outperforms existing memory-augmented RL frameworks, which differ in how they manage and update experience. MemRL, which uses RL solely to update its memory bank while keeping the policy frozen, fails to adapt to complex environments, yielding only 21.4% on ALFWorld. EvolveR, which jointly updates the policy and memory bank, shows improvement (43.8%) but remains limited by its reliance on rough trajectory storage. To provide a more competitive baseline, we implemented Mem0+GRPO, which combines a state-of-the-art prompt-based memory mechanism with an optimized policy model. While this hybrid approach improves performance to 54.7% on ALFWorld and 37.5% on WebShop, it still trails SkillRL by a wide margin (about 35.2% absolute success rate gap). These results validate our core hypothesis: effective experience transfer requires high-level skill abstraction and a co-evolving library rather than simple trajectory compression or prompt-based memory retrieval.

Comparison with Closed-Source Models. Remarkably, SkillRL with Qwen2.5-7B-Instruct significantly outperforms much larger closed-source models, as shown in Table 1. On ALFWorld, our method exceeds GPT-4o [24] by 41.9% and Gemini-2.5-Pro [5] by 29.6%. This demonstrates that effective skill learning can compensate for model scale, enabling smaller open-source models to achieve superior task performance through structured experiential knowledge.

Performance on Search-Augmented QA. As shown in Table 2, SkillRL achieves a state-of-the-art average score of 47.1%, significantly outperforming Search-R1 (38.5%) abd EvolveR (43.1%). Key observations include: 1) Superior multi-hop Reasoning: SkillRL excels in complex tasks like Bamboogle, surpassing EvolveR by 19.4%. This demonstrates that hierarchical skills effectively guide multi-step information synthesis. 2) Strong generalization: Despite being trained on limited datasets (NQ, HotpotQA), SkillRL maintains competitive performance on OOD tasks like TriviaQA and 2Wiki, confirming that distilled search strategies are task-agnostic.

In this section, we provide detailed analysis of each module’s effectiveness and the skill evolution dynamics.

Ablation Studies. We conduct ablation experiments to evaluate each component’s contribution, with results in Table 3. According to the results: (1) Removing hierarchical structure (i.e., task-specific skills only) decreases performance by 13.1% on ALFWorld and 11.3% on WebShop, indicating universal strategic principles provide essential foundational guidance. (2) Replacing the skill library with raw trajectories causes the largest degradation (up to 25%), which directly supports our motivation that abstraction is superior to memorization. Raw experiences introduce significant redundancy and noise that hinder effective knowledge transfer. (3) Cold-start SFT proves critical (20% drop without it), confirming that the base model requires an initial explicit demonstration phase to learn how to adaptively retrieve and utilize the abstracted skills before entering the RL stage. (4) Dynamic evolution contributes a 5.5% improvement by ensuring the skill library is a dynamic component rather than a static database. This co-evolution allows the agent to iteratively refine its internal policy by addressing emergent failure modes that were not covered by the initial skill set.

Per-Task Analysis on ALFWorld. Table 1 breaks down ALFWorld performance by task type. The largest gains are on PickTwo (+23%), Cool (+22%) and Heat (+15%), which are among the most challenging tasks requiring multi-step planning and state tracking. Task-specific skills are particularly valuable here, capturing strategies like “when picking two objects, verify the first is secured before searching for the second” that address common failure modes.

Skill Library Growth. Figure 3 shows how the skill library evolves during training. The initial skill library contains 55 skills (12 general, 43 task-specific). Through dynamic evolution, this grows to 100 skills by the end of training (Step 150). The growth is predominantly driven by task-specific skills (increasing from 43 to 80), while general skills show a steadier increase (from 12 to 20). Notably, we observe a balanced expansion across various task categories, ensuring the agent develops specialized expertise for each environment rollout. This overall expansion reflects the agent’s increasing ability to refine its repertoire and tackle diverse scenarios within specific task types.

Context Efficiency. To evaluate the impact of skill abstraction on inference overhead, we compare the average prompt length of SkillRL with a memory-augmented baseline using raw trajectories (Qwen2.5-7B with Raw Memory) in Figure 5. The results reveal that while the raw memory approach suffers from a high and fluctuating token footprint (averaging $\sim$1,450 tokens), SkillRL maintains a significantly leaner prompt (averaging $<$1,300 tokens), achieving approximately a 10.3% reduction in context length. This efficiency stems from our distillation mechanism, which compresses verbose environment interactions into high-density, actionable skills. Notably, SkillRL requires less context than the memory-based baseline to achieve superior performance, demonstrating that skill abstraction effectively mitigates the context-bloat problem common in traditional memory-based agents.

Evolution Dynamics. Figure 5 illustrates the reinforcement learning training curves with and without the recursive skill evolution mechanism. We observe that while SkillRL without evolution shows steady improvement, SkillRL with skill evolution exhibits a notably higher learning rate and superior asymptotic performance. Specifically, SkillRL achieves a success rate of over 80% within 60 training steps, whereas the baseline requires approximately 90 steps to reach a lower peak. This acceleration in convergence suggests that the dynamic introduction of new skills and refinement of existing ones effectively provide the agent with timely strategic guidance to overcome local optima. Furthermore, the higher performance ceiling validates that the co-evolution of the skill library and the policy allows the agent to adapt to increasingly complex task scenarios that static memory methods fail to resolve.

Qualitative Analysis. To further investigate how SkillRL utilizes the learned knowledge, we visualize the reasoning process on ALFWorld and WebShop in Figure 6. The case studies demonstrate that our trained agent can effectively retrieve and execute relevant skills from the SkillBank to guide its decision-making. For instance, in the WebShop task, the agent invokes general strategies like “Prioritize Core Keywords” alongside task-specific heuristics “Focus Key Query” to ensure the product meets all constraints within a limited budget. Similarly, in ALFWorld, the agent coordinates hierarchical skills, i.e., using “Progressive Goal Decomposition” for high-level planning and “No Appliance Before Object” to avoid common logical pitfalls. This seamless integration of general and specific skills confirms that the agent does not merely memorize trajectories, but rather develops a structured understanding of task logic, allowing for more robust and efficient problem-solving.

The following prompts are used during the online inference phase. These templates provide the agent with the current task description, a history of previous interactions, and a set of retrieved skills (experiences) to guide its decision-making process. The prompts explicitly enforce a Chain-of-Thought (CoT) reasoning step before action selection.

These prompts are utilized during the skill discovery and library initialization phases. They guide a high-capability teacher model to analyze interaction trajectories, identify failure modes, and distill reusable, actionable skills into a structured JSON format.

To bridge the gap between a base model and the target performance, we use the following prompts to generate high-quality synthetic trajectories for Supervised Fine-Tuning (SFT). These prompts instruct the teacher model to solve tasks while explicitly demonstrating the application of specific skills, thereby providing a clear learning signal for the student model.

All experiments were conducted on a cluster with 8 NVIDIA H100 80GB GPUs. Training times:

Total wall-clock time: approximately 30 hours per experiment.

The Qwen2.5-7B model falls below the capability threshold required to simultaneously navigate complex environments and abstract high-quality skills. However, our claim on experience-based skill distillation holds completely true once the base model has sufficient capacity. To demonstrate this, we conducted new experiments using a much stronger base model, i.e., Kimi-K2.5 [38] (1.1T parameters) on the Tinker platform [40]. As shown in Table 5, Kimi-K2.5 successfully distills skills from its own trial-and-error experience, significantly boosting its own performance (e.g., 71.2 to 88.5 on ALFWorld, 45.9 to 73.4 on WebShop) entirely without the o3 teacher. The introduction of o3 simply pushes the upper bound even higher. We will explicitly discuss this “capability threshold for skill abstraction" in the next version. We will clarify that it enables true autonomous experience-based skill discovery for stronger ones.