Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning

Yaorui Shi, Yuxin Chen, Zhengxi Lu, Yuchun Miao, Shugui Liu, Qi Gu, Xunliang Cai, Xiang Wang

and 1 more authors

An Zhang

arXiv:2605.06130v3 · This workspace stays on this version. Check arXiv for newer versions

Affiliations and author notes

University of Science and Technology of China
Meituan
National University of Singapore
Zhejiang University
Wuhan University, Equal contribution.Corresponding authors: guqi03@meituan.com, an_zhang@ustc.edu.cn

Some source details need review 9 to check

Original formulas, table values and figures are preserved. Comparison can miss differences; use the original PDF to check details.

Reading view checked: 193 math expressions, 0 rendering failures; 22 of 22 images available.

Compared 115 of 115 source passages against 20 of 20 PDF pages.

Some source graphics could not be placed in the reading view. (2)
A6_p2_pic1: the arXiv HTML draws 1 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 87% or more) without changing any text or formula. Text may look smaller than in the PDF.
A6_p3_pic1: the arXiv HTML draws 1 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 91% or more) without changing any text or formula. Text may look smaller than in the PDF.
A7_SS1_p4_pic1: the arXiv HTML draws 1 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 96% or more) without changing any text or formula. Text may look smaller than in the PDF.
A7_SS2_p1_pic1: the arXiv HTML draws 1 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 92% or more) without changing any text or formula. Text may look smaller than in the PDF.
A7_SS2_p4_pic1: the arXiv HTML draws 1 text or formula block(s) of this graphic outside its frame; each was scaled into its frame (to 88% or more) without changing any text or formula. Text may look smaller than in the PDF.
An unsupported comparison claim was rejected; source verification is incomplete.

HTML S1_p1_1 says “[Zhang et al., 2026b](#bib.bib14); [Muhtar et al., 2026](#bib.bib13); [Lu et al., 2026](#bib.bib32)”; PDF page 1 says “Zhang et al., 2026a; Muhtar et al., 2026; Lu et al., 2026.” Check PDF page 1

HTML S4_T1 says “RetroAgent (Zhang et al., 2026b)”, while PDF page 6 says “RetroAgent (Zhang et al., 2026a)”; the PDF image confirms the “a” suffix. Check PDF page 6

22 of 22 figure, table and graphic images captured. Open original PDF

Abstract

A persistent skill library allows language model agents to reuse successful strategies across tasks. Maintaining such a library requires three coupled capabilities. The agent selects a relevant skill, utilizes it during execution, and distills new skills from experience. Existing methods optimize these capabilities in isolation or with separate reward sources, resulting in partial and conflicting evolution. We propose Skill1, a framework that trains a single policy to co-evolve skill selection, utilization, and distillation toward a shared task-outcome objective. The policy generates a query to search the skill library, re-ranks candidates to select one, solves the task conditioned on it, and distills a new skill from the trajectory. All learning derives from a single task-outcome signal. Its low-frequency trend credits selection and its high-frequency variation credits distillation. Experiments on ALFWorld and WebShop show that Skill1 outperforms prior skill-based and reinforcement learning baselines. Training dynamics confirm the co-evolution of the three capabilities, and ablations show that removing any credit signal degrades the evolution. Our code is available at https://github.com/AlphaLab-USTC/Skill1.

Reinforcement learning (RL) (Sutton and Barto, 2018; Schulman et al., 2017; Shao et al., 2024) has become an important paradigm for training large language model (LLM) agents that interact with complex environments (Guo et al., 2025; Yang et al., 2024; Team et al., 2026; Touvron et al., 2023; Shridhar et al., 2021; Yao et al., 2022a; Xi et al., 2025). Standard RL training treats each task as an isolated episode, where the strategies that lead to success are absorbed only implicitly into the policy parameters and cannot be explicitly reused on future tasks. A natural solution is to augment agents with a persistent skill library that accumulates reusable strategies from past experience, so that the agent can draw on previously successful approaches instead of solving every task from scratch (Wang et al., 2023; Zhao et al., 2024; Xia et al., 2026; Zhang et al., 2026b; Muhtar et al., 2026; Lu et al., 2026). The workflow of these skill-augmented agents can be organized around a three-stage lifecycle (Jiang et al., 2026b): skill selection, where the agent selects a relevant skill from the library for the current task; skill utilization, where the agent executes guided by the selected skill; and skill distillation, where the agent derives new reusable skills from the trajectories.

Existing methods have advanced each stage through RL, improving how agents select skills (Zhang et al., 2026a; Wang et al., 2026; Li et al., 2026b; Wu et al., 2025), utilize them (Xia et al., 2026; Muhtar et al., 2026; Zhang et al., 2026b; Li et al., 2026b; Wang et al., 2025c), and distill reusable knowledge (Zhang et al., 2026b; Wang et al., 2025c; Muhtar et al., 2026; Wu et al., 2025). Yet two fundamental questions remain open. (1) How can an agent evolve all three capabilities simultaneously? Existing methods apply policy updates to only a subset of the lifecycle, leaving at least one capability unoptimized, leading to optimization bottlenecks (Xia et al., 2026; Muhtar et al., 2026; Zhang et al., 2026b; Wang et al., 2025c). For example, a policy that has learned to use skills well still underperforms if it keeps routing to sub-optimal ones. (2) How can the three capabilities co-evolve toward a shared objective? Prior designs draw the rewards from different sources (Li et al., 2026b; Zhang et al., 2026b; Muhtar et al., 2026). For example, one capability may receive task-outcome reward while another relies on an auxiliary signal such as self-assessed quality or heuristic matching scores. Since the three capabilities jointly determine task success, optimizing them with inconsistent signals creates conflicting pressures.

We present Skill1, a framework that achieves unified evolution of skill-augmented agents by training a single policy to co-evolve skill selection, utilization, and distillation. As illustrated in Figure 1, given a new task, the policy first generates a natural-language query to retrieve candidate skills from the library, and then re-ranks the retrieved candidates to select the best match. The policy then performs multi-turn interaction with the environment conditioned on the top-ranked skill. After execution, the policy distills reusable skills from the experience based on its rollouts.

We achieve co-evolution of all three capabilities through credit assignment on a single task-outcome signal $r(\tau)$ . The outcome directly measures how well the policy solves the current task and serves as the utilization reward. To credit selection and distillation, we decompose this signal into its low-frequency trend and high-frequency variation. The low-frequency trend is defined as the moving average of outcomes associated with each skill. This term reflects skill utility and guides the policy toward consistently effective skills. The high-frequency variation is approximated with the deviation of the current outcome from the trend. This term captures whether a newly distilled skill improves upon the library’s current boundary, and rewards the policy for producing useful skills.

Figure 1: **Training paradigms for skill-augmented agents.** (a) The skill-augmented agent loop consists of selection, utilization, and distillation. (b) Prior methods delegate some stages to external modules without policy gradients (e.g., freezes selection or uses an external teacher for distillation). Skill1 trains a single policy across all three stages with a shared task-outcome signal.

We empirically evaluate Skill1 on ALFWorld (Shridhar et al., 2021) and WebShop (Yao et al., 2022a). Skill1 achieves 97.5% success rate on ALFWorld, surpassing all other baseline skill-augmented agents. Training dynamics confirm that selection precision, utilization success rate, and library quality improve simultaneously under the shared signal. Ablations show that removing any single stage’s credit-assignment signal degrades all three capabilities, providing evidence for their mutual dependence.

We formulate the skill-augmented agent learning problem as a POMDP (Lauri et al., 2022) $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{O},T,\Omega,R,\gamma)$ . A state $S=(x,e,\mathcal{B})$ comprises a task instruction $x$ from dataset $\mathcal{D}$ , the environment state $e$ , and a persistent skill library $\mathcal{B}=\{s_{1},s_{2},\ldots\}$ . At each turn the agent selects an action $a\in\mathcal{A}$ to send to the environment. The observation function $\Omega$ exposes a partial view $o_{t}=(x,\,e_{t},\,z)$ , where $z$ is the skill selected from $\mathcal{B}$ via a frozen encoder $\mathcal{E}$ . The overall training objective for the workflow can be defined as:

\max_{\theta}\;\mathbb{E}_{x\sim\mathcal{D},\,\tau\sim\pi_{\theta}(\cdot\mid x)}\bigl[r(\tau)\bigr],

where $\pi_{\theta}$ is optimized with RL algorithms such as GRPO (Shao et al., 2024) (cf. Appendix B).

A skill $s\in\mathcal{B}$ consists of a natural-language strategy $s.\text{strat}$ that describes how to act and a scenario description $s.\text{desc}$ that characterizes when the skill applies. The agent maintains the skill library $\mathcal{B}=\{s_{1},s_{2},\ldots\}$ as it continuously explores the environment. To reuse a skill, the agent generates its action conditioned on the skill strategy:

a_{t}\sim\pi_{\theta}(\cdot\mid x,\,z.\text{strat},\,o_{\leqslant t}).

To interact with a skill library, the agent selects skills from $\mathcal{B}$ , utilizes them during execution (Eq. 2), and distills new skills back into $\mathcal{B}$ . In §3, we show how to optimize all three stages jointly through a single policy, deriving every learning signal from the task outcome $r(\tau)$ .

Figure 2: **Overview of the Skill1 framework.** (a) The policy generates a query and re-ranks retrieved candidates to select a skill. (b) The policy performs multi-turn interaction conditioned on the selected skill. (c) The policy reflects on the trajectory and distills a reusable skill. All learning signals are derived from the task-outcome $r(\tau)$ to achieve co-evolution of three capabilities.

We introduce Skill1, a framework that trains a single policy $\pi_{\theta}$ to co-evolve skill selection, utilization, and distillation toward a shared task-outcome objective (Figure 2). We first describe the workflow (§3.1), then derive all learning signals from the task outcome $r(\tau)$ (§3.2), and finally formulate the joint optimization objective (§3.3).

For each task $x\sim\mathcal{D}$ , the policy $\pi_{\theta}$ performs three stages in sequence. A complete trajectory takes the form $\tau=(q,\,z,\,a_{1},o_{1},\ldots,a_{T},o_{T},\,s_{\text{new}})$ , where $q$ is the selection query, $z$ is the selected skill (or $\emptyset$ ), the action–observation pairs constitute the multi-turn interaction, and $s_{\text{new}}$ is the distilled skill. The environment returns a terminal reward $r(\tau)\in\{0,1\}$ . Prompt templates are in Appendix G.

Given a task $x$ , the policy generates a natural-language query $q\sim\pi_{\theta}(\cdot\mid x)$ to search the skill library $\mathcal{B}$ . A frozen encoder $\mathcal{E}$ retrieves the top- $K$ candidates by semantic similarity:

\mathcal{B}_{K}=\operatorname{top\text{-}K}_{s\in\mathcal{B}}\;\operatorname{sim}\bigl(\mathcal{E}(q),\,\mathcal{E}(s.\text{desc})\bigr).

The policy then re-ranks these candidates by generating a permutation $\sigma\sim\pi_{\theta}(\cdot\mid x,\mathcal{B}_{K})$ , and the top-ranked skill $z$ is provided for utilization. Both query generation and re-ranking are produced by $\pi_{\theta}$ , so selection is directly optimizable through the policy gradient.

The policy interacts with the environment for up to $T$ turns conditioned on the selected skill: $\tau\sim\pi_{\theta}(\cdot\mid x,\,z.\text{strat},\,o_{\leqslant t})$ . For each task, $G$ rollouts are sampled independently, each performing its own selection, utilization, and distillation.

After each rollout, $\pi_{\theta}$ reflects on the trajectory to produce: (i) a reusable strategy $s_{\text{new}}.\text{strat}\sim\pi_{\theta}(\cdot\mid x,\tau)$ summarizing the approach, and (ii) a scenario description $s_{\text{new}}.\text{desc}\sim\pi_{\theta}(\cdot\mid x,\tau)$ characterizing when the skill applies. A new skill is admitted to $\mathcal{B}$ only when $r(\tau)=1$ . When the library reaches its capacity $|\mathcal{B}|=N_{\text{max}}$ , the skill with the lowest retirement score $U(s)\cdot\log\bigl(n(s)\bigr)$ is removed, where $n(s)$ is the number of times $s$ has been selected. This heuristic retires skills that are both low-utility and infrequently used while preserving well-tested high-utility skills.

Co-evolution requires that each capability receives targeted learning signals from the shared task outcome $r(\tau)$ . The challenge is that the three capabilities operate at different temporal scopes: utilization concerns the current episode, selection concerns which skills are consistently effective across episodes, and distillation concerns whether new experience improves upon what the library already covers. We address this by decomposing $r(\tau)$ into its low-frequency trend and high-frequency variation, assigning credit to each capability without auxiliary models or additional rollouts.

The task outcome directly measures how well the policy executes with the given skill and serves as the utilization reward:

R^{\text{util}}_{i}\;=\;r(\tau_{i}).

Selection improves through two mechanisms. First, the query $q$ is part of the rollout prefix and receives policy gradients through the utilization objective (Eq. 8). Better queries retrieve better candidates and lead to higher $r(\tau)$ , so query quality co-improves with task performance without a dedicated reward.

Second, re-ranking requires an explicit signal that reflects long-term skill quality rather than single-episode outcomes. We maintain the trend of each skill as a per-skill utility score, updated after each rollout via exponential moving average:

U(s)\;\leftarrow\;(1-\alpha)\cdot U(s)\;+\;\alpha\cdot r(\tau_{i}),\quad\forall\,s\in\mathcal{B}_{K}.

We update all retrieved candidates rather than only the selected one, treating co-retrieval as evidence of relevance to the same task distribution. The trend smooths out per-episode variance and accumulates each skill’s long-term contribution. We denote the best available utility as $\hat{U}_{i}=\max_{s\in\mathcal{B}_{K}^{i}}U(s)$ , which serves as the library baseline for subsequent reward derivations. The trend supervises re-ranking by rewarding the policy for producing a permutation $\sigma_{i}$ that agrees with the utility ordering. Here we use normalized discounted cumulative gain (NDCG) as the rubric:

R^{\text{rerank}}_{i}\;=\;\mathrm{NDCG}\bigl(\sigma_{i},\;\operatorname{argsort}(-U(\mathcal{B}_{K}^{i}))\bigr).

The ideal distillation signal would measure whether a newly distilled skill improves future task performance, but that future outcome is unavailable at training time. We approximate it with the variation of the current outcome relative to the library’s trend:

R^{\text{distill}}_{i}\;=\;r(\tau_{i})\;-\;\hat{U}_{i},

where $\hat{U}_{i}=\max_{s\in\mathcal{B}_{K}^{i}}U(s)$ is the highest trend among the retrieved candidates. A positive variation indicates that the current experience surpasses what the library already covers, so the distilled skill is worth admitting. A negative variation discourages redundant distillation.

Algorithm 1 Pseudo Code of Skill1

\pi_{\theta}

\mathcal{B}

\mathcal{E}

K

G

\lambda_{1}

\lambda_{2}

\alpha

2: for batch of

N

tasks, each with

G

rollouts do

3: for sample

i=1,\ldots,N\cdot G

\triangleright

Agent workflow (Sec 3.1)

q_{i}\leftarrow\pi_{\theta}(x_{i})

\triangleright

Skill selection: search

\mathcal{B}_{K}\leftarrow\operatorname{top\text{-}K}_{s\in\mathcal{B}}\,\operatorname{sim}\bigl(\mathcal{E}(q_{i}),\mathcal{E}(s.\text{desc})\bigr)

\sigma_{i}\leftarrow\pi_{\theta}(x_{i},\mathcal{B}_{K})

;

z_{i}\leftarrow\mathcal{B}_{K}[\sigma_{i}(1)]

\triangleright

Skill Selection: re-rank

\tau_{i}\sim\pi_{\theta}(\cdot\mid x_{i},\,z_{i}.\text{strat})

\triangleright

Skill utilization

(s_{\text{new},i}.\text{strat},\,s_{\text{new},i}.\text{desc})\leftarrow\pi_{\theta}(x_{i},\tau_{i})

\triangleright

Skill distillation

9: end for

10:

R^{\text{util}}_{i}\leftarrow r(\tau_{i})

;

\hat{U}_{i}\leftarrow\max_{s\in\mathcal{B}_{K}^{i}}U(s)

\triangleright

Reward assignment (Sec 3.2)

11:

R^{\text{distill}}_{i}\leftarrow r(\tau_{i})-\hat{U}_{i}

\triangleright

Variation as distillation credit

12:

R^{\text{rerank}}_{i}\leftarrow\mathrm{NDCG}(\sigma_{i},\,\operatorname{argsort}(-U(\mathcal{B}_{K}^{i})))

\triangleright

Trend as selection credit

13:

U(s)\leftarrow(1\!-\!\alpha)\,U(s)+\alpha\,r(\tau_{i}),\;\forall s\in\mathcal{B}_{K}^{i}

\triangleright

Update utility scores

14: Admit

s_{\text{new},i}

\mathcal{B}

r(\tau_{i})=1

\triangleright

Update skill library

15:

\theta\leftarrow\theta+\nabla_{\theta}\bigl[\mathcal{J}^{\text{util}}+\lambda_{1}\mathcal{J}^{\text{rerank}}+\lambda_{2}\mathcal{J}^{\text{distill}}\bigr]

\triangleright

Joint optimization (Sec 3.3)

16: end for

Each rollout $\tau_{i}$ is a concatenation of four generation segments produced by $\pi_{\theta}$ : the selection query $q_{i}$ , the re-ranking permutation $\sigma_{i}$ , the action sequence $a_{1:T}$ , and the distilled skill $s_{\text{new},i}$ . We assign each segment its own reward signal (§3.2) and optimize them jointly in a single gradient step using GRPO (Shao et al., 2024) (cf. Appendix B), which normalizes rewards within the $G$ rollouts of each task into group-relative advantages.

The action tokens $a_{1:T}$ are conditioned on $(x_{i},z_{i})$ and optimized by the task outcome $R^{\text{util}}_{i}=r(\tau_{i})$ . The query $q_{i}$ precedes the actions in the same sequence and receives gradients through the same objective:

\mathcal{J}^{\text{util}}(\theta)=\mathcal{J}_{\text{GRPO}}\bigl(\theta;\,\{\tau_{1},\ldots,\tau_{G}\},\{\hat{A}_{1},\ldots,\hat{A}_{G}\}\bigr).

The permutation $\sigma_{i}$ is generated conditioned on the task $x_{i}$ and retrieved candidates $\mathcal{B}_{K}^{i}$ , and reinforced by the ranking reward $R^{\text{rerank}}_{i}$ . Since different rollouts generate different queries, their retrieved candidate sets $\mathcal{B}_{K}^{i}$ differ, thus inner group comparison becomes meaningless. We thus optimize each permutation independently with a REINFORCE-style (Williams, 1992) objective:

\mathcal{J}^{\text{rerank}}(\theta)=\frac{1}{N\cdot G}\sum_{i}R^{\text{rerank}}_{i}\cdot\log\pi_{\theta}(\sigma_{i}\mid x_{i},\mathcal{B}_{K}^{i}).

The distilled skill tokens $(s_{\text{new},i}.\text{strat},\,s_{\text{new},i}.\text{desc})$ are generated conditioned on the task $x_{i}$ and trajectory $\tau_{i}$ , and reinforced by the variation $R^{\text{distill}}_{i}$ . Advantages $\hat{A}_{i}^{\text{distill}}$ are normalized separately from those of utilization since the two rewards measure different aspects of same outcomes:

\mathcal{J}^{\text{distill}}(\theta)=\mathcal{J}_{\text{GRPO}}\bigl(\theta;\,\{s_{\text{new},1},\ldots,s_{\text{new},G}\},\,\{\hat{A}_{1}^{\text{distill}},\ldots,\hat{A}_{G}^{\text{distill}}\}\bigr).

All terms are combined in a single update:

\mathcal{J}(\theta)=\mathcal{J}^{\text{util}}(\theta)+\lambda_{1}\,\mathcal{J}^{\text{rerank}}(\theta)+\lambda_{2}\,\mathcal{J}^{\text{distill}}(\theta).

The utility score $U(s)$ is updated non-parametrically via Eq. (5). The full procedure is summarized in Algorithm 1. Training hyperparameter settings are in Appendix C.

We evaluate on ALFWorld (Shridhar et al., 2021), a text-based household environment requiring multi-step planning and object interaction, and WebShop (Yao et al., 2022a), an online-shopping simulator where agents search and purchase products matching user specifications. We report success rate (%) on the test split for both environments.

For Skill1, the initial policy is Qwen2.5-7B-Instruct (Yang et al., 2024) and the frozen encoder $\mathcal{E}$ is all-MiniLM-L6-v2 (Reimers and Gurevych, 2019). We train with GRPO under $G=16$ and lr $=1\times 10^{-6}$ . The skill library is initialized empty with capacity $|\mathcal{B}|\leqslant 5000$ . The training data uses the train split of the corresponding environments. Full hyperparameters are in Appendix C.

We compare three categories of methods in Table 1: (1) training-free agents such as ReAct (Yao et al., 2022b), Reflexion (Shinn et al., 2023), Mem0 (Chhikara et al., 2025), and ExpeL (Zhao et al., 2024); (2) RL-trained methods without skills such as PPO (Schulman et al., 2017), RLOO (Ahmadian et al., 2024), GRPO (Shao et al., 2024), and GiGPO (Feng et al., 2025); and (3) RL-trained methods with skills such as EvolveR (Wu et al., 2025), Mem0 and SimpleMem (Liu et al., 2026a) optimized with GRPO, SkillRL (Xia et al., 2026), and RetroAgent (Zhang et al., 2026b). All baselines use the same base model Qwen2.5-7B-Instruct for fair comparison.

Original Table 1 — Table 1: Main results on ALFWorld and WebShop (Success Rate, %). **Bold** denotes best results; ↑ indicates improvement over the previous best. “Avg.” stands for average success rate and “Succ.” stands for success rate.

Table 1
	ALFWorld (Success %)	WebShop
Method	Pick	Look	Clean	Heat	Cool	Pick2	Avg.	Score	Succ.
w/o Training
Zero-Shot	33.4	21.6	19.3	6.9	2.8	3.2	14.8	26.4	7.8
ReAct (Yao et al., 2022b)	48.5	35.4	34.3	13.2	18.2	17.6	31.2	46.2	19.5
Reflexion (Shinn et al., 2023)	62.0	41.6	44.9	30.9	36.3	23.8	42.7	58.1	28.8
Mem0 (Chhikara et al., 2025)	54.0	55.0	26.9	36.4	20.8	7.7	33.6	23.9	2.0
ExpeL (Zhao et al., 2024)	21.0	67.0	55.0	52.0	71.0	6.0	46.3	30.9	11.2
RL-Trained w/o Skills
PPO (Schulman et al., 2017)	92.3	64.0	92.5	89.5	80.3	68.8	80.4	81.4	68.7
RLOO (Ahmadian et al., 2024)	87.6	78.2	87.3	81.3	71.9	48.9	75.5	80.3	65.7
GRPO (Shao et al., 2024)	90.8	66.1	89.3	74.7	72.5	64.7	77.6	79.3	66.1
GiGPO (Feng et al., 2025)	97.7	82.7	98.8	83.7	89.3	79.2	90.8	84.4	72.8
RL-Trained w/ Skills
EvolveR (Wu et al., 2025)	64.9	33.3	46.4	13.3	33.3	33.3	43.8	42.5	17.6
Mem0 (Chhikara et al., 2025) w/ GRPO	78.1	54.8	56.1	31.0	65.0	26.9	54.7	58.1	37.5
SimpleMem (Liu et al., 2026a) w/ GRPO	89.5	36.3	60.0	50.0	64.9	26.3	62.5	67.8	46.9
SkillRL (Xia et al., 2026)	97.9	71.4	90.0	90.0	95.5	87.5	89.9	85.2	72.7
RetroAgent (Zhang et al., 2026b)	97.9	90.9	99.2	92.9	85.3	91.0	94.9	88.9	82.3
Skill1 (Ours)	100.0↑2.1	98.6↑7.7	97.3	99.2↑6.3	96.1↑0.6	96.0↑5.0	97.5↑2.6	89.7	82.9

Table 1 presents the main results. We reproduce RetroAgent with the official implementation and borrow other baseline results from prior research (Feng et al., 2025; Xia et al., 2026; Jiang et al., 2025a). Skill1 results are averaged across three runs, and we report statistical analysis in Appendix D.

Skill1 achieves the highest overall performance. On ALFWorld, Skill1 reaches 97.5% average success rate, surpassing the previous best RetroAgent by 2.6 points and ranking first on five out of six task types. The exception is Clean, where Skill1 falls slightly below strongest baseline RetroAgent, yet this gap is not statistically significant according to Appendix D. On WebShop, Skill1 also demonstrates the best performance across all methods.

An explicit skill library complements parameter-only RL. GiGPO, the strongest RL-only method, absorbs strategies implicitly into parameters and cannot explicitly reuse them across tasks. Skill1 surpasses it by 6.7 points, with the largest gains on Look and Pick2 where composing multiple sub-procedures benefits most from reusable skills.

Unified optimization outperforms methods that leave part of the lifecycle unoptimized. RetroAgent optimizes utilization and distillation with separate intrinsic rewards but provides no gradient signal for selection. SkillRL freezes its selection mechanism after cold-start SFT. Skill1 optimizes all three stages jointly through a single task-outcome signal. The comparison reveals a clear trend that agent performance increases with the degree of co-evolution.

We remove workflow components and zero out auxiliary objective weights to isolate each design choice. All variants share the same base model and training budget. Results are reported in Table 2.

Original Table 2 — Table 2: Ablation study on ALFWorld (Success Rate %). Upper block ablates workflow components; lower block ablates training objectives.

Table 2
	Pick	Look	Clean	Heat	Cool	Pick2	Avg.
Skill1	100.0	98.6	97.3	99.2	96.1	96.0	97.5
w/o Select.	96.9	90.3	98.0	90.4	86.5	85.3	91.8
w/o Distill.	97.4	88.5	98.1	96.1	87.6	89.5	92.4
w/o Library	96.7	71.5	94.9	70.7	71.5	65.5	80.9
w/ $\lambda_{1}{=}0$	99.5	80.5	98.8	100.0	90.6	84.9	94.0
w/ $\lambda_{2}{=}0$	100.0	85.4	95.5	96.4	91.0	96.2	94.9
w/ $\lambda_{1}{=}\lambda_{2}{=}0$	98.1	74.9	95.6	95.6	79.5	87.2	90.2

The skill library is the foundation, and distillation makes it effective. Removing the library entirely causes the largest drop, from 97.5% to 80.9%, with Heat and Pick2 losing over 28 points each. These task types require composing multi-step sub-procedures that benefit most from reusable skills. Removing distillation while keeping the library still reduces performance by 5.1 points. Without distillation the library stores raw trajectories rather than condensed strategies, making selection noisier and reuse less effective.

Selection loss propagates to downstream stages. Without selection the average drops by 5.7 points, concentrated on Heat and Pick2 where routing to the correct multi-step skill matters most. Notably, this degradation occurs even though the utilization reward remains intact, showing that poor skill routing bottlenecks the entire pipeline regardless of the policy’s solving ability.

The two auxiliary objectives are complementary. Setting $\lambda_{1}{=}0$ or $\lambda_{2}{=}0$ individually reduces performance by 3.5 and 2.6 points respectively. Removing both yields a sharper decline to 90.2%, worse than removing each stage individually. This gap shows that the signals benefit utilization beyond their direct targets, confirming that both signals are necessary to sustain full co-evolution.

Figure 3 tracks three capability metrics across training: (1) selection precision, the average skill utility scores $U(s)$ ; (2) task-outcome reward $r(\tau)$ for utilization; and (3) distillation positive rate, the fraction of new rollouts exceeding the average of retrieved ones $\hat{U}_{i}$ . We compare the full system against ablations that progressively remove credit-assignment signals.

Figure 3: Training dynamics of the three capability metrics. Full Skill1 achieves fast and unified convergence across all stages. Removing selection signal (green) or both selection and distillation signals (orange) slows convergence of all capabilities.

The three capabilities exhibit mutual reinforcement under unified training. Selection precision converges first, reaching 0.95 by step 20. The resulting high-quality skill supply then accelerates the other two stages, with both utilization and distillation reaching 0.8 by step 60. This sequential acceleration shows that improvements in one stage propagate forward through the lifecycle.

Ablating any credit-assignment signal slows all three capabilities. Removing the selection signal reduces selection precision as expected, but also drags down utilization and distillation because the policy routes to sub-optimal skills more frequently. Further removing distillation causes utilization scores to drop, even though it still receives its own direct reward. This suggests that each signal contributes to the overall growing trend, which is a direct evidence of co-evolution.

The previous section shows that capability metrics rise together. Here we examine the qualitative nature of that improvement: does the policy actually learn to select more relevant skills and distill higher-quality ones?

The policy learns to generate increasingly precise selection queries. Figure 5 measures task-skill similarity at three checkpoints. Full Skill1 improves from 0.51 to 0.60 across training because the trend signal rewards queries that retrieve historically high-utility skills, gradually sharpening the policy’s ability to describe what it needs. Removing the selection signal slows this learning, and without learned selection entirely, similarity stays almost flat at the lowest level.

The library ceiling rises as the policy learns to distill better skills. Figure 5 tracks $\hat{U}$ , the utility of the top-ranked skill per task. A rising $\hat{U}$ means increasingly effective skills are entering the library, not merely more skills. Full Skill1 reaches 0.91 by step 85 while both ablations lag by approximately 0.10. The variation signal creates this pressure: producing a skill similar to existing ones yields little reward, so the policy must discover genuinely better strategies to obtain positive gradient.

We examine whether the library is utilized as a diverse collective asset or collapses to a few dominant entries. Figure 6 visualizes the converged libraries with and without credit-assignment signals.

Figure 6: T-SNE visualization of the skill libraries after convergence, with and without RL-trained selection and distillation. The top-10 percent most frequently used skills are highlighted. Skill1 activates nearly twice as many high-frequency skills, and these skills span a broader strategy space.

Co-evolution activates a broader set of skills. Skill1 uses a broader set of skills. As observed in Figure 6, the skill usage count distributes more uniformly in the left panel. Without evolving signals (i.e., Skill1 w/o Select. and Distill.), the skill usage count distribution sharpens, where only a small number of popular skills are intensively utilized.

Frequently used skills cover diverse strategies. We also observe that the active skills in Skill1 span a much broader region of the strategy space. On the contrary, the popular skills (red and purple ones) on the right subfigure huddle together with only limited coverage. In the design of our method, producing a under-performing skill similar to existing ones yields negative reward, so the policy is pressured to cover underserved scenarios rather than duplicating successful ones.

We compare wall-clock time and library size for Skill1, SkillRL, and two ablations under identical hardware of 8 H800 80GB GPUs.

Original Table 3 — Table 3: Computational cost on ALFWorld training. We report wall-clock time per step (seconds) and library size (number of skills) at three checkpoints.

Table 3
	Time / Step (s)	Library Size
Method	Step 20	Step 60	Step 100	Step 20	Step 60	Step 100
GRPO (no library)	301.3	274.1	296.7	—	—	—
SkillRL	368.1	319.0	326.6	60	71	83
Skill1	386.6	444.3	493.8	915	3,899	5,000
w/o Select.	367.4	406.7	521.8	892	3,693	5,000
w/o Distill.	508.8	750.1	738.4	2,212	5,000	5,000

Skill1 adds moderate overhead over baseline methods. GRPO without a library runs at approximately 290s per step. SkillRL maintains near-constant cost because its library grows minimally from 60 to 83 skills, but this static library limits final performance to 89.9% compared to 97.5% for Skill1. Skill1 operates at 387 to 494s, roughly 1.3 to 1.7 times as slow as GRPO, with the increase stemming from the growing library context. The selection step itself adds negligible overhead as query generation and re-ranking operate on short sequences compared to multi-turn interactions against the environment.

Distillation controls both library quality and computational cost. Without distillation, raw trajectories enter the library directly, growing it at 2.4 times the rate of Skill1. The larger library lengthens the selection context, making the variant without distillation 69% slower by step 60 and saturating the 5,000-skill cap far earlier. Distillation compresses experience into concise skills, improving library quality while keeping computational cost in check.

We present Skill1, a framework that trains a single policy to co-evolve skill selection, utilization, and distillation toward a shared task-outcome objective. By decomposing this signal into its low-frequency trend and high-frequency variation, Skill1 derives per-capability credit assignment without auxiliary rewards. Experiments on ALFWorld and WebShop show consistent gains over prior skill-based and RL baselines, and ablations confirm that the three capabilities evolve in a coupled manner. A natural next step is to extend this lifecycle to hierarchical or multi-agent settings, where skill sharing and conflict resolution introduce new challenges for unified credit assignment.

While Skill1 achieves strong performance, several limitations remain.

•
Environment coverage. Our evaluation is limited to two representative text-based agent environments. Whether the co-evolution framework generalizes to more environments (e.g., deep search environments) or those with visual observations remains unexplored.
•
Scalability of the skill library. The library capacity in this work is capped at 5,000 entries. As the diversity of tasks grows, the fixed-size library may become a bottleneck, and more sophisticated eviction or hierarchical organization strategies may be required.

References

Abel et al. (2023) D. Abel, A. Barreto, B. Van Roy, D. Precup, H. P. van Hasselt, and S. Singh A definition of continual reinforcement learning. Advances in Neural Information Processing Systems 36, pp. 50377–50407. Cited by: Appendix A.
Ahmadian et al. (2024) A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker Back to basics: revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12248–12267. Cited by: §4.1, Table 1.
Anthropic (2025) Anthropic Introducing agent skills. Claude Blog. Cited by: Appendix A.
Auer et al. (2002) P. Auer, N. Cesa-Bianchi, and P. Fischer Finite-time analysis of the multiarmed bandit problem. Machine learning 47 (2), pp. 235–256. Cited by: Appendix A.
Chen et al. (2025) K. Chen, M. Cusumano-Towner, B. Huval, A. Petrenko, J. Hamburger, V. Koltun, and P. Krähenbühl Reinforcement learning for long-horizon interactive llm agents. arXiv preprint arXiv:2502.01600. Cited by: Appendix A.
Chhikara et al. (2025) P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav Mem0: building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413. Cited by: Appendix C, §4.1, Table 1, Table 1.
Fang et al. (2025) R. Fang, Y. Liang, X. Wang, J. Wu, S. Qiao, P. Xie, F. Huang, H. Chen, and N. Zhang MemP: exploring agent procedural memory. arXiv preprint arXiv:2508.06433. Cited by: Appendix A.
Feng et al. (2025) L. Feng, Z. Xue, T. Liu, and B. An Group-in-group policy optimization for llm agent training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: Appendix A, Appendix C, §4.1, §4.2, Table 1.
Gao et al. (2025) J. Gao, L. Pan, Y. Wang, R. Zhong, C. Lu, Q. Cai, P. Jiang, and X. Zhao Navigate the unknown: enhancing llm reasoning with intrinsic motivation guided exploration. arXiv preprint arXiv:2505.17621. Cited by: Appendix A.
Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1.
Hübotter et al. (2026) J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. Kleine Buening, C. Guestrin, and A. Krause Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802. Cited by: Appendix A.
Jiang et al. (2026a) G. Jiang, Z. Su, X. Qu, et al. XSkill: continual learning from experience and skills in multimodal agents. arXiv preprint arXiv:2603.12056. Cited by: Appendix A.
Jiang et al. (2026b) Y. Jiang, D. Li, H. Deng, B. Ma, X. Wang, Q. Wang, and G. Yu SoK: agentic skills–beyond tool use in llm agents. arXiv preprint arXiv:2602.20867. Cited by: Appendix A, §1.
Jiang et al. (2025a) Y. Jiang, L. Jiang, D. Teney, M. Moor, and M. Brbic Meta-rl induces exploration in language agents. arXiv preprint arXiv:2512.16848. Cited by: §4.2.
Jiang et al. (2025b) Y. Jiang, L. Jiang, D. Teney, M. Moor, and M. Brbic Meta-rl induces exploration in language agents. arXiv preprint arXiv:2512.16848. Cited by: Appendix A.
Lauri et al. (2022) M. Lauri, D. Hsu, and J. Pajarinen Partially observable markov decision processes in robotics: a survey. IEEE Transactions on Robotics 39 (1), pp. 21–40. Cited by: §2.
Li et al. (2026a) H. Li, C. Mu, J. Chen, S. Ren, Z. Cui, Y. Zhang, L. Bai, and S. Hu Organizing, orchestrating, and benchmarking agent skills at ecosystem scale. arXiv preprint arXiv:2603.02176. Cited by: Appendix A.
Li et al. (2026b) Y. Li, R. Miao, Z. Qi, and T. Lan Arise: agent reasoning with intrinsic skill evolution in hierarchical reinforcement learning. arXiv preprint arXiv:2603.16060. Cited by: Appendix A, §1.
Liu et al. (2026a) J. Liu, Y. Su, P. Xia, S. Han, Z. Zheng, C. Xie, M. Ding, and H. Yao SimpleMem: efficient lifelong memory for llm agents. arXiv preprint arXiv:2601.02553. Cited by: Appendix A, Appendix C, §4.1, Table 1.
Liu and Van Der Schaar (2025) T. Liu and M. Van Der Schaar Position: truly self-improving agents require intrinsic metacognitive learning. In Forty-second International Conference on Machine Learning Position Paper Track, Cited by: Appendix A.
Liu et al. (2026b) Z. Liu, J. Kim, X. Luo, D. Li, and Y. Yang Exploratory memory-augmented llm agent via hybrid on- and off-policy optimization. In The Fourteenth International Conference on Learning Representations, Cited by: Appendix A.
Lu et al. (2026) Z. Lu, Z. Yao, J. Wu, C. Han, Q. Gu, X. Cai, W. Lu, J. Xiao, Y. Zhuang, and Y. Shen SKILL0: in-context agentic reinforcement learning for skill internalization. arXiv preprint arXiv:2604.02268. Cited by: Appendix A, §1.
Madaan et al. (2023) A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: iterative refinement with self-feedback. Advances in neural information processing systems 36, pp. 46534–46594. Cited by: Appendix A.
Muhtar et al. (2026) D. Muhtar, J. Liu, W. Gao, W. Wang, S. Xiong, J. Huang, S. Yang, W. Su, J. Wang, L. Pan, et al. Complementary reinforcement learning. arXiv preprint arXiv:2603.17621. Cited by: Appendix A, Appendix A, §1, §1.
Putta et al. (2024) P. Putta, E. Mills, N. Garg, S. Motwani, C. Finn, D. Garg, and R. Rafailov Agent q: advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199. Cited by: Appendix A.
Qiao et al. (2026) J. Qiao, W. Meng, Y. Cheng, Z. Lin, Z. Zhang, X. Tan, J. Gong, K. Shao, and Y. Xie Memory intelligence agent. arXiv preprint arXiv:2604.04503. Cited by: Appendix A.
Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-BERT: sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3982–3992. Cited by: §4.1.
Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1, §4.1, Table 1.
Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: Appendix A, Appendix B, §1, §2, §3.3, §4.1, Table 1.
Sheng et al. (2024) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu HybridFlow: a flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256. Cited by: Appendix C.
Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. Advances in neural information processing systems 36, pp. 8634–8652. Cited by: Appendix A, §4.1, Table 1.
Shridhar et al. (2021) M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. J. Hausknecht ALFWorld: aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, Cited by: §1, §1, §4.1.
Song et al. (2024) Y. Song, D. Yin, X. Yue, J. Huang, S. Li, and B. Y. Lin Trial and error: exploration-based trajectory optimization of llm agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 7584–7600. Cited by: Appendix A.
Sutton and Barto (2018) R. S. Sutton and A. G. Barto Reinforcement learning: an introduction. 2nd edition, MIT Press. Cited by: §1.
Team et al. (2026) M. L. Team, A. Gui, B. Li, B. Tao, B. Zhou, B. Chen, C. Zhang, C. Gao, C. Zhang, C. Han, et al. Longcat-flash-thinking-2601 technical report. arXiv preprint arXiv:2601.16725. Cited by: §1.
Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1.
Wang et al. (2023) G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar Voyager: an open-ended embodied agent with large language models. In Intrinsically-Motivated and Open-Ended Learning Workshop@ NeurIPS2023, Cited by: §1.
Wang et al. (2025a) H. Wang, C. T. Leong, J. Wang, J. Wang, and W. Li SPA-rl: reinforcing llm agents via stepwise progress attribution. arXiv preprint arXiv:2505.20732. Cited by: Appendix A.
Wang et al. (2025b) J. Wang, J. Liu, Y. Fu, Y. Li, X. Wang, Y. Lin, Y. Yue, L. Zhang, Y. Wang, and K. Wang Harnessing uncertainty: entropy-modulated policy gradients for long-horizon llm agents. arXiv preprint arXiv:2509.09265. Cited by: Appendix A.
Wang et al. (2026) J. Wang, Y. Ming, Z. Ke, S. Joty, A. Albarghouthi, and F. Sala SkillOrchestra: learning to route agents via skill transfer. arXiv preprint arXiv:2602.19672. Cited by: Appendix A, §1.
Wang et al. (2025c) J. Wang, Q. Yan, Y. Wang, Y. Tian, S. S. Mishra, Z. Xu, M. Gandhi, P. Xu, and L. L. Cheong Reinforcement learning for self-improving agent with skill library. arXiv preprint arXiv:2512.17102. Cited by: Appendix A, §1.
Wang et al. (2025d) S. Wang, Y. Wu, and Z. Xu Cogito, ergo ludo: an agent that learns to play by reasoning and planning. arXiv preprint arXiv:2509.25052. Cited by: Appendix A.
Wang et al. (2025e) Z. Wang, K. Wang, Q. Wang, P. Zhang, L. Li, Z. Yang, X. Jin, K. Yu, M. N. Nguyen, L. Liu, et al. Ragen: understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073. Cited by: Appendix A.
Wei et al. (2025a) Q. Wei, S. Zeng, C. Li, W. Brown, O. Frunza, W. Deng, A. Schneider, Y. Nevmyvaka, Y. K. Zhao, A. Garcia, and M. Hong Reinforcing multi-turn reasoning in llm agents via turn-level reward design. arXiv preprint arXiv:2505.11821. Cited by: Appendix A.
Wei et al. (2025b) T. Wei, N. Sachdeva, B. Coleman, Z. He, Y. Bei, X. Ning, M. Ai, Y. Li, J. He, E. H. Chi, et al. Evo-memory: benchmarking llm agent test-time learning with self-evolving memory. arXiv preprint arXiv:2511.20857. Cited by: Appendix A.
Williams (1992) R. J. Williams Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8 (3), pp. 229–256. Cited by: §3.3.
Wu et al. (2025) R. Wu, X. Wang, J. Mei, P. Cai, D. Fu, C. Yang, L. Wen, X. Yang, Y. Shen, Y. Wang, et al. Evolver: self-evolving llm agents through an experience-driven lifecycle. arXiv preprint arXiv:2510.16079. Cited by: Appendix A, Appendix C, §1, §4.1, Table 1.
Xi et al. (2025) Z. Xi, Y. Ding, W. Chen, B. Hong, H. Guo, J. Wang, X. Guo, D. Yang, C. Liao, W. He, et al. AgentGym: evaluating and training large language model-based agents across diverse environments. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pp. 27914–27961. Cited by: §1.
Xia et al. (2026) P. Xia, J. Chen, H. Wang, J. Liu, K. Zeng, Y. Wang, S. Han, Y. Zhou, X. Zhao, H. Chen, et al. Skillrl: evolving agents via recursive skill-augmented reinforcement learning. arXiv preprint arXiv:2602.08234. Cited by: Appendix A, Appendix C, §1, §1, §4.1, §4.2, Table 1.
Xu and Yan (2026) R. Xu and Y. Yan Agent skills for large language models: architecture, acquisition, security, and the path forward. arXiv preprint arXiv:2602.12430. Cited by: Appendix A.
Yang et al. (2024) A. Yang, B. Yang, B. Zhang, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Cited by: §1, §4.1.
Yang et al. (2026) Y. Yang, J. Li, Q. Pan, B. Zhan, Y. Cai, L. Du, J. Zhou, K. Chen, Q. Chen, X. Li, et al. AutoSkill: experience-driven lifelong learning via skill self-evolution. arXiv preprint arXiv:2603.01145. Cited by: Appendix A.
Yao et al. (2022a) S. Yao, H. Chen, J. Yang, and K. Narasimhan WebShop: towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, pp. 20744–20757. Cited by: §1, §1, §4.1.
Yao et al. (2022b) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao React: synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, Cited by: Appendix A, §4.1, Table 1.
Yao et al. (2023) W. Yao, S. Heinecke, J. C. Niebles, Z. Liu, Y. Feng, L. Xue, R. Murthy, Z. Chen, J. Zhang, D. Arpit, et al. Retroformer: retrospective large language agents with policy gradient optimization. arXiv preprint arXiv:2308.02151. Cited by: Appendix A.
Ye et al. (2026) T. Ye, L. Dong, Q. Dong, X. Wu, S. Huang, and F. Wei Online experiential learning for language models. arXiv preprint arXiv:2603.16856. Cited by: Appendix A.
Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: an open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. Cited by: Appendix A.
Zhan et al. (2025) R. Zhan, Y. Li, Z. Wang, X. Qu, D. Liu, J. Shao, D. F. Wong, and Y. Cheng Exgrpo: learning to reason from experience. arXiv preprint arXiv:2510.02245. Cited by: Appendix A.
Zhang et al. (2025a) H. Zhang, X. Liu, B. Lv, X. Sun, B. Jing, I. L. Iong, Z. Hou, Z. Qi, H. Lai, Y. Xu, R. Lu, H. Wang, J. Tang, and Y. Dong AgentRL: scaling agentic reinforcement learning with a multi-turn, multi-task framework. arXiv preprint arXiv:2510.04206. Cited by: Appendix A.
Zhang et al. (2026a) H. Zhang, Q. Long, J. Bao, T. Feng, W. Zhang, H. Yue, and W. Wang MemSkill: learning and evolving memory skills for self-evolving agents. arXiv preprint arXiv:2602.02474. Cited by: Appendix A, §1.
Zhang et al. (2026b) X. Zhang, Z. Liu, Y. Zhang, X. Hu, and W. Shao RetroAgent: from solving to evolving via retrospective dual intrinsic feedback. arXiv preprint arXiv:2603.08561. Cited by: Appendix A, Appendix A, §1, §1, §4.1, Table 1.
Zhang et al. (2025b) X. Zhang, Y. Zhang, H. Sun, K. Feng, C. Lu, C. Yang, and H. Meng Critique-grpo: advancing llm reasoning with natural language and numerical feedback. arXiv preprint arXiv:2506.03106. Cited by: Appendix A.
Zhao et al. (2024) A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. Huang Expel: llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, pp. 19632–19642. Cited by: Appendix A, §1, §4.1, Table 1.
Zhou et al. (2025) H. Zhou, Y. Chen, S. Guo, X. Yan, K. H. Lee, Z. Wang, K. Y. Lee, G. Zhang, K. Shao, L. Yang, et al. Memento: fine-tuning llm agents without fine-tuning llms. arXiv preprint arXiv:2508.16153. Cited by: Appendix A.
Zhou et al. (2024) Y. Zhou, A. Zanette, J. Pan, S. Levine, and A. Kumar ArCHer: training language model agents via hierarchical multi-turn rl. In International Conference on Machine Learning, pp. 62178–62209. Cited by: Appendix A.

Core algorithmic advances include GRPO (Shao et al., 2024), anchor-state grouping (Feng et al., 2025), and dynamic sampling with asymmetric clipping (Yu et al., 2025). Multi-turn RL methods address long-horizon challenges through hierarchical value functions (Zhou et al., 2024), leave-one-out advantage estimation (Chen et al., 2025), MCTS-guided search (Putta et al., 2024), exploration-based trajectory optimization (Song et al., 2024), multi-turn self-evolution (Wang et al., 2025e; Zhang et al., 2025a), and cross-episode meta-RL (Jiang et al., 2025b). Recent work further refines credit assignment via stepwise progress attribution (Wang et al., 2025a; Wei et al., 2025a) or intrinsic exploration signals (Gao et al., 2025; Wang et al., 2025b). Prompt-based methods such as ReAct (Yao et al., 2022b) and Reflexion (Shinn et al., 2023) enable reasoning without parameter updates but are upper-bounded by the frozen policy (Abel et al., 2023). Skill1 extends GRPO by decomposing a single task-outcome signal into stage-specific gradients for selection, utilization, and distillation within a unified RL objective.

Structuring past experience for reuse improves RL sample efficiency (Zhan et al., 2025; Ye et al., 2026; Muhtar et al., 2026), and explicit memory systems that store interaction histories (Wei et al., 2025b; Liu et al., 2026a; Liu et al., 2026b) or distilled lessons (Fang et al., 2025; Zhou et al., 2025; Wang et al., 2025d) support continuous adaptation. RetroAgent (Zhang et al., 2026b) combines intrinsic progress rewards with language-based lesson extraction and a utility-aware selection strategy (Auer et al., 2002). Critique-GRPO (Zhang et al., 2025b) integrates natural-language critiques with numerical rewards, and RL-based self-distillation (Hübotter et al., 2026) refines failed trajectories into policy updates. Retrospective self-correction through natural-language critiques (Madaan et al., 2023; Yao et al., 2023) further enables agents to learn from failures (Liu and Van Der Schaar, 2025). Skill1 builds on these insights but derives all learning signals from a single task-outcome signal, eliminating the need for separate intrinsic reward design.

A growing body of work equips LLM agents with persistent skill libraries (Jiang et al., 2026b; Xu and Yan, 2026; Li et al., 2026a; Jiang et al., 2026a; Anthropic, 2025). For selection, approaches include frozen embedding selectors (Xia et al., 2026; Muhtar et al., 2026), heuristic scoring (Zhang et al., 2026b), learned routing (Zhang et al., 2026a; Wang et al., 2026), and policy log-probability ranking (Li et al., 2026b; Wu et al., 2025). For utilization, RL-based methods condition the policy on selected skills (Xia et al., 2026; Muhtar et al., 2026; Zhang et al., 2026b; Li et al., 2026b; Wang et al., 2025c), sometimes with hierarchical rewards to incentivize skill use (Li et al., 2026b; Muhtar et al., 2026). For distillation, methods range from prompt-based extraction (Zhao et al., 2024) and training-free skill versioning (Yang et al., 2026) to teacher-driven generation (Xia et al., 2026), co-evolving extractors (Muhtar et al., 2026), and self-reflection (Zhang et al., 2026b; Wang et al., 2025c; Wu et al., 2025; Qiao et al., 2026). Existing methods have not yet achieved RL-optimized status on all three stages simultaneously, and those that optimize multiple stages use heterogeneous learning signals without a unified objective. Skill0 (Lu et al., 2026) internalizes skills into model parameters with zero external skills; Skill1 co-evolves all three stages through one policy model and a unified task outcome signal.

We use Group Relative Policy Optimization (GRPO) (Shao et al., 2024) as the optimization method, which eliminates the need for a separate value network by computing advantages relative to a group of rollouts sampled from the same task. For each task $d$ , a group of $G$ rollouts $\{\tau_{i}\}_{i=1}^{G}$ is sampled from $\pi_{\theta_{\text{old}}}$ . The group-relative advantage for rollout $i$ is:

\hat{A}_{i}=\frac{r(\tau_{i})-\operatorname{mean}(\{r(\tau_{1}),\ldots,r(\tau_{G})\})}{\operatorname{std}(\{r(\tau_{1}),\ldots,r(\tau_{G})\})}.

Let $\rho_{t}^{(i)}(\theta)=\pi_{\theta}(a_{t}^{(i)}\mid s_{t}^{(i)})/\pi_{\theta_{\text{old}}}(a_{t}^{(i)}\mid s_{t}^{(i)})$ denote the per-token importance ratio. The GRPO objective maximizes the clipped surrogate:

\mathcal{J}_{\text{GRPO}}(\theta)=\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|\tau_{i}|}\sum_{t=1}^{|\tau_{i}|}\min\!\bigl(\rho_{t}^{(i)}\hat{A}_{i},\;\operatorname{clip}(\rho_{t}^{(i)},1{-}\epsilon,1{+}\epsilon)\,\hat{A}_{i}\bigr)-\beta\,D_{\text{KL}}\!\bigl[\pi_{\theta}\|\pi_{\text{ref}}\bigr],

where $\epsilon$ is the clipping ratio, $\beta$ controls KL regularization toward a reference policy $\pi_{\text{ref}}$ , and $|\tau_{i}|$ is the number of tokens in rollout $i$ .

Skill1 is trained on 8 NVIDIA H800-80GB GPUs using the VeRL framework (Sheng et al., 2024) with Fully Sharded Data Parallelism (FSDP) under BFloat16 precision. Rollout generation uses vLLM with tensor parallelism of 4. Training converges in approximately 100 to 150 steps (roughly 30 hours on ALFWorld). The auxiliary objective weights are $\lambda_{1}=\lambda_{2}=0.3$ throughout all experiments unless otherwise specified.

We reproduce RetroAgent using its official implementation. $^{1}$ For SkillRL, EvolveR, Mem0, and SimpleMem, we use numbers reported in their respective papers (Xia et al., 2026; Wu et al., 2025; Chhikara et al., 2025; Liu et al., 2026a) under the same base model (Qwen2.5-7B-Instruct). GiGPO results are taken from Feng et al. (2025). All RL baselines use identical training budgets (150 epochs) and the same train/test splits to ensure fair comparison.

Table 4 lists the shared training hyperparameters across both environments. Table 5 lists the per-environment differences. Table 6 lists the skill library configuration.

Original Table 4 — Table 4: Shared training hyperparameters.

Table 4
Hyperparameter	Value
Optimization
Algorithm	GRPO
Learning rate	$1\times 10^{-6}$
KL loss coefficient	0.01
KL loss type	low-variance KL
PPO mini-batch size	256
PPO micro-batch size per GPU	16
Gradient checkpointing	True
Re-ranking loss weight $\lambda_{1}$	0.3
Distillation loss weight $\lambda_{2}$	0.3
Rollout
Group size $G$	16
Max prompt length	16,384 tokens
Max response length	2,048 tokens
vLLM tensor parallelism	4
GPU memory utilization	0.7
Validation temperature	0.4

Original Table 5 — Table 5: Per-environment hyperparameters.

Table 5
Hyperparameter	ALFWorld	WebShop
Training batch size	16	32
Validation batch size	64	128
Max environment steps	50	15

Original Table 6 — Table 6: Skill library configuration.

Table 6
Parameter	Value
Selection
Encoder	all-MiniLM-L6-v2 (384-dim)
Top- $K$ candidates	5
Training selection strategy	UCB
Evaluation selection strategy	Greedy
UCB exploration scale	1.0
Similarity weight $w_{\text{sim}}$	0.6
Library Management
Maximum library size	5,000
Utility EMA rate $\alpha$	0.05

We run all methods with 3 independent random seeds and report mean $\pm$ standard deviation (1- $\sigma$ ). The primary source of variability is the random seed, which affects parameter initialization, rollout sampling order, and skill library evolution trajectory. We use SciPy’s ttest_ind with equal_var=False (Welch’s t-test) to assess statistical significance.

We select RetroAgent as the strongest baseline and run it with 3 independent seeds under identical conditions to obtain variance estimates. Figure 7 reports per-task-type success rates (mean $\pm$ std) on ALFWorld.

Skill1 achieves statistically significant improvement over RetroAgent. On the aggregate metric (ALF All), Skill1 achieves 97.5 $\pm$ 0.6 versus RetroAgent’s 94.9 $\pm$ 0.9. A Welch’s t-test on the 3-seed averages yields $t=4.06$ , $\mathrm{df}=3.40$ , $p=0.021$ ( $<0.05$ ). The result confirms that the gain is not attributable to seed variance. Per-task significance is strongest on the tasks where RetroAgent struggles most: Heat ( $p=0.004$ ), Cool ( $p=0.005$ ), and Look ( $p=0.020$ ). The sole exception is Clean, where Skill1 trails RetroAgent by 1.9 points. This difference is not statistically significant ( $p=0.147>0.05$ ) and falls within normal seed variance.

Skill1 exhibits lower aggregate variance than RetroAgent. Skill1’s overall standard deviation (0.6) is smaller than RetroAgent’s (0.9), indicating more stable convergence across seeds. The unified evolution framework, where selection, utilization, and distillation reinforce each other, reduces sensitivity to initialization.

Figure 7: Per-task success rates (mean $\pm$ std over 3 seeds). Skill1 consistently outperforms best baseline RetroAgent on five of six task types and the average score.

This work develops a framework for LLM agents to autonomously acquire and reuse behavioral skills through reinforcement learning. On the positive side, the approach can reduce the manual engineering effort required to build capable agents and enable more sample-efficient learning in interactive environments.

On the negative side, agents that autonomously accumulate skills may exhibit emergent behaviors that are difficult to predict or audit. In high-stakes deployment scenarios, an unconstrained skill library could encode harmful action sequences, and brings new injection risks. We recommend deploying such systems with human-in-the-loop oversight and constraining the action space in safe domains.

We present two representative case studies from the ALFWorld evaluation, comparing Skill1 against RetroAgent on the same test task. Each case demonstrates a different transfer mechanism (failure avoidance and error correction) and highlights why unified evolution of selection, utilization, and distillation produces qualitatively different behavior from baselines that lack joint optimization.

Graphic from the paper; its text follows

Discussion. Both cases illustrate how the co-evolved skill library captures knowledge that goes beyond surface-level pattern matching. Case 1 demonstrates failure avoidance: RetroAgent lacks a high-utility skill encoding the stoveburner constraint because its selection mechanism is not optimized to route heat-tasks to the relevant skill. Skill1 retrieves the correct skill and explicitly cites it in its reasoning chain. Case 2 demonstrates error correction: RetroAgent picks the wrong alarmclock instance because its library does not preserve the targeting lesson from prior failures with sufficient utility. Skill1’s variation-driven distillation retains such lessons and the trend-driven selection surfaces them at test time. In both cases, Skill1 achieves near-optimal trajectories while the baseline exhausts steps on avoidable mistakes.

We list the prompt templates used in each stage of Algorithm 1:

•
Selection (Query generation) (line 3): $\pi_{\theta}$ generates query $q$ to retrieve candidates from $\mathcal{B}$ .
•
Selection (Re-ranking) (line 6): $\pi_{\theta}$ ranks $\mathcal{B}_{K}$ and selects the top skill $z$ .
•
Utilization (line 8): $\pi_{\theta}$ interacts with the environment conditioned on $z.\text{strat}$ .
•
Distillation (line 9): $\pi_{\theta}$ reflects on $\tau$ and produces $s_{\text{new}}$ .

Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning

Abstract

References

Meow summary

The idea and the problem

The central trick

Why it helps—and where it stops

Make notes of your own

Abstract

1 Introduction

2 Preliminary: LLM Agent with Skill Library

Task formulation.

Skills for LLM agents.

3 Method

3.1 Agent Workflow

Skill selection.

Skill utilization.

Skill distillation.

3.2 Reward Assignment

Crediting utilization.

Crediting selection.

Crediting distillation.

3.3 Joint Optimization

Utilization and query.

Re-ranking.

Distillation.

Total objective.

4 Experiments

4.1 Experimental Setup

Environments.

Training.

Baselines.

4.2 Main Results

4.3 Analysis

4.3.1 Ablation Study

4.3.2 Co-evolution Dynamics

4.3.3 Selection and Distillation Quality

4.3.4 Skill Library Diversity

4.3.5 Computational Overhead

5 Conclusion and Limitations

Conclusion.

Limitations.

References

Appendix A Related Work

Reinforcement Learning for LLM Agents.

Experience Reusing.

Skill Libraries for LLM Agents.

Appendix B Algorithm Details

Appendix C Implementation Details

Training infrastructure.

Baseline reproduction.

Hyperparameters.

Appendix D Statistical Analysis

D.1 Full Performance Breakdown

D.2 Analysis

Appendix E Broader Impacts

Appendix F Case Studies

Appendix G Prompt Templates

G.1 ALFWorld

G.2 WebShop

Meow summary

The idea and the problem

The central trick

Why it helps—and where it stops

Make notes of your own