Role-Agent vs Skill1: Making Agent RL Compound Without External Rewards

Quick answer

Role-Agent and Skill1 answer the same question with different machinery: how does an LLM agent trained with RL keep improving without an external reward model or a hand-built simulator? Role-Agent looks inward. One model alternates between acting and simulating, pays itself a predictive reward for forecasting environment states, and builds its own curriculum from failed trajectories. Skill1 looks outward. One GRPO policy learns to retrieve skills from a library, apply them, and write new ones, all under the same task-outcome reward.

On the benchmarks both papers report, Skill1 posts the stronger 7B numbers: 97.5% on ALFWorld against 93.8% for Role-Agent, with both papers measuring the same GiGPO baseline at 90.8% on the same backbone family. That head-to-head has caveats, and this page walks through them. The two methods are also not exclusive: Role-Agent’s modulation layer could sit on top of a Skill1-style policy.

The one design difference

Role-Agent’s two halves are World-In-Agent (WIA) and Agent-In-World (AIW), and the same LLM wears both hats. As the agent, it predicts the next H environment states after each action; a longest-matching-subsequence score between prediction and reality becomes a predictive reward that multiplies the task reward. As the environment, it reads failed trajectories, names the failure mode in words, and retrieves training tasks that share that mode. Nothing new is trained except the policy itself.

Skill1’s three operations are skill selection, skill utilization, and skill distillation, folded into one policy trained with GRPO. After each episode the policy can distill a new skill into a library capped at 5,000 entries; an exponential moving average tracks whether each skill actually pays off, and the library prunes what does not. Two auxiliary losses, each weighted 0.3, shape selection and distillation.

Both papers borrow the optimizer rather than inventing one. Role-Agent bolts onto GiGPO; Skill1 uses plain GRPO with a group size of 16 and a learning rate of 1e-6. The contribution in each case is credit assignment: where does the training signal come from, and how does it reach the parts of the system that need it?

Key numbers

Measurement	Role-Agent	Skill1	Setting and source
ALFWorld avg success, Qwen2.5-1.5B-Instruct	90.9% vs GiGPO 86.7% (+4.2)	not run	Role-Agent paper
ALFWorld hard subtasks, 1.5B	Pick2 +13.6%, Look +11.0% over GiGPO	not run	Role-Agent paper
WebShop, 1.5B	71.9% vs GiGPO 65.0% (+6.9)	not run	Role-Agent paper
Search QA, Qwen2.5-3B-Instruct	45.8% vs GiGPO 42.1%; 2WikiMultiHopQA +8.2, MuSiQue +5.2	not run	Role-Agent paper, trained on NQ and HotpotQA
ALFWorld avg success, Qwen2.5-7B-Instruct	93.8% vs GiGPO 90.8%	97.5% vs GiGPO 90.8% (paper reports +6.5)	both papers, same backbone family
WebShop, 7B	77.1% success vs GiGPO 72.8%	89.7 score, 82.9% success	different metrics; the success rates are the comparable pair
Skill1 per-category ALFWorld, 7B	not reported	Pick 100.0, Look 98.6, Heat 99.2, Clean 97.3, Cool 96.1, Pick2 96.0	Skill1 paper
Ablations	remove AIW: WebShop −5.0, ALFWorld −3.4; remove predictive reward: −3.6, −2.9	remove library: 97.5 → 80.9 (−16.6); remove selection: −5.7; remove distillation: −5.1	each paper’s own ablation
Internal signal detail	predictive reward correlates 0.41 with outcome reward (p < 0.01); prediction alignment 0.60 at init to the 0.70s near convergence; best horizon about 5% of max steps, 10% hurts	skill utility EMA rate 0.05; aux loss weight 0.3 each; library cap 5,000 skills	each paper
Training cost	WIA and AIW add runtime on top of baseline generation	100 to 150 steps, about 30 hours on 8 H800 GPUs	each paper

Two rows deserve a careful read. First, the ALFWorld 7B row is a genuine cross-paper anchor: both papers report GiGPO at 90.8% on Qwen2.5-7B, so the gain over a shared baseline is +3.0 points for Role-Agent and +6.5 as reported for Skill1. Each paper ran its own GiGPO baseline, so this is consistency between papers, not a shared harness. Second, the WebShop row mixes metrics: Skill1’s 89.7 is a score while its 82.9% and Role-Agent’s 77.1% are success rates. Comparing 89.7 against 77.1 directly would be wrong.

Why both beat the same baseline

GiGPO, group-relative policy optimization adapted to agents, is the incumbent both papers measure against, and both beat it by finding signal that pure outcome reward throws away. Role-Agent’s argument is that the model already knows more than the reward records: an agent that can predict what its action will do to the environment understands the dynamics better, so its credit should be scaled up. The predictive reward correlates only 0.41 with actual outcomes, which looks too weak to matter, but it multiplies the task reward rather than replacing it. That is enough to up-weight foreseeable actions and down-weight lucky ones.

Skill1’s argument is that experience itself is a signal the baseline discards. An agent that solved a task once should not relearn it from scratch, yet GiGPO starts every episode blank. Routing one outcome reward through selection, utilization, and distillation makes the library compound. The library ablation is the biggest single number on this page: removing it drops ALFWorld from 97.5% to 80.9%, a 16.6-point loss, far larger than any other ablation in either paper.

Neither result is evidence of a new state of the art in RL optimizers. Both are better credit assignment on top of group-relative updates, and both gains shrink or stay unproven as conditions move away from structured benchmarks with checkable success signals.

When to use which

Reach for Role-Agent when three things hold. Your tasks have sparse rewards and no dense simulator feedback. Your base model is small, roughly 1.5B to 7B, where the paper shows the gain. And you already run a GiGPO or GRPO loop and want a cheap addition: no extra labels, no extra trained models, though WIA and AIW do add per-step runtime.

Reach for Skill1 when your tasks come in families with repeated structure, the outcome reward is checkable, and failed or successful episodes should accumulate into something reusable. The 30-hour, 8 GPU training cost and the 100 to 150 step convergence are modest by agentic RL standards, but you pay for it in system complexity: an embedding encoder, an EMA utility tracker, and a pruning rule.

A few combinations and non-cases. If you have dense rewards or a clean simulator, Role-Agent’s predictive half buys little, and its own authors suggest keeping only the failure-retrieval curriculum. If your tasks never repeat, Skill1’s library has nothing to compound. If your base model is frontier-scale, neither paper has evidence for you: Role-Agent’s gain narrows from 1.5B to 7B, and Skill1 is tested only at 7B. Adjacent work points in the same direction: Ctx2Skill distills skills from context with self-play and no external reward, and OpenSkill builds skills plus their own verifiers from the open web.

Limits and open questions

Both evaluations live on Qwen2.5, both stop at 7B, and both lean on structured environments where success is checkable: ALFWorld’s household simulator, WebShop’s scripted web store. Role-Agent adds search QA as a third family; Skill1 does not. Neither tests code, tool-calling against live APIs, or long-horizon real-world tasks where outcomes are ambiguous.

Role-Agent’s open questions are about the internal signal. The failure-mode extraction depends on the same LLM correctly naming why it failed, with no quantitative check on how often that naming is right. The per-step runtime overhead is reported but not converted into a cost-per-point figure, so the trade against simply running more rollouts is unclear. The arXiv version is marked work in progress.

Skill1’s open questions are about the library. The 5,000-skill cap and EMA pruning are engineering choices without an analysis of what happens when the library saturates or skills conflict. A single 7B backbone leaves the scaling question open in both directions: a stronger base model might close the gap with simpler baselines, or a library might matter more at scale. Both papers, finally, assume a checkable outcome reward, which most messy real tasks do not provide.

FAQ

Which gets better ALFWorld numbers, Role-Agent or Skill1?

Skill1, on the one setting where the papers overlap. At Qwen2.5-7B, Skill1 reports 97.5% average success against Role-Agent’s 93.8%, and both papers measure GiGPO at 90.8%. Skill1 also reports per-category rates from 96.0% to 100.0%. Read this as Skill1’s library compounding well on a structured benchmark, not as a universal ranking.

Why do both papers benchmark against GiGPO?

GiGPO is the standard group-relative RL baseline for LLM agents, adapted from GRPO to agentic settings. Beating it is the entry ticket for any new agentic RL method in 2026, which is also why its 90.8% appears in both papers: it is the shared reference point that makes the cross-paper comparison possible.

Can Role-Agent’s predictive reward be combined with a Skill1-style library?

Nothing in either paper rules it out, and the mechanisms touch different parts of the loop. The predictive reward modulates credit within an episode; the library stores credit across episodes. A policy that retrieves skills and pays itself a forecast-based bonus is a plausible third system, but no published numbers exist for it yet.

What are the main limitations of each method?

Role-Agent is tested on Qwen2.5 at 1.5B to 7B, gains about 4% on average, and narrows as the base model grows; its self-reported failure analysis has no accuracy measurement. Skill1 is tested on a single 7B model across two structured benchmarks, with library management rules that are unprincipled and unproven at scale.

Do the gains survive as the base model gets bigger?

Unknown, and the one datapoint points the wrong way. Role-Agent’s gain shrinks from roughly 4.2 points at 1.5B to about 3.0 points at 7B, consistent with a stronger base model already internalizing some of what the predictive signal supplies. Skill1 has no larger-model result in either direction.