AI Agents · Multimodal Models

COLLEAGUE.SKILL vs MMSkills: What Belongs Inside an Agent Skill Package

Both papers make the agent skill a package, not a prompt or memory fragment. They disagree on what goes inside: a two-track text artifact distilled from one person, or multimodal packages mined from trajectories.

COLLEAGUE.SKILL vs MMSkills: What Belongs Inside an Agent Skill Package

The one design question

The agent-skills format went mainstream in 2025–2026: a “skill” ships as a folder an agent can load, not as a hidden memory store or a hand-tuned persona prompt. Once the container exists, the real research question becomes what goes inside it and where the contents come from. COLLEAGUE.SKILL and MMSkills answer with two different philosophies, and their numbers are so different in kind that comparing them honestly is most of the work.

COLLEAGUE.SKILL starts from human evidence: it distills one person’s heterogeneous work traces (documents, decisions, message threads) into a versioned skill package with two coordinated tracks. MMSkills starts from machine evidence: it mines public interaction trajectories (OpenCUA) into compact multimodal packages for visual agents. One optimizes the artifact for auditability and correction; the other optimizes it for state recognition in a live GUI or game.

Key numbers

MeasurementCOLLEAGUE.SKILLMMSkillsWhat it actually measures
Package contentsTwo tracks: capability (practices, mental models, heuristics) + bounded behavior (style, rules, correction history)Three parts: textual procedure + runtime state cards + multi-view keyframesDesign choice, not a score
Package sourceOne person’s work traces, distilled end-to-end4,205 Ubuntu + 3,157 macOS OpenCUA trajectories, mined automaticallyAuthorship pipeline
Library sizeGallery of 215 skills from 165 contributors247 packages for OSWorld, 248 for macOSWorld, 24 for VAB-Minecraft, 10 for Super Mario BrosEcosystem vs benchmark coverage
Adoption signal~18.5k GitHub stars; 100k+ cumulative stars across listed skill cardsNot reportedCommunity uptake; not a quality metric
OSWorld, Qwen3-VL-235BNot evaluated21.34% → 39.17% successTask success, same benchmark
OSWorld, Qwen3-VL-8B-InstructNot evaluated10.78% → 25.40%Task success; more than doubles the small model
OSWorld, Gemini 3.1 ProNot evaluated44.08% → 50.11%Task success on a frontier model
OSWorld, Gemini 3 FlashNot evaluated36.65% → 47.97%Task success on a fast tier
VAB-Minecraft, Qwen3-VL-8B-InstructNot evaluated23.28% → 38.79%Transfer beyond GUI to a game environment
Benchmark fidelity studyNone (no task-success or human study reported)None (no human-fidelity measure of skill authorship)The gap both papers leave

Read the table column by column and the asymmetry is the story. Every COLLEAGUE.SKILL number is an adoption or design number: stars, contributors, gallery size, rollback capability. Every MMSkills number is a benchmark delta on OSWorld, macOSWorld, VAB-Minecraft and Super Mario Bros. Neither paper measures what the other one does. COLLEAGUE.SKILL cannot tell you whether a distilled skill makes an agent better at a task, and MMSkills cannot tell you whether its mined packages faithfully encode anything a human would call expertise.

What goes inside the package

The two-track design of COLLEAGUE.SKILL separates what someone can do from how they do it. The capability track holds transferable know-how: heuristics, mental models behind decisions, standard practices. The bounded behavior track holds the interaction layer (tone, the rules the person holds to) plus a running correction log. “Bounded” is deliberate: behavior stays inside explicit, auditable limits rather than free-associating from a vibe, and the whole package can be read, diffed and reverted like code.

MMSkills answers a different failure. For a visual agent, the hard part of a procedure is rarely the action; it is reading the screen to know which state it is in and whether the last step worked. A text procedure cannot encode “the file picker looks like this” or “you are done when this badge turns green.” So each MMSkills package bundles three parts: the ordered textual plan with preconditions, state cards describing the runtime states the agent will pass through, and multi-view keyframes (screenshots anchored to those states as visual evidence of progress, success or failure). Loading is branch-based: the agent retrieves and loads only the package branch relevant to the current state, so it consults visual evidence without paying context-window cost for the whole library on every step.

How packages are authored

COLLEAGUE.SKILL is a trace-to-skill distillation pipeline with a lifecycle attached. Raw expert material goes in; a structured, inspectable package comes out; the owner then updates it by typing natural-language feedback (“you over-explain, be terser”), rolls back when an edit regresses, and installs the same package across different agent hosts. Domain presets cover common roles. The design bet is that the valuable unit is a correctable artifact, not a frozen prompt.

MMSkills is automatic all the way down: 4,205 Ubuntu and 3,157 macOS trajectories from OpenCUA are mined into 247 unique packages covering OSWorld’s 360 test cases and 248 covering macOSWorld’s 143 cases, with 24 packages for VAB-Minecraft and 10 for Super Mario Bros. Nobody hand-writes a procedure. The cost of that automation is inheritance: a task with no precedent in OpenCUA gets no package, and the library’s coverage is lopsided, 247 packages for OSWorld versus 10 for Super Mario Bros, which is why the game results should be read cautiously.

Reading the MMSkills numbers honestly

The pattern inside the MMSkills results matters more than any single row. The largest gain is on the smallest model: Qwen3-VL-8B-Instruct more than doubles, from 10.78% to 25.40% on OSWorld, and jumps from 23.28% to 38.79% on VAB-Minecraft. The 235B model gains 17.8 points, and the frontier models gain least in relative terms: Gemini 3.1 Pro moves 44.08% to 50.11%, about six points. The straightforward reading is that skill packages lend procedural competence which small models do not have and frontier models mostly already carry internally. That is exactly the economics of cost-sensitive GUI automation, where a small deployable agent is the point. It also sets expectations: if your base model is already frontier-tier, the same library buys you a modest single-digit-to-low-teens bump, not a doubling.

Reading the COLLEAGUE.SKILL numbers honestly

The headline figures (roughly 18.5k GitHub stars, a gallery of 215 skills from 165 contributors, over 100k cumulative stars across listed skill cards) are real-world adoption signals, and strong ones. But they are not quality measurements. The paper reports no benchmark, no controlled human study and no task-success comparison, so the central claim that a distilled skill reproduces an expert’s judgment is demonstrated by a working system and community uptake, not by measured fidelity. Two harder questions stay open: distilling a real person’s traces raises consent and likeness concerns that “controlled distribution” only gestures at, and bounded behavior is only as safe as the bounds someone remembered to write.

When to use which

  • Your bottleneck is a visual agent reading the screen (GUI automation, games): MMSkills-style multimodal packages. Text-only procedures cannot encode state recognition, and the OSWorld deltas say the format pays.
  • Your asset is one specific person’s expertise (a senior operator, a specialist’s judgment): COLLEAGUE.SKILL-style two-track packages, because you need the correction lifecycle and rollback more than you need benchmark deltas.
  • You are building a library at scale: mine trajectories like MMSkills; per-person distillation does not scale to hundreds of tasks.
  • You need auditable behavior limits: prefer the two-track split, where bounded behavior is an explicit, inspectable track with a correction history.
  • You need proof the skill works: neither paper gives you the full answer. MMSkills measures task success without fidelity, COLLEAGUE.SKILL offers fidelity-shaped design without measurement. Budget your own evaluation either way.

Limits and open questions

The missing experiment in this comparison is a single study that measures both task success and authorship fidelity for the same package format: packages distilled from a person and tested on a benchmark, or mined packages audited against human judgment. Until that exists, “multimodal packages boost task success” and “distilled packages carry human expertise” are two separate claims supported by two separate papers. There is also a quiet dependency inside MMSkills: branch loading only helps if retrieval picks the right package branch, and a wrong branch would actively mislead the agent, a component that is hard to audit from headline results. And both papers scope their claims tightly: MMSkills is evaluated on GUI control and games, not open-ended web or physical tasks, while COLLEAGUE.SKILL’s consent story for real-person distillation remains an open legal and social question.

FAQ

What is the difference between COLLEAGUE.SKILL and MMSkills?

COLLEAGUE.SKILL distills one person’s work traces into a versioned, two-track text package: a capability track for practices and decision heuristics, plus a bounded behavior track for style, rules and corrections. MMSkills mines public interaction trajectories into multimodal packages (textual procedure, runtime state cards and keyframes) that visual agents load on demand. One is about capturing how a person works; the other is about helping an agent read the screen.

How much does MMSkills improve OSWorld scores?

On OSWorld it raises Qwen3-VL-235B from 21.34% to 39.17%, Qwen3-VL-8B-Instruct from 10.78% to 25.40%, Gemini 3.1 Pro from 44.08% to 50.11%, and Gemini 3 Flash from 36.65% to 47.97%. The same packages lift the 8B model from 23.28% to 38.79% on VAB-Minecraft.

Does COLLEAGUE.SKILL have benchmark results?

No. The paper reports adoption figures, roughly 18.5k GitHub stars, 215 gallery skills from 165 contributors, over 100k cumulative stars across listed skill cards, but no benchmark or controlled human study measuring how well a distilled skill reproduces an expert’s judgment.

Where do MMSkills packages come from?

They are mined automatically from public interaction trajectories: 4,205 Ubuntu and 3,157 macOS trajectories from OpenCUA, yielding 247 packages for OSWorld, 248 for macOSWorld, 24 for VAB-Minecraft and 10 for Super Mario Bros.

Are multimodal skill packages only useful for small models?

No, but they help small models most. Qwen3-VL-8B-Instruct more than doubles its OSWorld success, while Gemini 3.1 Pro gains about six points, evidence that frontier models already carry much of the procedural knowledge and that packages lend the most competence to the smallest agents.