RL Elicits Contextual Learning of Unseen Language Translation

Quick answer

The paper trains an LLM with reinforcement learning to translate languages it has never seen, using only a surface-level chrF score as reward and a packet of linguistic context in the prompt: a dictionary, a few parallel sentences, and grammar-book excerpts. The headline is a contrast. On five completely unseen languages (Kalamang, Dinka, Wolof, Guarani, Kachin), the RL model on Qwen3-4B-Base averages 0.3335 chrF; supervised fine-tuning on the same data averages 0.2300, below the base model’s 0.2255. RL teaches the meta-skill of reading context; SFT memorizes the training languages and loses generalization. The pattern holds on Llama-3.2-3B-Instruct too (RL 0.3020 vs SFT 0.2125 on unseen).

The catch is in the other column. On seen Romansh varieties the ranking flips: SFT 0.6017, RL 0.5160. So this is not “RL beats SFT.” It is a generalization-versus-memorization trade, and which you want depends on whether your target language was in training.

The problem: memorizing languages does not scale

There are thousands of low-resource languages with almost no parallel data. The two standard fixes overfit. Continued pretraining on a target language teaches that language but transfers poorly to the next. Stuffing a grammar book into context (the MTOB-style approach) works for the language whose book you have, but the model treats it as a one-off rather than learning to read any grammar book. The authors’ framing: the model needs a meta-skill, extracting and applying in-context linguistic knowledge, not a bigger pile of memorized languages.

The context packet has five parts: a short linguistic introduction, two dictionary entries per source token retrieved by longest-common-subsequence matching, three to five parallel example sentences, two grammar-book passages averaging about 2.8k tokens, and a closing instruction asking for step-by-step metalinguistic reasoning. Training spans 14 languages: six Romansh varieties plus eight languages that ship with grammar books. Evaluation splits into seen Romansh, held-out same-family Romansh, and five unrelated unseen languages.

Why RL elicits contextual use when SFT does not

The mechanism is about what each objective rewards. SFT maximizes the likelihood of the gold translation. When your training set is dominated by Romansh, the shortest path is to memorize Romansh-to-German mappings, and a memorized mapping needs no dictionary or grammar from the prompt. So SFT scores high on seen languages while quietly learning to ignore the context. Point it at Kachin and it has nothing to fall back on: 0.0128 chrF, near zero.

A chrF-reward RL objective rewards a correct output however the model gets there. With a policy that already half-reads context, rollouts that consult the dictionary and grammar score higher, so the gradient pushes toward consulting them. The reward is deliberately thin, just r = chrF / 100 rescaled to [0,1], optimized with GRPO. The notable finding: a surface-level n-gram metric is enough signal to make the model learn a fairly abstract skill, evidence that outcome-based RL extends past math and code into language learning from context. This is the same skill-from-context direction explored for general tasks in Ctx2Skill, here pinned to translation with a hard external reward instead of self-play.

Key results

• Unseen languages (Qwen3-4B-Base, full context): RL 0.3335 chrF average vs SFT 0.2300 and base 0.2255. Romansh-trained SFT ends up worse than the untouched base on unrelated languages.
• The per-language SFT collapse is stark: Dinka RL 0.2291 vs SFT 0.0506; Wolof 0.2253 vs 0.0484; Guarani 0.2679 vs 0.0639; Kachin 0.2715 vs 0.0128; Kalamang 0.3464 vs 0.2860.
• Seen Romansh (German target), the trade reverses: SFT 0.6017 vs RL 0.5160 vs base 0.3413. Memorization wins where the target was in training.
• Held-out similar Romansh varieties: SFT 0.5464 vs RL 0.4785, a narrower edge than on seen languages.
• Llama-3.2-3B-Instruct, same pattern: unseen RL 0.3020 vs SFT 0.2125; seen SFT 0.5577 vs RL 0.4765. Not a Qwen artifact.
• Context carries unseen languages: RL scores 0.1815 with task instruction only vs 0.3335 with full context.

Which context piece does the work

The ablation has a surprise. The dictionary is the load-bearing component: removing it costs about 8.4 chrF on seen languages and drops Kalamang to 0.4483. Removing the grammar book costs only about 0.75 chrF on seen languages. The expensive 2.8k-token grammar passages, the thing the whole grammar-book-translation framing is built around, add almost nothing on top of a good dictionary and a few examples. Removing parallel sentences barely dents seen-language scores but costs 7.3 chrF on Kalamang, so examples matter most for genuinely unfamiliar languages.

The second finding is timing. Test-time context availability dominates training-time context. A model trained with RL that never saw retrieval during training can still exploit a full context at inference (about 0.28 chrF on English-to-Kalamang), while a model trained with full context but tested without it falls to 0.17. The learned skill is closer to “use whatever context you are given at inference” than “expect a specific training distribution.”

Limits and open questions

The unseen-language scores are low in absolute terms. A 0.33 chrF is a weak translation, not a usable one, so this is a result about relative learning behavior, not about solving low-resource MT. The grammar-book ablation cuts against the paper’s own framing: if removing the grammar costs under 1 chrF, it is unclear the model reads grammatical descriptions at all rather than leaning on the dictionary and examples, and the authors flag this. chrF is a surface metric, so a model that reproduces dictionary tokens in roughly the right order can score without real comprehension, and there is no COMET or human evaluation. Evidence covers two small models (4B and 3B) and a fixed retrieval pipeline, with no larger models and no test of whether the meta-skill survives a different context format.

FAQ

Does this RL method actually beat SFT for translation?

Only on languages the model never trained on. On the five unseen languages, RL averages 0.3335 chrF against SFT’s 0.2300, and SFT falls below the base model. On seen Romansh varieties SFT wins clearly, 0.6017 vs 0.5160. It is a generalization-versus-memorization result: RL learns to read context and transfers, SFT memorizes training languages and scores higher there while losing unseen ones.

Why does the RL-for-translation model learn to use the grammar book when SFT ignores it?

SFT maximizes gold-translation likelihood, and on a Romansh-heavy set the cheapest route is to memorize Romansh mappings, which need no context. A chrF-reward RL objective scores correct outputs regardless of route, so rollouts that consult the dictionary and examples earn more reward and the gradient reinforces them. The model is pushed toward the meta-skill, not a lookup table.

Is the grammar book the part doing the work?

No, and that is the surprising bit. Removing the grammar book costs only about 0.75 chrF on seen languages, while removing the dictionary costs about 8.4 and removing parallel sentences costs 7.3 on Kalamang. The dictionary and examples carry the result; the expensive grammar passages add little, which complicates the grammar-book-translation framing. The method shares Ctx2Skill’s skill-from-context premise but uses a hard chrF reward and GRPO instead of self-play.

One line: an RL objective with a cheap chrF reward teaches an LLM to translate unseen languages by reading the context instead of memorizing languages, 0.3335 vs 0.2300 chrF over SFT on five unseen languages, though the dictionary, not the grammar book, does most of the work. Read the original paper on arXiv.