Efficient AI · Language Models · Text-to-Image
Efficient Pretraining, Three Ways: HRM-Text vs Lens vs Chinchilla
HRM-Text bets on architecture plus objective, Lens on data density, Chinchilla on budget reallocation. Same goal of cutting training cost, three levers, three failure modes.
The question all three papers answer
Training cost is the tax every model pays before it is useful. The three papers in this comparison attack that tax with three different levers, and it is worth separating them because they are often blurred into one “efficiency” headline:
- HRM-Text changes the architecture and the objective, training a 1B hierarchical recurrent model only on instruction-response pairs with loss on response tokens.
- Lens changes the data, pairing 800M images with roughly 109-word GPT-4.1 captions so each gradient step carries more signal.
- Chinchilla changes nothing about the model or the data; it changes the budget split, reallocating the same FLOPs from a bigger undertrained model to a smaller, fully trained one.
Each is a bet that one lever dominates the others. None of the three is controlled against the others, so the honest comparison is about what each lever claims, what evidence backs it, and what it quietly assumes.
Key numbers
| Measurement | HRM-Text | Lens | Chinchilla | Same harness? |
|---|---|---|---|---|
| Parameters | 1B | 3.8B (diffusion backbone) | 70B | No, different tasks |
| Training data | 40B unique tokens, instruction-response pairs only | 800M image-text pairs (Lens-800M) | ~1.4T tokens | No |
| Reported training cost | ~$1,500, 46 hours on 16 H100s | ~192K A100 GPU-hours | Same FLOPs as 280B Gopher | No |
| Headline efficiency claim | 100-900x fewer tokens, 96-432x less compute than 2-7B open baselines | ~19.3% of Z-Image’s ~314K H800 GPU-hours | 4x smaller model, 4x more data, same budget | Each vs its own baseline set |
| Main quality scores | MMLU 60.7, ARC-C 81.9, DROP 82.2, GSM8K 84.5, MATH 56.2 | GenEval 0.525, OneIG 0.557, LongText 0.930 | 67.5% average on MMLU, beating Gopher 280B, GPT-3 175B, MT-NLG 530B | No |
| Inference cost | Not the focus | 3.15s per 1024px image on one H100 (0.84s, 4-step Turbo) | “Substantially less to fine-tune and serve” than Gopher | No |
| Baselines beaten | Llama 3.2 3B, Gemma 3 4B, OLMo 3 7B, Huginn, Ouro | 6B+ T2I models in the same score range | Gopher, GPT-3, Jurassic-1, MT-NLG | No |
Read the table for its empty cells as much as its filled ones. There is no setting where HRM-Text, Lens, and Chinchilla-trained models sit side by side under one evaluation, because they solve different problems. The comparison that matters is structural: each paper identifies a different “wasted spend” in standard training and removes exactly that.
Lever one: architecture plus objective (HRM-Text)
HRM-Text argues that the dominant waste in pretraining is the objective, not the model class. Next-token prediction on raw web text spends most gradients predicting connective tissue, and standard Transformers need enormous depth to build effective computation. Its fix is twofold: a hierarchical recurrent model where a slow strategic layer updates infrequently and a fast execution layer iterates many steps per strategic update, giving a 1B model the effective depth of something much larger; and a task-completion objective over instruction-response pairs only, with PrefixLM masking so the instruction is read bidirectionally and loss is computed only on answer tokens.
The numbers are striking if they hold: 60.7% MMLU and 84.5% GSM8K from 40B tokens and about $1,500 of GPU time. But notice what the bet requires. You must accept recurrent training instability (the paper’s MagicNorm and warmup credit assignment exist precisely because deep recurrence “blows up” otherwise), you must source or build a large corpus of high-quality instruction pairs, and you must accept that the benchmark suite (MMLU, GSM8K, MATH, ARC, DROP) is exactly the suite where this design should shine. Nothing in the paper establishes parity on open-ended generation, long-form coherence, or broad incidental world knowledge absorbed from raw text.
Lever two: data density (Lens)
Lens makes the opposite claim: the architecture is fine, the data is starving it. Short tag captions mean a text-to-image model must see millions of images before it learns one concept, so Lens pays GPT-4.1 once to write ~109-word captions covering layout, attributes, text content, and style for 800M images, and the resulting 3.8B model reaches GenEval 0.525 and OneIG 0.557 while consuming roughly 19.3% of the training compute reported for Z-Image.
The caption-length ablation is what separates this from a slogan: “denser data beats more parameters” is a claim that has been made before and is hard to attribute, and Lens at least shows the lever moving. The catches are equally concrete. The 19.3% figure compares A100-hours against H800-hours and only against Z-Image, so it is an illustrative ratio. The GPT-4.1 captioning pass is real compute and real money that sits outside the headline number, and it depends on access to a GPT-4.1-grade captioner. And 0.525 on GenEval is competitive, not state of the art: the honest reading of “matches 6B+ models” is “in the same range,” which the authors themselves implicitly accept by framing the result as parity at a fraction of cost.
Lever three: budget reallocation (Chinchilla)
Chinchilla is the baseline the other two are implicitly arguing against. It changes nothing about architectures or data pipelines. By fitting loss surfaces over 400 models from 70M to 16B+ parameters on 5B to 500B tokens, DeepMind showed that parameters and tokens carry roughly equal exponents, so for a fixed compute budget you should scale them together. The 2020-era practice of pouring almost everything into parameters had left giants like Gopher undertrained. Chinchilla is simply Gopher’s budget reallocated: 70B parameters, about 1.4T tokens, the same FLOPs, and a 7+ point jump to 67.5% average on MMLU.
The reason this remains the default is that it makes the fewest assumptions. It does not need a captioner, a curated instruction corpus, or a training-stability research program. Its known weaknesses are also well charted: the laws optimize training loss for one run, so high-volume serving deliberately overtrains past the Chinchilla point (the LLaMA argument), and the “just double the data” advice stops being free when high-quality tokens run scarce.
When to use which
- You want the safest efficiency win with no new components: reallocate the budget Chinchilla-style. Smaller model, more tokens, equal FLOPs. It is the only lever of the three with a closed-form rule and five years of replication behind it.
- You already generate or can afford captions at GPT-4.1 quality once: Lens-style data density. The economics are “pay the captioner once instead of paying GPU-hours forever,” which compounds across every future training run on that dataset.
- You control the full training stack and can tolerate a narrow evidence base: HRM-Text-style architecture plus objective. The payoff is the largest (orders of magnitude on paper) and the risk is the largest (stability engineering, benchmark-shaped evidence, no raw-text fallback).
Limits and open questions
The three levers are not mutually exclusive, and nobody has run the experiment that stacks them: a small recurrent or dense-caption-trained model at a Chinchilla-optimal budget, evaluated on broad open-ended tasks rather than a reasoning-heavy suite. Until that exists, treat each paper as an existence proof for its own lever, not as a ranking of levers. The one shared lesson is quieter but more reliable: every paper found its win by identifying where standard training spends compute that produces no signal, and deleting exactly that.
FAQ
Which is cheapest per unit of quality, HRM-Text, Lens, or Chinchilla?
They are not measured in the same units. HRM-Text reports dollars and benchmark accuracy for a language model; Lens reports GPU-hours and GenEval for an image model; Chinchilla reports FLOPs and MMLU for a 70B model. Any single “cheapest” ranking across these would be fabrication. What you can say: HRM-Text has the lowest absolute cost ($1,500), Lens has the largest relative-cost claim (19.3% of a rival), and Chinchilla has the most replicated claim.
Is the HRM-Text result reproducible by a university lab?
The training run itself is, in principle: 46 hours on 16 H100s is a cluster-week. The harder dependency is the 40B-token corpus of instruction-response pairs of sufficient quality, plus the engineering to keep deep recurrent training stable. The paper’s MagicNorm and warmup credit assignment are presented precisely because naive training fails.
Does Lens beat bigger text-to-image models?
Not on the leaderboard numbers. Its GenEval 0.525 and OneIG 0.557 put it in the same range as 6B+ models, which is the point: near-parity quality at a fraction of the training compute. If absolute top score is the goal, this is not the lever.
Is Chinchilla still the right recipe in 2026?
As a default, yes, with two amendments documented since: models served at very high volume are deliberately overtrained past the compute-optimal point to cut inference cost, and data scarcity makes the “double the data” half of the rule harder to satisfy than it was in 2022.