Language Models · Efficient AI
Chinchilla vs GPT-3: The Paper That Ended the 'Just Scale Parameters' Era
GPT-3 bet 175B parameters on roughly 300B tokens. Chinchilla re-ran the scaling math, showed data and parameters deserve equal scaling, and beat GPT-3 with a 70B model fed about 1.4T tokens under the same compute.
Two opposite bets, two years apart
Both papers come from the same scaling-laws tradition, and they reach opposite conclusions about how to spend a training budget. GPT-3 (2020) is the pure parameters bet: keep the GPT-2 architecture, scale it to 175 billion parameters (10x larger than any previous non-sparse language model), train on roughly 300 billion tokens, and let capability emerge from size. The 2020 Kaplan et al. scaling laws behind that bet said compute should pour mostly into parameters, with data growing only modestly.
Chinchilla (2022) is the correction. DeepMind re-ran the experiment at a granularity nobody had bothered with: over 400 models from 70M to 16B+ parameters, trained on 5B to 500B tokens, with the loss surface fit by three independent methods. All three methods agreed with each other and disagreed with the 2020 advice: parameters and tokens carry roughly equal exponents, so they should grow in lockstep. By that math, a 530B model trained compute-optimally would need on the order of 10 trillion tokens, far more than anyone was actually feeding these giants.
Key numbers
| Measurement | GPT-3 | Chinchilla | Setting and source | Same harness? |
|---|---|---|---|---|
| Parameters | 175B | 70B | Both papers | n/a |
| Training tokens | roughly 300B (per Chinchilla’s accounting) | ~1.4T, about 4x Gopher’s data | Chinchilla paper | Different corpora |
| Tokens per parameter | ~1.7 | ~20 | Arithmetic on the rows above | n/a |
| Compute relationship | n/a | Same FLOPs as the 280B Gopher | Chinchilla paper | Within DeepMind only |
| MMLU average | not reported on the GPT-3 page | 67.5%, a 7+ point jump over Gopher | Chinchilla paper | No direct GPT-3 MMLU number on-site |
| Downstream suite | uniformly below Chinchilla in that suite | uniformly above Gopher 280B, GPT-3 175B, Jurassic-1 178B, MT-NLG 530B | Chinchilla paper’s own suite | Yes, one suite, five models |
| Size gap to GPT-3 | baseline | 2.5x smaller (7.5x smaller than MT-NLG 530B) | Chinchilla paper | n/a |
| Scaling study range | down to 1.3B in comparisons | 400 models, 70M to 16B+, 5B to 500B tokens | Both papers | n/a |
| Headline capability | few-shot in-context learning: LAMBADA ~86%, TriviaQA ~71%, no gradient updates | training economics: a corrected budget split, no architecture change | Respective papers | Different questions |
Read the table for what it does not contain. There is no head-to-head MMLU number for GPT-3 on these pages (GPT-3 predates that eval habit), and the “uniformly beats” row comes from the Chinchilla paper evaluating all five models under its own downstream suite, not from a rerun of GPT-3 under identical conditions. The two papers also answer different questions, which is why the last row matters: GPT-3’s contribution is about what scale buys at inference time, Chinchilla’s is about how to spend a fixed budget at training time.
What GPT-3’s scale actually bought
The GPT-3 paper’s claim was never a single leaderboard number. It is that scale turns the prompt into a task interface: put the instruction and a few worked examples into the context, leave the weights frozen, and the model performs. The paper tests zero-shot, one-shot, and few-shot (typically 10 to 100 examples) on every task, and the real finding is the interaction: small models barely benefit from in-context examples, GPT-3 benefits a lot. That is the qualitative jump 175B bought.
The concrete few-shot numbers on the GPT-3 page: roughly 86% on the LAMBADA cloze task, well above prior zero-shot state of the art; around 71% on open-domain TriviaQA, competitive with fine-tuned systems that saw far more supervision; and synthetic on-the-fly tasks like letter unscrambling, using a newly defined word in a sentence, and 3-digit arithmetic, which earlier language models could not do from a prompt at all. Human evaluators performed near chance at telling GPT-3-generated short news articles from human-written ones. The lasting legacy is the interface shift: task adaptation moved out of the optimizer and into the context window, which is the template every later prompt-engineering and instruction-tuning line of work follows.
PaLM: the parameters-first bet at its purest
Same year as Chinchilla, opposite answer. PaLM is a 540-billion-parameter dense decoder-only Transformer that Google trained on 6,144 TPU v4 chips across two Pods using the Pathways system, deliberately not a mixture of experts, an argument that careful systems work could keep dense parameter scaling alive. The PaLM page does not state its training-token count, so the honest comparison is architectural: PaLM pushed the parameters-first recipe to its extreme the same year Chinchilla proved that recipe was misallocating its budget.
And the bet still bought real capability, per the PaLM page. With chain-of-thought prompting, PaLM 540B matched or beat fine-tuned state of the art on a suite of multi-step reasoning tasks using few-shot prompts alone, and it outperformed the average human score on the aggregate BIG-bench metric. Its BIG-bench curves improved discontinuously at the largest scale, the sharpest evidence for the “emergence” claim, which the paper itself leaves observed rather than explained. The same page is candid about the costs: a dense 540B model is expensive to train and to serve, larger models memorize more training data with privacy and copyright implications, and “beats average human” is an aggregate across many tasks, not a win on the hard ones. PaLM is the strongest case for GPT-3’s philosophy; Chinchilla is the strongest case against its budget split. Holding both explains the era.
What Chinchilla corrected
Chinchilla’s argument is that GPT-3 and its generation were not too small; they were badly underfed. Gopher, GPT-3, and MT-NLG were all trained on roughly 300 billion tokens regardless of whether they had 175B or 530B parameters. Chinchilla is the existence proof that reallocation beats size: 70B parameters, about 1.4 trillion tokens, the same compute budget as the 280B Gopher. No architecture trick, no new objective, just a corrected budget split.
The results, from the Chinchilla paper: 67.5% average on MMLU, state of the art at the time and a 7+ point improvement over Gopher; uniform wins over Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) across a large downstream suite, despite being 2.5x to 7.5x smaller than those models. And because the model is smaller, it costs substantially less to fine-tune and to serve, and the efficiency win compounds in production, not just during training.
Reading the head-to-head honestly
Three caveats before treating the table as a verdict. First, the downstream-suite row is the Chinchilla paper’s own evaluation; GPT-3’s page reports neither that suite nor MMLU, so “uniformly beats GPT-3” is one lab’s measurement, not a neutral rematch. Second, the token counts come from different corpora — GPT-3 trained on Common Crawl plus higher-quality sources, Chinchilla on DeepMind’s data, so the comparison of budget splits is cleaner than the comparison of data quality. Third, the contributions are complementary, not competitive: GPT-3 proved what a huge frozen model can do at inference time, Chinchilla proved how a fixed training budget should be split. A modern system wants both lessons: a well-fed model and a prompt that carries the task.
When to use which
- You are training a new base model and training compute is the constraint: start from the Chinchilla rule of thumb: double the parameters, double the tokens; roughly 20 tokens per parameter as a compute-optimal anchor. GPT-3’s 1.7-tokens-per-parameter split is now a documented mistake, not a style choice.
- The model will be served billions of times and inference cost dominates: the Chinchilla page itself carries the counterargument. Compute-optimal minimizes the cost of one training run; if the checkpoint will be run far more often than it was trained, deliberately overtrain a smaller model past its compute-optimal point, the LLaMA argument, to cut inference cost.
- High-quality text is the scarce resource: “just double the data” stops being free advice once the token budget runs into data-quality limits, which is exactly where the field has been pushing ever since.
- You are designing how users interact with the model: GPT-3’s lesson is untouched by any scaling-law correction. Zero/one/few-shot prompting, with its sensitivity to wording and example order, is the interface behavior this paper established, and it is orthogonal to how many tokens you trained on.
Limits and open questions
Scaling laws are empirical curve fits, not physics: the exponents move with data quality, tokenizer, architecture, and optimizer schedule, and the original Chinchilla coefficients have been re-derived, disputed, and partly adjusted in an erratum. Both papers are also pretraining-economics results that say nothing directly about instruction tuning, alignment, or tool use. GPT-3’s own page notes that later work exists precisely to patch the gaps it exposed. And neither paper runs the clean controlled experiment this comparison would want: GPT-3 retrained at 20 tokens per parameter under the same data and eval, to separate the budget split from everything else that differed between 2020 OpenAI and 2022 DeepMind.
FAQ
Is Chinchilla bigger than GPT-3?
No. Chinchilla has 70 billion parameters against GPT-3’s 175 billion, about 2.5x smaller. The difference is the data: Chinchilla saw roughly 1.4 trillion tokens, about 4 to 5 times GPT-3’s roughly 300 billion, which is why it could beat GPT-3 on the Chinchilla paper’s downstream suite despite being the smaller model.
Did Chinchilla actually beat GPT-3?
In the Chinchilla paper’s own downstream evaluation, yes: Chinchilla uniformly outperformed GPT-3 175B, Gopher 280B, Jurassic-1 178B, and MT-NLG 530B, and scored 67.5% on MMLU. There is no head-to-head MMLU number for GPT-3 on these pages, and the suite was Chinchilla’s, so read it as one lab’s controlled comparison rather than a neutral rematch.
How many tokens was GPT-3 trained on?
Roughly 300 billion, according to the accounting on the Chinchilla page, about 1.7 tokens per parameter, which is exactly the undertraining the Chinchilla paper attacked. GPT-3’s own page describes the corpus (Common Crawl plus higher-quality sources) without giving an exact token count.
What is the Chinchilla rule of thumb?
For compute-optimal training, scale model size and training tokens equally: every doubling of parameters calls for doubling the data. In practice that lands near 20 tokens per parameter, Chinchilla’s own ratio of 1.4T tokens to 70B parameters.
Should I train GPT-3 style, with a huge model and modest data?
Only in one regime: when the model will be served so many times that inference cost dominates, and you deliberately overtrain a smaller model past its compute-optimal point to make each query cheaper. For a single training run with a fixed budget, the Chinchilla split is the corrected default.