BERT vs DistilBERT vs MobileBERT: How Small Can an Encoder Get?

The question all three papers answer

BERT changed NLP by making one pretrained bidirectional encoder the default starting point for understanding tasks. It also left a standing problem: the model that set state of the art on 11 tasks at once is too heavy to run everywhere people want NLP. The two papers that followed, DistilBERT and MobileBERT, attack that problem with opposite instincts. DistilBERT cuts depth: half the layers, same width, distilled once at pretraining. MobileBERT cuts width: a thin model with bottleneck structures, squeezed out of a large teacher built for the purpose. Reading them together gives you the full menu of BERT compression (output-level distillation, depth removal, bottleneck design, teacher engineering) in three papers spanning 2018 to 2020.

Key numbers

Measurement	BERT	DistilBERT	MobileBERT	Comparison basis
Parameters	110M (BERT-base)	66M, 40% fewer, ~207MB	4.3x smaller than BERT_BASE	Params for BERT/DistilBERT stated in papers; MobileBERT as ratio vs BERT_BASE
Depth/width change	12 layers, full hidden size	6 layers, hidden size unchanged	thin version of BERT_LARGE with bottleneck structures	Different axes: depth vs width
Inference speed	baseline	~60% faster on CPU, ~71% faster on iPhone	5.5x faster than BERT_BASE, 62 ms latency on a Pixel 4	DistilBERT vs BERT-base; MobileBERT vs BERT_BASE
GLUE	BERT-large 80.5 (+7.7 abs over prior best)	~97% of BERT-base GLUE retained, averaged over 9 tasks	77.7, 0.6 below BERT_BASE in the same setup	See the harness column; not one common leaderboard
SQuAD v1.1	test F1 93.2 (BERT-large)	not the headline metric	dev F1 90.0, +1.5 above BERT_BASE	test vs split, large vs base; do not compare directly
SQuAD v2.0	test F1 83.1 (BERT-large)	not the headline metric	dev F1 79.2, +2.1 above BERT_BASE	test vs split, large vs base; do not compare directly
Training recipe	masked LM (15% masking) + next-sentence prediction, BooksCorpus + English Wikipedia	triple loss: soft-target distillation + MLM + cosine embedding alignment, same corpus, NSP dropped	distilled from a specially designed inverted-bottleneck BERT_LARGE teacher (IB-BERT), task-agnostic	All three pretrained once, then fine-tuned per task

Read the table for what it does not contain. The GLUE column mixes three different setups: BERT’s 80.5 is BERT-large on the original GLUE benchmark; DistilBERT’s 97% is a retention ratio against BERT-base, not an absolute score; MobileBERT’s 77.7 is an absolute number measured against its own BERT_BASE arm, 0.6 points lower. The defensible cross-model statement is narrow: DistilBERT loses only about 3% of BERT-base’s GLUE while MobileBERT loses 0.6 absolute points against the same-size teacher family; both land within roughly a point of each other in accuracy, at very different sizes. The SQuAD rows are even more careful territory: BERT’s 93.2/83.1 are test-set F1 for BERT-large, while MobileBERT’s 90.0/79.2 are dev-set F1 for a model 4.3x smaller than BERT-base. Same task, different split, different teacher size. Treat them as evidence that compression did not cost SQuAD accuracy, not as a ranking.

What each compression is actually doing

BERT itself is the uncompressed reference point: a 12-layer bidirectional Transformer encoder pretrained with masked language modeling and next-sentence prediction on BooksCorpus plus English Wikipedia. Its contribution is the recipe, not the architecture (pretrain once on unlabeled text, fine-tune with one output layer per task), and its 2018 results (GLUE 80.5, +7.7 absolute; MultiNLI 86.7; SQuAD v1.1 93.2 test F1) are the baseline both compressed models define themselves against.

DistilBERT compresses by removing depth. The student has 6 layers instead of 12, initialized from every other layer of the trained teacher, and keeps the teacher’s hidden size and attention design. On top of the depth cut it removes the token-type embeddings and the pooler, and drops next-sentence prediction entirely. Training is where the paper earns its keep: a triple loss that combines temperature-softened soft targets from the teacher, the standard MLM loss on real text, and a cosine embedding loss that aligns the direction of student and teacher hidden states. Soft targets matter because they carry the teacher’s opinion about which wrong answers are plausible, which is most of what a 6-layer model needs to stay close. The result is 66M parameters (40% fewer), roughly 60% faster CPU inference and 71% faster on-device, at about 97% of BERT-base’s GLUE score on average across nine tasks. Crucially it is distilled once at pretraining: one general checkpoint, fine-tuned per task like BERT, with no per-task distillation to redo.

MobileBERT compresses by removing width, and it brings a teacher engineered for the job. The student is a thin version of BERT_LARGE, not of BERT-base, built from bottleneck structures with a carefully designed balance between self-attention and feed-forward networks, so the model stays deep enough to represent but narrow enough to ship. Because a plain BERT_LARGE is a mismatched teacher for a thin student, the authors first train an intermediate teacher, an inverted-bottleneck incorporated BERT_LARGE (IB-BERT), and then transfer knowledge from that teacher into MobileBERT. The payoff: 4.3x smaller and 5.5x faster than BERT_BASE, with 62 ms latency on a Pixel 4 phone, a GLUE score of 77.7 that is only 0.6 below its own BERT_BASE arm, and SQuAD v1.1/v2.0 dev F1 of 90.0/79.2, which is 1.5 and 2.1 points above that same BERT_BASE arm. Like DistilBERT it is task-agnostic: one compressed checkpoint, fine-tuned anywhere.

The honest accuracy picture

“97% of BERT” and “0.6 points below BERT_BASE” sound like the same kind of claim but they measure different things. DistilBERT’s number is a ratio averaged over nine GLUE tasks, which hides per-task variance; the gap widens on harder reasoning-heavy tasks. MobileBERT’s number is an absolute delta against its own re-run of BERT_BASE under one setup, which is a cleaner same-harness comparison but only tells you about that setup. Neither number compares against stronger teachers like RoBERTa or later encoders, so “keeps almost all of BERT” is only as good as BERT itself. What both papers genuinely establish is that encoder accuracy tolerates aggressive compression far better than 2018 intuition suggested: cutting depth in half, or width by 4x, costs around a point of GLUE, and on SQuAD the compressed models can even come out ahead of their own base-size arm.

When to use which

• Accuracy is the constraint, latency is not: stay with a full-size encoder. BERT itself, or in practice a modern descendant, because a few points of GLUE and SQuAD headroom still matter for quality-critical ranking, retrieval, or extraction.
• You want the simplest drop-in speedup on server or desktop: DistilBERT. It keeps BERT’s hidden size and usage pattern, drops into the same fine-tuning flow, and the 40% size cut plus ~60% CPU speedup costs only about 3% of GLUE on average. It is the readable baseline the whole compression literature builds on.
• The target is a phone or an edge device: MobileBERT. The 4.3x size reduction and 5.5x speedup with a 0.6-point GLUE delta is the phone-class trade; its 62 ms Pixel 4 latency is measured, not extrapolated. The price is a more involved pipeline (you need the inverted-bottleneck teacher, not just the student), but you pay it once, at training time.
• You are choosing for a new system in 2026: treat all three as the reference points of a design space, not as the current frontier. The useful inheritance is the mechanism menu (soft-target distillation, depth removal, bottleneck width compression, teacher engineering), which is the same menu modern efficient-encoder and small-language-model work still orders from.

Limits and open questions

The comparison is honest only within its rows. No single study trains BERT, DistilBERT, and MobileBERT under one harness, so cross-model differences of a point or two are within setup noise; the table’s last column exists so you do not compare 80.5 with 77.7 as if they were a ranking. All three papers are English-only, so nothing here transfers to multilingual compression. DistilBERT’s speedups are inference-side (you still pay BERT-scale cost once to train the teacher), and its 97% is an average that thinner tasks can fall well below. MobileBERT’s headline depends on its specially trained IB-BERT teacher, which makes the recipe heavier to reproduce than the model size suggests. And the deepest caveat is temporal: BERT, DistilBERT, and MobileBERT are all classics, useful today mostly as the clearest worked example of how encoder compression trades size for accuracy.

FAQ

Which is smaller, DistilBERT or MobileBERT?

MobileBERT, by a wide margin. DistilBERT is 66M parameters, 40% smaller than BERT-base. MobileBERT is 4.3x smaller than BERT_BASE, roughly a quarter of the base model, because it thins width with bottleneck structures rather than halving depth. If raw size or phone-class latency is the constraint, MobileBERT is the tighter model; if you only need a moderate speedup with the least pipeline change, DistilBERT is simpler.

Is DistilBERT as accurate as BERT?

Close, not equal. On average across the nine GLUE tasks it retains about 97% of BERT-base’s score. That is an aggregate: harder reasoning-heavy tasks show larger drops, and the comparison is against BERT-base, not against stronger teachers. For latency-tolerant, quality-critical work, the full model still wins; for everything else the ~3% average cost is usually the right trade.

How is MobileBERT different from DistilBERT if both are distilled BERTs?

They compress along different axes with different teachers. DistilBERT halves depth: 6 layers instead of 12, same hidden size, initialized from every other teacher layer, trained with a triple loss of distillation, MLM, and cosine alignment against BERT-base. MobileBERT thins width: a bottlenecked thin version of BERT_LARGE, trained against a purpose-built inverted-bottleneck BERT_LARGE teacher (IB-BERT). Depth removal is the simpler recipe; width removal with an engineered teacher is what buys the 4.3x size cut.

What were MobileBERT’s exact results?

Against its own BERT_BASE arm: GLUE 77.7, which is 0.6 points lower; SQuAD v1.1 dev F1 90.0 (+1.5) and SQuAD v2.0 dev F1 79.2 (+2.1). The model is 4.3x smaller and 5.5x faster than BERT_BASE, with 62 ms latency measured on a Pixel 4. BERT’s own headline numbers (GLUE 80.5, SQuAD v1.1 test F1 93.2) come from BERT-large on different splits, so compare within rows, not across them.

Are these models still worth using today?

As reference points, absolutely: this three-paper sequence is the cleanest exposition of encoder compression mechanics that exists. As production defaults, only after checking modern alternatives; the same distillation and bottleneck ideas now live in more capable multilingual and long-context encoders. DistilBERT in particular remains one of the cheapest compressed encoders to deploy for English classification and ranking.