Small Language Models · Language Models · Efficient AI
DistilBERT: A Smaller, Faster BERT by Distillation
DistilBERT distills BERT during pretraining into a 6-layer model with 40% fewer parameters that runs about 60% faster and keeps roughly 97% of BERT-base's GLUE score.
Quick answer
DistilBERT compresses BERT during pretraining, not after fine-tuning, into a 6-layer model with 40% fewer parameters (66M vs 110M) that runs about 60% faster while keeping roughly 97% of BERT-base’s GLUE score. The contribution is showing that a general-purpose distilled model, trained once on the same corpus as the teacher, transfers to downstream tasks almost as well as the full model. You distill once and reuse the checkpoint everywhere.
The triple loss that does the distilling
DistilBERT trains the small student with three objectives at once. The distillation loss matches the teacher’s full soft-label distribution, using a temperature so the student also learns the teacher’s signal about which wrong answers are plausible. The standard masked-language-modeling loss keeps it grounded in real text. And a cosine embedding loss aligns the direction of the student’s hidden states with the teacher’s. Combining soft targets with MLM is what lets a half-depth model stay close to BERT, rather than distilling only on task labels after fine-tuning.
What was cut from BERT
The student keeps BERT’s hidden size and attention design (it inherits the Transformer encoder) but halves the number of layers from 12 to 6, and removes the token-type embeddings and the pooler. Crucially, the 6 student layers are initialized from every other layer of the trained teacher, so distillation starts from a strong point rather than random weights. Training uses the same data as BERT (English Wikipedia and the Toronto Book Corpus), large batches, and dynamic masking, and drops next-sentence prediction.
Key results
- 40% smaller: 66M parameters versus BERT-base’s 110M.
- About 60% faster inference on CPU, and roughly 71% faster on-device (measured on an iPhone), with the model around 207MB.
- About 97% of BERT-base’s GLUE score retained on average across the nine GLUE tasks.
- Distilled once at pretraining: the same DistilBERT checkpoint fine-tunes across tasks, unlike task-specific distillation that must be redone per task.
What “retains 97% of BERT” hides
The 97% is an average over GLUE; individual tasks vary, and the gap widens on the harder reasoning-heavy datasets. It is also retention against BERT-base, not against stronger teachers like RoBERTa or larger models, so “97% of the teacher” is only as strong as the teacher you started from. And the speedup is inference-side: you still pay BERT-scale cost once to train the teacher you distill from.
Limits and open questions
DistilBERT needs an already-trained teacher, so it does not remove the cost of pretraining BERT, only amortizes it. Later work pushed compression much further (TinyBERT, MobileBERT) using layer-wise and attention distillation, so DistilBERT is the readable baseline rather than the frontier of model compression. The original release is English-only, and the accuracy gap means quality-critical, latency-insensitive applications should still reach for the full model.
Should you use DistilBERT today (builder judgment)
For latency- or cost-sensitive NLP where a few points of accuracy are an acceptable trade, DistilBERT is still a sensible default and one of the easiest compressed encoders to deploy. If you need the best possible accuracy, use BERT-large or RoBERTa; if you need a much tighter memory or latency budget, evaluate the more aggressive distillation methods that followed. Its lasting value is methodological: it made general-purpose, pretraining-time distillation a standard tool.
FAQ
How does DistilBERT’s triple loss work?
It combines three objectives: a distillation loss matching the teacher’s temperature-softened logits, the standard masked-language-modeling loss on real text, and a cosine embedding loss aligning student and teacher hidden-state directions. The soft targets carry more information than hard labels, which is what keeps a 6-layer model close to BERT.
Does “retains 97% of BERT” mean 97% on every task?
No. The 97% is an average across the nine GLUE tasks; harder tasks show a larger drop, and the comparison is against BERT-base, not stronger teachers. Treat it as an aggregate, not a per-task guarantee.
What did DistilBERT remove from BERT, and how big is it?
It halves the layers (12 to 6), removes token-type embeddings and the pooler, and keeps the hidden size, giving 66M parameters versus 110M. The student is initialized from every other teacher layer before distillation.
DistilBERT vs TinyBERT or MobileBERT for compression?
DistilBERT is the simplest, most readable baseline and distills at the output and embedding level. TinyBERT and MobileBERT compress further with layer-wise and attention distillation, so pick them when you need a tighter budget than DistilBERT’s 40% reduction.
Read the original paper on arXiv.