Fine-Tuning & Adaptation · Efficient AI · Language Models
LoRA vs Full Fine-tuning: Same Accuracy, Different Weights
LoRA matches full fine-tuning within a point on standard tasks at 10,000x fewer trainable parameters, but it gets there through intruder dimensions that carry its forgetting and hurt under continual training.
What each method changes in the weights
Full fine-tuning updates every parameter. For a 175B model that means a second 175B copy per task, plus optimizer state, which under Adam is two more tensors per weight. LoRA freezes the pretrained matrix W and trains a low-rank correction B·A, with rank r typically between 1 and 64. At serving time B·A is added back into W, so the merged model has the original shape and no extra latency, unlike adapter layers that insert modules into the forward pass.
The 2021 LoRA paper measured the cost side of this trade: up to 10,000x fewer trainable parameters and 3x less GPU memory than fully fine-tuning GPT-3 175B with Adam, with accuracy on par or better across RoBERTa, DeBERTa, GPT-2 and GPT-3. The 2024 MIT study LoRA vs Full Fine-tuning: An Illusion of Equivalence measured the weight side and found the two methods do not learn the same thing. LoRA-trained matrices contain intruder dimensions: new singular vectors with large singular values that are nearly orthogonal to every pretrained singular vector. Fully fine-tuned matrices keep their top singular vectors close to where pretraining left them. Same accuracy, different solution.
Key numbers
| Measurement | Full fine-tuning | LoRA | Setting and source | Same harness? |
|---|---|---|---|---|
| Trainable parameters, GPT-3 175B | 175B | up to 10,000x fewer | LoRA paper, Adam | Yes |
| GPU memory during training | 1x | about 1/3 | GPT-3 175B, LoRA paper | Yes |
| Added inference latency | none | none after merging B·A into W | LoRA paper | Yes |
| Per-task artifact | hundreds of GB | megabytes | LoRA paper | Yes |
| RoBERTa-base MNLI accuracy | 0.8745 | r=8 0.8704, r=16 0.8739 | Illusion paper, Table 2, alpha = 2r | Yes |
| RoBERTa-base QQP accuracy | 0.9152 | r=8 0.9093 | Same | Yes |
| Intruder dimensions among top singular vectors | none | present; fewer as rank rises | Illusion paper, RoBERTa and LLaMA2-7B | Yes |
| Forgetting at matched task accuracy | more | less, in the default settings | Illusion paper, Figure 6 | Yes |
| Forgetting under aggressive alpha or learning rate | baseline | can exceed full fine-tuning | Illusion paper, Section 4 | Yes |
| Sequential fine-tuning on 6 tasks | retains earlier tasks better | accumulates intruders, retains worse | Illusion paper, Figure 9 | Yes |
| Post-hoc fix: scale down intruders, LLaMA2-7B MetaMath r=16 | n/a | forgetting down 28.2% for 2.7% accuracy | Illusion paper, Table 4 | n/a |
| Same fix, RoBERTa QQP r=8 | n/a | forgetting down 45.0% for 0.6% accuracy | Illusion paper, Table 3 | n/a |
The cost rows come from one paper and the behavior rows from another, and the behavior rows all use models under 7B parameters. Nothing in this table says how the trade looks at 70B or under quantized bases.
Where LoRA’s savings come from
The savings are real and they compound. No optimizer state for frozen weights is what cuts memory by 3x. Storing only A and B is what turns a task into a file of megabytes, and that is what makes population-scale personalization even conceivable: the position paper On the Scaling of PEFT argues for a million per-user adapters over one trillion-parameter base precisely because the adapter, not the base, is the unit that scales out. Code2LoRA goes further and generates a repository’s adapter from a hypernetwork instead of training it, reaching 66.2% in-repo exact match without spending any prompt tokens on repository context. None of these designs is possible with full fine-tuning, where each variant is a full model.
The forgetting trade, with the caveat that matters
LoRA usually forgets less, and the Illusion paper shows that holds even when task accuracy is matched, so it is not an artifact of underfitting. The reason is structural: a low-rank update cannot rewrite the pretrained spectrum, so most of the base model survives by construction. But the same paper shows the forgetting LoRA does have is concentrated in the intruder dimensions, and that their number depends on hyperparameters. Push alpha or the learning rate up and LoRA runs with identical task accuracy can forget more than full fine-tuning. The runs that generalize best have the fewest intruders. Two levers reduce them: higher rank (r=64 and above looks spectrally closer to full fine-tuning) and training only B over a fixed orthogonal A.
Continual learning flips the usual advice. Because each LoRA round adds its own intruders, six sequential tasks leave a LoRA model worse at earlier tasks than a fully fine-tuned one. If your workflow is one base model updated over and over, the full fine-tuning tax buys retention.
When to use which
- Many tasks, styles or users on one base; single GPU; adapters shipped as files: LoRA. This is the regime the method was built for and the cost table is decisive.
- One task that needs a large behavioral shift (new language, new modality head, heavy domain adaptation): full fine-tuning, or LoRA at high rank with the expectation that you are approaching full fine-tuning cost anyway.
- Repeated fine-tuning of the same model over time: full fine-tuning, or LoRA with periodic merging and a fresh adapter, to avoid stacking intruders.
- LoRA either way: use rank 16 or higher, keep alpha and learning rate moderate, and if pretraining-distribution loss matters, check for intruders and scale them down after training; the paper shows that recovers most forgetting at under a point of accuracy.
Limits and open questions
The accuracy comparisons that carry the “on par” claim are classification and short-generation tasks on models up to 7B; frontier-scale fine-tuning is uncharted in both papers. The intruder analysis uses vanilla LoRA, with variants such as DoRA only in an appendix. Forgetting is measured by pretraining loss and held-out tasks, which stand in for capability loss rather than measuring it directly. And the strongest practical result, scaling intruders down after training, is demonstrated but not turned into a procedure with a recommended factor. Anyone who reads the 2021 paper’s “matches full fine-tuning” as settled should read the 2024 paper’s “not always” next.
FAQ
Is LoRA as good as full fine-tuning?
On task accuracy, usually within a point: RoBERTa-base MNLI is 0.8745 fully fine-tuned against 0.8704 at LoRA rank 8 and 0.8739 at rank 16. On the weights, no: LoRA solutions contain intruder dimensions that full fine-tuning does not, and those affect forgetting and continual learning.
Does LoRA forget less than full fine-tuning?
In default settings yes, even at matched accuracy. But with high alpha or learning rate LoRA can forget more, and across sequential tasks it retains earlier tasks worse because intruder dimensions accumulate.
How much memory does LoRA save compared with full fine-tuning?
The LoRA paper reports 3x less GPU memory and up to 10,000x fewer trainable parameters when fine-tuning GPT-3 175B with Adam, because frozen weights need no optimizer state and only the small A and B matrices are stored per task.
What rank should you use if you choose LoRA over full fine-tuning?
The intruder-dimension study finds fewer intruders and behavior closer to full fine-tuning at rank 64 and above, while ranks 1 to 8 show the most. A practical floor is 16, with moderate alpha, and a post-training check that scales down remaining intruders if forgetting matters.