Fine-Tuning & Adaptation · Efficient AI · Language Models
LoRA vs Full Fine-tuning Explained: The Intruder Dimensions Paper
MIT shows LoRA and full fine-tuning reach similar accuracy through different weights: LoRA adds intruder dimensions that carry most of its forgetting and can be scaled down after training at almost no accuracy cost.
Quick answer
The paper asks whether LoRA and full fine-tuning learn the same solution when they reach the same accuracy, and answers no. Looking at the singular value decomposition of fine-tuned weight matrices, LoRA-trained matrices contain new singular vectors with large singular values that are nearly orthogonal to anything in the pretrained matrix. The authors call them intruder dimensions. Fully fine-tuned matrices do not have them; their top singular vectors stay close to the pretrained ones. The intruders matter because they carry the forgetting: shrinking their singular values after training cuts loss on the pretraining distribution by double-digit percentages while moving task accuracy by under a point in most settings. The paper is the reference for anyone who has read the LoRA claim that low-rank adaptation matches full fine-tuning and wants to know what that claim leaves out.
What an intruder dimension is
Take a pretrained weight matrix, fine-tune it, and compare the singular vectors before and after. Full fine-tuning on LLaMA2-7B leaves the similarity matrix close to the identity: the k-th singular vector after training points roughly where the k-th one pointed before. LoRA produces a different picture. Among the top singular vectors of the fine-tuned matrix there are new directions whose maximum cosine similarity to any pretrained singular vector is below a threshold (the paper sweeps it and uses 0.5 in the main text), and they sit at high rank, meaning large singular value. The detection procedure is Algorithm 1: for each of the top k singular vectors after fine-tuning, find the most similar pretrained singular vector; if that similarity is under the threshold, it is an intruder.
The paper shows the pattern on RoBERTa-base across six classification tasks and on LLaMA-7B and LLaMA2-7B fine-tuned on Alpaca, MetaMathQA and Magicoder-Evol-Instruct. Intruder dimensions appear at low LoRA ranks and shrink in number as rank increases; at r=64 and above LoRA solutions look more like full fine-tuning spectrally. Figure 5 tracks a single intruder over training: it appears early, its singular value grows, and it climbs the ranking as the run proceeds.
Key results
- Same accuracy, different weights (Table 2, RoBERTa-base, alpha = 2r): MNLI full fine-tuning 0.8745, LoRA r=8 0.8704, r=16 0.8739; QQP full 0.9152, LoRA r=8 0.9093. The accuracy gap is under a point while the spectral structure differs qualitatively.
- LoRA forgets less, even at matched fit (Figure 6): on LLaMA2-7B, measured on unrelated held-out tasks, and on RoBERTa via pretraining pseudo-loss, LoRA retains more of the pretraining distribution than full fine-tuning at the same downstream accuracy. The authors extend the earlier finding that LoRA forgets less by ruling out underfitting as the explanation.
- But not always: with different choices of LoRA alpha and learning rate, LoRA runs with nearly identical task accuracy show starkly different generalization, including cases where LoRA forgets more than full fine-tuning. Runs with the fewest intruder dimensions generalize best.
- Intruders cause the forgetting (Tables 3 and 4): scaling down the top intruder dimension in each matrix after training reduces pretraining loss far more than it reduces accuracy. RoBERTa QQP r=8: forgetting down 45.0% for a 0.6% accuracy drop; MNLI r=8: forgetting down 19.8% for 0.6%. LLaMA2-7B MetaMath r=16: forgetting down 28.2% for a 2.7% drop; Magicoder r=16: down 13.2% for 0.1%. Halving an intruder always costs under two percent accuracy on RoBERTa.
- Continual learning (Figure 9): training sequentially on six tasks, full fine-tuning retains earlier-task performance better than LoRA because LoRA accumulates intruder dimensions across tasks.
- What reduces intruders: higher rank, freezing a random orthogonal A and training only B (Figure 10), and lower learning rates or alpha (Appendix A.4).
Why the two methods diverge
The paper’s mechanistic account is the product parameterization. Full fine-tuning updates the whole matrix with a gradient that is, at the first step, a sum of rank-one terms aligned with the data. LoRA updates B and A separately, and because A is random at initialization, the update B A projects the gradient through a random low-dimensional subspace; the result is a direction that has no reason to align with the pretrained spectrum. Appendix A.2 shows that adding a random vector to the pretrained matrix reproduces intruder dimensions well, and A.3 derives the first-step difference between the two update rules. This is also why training only B on top of a fixed orthogonal A cuts intruders sharply.
Limits and open questions
The models are RoBERTa-base and 7B LLaMA variants; whether intruder dimensions behave the same in 70B-class models or under QLoRA-style quantization is not tested. Forgetting is measured by pretraining loss and by held-out task accuracy, both proxies for the capability loss users care about. The intervention of scaling down intruders is a post-hoc diagnostic, not a training recipe; the paper shows it works but does not propose a fixed schedule for choosing the scaling factor. And the study covers vanilla LoRA with a brief look at variants in Appendix O, so claims about DoRA or rank-stabilized variants should be checked against that appendix rather than assumed. The head-to-head with numbers from this paper and the original LoRA results is in LoRA vs Full Fine-tuning.
FAQ
What is the difference between LoRA and full fine-tuning according to this paper?
They reach similar task accuracy through different weight changes. LoRA introduces intruder dimensions, new high-ranking singular vectors nearly orthogonal to the pretrained spectrum, while full fine-tuning keeps the top singular vectors close to their pretrained directions.
Does LoRA forget less than full fine-tuning?
Usually, and the paper confirms it holds even when task accuracy is matched. But it depends on hyperparameters: with some alpha and learning-rate settings LoRA forgets more than full fine-tuning, and those runs have more intruder dimensions.
What happens if you scale down the intruder dimensions after LoRA fine-tuning?
Forgetting drops sharply while accuracy barely moves. On RoBERTa QQP with r=8, pretraining loss forgetting fell 45% for a 0.6% accuracy drop; on LLaMA2-7B MetaMath with r=16 it fell 28.2% for a 2.7% drop.
Which LoRA settings produce fewer intruder dimensions?
Higher rank (r=64 and above behaves more like full fine-tuning), training only the B matrix over a fixed orthogonal A, and lower learning rates or alpha. Continual fine-tuning across many tasks does the opposite and accumulates them.