ViT vs ResNet: When Transformers Beat ConvNets (and When They Don't)

The one design difference

Both architectures started from the same observation: depth is where the accuracy is, but depth is hard to train. ResNet (2015) attacked the problem inside the network. Its skip connection carries the input forward unchanged and adds it to a block’s output, so each block only has to learn a residual correction instead of a full mapping. That made 152-layer networks trainable for the first time (8x deeper than VGG at lower FLOPs), and the skip connection turned out to be a universal primitive: the residual stream inside every Transformer, including every language model, is the same idea.

ViT (2020) attacked the problem from the opposite direction. Instead of fixing the convnet, it deleted the convnet. Split a 224x224 image into 196 patches of 16x16 pixels, flatten each patch through one linear projection, add a [class] token and position embeddings, and feed the sequence to a near-unmodified language Transformer encoder. No convolutions, no locality prior, no translation equivariance: everything the model knows about images must be learned from data.

CLIP (2021) kept the ViT-style image encoder but changed the supervision signal: instead of predicting a fixed label set, two encoders learn to match 400 million internet image-text pairs with a contrastive objective, and classification at inference time is just comparing the image embedding against text embeddings of the candidate labels.

So the real comparison is not “old CNN vs new Transformer.” It is three bets on where the difficulty of vision lives: in optimization (ResNet), in architecture-neutral scale (ViT), and in supervision (CLIP).

Key numbers

Measurement	ResNet (2015)	ViT (2020)	CLIP (2021)	Same harness?
ImageNet headline	3.57% top-5 error, ensemble, ILSVRC 2015 1st	88.55% top-1, ViT-H/14, JFT-300M pretraining	Zero-shot accuracy matches the original ResNet-50	No, different years, metrics, and data
Training data behind the headline	1.28M labeled ImageNet images	300M images (JFT-300M, proprietary)	400M unlabeled web image-text pairs	No
CIFAR-10	Trained 100- and 1000-layer nets to study optimization	99.50% (ViT-H/14)	Not the focus	No
CIFAR-100	n/a	94.55% (ViT-H/14)	n/a	No
VTAB, 19-task suite	n/a	77.63% (ViT-H/14), at or above BiT-L CNNs	Transfers across 30+ datasets zero-shot	No
Transfer beyond classification	28% relative gain on COCO detection from the backbone	Attention spans whole image in early layers	Competitive with fully supervised baselines on many tasks	Different tasks
Pre-training compute	n/a	Fewer TPU-core-days than the BiT / Noisy Student CNNs it matches	Contrastive objective chosen because generative captioning was too slow to scale	ViT row compares against CNN baselines in its own paper

Read the table for what it does not contain: a single paper that trains all three under one protocol. The ResNet number is an ensemble top-5 from 2015; the ViT number is a single-model top-1 from 2020 on a proprietary 300M-image set; the CLIP number is deliberately a modest baseline (the original ResNet-50), because its claim is generality, not a leaderboard top spot. Any article that lines these up as if they were one race is misleading you.

The crossover that actually decides the comparison

The most quoted sentence in the ViT paper is the honest one: pre-train on ImageNet alone (~1.3M images) and ViT underperforms comparable ResNets. Pre-train on ImageNet-21k and it pulls roughly even. Pre-train on JFT-300M and it overtakes. That crossover is the whole story of CNNs versus Transformers in vision.

A ResNet bakes in priors (locality, translation equivariance, 2D neighborhood structure) that are simply correct for images, so on 1.3M images the model that starts with the right assumptions wins. At 300M images those priors stop being help and start being constraint; the Transformer is free to learn whatever structure the data actually has, and the paper shows it rediscovers spatial locality on its own, in the learned position embeddings and the early-layer attention maps. Scale buys what architecture used to.

This is also why “is ViT better than ResNet” is a badly posed question for your own project. The answer is conditional on a number the benchmarks never print: how much pre-training data you can reach.

The CLIP pivot: changing the game instead of winning it

CLIP’s relationship to the ResNet comparison is subtler. It does not beat the best supervised model of its era on ImageNet; the paper is explicit that it matches the original ResNet-50, a 2015 baseline, while using none of ImageNet’s 1.28M labels. What it changes is the cost of adding a task. A ResNet or a fine-tuned ViT needs new labels and a new training run per task; CLIP swaps the prompt (a photo of a {label}) and the same frozen weights move from ImageNet to OCR to geo-localization, non-trivially across more than 30 datasets.

The trade is documented in the paper’s own limits. Accuracy is sensitive to prompt wording, performance is weak on abstract tasks like counting, and the model inherits the biases of uncurated web data. CLIP is a representation breakthrough, not a reasoning one, but it made natural language a control surface for vision, which is the assumption every later multimodal model builds on.

When to use which

• Small or medium labeled dataset, tight compute budget: start ResNet-style or with a data-efficient ViT variant. ViT’s ImageNet-1k result trails ResNets; the 88.55% headline does not apply to your regime.
• You have (or can rent) very large pre-training data: ViT scales better. Attention’s advantage grows with model and data size, and the paper measured it reaching CNN-level accuracy with fewer TPU-core-days at the large end.
• Many tasks, few labels per task, text is available: CLIP-style contrastive pre-training, then zero-shot or linear-probe per task. Swapping prompts beats retraining heads when the task count is high and labels are scarce.
• Dense tasks (detection, segmentation): neither plain ViT nor the original convnet is the last word; the flat patch grid is a poor fit for dense prediction, which is why hierarchical designs came after. ResNet backbones and hierarchical ViT descendants both live here.

Limits and open questions

The flagship ViT numbers depend on JFT-300M, a proprietary Google dataset, so they are hard to reproduce externally; in the data regimes most teams actually have, the CNN remains the stronger and cheaper choice. CLIP’s zero-shot match of ResNet-50 is a statement about label efficiency, not peak accuracy, and prompt sensitivity means its scores move with phrasing. The ResNet 3.57% figure is an ensemble, so single-model comparisons should use a weaker number. And all three papers predate the modern recipe (self-supervised pretraining, masked image modeling, hierarchical attention) that is what actually fills the gap between these results and today’s practice. Treat this comparison as the clean statement of the trade-offs, not as a buying guide for 2026 model selection.

FAQ

Is a vision transformer better than a CNN?

Conditional on data scale. On ImageNet-scale data (~1.3M images) ViT underperforms comparable ResNets, as its own paper reports. Pre-trained on hundreds of millions of images, ViT-H/14 reaches 88.55% top-1 and matches or beats the best CNNs of its era (BiT-L) with less pre-training compute.

Why does ViT need more data than ResNet?

Because a ResNet has locality and translation equivariance built in, while ViT strips those priors out and must learn them. Below the data threshold the built-in priors win; above it they become constraints and the flexible architecture wins.

Did CLIP beat ResNet on ImageNet?

Not the way headlines suggest. Zero-shot CLIP matches the original ResNet-50 without any of ImageNet’s 1.28M training labels. It trails stronger supervised models on hard specialized datasets; its contribution is doing all of this with one frozen model across 30+ tasks, not a top score.

What do ResNet and Transformers have in common?

The skip connection. ResNet’s residual block (output the input plus a learned correction) is the same mechanism as the residual stream in every Transformer encoder layer, which is why ResNet is usually taught as a precursor to the modern deep-learning stack rather than a rival to it.