Vision Foundation Models · Segmentation
SAM vs SAM 2: 6x Faster on Images, but the Real Upgrade Is Video Memory
Meta's SAM 2 beats SAM on the same image suites (58.9 vs 58.1 one-click mIoU) while running 6x faster, and adds a streaming memory bank so a single prompt tracks an object through a whole video.
Two models, two problems
SAM solved a data problem. Its paper’s lasting contribution is as much SA-1B, 1.1 billion masks on 11 million licensed images, as it is the model itself: a three-stage data engine where the model and the dataset bootstrap each other, ending in a promptable segmenter that transfers zero-shot to images and tasks it was never trained on. SAM thinks in stills. Give it a point, box, or mask, and it returns a mask for that one frame.
SAM 2 solves a time problem. Promptable segmentation in video means the prompt may arrive on any frame and the mask must stay correct on every later frame, through occlusion, reappearance, and objects that deform or leave the scene. SAM 2 keeps the same prompt vocabulary, click, box, or mask, but wraps the mask decoder in a streaming loop: each frame’s prediction is written into a memory bank, and a memory-attention module lets the current frame read what the model has already seen. That is the entire architectural difference that matters, and it is what turns one prompt into a track.
Key numbers
Every SAM-versus-SAM-2 row below comes from the SAM 2 paper’s own evaluation, run under one harness by the same team, which is the closest thing to a controlled comparison that exists. Treat them as first-party numbers, not third-party reruns.
| Measurement | SAM | SAM 2 | Setting and source | Same harness? |
|---|---|---|---|---|
| Training data | SA-1B: 1.1B masks, 11M images | SA-1B + SA-V: 642.6K masklets, 50.9K videos (35.5M masks, 196 hours) | Both papers’ data sections | No, different modalities |
| Zero-shot image, 1-click mIoU (SA-23 avg) | 58.1 | 58.9 trained on SA-1B only; 61.9 with full data mix | SAM 2 paper, Table 5 | Yes |
| Zero-shot image, 5-click mIoU (SA-23 avg) | 81.3 | 81.7 trained on SA-1B only; 83.5 with full data mix | Table 5 | Yes |
| Speed, single A100, batch 1 | 21.7 FPS | 130.1 FPS | Table 5 | Yes, about 6x |
| Zero-shot video, 1-click J&F (17 datasets avg) | 56.9 with XMem++ add-on; 56.7 with Cutie add-on | 64.7 | Table 4 | Yes, against SAM-plus-tracker pipelines |
| Zero-shot video, 5-click J&F (17 datasets avg) | 70.6 (XMem++); 72.2 (Cutie) | 77.6 | Table 4 | Yes |
| Semi-supervised VOS, MOSE val J&F | Not applicable | 76.6 (Hiera-B+); 77.9 (Hiera-L); prior best Cutie-base+ 71.7 | Table 6, first-frame ground-truth mask | Yes, against published VOS systems |
| Interactive annotation efficiency | Not applicable | More than 3x fewer interactions than SAM+XMem++ or SAM+Cutie at comparable accuracy | Section 6.1, 9 densely annotated datasets, 3 clicks per frame | Yes |
Two readings of this table are easy to get wrong. First, the image mIoU gap is small: 58.1 versus 58.9 is 0.8 points, and even the fully trained SAM 2 is at 61.9. If your workload is still images, SAM 2 is not a quality revolution; it is a speed revolution that happens to come with a small quality gain. Second, SAM never appears in the video rows as itself, because it has no temporal mechanism. The honest baseline for “SAM on video” is a pipeline that feeds SAM masks into a dedicated video object segmentation tracker such as XMem++ or Cutie, and that pipeline loses by roughly 8 J&F points at one click while needing more human interactions to reach the same accuracy.
Where the 6x comes from
The speed gap is not a mystery and not primarily a software engineering feat. SAM runs a ViT-H image encoder; SAM 2 runs the smaller Hiera-B+ encoder, which the paper credits for most of the throughput difference, 21.7 versus 130.1 frames per second on a single A100. Because the encoder is smaller, the video memory module can afford to attend over stored frame features at interactive rates.
There is a cost hidden in this trade. Trained only on SA-1B, SAM 2 actually trails SAM on the 14 newly collected video datasets when their frames are evaluated as still images, 56.6 versus 59.1 one-click mIoU. It is the video data mix, SA-V’s 642.6K masklets (190.9K human-annotated, the rest generated by the data engine and quality-filtered), that lifts those numbers to 69.6 and pushes the SA-23 image average to 61.9. So the dataset, again, is doing the heavy lifting: same story as SA-1B in the first paper, one meta-level up.
What SAM 1 still does fine
SAM is not obsolete, and the numbers do not say it is. On fixed still-image tasks with abundant in-domain labels, a specialized supervised model such as Mask2Former can still beat a zero-shot promptable model, and SAM’s own paper concedes this. SAM also has years of ecosystem behind it: labeling tools, fine-tunes, deployed annotation pipelines, and hardware integrations that assume its exact API. If your product annotates product photos, medical slides, or map tiles one frame at a time, and the pipeline already works, the 0.8 mIoU and the 6x throughput are unlikely to justify a migration by themselves. SAM 2’s image checkpoint is also a different encoder family, so embeddings and any downstream heads trained on SAM features do not transfer.
When to use which
- Any video workload, annotation or inference: SAM 2, unambiguously. SAM plus a tracker is the old best practice, and SAM 2 beats that stack on accuracy, interaction count, and simplicity, one model instead of two.
- Interactive video annotation at scale: SAM 2 for the efficiency claim alone. More than 3x fewer interactions per accepted mask is a direct multiplier on annotation throughput, and SA-V itself was built with exactly this loop.
- Still-image pipelines that are already deployed: stay on SAM unless throughput hurts. The quality gain is real but small; migrate when you next touch the pipeline.
- Dense feature extraction rather than masks: neither, look at self-supervised backbones such as DINOv2, which is built for frozen features rather than promptable masks.
Limits and open questions
All same-harness numbers are first-party: Meta built SAM 2 and evaluated both models itself. There is no independent rerun in this comparison, and the community track record on SAM 1 showed that third-party results sometimes diverge from paper numbers on out-of-distribution imagery. The video evaluation is young: the 17 zero-shot video datasets and the J&F averages that SAM 2 reports were largely introduced or aggregated by the SAM 2 paper itself, so “state of the art on video” currently means state of the art on a benchmark the authors assembled. On the failure side, the paper’s own limitations section points at fast motion, complex occlusions, disappearance and reappearance, and small or thin objects, exactly the regimes where a memory bank has to guess across gaps. Finally, 451.7K of SA-V’s 642.6K masklets are model-generated rather than human-drawn, which is fine for training but worth remembering when citing dataset quality figures. Whether SAM 2’s memory architecture becomes the default for video perception, or is superseded by video-native multimodal models that segment as a side effect, is the open question this comparison cannot answer.
FAQ
Is SAM 2 strictly better than SAM on images?
No. Under the same evaluation, SAM 2 gains 0.8 mIoU at one click (58.9 versus 58.1) and 0.4 at five clicks when trained only on SA-1B, rising to 61.9 and 83.5 with the full data mix. The decisive image-side number is speed: 130.1 versus 21.7 FPS on a single A100. If your input is still images and your current pipeline is fast enough, the accuracy alone does not justify switching.
Can SAM do video if I add a tracker like XMem++ or Cutie?
That was the standard recipe, and the SAM 2 paper measures it directly: SAM masks piped into XMem++ or Cutie reach 56.9 and 56.7 average J&F at one click across 17 zero-shot video datasets, against 64.7 for SAM 2. In interactive annotation SAM 2 needs more than 3x fewer interactions to reach comparable accuracy. A two-model pipeline also doubles the integration and failure-mode surface compared to one model with a built-in memory bank.
Why is SAM 2 faster when it does more work per frame?
SAM 2 does more per frame, but its image encoder is smaller. SAM uses a ViT-H encoder at 21.7 FPS on one A100; SAM 2 uses Hiera-B+ at 130.1 FPS, and the paper attributes the speedup mainly to that smaller, more efficient encoder. The memory attention and memory bank add sequential work, but they operate on compact stored features, so the net effect is roughly 6x faster even with video machinery included.
Which SAM 2 checkpoint should I use, Hiera-B+ or Hiera-L?
Hiera-B+ is the default: 76.6 J&F on MOSE val in the semi-supervised VOS protocol. Hiera-L buys about one to one-and-a-half points (77.9 on MOSE, 78.4 SA-V test) at higher compute per frame. If you annotate video interactively or run near real time, take Hiera-B+; if you batch-process archival footage offline and want the best mask quality, take Hiera-L.
Is SAM 2’s SA-V dataset usable for training my own model?
Yes. Meta released the SA-V dataset (50.9K videos, 642.6K masklets, 196 hours) together with the model, training code, and a demo, under the same open-release pattern as SA-1B. Note the composition: 190.9K masklets are human-annotated and 451.7K are generated by the data engine and then quality-filtered, which matters if your use case depends on human-drawn precision.