Multimodal Models · Vision Foundation Models
Flamingo vs LLaVA: Few-Shot Prompting or Visual Instruction Tuning?
Flamingo froze both backbones and learned image tasks from 32 in-context examples; LLaVA fine-tuned Vicuna on 158K GPT-4-written instructions. Same frozen-encoder trick, opposite task interface.
Two answers to the same question
Both papers ask how to make a strong text model see. Flamingo (DeepMind, NeurIPS 2022) answers: freeze everything. The vision encoder and the language model stay untouched, only the bridge between them is trained, and the prompt carries the task. LLaVA (2023) answers the opposite: fine-tune the language model itself on instructions about images, and let a text-only GPT-4 write those instructions. Twelve months apart, the field’s center of gravity moved from one answer to the other, and understanding why is still the fastest way to understand how open multimodal models are built today.
The architectural details matter less than the interface each paper bets on. Flamingo bets that the prompt is the task: give the model a few <image> answer pairs and it continues the pattern, so one frozen set of weights serves captioning, VQA, multiple-choice, and video QA with zero gradient updates. LLaVA bets that users will not write few-shot prompts: they will type “what’s in my fridge?” and expect a chatty, instruction-following reply, which only fine-tuning delivers.
How each model connects vision to language
Flamingo keeps both backbones completely frozen. A Perceiver Resampler squeezes the vision encoder’s variable-size feature grid into a fixed set of visual tokens, and gated cross-attention layers interleaved into the frozen language model let text attend to those tokens. A learned gate initialized near identity means the model starts life as the plain language model and the visual path fades in during training. The largest variant, Flamingo-80B, pairs a 70B Chinchilla language model with a frozen vision encoder; Flamingo-3B and 9B use 1.4B and 7B backbones.
LLaVA is aggressively simpler. A frozen CLIP ViT-L/14 encodes the image, one trainable linear projection maps the features into Vicuna’s word-embedding space, and a Vicuna LLM reads the projected tokens as if they were text. Training runs in two stages: first only the projection matrix is trained on 595K filtered CC3M caption pairs, then the projection and the entire LLM are fine-tuned together on 158K instructions while the vision encoder stays frozen. No cross-attention stack, no resampler, one matrix as the connector.
Key numbers
| Measurement | Flamingo-80B | LLaVA (13B Vicuna) | Notes |
|---|---|---|---|
| Connector trained | Perceiver Resampler + gated cross-attention | Single linear projection | Flamingo paper Fig. 3 vs LLaVA Sec. 3 |
| Language backbone during training | Frozen 70B Chinchilla | Vicuna-13B unfrozen in stage 2 | Opposite strategies |
| Alignment training data | M3W 43M webpages, ALIGN 1.8B pairs, LTIP 312M pairs, VTP 27M videos | 595K CC3M captions + 158K GPT-4 instructions (58K conversation / 23K detail / 77K reasoning) | GPT-4 never saw a pixel |
| VQAv2 | 56.3 0-shot, 63.1 4-shot, 67.6 32-shot | Not reported | LLaVA defers academic VQA to LLaVA-1.5 |
| OK-VQA | 57.8 at 32 shots | Not reported | Fine-tuned SOTA was 54.4 with 10K examples |
| TextVQA | 37.9 at 32 shots | Not reported | Reading-centric VQA |
| COCO captioning (CIDEr) | 84.3 0-shot, 113.8 32-shot | Not reported | Fine-tuned SOTA 143.3, trained on 500K captions |
| LLaVA-Bench (COCO), relative to GPT-4 | Not applicable | 85.1 overall; 21.5 without instruction tuning | Conversation 83.1, detail 75.3, reasoning 96.5 |
| ScienceQA accuracy | Not reported | 90.92 alone, 92.53 with GPT-4 as judge | Human 88.40, prior SOTA MM-CoT 91.68, LLaVA 7B 89.84 |
| LLaVA-Bench (In-the-Wild), relative score | OpenFlamingo 19.1 | 67.3 ± 2.0 | BLIP-2 38.1; judged against text-only GPT-4 |
| Fine-tuned upper bound | VQAv2 82.0, COCO 138.1 after full fine-tuning | 92.53 ScienceQA with GPT-4 judge | Different tasks, no shared harness |
Read the table for what it cannot give you: a single row where Flamingo and LLaVA face the same benchmark under the same conditions. The LLaVA paper reports no VQAv2, no COCO captioning, no traditional VQA numbers at all; its authors explicitly defer academic VQA to the follow-up LLaVA-1.5. The closest cross-paper number is the LLaVA-Bench In-the-Wild row, where LLaVA is compared with OpenFlamingo, the open reimplementation of Flamingo: 67.3 versus 19.1 on a GPT-4-judged relative scale. That is a real gap, but it compares LLaVA against a reimplementation, not against Flamingo-80B itself, which was never released.
What few-shot prompting actually buys
Flamingo’s headline is scaling with examples rather than gradient steps. VQAv2 climbs from 56.3 with zero examples to 67.6 with 32, OK-VQA from 50.6 to 57.8, COCO captioning from 84.3 to 113.8 CIDEr. On six of the sixteen benchmarks, 32 in-context examples beat the contemporary fine-tuned state of the art, which had seen up to hundreds of thousands of task-specific labels. VQAv2’s 80.2 fine-tuned SOTA came from a model trained on 444K VQA pairs, 14,000 times more examples than Flamingo’s prompt. And one frozen model covers all of it, including video: MSVDQA reaches 52.3 and VATEX 65.1 at 32 shots.
The costs are real too. Few-shot numbers are sensitive to which examples are picked and in what order, so every figure above reflects a favorable prompt more than a guaranteed outcome. A 32-shot prompt is long, which makes inference expensive for a 70B-plus-vision-encoder stack. And the paradigm never teaches the model to chat: Flamingo continues a pattern, it does not follow an instruction phrased its own way. Fine-tuning Flamingo itself, the ablation the paper reports, pushes VQAv2 to 82.0 and COCO to 138.1 CIDEr, which quietly admits that when labels exist, fine-tuning wins.
What instruction tuning actually buys
LLaVA’s contribution is not an architecture, it is a recipe that a university lab can afford. The 158K instructions are written by text-only GPT-4 fed captions and bounding boxes as symbolic stand-ins for images; the vision encoder is an off-the-shelf CLIP; the connector is one matrix; the whole two-stage training runs on 8 A100s in roughly 4 hours of feature alignment plus 10 hours of instruction fine-tuning. The released artifacts made multimodal chat reproducible for everyone, which is why the paper has thousands of descendants.
The numbers that exist are strong but narrowly framed. 90.92 on ScienceQA without any help exceeds the 88.40 human average and nearly matches prior state of the art MM-CoT’s 91.68; adding GPT-4 as a judge on top pushes the combination to 92.53. The 85.1 relative score against GPT-4 on LLaVA-Bench is really a consistency measure, since GPT-4 both wrote the training data and serves as the judge, and the paper shows the score collapses to 21.5 without instruction tuning, proving the fine-tuning stage matters more than the connector. The In-the-Wild comparison against OpenFlamingo (67.3 versus 19.1) is the paper’s one direct strike at the Flamingo lineage, and it lands, but on a benchmark where the judge belongs to the winning team’s lineage.
Why the field moved from Flamingo to LLaVA
Neither result refutes the other; they optimize different things. Flamingo optimizes coverage: how many tasks one frozen model can address without any retraining, which matters when annotation is impossible. LLaVA optimizes usability: how naturally a person can talk to the model, which turned out to be what the market wanted. Once chat became the dominant interface, few-shot prompting lost its main advantage, because nobody wants to paste 32 examples before asking a question. Instruction tuning’s weakness, that it needs instruction data, was neutralized by LLaVA’s own trick of using GPT-4 as a data factory.
The lineages also merged. Every modern open VLM fine-tunes its language model (LLaVA’s choice) while keeping the vision encoder frozen (both papers’ choice), and the connector evolved from LLaVA’s linear layer toward richer designs that Flamingo’s gated cross-attention anticipated. Flamingo’s few-shot ability did not disappear either: in-context examples returned as multimodal few-shot evaluation, and its interleaved image-text format survives in models built for arbitrary image-text sequences.
When to use which
- You need one model to cover many vision tasks with no task-specific training data: Flamingo’s recipe. Few-shot prompting is the only option when annotated data does not exist, and 32 examples beating a model fine-tuned on 444K is the proof that it works.
- You are building a user-facing assistant that must follow open-ended instructions about images: LLaVA’s recipe. Instruction tuning is what makes “describe this photo in the style of a police report” work without engineering a prompt template.
- You have a modest compute budget and want reproducibility: LLaVA, decisively. One projection matrix, 8 consumer-grade GPUs for a day, and publicly released data, weights, and code.
- You care about video QA or interleaved image-text sequences in a single forward pass: Flamingo’s design handles native interleaving and shipped video benchmarks; LLaVA v1 is image-only single-turn.
- You need trustworthy benchmark numbers: read both papers’ tables with the judge and harness in mind: Flamingo’s are prompt-sensitive, LLaVA’s are GPT-4-judged.
Limits and open questions
The comparison’s structural gap is that these papers never met on a benchmark. LLaVA’s VQAv2 deferral means there is no number anywhere that answers “which model is better at vanilla VQA,” and LLaVA-Bench In-the-Wild, the one overlapping evaluation, used a reimplementation of Flamingo judged by the same GPT-4 family that trained the winner. Flamingo’s few-shot figures inherit prompt-selection sensitivity, so treat 67.6 VQAv2 as an upper bound rather than an expectation. LLaVA’s ScienceQA ensemble conflates model accuracy with judge quality: 92.53 requires GPT-4 in the loop, while LLaVA alone scores 90.92. And both are 2022–2023 results: current VLMs differ from both, so this comparison explains where the design space came from, not which recipe to copy for a frontier model.
FAQ
What is the difference between Flamingo and LLaVA?
Flamingo trains only a connector between a frozen vision encoder and a frozen language model, and tasks are specified by few-shot examples in the prompt. LLaVA fine-tunes the language model itself on 158K image instructions written by GPT-4, with a single linear projection as the connector. Same frozen-vision idea, opposite answer to where the task lives.
Is Flamingo few-shot and LLaVA fine-tuned?
Yes, exactly. Flamingo reaches 67.6 on VQAv2 using 32 in-context examples and no weight updates. LLaVA’s Vicuna backbone is fully fine-tuned in stage 2 on the 158K instruction set; without that instruction tuning, its LLaVA-Bench score drops from 85.1 to 21.5 relative to GPT-4.
Which is better on benchmarks, Flamingo or LLaVA?
They were never measured on the same benchmark, so there is no honest overall winner. Flamingo-80B reports 67.6 VQAv2 32-shot, 57.8 OK-VQA, 113.8 COCO CIDEr; LLaVA reports 90.92 ScienceQA alone and 85.1 LLaVA-Bench relative to GPT-4. On the one overlapping evaluation, LLaVA-Bench In-the-Wild, LLaVA scored 67.3 against OpenFlamingo’s 19.1.
Why is LLaVA so much cheaper to reproduce than Flamingo?
The connector is one linear layer instead of a Perceiver Resampler plus gated cross-attention stack, the instruction data is generated by GPT-4 rather than curated from 1.8B web image-text pairs and 27M videos, and training takes about 14 hours on 8 A100s. LLaVA’s authors released the data, weights, and code; Flamingo-80B was never released.
Is LLaVA’s 85.1% score against GPT-4 trustworthy?
Partially. The score means LLaVA’s answers were judged 85.1% as good as text-only GPT-4 with ground-truth captions on LLaVA-Bench. But GPT-4 wrote the training instructions and serves as the judge, so the number partly measures agreement within the same model family. The 21.5 score without instruction tuning is the more robust finding: fine-tuning is where the capability comes from.