Open Models · Language Models · Efficient AI
Mistral 7B vs Llama 2: The 7B Model That Reset the Open-Weights Baseline
Mistral 7B beat Llama 2 13B on every benchmark its authors ran, at half the size, with grouped-query and sliding-window attention under Apache 2.0; Llama 2 adds a 70B tier and the first documented RLHF chat recipe.
The two-line version
Mistral 7B is a 7-billion-parameter open model released in September 2023 under Apache 2.0, roughly half the size of Llama 2 13B, and it beat that 13B model on every benchmark the authors evaluated. Llama 2, released two months earlier by Meta, is a family at 7B, 13B and 70B whose lasting contribution was different: the first fully written-down RLHF chat pipeline shipped with commercially usable weights. The comparison matters for two reasons: it reset what “small open model” meant, and its most-cited numbers carry a harness caveat that most retellings skip.
Key numbers
The cleanest table in this comparison is Table 2 of the Mistral 7B paper, where Mistral’s team ran Mistral 7B, Llama 2 7B, Llama 2 13B and Code Llama 7B through the same evaluation pipeline. Every number below is from that single table.
| Benchmark (shots) | Llama 2 7B | Llama 2 13B | Mistral 7B | Gap vs 13B |
|---|---|---|---|---|
| MMLU (5-shot) | 44.4 | 55.6 | 60.1 | +4.5 |
| HellaSwag (0-shot) | 77.1 | 80.7 | 81.3 | +0.6 |
| WinoGrande (0-shot) | 69.5 | 72.9 | 75.3 | +2.4 |
| PIQA (0-shot) | 77.9 | 80.8 | 83.0 | +2.2 |
| ARC-Easy (0-shot) | 68.7 | 75.2 | 80.0 | +4.8 |
| ARC-Challenge (0-shot) | 43.2 | 48.8 | 55.5 | +6.7 |
| NaturalQuestions (5-shot) | 24.7 | 29.0 | 28.8 | −0.2 |
| TriviaQA (5-shot) | 63.8 | 69.6 | 69.9 | +0.3 |
| HumanEval (0-shot) | 11.6 | 18.9 | 30.5 | +11.6 |
| MBPP (3-shot) | 26.1 | 35.4 | 47.5 | +12.1 |
| MATH (4-shot maj@4) | 3.9 | 6.0 | 13.1 | +7.1 |
| GSM8K (8-shot maj@8) | 16.0 | 34.3 | 52.2 | +17.9 |
Three patterns in that table do the real explanatory work. First, the gaps are huge exactly where reasoning is: GSM8K +17.9, MBPP +12.1, HumanEval +11.6, MATH +7.1. Second, on pure world-knowledge tasks the 7B model cannot cheat its smaller capacity: Mistral 7B actually loses NaturalQuestions to Llama 2 13B by 0.2 points and edges TriviaQA by only 0.3. Third, commonsense and reading benchmarks sit in between, in the low single digits. The Mistral paper itself draws this line explicitly, estimating the model matches a Llama 2 more than 3x its size on reasoning and comprehension, but only about 1.9x on knowledge, because knowledge is parameter-bound and 7B simply stores fewer facts than 13B.
Why two papers give you two different Llama 2 scores
If you cross-check the table above against Meta’s own Llama 2 paper, the Llama 2 columns do not match. Meta reports MMLU 5-shot of 45.3 for the 7B and 54.8 for the 13B; Mistral’s rerun of the same models on the same benchmark produced 44.4 and 55.6. Both are “true.” They differ because evaluation harnesses (prompt formatting, answer extraction, few-shot sampling) move small-model scores by roughly a point in either direction. The honest way to compare across papers is to treat any single-digit difference as noise, and to keep each table in its own harness:
| Measurement | Mistral 7B (Mistral harness) | Llama 2 70B (Meta’s own paper) | Same harness? |
|---|---|---|---|
| MMLU (5-shot) | 60.1 | 68.9 | No |
| GSM8K (8-shot) | 52.2 | 56.8 | No |
| HumanEval (0-shot) | 30.5 | 29.9 | No |
| BIG-Bench Hard (3-shot) | not reported | 51.2 | n/a |
Even allowing for harness slop, the Llama 2 70B is the stronger model on raw capability: it leads Mistral 7B by 8.8 points on MMLU in their respective papers. Mistral 7B’s claim was never “best open model, period”; it was “best open model per parameter and per dollar,” and it held that claim at the 7B scale decisively.
Where the extra capability per parameter comes from
Two architectural levers do the mechanical work, both aimed at inference cost. Grouped-query attention shrinks the KV cache: Llama 2 7B and 13B give all 32 attention heads their own key and value heads, while Mistral 7B has 32 query heads but only 8 KV heads, so each KV head is shared by a group of four queries. A smaller KV cache means less memory bandwidth per generated token and higher throughput, which is the constraint that dominates real serving. Sliding-window attention caps what each token can attend to: the window is 4,096 tokens, and because windows stack across the model’s 32 layers the theoretical information span grows to roughly 131K tokens, with a rolling-buffer cache that reuses a fixed memory block. Mistral reports this as a 2x speedup at 16K sequence length. The stated context length is 8,192 tokens, double Llama 2’s 4,096.
Here is the honest caveat, and it is the one most “Mistral beats Llama” posts skip: the training data is not disclosed. The Mistral paper says nothing about pretraining data composition, and the instruction-tuned variant used only publicly available Hugging Face datasets with no proprietary data. So the benchmark win cannot be cleanly decomposed into “architecture gains” versus “data curation gains.” The NaturalQuestions and TriviaQA rows are the tell: if the architecture were doing all the work, knowledge-bound tasks would not flatline against a model with 6B more parameters.
What Llama 2 gives you that Mistral 7B does not
Scale tiers. Llama 2 ships 7B, 13B and 70B, and the 70B beats Mistral 7B head-to-head. If your task needs maximum quality from this generation, the 70B is the only option of the two.
A written chat recipe. Llama 2-Chat’s contribution is documentation, and it became the de facto textbook for open alignment: a few tens of thousands of high-quality SFT examples (Meta found quality beat quantity and stopped around 27K), over 1.4 million in-house human preference comparisons (2.9M including open-source data), two separate reward models for helpfulness and safety because one blended reward degraded both, iterative rejection sampling plus PPO, and Ghost Attention for keeping system instructions alive across turns. If you are building your own alignment pipeline, Llama 2’s paper tells you how; Mistral 7B’s tells you almost nothing about post-training.
A license with conditions, not freedom. Llama 2’s community license permits commercial use below 700 million monthly active users and blocks using its outputs to train competing models: open weights, not open source. Mistral 7B ships under Apache 2.0: commercial use, redistribution, fine-tuning, no strings. For a product team, that difference is often the deciding one, more than any benchmark.
Chat models: MT-Bench and a crowd vote
For the instruction-tuned variants, Mistral 7B Instruct scores 6.84 ± 0.07 on MT-Bench versus 6.65 for Llama 2 13B Chat, a 0.19 lead on a 10-point judge-graded scale, which is a modest edge, not a rout. The paper also reports a crowd-sourced head-to-head where Mistral outputs were preferred 5,020 times against 4,143 for Llama 2 13B Chat (about 54.8% of 9,163 votes as of October 6, 2023). Treat that second number as directional: it came from an open voting platform, not a controlled pairwise study, and the two models’ outputs were collected at different times. Meta’s own human evaluation for Llama 2-Chat 70B against ChatGPT (36% win, 31.5% tie) has the same vendor-run caveat and is not directly comparable to Mistral’s setup.
When to use which
- A base model at 7B to 13B scale for fine-tuning or on-device deployment: Mistral 7B, on every axis: better same-harness scores, half the parameters, 8 KV heads instead of 32, 8K context with a 131K theoretical span, Apache 2.0.
- Maximum capability from this generation, cost be damned: Llama 2 70B. Nothing in the Mistral 7B paper disputes that the 70B is stronger.
- You are building an alignment pipeline and want a written reference: Llama 2’s RLHF section (two reward models, rejection sampling plus PPO, Ghost Attention) remains the better document.
- A commercial product where license friction matters: Mistral 7B’s Apache 2.0 avoids Llama 2’s 700M-MAU clause and the output-restriction clause entirely.
- A new deployment in 2026: neither. Both are lineage; the direct successor line continues with Llama 3. This comparison explains why modern open models pair GQA with permissive licenses by default; pick a current-generation model and apply the same evaluation discipline this page uses.
Limits and open questions
The comparison is asymmetric in a way no table fixes: Meta disclosed training scale (2 trillion tokens) and pipeline details but not the data mixture either, and Mistral disclosed neither data nor training scale. Attribution of the win is therefore genuinely open. The chat-model numbers come from different evaluation styles (judge-graded MT-Bench, open crowd voting, vendor-run human evals) and should not be mixed into one ranking. Sliding-window attention’s practical long-context recall beyond the 8,192-token stated context is a theoretical-span claim, not a measured retrieval result. And every number here is from late 2023; the open-weights baseline has moved several times since, so use this page to understand the lineage and the evaluation caveats, not to choose a model to deploy today.
FAQ
Is Mistral 7B better than Llama 2?
At equal or smaller scale, yes, measurably. In the Mistral paper’s own harness, Mistral 7B beats Llama 2 13B on all 12 reported benchmarks, with the largest gaps on math and code (GSM8K +17.9, MBPP +12.1, HumanEval +11.6). Against Llama 2 70B, no: the 70B leads MMLU by 8.8 points across the two papers’ own tables, though on different harnesses.
Why does Mistral 7B beat Llama 2 13B with fewer parameters?
Partly architecture (grouped-query attention with 8 KV heads instead of 32, and sliding-window attention, which make inference cheaper rather than smarter) and partly an undisclosed training recipe. The paper’s own analysis bounds the claim: Mistral 7B matches a Llama 2 over 3x its size on reasoning and comprehension but only ~1.9x on knowledge, and on knowledge benchmarks it ties or loses to the 13B, which is the signature of data and recipe gains layered on a capacity-limited model.
What is the difference between the Mistral 7B and Llama 2 licenses?
Mistral 7B is Apache 2.0: commercial use, redistribution and fine-tuning with essentially no restrictions. Llama 2 uses Meta’s community license: commercial use is allowed below 700 million monthly active users, and you may not use Llama 2 outputs to improve other models. Practically, Apache 2.0 is safe default code-lawyer territory; the Llama 2 license needs a policy decision.
How do Mistral 7B Instruct and Llama 2 13B Chat compare?
On MT-Bench, Mistral 7B Instruct scores 6.84 ± 0.07 versus 6.65 for Llama 2 13B Chat. In the Mistral paper’s open crowd vote, Mistral outputs were preferred 5,020 to 4,143. Both are vendor-reported numbers on different setups, so read them as “roughly comparable, Mistral slightly ahead,” not as a precise ranking.
Does sliding-window attention mean Mistral 7B has a 131K context?
No. The stated context length is 8,192 tokens. The ~131K figure is a theoretical information-span estimate from stacking 4,096-token windows across 32 layers; it is a statement about how far information can propagate, not a measured guarantee of retrieval quality at that distance. Treat 8K as the supported context and the larger number as a cost-modeling artifact.