DeepSeek's small model beat its big one on nine benchmarks. DeepSeek picked all nine.

On the last day of July, DeepSeek quietly promoted a model. The build is called V4-Flash-0731, and the headline that travelled with it was irresistible: a smaller, cheaper model that beats the company's own flagship. Not ties it. Beats it — on all nine of the agentic and coding benchmarks DeepSeek published, from a model that activates a fraction of the parameters its bigger sibling does. Every bar on the chart went up, the reposts wrote themselves, and "the small one beat the big one" became the story of the release inside a day.

It is a genuinely interesting result, and it may even be true. But before we let it harden into a fact, ask the question almost nobody asked before hitting repost: it beat the big one on which nine benchmarks, chosen by whom, scored how, against what — and could you reproduce the number if you tried? Turn the release over and look at the seams, because that is where the real story is. And the real story is more careful, more checkable, and in one important respect more creditable than the headline that outran it.

What actually shipped

Start with what is not in dispute, because it matters. On 31 July, DeepSeek moved its V4-Flash API out of preview and into a general-availability beta under the build tag 0731. This was not a new architecture. By DeepSeek's own account, the model keeps the exact structure and size of the earlier V4-Flash-Preview; the only thing that changed is post-training — the reinforcement-learning and fine-tuning layer applied on top of the same base. Same skeleton, retrained reflexes. That is a specific, falsifiable claim, and I want to flag early that it is the honest kind: it tells you precisely what to credit the gains to and what not to.

The architecture underneath is a mixture-of-experts model — a large network that routes each token through only a small slice of its total parameters. The published figures put the total backbone at roughly 284 billion parameters, of which about 13 billion are active on any given token. The flagship it is being measured against, V4-Pro-Preview, activates something closer to 49 billion. So the compressed version of the claim — the version that went viral — is: a 13-billion-active model beat a 49-billion-active model. Smaller beats bigger. It is a good line. It is also where the first footnote lives.

Because "active parameters" is not a measure of size in the sense the headline implies. It is a measure of the compute spent per token, not of how much the model knows or how it was trained. A model can activate fewer parameters and still be the beneficiary of a longer, better, more expensive post-training run — which, by DeepSeek's own telling, is exactly what happened here. The gain did not come from shrinking the model. It came from training the same model harder. "Smaller model wins" quietly reframes a post-training result as an architecture result, and those are different claims with different implications. The interesting sentence is not "small beats big." It is "post-training moved the agentic scores by ten to twenty points without touching the weights' shape" — which is remarkable if it holds, and the rest of this piece is about that if.

Compared to what?

Here is the number everyone quoted: on Terminal-Bench 2.1, an agentic benchmark that scores a model on multi-step tasks in a real terminal, V4-Flash-0731 scored 82.7. The preview it replaced scored 61.8. The flagship V4-Pro-Preview scored 72.1. Read that ordering again, slowly, because it is the whole trick in miniature: the new model is being compared to two other DeepSeek models. Its own earlier self, and its own bigger sibling. Both comparisons are ones DeepSeek was always going to win, because it controls both reference points.

"Beats its own flagship" sounds like a competitive claim. It is an internal one. The comparison that would actually tell you where this model sits — against the frontier agentic systems from the labs DeepSeek is racing — is the comparison the launch chart does not foreground. Beating last quarter's preview and a sibling that was itself only ever a preview is the easiest bar in the building to clear. It is not nothing. But "our new small model beats our old bigger model" and "our new small model beats the best agent anyone else ships" are separated by an ocean, and only one of them was demonstrated.

"Beats its own flagship" sounds like a competitive claim. It is an internal one — DeepSeek controls both models on the chart, and picked which nine tests to run them on.

Then there is the suite itself. Nine benchmarks: Terminal-Bench, a software-engineering agent test, an offensive-security agent test called Cybergym, and the rest in the same agentic family. Every one of them was selected by DeepSeek for the launch post. There is nothing sinister in a company reporting the benchmarks it does well on — everyone does it — but the reader's job is to remember that a nine-for-nine sweep on a self-chosen suite carries exactly as much information as the selection process behind it, and we cannot see that process. Compared to what? Compared to the nine tests that made the slide. The ones that didn't are not on the slide.

Where the error bars went

Now the part that actually keeps me up, because it is the part that generalises past DeepSeek to the entire genre of agentic benchmarking. Every score in this release is a single number. 82.7. 54.4. 76.7. No range, no confidence interval, no indication of how many times each task was run or how much the result moved between runs. And of all the things you can measure a model on, agentic benchmarks are the ones where a bare point estimate is least trustworthy.

An agentic score is not like a multiple-choice exam, where the same question yields the same answer every time. It is the outcome of a model taking dozens of steps in a live environment — running commands, reading output, deciding what to do next — where a single early wrong turn cascades into a failed task. Change the scaffolding, the temperature, the retry policy, the timeout, or simply run it again on a different random seed, and the score moves, sometimes by several points. Independent researchers who re-run these suites routinely find run-to-run swings large enough to swallow the gap between two "different" leaderboard positions. Which means a headline like 82.7 versus 72.1 might be a real ten-point improvement — or it might be a four-point improvement wearing a good outfit, with the rest inside the noise nobody printed.

I am not saying the gains are fake. I am saying that as published, they are unfalsifiable in the specific sense that matters: you cannot tell, from the numbers given, whether the sweep would survive a second run. The fix is boring and well known — report the mean and the spread across multiple runs, hold out a fresh set of tasks the model was not tuned against, publish the harness — and it is boring precisely because it works. A confidence interval is not screenshottable. A single big number is. Guess which one shipped.

The silent swap

One detail slid by almost unremarked, and it deserves a paragraph of its own. For developers already calling the model, DeepSeek advertised the upgrade as zero-effort: same endpoint, same API key, same model name. Your code doesn't change. Which is convenient, and also means that anyone who was calling deepseek-v4-flash on 30 July was talking to a different model on 1 August without lifting a finger.

For most users that is a free improvement. For anyone running evaluations, it is a quiet hazard. If you benchmarked deepseek-v4-flash in July and again in August and compared the two, you did not measure drift or your own prompt changes — you measured a model substitution you were never told to control for. A pinned build tag exists (that is what 0731 is), but the default name floats. Reproducibility depends on people knowing to pin it, and most won't. The lesson is old and keeps needing to be relearned: a model behind a moving name is not a fixed object, and any number you cite about it has a silent "as of" attached.

The thing DeepSeek did right

I spend most of my time in this column taking numbers away from people, so let me hand one back, because it is the most important line here. You can check this one. DeepSeek published the 0731 weights to Hugging Face under an MIT licence — an open release, roughly 167 gigabytes of it, that anyone with the hardware can download, run, and re-benchmark on tasks DeepSeek never chose. (A few launch-day write-ups described the build as API-only; the repository says otherwise, which is itself a small reminder to check the source and not the recap.)

That openness is the difference between a claim and an argument. A closed model that reports a nine-for-nine sweep is asking for trust. An open model that reports the same sweep is making a bet that you can't refute it — and inviting you to try. Every criticism I have made above, someone with a few GPUs can now settle: run 0731 against a held-out agentic suite, ten times, with error bars, next to whatever frontier agent you like, and publish what falls out. The reason I can be this skeptical in public is the same reason my skepticism is cheap — the homework is downloadable. Most of the industry's leaderboard theatre happens behind an API precisely because it can't be.

So here is the verdict, with the error bars I wish the launch had carried. The 0731 build is probably a real improvement; retraining post-training to lift agentic performance is a legitimate mechanism, and DeepSeek was unusually specific about what it changed. But "beats its own flagship on all nine benchmarks" is, as published, an internal comparison on a self-selected suite with no variance reported and a floating model name — which is significant, in the sense of worth noticing, without yet being impressive, in the sense of proven. The good news is that DeepSeek, almost alone among the labs, shipped the one thing that lets us tell the difference. Someone should take them up on it. If the chart only goes up, look harder — and this time, you actually can.