Model as Verifier: The New Scaling Axis for LLMs
What LLM-as-a-Verifier is, how it differs from model as judge, and what best-of-N really costs — the technique that took DeepSeek V4 Flash from 79% to 88%.
A few weeks ago a result that looks like a measurement error went around X: DeepSeek V4 Flash, a cheap open model, jumping from 79% to 88% on Terminal-Bench without any retraining. Just by generating five solutions and verifying itself. It was posted by Jacky Kwok[4], first author of the paper behind the technique[1], and what matters for us is that the technique sits on top of the model-as-judge pattern many of us already have in production, changing only how you read its response.
The idea fits in one sentence. Stop asking the judge for a number and keep the whole distribution instead.
The tie: why a 1-to-5 judge falls short
A judge that scores from 1 to 5 ties almost every time the candidates are reasonable. If you already use the model-as-judge pattern you know what I mean: you hand it two long solutions, both compile, both do what the prompt asked, and the judge answers “4” for both. That leaves you nothing to choose between.
The authors measured it. On the same task, a 1-to-5 judge returned a tie in 88 out of 100 evaluations; with a continuous score and 20 levels of granularity, the ranking was correct in 77 out of 100[1]. The model did distinguish between them; what failed was asking it to cram that distinction into a single token.
Because the model does know more than what it writes down.
How do you get a continuous score out of logprobs?
When a model generates a token it isn’t choosing a word: it computes a probability for every possible token in the vocabulary and then samples one. If you ask it to score from 1 to 20, internally it spreads probability across “13”, “14”, “15”, and the rest, then writes down one of them. Logprobs are that spread, on a logarithmic scale, and several APIs return it to you if you ask for it.
Instead of reading the token that came out, you read the entire distribution over the scoring tokens and compute its expected value: R = Σ p(v_g) · φ(v_g), where φ normalizes each level to the [0, 1] interval. A judge torn between 13 and 14 no longer gives you a flat 13 — it gives you 0.672.
# From the scoring-token distribution to a continuous number between 0 and 1
import math
def puntuacion_continua(top_logprobs, granularidad=20):
# top_logprobs: [{"token": "13", "logprob": -0.21}, ...] at the score position
esperanza, masa = 0.0, 0.0
for cand in top_logprobs:
token = cand["token"].strip()
if not token.isdigit():
continue # skip anything that isn't a score
p = math.exp(cand["logprob"]) # logprob -> probability
esperanza += p * (int(token) / granularidad) # phi(): normalizes to [0, 1]
masa += p
if masa == 0:
return None # no score among the candidates: happens with tokenizers that split numbers
return esperanza / masa # renormalize over the valid tokensThe full formula in the paper is this same sum nested three times: R(x, τ) = (1/CK) Σ_c Σ_k Σ_g p·φ. The innermost sum runs over the G levels of the scale, the middle one over the K repetitions of the evaluation, and the outer one over the C criteria you’ve defined. Each of those three sums is a lever you can turn up or down, and that’s the part of the paper that actually changes how you build an evaluator.
For pairwise comparisons the framework does the same thing with Bradley-Terry: the probability that A beats B is the sigmoid of the difference between their continuous scores[1].
The three axes verification scales along
Each sum in the formula is an independent scaling axis, and the paper measures how much each one delivers on Terminal-Bench 2.0[1].
Granularity. Widening the scale from 1 to 20 levels raises the result from 73.1% to 77.5%, and the starting point of that ablation is G=1, not the 1-to-5 judge from the case above. It’s the cheapest change of the three: it doesn’t add a single call, you just change the criterion’s wording and the range you expect.
Repetition. Evaluating K times and averaging is plain Monte Carlo over the score: variance falls as O(1/K). From K=1 to K=16 the result goes from 74.7% to 77.4%, and then it flattens out. It doesn’t flatten out by accident. The judge’s errors are bias, not independent noise, and a bias repeated sixteen times is still the same bias.
Decomposed criteria. For code, instead of asking “is this good?”, the paper splits the evaluation into Specification, Output, and Errors. Each criterion on its own lands between 75.2% and 76.4%, but averaging all three gets you to 78.3%. It’s the only one of the three levers that attacks the bias instead of just averaging over it, because each criterion looks at a different part of the solution.
| Discrete judge (1-5) | Continuous verifier | |
|---|---|---|
| Output | the score token the model wrote | the expectation over the full distribution of score tokens |
| Ties | constant once candidates are reasonable | rare: the difference shows up in the third decimal |
| Scaling | widening the scale alone doesn’t fix the tie | granularity, repetitions, and criteria, each with its own measured curve |
| Ranking N candidates | O(N²) pairwise comparisons | O(Nk) with a probabilistic pivot tournament |
| API requirement | none | logprobs exposed by the provider |
A verifier that correctly ranks similar candidates is good for more than evaluating: it’s good for choosing.
Best-of-N: generate cheap, choose well
Best-of-N means generating N solutions to the same problem, scoring them with the verifier, and keeping the best one. The paper works with N between 3 and 5, and to avoid comparing every candidate against every other one, it uses a probabilistic pivot tournament that brings the cost down from O(N²) to O(Nk)[1].
The paper’s numbers, across the four suites it reports[1][2]: Terminal-Bench V2 86.5%, SWE-Bench Verified 78.2%, MedAgentBench 73.3%, and RoboRewardBench 87.4%. On Terminal-Bench V2 the context is what gives it scale: the same generator with no verification lands at 83.1% pass@1, GPT-5.5 reaches 84.7%, and an oracle that always picked the best candidate would score 92.1%.
That gap between 86.5% and 92.1% is the honest part of the result. The good candidates were already there; the verifier recovers a slice of that, not all of it. When it fails, it fails by picking a plausible-but-wrong solution, which is exactly the failure mode a judge has.
DeepSeek V4 Flash’s jump from 79% to 88% with five samples isn’t in the paper[4]. It’s a later experiment Kwok posted on X, on Terminal-Bench 2.1, with no review and no ablation table. Coming from the first author makes it credible, but it doesn’t make it a peer-reviewable result. I’d treat it as a strong signal that the technique generalizes to smaller models, and nothing more.
The fine print behind the “11x cheaper” claim
The 11x in the thread is price per token, not cost per solved task. DeepSeek lists V4 Flash at $0.14 per million input tokens and $0.28 for output[5], and compared with a frontier model’s price, the ratio checks out. The problem is the unit isn’t the same.
Do the math on what the pattern actually consumes. You generate N candidates, so you multiply the full generation by N. Then you verify each candidate with K repetitions and C criteria, and each of those calls eats the agent’s entire trajectory as input, which on Terminal-Bench means long trajectories. With N=5, K=4, and three criteria, the number of calls per task looks nothing like a direct run.
The framework mitigates this from both sides: the pivot tournament trims the comparisons, and version 0.2.0 adds prefix caching, which according to its changelog cuts uncached input tokens by about 3.4x on benchmarks with long trajectories, plus a token_usage() so you can measure it yourself[3]. Even so, the multiplier stays large, and the price per token is a number that moves: DeepSeek has announced a peak-hour surcharge structure that would change the whole arithmetic[5].
None of this makes it expensive in absolute terms. It’s cheap compared to using a frontier model for the same task, and it pays off whenever getting it right is worth more than the tokens: patches headed to production, migrations, anything where a human would have to review the failure if it goes wrong. For a high-volume endpoint answering online, it doesn’t pay off.
How to try it tomorrow
The framework is published under an MIT license and installs with pip install llm-verifier[3]. The selection function takes the problem, the candidates, and the criteria dictionary, and returns the winning index along with the continuous score for each candidate.
# Best-of-N with the framework from the paper: pip install llm-verifier
import llm_verifier
problem = "Write a function that reverses a string."
candidates = [
"def rev(s): return s[::-1]",
"def rev(s): return s",
"def rev(s): return ''.join(reversed(s))",
]
result = llm_verifier.select(
problem=problem,
candidates=candidates,
criteria={"Correctness": "Does the function actually reverse the string?"},
)
print(result.index) # index of the winning candidate
print(result.scores) # a continuous score per candidateThere are two more applications in the repository beyond best-of-N. With track() and ProgressTracker you score an agent’s trajectory step by step while it runs, and since trajectories that end well have a score that climbs with each step, you can abort the ones headed down the drain before paying for the rest. If you’ve already got how to evaluate AI agents in production set up, this slots in as the live signal an offline eval is missing. The other application is reinforcement learning: the continuous score works as a dense reward without training a separate reward model, with sample efficiency about 1.8x better on LIBERO[1].
And here’s the requirement that decides whether you can do this at all: the provider has to return logprobs. DeepSeek exposes them via logprobs and top_logprobs with up to 20 candidates per position, though its reasoning mode rejects them with an error, so the verifier has to run against the normal mode[6]. Gemini and Vertex give them via response_logprobs, and so does any OpenAI-compatible server, vLLM included. Anthropic’s API doesn’t expose logprobs, so Claude can’t be the verifier as-is; appendix B.6 of the paper describes a two-stage workaround for that case[1].
Common mistakes
Citing the 88% as if it were from the paper
It’s the easiest mistake to make and the most expensive to defend in front of your team. The paper reports 86.5% on Terminal-Bench V2 with Gemini 2.5 Flash as the verifier; DeepSeek V4 Flash’s 79%-to-88% is a later X thread, on a different version of the benchmark. They’re different numbers with different levels of rigor.
Budgeting off the price per token
If you drop the 11x into a spreadsheet without multiplying by N candidates and by K×C verifications, your estimate is going to fall short by a large factor. Measure it on a real sample of your tasks before promising any savings:
# problem and candidates: the same ones from the select() example above
llm_verifier.USAGE.reset()
llm_verifier.select(problem=problem, candidates=candidates, criteria={"Correctness": "..."})
print(llm_verifier.token_usage()) # calls, cached and uncached input tokens, output tokensSelf-verifying without accounting for self-preference bias
Having the generator and the verifier be the same model is exactly what makes the viral result attractive, and also what introduces the bias: a model scores its own solution style more favorably. Decomposing criteria dampens it, because asking “does this meet the spec?” leaves less room for aesthetic preference than asking “is this good?”. Dampening isn’t eliminating. If you’re going to use the same model on both sides, measure once against a different verifier and keep the gap as your margin of error.
Raising K expecting a proportional improvement
From K=1 to K=16 you gain 2.7 points, and you already have more than half of that at K=4: 74.7% with one evaluation, 76.1% with four, 77.4% with sixteen. Averaging reduces noise, not bias, and a judge’s errors are mostly bias.
Picking the provider before checking the logprobs
Finding out your API doesn’t return the distribution after you’ve already built the pipeline hurts. Before anything else, make a one-off call asking for logprobs and see if they come back, on the exact model and mode you’re going to use in production.
Implementation checklist
- The verifier’s API returns logprobs on the exact model and mode you’re going to use
- The scoring scale has enough granularity (20 levels, not 5)
- The criteria are decomposed and averaged, instead of a single “is this good?”
- You’ve measured pass@1 without verification as a baseline, so you know how much the verifier is really adding
- Cost is measured per solved task with
token_usage(), not estimated from the price per million tokens - Ranking N candidates uses the pivot tournament, not all-against-all
- If the generator and verifier are the same model, you’ve checked a sample against a different verifier
Sources
- LLM-as-a-Verifier: A General-Purpose Verification Framework — Kwok, Li, Atreya, Liu, Jiang, Finn, Pavone, Stoica, Mirhoseini (arXiv:2607.05391) — method, the continuous-score formula, ablations on granularity, repetitions, and criteria, best-of-N results, and appendix B.6.
- Project page — Stanford Scaling Intelligence Lab — confirmation of the results on Terminal-Bench V2, SWE-Bench Verified, MedAgentBench, and RoboRewardBench.
- llm-verifier framework on GitHub — MIT license,
select()/track()/ProgressTrackerAPI, supported providers, prefix caching, andtoken_usage()in v0.2.0. - Jacky Kwok’s thread on X — a later experiment beyond the paper: DeepSeek V4 Flash from 79% to 88% on Terminal-Bench 2.1 with five samples, and the “11x cheaper” claim.
- DeepSeek API pricing — $0.14 per million input tokens and $0.28 for output on V4 Flash, and the announced peak-hour surcharge.
- Chat Completions API — DeepSeek API Docs — the
logprobsandtop_logprobs(up to 20) parameters, and the error the reasoning mode returns if you request them.
Frequently Asked Questions
Can I use model as verifier with the Anthropic API?
Not as the verifier, at least not directly: Anthropic’s API doesn’t expose logprobs, and without the distribution over scoring tokens there’s no continuous score to compute. You have two ways out. Appendix B.6 of the paper describes a two-stage procedure for models that don’t give you logprobs, and the practical option is to split the roles: generate with Claude and verify with Gemini or DeepSeek, which do return them. That split also has the advantage that the generator and verifier are different models, which reduces self-preference bias.
Does this replace model as judge?
No. It’s the same pattern with the same prompt, reading the distribution instead of the token the judge wrote; for offline evaluation with a comparable history, the good old discrete judge still holds up.
How many candidates do I need for best-of-N?
The paper works with N between 3 and 5, and the author’s experiment with DeepSeek V4 Flash used five samples. Beyond that, cost grows linearly and improvement doesn’t, because the ceiling is set by the oracle: if the correct candidate isn’t among the ones you generated, no verifier is going to find it.
Does this work for monitoring agents in production?
Yes, and it’s the application I find most interesting of the three the framework ships. On trajectories that end well, the verifier’s score climbs as the agent moves forward, so you can score step by step with ProgressTracker and cut off a run that’s gone twenty steps without improving, instead of waiting for it to run out of budget. One caveat you can’t skip when taking this to production: the signal is validated on trajectories that end well, so calibrate the cutoff threshold against your own failed runs before you let it cut things off on its own. It also requires having your steps instrumented and accessible in real time, which is usually the real work.
Related articles
Model as Judge: Evaluate Your AI Agent's Responses
Learn how to use an AI model to automatically evaluate another agent's responses. Rubrics, evaluation types, and the most common pitfalls.
How to evaluate AI agents in production: metrics, evals, and observability
Technical guide to measuring the real reliability of AI agents in production: task completion rate, deterministic evals, LLM-as-judge, and silent degradation detection.