Why nobody trusts your LLM judge

Ask a team using LLM-as-judge how much they trust the scores, and the honest answer is usually somewhere between "directionally" and "not really". They keep running it anyway, because the alternative is reading transcripts by hand.

That distrust is not vague unease. The failure modes are specific, reproducible, and documented in the literature.

The five ways a judge is wrong

Position bias. Present the same two responses in the opposite order and the judge changes its winner. If your eval harness always puts the new model second, you have a systematic thumb on the scale and no way to see it.

Verbosity bias. Longer answers score higher, largely independent of quality. Any judge that has not been tested for this will reward a model that learned to pad.

Self-enhancement bias. A model rates its own family's output more favourably. Using GPT to judge GPT, or Claude to judge Claude, embeds a preference you did not ask for.

Prompt sensitivity. Rephrasing the rubric — not changing its meaning, just its wording — moves scores. Which means the number partly measures your prompt engineering rather than your system.

Shared blindness. A judge cannot catch an error it would make itself. If your generator and your judge misunderstand the same domain concept, the eval reports success with total confidence. This one is the most dangerous, because it fails silently and in exactly the cases you most need to catch.

Layered on top: the statistical problem. Most eval suites report a mean over a few hundred cases with no interval, then teams make ship decisions on a 2% difference that the sample size cannot support. The consensus in the literature is blunt — a large fraction of reported LLM evaluations are statistically invalid.

Why "more metrics" is the wrong response

The industry's answer has been breadth. One tool ships forty-plus metrics, another fifty. Faithfulness, answer relevance, context precision, toxicity, and so on.

Breadth does not address any of the five failure modes above. Forty uncalibrated metrics are forty numbers of unknown reliability rather than five. In practice breadth makes the problem worse, because a long dashboard of green scores feels like evidence.

The question a team actually needs answered is not "how many metrics do you have" but "how do I know this metric is right about my data".

What calibration means concretely

Calibration is a measurement discipline, not a feature checkbox. It means four things, and they are all measurable:

Agreement with humans, tracked continuously. Pick a sample, have people label it, and report the judge's agreement with those labels as a live number — not once during setup, but continuously, because agreement drifts as your traffic changes.

Confidence intervals on every score. If the interval on your faithfulness score is ±6 points, a 2-point regression is not a regression. Reporting the interval is what stops eval theatre.

Bias diagnostics by name. Test for position bias by running pairs in both orders and measuring flip rate. Test for verbosity bias by regressing score on length. Test for self-preference by swapping judge families. These are cheap and almost nobody runs them.

An explicit unreliability signal. The most useful output a judge can produce is sometimes "I am not reliable for this class of input." A system that always returns a confident number is hiding information you need.

The uncomfortable part

Calibration costs something. It requires human labels — not many, but some, and continuously rather than once. Teams who want evaluation to be free will not like that, and vendors who want evaluation to look free have an incentive not to mention it.

We think that trade is obviously worth making, because the alternative is making release decisions on numbers whose error bars you have never measured. But it should be stated plainly rather than buried: trustworthy evaluation is not a model you can drop in. It is a loop involving people.

---

This is the product we are building toward, and it is worth being direct about status: it is in development, not shipping. What ships today is the observability layer underneath it — traces, accurate cost, and replay. Evaluation you cannot trust is worse than no evaluation, so we would rather ship it late than ship it uncalibrated.