An LLM judge evaluates outputs at scale, and human experts should label only where it is least sure. The judge’s natural escalation signal, the spread of its predicted label distribution, conflates two uncertainties: aleatoric, real disagreement in the expert pool, which more labels cannot reduce, and epistemic, the judge’s ignorance of what the experts would say, which even one label reduces. A small Bayesian model separates them: a regression on the labels already collected learns how far to trust a black-box judge’s prediction on each item, and both components then follow as closed-form expressions, with no sampling and no further judge calls.

On a real LLM judge scored against exactly known truth (500 three-sided dice), the two estimated components isolate: estimated aleatoric tracks the true disagreement and not the judge’s error, estimated epistemic tracks the judge’s error and not the disagreement. The judge’s own stated confidence is no guide to its actual error. On real human disagreement (ChaosNLI, 100 annotations per item), ranking by the epistemic estimate removes 83% more error than ranking by total uncertainty for the same expert labels, though simply escalating the least-labelled items does as well there.