WRITING / AI STRATEGY
Your Judge Needs an Eval Too: Validating LLM-as-Judge in Production
An LLM judge is a measurement instrument, so measure it first: pairwise labels, kappa against human raters and a version pinned in telemetry.
An LLM-as-judge is not a measurement instrument you can assume correct—it's code with its own failure modes, and it needs its own eval before you trust a single dashboard it produces.
Published January 24, 2026. Reading time: 9 min read. Author: Morgan Wilson.