LLM-as-judge: useful, biased, calibratable

LLM-as-judge is the most cost-effective way to score open-ended generations at scale. It is also the most common source of false confidence in AI QA. Treat your judge like an unreliable contractor — measure it, calibrate it, constrain it.