The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scale | by Netflix Technology Blog | Sep, 2026 | Netflix TechBlog
Learn about Netflix’s world class engineering efforts, company culture, product developments and more. Follow publication Top highlight 1 621 11 5 Listen Share By Emma Yanyang Kong, JJ Tan, Ishan Gupta, Lars Olds, Claire Campbell, David Fagnan, Ratna Kavuri, Rohan Gosain, Veli Balin, Minsu Jang, Louis Garcia Any team generating text with an LLM at scale hits the same problem. You cannot evaluate everything manually. You don't know which outputs are great, which are mediocre, and which are harmful, because you simply cannot read them all. The standard solution is LLM-as-a-Judge, a second model that scores the first model’s output as a proxy for human raters. It is fast, cheap, and easy to deploy. However, it introduces a new and critical question: Who evaluates the evaluator? Furthermore, if the judge drifts, quietly degrading and starting to approve bad output, nothing downstream will flag it. This post is about how we solve that problem, by building, aligning, and continuously monitor