Introduction
An LLM-as-a-judge pipeline treats a language model as an evaluator: it reads candidate outputs and scores them against a written rubric instead of producing free-form answers for end users. This approach can scale review work, catch regressions between releases, and flag outputs that need human attention. It is not a replacement for human judgment. Like any automated check, a judge pipeline can be miscalibrated, biased toward confident-sounding text, or inconsistent across runs. This guide develops an original, practical evaluation pipeline around three practices: a golden set of labeled examples, an evaluator-optimizer loop, and inter-rater agreement checks using Cohen's kappa. The workflow is a design exercise, not a measured benchmark. Treat all example numbers as hypothetical.
Write a Scoring Rubric First
Before calling any model as a judge, define what good output looks like for your task. A useful rubric names each score level, lists the evidence required for that level, and gives at least one positive and one negative example. For a customer-support reply, a three-point rubric might look like this:
- 2 points: the answer addresses the user's question, cites only supplied facts, and makes no unsupported claims.
- 1 point: the answer addresses part of the question but includes an unsupported claim or omits a required detail.
- 0 points: the answer does not address the question, contradicts the supplied facts, or refuses without reason.
The rubric should also state how to handle ambiguity. If a response mixes correct and incorrect statements, does it score as partially correct or incorrect? Decide before evaluation begins, not while scoring individual cases. A rubric that changes during scoring cannot produce comparable results.
Build a Golden Set
A golden set is a small, labeled collection of inputs paired with reference judgments. It serves as the baseline against which judge behavior is measured. Start with twenty to fifty cases that cover ordinary inputs, edge cases, ambiguous inputs, and inputs with no acceptable answer. For each case, record the input, the candidate output being judged, the reference score, and a short justification.
The justification matters as much as the score. If two reviewers disagree on whether a response deserves one point or two, the written justification reveals whether the disagreement is about the rubric or about the specific output. Review the golden set with at least two people before using it to evaluate a judge. Resolve disagreements by clarifying the rubric, not by averaging scores.
Do not treat the golden set as a training dataset for the judge. It is an evaluation artifact. If you use the same cases to tune the judge's prompt and to measure its performance, the measured improvement will be inflated. Keep a separate held-out set for final assessment.
Structure the Judge Prompt
Give the judge model a clear role, the rubric, the input, the candidate output, and an explicit output format. Request a score followed by a brief justification. Avoid asking the judge to "decide if the answer is good" without criteria, because "good" is too vague to score consistently. A usable prompt template might look like this:
You are an evaluation judge. Score the candidate reply below using this rubric:
2: addresses the question, cites only supplied facts, no unsupported claims.
1: addresses part of the question but has an unsupported claim or omission.
0: does not address the question, contradicts supplied facts, or refuses without reason.
Return only:
Score: <0, 1, or 2>
Justification: <one sentence>
Input: {{user_question}}
Supplied facts: {{facts}}
Candidate reply: {{candidate_reply}}
Use placeholders for variable parts so the same template can be reused across cases. Keep the prompt concise. Long, wordy instructions increase the chance that the judge ignores part of the rubric.
Use an Evaluator-Optimizer Loop
Anthropic's guidance on building effective agents describes an evaluator-optimizer pattern: one LLM call generates a response while another provides evaluation and feedback in a loop. This is recommended when clear evaluation criteria exist and when iterative refinement provides measurable value. In a judge pipeline, the generator produces candidate outputs and the judge scores them. When the judge flags a low-scoring output, feed the judge's justification back to the generator as a revision instruction and produce a new candidate. Repeat until the score meets a threshold or a maximum number of attempts is reached.
This loop is a bounded refinement step, not an open-ended autonomous process. Set a maximum attempt count and stop when it is reached. If the judge keeps flagging the same issue, the problem may be in the rubric, the input data, or the generator's capabilities rather than in a fixable formatting mistake.
Measure Agreement with Cohen's Kappa
Cohen's kappa measures agreement between two raters beyond what would be expected by chance. A kappa of 1 means perfect agreement; a kappa of 0 means agreement no better than random guessing; negative values mean disagreement worse than chance. To apply this to a judge pipeline, have two independent judges (human or model) score the same golden-set cases, then compute kappa from their score tables.
If the two judges are both language models, run them with different prompts or different temperature settings so their scores are not trivially identical. If one judge is a person and one is a model, the comparison shows where the model diverges from human judgment. Low kappa means the rubric is ambiguous, the cases are genuinely borderline, or the judge model is inconsistent. Do not publish a judge pipeline whose scores disagree with human judgment more often than they agree, without documenting the discrepancy and its impact.
Separate Retrieval from Judgment
If your pipeline judges answers that depend on retrieved documents, evaluate retrieval and judgment separately. First give the judge the correct supporting documents and measure whether it scores good answers highly and bad answers poorly. This isolates the judge's ceiling performance. Then run the full pipeline, including retrieval, and compare. If scores drop when retrieval is added, the problem may be missing or noisy context rather than the judge itself.
This separation prevents a common error: blaming the judge when the real failure is that the right evidence was never retrieved. Record both stages separately so that improvements can be targeted.
Record Every Score
Store the judge model identifier, prompt version, input hash, candidate output, score, justification, and timestamp for every scored case. A score without its context cannot be audited later. When a score changes between runs, the record should make it possible to tell whether the change came from a different judge model, a revised prompt, a new golden set, or a different input.
Keep the record separate from the application's normal logs if the inputs contain sensitive information. An evaluation log can be smaller and more restricted than a production log while still preserving enough detail to reproduce a result.
Define Stop Conditions
Decide in advance what makes a judge pipeline ready for use. Possible conditions include: kappa above a chosen threshold on the held-out set, no unexplained score changes across three consecutive runs, and a human reviewer approving a sample of judged outputs. Also define what triggers reassessment: a new judge model, a changed rubric, a shift in input distribution, or a reported scoring error. Without stop conditions, tuning a judge becomes an open-ended task with no defined completion point.
Limitations
An LLM judge can favor longer answers, agree with its own previous outputs, or miss subtle errors that a human would catch. It does not understand policy the way a person does, and it can reproduce common mistakes from its training data. A high kappa between two models does not mean both models are correct; it only means they agree. Treat judge scores as signals to investigate, not as final verdicts. Human review remains essential for high-stakes decisions.
Conclusion
A practical LLM-as-a-judge pipeline starts with a written rubric, a labeled golden set, and a structured judge prompt. It uses an evaluator-optimizer loop for bounded refinement, measures agreement with Cohen's kappa, and separates retrieval quality from judgment quality. Record every score with its context, define stop conditions, and reassess when the model, rubric, or inputs change. These practices make judge behavior inspectable, correctable, and safe to operate.
Sources
- Anthropic: Building effective agents, for the evaluator-optimizer workflow distinction.
- Ragas: Available metrics, for faithfulness, context precision, context recall, and response relevancy as LLM-based evaluation metrics.
For related planning guidance, use the workflow design guide to decide where evaluation fits in your system.