Quick answer: Use LLM as a judge when exact-match checks cannot grade open-ended answers, but give it a written rubric, a rationale-before-score format and a check against human ratings before you trust its numbers.
If the outputs you are grading come from a model you trained yourself, the wider workflow is covered in the guide on how to fine tune an llm.
Choosing between pairwise comparison and direct scoring
The first decision is the shape of the verdict. A judge can either compare two outputs or grade one output on its own, and Evidently AI’s guide notes that judges handle both. Pick pairwise comparison when your question is “which version is better?” and direct scoring when you need a label or number for each output.
| Approach | What the judge returns | Planning use |
|---|---|---|
| Pairwise comparison | The better of two responses to the same input | Suits head-to-head tests of two prompts or two model versions, where relative preference matters more than an absolute grade. |
| Direct scoring, numeric | A number for continuous judgments such as helpfulness from 0 to 1 | Suits tracking one quality dimension across many outputs, as long as the scale is defined in the rubric. |
| Direct scoring, categorical | An explicit label such as correct, partially_correct or incorrect | Suits pass/fail style reporting, where each label needs a plain definition a reviewer would agree with. |
Pairwise comparison is described by Evidently AI as giving the model two responses and asking it to choose the better one, while the numeric and categorical score types come from Langfuse’s documentation.
Choose pairwise when you are deciding between two candidates and do not need a standalone grade. Choose direct scoring when you want one output judged against a fixed standard. Pairwise results also give you a natural bias test, covered below.
How an LLM judge works
An LLM judge is a model that reads an output, applies criteria you wrote and returns a verdict. According to MLflow’s documentation, the judge produces a score and a written justification. The approach emerged as LLMs began to be used as evaluators for complex tasks, as the survey on LLM-as-a-Judge describes.
The reason to use one is that many correct answers differ in wording, so matching against a reference answer fails; Evidently AI points out there are many ways to be right without matching the example answer. Human review catches these issues but, in MLflow’s words, can only cover a limited number of outputs per day. A judge fills that gap.
The method was introduced in the MT-Bench and Chatbot Arena paper, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, as the Hugging Face cookbook recounts. Any model can fill the role: MLflow lists OpenAI, Claude, Gemini and open source models. Treat it as an approximation of human labelling. Evidently AI is explicit that it is not an evaluation metric in the same sense as accuracy, and that success depends on implementation details such as the model, the prompt design and the task complexity.
Writing the judge prompt
A good judge prompt names the task, defines the criteria in a rubric, supplies the material to judge and fixes the output format. Langfuse structures its judge prompt around a rubric, the input, the output and an optional reference answer holding ground truth or the expected output.
A reusable prompt skeleton
Build the prompt from these parts, in this order:
- Task statement. Start with a plain instruction, in the style of Evidently AI’s helpfulness example: Given a QUESTION and RESPONSE, evaluate if the response is helpful.
- Rubric. Define each level. Langfuse’s example scores a factual-accuracy criterion as Score 1 if the answer is factually incorrect, 5 if fully accurate and well-sourced.
- Inputs. Insert the question, the response and, when you have one, the reference answer.
- Reasoning slot. Ask for an evaluation field before the final rating. The Hugging Face cookbook recommends leaving the model more time for thought by adding an Evaluation field before the final answer.
- Constrained verdict. Limit the answer to a fixed set. Evidently AI’s example asks for a binary helpful or unhelpful label, and MLflow shows a judge whose feedback values are limited to accurate or inaccurate.
Scale and rubric design
Keep the scale small. The Hugging Face cookbook advises a small integer scale like 1-4 or 1-5 rather than a large float scale. A five-level rubric works only when each level has a distinct written definition; a vague middle level is where judges and reviewers drift apart.
Use numeric scores when the quality is continuous, and categorical labels when you want explicit buckets. If you cannot write down in a sentence what separates two adjacent levels, merge them.
Biases to test before trusting a judge
Judges are fallible. The survey of LLM-as-a-Judge calls reliability a significant challenge that requires careful design and standardization, and lists improving consistency and mitigating biases among its strategies. It also proposes methods for evaluating judge reliability, supported by a benchmark.
You do not need to wait for a perfect theory of judge bias to test your own setup. Two cheap checks cover the most obvious risks:
- Position swap. Run every pairwise comparison twice with the two responses in opposite order. If the winner changes with the order, the verdict reflects placement rather than quality, and those cases should be counted as ties or sent to a human.
- Length probe. Compare the judge’s preferences against response length on your own data. If the longer answer wins regardless of content, tighten the rubric to reward accuracy and concision explicitly.
- Rubric sensitivity. Reword the criteria without changing their meaning and re-run a sample. Large shifts in results point to a prompt problem, not a quality difference.
Log the swapped results next to the originals in a spreadsheet, with a column for agreement between the two orders and the win rate for each candidate. A pairwise setup makes this audit easy; direct scoring needs the length and rewording probes instead.
Validating the judge against human ratings
Before relying on a judge, score a labelled sample with it and compare against reviewers. Langfuse puts it simply: check that it scores cases the way your reviewers would before you rely on it. Calibrate against labelled examples or human scores, then keep a human-rated subset for periodic re-checks.
Interpret agreement against a human baseline, not against perfection. In the Hugging Face cookbook example, two human raters had a Pearson correlation of 0.563, which the authors call not that good, while the initial LLM judge reached 0.567 with the human raters. A judge that matches human-to-human agreement on your sample is behaving like a reviewer; one far below it needs a better prompt or a different model.
Work through low agreement in this order: read the judge’s written rationales on the disagreements, find which rubric level they hinge on, rewrite that definition, and re-run the same sample. Keep the sample fixed so changes in agreement come from the prompt and not the data.
Running the judge across a dataset
Once the prompt is validated, automate it: loop over your outputs, send each one with the rubric to the judge model, parse the constrained verdict and store the score with its justification. Keep the parsed label and the rationale as separate fields so you can audit disagreements later.
Use the judge for regression checks and comparisons between candidates, such as a model tuned with the preference methods in dpo vs rlhf. For standardised test sets that sit alongside a judge, llm benchmarks explained lists the common options.
Troubleshooting a judge that disagrees with you
- Verdicts flip when positions swap. Treat those cases as ties, and consider direct scoring against a reference for that task.
- Everything gets a high score. Add concrete failure descriptions to the lower rubric levels and shrink the scale toward the small integer range.
- Scores look arbitrary. Require the evaluation text before the rating, so the number follows stated reasoning.
- Agreement stays low. Compare it with how well your human reviewers agree with each other. If they disagree as much, the rubric is the problem, not the judge.
- Results differ by model. Judge quality depends on the model, prompt design and task complexity, so re-validate whenever you change the judge model.
Frequently asked questions
Does the judge have to be a different model from the one being tested?
Any model can act as a judge, including OpenAI, Claude, Gemini and open source models. Whichever you pick, validate it against human ratings on your own task before treating its scores as reliable.
Is an LLM judge a metric like accuracy?
No. It is a technique that approximates human labelling, so its output depends on the model, the prompt and the task, and it needs checking rather than assuming.
Why not just compare answers to a reference?
Many valid answers do not match the reference wording. A judge reads meaning against your rubric, though an optional reference answer can still be supplied as ground truth.
Do I still need human reviewers?
Yes, for the labelled sample used to calibrate the judge and for periodic re-checks. Humans cannot review every output, which is exactly why the judge handles volume.
What format should the judge’s answer take?
Ask for a short rationale first, then a constrained verdict such as a small integer scale, a fixed label set, or a choice between two responses.
