Definition
What is LLM-as-a-judge?
LLM-as-a-judge is LLM-as-a-judge is the practice of using one AI model to grade the outputs of another against a rubric, so teams can score thousands of answers without a person reading each one.
What it is
LLM-as-a-judge means you hand a language model a piece of work and a grading guide, and ask it to score the work. The work might be a chatbot reply, a summary, a code change or an agent's finished task. The guide might be a yes/no question ("Does this answer cite its source?") or a rubric with levels from one to five.
Think of a teacher's assistant who grades the first pass of every essay. A human still writes the grading guide and spot-checks the pile, but the assistant does the bulk of the reading.
How it works
A typical setup has three parts:
- The input and the output. What the system was asked, and what it answered.
- The rubric. Plain-language criteria that decide each score. Good rubrics are specific enough that two different people, or two different runs of the judge, would land on the same answer.
- The verdict. A score, a label or a pass/fail, often with a short reason.
Teams use this to run evals at a scale people cannot match. Instead of reading 5,000 test replies after every change, they let a judge model grade all of them and read only the ones it flags.
Where it goes wrong
A judge is itself a model, so it makes mistakes. Researchers have documented several recurring biases:
- Position bias. When comparing two answers, a judge can favor whichever one it sees first.
- Length bias. Longer answers can score higher even when they are not better.
- Self-preference. A model can rate text written by itself or a close relative more kindly.
- Overconfidence. A judge can state a score firmly when the case is genuinely unclear, which is a model calibration problem.
How to use it well
- Write the rubric first, and test it on examples where you already know the right answer.
- Keep a sample of human-graded cases and check how often the judge agrees with the people.
- Swap the order of answers when comparing two, and average the results.
- Use a different model as the judge than the one being graded when you can.
- Prefer narrow questions over "rate this from 1 to 10."
How it relates to other terms
LLM-as-a-judge is one way to run an AI benchmark or a private eval. It is also a close cousin of RLHF, where human preferences train a model, and of reward models that stand in for those preferences. Judges can also be fooled in the same way any model can, including by prompt injection hidden in the text being graded.
The short version
A judge model is a cheap, fast first reader, not the final authority. Treat its scores as measurements with error bars, check them against people, and you get a testing loop that scales. Skip the checking and you are measuring the judge's habits instead of your product.
Questions people ask
What is LLM-as-a-judge?
It is using one AI model to score another model's outputs against a rubric, so large test sets can be graded without a person reading every answer.
Is LLM-as-a-judge reliable?
It can be, but only when checked. Judges show known biases such as favoring longer answers or the first answer shown, so teams compare them against human-graded samples.