A reward model is a scoring model. You give it a prompt and a response, and it returns a number that says how good the response is. That number is used to steer another model toward better behavior during training.
Why it exists
Some qualities are easy to write down as rules, like "the code passes its tests." Others are not: helpful, clear, polite, safe. You cannot write a formula for them. So builders collect human judgments instead. People are shown two answers to the same prompt and pick the better one. A reward model is trained on thousands of such comparisons until it can predict which answer a person would prefer.
Where it fits in training
The reward model is a key piece of RLHF, reinforcement learning from human feedback. The steps look like this:
- Start with a base model that has been through pretraining.
- Collect human comparisons between pairs of answers.
- Train the reward model to predict those preferences.
- Use reinforcement learning to adjust the main model so its answers earn higher scores from the reward model.
This falls under post-training, the stage that turns a raw text predictor into an assistant.
Reward models for reasoning
Some reward models grade only the final answer. Others grade each step of a solution, which gives a denser signal for math and code. For tasks with a checkable answer, builders often skip a learned model and use an automatic check instead, such as running unit tests.
The weak spot
A reward model is a stand-in for what people want, not the real thing. If the main model finds quirks the scorer likes, such as longer answers or confident tone, it can chase the score instead of the goal. That failure is called reward hacking, and it is one reason models can become overly agreeable, a problem known as sycophancy.
Why you should care
When a model feels polite, cautious or eager to please, the reward model it was trained against helped shape that. It also explains why two models with similar knowledge can behave very differently: they were rewarded for different things.
A related idea is LLM-as-a-judge, where a general model grades outputs at evaluation time. A reward model is built specifically to supply the training signal.