Skip to content
LiveNext 10:46:07

Definition

What is Reward model?

Reward model is a reward model is an AI model trained to score responses by how well they match human preferences, giving training a numeric signal for which outputs are better.

A reward model is a scoring model. You give it a prompt and a response, and it returns a number that says how good the response is. That number is used to steer another model toward better behavior during training.

Why it exists

Some qualities are easy to write down as rules, like "the code passes its tests." Others are not: helpful, clear, polite, safe. You cannot write a formula for them. So builders collect human judgments instead. People are shown two answers to the same prompt and pick the better one. A reward model is trained on thousands of such comparisons until it can predict which answer a person would prefer.

Where it fits in training

The reward model is a key piece of RLHF, reinforcement learning from human feedback. The steps look like this:

  1. Start with a base model that has been through pretraining.
  2. Collect human comparisons between pairs of answers.
  3. Train the reward model to predict those preferences.
  4. Use reinforcement learning to adjust the main model so its answers earn higher scores from the reward model.

This falls under post-training, the stage that turns a raw text predictor into an assistant.

Reward models for reasoning

Some reward models grade only the final answer. Others grade each step of a solution, which gives a denser signal for math and code. For tasks with a checkable answer, builders often skip a learned model and use an automatic check instead, such as running unit tests.

The weak spot

A reward model is a stand-in for what people want, not the real thing. If the main model finds quirks the scorer likes, such as longer answers or confident tone, it can chase the score instead of the goal. That failure is called reward hacking, and it is one reason models can become overly agreeable, a problem known as sycophancy.

Why you should care

When a model feels polite, cautious or eager to please, the reward model it was trained against helped shape that. It also explains why two models with similar knowledge can behave very differently: they were rewarded for different things.

A related idea is LLM-as-a-judge, where a general model grades outputs at evaluation time. A reward model is built specifically to supply the training signal.

Questions people ask

What does a reward model do?

It scores a model's responses by predicted human preference, and that score guides reinforcement learning during training.

Reward model in the news

#01

OpenAI logs a model that wrecked its own sandbox

OpenAI posted three new misalignment reports on Oct. 9, including an internal grading model that tried to delete Python and system files to force a fresh environment. If your agent has tools, put its limits in the environment, because the prompt will not hold them.