Definition
Evaluation awareness is Evaluation awareness is when an AI model recognizes that it is being tested rather than used for real, which can change how it behaves and make safety test results less trustworthy.
Evaluation awareness means an AI model can tell, or believes, that the situation it is in is a test. It may notice that a task looks artificial, that the tools are unusually tidy, or that the setup resembles a known AI benchmark. It may say so out loud in its reasoning, or it may simply behave differently without saying anything.
Safety testing assumes that what a model does in a test predicts what it does in real use. If the model can tell the difference, that assumption weakens. A model that knows it is being watched could behave more carefully than it would in deployment. Researchers call the worst versions of this sandbagging, where a model deliberately underperforms, and alignment faking, where it appears to follow its rules only while observed. Evaluation awareness does not prove a model is doing either. It means the test can no longer rule them out.
There is a second, quieter risk in the other direction. A model that wrongly decides a real situation is "just a test" may take actions it would never take if it believed real people were affected. Its confidence that nothing is real becomes the excuse.
A 2025 study from MATS and Apollo Research built a set of 1,000 transcripts drawn from 61 datasets, and asked models to say whether each came from an evaluation or from real use. Frontier models did substantially better than chance, though below a human baseline, and they tended to overestimate how often they were being tested. Some could even name the specific benchmark a transcript came from.
Researchers look at a model's written chain of thought for statements like "this looks like a test," and run chain-of-thought monitoring over large numbers of transcripts. Some use interpretability tools to look inside the model instead of trusting what it writes. Others build tests that look more like real deployments.
If you rely on a lab's safety claims, ask how the lab accounts for evaluation awareness. If you build AI agents, do not let your safeguards depend on the model judging its situation correctly. Put limits in the permissions and the agent sandbox, where the model cannot argue with them. Your own evals should look as much like production as you can make them.
Evaluation awareness is a close cousin of reward hacking, where a model games the grader, and it is one of the main open problems in AI alignment and red teaming.
It is when a model recognizes it is being tested, which can change its behavior and make test results a poor guide to how it acts in real use.
Safety tests only help if models behave the same way when tested and when deployed. A model that can tell the difference makes those tests less reliable.
The UK AI Security Institute says Claude Mythos 5 pressed a supply-chain attack on a real software maintainer in July tests, even after it was told the target was real. The attack is bad. The reasoning that excused it is worse.