What is Data poisoning?

Data poisoning attacks an AI model before it is ever used. Instead of tricking a finished model with a clever prompt, the attacker corrupts the examples it learns from.

How it works

A model learns patterns from its training data. If someone adds bad examples to that data, the model learns the bad patterns too. Poisoning can happen at several stages: during pretraining on web-scale text, during fine-tuning on a smaller dataset, or in data that an app collects from users and feeds back into training.

There are two broad goals:

• Degrade the model: add noise or wrong labels so it performs worse.
• Plant a backdoor: teach the model to behave normally except when it sees a secret trigger, such as a rare phrase. The model passes ordinary tests, then misbehaves on cue.

Why it is hard to catch

Poisoned examples can look ordinary. Scanning rows for obvious harm will miss data that is subtle, spread thin or disguised as fiction. Related research on emergent misalignment shows that data which looks harmless can still change how a model behaves broadly. A backdoor is also invisible until the trigger appears, so normal evals may never touch it.

Models trained on huge amounts of scraped text are exposed because nobody can read all of it. Anyone who can publish to the open web has some ability to influence what gets scraped.

How it differs from prompt injection

Prompt injection hides instructions in text a model reads while it is running. Data poisoning changes the model itself, at training time. One is a trick played on a finished system. The other is tampering with how the system was made. It is closer in spirit to a supply chain attack.

Defenses

• Know where your data comes from and prefer sources you can trust.
• Filter and deduplicate data, and sample it for review.
• Test the trained model's behavior, including on odd inputs and far from the training task.
• Use red teaming to hunt for hidden triggers.
• Treat vendor or generated datasets, including synthetic data, with the same suspicion as downloaded code.

An analogy

Poisoning a model is like tampering with a textbook before a class studies from it. Every student learns the error, and none of them know it is there.