Definition
What is Overfitting?
Overfitting is Overfitting is when a model learns its training examples too closely, including their noise, so it scores well on data it has seen and poorly on new data.
Overfitting happens when a model memorizes its training data instead of learning the pattern underneath. It looks excellent on the examples it studied and falls apart on anything new.
An everyday analogy
Imagine a student who memorizes last year's exam answers word for word. They ace a repeat of that exam. Give them new questions on the same subject and they struggle, because they learned the answers, not the material. A model that overfits is that student.
How it happens
Models learn by adjusting their parameters to reduce error on training data. Given enough parameters and enough passes over the data, a model can fit even random quirks and mistakes in the examples. Overfitting is most likely when:
- The dataset is small compared with the model.
- The model trains for too many rounds on the same examples.
- The data has many near-duplicates.
- Fine-tuning uses a narrow set of examples, so the model gets very good at that slice and worse elsewhere.
How people spot it
The standard check is to hold back data the model never trains on. If accuracy on training data keeps climbing while accuracy on the held-out data stalls or drops, the model is overfitting. This is why benchmarks and evals use separate test sets.
How it is reduced
- Collect more and more varied data, or add synthetic data.
- Stop training when held-out performance stops improving.
- Use a smaller model or add regularization, which penalizes overly complex solutions.
- Keep test data strictly separate.
A cousin to watch
Overfitting is about the model memorizing the training set. A related problem, benchmark contamination, is when test questions leak into the training data, so a high score reflects memory, not skill. Both make a model look better than it is.
What it means when you use AI
If a model aces a published test but stumbles on your own tasks, overfitting or contamination is one possible reason. The practical fix is to test on your own realistic examples, which the model could not have seen. And if you fine-tune a model yourself, keep a held-out set and compare against the original model before you trust the results.
The goal of training is not to remember. It is to generalize.
Questions people ask
What is overfitting in machine learning?
It is when a model fits its training data so closely, noise included, that it performs worse on new data it has not seen.