Generalization is a model's ability to perform well on new, unseen data, not just the data it was trained on.
A model that generalizes well has learned the underlying patterns in its training data rather than memorizing them. The practical test of any model isn't how it performs on training data, but whether the learned patterns hold up when the model encounters real-world inputs it has never seen before.
These are the two main failure modes that undermine generalization.
- Overfitting occurs when a model is too closely tailored to its training data, picking up noise and random fluctuations as if they were meaningful patterns.
- Underfitting sits at the other extreme, where the model is too simple to capture the actual structure in the data and performs poorly on both training and new data.
Techniques that can improve a model's ability to generalize include regularization, which penalizes overly complex models to prevent them from fitting too closely to training data, and cross-validation, which tests the model on multiple subsets of data to give a more reliable picture of how it will perform on data it hasn't seen before.