Overfitting vs Underfitting: The Two Numbers That Diagnose Your Model

But those two numbers only mean something if one habit comes first.

Rule zero: seal the test set before anything else

The moment your data is clean, before you explore further, engineer a feature or train anything, split off a test set and do not touch it again until the very end. Every decision (which features, which model, which settings) is made on the training set only.

from sklearn.model_selection import train_test_split

# Do this FIRST, before exploring, before tuning, before anything
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0
)

# X_test and y_test are now sealed.
# Every decision from here on uses ONLY X_train and y_train.

Use the test set to make even one decision and you have leaked information into your process. The number it eventually reports stops being honest. It is the difference between a real exam and one where you have already seen the answer key.

The experiment: same data, rising complexity

Here, sixty noisy points are fitted with polynomials of increasing degree, from a straight line up to a wildly flexible degree-15 curve. At each step we record the mean squared error on the training set and on the sealed test set.

for degree in [1, 5, 9, 15]:
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(X_train, y_train)
    train_mse = mean_squared_error(y_train, model.predict(X_train))
    test_mse  = mean_squared_error(y_test,  model.predict(X_test))

# OUTPUT
# degree  1: train= 3.31  test=  3.17
# degree  5: train= 1.55  test=  1.13
# degree  9: train= 1.48  test=  1.13
# degree 15: train= 1.14  test=195.86

Look at what happened between degree 9 and degree 15. Training error kept improving, from 1.48 to 1.14, which read on its own looks like progress. Test error exploded from 1.13 to 195.86. The model did not get better. It got catastrophically worse at the only thing that matters, while looking better on the metric that does not.

Reading the diagnosis

Training error	Test error	Diagnosis	What to do
High	High	Underfitting (high bias): too simple	Add complexity, better features, a richer model
Low	High	Overfitting (high variance): memorised noise	Simplify, gather more data, regularise
Low	Low, and close	Good fit	Stop tuning and ship

Degree 1 is the first row: both errors are high because a straight line cannot bend to follow a curve. Degree 15 is the second row. Degrees 5 and 9 sit in the third. Wondering whether more data would help? Try our learning curves tutorial in Python to compare training and validation errors as the training sample grows.

Think of a student who memorises the exact wording of every practice question instead of learning the method. Perfect score on the practice set; falls apart when the exam asks the same thing in slightly different words. That is degree 15. The same thing happens with decision trees: an unlimited tree scores 100% on training data and gains nothing on the test set, as shown in logistic regression vs random forest vs gradient boosting.

A warning about cross-validation

Cross-validation gives you a more reliable estimate than a single split, but it has its own trap: when rows have a time order, ordinary k-fold quietly leaks the future into training. We cover that failure, and the fix, in When Cross-Validation Lies to You.

Key takeaways

• Seal the test set before anything else. Open it once, at the end.
• Underfitting: both errors high. Overfitting: training error low, test error high.
• Training error almost always improves with complexity. That is exactly why you cannot judge a model by it.
• In this experiment, test error jumped from 1.13 to 195.86 while training error looked better than ever.

---

Learn this properly, with code that actually ran

This experiment comes from Chapter 12 of the Machine Learning Starter Kit, a 172-page handbook where every listing was executed and every output is the real one. Just starting out? Get the free 90-Day Machine Learning Roadmap first.