When Cross-Validation Lies to you

You train a model, run cross-validation, and the score looks great. You ship it. A month later the model is making bad calls and that great score is nowhere to be found. If your data has a time order to it, this happens more often than people admit, and the cause is usually the cross-validation itself.

The right split depends on what future predictions must generalise to: new independent examples, unseen groups, or later dates. A high score is meaningful only when the validation design matches that use.

Learn to evaluate models honestly. Read the free Starter Kit excerpt on test sets before deciding whether the €19 handbook is right for you.

What k-fold assumes

Plain k-fold cross-validation, the kind you get from scikit-learn’s KFold, cuts your rows into chunks, trains on some chunks, and tests on the rest. It repeats this so every row lands in the test set once. The whole idea rests on one assumption: the rows are independent. Shuffle them, split them, and nothing leaks from train into test.

That assumption is fine for a table of customers where each row stands on its own. It falls apart the moment your rows are points in time.

Why time breaks it

For forecasting, a shuffled split can train on future observations and validate on earlier ones. Repeated entities and overlapping label windows can introduce further dependence or leakage. Sharing historical input values does not by itself prove that the answer has leaked; inspect which information was actually available at prediction time.

The resulting score may be optimistic because the validation task differs from the intended production task.

A leakage demonstration can produce an inflated score when its construction exposes target information across the split. There is no universal R-squared value to expect, and genuinely independent noise does not become predictable merely because k-fold is used. Interpret any synthetic result in the context of its data-generation and splitting code.

Choose the split before choosing the model

For forecasting, begin with chronological or walk-forward validation. For repeated people, accounts, or devices, consider group-aware splits when evaluating generalisation to unseen groups. Fit learned preprocessing only within each training fold. See the scikit-learn cross-validation guide.

Choose the metric after the split

A suitable split tells you where to evaluate; the metric tells you what to measure. For classification, use precision and recall for decisions, ROC curves and AUC for ranking across thresholds, and Brier score for probability prediction error. If only a small fraction can be investigated, compare lift curves at that review budget.

Follow the model-evaluation checklist to combine an appropriate split, a baseline and a decision-relevant metric. Changing the metric cannot repair leakage in the data split.

When purging and embargo are relevant

The fix is not a fancier model. It is a fairly small change to the split.

Two ideas do most of the work:

• Purging. Before testing on a block, drop any training rows whose label overlaps that block in time. No shared answers.
• Embargo. Also drop a short window of training rows right after the test block, because data that close is still correlated with it.

Keep the time order and cut the rows that leak. Then the model has to earn its score.

You can write this yourself, and it is a good exercise. A small library called purgedcv does it for you. It plugs into scikit-learn the same way KFold does, so you pass it to cross_val_score and the rest of your code stays put. It also handles the harder cousins of this problem, like combinatorial splits that give you several backtest paths instead of one. The documentation has runnable examples if you want to try it on your own data.

How uncertain is the final score?

Once the split is sound, there is another question: how much might your score vary with a different evaluation sample? Our bootstrap confidence intervals tutorial in Python uses paired resampling to compare two fixed models on the same held-out rows. Its simple row bootstrap assumes independent observations; time series and repeated groups need a resampling design that respects their dependence. Bootstrapping cannot repair leakage.

The takeaway

If your data has a clock attached to it, treat a high cross-validation score with suspicion until you know the split respects that clock. Purging is not there to make your numbers look better. It is there to stop them looking better than they are. A model that survives an honest split is one you can actually trust outside your notebook.

Put model evaluation into a broader learning plan with the Core ML phase of our free 90-day roadmap. For a practical coding companion, explore the Machine Learning Starter Kit.