Coderhouse goes global

🌍

Read more

What Is Data Leakage in Machine Learning and How Teams Prevent It

AI Engineering

Build and ship real applications with LLMs, agents and automation.

8 weeks · Live online · View course →

Data Analytics

Go from zero to job-ready analyst with SQL, Python and BI tools.

10 weeks · Live online · View course →

Dan Patiño

AI Strategy & Innovation at Coderhouse

Artificial Intelligence

What Is Data Leakage in Machine Learning and How Teams Prevent It

What Is Data Leakage in Machine Learning and How Teams Prevent It

Data leakage happens when a machine learning model is trained with information that would not be available at the moment it has to make a real prediction. The model looks brilliant during evaluation — sometimes suspiciously so — and then performs much worse in production. It is one of the most common and expensive mistakes in applied machine learning, and it is also one of the hardest to notice, because nothing crashes: the metrics simply lie.

The scikit-learn guide to common pitfalls lists leakage among the top mistakes practitioners make, and the Kaggle Learn lesson on data leakage uses it to explain why some competition-winning models are useless in the real world. As more teams ship models quickly with AutoML and AI coding assistants, understanding leakage has become a basic skill rather than an advanced one.

The two main types of data leakage

Target leakage

Target leakage occurs when a feature contains information created after or because of the outcome you are predicting. A classic example: predicting whether a patient has pneumonia using a column that says whether they took antibiotics. The antibiotics were prescribed because of the diagnosis, so the model learns a shortcut that will not exist when a new patient arrives. In business data, think of predicting churn with a “cancellation reason” field, or predicting loan default with a “collections status” column.

Train-test contamination

This type happens when information from the validation or test data leaks into training. The most frequent cause is preprocessing the whole dataset before splitting it: scaling, imputing missing values, selecting features, or oversampling with statistics computed on all rows. The test set is no longer truly unseen, so the score is optimistic.

Other common sources of leakage

  • Time travel in time series: randomly shuffling data that has a time order, so the model trains on the future to predict the past. Use time-based splits such as TimeSeriesSplit.

  • Duplicate or related rows across splits: the same customer, patient, or device appearing in both training and test data. Split by group (customer ID) instead of by row.

  • Aggregates computed with future data: features like “total purchases this year” calculated at the end of the year and used for predictions made in March.

  • Leaky IDs and timestamps: record IDs or processing dates that correlate with the label because of how the data was collected.

How to detect data leakage

1. Be suspicious of results that are too good

An accuracy or AUC far above what similar problems achieve is a warning sign, not a celebration. Compare against a simple baseline and published benchmarks for the domain.

2. Check feature importance

If one feature dominates the model, ask exactly when and how that value is created. Very often the top feature is the leak.

3. Rebuild the prediction timeline

For every feature, write down the moment it becomes known and compare it with the moment the prediction is made. Anything known later must go, or must be recomputed with a cutoff date.

4. Validate like production

Hold out the most recent period, or a set of customers never seen in training, and evaluate there. A large gap between cross-validation scores and this realistic holdout points to leakage.

How to prevent it

Split first, preprocess second. Always separate training and test data before any transformation that learns from the data.

Use pipelines. Tools like scikit-learn Pipelines fit scalers, imputers, and encoders only on the training folds during cross-validation, which removes a whole class of contamination bugs.

Engineer features with a cutoff date. Good feature engineering computes every aggregate “as of” the prediction time. Feature stores and point-in-time joins exist precisely for this.

Review features with domain experts. Someone who knows the business process can spot in minutes that a field is filled in only after the outcome happens.

Data leakage vs overfitting

Both make models fail on new data, but for different reasons. Overfitting means the model memorized noise in the training data, and a proper validation set usually reveals it. Leakage fools the validation set itself, so the evaluation looks fine until the model meets real users.

Leakage in the age of LLMs

The same idea applies to large language models, where it is often called benchmark contamination: if test questions appeared in the training data, benchmark scores overstate real ability. Teams building AI products protect themselves by creating private evaluation sets from their own use cases and keeping them out of any fine-tuning data.

FAQ

Is data leakage always intentional?

Almost never. It is usually an accident caused by preprocessing order, poorly understood columns, or random splits on time-ordered data.

Can cross-validation prevent data leakage?

Only if preprocessing happens inside each fold, for example with a pipeline, and if the split respects time and groups. Otherwise cross-validation can leak too.

How do I fix a model that has leakage?

Remove or recompute the leaky features, rebuild the preprocessing inside a pipeline, re-split the data realistically, and retrain. Expect the scores to drop — the new, lower number is the honest one.

Keep learning

Trustworthy models start with trustworthy data and honest evaluation. If you want to build those skills:

  • AI Engineering Course — learn to build, evaluate, and ship machine learning and AI applications that hold up in production.

  • Data Analytics Course — learn to clean, prepare, and analyze data so every feature you use is reliable.

Dan Patiño

I'm Dan Patiño, head of AI Strategy & Innovation at Coderhouse. My day-to-day work involves merging the tactical management of e-commerce (CRO, Email Marketing and SEO) with the development of disruptive solutions. I specialize in building internal AI-powered apps to automate tasks and boost innovation within the team. I firmly believe that technology is strategy's best ally. To dive deeper into my professional journey, I'll be waiting for you on my LinkedIn profile.