Standard 1 stop to get here

Data Cleaning

Finding and fixing errors, inconsistencies and gaps in a dataset (duplicates, typos, missing values, wrong labels) before a model learns from it.

Your route here

1 stop · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Data Cleaning · you are here ✓ understood

Data cleaning is the unglamorous work of making a dataset correct before anything learns from it. Rahm and Do define it as detecting and removing errors and inconsistencies to improve data quality. A model trained on dirty data learns the dirt along with the signal, and no amount of modelling downstream fully undoes it.

What goes wrong

  • Duplicates: the same record entered twice, or the same customer spelled two ways. Duplicates that land in both training and test sets inflate scores, a form of data leakage.
  • Missing values: blank fields that must be filled in, flagged, or dropped, each choice with its own bias.
  • Invalid or inconsistent values: typos, impossible ages, dates in three formats, the same unit recorded as metres in one table and feet in another.
  • Outliers: some are errors, some are the most important rows in the data. Anomaly detection can flag them; a person usually has to decide which is which.
  • Wrong labels: for supervised learning, the answers themselves can be wrong.

Problems multiply when data is merged from several sources, which rarely agree on formats or conventions; Rahm and Do note that integrating sources sharply increases the need for cleaning.

Labels count too

Label errors are easy to overlook because they look like data. Northcutt and colleagues estimated an average of at least 3.3% label errors across the test sets of ten widely used benchmarks, including at least 6% of the ImageNet validation set. Errors at that level can change which model looks best.

In practice

Cleaning overlaps with data preprocessing but comes first: fix what is wrong before you transform what is right. Write the rules down as code so they can be rerun and reviewed, keep the raw data untouched, and version the cleaned copy, because cleaning decisions shape results as much as model choices do.

Where it sits

Before this

Dataset
Data Cleaning

Leads to

Nothing yet: a destination in its own right.

Explore nearby

Foundations Data Preprocessing Cleaning, transforming, and preparing raw data for model training (handling missing values, normalization, encoding). Evaluation Data Leakage When information from outside the training data is used to create the model, leading to overly optimistic performance estimates. Foundations Labeled Data Data with associated target outputs or annotations, required for supervised learning tasks. Foundations Anomaly Detection Identifying unusual patterns or outliers in data that don't conform to expected behavior, used for fraud detection and monitoring. Foundations Feature Engineering The process of selecting, transforming, and creating input features to improve model performance. Shipping AI Data Versioning Tracking different versions of datasets to ensure reproducibility and manage changes.

Sources

  1. Erhard Rahm and Hong Hai Do, "Data Cleaning: Problems and Current Approaches" . IEEE Data Engineering Bulletin 23(4), 2000
  2. Curtis G. Northcutt, Anish Athalye and Jonas Mueller, "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks" . arXiv 2103.14749, 2021