A data pipeline is the plumbing that gets data ready for a model, run automatically rather than by hand. It pulls raw records from wherever they live, checks them, reshapes them into features, and delivers the result to training jobs or to a model in production. When people say a model is only as good as its data, the pipeline is what decides which data that is.
In a notebook, data preprocessing is a few cells you run once. In production the same steps must run every day, on new data, identically each time, and fail loudly when something is off. That is the job of a pipeline.
Validate early
Polyzotis and colleagues at Google argue that errors in input data can wipe out the benefits of faster, more accurate training and inference, and describe a validation system built into Google’s TFX platform. Typical checks compare incoming data against an expected schema, flag missing or out-of-range values, and look for differences between the data used in training and the data seen at serving time.
Keep training and serving consistent
A classic failure is computing a feature one way for training and another way at inference, so the model sees inputs in production unlike anything it learned from. Sharing one transformation codebase, or a feature store, helps avoid this. Data versioning records exactly which data produced which model.
Avoid the jungle
Sculley and colleagues warn about “pipeline jungles”, data preparation that grows organically as new sources are bolted on until it becomes hard to test or change. They also note that data dependencies can build up debt much as code dependencies do. Pipelines that are designed deliberately, tested, and monitored for data drift are a core part of MLOps.