Bagging, short for bootstrap aggregating, is a way to make an unstable model steadier by training many copies of it and combining their answers. Leo Breiman introduced it in 1994. It is one of the classic ensemble strategies, alongside boosting and stacking.
The recipe has two steps. First, make many bootstrap samples: new training sets the same size as the original, drawn at random with replacement, so each one repeats some examples and leaves others out (about a third, on average). Train one model on each. Second, aggregate: average the predictions when the target is a number, take a plurality vote when it’s a class.
Why it works
Some models change a lot when the training data changes a little. A deep decision tree is the textbook case: move a few examples and the splits near the top can flip. That sensitivity is variance, and it is a major source of overfitting. Averaging many such models, each fitted to a slightly different resample, smooths out the idiosyncrasies each one picked up while keeping what they agree on. Breiman’s paper puts the condition plainly: bagging helps when perturbing the training set changes the predictor significantly.
The corollary matters too. A stable model, such as a regularized linear model, barely changes between resamples, so bagging it adds cost and little else. In the terms of the bias-variance tradeoff, bagging mainly lowers variance and does little about bias.
Where you meet it
A random forest is bagged decision trees with one extra twist: each split considers only a random subset of features, which makes the trees less alike and the average stronger (the paper). Gradient boosting takes the other route, training models one after another, each correcting the last.