Landmark 2 stops to get here · leads to 7

Regression

A supervised learning task where the model predicts continuous numerical values rather than discrete categories.

Your route here

2 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Supervised Learning ✓ understood

    Learning from examples paired with the correct answer, so a model can predict answers for new inputs it hasn't seen.

  3. Regression · you are here ✓ understood

Picture it

Regression

  • Predicts a continuous number
  • e.g. house price: $412,500
  • Loss: mean squared error
  • Judged by RMSE or R²

Classification

  • Predicts a discrete category
  • e.g. spam or not spam
  • Loss: cross-entropy
  • Judged by accuracy, precision, recall
Notice the only real difference is the output: regression answers "how much?" while classification answers "which one?"

Regression is the kind of supervised learning where the answer is a number: a house price, tomorrow’s temperature, how many minutes a delivery will take. The model sees examples of inputs paired with the correct number and learns a function that maps new inputs to a numeric prediction.

Its sibling is classification, where the answer is a category. The diagram lines the two up, and the distinction is only about the output. The same inputs, say a house’s size, location and age, could feed a regression model predicting its price or a classifier predicting whether it will sell within a month.

How it works

The simplest version is linear regression. Each input feature gets a weight, the prediction is the weighted sum plus a constant, and the weights are chosen to make the squared gaps between predictions and true values as small as possible. That criterion, least squares, has a closed-form solution, so a linear regression can be fitted in one step rather than by trial and error.

Squaring the errors is a deliberate choice. It makes the loss smooth and easy to optimize, and it punishes big misses much more than small ones: being off by 10 costs a hundred times as much as being off by 1. That makes least squares sensitive to outliers, which is why some problems use absolute error instead.

The model doesn’t have to be a straight line. Decision trees, gradient-boosted ensembles and neural networks all do regression too. Only the output layer and the loss change: one number out, and an error measured as a distance.

How it’s judged

  • Mean squared error, the average of the squared gaps, or its square root (RMSE), which is back in the target’s units: “off by about $18,000 on average”.
  • R², the share of the variation in the target that the model explains, compared with always guessing the average.

The catch

A regression model predicts a single best guess, usually something like the average outcome for inputs like these. It says nothing about how uncertain that guess is unless you model the spread as well. And like any learned model it is most trustworthy inside the range it was trained on. Extrapolating a trend line far past the data, as in time-series forecasting years ahead, is where confident regression predictions most often go wrong.

Where it sits

Explore nearby

Sources

  1. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning . MIT Press, 2016; section 5.1.4, Linear Regression
  2. scikit-learn User Guide, Linear Models: Ordinary Least Squares
  3. scikit-learn User Guide, Metrics and scoring: regression metrics . R² and mean squared error