Landmark 2 stops to get here · leads to 6

Activation Function

A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.

Your route here

2 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  3. Activation Function · you are here ✓ understood

Picture it

input xoutputsigmoidtanhReLU
Notice how sigmoid and tanh squash inputs into a range while ReLU passes positives and zeroes negatives: none is a straight line.

An activation function is the small non-linear function each neuron applies to its weighted sum of inputs before passing the result on. It sounds like a detail, but without it a neural network could not learn anything interesting.

Here is why. Each layer multiplies its inputs by weights and adds them up, which is a linear operation. Stack two linear operations and you get another linear operation. So a hundred layers with no activation function collapse into the equivalent of one, and the network can only draw straight lines through the data. Putting a non-linear function between layers breaks that collapse and lets depth add real expressive power.

The main kinds

The diagram plots the classic shapes side by side.

  • Sigmoid squashes any input into the range 0 to 1. Tanh is the same S-curve rescaled to −1 to 1. Both were the default for decades.
  • ReLU (rectified linear unit) outputs the input if it is positive and zero otherwise. It is cheap to compute, and the standard textbook calls it the default recommendation for modern networks.
  • GELU is a smooth cousin of ReLU that multiplies each input by the probability that a standard normal variable falls below it, so it bends smoothly instead of cutting hard at zero. It is a common choice in transformers.
  • Softmax is a special case used at the output: it turns a vector of scores into probabilities that sum to 1.

Why ReLU took over

The S-curves saturate. Far from zero they go flat, so their slope is nearly zero, and gradient-based training gets almost no signal through them. Stack many such layers and the gradient shrinks at each one: the vanishing gradient problem. ReLU’s slope is exactly 1 for every positive input, so gradients pass through intact. In the 2012 AlexNet paper, a four-layer convolutional network with ReLUs reached 25% training error on CIFAR-10 six times faster than the same network with tanh.

The catch

ReLU has its own failure mode. A unit whose input is negative for every example outputs zero and gets zero gradient, so it stops learning; these are often called dead units. Variants such as Leaky ReLU keep a small slope below zero to avoid that. In practice, most people use whatever the architecture they are building on already uses.

Where it sits

Before this

Neural Network
Activation Function

Leads to

Softmax ReLU Sigmoid Vanishing Gradient Maxout Xavier Initialization

Explore nearby

Neural Networks ReLU Rectified Linear Unit - an activation function that outputs the input if positive, zero otherwise. f(x) = max(0, x). Neural Networks Sigmoid An activation function that squashes values to range (0,1), often used for binary classification and gates in LSTMs. Neural Networks Softmax A function that turns a list of scores (logits) into probabilities that are all positive and sum to 1; the standard output of classifiers and language models. Neural Networks GELU Gaussian Error Linear Unit - a smooth activation function combining properties of dropout and ReLU, used in BERT and GPT. Training Vanishing Gradient A problem where gradients become extremely small during backpropagation, preventing deep networks from learning effectively.

Sources

  1. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning . MIT Press, 2016; section 6.3, Hidden Units
  2. Krizhevsky, Sutskever and Hinton, "ImageNet Classification with Deep Convolutional Neural Networks" . NeurIPS 2012; section 3.1, ReLU Nonlinearity
  3. Hendrycks and Gimpel, "Gaussian Error Linear Units (GELUs)" . 2016