Standard 3 stops to get here · leads to 2

Weight Initialization

How a network's weights are set before training starts: random values scaled so that signals and gradients neither shrink nor blow up across layers.

Your route here

3 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  3. Weight ✓ understood

    Learnable parameters connecting neurons in neural networks, determining the strength of connections.

  4. Weight Initialization · you are here ✓ understood

Weight initialization is the choice of starting values for a network’s weights before training begins. Training only nudges weights from wherever they start, so a bad start can leave a deep network unable to learn at all.

Two requirements shape every scheme. First, the weights must break symmetry: if two units in a layer start with identical weights, they receive identical gradients and stay identical forever, so the layer behaves like a single unit. Random values fix that. Second, the random values need the right scale. Each layer multiplies its input by its weights; if they are slightly too large, activations and gradients grow layer after layer until they explode, and if slightly too small they shrink until they vanish.

Scaling to the layer

The standard fix is to draw weights with a variance tied to how many inputs a unit has, so the signal keeps roughly the same size as it passes through.

  • Xavier (Glorot) initialization balances the number of inputs and outputs of a layer. Glorot and Bengio derived it in 2010 for tanh-like activations.
  • He initialization uses a standard deviation of √(2/n) for n inputs, compensating for ReLU zeroing out half its inputs. He et al. used it to train very deep networks from scratch.

Biases are usually just set to zero.

Less fragile than it was

Later techniques made training much less sensitive to the starting point. Normalization layers rescale activations as they flow, and residual connections give gradients a direct path back through the network. Initialization still matters, but it is now one safeguard among several rather than the thing that decides whether a deep network trains.

Where it sits

Before this

Weight Neural Network
Weight Initialization

Leads to

He Initialization Xavier Initialization

Explore nearby

Neural Networks Xavier Initialization A weight initialization strategy maintaining variance across layers, improving training of deep networks. Neural Networks He Initialization Weight initialization designed for ReLU activations, preventing vanishing/exploding gradients in deep networks. Training Vanishing Gradient A problem where gradients become extremely small during backpropagation, preventing deep networks from learning effectively. Training Exploding Gradient A problem where gradients become extremely large during backpropagation, causing unstable training and NaN values. Neural Networks Batch Normalization A technique that normalizes layer inputs to stabilize and accelerate training by reducing internal covariate shift. Neural Networks Activation Function A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.

Sources

  1. Xavier Glorot and Yoshua Bengio, "Understanding the difficulty of training deep feedforward neural networks" . AISTATS 2010; proposes the normalized (Xavier) initialization
  2. Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun, "Delving Deep into Rectifiers" . ICCV 2015; derives an initialization for ReLU networks
  3. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning . MIT Press, 2016; section 8.4, Parameter Initialization Strategies