Landmark 2 stops to get here · leads to 7

Recurrent Neural Network

A neural network architecture with loops that allow information to persist, designed for sequential data like text and time series.

Your route here

2 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  3. Recurrent Neural Network · you are here ✓ understood

Picture it

  1. 01 x₁ → h₁ Read the first input, produce a hidden state
  2. 02 x₂ + h₁ → h₂ Combine the next input with the previous state
  3. 03 x₃ + h₂ → h₃ Same weights reused at every time step
  4. 04 Output from hₜ The final state summarizes the whole sequence
This is the RNN's loop unrolled over time: the hidden state carries memory forward, one step at a time, with the same weights at each step.

A recurrent neural network (RNN) is a neural network built for sequences. It reads its input one step at a time, a word, a character or a sensor reading, and keeps a hidden state: a vector that acts as a running memory of everything it has seen so far. At each step it combines the new input with the previous state to produce the next state.

That loop is the whole idea. A plain feed-forward network takes a fixed-size input all at once and has no notion of order. An RNN can take a sequence of any length, and because the state is passed forward, what it does with word five depends on words one to four.

How it works

The diagram unrolls the loop across time. Two things are worth noticing. First, the same weights are used at every step. That parameter sharing is what lets one small network handle a sentence of 5 words or 500, and generalize to lengths it never saw in training. Second, the final hidden state has to carry a summary of the whole sequence, which is a lot to ask of one fixed-size vector.

RNNs are trained by unrolling them like this and running backpropagation through the unrolled graph, a procedure called backpropagation through time. The error at the end is pushed back step by step to the start.

Why they struggle with long sequences

Going backward through many steps means multiplying by the same weights over and over. The gradient tends to shrink toward zero (vanishing gradients) or, more rarely, blow up (exploding gradients). Vanishing gradients mean the network barely learns from anything far back in the sequence. Exploding gradients are usually tamed by gradient clipping, which caps the gradient’s size.

The standard fix for vanishing gradients is gating. The LSTM, introduced in 1997, and the later, simpler GRU add learned gates that control what is written into memory, kept and read out, giving information a path to survive across many steps. Through the 2010s, LSTMs were a standard choice for language and speech. In 2014, a sequence-to-sequence system built from two LSTMs, one reading and one writing, reached a BLEU score of 34.8 on English-to-French translation.

Where they stand now

Transformers replaced RNNs for most language work. An RNN must process step 50 after step 49, so training can’t spread across a sequence in parallel, and distant words are still many steps apart. Attention links any two positions directly. RNNs do keep one advantage: they need the same small amount of memory per step however long the sequence gets, which suits streaming input.

Where it sits

Explore nearby

In the research

All papers →

4 papers that build on Recurrent Neural Network .

Sources

  1. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning . MIT Press, 2016; chapter 10, Sequence Modeling: Recurrent and Recursive Nets
  2. Pascanu, Mikolov and Bengio, "On the difficulty of training Recurrent Neural Networks" . 2012; ICML 2013
  3. Sutskever, Vinyals and Le, "Sequence to Sequence Learning with Neural Networks" . 2014