A recurrent neural network (RNN) is a neural network built for sequences. It reads its input one step at a time, a word, a character or a sensor reading, and keeps a hidden state: a vector that acts as a running memory of everything it has seen so far. At each step it combines the new input with the previous state to produce the next state.
That loop is the whole idea. A plain feed-forward network takes a fixed-size input all at once and has no notion of order. An RNN can take a sequence of any length, and because the state is passed forward, what it does with word five depends on words one to four.
How it works
The diagram unrolls the loop across time. Two things are worth noticing. First, the same weights are used at every step. That parameter sharing is what lets one small network handle a sentence of 5 words or 500, and generalize to lengths it never saw in training. Second, the final hidden state has to carry a summary of the whole sequence, which is a lot to ask of one fixed-size vector.
RNNs are trained by unrolling them like this and running backpropagation through the unrolled graph, a procedure called backpropagation through time. The error at the end is pushed back step by step to the start.
Why they struggle with long sequences
Going backward through many steps means multiplying by the same weights over and over. The gradient tends to shrink toward zero (vanishing gradients) or, more rarely, blow up (exploding gradients). Vanishing gradients mean the network barely learns from anything far back in the sequence. Exploding gradients are usually tamed by gradient clipping, which caps the gradient’s size.
The standard fix for vanishing gradients is gating. The LSTM, introduced in 1997, and the later, simpler GRU add learned gates that control what is written into memory, kept and read out, giving information a path to survive across many steps. Through the 2010s, LSTMs were a standard choice for language and speech. In 2014, a sequence-to-sequence system built from two LSTMs, one reading and one writing, reached a BLEU score of 34.8 on English-to-French translation.
Where they stand now
Transformers replaced RNNs for most language work. An RNN must process step 50 after step 49, so training can’t spread across a sequence in parallel, and distant words are still many steps apart. Attention links any two positions directly. RNNs do keep one advantage: they need the same small amount of memory per step however long the sequence gets, which suits streaming input.