Language & LLMs Sep 2014

Neural Machine Translation by Jointly Learning to Align and Translate

Dzmitry Bahdanau et al. · ICLR 2015

arXiv:1409.0473

In short

Squeezing a whole sentence into one fixed-length vector was the bottleneck of neural translation. This paper lets the decoder look back over every source word and weight the relevant ones for each word it produces: the first attention mechanism.

Why it matters

Attention was born here, three years before it became “all you need”.

Read first

The 3 Field Guide ideas this paper leans on.

Neural Networks · Landmark Recurrent Neural Network A neural network architecture with loops that allow information to persist, designed for sequential data like text and time series. Language & LLMs · Standard Encoder-Decoder A architecture where the encoder processes input and the decoder generates output, used in translation and sequence-to-sequence tasks. Language & LLMs · Landmark Attention Mechanism A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step.

Starting from scratch? The full route 12 ideas · basics first

Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Neural Network ✓ understood
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
Recurrent Neural Network · read first ✓ understood
A neural network architecture with loops that allow information to persist, designed for sequential data like text and time series.
Sequence-to-Sequence ✓ understood
Models that transform input sequences to output sequences, used for translation, summarization, and generation.
Dataset ✓ understood
A collection of data examples used for training, validating, or testing machine learning models.
Feature ✓ understood
A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.
Deep Learning ✓ understood
A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.
Representation Learning ✓ understood
Learning useful features or representations of data automatically, rather than hand-crafting them.
Embedding ✓ understood
A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.
Attention Mechanism · read first ✓ understood
A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step.
Transformer ✓ understood
A neural network architecture, introduced in 2017, built from stacked self-attention and feed-forward layers; the basis of nearly every modern large language model.
Encoder-Decoder · read first ✓ understood
A architecture where the encoder processes input and the decoder generates output, used in translation and sequence-to-sequence tasks.

Nearby papers

Canon · 1997 Long Short-Term Memory LSTMs powered speech recognition, translation and text generation for twenty years, until transformers took over. Canon · 2017 Attention Is All You Need Every major LLM, and most modern vision and speech models, is a transformer. Frontier · #42 Kimi Linear: An Expressive, Efficient Attention Architecture It is a credible claim that linear attention can replace full attention without a quality penalty. Frontier · #81 Mamba-3: Improved Sequence Modeling using State Space Principles Inference cost now dominates, which makes fast, constant-memory architectures matter again.