Language & LLMs Jun 2017

Attention Is All You Need

Ashish Vaswani et al. · NeurIPS 2017

arXiv:1706.03762

In short

The Transformer drops recurrence and convolution entirely and builds a translation model from stacked self-attention and feed-forward layers. It trains in parallel, set new translation records, and cost a fraction of the compute of earlier models.

Why it matters

Every major LLM, and most modern vision and speech models, is a transformer.

Read first

The 4 Field Guide ideas this paper leans on.

Language & LLMs · Landmark Attention Mechanism A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step. Language & LLMs · Landmark Self-Attention A mechanism where each token attends to all other tokens in the sequence to understand contextual relationships. Language & LLMs · Standard Encoder-Decoder A architecture where the encoder processes input and the decoder generates output, used in translation and sequence-to-sequence tasks. Language & LLMs · Landmark Transformer A neural network architecture, introduced in 2017, built from stacked self-attention and feed-forward layers; the basis of nearly every modern large language model.

Starting from scratch? The full route 13 ideas · basics first

Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Neural Network ✓ understood
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
Dataset ✓ understood
A collection of data examples used for training, validating, or testing machine learning models.
Feature ✓ understood
A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.
Deep Learning ✓ understood
A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.
Representation Learning ✓ understood
Learning useful features or representations of data automatically, rather than hand-crafting them.
Embedding ✓ understood
A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.
Attention Mechanism · read first ✓ understood
A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step.
Self-Attention · read first ✓ understood
A mechanism where each token attends to all other tokens in the sequence to understand contextual relationships.
Recurrent Neural Network ✓ understood
A neural network architecture with loops that allow information to persist, designed for sequential data like text and time series.
Sequence-to-Sequence ✓ understood
Models that transform input sequences to output sequences, used for translation, summarization, and generation.
Transformer · read first ✓ understood
A neural network architecture, introduced in 2017, built from stacked self-attention and feed-forward layers; the basis of nearly every modern large language model.
Encoder-Decoder · read first ✓ understood
A architecture where the encoder processes input and the decoder generates output, used in translation and sequence-to-sequence tasks.

Nearby papers

Canon · 2014 Neural Machine Translation by Jointly Learning to Align and Translate Attention was born here, three years before it became “all you need”. Canon · 2018 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding It established “pre-train once, fine-tune everywhere”, and still powers much of search and classification. Canon · 2020 An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale It brought vision and language onto one architecture, which made today’s multimodal models possible. Canon · 2022 FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness It is a big part of why long context windows became affordable, and it ships inside nearly every LLM stack.