Landmark 8 stops to get here · leads to 6

Self-Attention

An attention step in which every token in a sequence looks at every token in that same sequence and builds a new representation from a weighted mix of them.

Your route here

8 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  3. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  4. Feature ✓ understood

    A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.

  5. Deep Learning ✓ understood

    A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.

  6. Representation Learning ✓ understood

    Letting a model learn its own internal description of the data, such as embeddings or layered visual features, instead of relying on features designed by hand.

  7. Embedding ✓ understood

    A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.

  8. Attention Mechanism ✓ understood

    A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step.

  9. Self-Attention · you are here ✓ understood

Picture it

EACH ROW ATTENDS TO THE COLUMNS · ROWS SUM TO 1thethe0.18catcat0.39satsat0.42onon0.26thethe0.18matmat0.38computed from toyword vectors:softmax(q·k / √d)
Each row shows how much one token (here, a word) attends to every token in the sentence, including itself; the weights in each row sum to 1.

Self-attention is the operation at the heart of the transformer. Each token in a sequence looks at every token in that same sequence, itself included, decides how relevant each one is, and replaces its own vector with a weighted blend of what it found. After one layer, the vector for “it” in “the cat sat because it was tired” can carry information pulled from “cat”.

Ordinary attention was first used to let a translation model’s decoder look back at the source sentence: two different sequences. Self-attention points the same machinery inward. The 2017 transformer paper notes it was “sometimes called intra-attention” and had already been used in reading-comprehension models.

How it works

Every token’s vector is multiplied by three learned matrices to produce a query, a key and a value. The query says what the token is looking for, the key says what it offers, and the value is the content it passes on if chosen.

  1. Score each pair by taking the dot product of one token’s query with another token’s key.
  2. Divide by the square root of the key dimension, which stops large dot products from saturating the next step. This is the “scaled” in scaled dot-product attention.
  3. Apply a softmax across each row, so a token’s weights over all positions are positive and sum to 1.
  4. Output a weighted sum of the value vectors using those weights.

The chart is that weight matrix for one sentence: read a row to see where one token put its attention. Because all rows are computed together as a few matrix multiplications, the whole sequence is processed in parallel, which is what made transformers so fast to train.

Variations you’ll meet

  • Multi-head attention runs several self-attention operations side by side with separate projections, so one head can track syntax while another tracks coreference.
  • Causal (masked) self-attention blocks each token from seeing later positions. Decoder-only language models use it so that predicting the next word can’t peek at the answer.
  • Cross-attention is the non-self version: queries come from one sequence, keys and values from another.

The catch

Every token scores every other token, so the work and the score matrix grow with the square of the sequence length. That is the main cost of a long context window, and it drives techniques like FlashAttention that reorganize the computation to use memory more efficiently. Self-attention also has no built-in sense of order, which is why transformers add positional encodings.

Where it sits

Before this

Attention Mechanism
Self-Attention

Leads to

Cross-Attention Masking Multi-Head Attention Positional Encoding Attention Mask Bidirectional Attention

Explore nearby

Language & LLMs Multi-Head Attention Running multiple attention operations in parallel with different learned projections, capturing diverse relational patterns. Language & LLMs Query-Key-Value The three learned projections in attention mechanisms used to compute attention weights and outputs. Language & LLMs Transformer A neural network architecture, introduced in 2017, built from stacked self-attention and feed-forward layers; the basis of nearly every modern large language model. Language & LLMs Cross-Attention Attention between two different sequences, where queries come from one and keys/values from another. Language & LLMs Scaled Dot-Product Attention The attention computation using dot product of queries and keys, scaled by dimension to stabilize gradients.

In the research

All papers →

7 papers that build on Self-Attention ; showing 5, canon first.

Canon · 2017 Attention Is All You Need Every major LLM, and most modern vision and speech models, is a transformer. Canon · 2022 FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness It is a big part of why long context windows became affordable, and it ships inside nearly every LLM stack. Frontier · Dec 2025 · 671 citations DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models It shows an open model competing with the best closed ones on reasoning while getting cheaper to run. Frontier · Sep 2025 · 189 citations LongLive: Real-time Interactive Long Video Generation Minute-long, steerable video at 20 frames per second on one GPU makes interactive video generation real. Frontier · Oct 2025 · 116 citations Kimi Linear: An Expressive, Efficient Attention Architecture It is a credible claim that linear attention can replace full attention without a quality penalty.

Sources

  1. Vaswani et al., "Attention Is All You Need" . 2017; section 3.2, Scaled Dot-Product Attention
  2. Cheng, Dong and Lapata, "Long Short-Term Memory-Networks for Machine Reading" . 2016; an early intra-attention model