Standard 10 stops to get here · leads to 2

Masking

Hiding some positions from a model, either to control what attention may look at (causal and padding masks) or to create a fill-in-the-blank training task.

Your route here

10 stops · basics first

Token ✓ understood
The unit of text a language model reads and writes: a word, part of a word, or a character, drawn from the model's fixed vocabulary.
Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Neural Network ✓ understood
A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.
Dataset ✓ understood
A collection of data examples used for training, validating, or testing machine learning models.
Feature ✓ understood
A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.
Deep Learning ✓ understood
A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.
Representation Learning ✓ understood
Letting a model learn its own internal description of the data, such as embeddings or layered visual features, instead of relying on features designed by hand.
Embedding ✓ understood
A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.
Attention Mechanism ✓ understood
A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step.
Self-Attention ✓ understood
An attention step in which every token in a sequence looks at every token in that same sequence and builds a new representation from a weighted mix of them.
Masking · you are here ✓ understood

Picture it

Masking what it sees

Attention masks block some positions
Causal mask: no peeking at later tokens
Padding mask: ignore filler tokens

Masking what it learns

Hide part of the input on purpose
Train the model to fill the gap
BERT hides 15% of tokens; MAE 75% of patches

The same word covers two jobs: controlling which positions a model may look at, and hiding data to create a training task.

Masking means deliberately hiding some positions from a model. The word covers two related jobs in modern machine learning, and it helps to keep them apart.

Masks that control attention

In a transformer, self-attention lets every token look at every other token. An attention mask switches some of those connections off, in practice by setting their attention scores to minus infinity before the softmax so they receive zero weight.

A causal mask stops each position from seeing the positions after it. The original transformer used this in its decoder so that predicting the next word can’t cheat by looking ahead, and every GPT-style model relies on it.
A padding mask tells the model to ignore the filler tokens added to make sequences in a batch the same length.

Here masking doesn’t change the data; it changes what the model is allowed to use.

Masks that create a training task

The second job hides part of the input on purpose and trains the model to reconstruct it, which turns unlabeled data into a supervised task. Masked language modeling, introduced with BERT, hides 15% of the tokens in each sequence and asks the model to predict them from context on both sides; see the BERT paper. Vision borrowed the idea: masked autoencoders hide 75% of an image’s patches and learn by filling them back in.

The difference that matters

A causal mask makes a model generative: it can only use the past, so it can write text one token at a time. A fill-in-the-blank objective with bidirectional attention makes a model a strong reader but not a natural writer. That split is much of the difference between GPT-style and BERT-style models.

Where it sits

Before this

Self-Attention Token

Masking

Leads to

Attention Mask Causal Mask

Explore nearby

Language & LLMs Attention Mask A binary mask indicating which tokens should be attended to, used to handle padding and causal masking. Language & LLMs Causal Mask An attention mask ensuring tokens can only attend to previous positions, crucial for autoregressive generation. Language & LLMs Masked Language Modeling A pre-training objective where random tokens are masked and the model learns to predict them from context. Language & LLMs Padding Token A special token used to make sequences the same length in a batch, typically ignored during computation. Language & LLMs Bidirectional Attention Allowing tokens to attend to both past and future context, used in encoder models like BERT.

Sources

Ashish Vaswani et al., "Attention Is All You Need" . NeurIPS 2017; masked decoder self-attention
Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" . NAACL 2019; the masked language modeling objective
Kaiming He et al., "Masked Autoencoders Are Scalable Vision Learners" . CVPR 2022