Landmark 2 stops to get here · leads to 2

Self-Supervised Learning

Training on unlabeled data by making the data supply its own answers: hide or alter part of an input and have the model predict or match it.

Your route here

2 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Unsupervised Learning ✓ understood

    Learning from unlabeled data to discover hidden patterns, structures, or relationships without explicit target outputs.

  3. Self-Supervised Learning · you are here ✓ understood

Picture it

  1. 01 Unlabeled data e.g. raw text scraped at scale
  2. 02 Hide part of it "The cat [MASK] on the mat"
  3. 03 Predict the hidden part The data itself supplies the answer: "sat"
  4. 04 Learned representations Reused for downstream tasks via fine-tuning
Notice that no human labels appear anywhere: the task is carved out of the data itself, so it scales to huge unlabeled corpora.

Self-supervised learning trains a model on raw, unlabeled data by turning the data into its own answer key. Hide a word and ask the model to fill it in. Cover part of a photo and ask it to reconstruct the missing patch. The correct answer was there all along, so no human has to label anything.

Technically it’s a form of unsupervised learning, since no one supplies labels. In practice it trains exactly like supervised learning, with a prediction, a right answer and a loss. That combination is its advantage: the machinery of supervised training, applied to data sets far larger than anyone could label.

The main recipes

  • Predict what’s missing. BERT masks 15% of the tokens in each sentence and learns to recover them; that’s masked language modeling, the example in the diagram. GPT-style models do a variant: predict the next token from everything before it. Every sentence on the internet becomes thousands of training examples.
  • Rebuild the input. Masked autoencoders hide a large share of an image’s patches, 75% in the original paper, and learn to reconstruct the pixels.
  • Match views of the same thing. Contrastive learning takes two randomly altered copies of one image, cropped and colour-shifted, and trains the model to recognise them as the same while pushing apart copies of different images. SimCLR, in 2020, used this to match a supervised ResNet-50 on ImageNet with a simple linear classifier on top.

Where it fits

The hidden-word task isn’t the goal. It’s a pretext that forces the model to learn how language or images are structured, because you can’t reliably guess the missing piece without that knowledge. What survives is the internal representation. That’s the pre-training stage behind most foundation models, followed by fine-tuning on a small labeled set for the task you actually care about.

The catch

The pretext task has to be hard in the right way. Too easy and the model learns shortcuts; a masked image with only a few patches hidden can be recovered from neighbouring patches with little real understanding, which is why MAE hides so many. Contrastive methods, meanwhile, lean heavily on which augmentations you choose and benefit from very large batches. And whatever biases sit in the unlabeled data get absorbed along with everything else, because nobody curated it example by example.

Where it sits

Self-Supervised Learning

Explore nearby

In the research

All papers →

A paper that builds on Self-Supervised Learning .

Sources

  1. Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" . 2018; section 3.1, masked LM
  2. Chen et al., "A Simple Framework for Contrastive Learning of Visual Representations" . 2020 (SimCLR)
  3. He et al., "Masked Autoencoders Are Scalable Vision Learners" . 2021 (MAE)