Self-supervised learning trains a model on raw, unlabeled data by turning the data into its own answer key. Hide a word and ask the model to fill it in. Cover part of a photo and ask it to reconstruct the missing patch. The correct answer was there all along, so no human has to label anything.
Technically it’s a form of unsupervised learning, since no one supplies labels. In practice it trains exactly like supervised learning, with a prediction, a right answer and a loss. That combination is its advantage: the machinery of supervised training, applied to data sets far larger than anyone could label.
The main recipes
- Predict what’s missing. BERT masks 15% of the tokens in each sentence and learns to recover them; that’s masked language modeling, the example in the diagram. GPT-style models do a variant: predict the next token from everything before it. Every sentence on the internet becomes thousands of training examples.
- Rebuild the input. Masked autoencoders hide a large share of an image’s patches, 75% in the original paper, and learn to reconstruct the pixels.
- Match views of the same thing. Contrastive learning takes two randomly altered copies of one image, cropped and colour-shifted, and trains the model to recognise them as the same while pushing apart copies of different images. SimCLR, in 2020, used this to match a supervised ResNet-50 on ImageNet with a simple linear classifier on top.
Where it fits
The hidden-word task isn’t the goal. It’s a pretext that forces the model to learn how language or images are structured, because you can’t reliably guess the missing piece without that knowledge. What survives is the internal representation. That’s the pre-training stage behind most foundation models, followed by fine-tuning on a small labeled set for the task you actually care about.
The catch
The pretext task has to be hard in the right way. Too easy and the model learns shortcuts; a masked image with only a few patches hidden can be recovered from neighbouring patches with little real understanding, which is why MAE hides so many. Contrastive methods, meanwhile, lean heavily on which augmentations you choose and benefit from very large batches. And whatever biases sit in the unlabeled data get absorbed along with everything else, because nobody curated it example by example.