Standard 11 stops to get here

Latent Diffusion

A diffusion model that runs its denoising in the compressed latent space of a pretrained autoencoder instead of on raw pixels, cutting compute while keeping detail.

Your route here

11 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  3. Diffusion Model ✓ understood

    A generative model trained to remove noise a step at a time, so it can turn pure random noise into a new image, audio clip or other sample.

  4. Unsupervised Learning ✓ understood

    Learning from unlabeled data to discover hidden patterns, structures, or relationships without explicit target outputs.

  5. Autoencoder ✓ understood

    An unsupervised neural network that learns to compress data into a latent representation and reconstruct it, useful for dimensionality reduction.

  6. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  7. Feature ✓ understood

    A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.

  8. Deep Learning ✓ understood

    A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.

  9. Representation Learning ✓ understood

    Letting a model learn its own internal description of the data, such as embeddings or layered visual features, instead of relying on features designed by hand.

  10. Embedding ✓ understood

    A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.

  11. Latent Space ✓ understood

    A compressed, learned representation space where similar data points are close together, used in autoencoders and VAEs.

  12. Latent Diffusion · you are here ✓ understood

Picture it

  1. 01 Encode An autoencoder squeezes the image into a small latent
  2. 02 Add noise Training corrupts the latent step by step
  3. 03 Denoise A U-Net learns to undo the noise, guided by the prompt
  4. 04 Decode The decoder turns the clean latent back into pixels
Notice that the expensive denoising loop never touches pixels: it runs on a compressed latent, and only the final decode returns to full size.

Latent diffusion is the trick that made high-resolution image generation affordable. A standard diffusion model learns to reverse a gradual noising process, and it does that work directly on pixels. Every denoising step touches every pixel, so training and sampling get expensive fast: Rombach and colleagues noted that training powerful pixel-space models often consumed hundreds of GPU days.

Latent diffusion splits the job in two. First, an autoencoder is trained once to compress images into a much smaller grid of numbers, a point in its latent space, and to decode them back with little visible loss. Then the diffusion model is trained entirely on those latents. The autoencoder handles fine perceptual detail; the diffusion model handles the content and layout of the image.

How much to compress

The paper labels each model by its spatial downsampling factor. Compress too little and you keep most of the cost; compress too much and detail is lost before diffusion ever sees it. Factors between 4 and 16 struck the best balance between efficiency and faithful results in their experiments.

Conditioning on a prompt

To steer generation, the denoising U-Net gets cross-attention layers that read an encoding of the condition, such as a text prompt or bounding boxes. The same design covered text-to-image synthesis, inpainting, semantic scene synthesis and super-resolution.

This is the recipe behind Stable Diffusion, which pairs latent diffusion with a pretrained text encoder and large-scale training. For a summary of the original work, see the latent diffusion paper.

Where it sits

Before this

Diffusion Model Autoencoder Latent Space
Latent Diffusion

Leads to

Nothing yet: a destination in its own right.

Explore nearby

Vision & Multimodal Stable Diffusion A latent diffusion model for text-to-image generation that operates in compressed latent space for efficiency. Neural Networks Variational Autoencoder A generative model that learns a probabilistic latent space, allowing sampling of new data points similar to training data. Vision & Multimodal U-Net A CNN architecture with encoder-decoder structure and skip connections, widely used for image segmentation tasks. Language & LLMs Cross-Attention Attention between two different sequences, where queries come from one and keys/values from another. Vision & Multimodal Image Generation Creating new images, often from a text prompt, with generative models such as GANs, VAEs and diffusion models that learn to turn random noise into realistic pictures. Vision & Multimodal CLIP Contrastive Language-Image Pre-training - a model jointly trained on images and text, enabling zero-shot image classification.

Sources

  1. Rombach, Blattmann, Lorenz, Esser and Ommer, "High-Resolution Image Synthesis with Latent Diffusion Models" . CVPR 2022 (arXiv 2021)
  2. Ho, Jain and Abbeel, "Denoising Diffusion Probabilistic Models" . NeurIPS 2020