Image generation is the task of producing new images rather than analysing existing ones. A generative model learns what a collection of images looks like, statistically, and then produces fresh samples that could plausibly have come from it. Add conditioning, usually a text prompt, and you get text-to-image systems that draw “a lighthouse in a thunderstorm, oil painting” on request.
The hard part is scale. A 512×512 colour image has nearly 800,000 numbers, and almost every random arrangement of them is static. The model has to learn the tiny region of that space where real-looking images live.
Three families
The diagram shows that all three main approaches start from randomness and learn to turn it into an image. They differ in how that transformation is learned.
- GANs (2014) pit two networks against each other. A generator turns random noise into an image; a discriminator tries to tell its output from real photos. The original paper frames this as a two-player minimax game. GANs are fast, one pass per image, but notoriously unstable to train and prone to covering only part of the data.
- VAEs (2013) learn an encoder that compresses images into a smooth latent space and a decoder that maps points back to images. Sample a point, decode it, get a new image. Training is stable, but on their own VAEs tend to produce blurry results.
- Diffusion models learn to reverse a noising process. Training takes real images, adds noise in many small steps, and teaches a network to predict and remove it. Generation starts from pure noise and denoises step by step. The 2020 DDPM paper showed this could produce high-quality images, and diffusion became the dominant approach.
How text-to-image works today
Latent diffusion (2021) made diffusion practical by running it in the compressed latent space of a pre-trained autoencoder instead of on raw pixels, which cut compute sharply, and by adding cross-attention layers so text could steer every denoising step. It is the basis of Stable Diffusion. The prompt is turned into vectors by a text encoder, often one trained to match images with captions, such as CLIP. So a modern system combines a text encoder, a diffusion model and a VAE-style decoder: several of these families working together.
The catch
Generators reproduce the patterns in their training data, biases included. They have historically struggled with fine structure that needs exact consistency, such as legible lettering or the right number of fingers, though each generation of models improves. And because output can now be photorealistic, telling generated images from real ones has become its own problem.