← All topics

Guide · Models · Topic 10 of 32

Stable Diffusion and latent diffusion

One real image generator taken apart: the autoencoder, the text encoder, the denoiser, guidance and steps, and why robotics builds on it.

9 minute readBuilds on: Generative modeling, Attention in depth

The other backbone

Vision-language models understand images. Diffusion models produce them. Stable Diffusion, released in 2022, is the model that made image generation fast and open enough for everyone to build on, and its design, called latent diffusion, is the template for the image and video generators that robotics now uses as its second backbone.

The generative modeling note explains how diffusion works in principle. This note takes one real model apart. Every image below was produced by running Stable Diffusion 1.5 for this note.

Three parts

A text prompt goes into a text encoder of 123 million parameters, which passes 77 text tokens to a denoiser of 860 million parameters. Random noise of size 4 by 64 by 64 also enters the denoiser, which runs 4 to 50 times with its output fed back in. The clean latent then goes to a decoder, an autoencoder of 84 million parameters, which outputs a 512 by 512 image.
Stable Diffusion 1.5. The parameter counts were read from the model itself.
  • An autoencoder that compresses an image into a small grid of numbers, the latent, and rebuilds an image from one. It is trained first and then frozen.
  • A text encoder that turns the prompt into a sequence of vectors. It is the text half of a CLIP model, also frozen.
  • A denoiser that takes a noisy latent, the text vectors and the current noise level, and predicts the noise. It holds most of the parameters and is the only part trained for generation.

Generation runs the denoiser repeatedly on a latent that starts as pure noise, and decodes once at the end.

Why it works in a latent

A 512 by 512 robot camera frame beside the four channels of its latent, each 64 by 64. The channels are coarse, noisy-looking versions of the image in which the mugs, plates and robot can still be made out.
A real robot camera frame and the four channels of its latent. The layout of the scene survives. 786,432 numbers have become 16,384.

The autoencoder shrinks each side of the image by eight and keeps four channels, a reduction of 48 times in the amount of data. The denoiser therefore works on a 64 × 64 grid instead of 512 × 512, and that is the whole reason the model can run on an ordinary graphics card.

A robot camera frame, the same frame after passing through the autoencoder, and close-ups of the gripper from each. The rebuilt frame looks the same at full size. In the close-up, edges and fine markings on the gripper are slightly softened.
The same frame encoded and decoded with no diffusion involved. At full size the two match. Close up, fine edges on the gripper are slightly softened.

The compression has a price, and this round trip shows it. The rebuilt frame scores 35.3 dB against the original, which is good, and the loss is concentrated in exactly the small, sharp detail a robot cares about: edges of fingers, thin parts, small text. Whatever the autoencoder cannot rebuild, the generator can never produce. Before building on any latent model, encode and decode real images from the target domain and look closely.

Generation, one step at a time

Six images decoded from the latent at steps 1, 5, 10, 15, 20 and 25 of a 25-step generation. The first two are pure noise. At step 10 a rough shape appears. By step 20 a robot holding a mug is clear, and step 25 is clean.
One generation, decoded at six points along the way. The prompt was “a robot arm picking up a white mug from a wooden table, photograph”. Layout is settled in the middle steps and detail in the last ones.

The latent starts as random numbers. At each step the denoiser estimates the noise present, and a small rule called the sampler removes part of it. Early steps decide the composition: where things are and roughly what they are. Late steps add texture and sharp edges. This is worth knowing because it tells you where a change will take effect. Changing the prompt late alters detail and leaves the layout alone.

The result is also a lesson in what the model knows. The prompt asked for a robot arm. The model drew a friendly humanoid, because that is what “robot” looks like in the images it learned from.

The noise schedule

Left: over 1,000 noise steps the share of image in the mixture falls from 1 to 0.07 while the share of noise rises from 0 to nearly 1; the two cross near step 350. Right: the signal-to-noise ratio on a log axis falls from over 100 to below 0.01.
Read from the model’s own scheduler. Left: the two coefficients of the mixing formula, αˉt\sqrt{\bar\alpha_t} and 1−αˉt\sqrt{1 - \bar\alpha_t}. Right: their squared ratio, the signal-to-noise ratio.

Training picks a step tt at random, mixes the clean latent with noise in the proportions on the left, and asks the denoiser for the noise. This model uses 1,000 steps with β\beta rising from 0.00085 to 0.012. Signal and noise are equal near step 350.

The final value is worth noticing. At the last step αˉT\bar\alpha_T is 0.005, so about 7% of the image’s amplitude is still present. Training never shows the model pure noise, and sampling starts from pure noise. That mismatch is one reason this generation of models struggles with very dark or very bright images, and later models corrected the schedule or changed what the network predicts. Three targets are in use: the noise ϵ\epsilon, the clean latent x0x_0, and the velocity v=αˉt ϵ−1−αˉt x0v = \sqrt{\bar\alpha_t}\, \epsilon - \sqrt{1 - \bar\alpha_t}\, x_0, which behaves well at both ends of the schedule.

How the prompt gets in

The prompt is encoded once into 77 token vectors. Inside the denoiser, cross-attention layers let every position of the image latent read those vectors, so each region can attend to the words that concern it. The image is the queries and the text is the keys and values.

Four images from the same starting noise at guidance strengths 1, 3, 7.5 and 15. With guidance off the image is an incoherent tangle. At 3 it is a recognisable but muddled machine. At 7.5 it is a clean image of a robot holding a mug. At 15 it is simplified and oversaturated.
The same starting noise at four guidance strengths. Too little and the prompt is barely followed. Too much and the image becomes simplified and harsh.

Reading the prompt is not enough to make the model follow it closely. Classifier-free guidance supplies the rest. At each step the denoiser is run twice, once with the prompt and once with an empty prompt, and the result is pushed further in the direction that separates the two:

prediction = unconditional + scale × ( conditional − unconditional )

A scale of 1 is no guidance. Values around 5 to 8 are typical. It costs a second pass at every step, and it trades variety for obedience.

Steps, time and quality

Four images from the same starting noise using 4, 10, 25 and 50 steps, taking 0.15, 0.37, 0.91 and 1.83 seconds. At 4 steps the image is a blurred smear. At 10 it is a recognisable robot with errors. At 25 and 50 it is clean and nearly identical.
The same starting noise with four step counts, timed on a desktop GPU. Time is proportional to steps. Quality stops improving well before 50.

Each step is a full pass through the denoiser, twice with guidance, so time is proportional to steps: 0.15 seconds for 4 and 1.83 for 50 here. This model needs about 25. Four are not enough. Getting good results in very few steps takes a model trained for it, by distillation or with a flow-matching objective, and that is exactly the property a robot needs from a generator that runs inside a control loop.

Conditioning on more than text

A real robot camera frame followed by four versions produced by image-to-image generation at strengths 0.2, 0.4, 0.6 and 0.8. At 0.2 the scene is nearly unchanged. At 0.4 the mugs and plates gain new patterns. At 0.6 the gripper and objects change shape. At 0.8 the scene is a different image of cups on a marble surface.
Image-to-image on a real robot frame, with a prompt asking for a marble kitchen counter. Strength sets how much noise is added before denoising, and so how much of the original survives.

Image-to-image starts from a real image instead of pure noise. The image is encoded, noised to an intermediate step, and denoised from there with a new prompt. The strip shows the trade-off that makes this hard to use for robot data. At low strength nothing meaningful changes. At high strength the background changes, and so do the gripper and the objects, which means the recorded actions no longer match the picture.

Several mechanisms give finer control, and they are what generated data for robots depends on.

MechanismWhat it addsUse
InpaintingA mask: only the masked region is regeneratedChange a background or swap an object and leave the robot untouched
Structural controlAn extra network that feeds an edge map, depth map or pose into the denoiserKeep the geometry of a scene and change its appearance
Low-rank adaptersSmall trained matrices added to the attention layersTeach the model a new object or style from a few images
Image promptsFeatures of a reference image passed through cross-attentionGenerate variations of a specific object
Negative promptsThe unconditional branch of guidance is replaced with a prompt to avoidSteer away from unwanted content

For robot data the useful combination is inpainting or structural control with the robot and the manipulated object masked out, so that what the policy must attend to is preserved exactly and everything else varies.

What changed since

GenerationDenoiserText encoderObjectiveWhat it added
Stable Diffusion 1.xConvolutional U-Net with attentionCLIP text encoderPredict the noiseLatent diffusion, openly released
SDXLA larger U-NetTwo CLIP text encodersPredict the noiseHigher resolution and better composition
Diffusion transformersA transformer over latent patchesVariesDiffusion or flowScales like a language model; no convolutions in the denoiser
Stable Diffusion 3, FLUXA transformer with joint text and image attentionCLIP plus a large language-model encoderFlow matchingBetter prompt following and rendered text; fewer steps
Video modelsA transformer over space and timeA large language-model encoderMostly flow matchingAn autoencoder that also compresses time

Three trends run through the table. The denoiser moved from convolutions to transformers. The objective moved from noise prediction to flow matching. And the text encoder grew, because how well a model follows a prompt is limited by how well it reads it.

Why robotics builds on it

A model that can generate plausible images has learned a good deal about how the world looks and, for video, how it moves. Robotics uses that in four ways.

  • As a world model. A video generator conditioned on the robot’s actions predicts what the cameras will see next. The world models note covers this.
  • As a policy backbone. World-action models start from a pretrained video generator and train it to produce actions together with future frames.
  • As the method, not the model. Diffusion Policy and flow-matching action heads borrow the training recipe and apply it to action chunks, with no image generation involved.
  • As a source of data. Image and video generators change backgrounds, lighting and objects in existing demonstrations.

The properties learned on this small model carry over directly. The autoencoder sets what can be represented, so fine contact detail is at risk. The number of steps sets whether it can run in real time. And the model’s idea of the world comes from its training images, which contained many pictures of robots and very few taken by one.

Common mistakes

  • Skipping the autoencoder check. Encode and decode real frames from your domain before anything else.
  • Raising guidance to fix everything. Past a point it damages the image without improving obedience.
  • Assuming few steps will do. A model must be trained for few-step sampling. Otherwise quality falls off sharply.
  • Reading generated video as physics. Plausible is not the same as correct.

Terms

Latent diffusion
Running the diffusion process in an autoencoder’s compressed space instead of on pixels.
Latent
The compressed representation of an image produced by the autoencoder.
Denoiser
The network that predicts the noise in a noisy latent.
Sampler
The rule that turns the denoiser’s prediction into the next, less noisy latent.
Classifier-free guidance
Exaggerating the difference between prompted and unprompted predictions to follow the prompt more closely.
Diffusion transformer
A denoiser built from transformer blocks over latent patches.
Noise schedule
The rule that sets how much noise is mixed in at each step.
Signal-to-noise ratio
The ratio of image to noise power at a given step.
Inpainting
Regenerating only a masked region of an image.

Further reading