← All topics

Guide · Models · Topic 9 of 32

Generative modeling

Autoregression, diffusion and flow matching, and the latent spaces they work in.

9 minute readBuilds on: The transformer

Two ways to generate

A generative model learns a data distribution so that it can produce new samples from it. There are two main families.

AutoregressiveDiffusion and flow matching
How it generatesOne piece at a time: predict the next token, append it, repeatAll at once: start from noise and refine the whole sample over several steps
Training lossCross entropy on the next tokenRegression: predict the noise, or the direction toward clean data
StrengthsSimple, handles variable lengthSharp continuous outputs; represents several valid answers
LimitsOne step per token; continuous values must be binned firstNeeds several refinement steps per sample
Used forLanguage, reasoning, tokenised actionsImages, video, robot action chunks

Both appear in robotics. Action tokens are generated one at a time. Continuous action chunks are generated by diffusion or flow matching.

Diffusion and flow matching, step by step

Training destroys data with noise, and the model learns to undo it. The model never learns to draw a whole sample in one go. It learns one small denoising step, which is then applied repeatedly.

Two panels of two-dimensional samples and the paths they took from noise: wandering paths for diffusion, nearly straight paths for flow matching
A real toy model: one small network trained two ways on 2-D data with two clusters. Dots are final samples and faint lines are their paths from noise. Left, diffusion in 50 steps. Right, flow matching in 10.

Training. Take a real sample, pick a random noise level, mix the two, and ask the network to predict the noise, or the velocity that points from noise toward data. Sampling. Start from pure noise and step along the predicted direction. Conditioning. Feed the network text, images or robot state, and the same noise becomes a sample that fits the condition.

Diffusion wanders because fresh noise is injected at every step. Flow matching moves in nearly straight lines, so it needs far fewer steps. That difference is why recent robot policies favour it.

The two objectives, written out

Both are plain regression losses. What differs is the path between noise and data.

The diffusion forward process and the noise-prediction loss, typeset
Diffusion. The first line is the fixed process that adds noise. The second is the loss: the network sees the noisy sample, the step and the context, and predicts the noise that was added.
The flow matching path, its velocity, the loss and the sampling update, typeset
Flow matching. Choose a straight path between a noise sample and a data sample, note that its velocity is constant, and regress the network onto that velocity. Sampling is a few equal steps from noise to data.

Flow matching keeps diffusion’s regression trick and removes the noise schedule, which is one fewer thing to tune.

The mathematics in one place

Every generative model in use is fitted to maximise, or to bound, the probability of the data. The families differ in how they make that tractable.

Autoregressive

log⁡pθ(x)=∑tlog⁡pθ(xt∣x<t)\log p_\theta(x) = \sum_{t} \log p_\theta\big(x_t \mid x_{<t}\big)

The likelihood is exact, training is one parallel pass, and sampling takes one model call per token.

Diffusion

A fixed forward process adds Gaussian noise over TT steps. Its useful property is that any step can be reached directly from the data:

xt=αˉt x0+1−αˉt  ϵϵ∼N(0,I)x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\; \epsilon \qquad \epsilon \sim \mathcal{N}(0, I)

where αˉt\bar\alpha_t falls from 1 to nearly 0 as tt grows. The network ϵθ\epsilon_\theta is trained to recover the noise that was added:

Ldiffusion=Ex0, ϵ, t ∥ϵ−ϵθ(xt, t)∥2\mathcal{L}_{\text{diffusion}} = \mathbb{E}_{x_0,\, \epsilon,\, t}\, \big\| \epsilon - \epsilon_\theta(x_t,\, t) \big\|^2

This simple loss is a re-weighted form of a bound on the log-likelihood. Predicting the noise is also equivalent, up to scale, to predicting the score, the gradient of the log-density of the noisy data: ∇xtlog⁡pt(xt)=−ϵ/1−αˉt\nabla_{x_t} \log p_t(x_t) = -\epsilon / \sqrt{1 - \bar\alpha_t}. Sampling follows that gradient from noise toward data.

Flow matching

Define a straight path between a noise sample x0x_0 and a data sample x1x_1, and train a network vθv_\theta to output the velocity along it:

xt=(1−t) x0+t x1Lflow=Ex0, x1, t ∥vθ(xt, t)−(x1−x0)∥2x_t = (1 - t)\, x_0 + t\, x_1 \qquad\qquad \mathcal{L}_{\text{flow}} = \mathbb{E}_{x_0,\, x_1,\, t}\, \big\| v_\theta(x_t,\, t) - (x_1 - x_0) \big\|^2

Sampling integrates the ordinary differential equation dx/dt=vθ(x,t)\mathrm{d}x / \mathrm{d}t = v_\theta(x, t) from t=0t = 0 to 11, in the simplest case with Euler steps: x←x+vθ(x,t) Δtx \leftarrow x + v_\theta(x, t)\, \Delta t. The learned velocity is an average over all the straight paths that pass through a point, so the trajectories that result curve a little, and the straighter they are the fewer steps are needed.

Variational autoencoders

log⁡pθ(x)  ≥  Eqϕ(z∣x)[log⁡pθ(x∣z)]−KL(qϕ(z∣x) ∥ p(z))\log p_\theta(x) \;\ge\; \mathbb{E}_{q_\phi(z \mid x)} \big[ \log p_\theta(x \mid z) \big] - \mathrm{KL}\big( q_\phi(z \mid x) \,\|\, p(z) \big)

The first term rewards reconstruction and the second keeps the codes close to a simple prior. In the autoencoders used for latent diffusion the second term is given a very small weight, so they behave as compressors with a smooth latent space.

Flow matching against diffusion, measured

Left: sixty trajectories of a trained flow-matching model on a two-dimensional toy dataset shaped like two crescents, each moving from a random starting point to the data in eight steps along gently curved paths. Right: distance between generated samples and the data against the number of sampling steps for flow matching and two diffusion samplers. Flow matching is lower at every step count up to 16. At four steps it is five times closer than diffusion.
Trained for this note: the same small network learns a two-crescent dataset with a flow-matching loss and with a diffusion loss. Right: energy distance between 3,000 samples and the data. The dashed line is the distance between two sets of real data.

The two models have the same architecture and saw the same data for the same number of steps. With 32 or more sampling steps they are equally good. With few, they are not: at four steps flow matching is five times closer to the data than diffusion, and at one or two steps diffusion produces nothing usable. The paths on the left show why. They are nearly straight, so a few large steps follow them well.

The deterministic diffusion sampler also gets worse again beyond 32 steps here, a reminder that a sampler is a numerical method with its own errors, and that more steps are not automatically better.

Latent spaces: generate in a compressed world

Pixels pass through an encoder into a small latent grid, a generator works on the latent, and a decoder expands the result back to pixels
Modern generators work on a compact code and decode it at the end.

Pixels are expensive. A short 480p video clip is tens of millions of numbers. An autoencoder compresses it, often by eight times in each spatial direction and four times in time, and the generator works on that small grid. This is what makes video generation affordable.

The price is a quality ceiling set by the autoencoder. If it cannot reconstruct small objects, thin structures or fast motion, the generator cannot produce them. Robot grippers and contact points are exactly that kind of detail, so reconstruct real data first and look at it.

Autoencoders and tokenisers

The latent space above has to come from somewhere. It comes from an autoencoder: an encoder that squeezes the input into a small code and a decoder that rebuilds the input from it, trained together so the rebuilt input matches the original.

An image passes through an encoder to a small grid of vectors, then through a decoder to a rebuilt image. A second path maps each vector to its nearest entry in a codebook, giving a grid of integer token numbers.
One encoder, two kinds of code. The upper path keeps continuous vectors. The lower path replaces each vector with the number of its nearest codebook entry.
  • Continuous latents. A variational autoencoder, or VAE, also keeps its codes smooth and well spread, so that a point between two codes still decodes to something sensible. Diffusion and flow models generate in this space.
  • Discrete tokens. A vector-quantised autoencoder snaps each code to the nearest entry in a learned codebook and keeps only the entry’s number. An image becomes a short list of integers, which a language-model-style generator can predict one at a time.

The same idea is applied to robot actions. A chunk of actions is compressed into a few tokens, which is how a language model can output smooth motion without one token per number.

Making sampling controllable and fast

  • Guidance. Run the model with and without the condition and exaggerate the difference. Samples follow the prompt more closely, at the cost of variety.
  • Few-step sampling. Each step is a full pass through the network. Flow matching and better solvers cut fifty steps to under ten. Robot action heads commonly use four to ten.
  • Distillation. Train a student to match many teacher steps in one, trading a little quality for a large speed-up.

Few-step sampling is what lets these models run inside a robot’s control loop. An action generator that needs fifty passes cannot run at 10 Hz. One that needs four can.

Video: all at once, or frame by frame

Video adds a choice that images do not have. A model can denoise a whole clip together, which keeps it consistent from start to end and fixes its length. Or it can generate one frame, or a few, at a time, each conditioned on the ones before. That allows any length and lets new input arrive mid-stream, at the cost of errors building up.

Recent models combine the two by giving each frame its own noise level: frames already produced are clean, and frames still to come are noisy. This matters for robots, because a world model has to keep generating as new actions and observations arrive.

Scoring generated samples

Generated images and video have no single right answer to compare against, so the standard metrics compare populations. FID for images and FVD for video pass many real and many generated samples through a pretrained network and measure how far apart the two sets of features are. Lower is better.

These scores need thousands of samples to be stable, and they measure realism, not correctness. A video can score well and ignore the action it was conditioned on. Use them to track progress, and look at the samples.

Common mistakes

  • Judging a generator without checking its autoencoder. Blurry hands or grippers are often a reconstruction limit, not a generation failure.
  • Turning guidance up. Past a point, outputs become saturated and unnatural.
  • Counting steps as free. Every extra step is latency.

Terms

Latent
A compressed representation that a generator works in.
Denoising step
One refinement of a noisy sample toward clean data.
Guidance
A knob that trades variety for closeness to the condition.
VAE
An autoencoder that maps data to a smooth latent space and back.
Autoencoder
An encoder and decoder trained to compress an input and rebuild it.
Codebook
A learned list of vectors; a discrete token is the number of the nearest one.
Score
The gradient of the log-density of the data; the direction in which samples become more probable.
ODE sampler
A sampler that follows a deterministic differential equation from noise to data.
ELBO
A lower bound on the log-likelihood that can be optimised when the likelihood itself cannot.

In my work

Further reading