← All topics

Guide · Robot learning · Topic 20 of 32

World models

Models that predict what happens next, and models that predict and act together.

7 minute readBuilds on: Generative modeling, Robot policies today

Learning to predict what happens next

A policy answers “what should I do?”. A world model answers “what will happen if I do this?”. It takes the current situation and a proposed action and predicts the outcome.

Two current camera frames plus proposed actions enter a world model, which outputs three future frames
The frames on the left are real observations. The frames on the right stand in for a prediction and are later frames of the same episode.

The idea is old in control: build a model of the dynamics and use it to plan. What is new is learning that model from video at scale, using the generative methods in the generative modeling note. An action-conditioned video model takes recent frames and a sequence of robot actions and generates the frames that would follow.

Robots want this for two reasons. They can try actions in imagination before the real world, and video is plentiful, so it teaches physics without anyone labelling actions. The bar is high, though. The model must get contact, occlusion and object permanence right, and it must show the consequences of the action, not just plausible video.

The objectives

Write oto_t for an observation, ata_t for an action and ztz_t for a compact latent state. A world model is a learned approximation to how the world evolves under actions:

in pixels: pθ(ot+1:t+H∣o≤t, at:t+H)in latents: zt+1=fθ(zt,at),zt=enc(ot)\text{in pixels: } p_\theta\big(o_{t+1:t+H} \mid o_{\le t},\, a_{t:t+H}\big) \qquad\qquad \text{in latents: } z_{t+1} = f_\theta(z_t, a_t), \quad z_t = \mathrm{enc}(o_t)

The pixel form is trained as a conditional generative model, usually by diffusion or flow matching in an autoencoder’s latent space, with the actions as conditioning. The latent form is trained by regression in representation space against a slow-moving target encoder:

Llatent=∥fθ(enc(ot), at)−enc‾(ot+1)∥2\mathcal{L}_{\text{latent}} = \big\| f_\theta\big(\mathrm{enc}(o_t),\, a_t\big) - \overline{\mathrm{enc}}(o_{t+1}) \big\|^2

A world-action model generates actions and future observations together, with one network and a joint objective:

LWAM=Lvideo(ot+1:t+H)+λ Laction(at:t+H)\mathcal{L}_{\text{WAM}} = \mathcal{L}_{\text{video}}\big(o_{t+1:t+H}\big) + \lambda\, \mathcal{L}_{\text{action}}\big(a_{t:t+H}\big)

Both terms are denoising losses of the same form, applied to different tokens of one sequence. An inverse dynamics model is the conditional the other way round, p(at∣ot,ot+1)p(a_t \mid o_t, o_{t+1}), which is why it is easier to learn: both ends of the motion are visible.

World-action models: predict and act in one model

A newer approach trains one network to produce both the future and the actions that cause it, where older designs kept the policy and the world model separate.

Frames, an instruction and joint state are encoded and enter a causal diffusion transformer, whose output is decoded into an imagined future and into an action chunk
Wiring of one published world-action model, DreamZero. Figure from my video on it.

The bet is that a model which can predict what its actions will cause has learned something a pure policy has not. Predicting the future gives a training signal on every pixel, or every latent value, of every frame, while imitating actions gives only a few numbers per step. At run time the future prediction can be skipped to save compute.

The main design choice is what to predict. Pixels are easy to inspect but spend capacity on appearance. Latents are cheaper and focus on what matters but are harder to check. This is the same debate as in representation learning. It is an active research direction and the results are still coming in, so treat it as a bet and not a settled answer.

Getting actions out of video

Video is plentiful and has no action labels. Two methods supply them.

Top: two video frames go into an inverse dynamics model, which outputs a seven-number robot action. Bottom: two frames go into an encoder that outputs one discrete code, and a decoder must rebuild the later frame from the earlier frame and that code.
Two ways to label video. The upper route outputs real robot actions and needs some robot data to train. The lower route invents its own action codes and needs none.
  • Inverse dynamics. Train a model on a small amount of robot data to answer: given this frame and the next, what action was taken? Then run it over unlabelled video. This is an easier problem than control, because the model sees the outcome.
  • Latent actions. Train an encoder to summarise the change between two frames as one of a small number of codes, and a decoder to predict the next frame from the current frame and that code. The codes end up meaning things like “move left” or “close”. A policy can be pretrained to predict them and then mapped to real actions with a little robot data.

Both turn human and internet video into something a robot policy can train on. Neither recovers force, and both inherit whatever the video leaves out of view.

Planning inside the model

A camera image is encoded to a state. Five imagined action sequences fan out from it, each ending in a predicted reward. The sequence with the highest reward is highlighted, and its first action is executed before planning again.
Planning without rendering a single frame. The rollouts happen in the model’s compact internal state.

A world model does not have to produce video to be useful. A latent dynamics model encodes the observation into a small state and predicts the next state and the reward directly. Because no pixels are generated, thousands of imagined futures cost little.

Two uses are established in reinforcement learning. One trains a policy entirely on imagined rollouts. The other plans at run time, as in the figure: sample action sequences, score them in the model, execute the first action of the best, and repeat. The second is model-predictive control with a learned model.

The baseline every world model must beat

A robot camera frame, the frame one second later, and two difference maps. After one second only the robot arm region has changed. After five seconds the arm and the object it moved have changed and the rest of the scene is still black.
Two real frames one second apart, and maps of which pixels changed after one and after five seconds. The table, the stove and the background do not move.
Left: the score of predicting that nothing changes, in decibels, against how far ahead is predicted. 25.6 at a tenth of a second, 20.1 at half a second, 18.3 at one second, 17.4 at five. Right: the share of pixels that changed, rising from 4 percent at a tenth of a second to 13 percent at one second and 17 percent at five.
Measured on 60 real episodes. Left: the image-similarity score earned by copying the current frame forward. Right: how much of the image actually changes.

In manipulation video, almost nothing moves. One second ahead, 13% of the pixels have changed, and after five seconds 17%. A model that outputs the current frame unchanged scores 25.6 dB at a tenth of a second and still 18.3 dB at one second, without having predicted anything.

Two consequences follow. A reported image-similarity score means little without this baseline beside it, because most of the score is the static background. And the pixels that do change are the arm and the object it touches, which are exactly the ones a policy cares about and the hardest to predict. A useful evaluation weights or crops to the region that moves, or bypasses pixels and asks whether the predicted video implies the right actions.

Three uses, and one caution

  • Evaluate policies. Roll a policy out inside the model and not on a robot. It is cheap, fast and safe, and useful for ranking checkpoints before real tests.
  • Generate data. Produce new scenes, lighting and viewpoints to stretch scarce demonstrations. The generated video needs actions that match it.
  • Plan. Imagine several action sequences, score the outcomes and execute the best. This is slow today and used mostly for high-level decisions.

The caution is that a learned model can produce physics that looks right and is wrong, and its errors grow with the length of the rollout. An optimistic simulator will rank bad policies highly. The check that matters is whether success inside the model predicts success on the robot, for the same set of policies.

How world models are scored

MeasureWhat it tells youWhat it misses
Frame similarity (PSNR, SSIM, LPIPS)How close each predicted frame is to the real oneRewards blur; punishes a plausible future that differs from the recorded one
FVDWhether the generated clips look like real ones as a setSays nothing about following the action
Action followingWhether the predicted video shows the commanded motion, for example by recovering the action from itNeeds a second model to judge
Policy rankingWhether policies that do well in the model do well on the robotCosts real robot trials to establish

Only the last row measures what a world model is for. The others are useful on the way there.

Common mistakes

  • Judging by how the video looks. Plausible is not the same as correct. Check outcomes against reality.
  • Long rollouts. Small errors compound, as they do for policies.
  • Ignoring the action. A model that produces the same future whatever the action is a video model, not a world model.

Terms

World model
A model that predicts future observations given actions.
Rollout
A sequence of predictions, each built on the last.
Action-conditioned
Predictions that depend on the action given.
World-action model
One model that outputs both future observations and actions.
Inverse dynamics model
A model that infers the action taken between two observations.
Latent action
A learned code that stands for the change between two frames, with no robot action label.
Model-predictive control
Planning a short sequence, executing its first action, and planning again.
Persistence baseline
Predicting that the future frame equals the current one; the score any world model must beat.
PSNR
Peak signal-to-noise ratio: a logarithmic measure of pixel error between two images, in decibels.

In my work

Further reading