← All topics

Guide · Robot learning · Topic 17 of 32

Robot policies today

How a model that understands images and language is turned into one that moves a robot.

9 minute readBuilds on: Generative modeling, Foundation models and adaptation, Robot learning basics

Why actions are generated, not regressed

The obvious way to train a policy is regression: show the model an observation and penalise the distance between its output and the demonstrated action. This fails for a simple reason. Demonstrations contain more than one good answer.

Two demonstrated paths pass an obstacle on either side; their average runs straight into it
Both demonstrations are fine. Their average is a collision.

One person steers left around an obstacle and another steers right. A model trained with mean squared error learns the average of the two, which goes straight through the obstacle. The fix is to treat action prediction as generation: start from random noise and refine it into one consistent chunk of future actions, conditioned on recent camera images and the robot’s state. Diffusion Policy showed in 2023 that this gives large gains on manipulation tasks. Most recent systems use flow matching for the same job because it needs fewer refinement steps. The note on generative modeling explains both.

Anatomy of a vision-language-action model

A vision-language-action model, or VLA, is a vision-language model that has been taught to output robot actions.

Camera images, an instruction and robot state pass through encoders into a VLM backbone and then an action head that outputs an action chunk
Read left to right. Inputs become tokens, a pretrained backbone reads them together, and an action head turns the result into motion.

Camera images go through one or two vision encoders. The instruction is tokenised as text. The robot’s own state is projected into a token. All of these enter a pretrained language-model backbone, which reads them together. An action head then turns the backbone’s output into a short sequence of future actions, called a chunk.

The division of labour matters. The vision encoder and backbone arrive already trained on web-scale images and text, so they know what a cup is and what “on the shelf” means. Robot data only has to teach the acting part: the state projector, the action head, and usually a fine-tune of the backbone.

A real example: π0

The pi-zero architecture: a PaliGemma vision-language model and a smaller action expert sharing one attention stream
Two transformers sharing one attention stream. Figure from my π0 video.

π0, published by Physical Intelligence in 2024, follows this pattern closely. The backbone is PaliGemma, a vision-language model of about 3 billion parameters built from the SigLIP vision encoder and the Gemma language model. Beside it sits a separate transformer of about 300 million parameters, the action expert, whose tokens attend to everything the backbone computed.

The action expert is trained with flow matching and produces a chunk of 50 continuous actions in about ten refinement steps. At 50 Hz that chunk is one second of motion. Most of the parameters are spent on understanding the scene; a small expert turns that understanding into movement.

The flow-matching action head, written out

Let A=(at,…,at+H−1)A = (a_t, \dots, a_{t+H-1}) be the chunk of actions to predict and oo the observation: images, instruction and body state. Training draws noise ϵ∼N(0,I)\epsilon \sim \mathcal{N}(0, I) and a time τ\tau between 0 and 1, forms a point on the straight line between noise and the true chunk, and trains the network to output the direction of that line:

Aτ=(1−τ) ϵ+τAL=EA, ϵ, τ∥vθ(Aτ, τ, o)−(A−ϵ)∥2A^{\tau} = (1 - \tau)\, \epsilon + \tau A \qquad\qquad \mathcal{L} = \mathbb{E}_{A,\, \epsilon,\, \tau} \big\| v_\theta(A^{\tau},\, \tau,\, o) - (A - \epsilon) \big\|^2

At run time the chunk starts as noise and is moved along the predicted direction in KK Euler steps of size 1/K1/K:

Aτ+1/K=Aτ+1K vθ(Aτ, τ, o)A^{\tau + 1/K} = A^{\tau} + \frac{1}{K}\, v_\theta(A^{\tau},\, \tau,\, o)

Ten steps is typical. Each step is a pass through the small action network only, because the observation has already been encoded and cached. Sampling τ\tau from a distribution that favours noisier values, instead of uniformly, is a common refinement: for control, getting the coarse shape of the motion right matters more than polishing its last details.

Two ways to output actions

A histogram over action bins, with the arithmetic 50 timesteps times 14 dimensions equals 700 tokens
Actions as tokens: every value becomes a choice among bins. One second of motion for a two-arm robot is 700 tokens.
Discrete action tokensContinuous action head
HowSort each action value into a bin and predict the bins like wordsA small network refines a whole chunk from noise, reading the backbone’s features
StrengthsNo new architecture; works with any language model; trains stablySmooth and precise; a few steps yield many actions
CostsOne token per value is slow; bins limit precisionAn extra module and loss; it must not degrade the backbone

Neither has won outright. Newer tokenisers compress a whole chunk into far fewer tokens, and several systems train with tokens but act with a continuous head.

Action tokens, and compressing them

The token route writes each action value as one of BB bins. On normalised real actions the mean rounding error is 0.0015 with 256 bins and 0.026 with 16, measured on the LIBERO data, so 256 is precise enough. The trouble is length. A 50-step chunk of 7 values is 350 tokens, generated one at a time, and neighbouring tokens are nearly identical, which makes the next token so easy to predict that training learns little from it.

Left: one action value over a five-second chunk, with reconstructions from 4 and from 8 frequency coefficients; the 8-coefficient version follows the recording closely. Middle: the size of its 50 frequency coefficients, concentrated in the first few. Right: the share of signal energy captured against the number of coefficients kept, across all chunks; 5 of 50 hold 95 percent.
Real action chunks from 200 LIBERO episodes, transformed with a discrete cosine transform. Robot motion is smooth, so almost all of it lives in a few low frequencies.

The remedy is the one used for images and audio: change to a basis in which the signal is sparse. Apply a discrete cosine transform along time, round the coefficients, and discard the zeros. Across these chunks, 5 coefficients out of 50 carry 95% of the energy. Rounding the coefficients leaves an average of 95 non-zero values per chunk out of 350, with a reconstruction error of about 1% of the action range. A further lossless compression step on the result, of the kind used for text, brings the count down again. This is the idea behind frequency-space action tokenisers, and it is what lets a token-based policy match a flow-matching one on dexterous, high-rate tasks.

Long tasks

A policy that maps the current image to the next second of motion can pick up a cup. Clearing a table needs a plan and a memory. Two notes cover this: long-horizon tasks for the methods (chunks, sub-task decomposition, reasoning before acting) and system architectures for how a slow planning model and a fast acting model are arranged.

One policy for many robots

No single robot has enough data. Training one policy on data from many different robots is how the field pools what exists, and it works because most of what a policy learns is not about the body: what objects are, what the instruction means, what the task requires. The body-specific part is the output.

Three designs for training one policy on several robots. Shared action space: every arm outputs the motion of its hand in one seven-value format. One padded vector: an action vector sized for the largest robot, with unused slots masked. A head per robot: a shared backbone with a small output module for each body.
Three ways to let one backbone drive different bodies. They are often combined.
  • A shared action space. Write every robot’s action as the motion of its hand. This is why end-effector actions dominate cross-robot datasets. It covers arms and breaks down for hands, legs and mobile bases.
  • One padded vector. Fix the action size to that of the largest robot and mask what a smaller robot does not use. It is simple, and the model has to learn from context which slots mean what.
  • A head per robot. Share the backbone and give each body a small input and output module. A new robot needs only a new head, trained on a little of its own data.

Whichever is used, three details decide whether it works. Tell the model which robot it is, through a token or through the body state, or it will average incompatible conventions. Normalise each robot’s actions with its own statistics. And weight the mixture so the target robot is not drowned out, as the note on robot data shows.

Transfers wellTransfers poorly
Recognising objects and scenesReach and workspace
Understanding the instructionGripper or hand geometry
The order of steps in a taskCamera placement
Rough motion toward a targetControl rate, forces and contact

The realistic claim is modest. Cross-robot training does not usually let a policy run on a body it has never seen. It means a new body needs far fewer demonstrations than starting from nothing. A further step is to predict something with no body in it at all, such as future video, tracked points or latent actions, and leave a small robot-specific module to turn that into motor commands.

Keeping what the backbone knew

Fine-tuning a vision-language model on robot data can damage the knowledge that made it worth starting from. The action loss is new, its gradients are large and unlike anything in pretraining, and they flow into the backbone. Three practices limit the damage.

  • Co-training. Keep training on web image-and-text tasks alongside the robot data, so the backbone is still held to what it knew.
  • Discrete targets for the backbone. Train the backbone on tokenised actions with ordinary cross entropy, a loss of the kind it was pretrained with, and train the continuous action head on its features.
  • Stopping the gradient. Do not let the action head’s loss flow back into the backbone at all. The backbone learns from its own token loss, and the head adapts to whatever features result.

The combination of the last two has been reported to train faster and to follow language better than letting everything flow. The general lesson is that how the losses are routed matters as much as which architecture is used.

  • Dual-system designs. A large, slow model understands the scene a few times a second while a small, fast model produces actions at control rate.
  • Embodied reasoning. Before acting, the model states the sub-task, the objects involved and where they are. This helps on long tasks and makes failures easier to explain.
  • Cross-embodiment training. One model is trained on data from many different robots, so a new robot needs less data of its own.

Common mistakes

  • Mixing action spaces. A dataset recorded as absolute joint positions fed to a policy that expects relative hand motion will not crash. The robot will simply move wrongly.
  • Trusting offline error. A policy with a low action error on held-out data can still fail when its own actions change what it sees next.
  • Adding refinement steps. More steps cost latency, and a late action is a wrong action.

Terms

Policy
The function from what the robot senses to what it does.
Action chunk
A short sequence of future actions produced in one call.
Action head
The part of the model that outputs actions.
VLA
A vision-language model trained to output robot actions.
Sub-task
One step of a longer task, small enough for the low-level policy to carry out.
Embodiment
A particular robot body: its joints, reach, hand and sensors.
Cross-embodiment training
Training one policy on data from several different robot bodies.
Action expert
A small network attached to a vision-language backbone that generates continuous actions.
Discrete cosine transform
A change of basis that writes a signal as a sum of cosines of increasing frequency.
Stop-gradient
Blocking a loss from updating part of a network.

In my work

Further reading