← All topics

Guide · Practice · Topic 31 of 32

Deployment on the robot

Making the model small enough and fast enough, then learning from the field.

10 minute readBuilds on: Robot policies today

Making the model fit

A model that needs a data-centre GPU is no use on a battery-powered robot. There are three ways to shrink one.

  • Quantisation. Store and compute with fewer bits per number. It is the first tool to reach for.
  • Pruning. Remove weights, attention heads or whole layers that contribute little.
  • Distillation. Train a small student to copy a large teacher’s outputs.
Rows of coloured squares showing the sign, exponent and mantissa bits of FP32, BF16, FP16, FP8 and FP4 number formats
Number formats, bit by bit. Fewer bits mean less memory and faster inference, and less range or precision.

Sixteen-bit inference is usually free in practice. Eight-bit integers need representative calibration data to choose their ranges. Below that, care is needed, because of how low-bit scales behave.

A histogram of weights clustered near zero with a single outlier far to the right
One outlier stretches a 4-bit scale so far that almost all the weights are crushed into two levels.
A curve of quality loss rising steeply as bits per weight fall below four
The shape of the trade-off, drawn schematically: nearly free at 8 bits, careful work at 4, a cliff at 3.

Always re-measure task success afterwards. A tiny numerical error in each action can accumulate over hundreds of control steps, so compare closed-loop success, not just how far the outputs moved. On robot accelerators the supported operations and precisions are fixed by the hardware, so check them early.

Quantisation, written out and measured

Uniform quantisation maps a real number xx to one of 2b2^b integer levels using a scale ss and an offset zz:

q=clip ⁣(round(xs)+z, 0, 2b−1)x^=s (q−z)q = \mathrm{clip}\!\Big( \mathrm{round}\big(\tfrac{x}{s}\big) + z,\ 0,\ 2^b - 1 \Big) \qquad\qquad \hat x = s\,(q - z)

The scale is the range being covered divided by the number of levels, so the rounding error is at most s/2s / 2. Everything about quantisation follows from that: a wider range or fewer bits means a larger step. A single outlier that stretches the range makes every other value coarser, which is why ranges are usually set from percentiles of calibration data and not from the extremes, and why weights are often given a separate scale for each output channel.

Left: change in the policy's output against bits per weight, as a percentage of one action step, falling from about 290 percent at 2 bits to 6.5 percent at 8 bits and 0.02 percent at 16. Right: closed-loop success against bits per weight: 0 percent at 2 bits, 19 at 4, 70 at 5, 94 at 6, 98 at 8 and 99 at 16, against 99 at full precision.
Measured for this note: the weights of a small trained policy rounded to fewer bits, with no retraining. Left: how far its output moves. Right: what that does to success in the loop.

At 16 bits the policy is unchanged. At 8 bits its output moves by 6.5% of one step and success falls by one point, from 99% to 98%. Below that the decline is steep: 94% at 6 bits, 70% at 5, 19% at 4.

The two panels together make the point about closed-loop evaluation. At 6 bits the action error is a quarter of a step, which sounds serious, and the policy still succeeds 94% of the time, because each observation gives it the chance to correct. At 5 bits the error has only doubled and success has collapsed. The error measured on recorded data does not predict where that cliff is. Only running the policy does.

  • Post-training quantisation. Round a trained model, using a small calibration set to choose the ranges of the activations. It is what the figure does to the weights, and it is usually enough at 8 bits.
  • Quantisation-aware training. Simulate the rounding during training, so the weights settle where rounding hurts least. It recovers most of what is lost at 4 to 6 bits, at the cost of a training run.
  • Mixed precision. Keep the sensitive layers, often the first, the last and the normalisations, at higher precision, and quantise the rest.

The latency budget

The model’s answer always arrives late. The design goal is to make that harmless.

A timeline of one control cycle: camera frames, a single VLM encoding step, ten short action-expert steps, then fifty actions sent to the robot
One control cycle of the π0 model, from the timing table in its paper. Figure from my video on it.

In this example the images are encoded once and cached, the small action expert then runs ten short steps, and the result is fifty actions: one second of motion at 50 Hz, produced by 73 milliseconds of computation. The next chunk is ready long before the current one runs out.

  • Chunk and overlap. Each call returns a stretch of motion, so the next call can run while this one executes.
  • Asynchronous inference. Motion never pauses to wait for the model.
  • Real-time chunking. Plan the new chunk knowing it will arrive late, and blend it into the actions already committed.
  • Split the work. Run the large model a few times a second and only the small action network every cycle.

Latency is a correctness issue. An action computed from an old image is an action for a world that no longer exists.

Calculating latency: count the tokens

The time from an image to an action is the sum of four things: reading the sensors, preparing the inputs, running the model, and sending the command. The model usually dominates, and its time can be estimated before anything is built.

Model time = number of passes × time per pass. The time per pass is set mainly by how many tokens are in the sequence, so the first step is to count them. For images:

tokens = cameras × frames × (height × width) ÷ (patch size)²

A 224-pixel image cut into 14-pixel patches gives 16 × 16 = 256 tokens. Two cameras give 512. Models that work on compressed video divide further, by the autoencoder’s compression in space and in time. Add the instruction (5 to 30 tokens), the body state (one), and the action tokens being generated.

Measured time for one pass of a transformer against the number of tokens, on log axes, for a 12-layer and a 24-layer model at 32-bit and 16-bit precision. All four curves are nearly flat below about 256 tokens and then rise steeply. The 24-layer model at 32-bit takes 16 milliseconds at 512 tokens and 228 at 4,096.
Measured for this note on a desktop GPU (RTX 4080): one pass of a standard transformer with 85 million parameters (12 layers) and 302 million (24 layers). The numbers will differ on robot hardware. The shapes will not.

Three things can be read off this measurement. Below a few hundred tokens the time is flat: fixed overheads dominate, and trimming tokens buys nothing. Above that it climbs faster than the token count, because attention compares every token with every other. Going from 2,048 to 4,096 tokens multiplied the time by about 2.5, not 2. And switching from 32-bit to 16-bit numbers made the large model three to four times faster at every length.

The knobs that control it

Bars of image token count for six configurations, each with the measured time for one pass of the 24-layer model. Two cameras at 224 pixels: 512 tokens, 5 milliseconds. A third camera: 768, 8. One frame of history: 1,024, 10. Double the resolution: 2,048, 22. Merge every two by two patches: 128, 3. One camera: 256, 3.
What each choice does to the token count, with the measured time for one pass at 16-bit precision. Doubling the resolution costs four times the tokens. A camera or a frame of history costs its share.
KnobEffect on timeWhat it costs
Number of camerasTokens grow in proportionViewpoints, and with them occluded detail
Image resolutionTokens grow with the squareSmall objects and fine detail
Patch size, or merging tokensTokens fall with the squareFine detail
Frames of historyTokens grow in proportionMotion cues and short-term memory
Compression (video models)Fewer tokens per frameDetail the autoencoder cannot rebuild
Refinement stepsPasses grow in proportionSample quality, below a handful of steps
Depth and widthTime grows with depth, and faster with widthCapability
Number precision16-bit was 3 to 4 times faster here; 8-bit goes furtherAccuracy, if calibration is poor
CachingThe cameras are encoded once, not once per stepNothing, if the architecture allows it
Chunk lengthMore actions per call, so fewer callsReactivity
Three stacked bars of the time to produce one chunk of actions. Running the whole model for 10 steps: 53 milliseconds. Encoding the cameras once and running a small action network for 10 steps: 22. Encoding once and 4 small steps: 12.
The same measurements combined. Running the large model on every refinement step costs 53 ms. Encoding the cameras once and refining with a small network costs 22 ms at ten steps and 12 ms at four.

The last figure is the argument for the design described in robot policies today: a large model that reads the scene once, and a small one that turns it into actions over several quick steps. Two knobs, caching and step count, cut the time by more than four with the same large model.

Model time is not the same as reaction time. A robot that executes half a second of a chunk before planning again can take up to half a second, plus the model time, to respond to something new. The note on senses shows what that delay means for a slipping object. Shortening the stretch that is executed before re-planning is a latency knob too, and often the largest.

Where the model runs

Three stacked bars of time from image to motion. On a small on-board computer: 200 milliseconds, almost all in the model. On-board with an optimised model: 90 milliseconds. On a nearby server with a large GPU: 95 milliseconds, of which 35 are network.
Illustrative timings for one inference. Moving the model off the robot trades computation time for network time.
On the robotOn a nearby server
ComputeLimited by power, weight and heatAs large as needed
LatencyFixed and predictableDepends on the network, and varies
If the link dropsNothing changesThe robot must stop safely
Updating the modelEvery robot is updatedOne place to update

Many systems split the work: the large, slow model runs wherever compute is available, and the small, fast action network runs on the robot, so that a network stall degrades the plan and not the motion.

Export and runtimes

The framework a model is trained in is rarely what runs on the robot. The model is exported to an exchange format and then compiled by a runtime built for the target chip. Three problems come up every time.

  • Unsupported operations. The runtime may not implement every layer. Find out before the architecture is final.
  • Fixed shapes. Many runtimes want the batch size, image size and sequence length fixed at compile time.
  • Numerical differences. Compare the exported model’s outputs with the original on the same inputs, then compare task success.

Then profile each stage separately: image capture, preprocessing, the vision encoder, the action network, and sending the command. The slowest stage is often not the one expected, and preprocessing on the CPU is a frequent culprit.

After deployment

A deployed robot meets situations the training set never covered. Monitoring what happens, capturing the failures and feeding them back into training is the subject of its own note, the data flywheel. What deployment has to provide for it is logging: every episode recorded with its images, body state, commanded and achieved actions, timestamps and the release that was running.

Distillation

Distillation trains a small student to reproduce a large teacher. For a classifier or a token-based policy the student is trained on a mixture of the true labels and the teacher’s full output distribution, softened with a temperature TT:

L=(1−λ) CE(y, ps)+λ T2 KL(pt(T) ∥ ps(T))p(T)=softmax(z/T)\mathcal{L} = (1 - \lambda)\, \mathrm{CE}\big(y,\ p_s\big) + \lambda\, T^2\, \mathrm{KL}\big(p_t^{(T)} \,\|\, p_s^{(T)}\big) \qquad p^{(T)} = \mathrm{softmax}(z / T)

The teacher’s distribution carries more information than a label: it says which wrong answers are nearly right. For generative policies the same idea is applied to sampling. A student is trained to produce in one or two steps what the teacher produces in ten, which is how few-step action heads are obtained.

For robots there is a further option that needs no labels at all. Run the large policy, record what it does, and train the small one on those recordings. The student sees more data than any demonstration set contains, and inherits the teacher’s mistakes with it.

Versions and rollback

What runs on a robot is more than the weights. A release is the weights, the normalisation statistics, the preprocessing, the action convention and the controller settings together, and a mismatch between any two of them is a failure that looks like a bad model.

  • Version them as one unit. Record which release produced every episode, so a change in success can be traced.
  • Roll out in stages. One robot, then a few, then the fleet, with success compared against the previous release at each step.
  • Keep the way back. The previous release stays installed, and returning to it is one command.

Common mistakes

  • Checking output difference only. After quantising, measure whether the task still succeeds.
  • Measuring the model alone. The camera, preprocessing and communication all spend the budget too.
  • Shipping without monitoring. Then the first sign of a problem is a complaint.

Terms

Quantisation
Representing numbers with fewer bits.
Calibration
Choosing the numeric ranges for low-bit formats from sample data.
Latency
The delay between sensing and acting.
Drift
Deployment inputs slowly departing from the training data.
Runtime
The software that executes an exported model on a specific chip.
Release
The weights plus every setting needed to run them, versioned together.
Latency
The delay between an observation and the action that responds to it.
Sequence length
The number of tokens the model processes in one pass.
Calibration data
A small representative dataset used to choose quantisation ranges.
Quantisation-aware training
Training with rounding simulated in the forward pass, so the model adapts to it.
Distillation
Training a smaller model to reproduce the outputs of a larger one.

In my work

Further reading