← All topics

Guide · Robot learning · Topic 15 of 32

Robot learning basics

The control loop, action spaces, imitation learning and where reinforcement learning fits.

10 minute readBuilds on: Foundations that survived, Robotics fundamentals

The control loop

A policy is a function from what the robot senses to what the robot should do next, called many times a second.

A loop of four boxes: sense, policy, robot, world, connected by observation, action, physical effect and new state
One cycle. Each action changes what the robot sees next.

The observation is camera images plus the instruction. Proprioception is the robot’s own state: joint angles and how far the gripper is open. The action is the command sent to the robot. Learned policies typically run at 10 to 50 Hz, with a classical motor controller underneath running at 500 Hz or more.

This loop is the biggest conceptual shift for anyone coming from ordinary machine learning. A classifier’s test inputs do not depend on the classifier. A policy’s outputs change its own future inputs, so a small mistake moves the robot somewhere the model may never have seen.

Action spaces: what exactly does the policy output?

“Move the gripper to the cup” can be written as numbers in several ways. The choice affects how easily the policy learns, how errors build up, and whether data from two robots can be combined. There are four separate decisions.

Joints or hand

Joint spaceEnd-effector (Cartesian) space
The policy outputsA target for every jointA position and rotation for the hand
AdvantagesExact; no solver; the whole arm’s shape is controlled, which matters near obstaclesClose to how tasks are described; the same numbers mean the same thing on different arms
CostsTied to one robot; the link between a joint and the task is indirectNeeds inverse kinematics underneath; the elbow goes wherever the solver puts it; rotation formats differ
Typical useTwo-arm and humanoid systems trained on one bodySingle arms and cross-robot datasets

Absolute or relative

The same 21-second robot motion drawn two ways. Top: the hand's absolute position in three axes, as smooth curves drifting up to 30 centimetres from the start. Bottom: the relative command at each step in the same three axes, a series of bumps around zero.
One real episode, written both ways. The absolute form depends on where the task happens to be. The relative form looks the same wherever it is performed.

An absolute action says where to be. Errors do not build up, and the numbers depend on where the robot and the objects are, so the policy has to learn the workspace. A relative action, also called a delta, says how far to move from here. The same motion looks the same anywhere, which helps generalisation, and each small error is added to the last.

Relative to what

Position error over 100 steps for two ways of writing a relative action. When each action is relative to the previous step the error grows steadily. When each action is relative to the start of its 20-step chunk the error resets at every chunk boundary and stays small.
A simulation with the same small error on every predicted action. Measured from the previous step, the errors add up without limit. Measured from the pose at the start of the chunk, they are wiped out each time a new chunk begins.

A delta needs a reference, and there are two questions about it. The first is the starting point. Step-to-step deltas chain, so their errors accumulate. Policies that predict chunks usually express every action in the chunk relative to the pose at the moment of prediction, which stops the build-up at each new chunk. The second is the frame. “Move 2 cm forward” in the robot’s base frame and in the gripper’s own frame are different motions as soon as the gripper is tilted. Gripper-frame deltas transfer better between setups, and base-frame deltas are simpler to read.

Position, velocity, and what was recorded

Most learned policies output position targets and let the controller work out the speed. Some output velocities, which suits continuous motions and makes the result depend on timing. Separately, a dataset can store what was commanded or what was reached. Training on reached positions teaches the policy to ask for where the arm got to, which is always a little short of where it was sent, and the robot ends up moving sluggishly.

The gripper is its own small case: one number, open to closed, with the sign and the range differing between datasets. Before training on any dataset, write down its answer to all four questions. Mixing conventions does not raise an error. The robot just moves strangely. The fundamentals note covers the kinematics and rotation formats these choices rest on.

Normalise actions and state

The numbers in a robot action do not share a scale, and a network trained on them as recorded will quietly ignore the small ones.

Left: the recorded range of seven action values. The three position values span about plus or minus 0.8, the three rotation values about plus or minus 0.1 to 0.3, and the gripper the full plus or minus 1. Right: the share of a squared-error loss. As recorded, the gripper takes 74 percent, position 25 percent and rotation 0.9 percent. After normalising, the three groups are in proportion to their number of values.
Measured on 101,469 recorded actions from 379 LIBERO demonstrations. Left: the range of each value. Right: how much of a squared-error loss each group accounts for when every value is predicted equally well in relative terms.

In this dataset the gripper command is either −1 or +1, the position values are a few tenths, and the rotation values are a few hundredths. Under a squared-error loss, 74% of the training signal is about the gripper and under 1% is about rotation. The model would learn to open and close on time and never learn to turn the wrist.

The fix is to rescale every value to a common range before training, using statistics from the training set: subtract the mean and divide by the spread, or map the 1st and 99th percentiles to −1 and +1. The percentile form is less disturbed by rare extreme values. Three rules go with it.

  • Save the statistics with the model. They are part of the checkpoint. A model with the wrong statistics outputs confident nonsense.
  • Undo it on the way out. The policy predicts normalised actions. Convert them back with the same statistics before sending them to the robot.
  • Normalise the body state too. Joint angles and gripper width have the same problem as actions.

A mismatch here is among the most common silent bugs in robot learning. Fine-tuning a pretrained policy with new statistics, or evaluating with the statistics of a different dataset, produces a robot that moves smoothly to the wrong place.

Imitation learning and its weak point

The dominant way to teach robots today is imitation: record a person doing the task, then train the policy to copy. In its simplest form, behaviour cloning, this is ordinary supervised learning with observations as inputs and demonstrated actions as targets.

A dashed straight line for the demonstrated path and a solid line that drifts further away from it over time
Compounding error. The policy drifts into states the demonstrations never visited.

Its weakness is compounding error. Each small mistake changes the next observation. Away from the demonstrations the policy has no data on how to recover, so the errors grow. The fixes are more varied data, demonstrations of recovery, and action chunking.

Behaviour cloning in symbols, and why errors compound

Behaviour cloning fits a policy πθ\pi_\theta to a dataset D\mathcal{D} of observations and expert actions by maximum likelihood:

θ∗=arg⁡max⁡θ  E(o,a)∼D[log⁡πθ(a∣o)]\theta^{*} = \arg\max_\theta \; \mathbb{E}_{(o, a) \sim \mathcal{D}} \big[ \log \pi_\theta(a \mid o) \big]

With a Gaussian policy of fixed variance this is mean squared error on the action. The catch is in the expectation. It is taken over the states the expert visited. At test time the policy visits the states its own actions lead to, and nothing in the objective covers those.

Distance between where a cloned policy is and where the expert would be, against steps into the episode, for policies trained on 10, 50 and 200 demonstrations. With 10 demonstrations the gap grows throughout, reaching 41 centimetres at step 60. With 50 it peaks at 9 centimetres and falls back. With 200 it stays under 3.
Measured on the toy reaching task: policy and expert start from the same place, and the gap between them is tracked. Mean of four training runs and 300 starts.

The policy trained on 10 demonstrations has a one-step action error of 1.8 cm when it is shown states from expert demonstrations. Run on its own, it ends 41 cm from where the expert would be. The one-step error says almost nothing about that outcome, because each error moves the policy to a state slightly further from the data, where the next error is larger.

The classical analysis makes this precise. If the policy errs with probability ϵ\epsilon per step on the expert’s states, then over an episode of TT steps the expected number of mistakes can grow as

behaviour cloning: O(ϵ T2)with corrective data: O(ϵ T)\text{behaviour cloning: } O(\epsilon\, T^2) \qquad\qquad \text{with corrective data: } O(\epsilon\, T)

The quadratic term is the compounding. The linear bound is achieved by DAgger: run the current policy, ask the expert what it would have done in the states the policy reached, add those to the dataset, and retrain. The data then covers the states the policy actually visits. The note on the data flywheel measures the effect.

Action chunking

Three curves of real robot actions over twelve seconds, x motion, z motion and gripper, with alternating shaded bands marking chunks
Real actions from one episode of the LIBERO benchmark, cut into chunks of ten steps.

Instead of predicting one action, the policy predicts a short sequence of future actions in one call. This shortens the number of decisions in a task, keeps motion smooth, and gives a slow model time to compute the next chunk while the current one executes. Re-plan before the chunk ends to stay responsive. It is a simple trick, and it is a large part of why imitation learning started working well.

Chunking, measured

A chunking policy outputs HH future actions at once, πθ(at:t+H∣ot)\pi_\theta(a_{t:t+H} \mid o_t). Two parameters matter: the chunk length HH, and how many of those actions are executed before the policy looks again.

Success against chunk length from 1 to 32 actions, all executed before the policy looks again, for undisturbed episodes and for episodes with one push. Success is about 83 to 85 percent up to a chunk of 8, then falls to 71 percent at 32. The pushed episodes are 3 to 4 points lower throughout.
Measured on the toy task with 60 demonstrations. Every action in a chunk is executed before the next observation is used.

On this task chunks of up to eight actions cost nothing, and longer ones lose accuracy because the policy is predicting further ahead from the same observation. A single push mid-episode costs three or four points at every length, since the policy cannot respond until its chunk runs out. The toy has no pauses or hesitations in its demonstrations, which is where chunking earns its largest gains on real data, so this figure shows the cost side of the trade more clearly than the benefit.

When chunks overlap, several predictions exist for the same moment. Temporal ensembling averages them, with weights that favour either older or newer predictions:

at=∑iwi a^t(i)∑iwiwi=e−m ia_t = \frac{\sum_{i} w_i\, \hat a_t^{(i)}}{\sum_i w_i} \qquad w_i = e^{-m\, i}

where a^t(i)\hat a_t^{(i)} is the prediction for time tt made ii steps ago. It smooths the joins between chunks at the cost of blending plans that may disagree.

How much of the past to show

A policy can be given only the current frame, or the last several. More history sounds strictly better and is not. Past frames reveal velocity and what happened behind an occlusion. They also make it easy for the policy to learn “keep doing what I was doing”, the copying shortcut shown in the first note, which scores well on recorded data and stalls on the robot.

The common setting is therefore short: one or two frames of images, the current body state, and a predicted chunk of future actions. When a task truly needs memory, add it deliberately, and check that success in the loop went up.

Reinforcement learning and the layer underneath

Reinforcement learning learns by trial and error from a reward. It needs enormous numbers of trials, so it works where a fast, accurate simulator exists: walking, running and balance. It struggles where trials are slow or unsafe, rewards are hard to write, and contact is hard to simulate, which describes most everyday manipulation. Imitation leads there, and reinforcement learning is increasingly used afterwards to raise an imitation policy’s success rate.

A stack of four layers from top to bottom: learned policy, kinematics and planning, joint controller, motors and safety limits
Learned policies do not replace control engineering. They sit on top of it.

Common mistakes

  • Mixing action spaces. The most common silent bug in robot learning.
  • Collecting only perfect demonstrations. The policy then never sees how to recover from a mistake.
  • Judging by offline error. Low error on recorded data says little about closed-loop behaviour.
  • Skipping normalisation, or mismatching it. Check that the statistics used in training, saved in the checkpoint and applied on the robot are the same.
  • Chaining step-to-step deltas over a long chunk. Express a chunk’s actions relative to its starting pose.

Terms

Policy
The function from observations to actions.
Proprioception
The robot’s sense of its own joints.
Behaviour cloning
Supervised learning of actions from demonstrations.
Closed loop
Actions affect the next observation.
Normalisation statistics
The per-value mean and spread, or percentiles, used to rescale actions and state.
Delta action
An action given as a change from a reference pose, not as a destination.
Cartesian space
Describing the hand by its position and orientation, as opposed to by joint angles.
Covariate shift
The difference between the states a policy was trained on and the states it visits when run.
DAgger
Collecting expert labels in the states the learned policy reaches, and retraining on them.
Temporal ensembling
Averaging the overlapping predictions that successive action chunks make for the same moment.

In my work

Further reading