← All topics

Guide · Data and hardware · Topic 26 of 32

Data efficiency and generalisation

How much data a robot needs, what makes it need less, and what the result carries over to.

7 minute readBuilds on: Robot learning basics, Data for robots

Two questions about any robot policy

Robot data is the most expensive ingredient in the field, so two questions follow every result. How much data did it take? That is data efficiency. What does it work on besides the data it saw? That is generalisation. They are linked: a method that generalises well needs fewer demonstrations of each new situation.

Neither has a single-number answer. Data efficiency is a curve, success against the amount of data. Generalisation is a set of separate tests, one for each way the world can differ from the training set.

A learning curve you can run yourself

Left: a square workspace with a round obstacle and several curved paths from start dots to target stars. Right: closed-loop success against the number of demonstrations for four training setups. All four rise from under 15 percent at 2 demonstrations to about 98 percent at 200, and the setup with both additions is highest throughout.
A small experiment, run for this note. A point robot must reach a target without touching the obstacle. A small network is trained by behaviour cloning and then run in the loop on 300 unseen start and target pairs. Lines are the mean of six runs.

The grey line is plain behaviour cloning. The other lines add one thing each. Recovery data means the demonstrator was nudged off course while recording, so the data shows how to get back. Better inputs means the policy is given the target and obstacle positions relative to itself, which is the form of the problem that matters.

DemonstrationsPlain+ recovery data+ better inputs+ both
1012%23%19%41%
2033%50%49%62%
5075%81%80%84%
20098%99%98%99%

Three things in this table hold well beyond the toy. The additions matter most when data is scarce: at 10 demonstrations they more than triple success. They combine. And they fade as data grows, because enough data teaches the same thing the hard way. A method that only looks better at one data size has not been shown to be more efficient.

The shape of the curve

Failure rate against number of demonstrations on log axes for the toy experiment, with a straight-line fit through the points from 10 demonstrations upward. The fitted slope is minus 1.24.
The plain behaviour-cloning curve from above, replotted as failure rate on log axes. From 10 demonstrations on, the points lie close to a straight line.

On log axes the failure rate falls along a straight line, which means it follows a power law:

failure rate≈c N−α⟺log⁡(failure)≈log⁡c−αlog⁡N\text{failure rate} \approx c\, N^{-\alpha} \qquad\Longleftrightarrow\qquad \log(\text{failure}) \approx \log c - \alpha \log N

Here α\alpha is 1.24. The exponent is the single number that summarises data efficiency. With α=1.24\alpha = 1.24, halving the failure rate takes about 1.75 times the data, and reaching 99% success would take around 500 demonstrations by extrapolation. A method that is more data-efficient shows up as a steeper line or one shifted down, and the comparison should be made on these axes, where a constant factor in data is a constant horizontal shift.

The fit is an extrapolation from five points on a toy, and it describes a regime, not a law: below about ten demonstrations the policy fails almost always and the line does not apply. Real studies find the same form, with variety of environments and objects on the horizontal axis in place of raw demonstration count.

What buys efficiency

Every lever below does the same job: it supplies knowledge from somewhere cheaper than new demonstrations.

LeverWhere the knowledge comes fromLimit
Pretrained vision and languageWeb images and text: what objects are and what words meanKnows nothing about forces or the robot’s body
Pretrained robot policyThousands of hours of other robots’ dataHelps most when the new robot and tasks resemble the old
Co-trainingTraining on the new data together with larger related datasetsThe mixture has to be weighted with care
Action chunkingFewer decisions per task, so fewer chances to driftLess reactive within a chunk
AugmentationKnown invariances: lighting, small shifts, colourCannot invent new behaviour
Recovery data and correctionsStates the policy actually reaches when it errsNeeds a person in the loop
Better observationsA wrist camera, relative actions, body stateHas to be decided before data collection
Simulation and generated dataPhysics engines and generative modelsThe gap to reality
Human videoPeople doing the taskNo robot actions; they must be inferred

The first two are why “pretrain once, adapt many times” took over robotics. The notes on foundation models and robot data cover them in detail.

What pretraining buys, measured

Success in a new layout against demonstrations collected there, for a policy trained from scratch and one pretrained on four other layouts. From scratch: 3 percent with 2 demonstrations, 13 with 10, 40 with 20, 79 with 50. Pretrained: 71 percent with 2 demonstrations and 77 with 50. With no new data the pretrained policy scores 70.
The toy task with the obstacle in a new place. One policy sees only demonstrations from the new layout. The other was first trained on 240 demonstrations from four other layouts. Mean of five runs.

The pretrained policy succeeds 70% of the time in the new layout before seeing a single demonstration of it, and 71% after two. Trained from scratch, two demonstrations give 3%, and it takes about 50 to catch up. This is the entire argument for foundation models in robotics, in one figure: the knowledge that transfers does not have to be collected again.

The figure also shows the limit. The pretrained curve is nearly flat. Adapting conservatively, with the old data mixed in to avoid forgetting, it gains only six points from 50 new demonstrations, and from scratch overtakes it at that point. Pretraining buys a starting level, and the last points still have to come from data in the target setting.

How much is typical

Published results give rough orders of magnitude, not rules. A single task learned from scratch on one robot usually takes from tens to a few hundred demonstrations. Adapting a pretrained generalist policy to a new task takes a similar count and tends to give a more robust result. Working in places the robot has never seen is a different matter and depends on variety far more than on volume.

One careful study of this (Lin and colleagues, 2024) collected over 40,000 demonstrations and found that success in new environments with new objects grew roughly as a power law in the number of different environments and objects trained on. More demonstrations in the same environment helped quickly at first and then stopped helping. Their practical recipe was many environments with a modest number of demonstrations each. The rule of thumb is to spend the collection budget on variety.

Generalisation is several things

Four robot camera frames in a row. The first is a training scene with groceries and a basket. The second has the same objects moved. The third has a different instruction. The fourth is a different scene with a stove and a moka pot.
Frames from the LIBERO dataset, used here to show the kinds of change. Each step to the right asks more of the policy.
What changesWhy it is hardWhat helps
Object positionsThe policy may have memorised where things wereVary positions during collection; use a wrist camera
Object instancesA new mug has a new shape, colour and handleMany instances per category; pretrained vision
Scene and lightingBackgrounds leak into what the policy attends toMany environments; augmentation
Instruction wordingThe same task can be phrased many waysA language-model backbone; paraphrased labels
Camera placementThe mapping from pixels to motion shiftsFix the cameras, or vary them on purpose
TaskNew combinations of known skills, or new skillsBroad multi-task data; the hardest of all
Robot bodyDifferent joints, reach and gripperCross-embodiment training; shared action spaces

“The model generalises” means little until the axis is named. Policies today are strongest on the first four rows and weakest on the last two.

Measuring it

  • Report the curve. Give success at several data sizes, not one. State how many demonstrations and how many hours.
  • Hold out one axis at a time. Test new positions, new objects and new scenes separately, so a failure can be traced.
  • Compare at equal data. A method given more or better data has not been compared.
  • Count pretraining. A model that needs 50 demonstrations after 10,000 hours of pretraining is efficient in new data, not in total data. Say which you mean.

The evaluation note covers how many trials each of these measurements needs.

Common mistakes

  • Quoting one data size. Efficiency is the shape of the curve. One point cannot show it.
  • Collecting volume without variety. A thousand demonstrations in one scene teach that scene.
  • Testing on the training layout. Success where the objects stood during collection says little about anywhere else.

Terms

Data efficiency
How much task success a method gets from a given amount of data.
Learning curve
Performance plotted against the amount of training data.
In-distribution
Test conditions that resemble the training data.
Out-of-distribution
Test conditions that differ from the training data along some axis.
Co-training
Training on a small target dataset together with larger related ones.
Few-shot
Adapting to something new from a handful of examples.
Power law
A relationship in which one quantity is proportional to a fixed power of another; a straight line on log-log axes.
Zero-shot
Performing a task or working in a setting with no examples of it.

In my work

Further reading