← All topics

Guide · Data and hardware · Topic 25 of 32

Data quality and curation

What makes a good demonstration, how a large dataset is checked and scored, and how to measure what data is worth.

7 minute readBuilds on: Data for robots

Quality is a property of the dataset, not of the model

The same model trained on two datasets of the same size can differ more than two different models trained on one dataset. Data quality is therefore the cheapest lever in robot learning and the least glamorous. This note covers it in three parts: what a good demonstration is, how a dataset too large to watch is checked, and how to tell what a given batch of data was worth.

The note on data for robots covers where the data comes from, and the data flywheel covers data that arrives after deployment.

What makes a good demonstration

More data only helps if it is good data. Slow, hesitant or failed demonstrations teach the policy to be slow, hesitant and to fail. A good demonstration has seven properties, and most of them are decided by the instructions given to the operator before collection starts.

PropertyWhat it meansWhy it matters
ConsistentOne strategy per task, written into the operator instructionsTwo strategies for the same situation give the policy two answers to average
ObservableThe operator acts only on what the robot’s cameras showIf the operator looks over the robot’s shoulder, the data contains decisions the policy cannot reproduce
DecisiveSteady speed, with no idle pauses or fidgetingA policy copies hesitation, and pauses teach it that doing nothing is a valid action
FeasibleInside the robot’s joint, speed and gripper limitsMotions at the limits are clipped by the controller and recorded wrongly
Includes recoveryA small share of episodes show a slip or a bad grasp being correctedOtherwise the policy has never seen how to get back
Diverse where it mattersObjects, layouts and starting poses vary. The strategy does not.Variety in the scene generalises. Variety in the method confuses.
Cleanly recordedSensors in sync, steady frame timing, correct task labels and sub-task boundariesEvery one of these, when wrong, is a silent bug

“Consistent” and “diverse” pull in different directions only on the surface. Vary what the world can vary. Hold fixed what the operator chooses.

Curation at scale

Past a few hundred episodes nobody can watch everything, so quality has to be checked by program, with people looking at what the programs flag.

A scatter of 379 episodes: episode length relative to the median for its task, against the share of time spent not moving. Most episodes cluster near a length of 1.0 with almost no idle time. Twenty-two are marked as flagged, because they are unusually long or have unusual idle time.
Real data: every episode in the LIBERO set scored on two cheap measures. With thresholds at 1.4 times the task’s median length and at the 95th percentile of idle time, 22 of 379 episodes are flagged for a person to review.
StageWhat it does
Sanity checksTimestamp gaps, dropped frames, sensors out of sync, calibration drift. Reject or repair automatically.
Episode scoringSuccess label, duration against similar episodes, jerkiness, pauses, repeated grasps. The figure above uses two of these.
Consistency checksGroup the demonstrations of one task and flag outlier strategies and outlier operators.
De-duplicationEmbed every episode and drop near-copies, which add volume and no information.
Coverage mappingFind what is missing among objects, scenes and layouts, and use it to plan the next collection.
LabellingA vision-language model writes a first pass of instructions and sub-task boundaries. People check a sample.
Closing the loopFailures found in evaluation become the brief for the next round of collection.

The last row is what turns curation from cleaning into a strategy. The dataset is steered by what the policy gets wrong.

Scores computed on every episode

The scores used for curation are simple functions of the recorded motion. Let ptp_t be the hand position at step tt and Δt\Delta t the time between steps.

path length: ∑t∥pt+1−pt∥jerk: 1T∑t∥pt+3−3pt+2+3pt+1−ptΔt3∥2\text{path length: } \sum_{t} \| p_{t+1} - p_t \| \qquad\qquad \text{jerk: } \sqrt{\frac{1}{T} \sum_t \Big\| \frac{p_{t+3} - 3 p_{t+2} + 3 p_{t+1} - p_t}{\Delta t^3} \Big\|^2}

Path length measures directness. Jerk, the third derivative of position, measures smoothness: a hesitant or corrected motion has large jerk even when its path is short. Idle time is the share of steps with a near-zero command, and duration is the episode length. Each score is divided by the median for its task, because tasks differ.

Left: the hand path of a typical demonstration of one task, 137 centimetres long, over the paths of 25 others. Middle: the longest path for the same task, 195 centimetres, which wanders well outside the others. Right: path length against jerk for all 379 episodes, each relative to its task median, showing a cloud centred on one with no clear trend.
Real data. Left and middle: two demonstrations of one task, seen from above, over 25 others in grey. Right: every episode on two scores, each relative to the median for its task.

The middle panel is the kind of episode these scores exist to find. Its path is 42% longer than the median for the task and it wanders outside the envelope of the others. It may be a demonstration of recovery, which is valuable, or an operator hesitating, which is not, and only a person looking at it can say which. The scores decide what gets looked at.

The right-hand panel shows why one score is not enough. Path length and jerk are almost uncorrelated across these episodes, with a correlation of 0.05, so each flags different demonstrations. Path length and duration, by contrast, correlate at 0.63 and are partly redundant.

Measuring consistency

Consistency between demonstrations of the same task can be measured directly. Resample every demonstration to a common number of steps, and at each step compute the spread across demonstrations:

σ(s)=1N∑i=1N∥pi(s)−pˉ(s)∥2\sigma(s) = \sqrt{\frac{1}{N} \sum_{i=1}^{N} \big\| p_i(s) - \bar p(s) \big\|^2}

Low spread where the task needs precision, at a grasp or an insertion, is the sign of a clean dataset. The note on failure triage plots exactly this for one task. Two refinements are common. Aligning demonstrations in time before comparing them, by warping the time axis so that matching events line up, removes differences that are only in speed. And comparing operators against each other shows whether the spread comes from the task or from one person’s habits.

A learned alternative asks how predictable each demonstration’s actions are from its observations, given the rest of the dataset. Demonstrations whose actions are poorly predicted are the inconsistent ones. This is the idea behind estimating data quality from mutual information between states and actions.

Measuring what data is worth

Hours collected is the easy number to report and the wrong one to manage by. The question is how much each kind of data improves the policy.

Bars of success points gained per added demonstration as a dataset grows, for plain demonstrations and for demonstrations with recovery. Gains are between 1 and 3 points per demonstration up to 20 demonstrations and fall to under 0.1 between 100 and 200.
From the experiment in the data-efficiency note. The value of one more demonstration is high while data is scarce and falls by more than twenty times once the curve flattens. Recovery data is worth more early and no more later.
  • Ablate it. Train with and without a batch or a source and compare on the evaluation suite. This is the only direct measurement.
  • Compare properly. Use the same starting conditions for both checkpoints, interleave the trials in time, and report intervals, as the evaluation note describes. A data ablation judged on 20 trials measures noise.
  • Use cheap stand-ins first. Fine-tune a small model on subsets before committing to a full training run.
  • Estimate influence. Newer methods estimate which individual demonstrations helped or hurt specific outcomes without retraining for each one. They are approximate and good for finding harmful episodes.
  • Track a curve per source. Plot the gain as each source grows, as in the figure, and notice where it flattens.
  • Count in money. The useful unit is evaluation points gained per dollar, which lets a cheap, weak source be compared with an expensive, strong one.
  • Move the budget. Spend on the source whose curve is still climbing.

Common mistakes

  • Managing by hours collected. Track what each source adds to evaluation results per dollar.
  • Reviewing by sampling alone. A random sample finds common problems. Rare ones need a score computed on every episode.
  • Filtering without measuring. Check that removing the flagged episodes improves the policy. Sometimes the unusual ones were the recovery data.

Terms

Curation
Checking, scoring, filtering and labelling a dataset before training on it.
Ablation
Removing one thing and measuring the difference, to learn what it contributed.
Jerk
The rate of change of acceleration; a measure of how smooth a motion is.
Influence
An estimate of how much one training example changed a model’s behaviour on a given test case.

Further reading