← All topics

Guide · Data and hardware · Topic 27 of 32

The data flywheel

Turning what a deployed robot gets wrong into the data that fixes it: what to capture, how to use it, and how to keep the loop honest.

10 minute readBuilds on: Data for robots, Data quality and curation, Deployment on the robot

What a data flywheel is

A policy trained on demonstrations is only as good as the situations those demonstrations covered. Once it is deployed, it meets the rest. A data flywheel is the loop that turns those encounters into training data: every failure in the field becomes a case the next release handles.

A loop of eight stages: deploy the current release, monitor outcomes and alarms, capture failures and take-overs, curate, label corrections and outcomes, train on old and new together, evaluate on a fixed suite, and release in stages, returning to deploy.
One turn of the wheel. The stages in orange are where most of the engineering effort goes, and where most flywheels stall.

The name comes from the way the loop feeds itself. A better policy can be trusted with more work, more work produces more data, and more data makes a better policy. The loop is simple to draw and hard to run, because each stage has a way of quietly failing. This note goes through them in order.

What to capture

A fleet produces far more data than anyone can store, review or train on, and most of it shows the policy doing what it already does well. The first design decision is what to keep.

KeepWhy it is valuableHow it is detected
Take-oversA person stepping in marks the exact moment the policy was wrong, and what they do next is a demonstration of the fixThe operator’s control input
FailuresShows what the policy cannot doSuccess checks, timeouts, safety stops, operator reports
Near missesRecovered slips and retries are failures that almost happenedRepeated grasps, long episodes, force spikes
Unfamiliar situationsPlaces the training data did not reachDistance from training data, low confidence, disagreement between predicted chunks
A random sample of everythingAn unbiased picture of normal operation, for monitoring and for evaluationChosen by chance, not by a trigger

The last row is easy to skip and important. Data chosen by triggers is biased toward what the triggers can see. A small random sample is the only way to notice the failures no monitor was built for.

Why corrections are worth more than new demonstrations

Success against rounds of data collection for two strategies that add the same amount of labelled data each round. Adding ordinary new demonstrations rises from 11 percent to 89 percent over six rounds. Adding corrections where the policy went wrong rises faster, to 76 percent after two rounds and 97 percent after five.
An experiment run for this note, on the task from the data-efficiency note. Both strategies start from the same 10 demonstrations and add 300 labelled steps per round. One adds new expert demonstrations. The other runs the current policy, takes the states where it failed, and asks the expert what it should have done there. Mean of five runs.

The two strategies cost the same in expert effort. After two rounds the flywheel is at 76% and ordinary collection at 55%. After five it is at 97% against 86%.

The reason is where the data comes from. Expert demonstrations show the states an expert visits, and an expert rarely gets into trouble. The policy’s mistakes take it to states the expert never saw, and those are the states it needs help with. Collecting corrections there puts the data exactly where the policy’s errors are. This is the idea behind a family of methods that began with DAgger, and it is why take-overs are the most valuable thing a fleet produces.

The experiment also shows the flywheel’s built-in limit. As the policy improved, its failure rate during collection fell from 86% in the first round to 2% in the last. Each turn of the wheel leaves fewer failures to learn from, so the same amount of useful data takes more and more robot hours to find. A flywheel does not spin faster for ever. It needs a growing fleet, or harder tasks, to keep its supply.

The loop in symbols

Let πi\pi_i be the policy after round ii, π∗\pi^{*} the expert, and Di\mathcal{D}_i the dataset. One turn of the loop is:

Di+1=Di∪{(s, π∗(s)):s∼dπi}πi+1=arg⁡min⁡π∑(s,a)∈Di+1ℓ(π(s), a)\mathcal{D}_{i+1} = \mathcal{D}_i \cup \big\{ (s,\ \pi^{*}(s)) : s \sim d_{\pi_i} \big\} \qquad\qquad \pi_{i+1} = \arg\min_\pi \sum_{(s, a) \in \mathcal{D}_{i+1}} \ell\big(\pi(s),\ a\big)

dπid_{\pi_i} is the distribution of states the current policy visits. That subscript is the whole idea: the new labels are collected where the policy goes, not where the expert goes. This is the DAgger algorithm, and it is what the experiment above implements.

In deployment the expert cannot label every state, so people intervene only when they judge it necessary. The policy runs until a person takes over, the person’s actions during the take-over are recorded, and control is handed back. The data is then of two kinds, and they are usually weighted differently in training:

L=∑(s,a)∈Ddemoℓ+wint∑(s,a)∈Dinterventionℓ+wauto∑(s,a)∈Dautonomousℓ\mathcal{L} = \sum_{(s, a) \in \mathcal{D}_{\text{demo}}} \ell + w_{\text{int}} \sum_{(s, a) \in \mathcal{D}_{\text{intervention}}} \ell + w_{\text{auto}} \sum_{(s, a) \in \mathcal{D}_{\text{autonomous}}} \ell

Intervention data is weighted up, because it is scarce and marks exactly where the policy was wrong. Autonomous data from just before a take-over is weighted down or dropped, since those are the actions that made the take-over necessary. Autonomous data from successful stretches can be kept at a low weight.

From captured episodes to a training set

A funnel on a log scale with illustrative numbers: 10,000 episodes run, 900 flagged by a monitor or an operator, 420 confirmed as a failure or take-over, 310 passing quality checks, and 310 labelled and added to training.
Illustrative numbers for one week. About three episodes in a hundred end up in the training set.
  • Confirm. Monitors raise false alarms. A person, or a second model, checks that a flagged episode is what the flag says.
  • Curate. The same checks as any dataset: synchronised sensors, no dropped frames, sensible duration. The note on robot data lists them.
  • Cut. Keep the stretch around the event, not the whole shift. For a take-over, that is a few seconds before it and the correction after.
  • Label. A take-over arrives with its own labels: the person’s actions. An autonomous failure has none, and needs either a person to demonstrate the right behaviour from that state or a method that can learn from an outcome alone.
  • File it. Tag every episode with the release that produced it, the task, and the failure type. Without the release, nobody can later tell whether a problem was fixed.

Training on it without breaking what worked

Success on old situations and on new situations for three models. Before the update: 93 percent on old, 36 percent on new. Trained further on the new data only: 54 percent on old, 43 percent on new. Retrained on old and new together: 96 percent on old, 97 percent on new.
A second experiment on the same task. A policy trained on one half of the workspace is updated with data from the other half. Training on the new data alone damaged what it knew and did not fix the new cases. Training on both fixed both.

The obvious way to use new data is to continue training on it. The figure shows what that does. Success on the old situations fell from 93% to 54%, and the new situations barely improved. A network trained only on new data moves its weights to fit that data and loses what the old data was holding in place. This is called catastrophic forgetting.

  • Train on old and new together. Either retrain from the pretrained starting point on the whole set, or keep mixing a share of old data into every batch.
  • Weight the new data up, within reason. Fresh corrections are few, so they are sampled more often than their numbers warrant. Too much, and the policy overfits to last week’s failures.
  • Keep every earlier fix in the test suite. A failure that was fixed once should stay fixed. The next section covers this.

Learning from episodes with no correction

Most of what a fleet does is unattended, so most episodes have no human correction attached. Two things can still be done with them.

  • Successes as demonstrations. A successful autonomous episode is a free demonstration. Used alone, this drifts toward the easy cases the policy already handles, so it is a supplement to corrections, not a replacement.
  • Outcomes as reward. If each episode is labelled only as succeeded or failed, reinforcement learning can use both kinds, learning which actions led toward success. This is how practice on the robot is turned into improvement, and it is covered in the reinforcement learning note.

Both depend on the success label being right. An automatic success detector that is wrong one time in ten teaches the policy that one failure in ten is the goal.

Choosing what to label

When labelling capacity is the bottleneck, the question is which episodes to spend it on. This is active learning, and three signals are used to rank episodes.

SignalHow it is computedFinds
UncertaintyDisagreement between an ensemble of policies, or between several samples from one generative policyStates where the policy does not know what to do
NoveltyDistance from the nearest training examples in an embedding spaceSituations the data never covered
OutcomeFailures, take-overs, long or repeated attemptsStates where the policy was demonstrably wrong

Outcome is the most reliable, because it does not depend on the model’s opinion of itself. Uncertainty and novelty find problems before they become failures, and they also flag harmless oddities, so they need a person or a cheaper check behind them. Whichever is used, keep the random sample alongside it. A selection rule can only find the failures it was designed to see.

Keeping the loop honest

A flywheel feeds a model’s own behaviour back into its training. That makes it easy to fool yourself, in four ways.

RiskWhat happensGuard
The test set leaksField failures are added to training, and the same failures are used to measure improvementA held-out evaluation set that is never trained on, refreshed when it has been looked at too often
Old fixes come undoneA new release repairs this month’s failure and reintroduces last month’sA regression suite: every fixed failure becomes a permanent test
The policy shapes its own dataIt stops attempting what it is bad at, so those cases vanish from the data and look solvedThe random sample; and track what is attempted, not only what succeeds
The conditions changed, not the policySuccess rises because the week’s work was easierCompare releases side by side on the same conditions, with intervals

The evaluation note covers how to compare two releases properly, and failure triage covers how to tell what a given failure was caused by before deciding that more data is the answer.

Measuring the flywheel itself

  • Loop time. The time from a failure being seen in the field to a release that handles it reaching the fleet. This is the flywheel’s speed, and usually the number most worth improving.
  • Yield. The share of captured episodes that end up in training. A low yield means the triggers are noisy or the curation is too slow.
  • Interventions per hour, by release. The outcome the whole loop exists to move.
  • Failures by type, by release. Whether each turn removed the failure it targeted.
  • Cost per point. What a point of improvement cost in robot hours, review time and compute, compared with other ways to spend the same money.

Teams tend to measure the size of the dataset. The numbers above measure whether the wheel is turning.

Common mistakes

  • Storing everything and reviewing nothing. Data that is never curated or labelled does not improve anything.
  • Fine-tuning on the newest data alone. It forgets. Train on old and new together.
  • Measuring on the failures you trained on. Keep a held-out set and a regression suite.
  • Expecting the wheel to speed up by itself. A better policy fails less, so each turn takes more deployment to supply.

Terms

Data flywheel
The loop in which a deployed model’s failures are captured and used to train its successor.
Take-over
A person assuming control from the policy; also called an intervention.
Catastrophic forgetting
Losing earlier abilities when a network is trained further on new data alone.
Regression suite
A fixed set of tests for failures that were fixed before, run on every release.
Loop time
The time from seeing a failure in the field to deploying a release that handles it.
Yield
The share of captured episodes that end up in the training set.
Active learning
Choosing which examples to label so that each label is as informative as possible.
Intervention weighting
Giving human corrections more weight than other data during training.

Further reading