← All topics

Guide · Practice · Topic 30 of 32

Failure triage

When the robot fails: how to tell whether the controller, the pipeline, the data or the policy is at fault.

6 minute readBuilds on: Robotics fundamentals, Robot learning basics, Evaluation

Find the layer before fixing anything

When a robot fails, the instinct is to retrain the policy. That is the slowest fix and often the wrong one. A learned policy sits on a stack: a controller underneath, a data pipeline feeding it, and a dataset behind it. A fault in any layer looks, from the outside, like a bad policy.

A flow of four yes or no questions. Did the robot do what was commanded? If not: controller, calibration or timing. Did the policy see what it saw in training? If not: training and deployment do not match. Is this situation in the data? If not: data coverage. Do the demonstrations agree here? If not: data consistency. Only if all four are yes: the policy or its training.
The order to check in. Each question is cheaper to answer than the one below it, and each rules out a whole layer.

Work down the list and stop at the first “no”. The order is by cost. The first two questions take minutes and need no training run. The last one can take days.

1. Did the robot do what was commanded?

Take the controller out of the question first. Log the actions the policy sent and the motion the robot actually made, and compare them.

Left: a scatter of commanded against achieved motion per step for 379 episodes, lying close to a straight line with correlation 0.97. Middle: commanded and achieved motion over time in recorded data, nearly on top of each other. Right: a simulated faulty controller where the achieved motion lags the command and is cut off at a limit.
Left and middle: real recorded data. Across 379 LIBERO episodes the achieved motion follows the command closely (correlation 0.97; a command of 1.0 moves the hand about 1.2 cm per step). Right: the same command passed through a simulated fault, a delay and a speed limit, to show what to look for.

A healthy system looks like the middle panel. Three patterns point at the controller or the hardware and not at the policy: the achieved motion lags the command, it is clipped at a limit the policy does not know about, or it is offset by a constant, which suggests calibration.

The decisive test is replay. Take a recorded demonstration and send its actions to the robot with no policy involved. If the replay fails, no policy trained on that data can succeed, and the problem is below the policy.

2. Did the policy see what it saw in training?

The second layer is everything between the sensors and the network. These faults are silent, and each has been met before in this guide.

CheckHow to check it
Camera placement and orderOverlay a live frame on a training frame. Confirm that the wrist and scene cameras are not swapped.
Image processingSame crop, resize, colour order and compression as in training.
Normalisation statisticsThe checkpoint’s statistics are the ones applied, in both directions.
Action conventionSame frame, same units, absolute or relative, same gripper sign.
TimingSame control rate; images and body state no staler than in training.

A useful shortcut is to feed the policy a training episode’s observations, offline, through the deployment code path. Its outputs should match the recorded actions about as well as they did in training. If they do not, the pipeline differs.

3 and 4. Is it in the data, and does the data agree?

If the robot executes commands faithfully and the inputs match, look at the dataset before the model.

Left: a heat map of where the robot's hand went across 35 demonstrations of one task, with a marked test position in an empty region. Right: hand height against progress through the episode for every demonstration, with one stage highlighted where the curves spread apart most.
Real data: 35 demonstrations of one LIBERO task. Left: where the hand went, with a hypothetical test position the data never visited. Right: hand height through the task. The demonstrations spread most around 40% of the way through.
  • Coverage. Was the failing situation ever demonstrated? Find the nearest training episodes to the failure, by object position or by image embedding. If the nearest ones are far away, the policy was never shown what to do, and the fix is to collect that case.
  • Consistency. Where the data does cover the situation, do the demonstrations agree? In the right-hand panel they differ by several centimetres at one stage. Wide spread where a precise motion is needed means the policy is being taught several answers. The fix is to clean or re-collect, not to train longer.

These two have opposite cures, which is why they are separate questions. More data fixes a coverage gap and can make a consistency problem worse.

Is this situation in the data? A measured check

Question 3 can be answered with a number. Embed the frames of the failing episode with a pretrained image encoder, and for each one find the distance to the nearest frame in the training set. Frames from familiar situations sit close to the data. Frames from unfamiliar ones do not.

Two histograms of distance from the nearest training frame. Frames from held-out episodes of tasks in the training set cluster near 0.02. Frames from tasks never in the training set are spread from 0.02 to 0.11 with a peak near 0.08. A threshold at 0.036 has 5 percent false alarms and catches 74 percent of the unfamiliar frames.
Measured on the LIBERO frames. The training set holds half the episodes of seven tasks. Green: frames from the other episodes of those tasks. Orange: frames from three tasks left out entirely.

Distance to the nearest training frame separates the two groups with an area under the curve of 0.93. A threshold that raises a false alarm on 5% of familiar frames catches 74% of the unfamiliar ones. The quarter that it misses are frames from new tasks that happen to look like old ones, typically the start of an episode before anything distinctive has happened.

In triage this gives a quick, objective answer. If the failing episode’s frames are far from the training set, the problem is coverage and the remedy is data. If they are as close as any held-out episode, coverage is not the problem and the next question applies. The same score, computed live, is a run-time monitor, as the note on understanding and safety describes.

It has two blind spots. It measures similarity of appearance as one particular encoder sees it, so a change the encoder ignores, such as a small shift in an object’s position, goes undetected. And it says nothing about whether the training data near that point was any good.

5. Only then, the policy

If all four answers are yes, the fault is in the model or its training: too little capacity, an objective that averages over valid choices, too much or too little history, a chunk that is too long to react, or a run that had not converged. Change one of these at a time and compare against the unchanged run on the same test positions.

Habits that make triage possible

  • Log everything, every episode. Images, body state, commanded and achieved actions, timestamps, and the release that was running. A failure that was not recorded cannot be diagnosed.
  • Keep a failure taxonomy. Label each failure with a stage and a cause, and count. The largest group is the next piece of work.
  • Reproduce before fixing. Find the starting conditions that fail repeatedly. A fix cannot be verified against a failure that happened once.
  • Change one thing. Then re-run the same failing cases.

Common mistakes

  • Retraining first. It is the most expensive test and it changes several things at once.
  • Collecting more data for a consistency problem. More conflicting demonstrations do not resolve a conflict.
  • Diagnosing from the success rate. The number says that something failed. Only the logs say what.

Terms

Replay
Sending a recorded episode’s actions to the robot with no policy in the loop.
Coverage
Whether the training data includes situations like the one being tested.
Consistency
Whether demonstrations of the same situation agree with each other.
Failure taxonomy
A fixed list of failure stages and causes used to label and count failures.
Nearest-neighbour distance
The distance from a sample to the closest example in a reference set; a simple measure of familiarity.

Further reading