← All topics

Guide · Practice · Topic 32 of 32

Understanding and safety

Looking inside a model, knowing when not to trust it, and keeping people safe regardless.

7 minute readBuilds on: Robot policies today

Looking inside the model

These are tools for asking what a model has learned. They suggest hypotheses. An ablation or a controlled test confirms them.

ToolWhat it doesHow far to trust it
ProbingTrains a small classifier on internal features. If it can read out object position, the model encodes it.A cheap first check of what a layer knows
Attention and saliency mapsShow which pixels or tokens influenced an output.Good for spotting shortcuts; suggestive, not proof
Sparse autoencodersDecompose activations into many simple features, each often with a readable meaning.An active research area
AblationRemoves or zeroes a component, or masks an input such as one camera, and measures the change.The most direct causal evidence

A probe, run for real

A probe asks whether some fact can be read from a model’s internal features by a simple classifier. Here the question is whether a pretrained image encoder’s features contain the state of the gripper, something it was never trained to notice.

FeaturesAccuracy at telling whether the gripper is closed
Always guessing the more common answer69%
Raw pixels, 16 × 1681%
Frozen CLIP image embedding86%

This was measured on 1,516 LIBERO frames with whole episodes held out. The features do contain the information, since 86% is well above the 69% of guessing. They do not contain much more of it than a crude thumbnail does, which says the encoder’s summary vector is not where fine gripper detail lives.

The result illustrates the two standard cautions about probing. A probe can succeed because the answer is recoverable from something simple, which the pixel baseline exposes, so a probe without a baseline shows little. And a successful probe shows that information is present, not that the model uses it. That takes an intervention: remove or alter the feature and see whether the behaviour changes.

Knowing when not to trust it

A policy facing an unfamiliar scene should slow down or stop, and to do that the system has to notice.

  • Uncertainty. Estimate how sure the model is, with an ensemble, by sampling several outputs, or with calibrated confidence scores. This lets the system ask for help.
  • Out-of-distribution detection. Flag inputs unlike the training data, for example by comparing their embeddings with those seen in training, and trigger a safe stop or a human check.

Calibration and out-of-distribution detection, measured

A model is calibrated if its stated confidence matches how often it is right. The standard summary is the expected calibration error. Predictions are grouped into bins by confidence, and the gap between confidence and accuracy is averaged over the bins BmB_m:

ECE=∑m∣Bm∣n ∣acc(Bm)−conf(Bm)∣\mathrm{ECE} = \sum_{m} \frac{|B_m|}{n}\, \big| \mathrm{acc}(B_m) - \mathrm{conf}(B_m) \big|

A small classifier trained for this note on 150 images was right 94.4% of the time on held-out data while reporting an average confidence of 96.5%. It was overconfident, as networks trained to zero training loss generally are. The standard fix is temperature scaling: divide the logits by a single number TT chosen on held-out data. Here the best TT was 1.81, which brought average confidence down to 92.6% and the calibration error from 2.7% to 2.2%. It changes no prediction, only how sure the model claims to be.

Calibration holds only for inputs like the ones it was measured on. On unfamiliar inputs a model’s confidence means little, which is why a separate check for unfamiliarity is needed. The simplest, distance from the nearest training example in an embedding space, is measured in the note on failure triage: it separated frames of unseen tasks from familiar ones with an area under the curve of 0.93, catching 74% of them at a 5% false-alarm rate.

That last pair of numbers is the practical design problem. A monitor that stops the robot on 5% of perfectly normal frames is unusable at 10 frames a second, so run-time monitors average over a window, require several consecutive alarms, or are used to slow the robot down before they are allowed to stop it.

Physical safety sits outside the model

A learned policy will sometimes be wrong, so safety has to be enforced outside it.

Five stacked layers from outermost to innermost: human supervisor, runtime monitor, learned policy, low-level controller, hardware
Safety layers, outermost first. The policy is one layer, and it is assumed to be fallible.
  • Defence in depth. Use several independent layers, each able to stop the robot.
  • Lower layers are simpler. Limits in firmware and hardware are far easier to verify than a neural network.
  • Fail safe. On lost communication or stale commands, stop.
  • Limit energy. Cap speed and force near people. Collaborative robots are built for this.
  • Test the stop. Exercise the emergency stop regularly, and make sure new team members know where it is.

Never rely on the policy to be safe. Put hard limits underneath it and a human above it, and run any policy under test at reduced speed first.

The standards

Robot safety is governed by published standards, and a learned policy does not change which ones apply.

StandardCovers
ISO 12100How to assess and reduce risk for any machine
ISO 10218, parts 1 and 2Industrial robots and the systems built around them. The 2025 edition includes the requirements for working alongside people.
ISO/TS 15066Collaborative operation, including limits on force and pressure in contact with a person. Its content was folded into the 2025 edition of ISO 10218.
ISO 13482Personal care robots, including mobile and wearable ones
A stepped line of allowed robot speed against distance to the nearest person: zero inside a stop zone, limited in a middle zone, and full beyond it.
Speed and separation monitoring, one of the standard ways to share a space. The robot’s permitted speed depends on how close a person is.

The two approaches most used for shared spaces are in this family. Speed and separation monitoring slows and then stops the robot as a person approaches. Power and force limiting lets contact happen and keeps it below levels that cause harm. Both are enforced by certified safety hardware, below the learned policy and independent of it. This table is an orientation, not compliance advice.

Predicting failure at run time

A policy does not announce that it is about to fail, and separate monitors are used to notice.

  • Consistency of its own plans. Successive action chunks that disagree sharply suggest the policy is unsure.
  • Distance from the training data. Compare the embedding of the current image with those seen in training.
  • A second model as observer. A vision-language model asked “is the task progressing?” at intervals.
  • Plain limits. A time limit per sub-task, and a count of retries.

When a monitor fires, the response is to slow down, stop, or ask a person. Monitors raise false alarms, so measure how often on normal runs before relying on one.

Instructions are an attack surface

A robot that follows language can be told to do the wrong thing, by a person or by text in its camera view. A printed sign reading “ignore previous instructions” is an input like any other. The defences are structural: limit the tasks a robot will accept to an approved list, require confirmation for anything risky, and keep the physical limits where no instruction can reach them.

Common mistakes

  • Treating a saliency map as an explanation. It shows where the model looked, not why it decided.
  • Trusting raw confidence. Uncalibrated scores can be high on inputs the model has never seen.
  • Putting safety in the policy. A constraint the network learned is a tendency, not a guarantee.

Terms

Ablation
Removing a component to measure its effect.
Calibration
Agreement between stated confidence and actual accuracy.
Out of distribution
Unlike the data the model was trained on.
Emergency stop
A hardware control that cuts power to motion.
Speed and separation monitoring
Reducing a robot’s speed as a person gets closer, and stopping it at a set distance.
Power and force limiting
Designing and controlling a robot so that contact with a person stays below harmful levels.
Expected calibration error
The average gap between a model’s confidence and its accuracy, over bins of confidence.
Temperature scaling
Dividing a model’s logits by a constant fitted on held-out data, to correct its confidence.
False-alarm rate
How often a monitor fires on inputs that are in fact normal.

Further reading