Looking inside the model
These are tools for asking what a model has learned. They suggest hypotheses. An ablation or a controlled test confirms them.
| Tool | What it does | How far to trust it |
|---|---|---|
| Probing | Trains a small classifier on internal features. If it can read out object position, the model encodes it. | A cheap first check of what a layer knows |
| Attention and saliency maps | Show which pixels or tokens influenced an output. | Good for spotting shortcuts; suggestive, not proof |
| Sparse autoencoders | Decompose activations into many simple features, each often with a readable meaning. | An active research area |
| Ablation | Removes or zeroes a component, or masks an input such as one camera, and measures the change. | The most direct causal evidence |
A probe, run for real
A probe asks whether some fact can be read from a model’s internal features by a simple classifier. Here the question is whether a pretrained image encoder’s features contain the state of the gripper, something it was never trained to notice.
| Features | Accuracy at telling whether the gripper is closed |
|---|---|
| Always guessing the more common answer | 69% |
| Raw pixels, 16 × 16 | 81% |
| Frozen CLIP image embedding | 86% |
This was measured on 1,516 LIBERO frames with whole episodes held out. The features do contain the information, since 86% is well above the 69% of guessing. They do not contain much more of it than a crude thumbnail does, which says the encoder’s summary vector is not where fine gripper detail lives.
The result illustrates the two standard cautions about probing. A probe can succeed because the answer is recoverable from something simple, which the pixel baseline exposes, so a probe without a baseline shows little. And a successful probe shows that information is present, not that the model uses it. That takes an intervention: remove or alter the feature and see whether the behaviour changes.
Knowing when not to trust it
A policy facing an unfamiliar scene should slow down or stop, and to do that the system has to notice.
- Uncertainty. Estimate how sure the model is, with an ensemble, by sampling several outputs, or with calibrated confidence scores. This lets the system ask for help.
- Out-of-distribution detection. Flag inputs unlike the training data, for example by comparing their embeddings with those seen in training, and trigger a safe stop or a human check.
Calibration and out-of-distribution detection, measured
A model is calibrated if its stated confidence matches how often it is right. The standard summary is the expected calibration error. Predictions are grouped into bins by confidence, and the gap between confidence and accuracy is averaged over the bins :
A small classifier trained for this note on 150 images was right 94.4% of the time on held-out data while reporting an average confidence of 96.5%. It was overconfident, as networks trained to zero training loss generally are. The standard fix is temperature scaling: divide the logits by a single number chosen on held-out data. Here the best was 1.81, which brought average confidence down to 92.6% and the calibration error from 2.7% to 2.2%. It changes no prediction, only how sure the model claims to be.
Calibration holds only for inputs like the ones it was measured on. On unfamiliar inputs a model’s confidence means little, which is why a separate check for unfamiliarity is needed. The simplest, distance from the nearest training example in an embedding space, is measured in the note on failure triage: it separated frames of unseen tasks from familiar ones with an area under the curve of 0.93, catching 74% of them at a 5% false-alarm rate.
That last pair of numbers is the practical design problem. A monitor that stops the robot on 5% of perfectly normal frames is unusable at 10 frames a second, so run-time monitors average over a window, require several consecutive alarms, or are used to slow the robot down before they are allowed to stop it.
Physical safety sits outside the model
A learned policy will sometimes be wrong, so safety has to be enforced outside it.

- Defence in depth. Use several independent layers, each able to stop the robot.
- Lower layers are simpler. Limits in firmware and hardware are far easier to verify than a neural network.
- Fail safe. On lost communication or stale commands, stop.
- Limit energy. Cap speed and force near people. Collaborative robots are built for this.
- Test the stop. Exercise the emergency stop regularly, and make sure new team members know where it is.
Never rely on the policy to be safe. Put hard limits underneath it and a human above it, and run any policy under test at reduced speed first.
The standards
Robot safety is governed by published standards, and a learned policy does not change which ones apply.
| Standard | Covers |
|---|---|
| ISO 12100 | How to assess and reduce risk for any machine |
| ISO 10218, parts 1 and 2 | Industrial robots and the systems built around them. The 2025 edition includes the requirements for working alongside people. |
| ISO/TS 15066 | Collaborative operation, including limits on force and pressure in contact with a person. Its content was folded into the 2025 edition of ISO 10218. |
| ISO 13482 | Personal care robots, including mobile and wearable ones |

The two approaches most used for shared spaces are in this family. Speed and separation monitoring slows and then stops the robot as a person approaches. Power and force limiting lets contact happen and keeps it below levels that cause harm. Both are enforced by certified safety hardware, below the learned policy and independent of it. This table is an orientation, not compliance advice.
Predicting failure at run time
A policy does not announce that it is about to fail, and separate monitors are used to notice.
- Consistency of its own plans. Successive action chunks that disagree sharply suggest the policy is unsure.
- Distance from the training data. Compare the embedding of the current image with those seen in training.
- A second model as observer. A vision-language model asked “is the task progressing?” at intervals.
- Plain limits. A time limit per sub-task, and a count of retries.
When a monitor fires, the response is to slow down, stop, or ask a person. Monitors raise false alarms, so measure how often on normal runs before relying on one.
Instructions are an attack surface
A robot that follows language can be told to do the wrong thing, by a person or by text in its camera view. A printed sign reading “ignore previous instructions” is an input like any other. The defences are structural: limit the tasks a robot will accept to an approved list, require confirmation for anything risky, and keep the physical limits where no instruction can reach them.
Common mistakes
- Treating a saliency map as an explanation. It shows where the model looked, not why it decided.
- Trusting raw confidence. Uncalibrated scores can be high on inputs the model has never seen.
- Putting safety in the policy. A constraint the network learned is a tendency, not a guarantee.
Terms
- Ablation
- Removing a component to measure its effect.
- Calibration
- Agreement between stated confidence and actual accuracy.
- Out of distribution
- Unlike the data the model was trained on.
- Emergency stop
- A hardware control that cuts power to motion.
- Speed and separation monitoring
- Reducing a robot’s speed as a person gets closer, and stopping it at a set distance.
- Power and force limiting
- Designing and controlling a robot so that contact with a person stays below harmful levels.
- Expected calibration error
- The average gap between a model’s confidence and its accuracy, over bins of confidence.
- Temperature scaling
- Dividing a model’s logits by a constant fitted on held-out data, to correct its confidence.
- False-alarm rate
- How often a monitor fires on inputs that are in fact normal.