Six principles that outlast any model
Model names change every few months. These habits do not, and they decide whether a project works more often than the choice of architecture does.
- Data beats clever algorithms. The scale, variety and quality of the data set the ceiling. A better dataset usually helps more than a new architecture, and the largest recent gains in robotics came from more varied data.
- General methods that scale win. This is Rich Sutton’s “bitter lesson”: methods that improve with more data and compute keep overtaking methods that encode human knowledge by hand. Prefer designs that get better as the data grows.
- All models are wrong. A good score on held-out data is not a working robot. Judge a model in the loop it will actually run in.
- Failing models don’t throw exceptions. A broken pipeline still produces a loss curve. The only way to know is to look at the inputs, labels and outputs yourself.
- Evaluation leaks. Every choice made against the test set moves a little of it into the model. Public benchmarks may already be in the pretraining data.
- One variable per experiment. Change one thing, keep a matched control, and know what a trivial baseline scores before you start.

Silent failure: what it looks like
Deep learning code fails quietly. Nothing crashes, the loss goes down, and the model has still learned the wrong thing. Four failures account for most of these cases.

| Failure | What happened | Symptom |
|---|---|---|
| Label or input bug | Images and labels are misaligned, normalisation is wrong, or two cameras are swapped | The loss falls slowly and levels off high |
| Data leak | Test samples, or near-copies of them, are in the training set | The validation score looks too good |
| Shortcut learning | The model uses a cue that happens to correlate with the answer | Great on the benchmark, fails in new scenes |
| Evaluation bug | The metric is computed on the wrong split or the wrong output | The numbers do not match what you see |
Shortcut learning deserves a closer look because it is the hardest to spot. A classifier can learn the background in place of the object. A robot policy can learn to copy its previous action from its own joint readings and look fine on recorded data, then freeze on the real robot.

That is why the dumb baseline matters. On recorded data, “repeat the last action” is a strong score to beat, and a model that only matches it has learned nothing about the task.
How much of a result is noise
Before asking whether a change helped, ask how much the number moves when nothing changes. Training is random in its initial weights, in the order of the data and in dropout, and that randomness shows up in the final score.

Here the same code on the same data gives anything from 90.3% to 94.2%. A “1.5-point improvement” from a single run of each configuration would be well inside that spread, and would mean nothing. Three practices follow.
- Run several seeds. Report the mean and the spread. Three is a minimum; five or more for a claim that matters.
- Compare the gap with the spread. A difference between two means is only interesting when it is large compared with , the standard deviation of the runs divided by the square root of how many there were.
- Fix what can be fixed. Use the same seeds, data order and evaluation set for both arms, so that the remaining difference is the change being tested.
On a robot the same problem is larger, because each evaluation is a few dozen physical trials on top of the training noise. The note on evaluation works out how many trials a comparison needs.
Numbers you can check before trusting a run
Several quantities have a known correct value. Checking them costs minutes and catches most broken pipelines.
| Check | What it should be | What a wrong value means |
|---|---|---|
| Loss at the first step | For a classifier with balanced classes, . The small network above starts at 2.308, and . | A much larger value points to bad initialisation or unnormalised inputs; a much smaller one to a label leaking into the input |
| Overfitting a single batch | Loss goes to zero and the model reproduces those samples exactly | If it cannot, there is a bug in the model, the loss or the labels |
| A shuffled-label run | Validation performance at chance | Anything better means information is leaking from the evaluation set |
| Input ablation | Blanking an input the task needs makes performance collapse | If nothing changes, the model is not using that input |
| Predicted against recorded, by eye | Plots of predicted and recorded actions for whole episodes overlap | A constant offset or a lag points to normalisation or timing |
Leakage in robot data
Leakage is information about the evaluation set reaching the model. Robot datasets make it easy, because consecutive frames are nearly identical and episodes from one session share almost everything.
- Split by episode, never by frame. Frames 100 and 101 of one episode are the same image. A random split by frame puts near-copies of every validation frame in the training set, and validation loss then measures memory.
- Then ask what else is shared. Episodes from the same scene, object set or operator share more than the split suggests. To claim generalisation to new scenes, hold out whole scenes.
- Compute statistics on the training split only. Normalisation means and ranges computed over all the data carry information from the validation set.
- Count the choices made on the test set. Every checkpoint, threshold or hyperparameter chosen by looking at test results fits the test set a little. Keep a final set that is looked at once.
Habits that catch it
- Look at the data. View raw samples after every transform, not before.
- Overfit one batch. If the model cannot memorise ten samples, there is a bug.
- Beat a dumb baseline. Compare with “predict the mean” or “repeat the last action”. If you cannot beat it, stop and find out why.
- Ask what shortcut exists. Could the model score well without solving the task? Design a test that separates the two.
- Test in the loop. Run the policy. Do not only score it.
A falling loss proves that the code runs. It does not prove the model learned what you intended.
Common mistakes
- Changing the model first. When a result surprises you, check the data and the evaluation before touching the architecture.
- Trusting an aggregate. One average hides which cases fail. Break results down before drawing a conclusion.
- Skipping the control. Without a matched run that differs in one way, an improvement cannot be attributed to anything.
Terms
- Baseline
- The simplest method that could plausibly work; the number to beat.
- Control
- A run identical to the experiment except for the one thing being tested.
- Shortcut
- A cue that predicts the answer in the training data without the task being solved.
- Leak
- Information from the test set reaching the model during training or tuning.
- Seed
- The starting value of the random number generator; it fixes the initial weights and data order.
- Standard error
- The standard deviation of a mean: the spread of individual runs divided by .