← All topics

Guide · Foundations · Topic 2 of 32

Foundations that survived

The training loop, learning rates, losses, overfitting and the stabilisers that make deep networks trainable.

11 minute read

The training loop

Every model in this guide, from a small classifier to a robot foundation model, is trained the same way: four steps, repeated millions of times.

Four boxes in a row: forward pass, loss, backward pass, update, with an arrow looping back to the start
One step of training. The loop repeats with the next batch of data.

The forward pass turns a batch of inputs into predictions. The loss scores how wrong those predictions are, as a single number. The backward pass, called backpropagation, applies the chain rule layer by layer to find how much each weight contributed to the loss. That is the gradient. The update nudges every weight a small step against its gradient.

The optimizer decides the size of that step. AdamW is the default today. It keeps a running average of each weight’s gradient and of its size, which gives every weight its own step size, and it applies weight decay, a gentle pull toward zero, as a separate step.

The same loop in symbols

Write the model as a function fθf_\theta with parameters θ\theta, and the loss on a batch BB as the average loss over its examples. One step of training is:

gt=∇θ 1∣B∣∑(x,y)∈Bℓ(fθ(x), y)θt+1=θt−η gtg_t = \nabla_\theta \, \frac{1}{|B|}\sum_{(x, y) \in B} \ell\big(f_\theta(x),\, y\big) \qquad\qquad \theta_{t+1} = \theta_t - \eta\, g_t

The gradient gtg_t is computed by backpropagation, which is the chain rule applied from the loss backwards. If layer ll computes hl=ϕ(Wlhl−1)h_l = \phi(W_l h_{l-1}), then the gradient with respect to its input follows from the gradient with respect to its output:

∂ℓ∂hl−1=Wl⊤(ϕ′(Wlhl−1)⊙∂ℓ∂hl)∂ℓ∂Wl=(ϕ′(Wlhl−1)⊙∂ℓ∂hl) hl−1⊤\frac{\partial \ell}{\partial h_{l-1}} = W_l^{\top}\Big(\phi'(W_l h_{l-1}) \odot \frac{\partial \ell}{\partial h_l}\Big) \qquad\qquad \frac{\partial \ell}{\partial W_l} = \Big(\phi'(W_l h_{l-1}) \odot \frac{\partial \ell}{\partial h_l}\Big)\, h_{l-1}^{\top}

Two consequences are worth carrying around. The backward pass costs about the same as the forward pass, so a training step is roughly three times the cost of inference. And the gradient at an early layer is a product of one term per later layer, which is why depth is fragile: if those terms are mostly smaller than one the product vanishes, and if larger it explodes. A later section measures this.

The batch gradient is a noisy estimate of the gradient over the whole dataset, with noise that shrinks as 1/∣B∣1/\sqrt{|B|}. That noise is not purely a nuisance. It is part of why stochastic gradient descent finds solutions that generalise.

Optimisers

Plain gradient descent uses the same step size for every parameter and forgets the past. The optimisers in use add memory and per-parameter scaling.

Momentum keeps a running average of past gradients and steps along that, which damps oscillation and speeds progress along consistent directions:

vt=μ vt−1+gtθt+1=θt−η vtv_t = \mu\, v_{t-1} + g_t \qquad\qquad \theta_{t+1} = \theta_t - \eta\, v_t

Adam keeps a running average of the gradient and of its square, and divides one by the root of the other. Every parameter gets its own effective step size, large where gradients have been small and small where they have been large:

mt=β1mt−1+(1−β1) gtvt=β2vt−1+(1−β2) gt2θt+1=θt−η m^tv^t+ϵm_t = \beta_1 m_{t-1} + (1 - \beta_1)\, g_t \qquad v_t = \beta_2 v_{t-1} + (1 - \beta_2)\, g_t^2 \qquad \theta_{t+1} = \theta_t - \eta\, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}

The hats denote a correction for the averages starting at zero: m^t=mt/(1−β1t)\hat m_t = m_t / (1 - \beta_1^t) and likewise for vv. Typical values are β1=0.9\beta_1 = 0.9 and β2=0.999\beta_2 = 0.999. AdamW adds weight decay as a separate step, θ←θ−ηλθ\theta \leftarrow \theta - \eta \lambda \theta, instead of folding it into the gradient where Adam’s scaling would distort it. It is the default for transformers.

Validation loss over 30 passes for three optimisers on the same network. Plain SGD falls slowly to 0.19. SGD with momentum reaches 0.049 and AdamW 0.064.
The same network and data with three optimisers, each at a reasonable learning rate. Plain gradient descent is still descending when the other two have finished.

The figure is a reminder that “which optimiser is best” has no general answer. Momentum edges out AdamW on this small problem. On large transformers AdamW is far more forgiving of the learning rate, and that robustness is why it is the default.

The learning rate

The learning rate η\eta is the size of each update, and it is the most important number to tune.

Validation loss after 12 passes against learning rate on a log axis, for SGD and Adam. Each has a basin where it works. Adam works from about 0.0003 to 0.03 and diverges above 0.3. SGD needs a rate about a hundred times larger, working near 0.1 to 0.3.
Measured for this note on a small classifier, mean of three runs per point. Each optimiser has a basin: too low and the loss never leaves the level of guessing, too high and training diverges.

Two things in this sweep generalise. The usable range is roughly one to two orders of magnitude wide, so a sweep on a log scale finds it in a handful of runs. And the range depends on the optimiser: Adam’s basin here sits about a hundred times lower than plain gradient descent’s, which is why a rate copied from one setup rarely transfers to another.

In practice the rate is not constant. It ramps up linearly from zero over the first steps, because early gradients are large and noisy, and then decays, usually along a cosine:

ηt={ηmax⁡ t/Twt<Tw12 ηmax⁡(1+cos⁡π(t−Tw)T−Tw)t≥Tw\eta_t = \begin{cases} \eta_{\max}\, t / T_w & t < T_w \\ \tfrac{1}{2}\,\eta_{\max}\left(1 + \cos\frac{\pi (t - T_w)}{T - T_w}\right) & t \ge T_w \end{cases}

One trap is worth knowing early. A metric that stops improving only because the schedule has reached zero has not converged. Check that the model levelled off while the learning rate was still meaningfully above zero.

Loss functions: telling the model what “wrong” means

The loss is the only signal the model gets, so choosing it is choosing what the model cares about. Pick the one that matches the kind of output.

LossUse it when the output isIntuitionWhere you will see it
Cross entropyOne of several classes, or the next tokenPenalises low probability on the correct answerClassifiers, language models, action tokens
MSEA continuous numberPenalises large errors heavily; predicts the averageRegression, diffusion and flow-matching targets
L1 or HuberA continuous number with outliersPenalises errors evenlyAction regression, depth, boxes
KL divergenceA probability distributionHow different two distributions areVAEs, distillation from a teacher
ContrastiveAn embeddingPull matching pairs together, push the rest apartCLIP-style models, retrieval

The second row matters for robots. MSE predicts the mean, so a policy trained with it averages two valid ways of doing a task. The note on robot policies shows why that fails.

Why these losses: maximum likelihood

The losses in the table above are not arbitrary. Each one says: make the training data as probable as possible under the model. Cross entropy is exactly that for a choice among classes. Mean squared error is the same idea under the assumption that the right answer is one value blurred by bell-shaped noise.

That assumption is why squared error predicts the average. If the data contains two different right answers, a single bell curve fitted to both sits in the middle. Generative models exist to replace that assumption with something that can hold several answers at once.

Softmax and cross entropy, written out

A classifier outputs one score per class, called logits zz. The softmax turns them into probabilities, and cross entropy is the negative log of the probability given to the correct class yy:

pi=ezi∑jezjℓ=−log⁡py=−zy+log⁡∑jezjp_i = \frac{e^{z_i}}{\sum_j e^{z_j}} \qquad\qquad \ell = -\log p_y = -z_y + \log \sum_j e^{z_j}

Its gradient with respect to the logits is remarkably simple: ∂ℓ/∂zi=pi−1[i=y]\partial \ell / \partial z_i = p_i - \mathbb{1}[i = y], the predicted probabilities minus the truth. A language model applies exactly this at every position, with the vocabulary as the classes. Robot policies that write actions as tokens do the same.

Two refinements are common. Label smoothing trains toward a target of 1−ϵ1 - \epsilon on the correct class instead of 1, which stops the logits growing without bound and improves calibration. For regression with outliers, the Huber loss is quadratic for small errors and linear for large ones, so a few bad labels do not dominate the gradient.

Overfitting and capacity

Training and validation loss over 150 passes on 100 training images. Without regularisation, training loss reaches zero and validation loss is lowest at pass 27 and then creeps upward. With dropout and weight decay, the best validation loss is lower and comes later.
Measured for this note: a network with far more parameters than it has training images. Training loss goes to zero in both runs. Validation loss is the one to read.

The goal is low error on data the model has never seen. Capacity is how complex a function the model can represent. A model with too little, or a broken training run, gives high loss on both training and validation data, which is underfitting. A model that memorises shows training loss still falling while validation loss rises, which is overfitting. In the left panel the best validation loss comes at pass 27; every pass after that makes the model worse while the training loss says it is getting better.

The classical account splits the expected error of a model into three parts:

E[(y−f^(x))2]=(E[f^(x)]−f(x))2⏟bias2+Var[f^(x)]⏟variance+σ2⏟noise\mathbb{E}\big[(y - \hat f(x))^2\big] = \underbrace{\big(\mathbb{E}[\hat f(x)] - f(x)\big)^2}_{\text{bias}^2} + \underbrace{\mathrm{Var}\big[\hat f(x)\big]}_{\text{variance}} + \underbrace{\sigma^2}_{\text{noise}}

Bias is error from a model too simple to fit the truth. Variance is how much the fitted model changes with the particular training sample. Small models are high in bias and large ones in variance, which predicts a sweet spot in between. The modern twist is that very large models trained on very large datasets often keep improving past the point where this picture says they should fail, an effect called double descent. The practical rule has not changed: hold out data, plot both curves, keep the checkpoint with the best validation score, and reach for more data before anything else.

Convolutions: the older building block that stayed

Before transformers, vision models were built from convolutions. A convolution slides a small grid of weights over the image and, at each position, outputs one number saying how well the patch underneath matches the grid.

A robot camera frame in grey, a three by three grid of weights with negative values on the left and positive on the right, and the result, which is bright along vertical edges of the objects.
A real camera frame passed through one hand-written 3 × 3 filter. The output is bright wherever brightness changes from left to right. A trained network learns thousands of such filters and stacks them.

Two properties made this work. The same weights are used at every position, so a feature learned in one corner is recognised in another. And each output looks only at a small neighbourhood, so early layers find edges and later layers combine them into parts and objects.

Transformers have replaced convolutions as the main architecture, and convolutions are still in most systems. The step that cuts an image into patch tokens is a convolution. The autoencoders that compress images and video are convolutional. Several robot policies still use a convolutional network for vision.

Why deep networks train at all

Left: validation accuracy against the number of layers. Plain layers fall from 95 percent at 2 layers to 38 percent at 16 and to chance at 32. Layers with residual connections and normalisation stay at 97 to 99 percent up to 64 layers. Right: the size of the gradient reaching each layer of a 32-layer network at initialisation. In the plain network it shrinks by eleven orders of magnitude from the output side to the input side. In the residual network it stays near one.
Measured for this note. Left: the same network with more layers. Right: the gradient reaching each layer of the 32-layer versions before any training.

Stacking plain layers stops working quickly. At 32 layers the plain network does no better than guessing, and the right-hand panel shows why: the gradient that reaches the first layer is eleven orders of magnitude smaller than the gradient at the last, so the early layers never move. This is the product of per-layer terms from the backpropagation equation, measured.

Three ideas fix it, and every modern network uses all three.

  • Initialisation. Draw the initial weights with a variance chosen so that signals keep roughly the same size from layer to layer. For a layer with ninn_{\text{in}} inputs and a ReLU, that is Var(W)=2/nin\mathrm{Var}(W) = 2 / n_{\text{in}}.
  • Normalisation. Rescale activations inside the network. Layer normalisation computes, for each token, LN(x)=γ⊙(x−μ)/σ+β\mathrm{LN}(x) = \gamma \odot (x - \mu) / \sigma + \beta, with μ\mu and σ\sigma taken over that token’s features. RMSNorm drops the mean and divides by the root mean square alone, which is cheaper and works as well in transformers.
  • Residual connections. Compute hl+1=hl+F(hl)h_{l+1} = h_l + F(h_l) instead of hl+1=F(hl)h_{l+1} = F(h_l). The derivative is then I+∂F/∂hlI + \partial F / \partial h_l. The identity term gives the gradient a path that is not multiplied down at every layer, which is the flat line in the figure.

With these, the 64-layer network trains as well as the 2-layer one.

The stabilisers

Three small ideas make deep networks trainable and keep them from memorising.

  • Regularisation. Weight decay pulls weights toward zero. Dropout randomly silences units during training. Augmentation, such as random crops and colour shifts, acts as free extra data and matters a great deal when robot data is scarce. Early stopping keeps the checkpoint with the best validation score.
  • Normalisation. It keeps the numbers flowing through the network at a steady scale. Transformers use LayerNorm or RMSNorm; older convolutional networks use BatchNorm.
  • Residual connections. Each layer adds a correction to its input, so the output is x + f(x). Gradients get a direct path back to the early layers, which is what allows networks hundreds of layers deep.

Common mistakes

  • Tuning everything except the learning rate. If training diverges or stalls, look there first.
  • Stopping at the step budget. A run is finished when the validation curve has flattened, not when the counter runs out.
  • Reading the training loss alone. It always improves. Only the validation curve says whether the model is learning or memorising.

Terms

Gradient
How the loss changes as each weight changes.
Batch
The handful of samples used for one update.
Epoch
One full pass over the training data.
Weight decay
A small pull of every weight toward zero at each step.
Convolution
A small grid of weights applied at every position of an image.
Logit
A raw score output by a network before it is turned into a probability.
Momentum
A running average of past gradients used as the update direction.
Vanishing gradient
A gradient that shrinks toward zero as it is propagated back through many layers.

Further reading