← All topics

Guide · Practice · Topic 28 of 32

Training at scale

A recipe for starting any training project, and the engineering around a large run.

7 minute readBuilds on: Foundations that survived

A recipe for training neural networks

Andrej Karpathy’s 2019 recipe is still the best checklist for starting a training project. Its lesson is the ordering: build up from something simple that works, and never start with the full system.

Five boxes in a row: know the data, skeleton and baseline, overfit, regularise, tune
Each stage gives you something known to work before you add complexity.
  • Know the data. Look at hundreds of samples. Find the duplicates, the bad labels and the imbalance before the model does.
  • Build the skeleton and a baseline. Set up the full training and evaluation loop with a tiny model. Fix the random seed, and check that the loss at the very start is what you would expect from chance.
  • Overfit. Make the model big enough to memorise the training set. If it cannot, there is a bug.
  • Regularise. Add data first. Then add augmentation, dropout and weight decay, trading some training loss for validation loss.
  • Tune. Search the learning rate and a few key settings, one at a time, and train to convergence.

The engineering around a run

Large training runs are software projects. Six practices keep them fast, comparable and recoverable.

PracticeWhat it isWatch for
Mixed precisionCompute in 16-bit numbers and keep the sensitive parts in 32-bit. Roughly halves memory.Overflow and NaNs
Distributed trainingSplit each batch across several GPUs for a shorter wall-clock time.Data loading becomes the bottleneck
Experiment trackingLog every run: configuration, code version, metrics and sample outputs.Run names you cannot decode in a year
Pinned configurationsSet every hyperparameter explicitly and compare the full configuration with the reference run before launch.Defaults that changed between code versions
CheckpointingSave the model, optimizer and step regularly, and keep the best checkpoint by validation score.A resumed run that differs from an uninterrupted one
ConvergenceA run is done when the validation curve has flattened.A curve that flattened only because the learning rate reached zero

The configuration row deserves emphasis. The most common way an experiment silently changes two things at once is a default that moved. A result you cannot reproduce or compare is not a result.

Two small additions that prevent large failures

Left: held-out error for the weights being trained, which is noisy, and for a running average of those weights, which is smoother and about three times lower. Right: with a few bad batches, error without clipping spikes by a factor of several hundred each time, while the clipped run stays level.
A small regression model trained with noisy gradients, run for this note. Left: averaging the weights. Right: the same model with 1% of batches corrupted, with and without clipping.
  • Averaged weights (EMA). Keep a second copy of the model that is a slow running average of the trained weights, and evaluate and deploy that copy. It filters out step-to-step noise. In the run above its error is about a third of the raw weights’. It is standard for diffusion and flow-matching policies, where it makes a visible difference to the motion.
  • Gradient clipping. If a batch produces an unusually large gradient, scale it down to a fixed size before the update. One bad batch then costs one ordinary step. In the run above, an unclipped bad batch raised the error several hundredfold and took a hundred steps to recover from.

Batch size, learning rate and the data loader

A larger batch gives a less noisy gradient, which permits a larger learning rate. A common starting rule is to scale the two together. When the batch you want does not fit in memory, gradient accumulation adds up the gradients of several small batches before each update. It gives the same result and takes longer.

In robot learning the slow part is often not the GPU. Decoding video and resizing images on the CPU can leave the GPU idle most of the time. Check GPU utilisation first. If it is low, add loader workers, decode at a lower resolution, or pre-compute and cache what does not change.

Where the time and memory go

Left: training speed in samples per second against batch size for 32-bit and mixed-precision training of the same transformer. At 32 bits it levels off near 420 from a batch of 16. With mixed precision it reaches about 1,080 at a batch of 32. Right: peak memory against batch size, rising in proportion to it; at a batch of 128, 9.9 gigabytes at 32 bits and 5.6 with mixed precision.
Measured for this note on a desktop GPU: one training step of an eight-layer transformer on sequences of 256 tokens.

Three facts about a training run can be read from this measurement.

  • Throughput saturates. Larger batches are faster per sample only until the GPU is fully used, here at a batch of 16 to 32. Past that point a larger batch buys nothing in speed.
  • Mixed precision is the cheapest speed-up there is. Computing in 16-bit numbers made this model 2.6 times faster and nearly halved its memory. It is one line of code, and the price is the occasional overflow.
  • Memory grows with the batch. Training memory is roughly weights + gradients + optimiser state + activations. The first three are fixed: with AdamW, about four numbers per parameter. Activations are kept for the backward pass and scale with batch size × sequence length × layers × width, which is why the right-hand panel is a straight line.

When the batch you want does not fit, there are three levers before a bigger GPU. Gradient accumulation sums the gradients of kk small batches before each update, which gives the same result as one large batch and takes kk times as long. Gradient checkpointing discards activations and recomputes them during the backward pass, trading about a third more compute for much less memory. And splitting the batch across several GPUs, each computing the gradient of its share, scales nearly linearly until the data loader or the network between machines becomes the limit.

On the data side, one processor core decoded the video in the LIBERO dataset at about 9,500 frames a second. At that resolution, 256 pixels, decoding is not a bottleneck for a model that consumes a thousand samples a second. At the resolution of a real camera it is twenty to forty times slower per frame, and it becomes the bottleneck unless frames are decoded in parallel or stored pre-resized.

The numbers that go with batch size

Three hyperparameters are tied to the batch size, and changing one without the others is a common way to break a working configuration.

Learning rate. A batch kk times larger gives a gradient with k\sqrt{k} less noise, and the usual rule of thumb is to scale the learning rate with the batch, η∝∣B∣\eta \propto |B|, up to a limit, with a longer warmup for the larger rate. Adaptive optimisers are less sensitive than plain gradient descent, and a square-root scaling is often safer for them.

Averaged weights. The averaged copy of the model is updated after every step as

θˉt=β θˉt−1+(1−β) θt\bar\theta_t = \beta\, \bar\theta_{t-1} + (1 - \beta)\, \theta_t

which averages over roughly the last 1/(1−β)1 / (1 - \beta) steps: 1,000 steps at β=0.999\beta = 0.999. If the batch is doubled and the number of steps halved, the same β\beta now averages over twice as much data, so β\beta should be adjusted to keep the window the same in samples.

Clipping threshold. Gradient clipping rescales the gradient when its norm exceeds a threshold cc:

g←g⋅min⁡ ⁣(1, c∥g∥)g \leftarrow g \cdot \min\!\Big(1,\ \frac{c}{\| g \|}\Big)

The typical gradient norm changes with batch size and with the loss, so the threshold should be set from a logged distribution of norms, a little above the usual value, and not copied from another project.

When the loss becomes NaN

NaN means “not a number”: some value overflowed or an impossible operation was attempted, and it has spread to everything. Work through the causes in order of likelihood.

CheckWhat to look for
The learning rateHalve it. If the problem goes away, it was too high, or the warmup was too short.
The dataNaN or infinite values in the inputs; actions that were never normalised; a division by a zero spread in the normalisation.
Mixed precisionSwitch it off. If the run survives, a value overflowed in 16 bits.
The first bad stepLog the gradient size every step and find where it jumps. Inspect that batch.
The loss itselfA logarithm of zero or a division by a very small number.

Common mistakes

  • Starting with the full system. When it fails, there are too many places to look.
  • Changing several things at once. The result cannot then be attributed to any of them.
  • Declaring victory at the step limit. Check the curve, not the counter.
  • Evaluating the raw weights of a diffusion policy. Use the averaged copy.

Terms

Seed
The number that fixes the random choices in a run, so it can be repeated.
Checkpoint
A saved copy of the model partway through training.
Hyperparameter
A setting chosen by you, not learned by the model.
Convergence
The point where more training no longer improves the validation metric.
EMA weights
A slow running average of the model’s weights, used for evaluation and deployment.
Gradient clipping
Scaling down a gradient that exceeds a set size.
Gradient accumulation
Summing gradients over several small batches before one update.
Mixed precision
Computing mostly in 16-bit numbers while keeping sensitive values in 32 bits.
Gradient checkpointing
Recomputing activations during the backward pass instead of storing them, to save memory.

Further reading