← All topics

Guide · Models · Topic 11 of 32

Foundation models and adaptation

Pretrain once, adapt many times: the ladder from prompting to fine-tuning.

7 minute readBuilds on: The transformer, Representation learning

Pretrain once, adapt many times

Broad data feeds expensive pretraining, which produces a foundation model that is cheaply adapted to a specific task
The economics of the field in one line.

A foundation model is trained on broad data at great cost and then reused for many tasks at low cost. Pretraining takes enormous data and compute and is done by a few groups. Adaptation takes a small dataset and a modest budget and is done by everyone. Almost no one trains from scratch any more. The skill is choosing a starting model and adapting it well.

Two kinds matter most here. Language models are decoder transformers trained to predict the next token; they carry broad knowledge and follow instructions. Vision-language models add a vision encoder in front, so they can answer questions about images and video. A vision-language model is the usual starting point for a robot policy.

Transfer works because general features carry over and a little task data goes a long way. The risk travels with it: the model’s gaps and biases transfer too.

Language models that see

A vision-language model is a language model with a vision encoder in front. Image patches are encoded, projected into the language model’s token space, and placed in the prompt beside the text. It is the usual starting point for a robot policy, and it has a note of its own.

Why scale works

The case for foundation models rests on an empirical regularity. Across model families, the loss falls smoothly as a power law in each of three quantities: parameters, data and compute.

L(N)≈(NcN)αN+L∞L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N} + L_\infty

with similar forms for data and compute, and small exponents. A small exponent means that each constant improvement needs a multiplied budget, which is why progress has come from orders of magnitude and not from percentages. It also means the returns are predictable, which is what justified building very large models in the first place.

Three practical consequences follow. Parameters and data have to grow together, because a large model on little data overfits and a small model on much data saturates. A model of a given size is best trained on an amount of data in proportion to it. And the same curves appear to hold in robotics for data variety, as the note on data efficiency describes.

The adaptation ladder

There are several ways to get a pretrained model to do your task. Try the cheap rungs first, and climb only as far as the task needs. Each rung costs more data, compute and upkeep.

MethodWhat changesData neededReach for it when
PromptingOnly the input textNoneThe model already knows how and needs instructions
In-context examplesThe input, with a few worked examplesA handfulThe output format or style is specific
Retrieval augmentationThe input, with documents fetched at run timeA document storeAnswers depend on facts the model lacks
LoRA and adaptersSmall added weight matrices; the base is frozenHundreds to thousandsA new skill or domain on a limited budget
Full fine-tuningAll the weightsThousands and upA large shift, such as a new robot or new kind of output
DistillationA smaller student copies the adapted modelThe teacher’s outputsThe result must run on a small device

LoRA stands for low-rank adaptation. Instead of updating a large weight matrix, it learns two small matrices whose product is added to it, training a fraction of a percent of the parameters. For robot policies, full fine-tuning is still common because actions are a new kind of output, but a frequent compromise is LoRA on the language backbone plus a newly trained action head. The data counts above are rough orders of magnitude.

Low-rank adaptation, measured

Full fine-tuning updates every weight. Low-rank adaptation freezes the pretrained matrix W0W_0 and learns a correction that is the product of two thin matrices:

W=W0+αr BAB∈Rdout×r,A∈Rr×din,r≪min⁡(din,dout)W = W_0 + \frac{\alpha}{r}\, B A \qquad B \in \mathbb{R}^{d_{\text{out}} \times r},\quad A \in \mathbb{R}^{r \times d_{\text{in}}},\quad r \ll \min(d_{\text{in}}, d_{\text{out}})

BB starts at zero, so training begins from exactly the pretrained model. The number of trained parameters per matrix falls from dindoutd_{\text{in}} d_{\text{out}} to r(din+dout)r (d_{\text{in}} + d_{\text{out}}). For a 2,048 × 2,048 matrix at rank 8 that is 32,768 in place of 4.2 million, under 1%. After training, the product can be added into W0W_0, so inference costs nothing extra.

Left: the cumulative share of the weight change from full fine-tuning captured by its largest directions. Twelve of 256 directions hold 90 percent. Right: accuracy on a new task against the number of parameters trained, for adapters of rank 1 to 64. Rank 4, with 4,392 parameters, reaches 96.6 percent. Rank 16 reaches 98.1 percent. Full fine-tuning of 85,002 parameters reaches 99.0 percent.
Measured for this note. A small network trained on one task is adapted to a second. Left: the change that full fine-tuning made to one weight matrix, decomposed. Right: adapters of increasing rank.

The left panel is the reason it works. When this network was fully fine-tuned, 90% of the change to its middle weight matrix lay in 12 of 256 possible directions. The update was already nearly low-rank, so restricting it to be low-rank loses little. On the right, rank 4 reaches 96.6% with 5% of the parameters, against 99.0% for full fine-tuning.

The same experiment shows what adaptation costs. Before adapting, the network scored 98.4% on its original task. After full fine-tuning on the new one it scored 38.4%. An adapter avoids this for free: remove it and the original model is back, and several adapters can be kept for one base model and swapped per task.

Sampling: temperature

A language model outputs a probability for every possible next token. How one is picked from that list is a setting, not part of the model.

The same six candidate words at three temperatures. At 0.3 the top word has 87 percent. At 1.0 it has 49 percent. At 2.0 it has 33 percent and the others are close behind.
One set of scores, three temperatures. Low temperature concentrates probability on the top choice. High temperature spreads it out.

Temperature rescales the scores before they become probabilities. Near zero the model almost always takes its top choice, which is repeatable and can be dull. Higher values give variety and more mistakes. A common companion, top-p sampling, discards the unlikely tail before picking. For structured output and for robot actions written as tokens, use a low temperature.

From answering to acting

  • Reasoning. Have the model write intermediate steps before its answer, often called chain of thought. This trades extra compute at run time for accuracy.
  • Tool use. The model emits a structured call, such as a search or a piece of code to run, and the result is fed back as new input. It can then act on facts it was never trained on.
  • Agents. A loop of observe, decide, act and observe again, useful for multi-step work.

These are the bridge from chat models to systems that do work. A robot is an agent whose tools are motors, and the reasoning idea returns in robot policies as stating the plan before acting.

Context length and made-up answers

The context window is how many tokens the model can read at once: the instructions, the documents, the images and its own output so far. Anything outside it does not exist for the model. Long contexts cost more, respond more slowly, and details in the middle are recalled less reliably than those at the start and end.

A language model always produces fluent text, including when it has nothing to base it on. A confident wrong answer of this kind is called a hallucination. The defences are to give the model the facts through retrieval, to ask for output in a form that can be checked, and to verify anything that matters. For a robot the check is physical: confirm that the object the model named is actually in view before acting on it.

Common mistakes

  • Fine-tuning first. Many tasks are solved by a better prompt or by retrieval, at a fraction of the cost.
  • Forgetting what was lost. Fine-tuning on a narrow dataset can erase general abilities the base model had.
  • Skipping the baseline. Measure the unadapted model first, or you cannot say what adaptation bought.

Terms

Foundation model
A large model pretrained on broad data and reused for many tasks.
Fine-tuning
Continuing training on task data.
LoRA
Adapting a model by training small added matrices.
Context window
The amount of input the model can read at once.
Projector
The small layer that maps image features into a language model’s token space.
Temperature
A setting that sharpens or flattens the probabilities a model samples from.
Context window
The number of tokens a model can read at once.
Hallucination
Fluent output that is not supported by the input or by fact.
Rank
The number of independent directions in a matrix; a low-rank matrix can be written as a product of two thin ones.
Adapter
A small trained module added to a frozen pretrained model.
Scaling law
An empirical power-law relation between model performance and parameters, data or compute.

Further reading