Pretrain once, adapt many times

A foundation model is trained on broad data at great cost and then reused for many tasks at low cost. Pretraining takes enormous data and compute and is done by a few groups. Adaptation takes a small dataset and a modest budget and is done by everyone. Almost no one trains from scratch any more. The skill is choosing a starting model and adapting it well.
Two kinds matter most here. Language models are decoder transformers trained to predict the next token; they carry broad knowledge and follow instructions. Vision-language models add a vision encoder in front, so they can answer questions about images and video. A vision-language model is the usual starting point for a robot policy.
Transfer works because general features carry over and a little task data goes a long way. The risk travels with it: the model’s gaps and biases transfer too.
Language models that see
A vision-language model is a language model with a vision encoder in front. Image patches are encoded, projected into the language model’s token space, and placed in the prompt beside the text. It is the usual starting point for a robot policy, and it has a note of its own.
Why scale works
The case for foundation models rests on an empirical regularity. Across model families, the loss falls smoothly as a power law in each of three quantities: parameters, data and compute.
with similar forms for data and compute, and small exponents. A small exponent means that each constant improvement needs a multiplied budget, which is why progress has come from orders of magnitude and not from percentages. It also means the returns are predictable, which is what justified building very large models in the first place.
Three practical consequences follow. Parameters and data have to grow together, because a large model on little data overfits and a small model on much data saturates. A model of a given size is best trained on an amount of data in proportion to it. And the same curves appear to hold in robotics for data variety, as the note on data efficiency describes.
The adaptation ladder
There are several ways to get a pretrained model to do your task. Try the cheap rungs first, and climb only as far as the task needs. Each rung costs more data, compute and upkeep.
| Method | What changes | Data needed | Reach for it when |
|---|---|---|---|
| Prompting | Only the input text | None | The model already knows how and needs instructions |
| In-context examples | The input, with a few worked examples | A handful | The output format or style is specific |
| Retrieval augmentation | The input, with documents fetched at run time | A document store | Answers depend on facts the model lacks |
| LoRA and adapters | Small added weight matrices; the base is frozen | Hundreds to thousands | A new skill or domain on a limited budget |
| Full fine-tuning | All the weights | Thousands and up | A large shift, such as a new robot or new kind of output |
| Distillation | A smaller student copies the adapted model | The teacher’s outputs | The result must run on a small device |
LoRA stands for low-rank adaptation. Instead of updating a large weight matrix, it learns two small matrices whose product is added to it, training a fraction of a percent of the parameters. For robot policies, full fine-tuning is still common because actions are a new kind of output, but a frequent compromise is LoRA on the language backbone plus a newly trained action head. The data counts above are rough orders of magnitude.
Low-rank adaptation, measured
Full fine-tuning updates every weight. Low-rank adaptation freezes the pretrained matrix and learns a correction that is the product of two thin matrices:
starts at zero, so training begins from exactly the pretrained model. The number of trained parameters per matrix falls from to . For a 2,048 × 2,048 matrix at rank 8 that is 32,768 in place of 4.2 million, under 1%. After training, the product can be added into , so inference costs nothing extra.

The left panel is the reason it works. When this network was fully fine-tuned, 90% of the change to its middle weight matrix lay in 12 of 256 possible directions. The update was already nearly low-rank, so restricting it to be low-rank loses little. On the right, rank 4 reaches 96.6% with 5% of the parameters, against 99.0% for full fine-tuning.
The same experiment shows what adaptation costs. Before adapting, the network scored 98.4% on its original task. After full fine-tuning on the new one it scored 38.4%. An adapter avoids this for free: remove it and the original model is back, and several adapters can be kept for one base model and swapped per task.
Sampling: temperature
A language model outputs a probability for every possible next token. How one is picked from that list is a setting, not part of the model.

Temperature rescales the scores before they become probabilities. Near zero the model almost always takes its top choice, which is repeatable and can be dull. Higher values give variety and more mistakes. A common companion, top-p sampling, discards the unlikely tail before picking. For structured output and for robot actions written as tokens, use a low temperature.
From answering to acting
- Reasoning. Have the model write intermediate steps before its answer, often called chain of thought. This trades extra compute at run time for accuracy.
- Tool use. The model emits a structured call, such as a search or a piece of code to run, and the result is fed back as new input. It can then act on facts it was never trained on.
- Agents. A loop of observe, decide, act and observe again, useful for multi-step work.
These are the bridge from chat models to systems that do work. A robot is an agent whose tools are motors, and the reasoning idea returns in robot policies as stating the plan before acting.
Context length and made-up answers
The context window is how many tokens the model can read at once: the instructions, the documents, the images and its own output so far. Anything outside it does not exist for the model. Long contexts cost more, respond more slowly, and details in the middle are recalled less reliably than those at the start and end.
A language model always produces fluent text, including when it has nothing to base it on. A confident wrong answer of this kind is called a hallucination. The defences are to give the model the facts through retrieval, to ask for output in a form that can be checked, and to verify anything that matters. For a robot the check is physical: confirm that the object the model named is actually in view before acting on it.
Common mistakes
- Fine-tuning first. Many tasks are solved by a better prompt or by retrieval, at a fraction of the cost.
- Forgetting what was lost. Fine-tuning on a narrow dataset can erase general abilities the base model had.
- Skipping the baseline. Measure the unadapted model first, or you cannot say what adaptation bought.
Terms
- Foundation model
- A large model pretrained on broad data and reused for many tasks.
- Fine-tuning
- Continuing training on task data.
- LoRA
- Adapting a model by training small added matrices.
- Context window
- The amount of input the model can read at once.
- Projector
- The small layer that maps image features into a language model’s token space.
- Temperature
- A setting that sharpens or flattens the probabilities a model samples from.
- Context window
- The number of tokens a model can read at once.
- Hallucination
- Fluent output that is not supported by the input or by fact.
- Rank
- The number of independent directions in a matrix; a low-rank matrix can be written as a product of two thin ones.
- Adapter
- A small trained module added to a frozen pretrained model.
- Scaling law
- An empirical power-law relation between model performance and parameters, data or compute.
Further reading
- Hu et al., 2021. LoRA: Low-Rank Adaptation of Large Language Models.
- Lewis et al., 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
- Wei et al., 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.
- Liu et al., 2023. Visual Instruction Tuning (LLaVA).
- Holtzman et al., 2019. The Curious Case of Neural Text Degeneration (top-p sampling).
- Kaplan et al., 2020. Scaling Laws for Neural Language Models.
- Hoffmann et al., 2022. Training Compute-Optimal Large Language Models.