← All topics

Guide · Models · Topic 12 of 32

Vision-language models

How a language model is taught to read images, what the image costs, what these models get wrong, and how one becomes a robot policy.

8 minute readBuilds on: Attention in depth, Contrastive and multimodal learning, Foundation models and adaptation

What a vision-language model is

A vision-language model, or VLM, is a language model that can also read images. Give it a picture and a question and it writes an answer. It is one of the two backbones that robot foundation models are built on. The other, the diffusion model, is covered in its own note.

A VLM is valuable to robotics for what it already knows. It has read a large part of the internet and seen billions of captioned images, so it arrives knowing what a mug is, what “left of” means, and that a drawer has to be opened before something is put in it. A robot policy built on it inherits all of that and only has to learn how to move.

Three ways to join vision and language

Three designs. Two encoders, matched: an image encoder and a text encoder each produce one vector, and the vectors are compared. A reader in between: a small module reads the image encoder’s output and passes a fixed, small set of tokens to a language model. One sequence: image patches pass through a projector and join the text tokens in one sequence read by the language model.
The three families, in the order they appeared. Almost all current models, and almost all robot policies, use the third.
Two encodersA reader in betweenOne sequence
Image enters asOne vectorA small fixed set of tokensHundreds of patch tokens
Can generate textNoYesYes
New parts to trainBoth encodersThe reader, and cross-attention layersOne small projector
Cost per imageLowestLowHighest: the tokens fill the prompt
Fine spatial detailLostPartly keptBest kept
Used forRetrieval, zero-shot labelsMany images or video on a budgetGeneral assistants, robot policies

The single-sequence design won on simplicity. It adds almost nothing to a language model, it reuses the language model’s attention to relate words to patches, and it improves automatically when either half is replaced with a better one. The note on attention shows a real example of a word attending to the right patches.

A real model, by the numbers

A bar divided into three parts by parameter count. Vision encoder, SigLIP: 412 million. Projector: 2.4 million. Language model, Gemma: 2,509 million.
PaliGemma, counted from the model itself. The projector that joins the two halves is under a tenth of a percent of the total.
PartStructureParameters
Vision encoder27 transformer layers, width 1,152, 14-pixel patches, 224-pixel input412 million
ProjectorOne linear layer from width 1,152 to 2,0482.4 million
Language model18 layers, width 2,048, MLP width 16,384, vocabulary of 257,2162.51 billion

A 224-pixel image becomes (224/14)2=256(224 / 14)^2 = 256 patch tokens. The projector maps each from the encoder’s width to the language model’s, and they are placed at the start of the sequence, followed by the text. The whole design is in that one sentence: everything else is two existing models.

The training objective is the language model’s own. Given image tokens II and a text prefix, the model is trained to predict each token of the answer yy:

L=−∑tlog⁡pθ(yt∣I, prefix, y<t)\mathcal{L} = -\sum_{t} \log p_\theta\big( y_t \mid I,\ \text{prefix},\ y_{<t} \big)

The attention mask is a block mask. The image and the prefix attend to each other in both directions, and the answer attends to everything before it. The image and prefix can therefore be encoded once and cached, which robot policies exploit.

The image is the expensive part

Image tokens per image for six configurations on a log scale. 224 pixels with 14-pixel patches: 256. 448 pixels: 1,024. 896 pixels: 4,096. 448 pixels then merging two by two: 256. 896 pixels then merging four by four: 256. 384 pixels with three by three merging: 81.
Tokens per image. Resolution multiplies them; merging neighbouring patches before the language model divides them again.

A sentence is a few dozen tokens. An image is hundreds or thousands, and the language model’s cost grows faster than its token count. Every VLM design is therefore a decision about how many image tokens to spend, and the answer shapes what the model can see.

  • Low resolution. Cheap, and small objects and text vanish. A gripper’s fingertips can be a handful of pixels.
  • High resolution. Sees detail, at four times the tokens for each doubling.
  • Tiling. Cut a large image into tiles, encode each at the encoder’s native size, and add a low-resolution overview. Detail is kept and the token count multiplies.
  • Merging and pooling. Combine each small block of neighbouring patch features into one token before the language model. The encoder still sees full resolution, and the language model pays for far fewer tokens.
  • A reader. Compress to a fixed count regardless of resolution, at some loss of spatial precision.

Robot policies sit at the frugal end. They need several cameras, many times a second, so they use low resolution or heavy merging, and often a compact model. The deployment note measures what each choice costs in time.

What resolution buys

The same region of a photograph of a supermarket shelf, as the vision encoder receives it at three input sizes. At 224 pixels the region is 45 pixels across and the product labels are unreadable blocks. At 448 it is 89 pixels and the brand name can be made out. At 896 it is 179 pixels and the label text and price tag are legible.
One region of a real photograph at the three input sizes this model family is released in. This is what the vision encoder is given. No amount of language modelling can recover what the left-hand version has lost.

The token count at each size is (r/14)2(r / 14)^2: 256, 1,024 and 4,096. Reading a label or locating a small part needs the middle or right-hand version, and each step multiplies the image tokens by four and the cost of the language model by more than that. Robot policies mostly run at the left-hand size, with several cameras, many times a second. That is a deliberate trade, and it is the reason wrist cameras matter: they put the pixels where the detail is, instead of paying for resolution everywhere.

How one is trained

Four stages in a row. Separate pretraining: the vision encoder on image and caption pairs and the language model on text. Alignment: train the projector on captioned images. Instruction tuning: many tasks at once. Robot data: add an action output and train on demonstrations.
The usual sequence. Each stage is cheaper and uses less data than the one before it.

The vision encoder and the language model are trained separately first, each on its own enormous dataset. The vision encoder is usually trained contrastively against captions, as the note on contrastive learning describes, which is why its features already line up with language. Alignment then trains only the projector, on captioned images, until image tokens are usable by the language model. Instruction tuning trains on a mixture of tasks phrased as instructions, with more of the model unfrozen.

What gets frozen at each stage is a real decision. Freezing the vision encoder preserves its general features and limits how far it can adapt to, say, the view from a wrist camera. Unfreezing everything adapts fully and risks forgetting.

Beyond text answers

A language model can only emit tokens, so every other output is written as tokens. Locations are the important case for robots. Some models add special location tokens to their vocabulary, a thousand or so values per axis, and a box becomes four of them. Others write coordinates as ordinary numbers or output points.

The same robot camera frame twice. Asked to detect plate, the model draws correct boxes around both plates. Asked to detect mug, it draws one box on the robot’s gripper and none on the three mugs.
Real output from a public vision-language model (PaliGemma) on a LIBERO frame. The boxes are decoded from the location tokens it wrote. It finds both plates and, asked for a mug, boxes the gripper.

The same idea carries straight through to robot control. If a location can be a token, so can an action, and a VLM that writes action tokens is the simplest kind of robot policy.

What they get wrong

Asked about the same frame, two public models gave these answers. There are three mugs on the table, and the mug on the left is white and yellow.

QuestionModelAnswer
Describe the imagePaliGemma“a robot’s hand picks up a cup and puts it on a saucer”
Describe the imageSmolVLM2“A coffee maker is on a table with two cups and a plate.”
How many mugs are on the table?PaliGemma“couple”
How many mugs are on the table?SmolVLM2“There are two mugs on the table.”
What colour is the mug on the left?PaliGemma“red”

Each answer is fluent and each is wrong in a way that would matter to a robot. Both miscount. One takes the robot’s gripper for a coffee maker. The colour named belongs to a different mug. These are small models on a simulated image, and larger models do better, but the kinds of error are typical.

  • Counting and spatial relations. Left, right, behind and how many are consistently the weakest skills.
  • Fine detail at low resolution. What the vision encoder discards, the language model cannot recover.
  • Unfamiliar views. Wrist cameras, simulation and robot hardware are rare in web data.
  • Confident invention. The model answers in the same tone whether or not it can see the thing.
  • No sense of force or time. A single image says nothing about weight, friction or what happened a moment ago.

From a VLM to a robot policy

Turning a VLM into a policy means giving it a way to output actions, and there are three ways in use.

  • Action tokens. Write each action as tokens from the vocabulary and train on demonstrations. Nothing is added to the architecture.
  • An action head. Attach a small network that reads the VLM’s internal features and generates continuous actions, usually by flow matching. The VLM’s language abilities are left largely intact.
  • A planner above a policy. Leave the VLM as it is and have it name sub-tasks for a separate low-level policy.

A further practice is co-training: continuing to train on web image-and-text tasks alongside the robot data, so that learning to act does not erase what the model knew about the world. The note on robot policies covers all of these in detail.

Common mistakes

  • Choosing the resolution last. It decides what the model can see and how fast it runs. Decide it first.
  • Trusting a description. Check counts, colours and positions against the image before acting on them.
  • Fine-tuning on robot data alone. The model’s general knowledge erodes. Mix in the data it was built on.

Terms

Vision-language model
A language model that takes images as well as text as input.
Projector
The small layer that maps image features into the language model’s token space.
Alignment
The training stage that teaches a language model to use image tokens.
Instruction tuning
Training on many tasks phrased as instructions.
Location token
A vocabulary entry that stands for a coordinate, used to write boxes and points.
Co-training
Training on robot data and web data together.
Prefix
The part of the input given to the model, as opposed to the part it must generate.
Tiling
Splitting a large image into pieces that are each encoded at the encoder’s native size.

Further reading