← All topics

Guide · Models · Topic 6 of 32

Representation learning

How models learn useful features from data with no labels.

6 minute readBuilds on: The transformer

Labels made from the data itself

Labelled data is expensive. Self-supervised learning avoids labels by inventing a prediction problem whose answer is already in the data. This is why models can train on billions of images and hours of video. Three recipes dominate.

Masked modeling

Hide part of the input and predict the missing part. For text, that means predicting hidden or next words. For images and video, it means predicting hidden patches. The model has to learn structure and context to fill the gaps.

A night street photo with one large visible block outlined in blue and four smaller hidden blocks outlined in orange
One image, a visible context block and several hidden target blocks to be predicted.

Contrastive learning

Take two things known to belong together and make their embeddings match, while pushing apart the embeddings of things that do not. With images and their captions as the pairs, this gives CLIP-style models.

An eight by eight grid of image and caption pairs with the diagonal cells highlighted
A batch of eight images and eight captions. Each image must pick its own caption: the diagonal is the truth and every other cell is a wrong pairing to reject.

Self-distillation

A student network predicts the output of a teacher that is a slowly updated copy of itself, seeing a different view of the same image. It sounds circular, but it works, and it gives the strong visual features of the DINO family. The slow teacher is essential.

Two curves of representation variance during training: one stays high, the other decays toward zero
A toy experiment. With a slowly updated teacher the features stay varied; with gradients through both branches they collapse to a constant.

The objectives, written out

A robot camera frame shown three ways. The full image. The same image with three quarters of its patches hidden, for masked autoencoding. The same image with one block marked as the target and a ring around it hidden, for joint-embedding prediction.
Two pretext tasks on one real frame. Masked autoencoding hides most patches and rebuilds their pixels. Joint-embedding prediction hides a block and predicts its features.

Each family of self-supervised learning is a different loss on the same kind of data. Let xx be an image, mm a set of hidden patches and fθf_\theta the encoder.

Masked autoencoding encodes only the visible patches, and a small decoder gg rebuilds the hidden ones. The loss is the squared error on the hidden pixels:

LMAE=1∣m∣∑i∈m∥g(fθ(x∖m))i−xi∥2\mathcal{L}_{\text{MAE}} = \frac{1}{|m|} \sum_{i \in m} \big\| g\big(f_\theta(x_{\setminus m})\big)_i - x_i \big\|^2

Hiding as much as 75% makes the task hard enough to need real understanding and makes training cheap, because the encoder only processes the quarter that is visible.

Self-distillation, the DINO family, shows two different crops of an image to a student and a teacher. The student is trained to match the teacher’s output distribution, and the teacher is an exponential moving average of the student:

LDINO=−∑kpteacher(k)(x1) log⁡pstudent(k)(x2)θteacher←τ θteacher+(1−τ) θstudent\mathcal{L}_{\text{DINO}} = -\sum_{k} p_{\text{teacher}}^{(k)}(x_1)\, \log p_{\text{student}}^{(k)}(x_2) \qquad\qquad \theta_{\text{teacher}} \leftarrow \tau\, \theta_{\text{teacher}} + (1 - \tau)\, \theta_{\text{student}}

Nothing here stops both networks outputting the same thing for every image. Two devices do: the teacher’s outputs are centred, by subtracting a running mean, and sharpened with a low temperature. Together with the slow-moving teacher they keep the representation from collapsing.

Joint-embedding prediction replaces pixels with features as the target. A context encoder sees part of the image, and a predictor hh must output the features that a slow-moving target encoder fˉ\bar f produces for the hidden block:

LJEPA=1∣m∣∑i∈m∥h(fθ(x∖m))i−fˉ(x)i∥2\mathcal{L}_{\text{JEPA}} = \frac{1}{|m|} \sum_{i \in m} \big\| h\big(f_\theta(x_{\setminus m})\big)_i - \bar f(x)_i \big\|^2

The contrastive objective, the fourth family, has a note to itself.

Vision encoders: the eyes of every modern system

A vision encoder turns an image into a set of feature vectors that other models consume. Almost all of them are Vision Transformers. What differs is how they were trained, and that decides what their features are good at.

Encoder familyTrained byIts features are good atTypical use in robotics
CLIP and SigLIPMatching images with captionsMeaning that lines up with languageVision-language models, open-vocabulary recognition
DINO familySelf-distillation on images onlyFine spatial detail, parts, correspondenceManipulation features, tracking, depth
Masked autoencodersReconstructing hidden patchesLow-level structurePretraining on unlabelled domain data
Video encodersMasked or predictive training on clipsMotion and change over timeDynamics, world models
Three photos labelled mug, basketball and dog by comparing each image with text prompts
A language-aligned encoder classifies with no task training: compare the image embedding with the embedding of each candidate label.

Many current robot models feed two encoders at once: a language-aligned one for meaning and a DINO-style one for spatial precision. Choose the encoder by what the features must capture.

How a representation is judged

A representation is useful if simple things can be read from it with little data. The standard test is a linear probe: freeze the encoder, train a single linear layer on its features for some task, and report the accuracy. If a linear layer suffices, the information was already laid out in an accessible form.

Accuracy of a linear classifier against the number of labelled frames per class, for frozen CLIP features and for raw pixels, on two tasks. Identifying which of 10 tasks a frame shows: CLIP goes from 78 to 98 percent and pixels from 70 to 95. Identifying which of four stages of the episode: both start near chance, and CLIP reaches 73 percent against 63 for pixels at 50 labels.
Measured for this note on 1,516 robot frames, with whole episodes held out. A linear classifier is trained on frozen features from a few labelled frames per class. Mean of eight runs.

Two findings are worth reading closely. With one labelled frame per task, frozen CLIP features identify the task 78% of the time, which is the point of pretraining: the work was done before any labels arrived. And the pixel baseline is not far behind on this question, 70%, because these scenes look very different from one another. On the harder question, how far through the episode a frame is, both start at chance and pretrained features pull ahead only once there are twenty labels.

The lesson is that “good features” is always relative to a question. A language-aligned encoder knows what objects are in the scene. It was never trained to notice how far a gripper has closed.

ProtocolWhat is trainedWhat it measures
k-nearest neighboursNothingWhether similar inputs are already close together
Linear probeOne linear layerWhether the information is linearly readable
Attentive probeA small attention layer over patch featuresWhether it is present in the patches, if not in one summary vector
Fine-tuningEverythingHow good a starting point the weights are

Predicting in feature space

A model forced to predict every pixel spends capacity on detail that cannot be predicted and does not matter, such as texture and sensor noise. A joint-embedding predictive architecture, or JEPA, predicts the representation of the hidden or future part instead.

Visible input goes through an encoder and a predictor; hidden input goes through a slow target encoder; a loss compares the two embeddings
Predict the embedding of the hidden part from the visible part.

For robots this is attractive. What matters for a decision is where the cup will be, not the grain of the table under it.

Embeddings and retrieval

Once similar things land close together in embedding space, hard problems turn into nearest-neighbour lookups. Search becomes “find the closest vectors”. Metric learning trains that distance directly, and it is how product search, re-identification and warehouse item matching work.

Common mistakes

  • Using one encoder for everything. A language-aligned encoder can be weak on fine spatial detail, and the reverse.
  • Assuming features transfer. Features learned on web photos can miss what matters in your domain, such as packaging or radar imagery.
  • Ignoring collapse. Self-supervised training can quietly converge to a constant output. Monitor the variance of the features.

Terms

Embedding
A vector that represents an input so that similar inputs are close.
Pretext task
The invented prediction problem used for self-supervision.
Collapse
When every input maps to nearly the same embedding.
Zero-shot
Doing a task with no training examples for it.
Linear probe
A linear classifier trained on frozen features, used to measure what the features encode.
Pretext task
A task invented so that a model can be trained without labels.
Momentum encoder
A copy of a network whose weights are a slow moving average of another’s.

In my work

Further reading