← All topics

Guide · Models · Topic 7 of 32

Contrastive and multimodal learning

Learning by comparison: triplet and batch-wide losses, mining, and how two kinds of data are brought into one space.

8 minute readBuilds on: Representation learning

Learning by comparison

Most losses compare a prediction with a correct answer. A contrastive loss needs no answer at all. It is told only which things belong together, and it shapes an embedding space so that those land close and everything else lands far apart.

That is enough to learn a great deal, because “belong together” is cheap to obtain. Two photos of the same product belong together. An image and its caption belong together. A camera frame and the touch reading recorded in the same instant belong together. The second and third are how models are taught to relate different kinds of data, which is what multimodal learning means.

The losses

Two diagrams of an anchor point, a positive and a negative. Before: the negative is closer to the anchor than the positive. After: the negative has been pushed out beyond the positive’s distance plus a margin.
A triplet. Training moves the points until the negative is further from the anchor than the positive is, by at least the margin.

The triplet loss works on three items at a time: an anchor, a positive that should be near it, and a negative that should not.

L = max( 0, d(anchor, positive) − d(anchor, negative) + margin )

The loss is zero once the negative is further away than the positive by the margin. That zero is its weakness as well as its point. Most randomly chosen triplets already satisfy it and teach nothing, so training depends on finding the triplets that do not. That search is called mining.

LossComparesStrengthWeakness
Contrastive (pairwise)Two items: pull together if matched, push apart up to a margin if notSimplestUses little of each batch
TripletAnchor, positive, negativeDirectly states the ranking wantedMost triplets are uninformative; needs mining
InfoNCEOne positive against every other item in the batch, as a classificationUses every negative in the batch; no miningWants large batches; has a temperature to tune
Supervised contrastiveAs InfoNCE, with every same-class item as a positiveUses labels when they existNeeds labels
Sigmoid (SigLIP)Every pair in the batch, each as its own yes or noWorks at small and very large batch sizesNeeds a bias term and care with the many negatives

InfoNCE is the loss behind CLIP. Take a batch of N matched pairs. For each item, compute its similarity to all N candidates, divide by a small number called the temperature, and apply cross entropy with the true partner as the correct class.

L = − log [ exp( sii ÷ τ ) ÷ Σj exp( sij ÷ τ ) ]

Every other item in the batch is a negative for free, which is why these models are trained with batches of thousands. The temperature sets how hard the loss concentrates on the most confusing negatives: a low value attends almost only to the hardest ones.

Three losses on the same data

Two scatter plots of handwritten-digit embeddings in two dimensions, coloured by digit. Before training the ten digits overlap in one cloud. After training with a triplet loss they form ten separate clusters.
Run for this note on a public set of 1,797 handwritten digits. A small network maps each image to a point in two dimensions. Left: before training. Right: after training with a triplet loss. Nothing told it there were ten classes, only which images matched.
Retrieval accuracy against training passes for three losses. Triplet with random negatives reaches 98.0 percent. Triplet with the hardest negatives in the batch reaches 98.9. Contrast against the whole batch reaches 99.2 and gets there fastest.
The same data with a 16-dimensional embedding, mean of five runs. The score is how often an image’s nearest neighbour is the same digit. An untrained network scores 84%.

Two lessons hold beyond this small case. Choosing negatives matters: taking the hardest ones in each batch cut the error of the triplet loss roughly in half, from 2.0% to 1.1%. And contrasting against the whole batch did a little better again, and learned fastest, with nothing to mine. This is why InfoNCE-style losses displaced triplets for large-scale pretraining, and why triplets with hard mining remain common where data is labelled and batches are small.

What the loss does to the space

With embeddings normalised to unit length, every point lies on a sphere and similarity is the cosine of the angle between two points. A contrastive loss can then be understood as pursuing two goals at once.

Lalign=E(x,x+)∥f(x)−f(x+)∥2Luniform=log⁡ Ex,y  e−2∥f(x)−f(y)∥2\mathcal{L}_{\text{align}} = \mathbb{E}_{(x, x^+)} \big\| f(x) - f(x^+) \big\|^2 \qquad\qquad \mathcal{L}_{\text{uniform}} = \log\, \mathbb{E}_{x, y}\; e^{-2 \| f(x) - f(y) \|^2}

Alignment pulls matched pairs together. Uniformity spreads all points evenly over the sphere, which preserves as much information as possible. In the InfoNCE loss the numerator is the alignment term and the denominator, summed over the batch, is the uniformity term. This also explains collapse: alignment alone is minimised by mapping everything to one point, and it is the negatives that prevent it.

For two modalities the loss is applied in both directions and averaged. With image embeddings uiu_i, text embeddings viv_i and a learned temperature τ\tau:

LCLIP=−12N∑i=1N[log⁡eui⊤vi/τ∑jeui⊤vj/τ+log⁡eui⊤vi/τ∑jeuj⊤vi/τ]\mathcal{L}_{\text{CLIP}} = -\frac{1}{2N} \sum_{i=1}^{N} \left[ \log \frac{e^{u_i^{\top} v_i / \tau}}{\sum_{j} e^{u_i^{\top} v_j / \tau}} + \log \frac{e^{u_i^{\top} v_i / \tau}}{\sum_{j} e^{u_j^{\top} v_i / \tau}} \right]

The sigmoid variant drops the softmax and treats every pair in the batch as its own yes-or-no question, with a label zijz_{ij} of +1 on the diagonal and −1 elsewhere:

LSigLIP=−1N∑i∑jlog⁡σ(zij (t ui⊤vj+b))\mathcal{L}_{\text{SigLIP}} = -\frac{1}{N} \sum_{i} \sum_{j} \log \sigma\big( z_{ij}\, ( t\, u_i^{\top} v_j + b ) \big)

Because no term depends on the rest of the batch, the loss can be computed in pieces across machines, which is why it scales to very large batches and also works at small ones.

How much do temperature and batch size matter? On the small digits problem above, not at all: sweeping the temperature from 0.01 to 1 moved retrieval accuracy only between 98.5% and 99.1%, and batch sizes from 8 to 256 between 98.2% and 98.9%. They matter at scale, where the supply of hard negatives in a batch is what limits learning. A hyperparameter’s reputation comes from the regime it was studied in.

Two modalities, one space

Apply the same loss across two kinds of data and the result is a shared embedding space. CLIP has an image encoder and a text encoder, and trains them so that an image and its caption land at the same point. Afterwards a picture can be compared with a sentence directly, which gives recognition from a text description with no training on the classes involved.

A ten by ten grid of probabilities that each of ten robot camera frames matches each of ten task instructions, from a CLIP model. The correct matches lie on the diagonal. The model’s choice is correct for only two of the ten frames, and several frames are assigned the same instruction.
A real test for this note: a public CLIP model given ten LIBERO frames and their ten task instructions, asked which instruction goes with which frame. It gets 2 of 10.

The result is a useful warning. The same model recognises everyday photos very well. It fails here for three reasons that recur in robotics. The scenes are nearly identical tabletops that differ in small objects. The instructions are long and compositional, and a single vector cannot hold “the white mug on the left plate and the yellow mug on the right”. And the images are simulated, which is far from what the model was trained on. Matching one vector against one vector is a blunt instrument, and this is the main reason robot models use the vision encoder’s patch features inside a language model instead. The note on vision-language models picks up there.

Beyond images and text

  • More senses. The same recipe aligns sound, depth, touch and motion with images. Because each is paired with images, they become comparable with each other without ever having been paired directly.
  • Time. Frames close together in a video are positives and distant frames are negatives. The embedding learns what changes slowly, which is often the task-relevant state.
  • Views. Two cameras looking at the same moment form a positive pair, which teaches features that do not depend on viewpoint.
  • Touch and vision. A fingertip image and the camera image of the same contact, paired, give a touch encoder a starting point that does not depend on scarce labelled touch data.

How it goes wrong

  • Collapse. If nothing pushes points apart, mapping everything to one point satisfies the loss. Negatives prevent it, and methods without negatives need another device, as the representation learning note shows.
  • False negatives. Two different images of the same thing in one batch are treated as a mismatch and pushed apart. It worsens as batches grow.
  • Shortcuts. If positives share something trivial, such as the same background or camera, the model learns that and nothing else.
  • The modality gap. Even after training, images and texts tend to occupy separate regions of the shared space. Distances across modalities are meaningful only in relative terms.

Common mistakes

  • Training a triplet loss without mining. Most triplets contribute nothing after the first few passes.
  • Leaving the temperature at a default. It changes which negatives matter and is worth tuning.
  • Expecting one vector to carry a sentence. Fine for “a photo of a mug”. Not for a multi-step instruction.

Terms

Embedding
A vector that represents an item so that similar items are close.
Positive and negative
Items that should and should not be close to the anchor.
Hard negative
A negative that the model currently places close to the anchor.
Mining
Selecting informative pairs or triplets to train on.
Temperature
A scale applied to similarities before the softmax; lower values sharpen the focus on the hardest negatives.
Modality gap
The tendency of different data types to occupy separate regions of a shared embedding space.
Alignment and uniformity
The two things a contrastive loss optimises: matched pairs close together, and all points spread evenly.

Further reading