Contrastive learning in VLMs
A simple idea that made text and vision alignment possible
If there is one idea that completely transformed how machines understand images and text together, it is contrastive learning. It’s a simple idea mathematically, but its impact was so big that it enabled OpenAI’s CLIP model - one of the most influential vision-language models ever created.
In this article, we will break down:
What contrastive learning really means
Why it became essential for vision-language models
The math behind contrastive loss
How CLIP uses this idea to align images and text
Why temperature (τ) in the loss formula matters more than you think
1. The Problem: How Do You Make Images and Text Speak the Same Language?
A basic vision-language model has two parts:
An image encoder (like a Vision Transformer or CNN)
A text encoder (like BERT or a Transformer decoder)
Both convert their inputs into vector embeddings. Now the challenge is this — if we give an image of a dog and the text “a happy dog running towards the camera,” both their embeddings should be close in the joint embedding space.
If we can achieve this reliably, then tasks like image search, captioning, and text-to-image similarity become natural.
But how do we train a model to do this?
2. The Core Idea: Contrastive Learning
Contrastive learning solves one simple problem —
Bring similar things closer, and push dissimilar things apart in the embedding space.
To do this, we create two types of pairs:
So the model learns a metric space — where correct image-text pairs are close, and unrelated pairs are far apart.
3. Positive Pairs Through Augmentation: Creating Two Views of One Image
In pure image-only contrastive learning (like SimCLR), positive pairs are created by applying different augmentations to the same image:
Random cropping
Horizontal flip
Color jitter
Grayscale
Rotation
Gaussian blur
Both augmented images still mean the same thing — so the model must learn that they belong together.
4. The Math Behind Contrastive Loss
Let’s say:
We take an image as anchor → create two augmented versions
Pass them through an encoder + projection head
Get two embeddings: zᵢ and zⱼ
We want them to be similar, so we use cosine similarity:
To convert this into a loss, we use softmax + negative log likelihood:
Where:
τ is temperature
Lower τ makes softmax sharper → model focuses more on hard negatives
Higher τ gives smoother gradients → slower, but more stable learning
5. CLIP: Contrastive Learning with Images and Text
OpenAI took this idea and scaled it to 400 million image-text pairs from the internet.
For a batch of N images and N texts:
Image embeddings → I₁, I₂, ..., Iₙ
Text embeddings → T₁, T₂, ..., Tₙ
The only correct pairs are (I₁, T₁), (I₂, T₂), ..., (Iₙ, Tₙ).
Everything else is a negative pair.
So CLIP uses symmetric contrastive loss:
This makes sure that:
Each image is closest to its caption
Each caption is closest to its image
This simple idea made CLIP capable of zero-shot image classification. No need for ImageNet fine-tuning. Just type:
“A photo of a golden retriever.”
And compare its embedding with image embeddings. The highest cosine similarity wins.
6. Why Contrastive Learning Works So Well
✔ No need for labelled datasets like ImageNet.
✔ Only needs (image, caption) pairs — which are freely available on the internet.
✔ Learns a universal visual-semantic understanding.
✔ Works for classification, retrieval, and multimodal reasoning.
7. What’s Next?
In the next lecture/article, we will build a Nano Vision-Language Model from scratch and implement the same CLIP-style loss function.
You will see:
Custom image and text encoder
Contrastive loss implementation
Training on a mini dataset
YouTube lecture
Join Vision Transformer PRO
– Access to all lecture videos
– Hand-written notes
– Private GitHub repo
– Private Discord
– “Transformers for Vision” book by Team Vizuara (PDF)
– Email support
– Hands-on assignments
– Certificate