{ "id": "epe84wmpba", "slug": "multimodal-contrastive-learning", "title": "Multimodal contrastive learning", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai" }, { "id": "technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_computation" }, { "id": "technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.ml_methods" }, { "id": "technology.computing.ai.supervised_unsupervised", "label": "Supervised, unsupervised, and semi-supervised learning", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.ai.supervised_unsupervised" } ], "geo": [ { "id": "geo.nongeo.t2001.technology.computing.ai.supervised_unsupervised", "label": "Non-geographic · 2001 to 2020: Supervised, unsupervised, and semi-supervised learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.supervised_unsupervised", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology" }, { "id": "geo.nongeo.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing" }, { "id": "geo.nongeo.t2001.technology.computing.ai", "label": "Artificial intelligence and data", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_computation", "label": "Machine learning and neural computation", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_computation" }, { "id": "geo.nongeo.t2001.technology.computing.ai.ml_methods", "label": "Machine learning methods", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.ml_methods" }, { "id": "geo.nongeo.t2001.technology.computing.ai.supervised_unsupervised", "label": "Supervised, unsupervised, and semi-supervised learning", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.ai.supervised_unsupervised" } ] } ], "excerpt": "Multimodal contrastive learning is a machine learning approach that aligns embeddings of paired data, such as images and text, prominent through CLIP and ALIGN in 2021.", "snippet": "Multimodal contrastive learning is a machine learning approach that aligns embeddings of paired data, such as images and text, prominent through CLIP and ALIGN in 2021.", "node": "technology.computing.ai.supervised_unsupervised", "markdown": "# Multimodal contrastive learning\n\nMultimodal contrastive learning is a machine learning approach that trains separate encoders for different data modalities, such as images and text, so that matched pairs land close together and mismatched pairs far apart in a shared embedding space. The resulting aligned embeddings support zero-shot classification, cross-modal retrieval, and conditioning of generative models. The approach became prominent with CLIP[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf) and ALIGN[2](https://proceedings.mlr.press/v139/jia21b/jia21b.pdf) in 2021, which showed that contrastive alignment on hundreds of millions of noisy web pairs produces vision-language representations that transfer to tasks the models were never explicitly trained on.\n\n| Key fact | Value |\n|---|---|\n| Core objective | Symmetric cross-entropy over cosine similarities of the \\( N \\) real pairs versus the \\( N^{2}-N \\) incorrect pairings in a batch[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf) |\n| CLIP training data | 400 million web image-text pairs gathered with 500,000 queries, up to 20,000 pairs per query[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf) |\n| CLIP training scale | Minibatch of 32,768; largest ResNet took 18 days on 592 V100 GPUs[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf) |\n| Zero-shot result | CLIP matches the original ResNet-50 on ImageNet without using its 1.28M labeled examples; ALIGN reaches 76.4% top-1 zero-shot[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf) • [2](https://proceedings.mlr.press/v139/jia21b/jia21b.pdf) |\n| Compute reduction | SigLIP's sigmoid loss trains a Large LiT model to 84.5% ImageNet zero-shot on four TPUv4 chips in two days[3](https://openaccess.thecvf.com/content/ICCV2023/papers/Zhai_Sigmoid_Loss_for_Language_Image_Pre-Training_ICCV_2023_paper.pdf) |\n| Modality gap | Images and text embed at arm's length; the default gap of 0.82 on the unit sphere achieves the global minimum of CLIP's loss at its learned temperature[4](https://proceedings.neurips.cc/paper_files/paper/2022/file/702f4db7543a7432431df588d57bc7c9-Paper-Conference.pdf) |\n\n## How it works\n\nGiven a batch of \\( N \\) (image, text) pairs, the model is trained to predict which of the \\( N \\times N \\) possible pairings actually occurred. An image encoder and a text encoder produce embeddings; after L2 normalization, the pairwise cosine similarities are scaled by a temperature to form logits, and a symmetric cross-entropy loss with labels \\( \\mathrm{arange}(n) \\) is computed in both directions and averaged, \\( \\mathcal{L} = (\\mathcal{L}_{i} + \\mathcal{L}_{t})/2 \\).[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf) Minimizing this loss raises the similarity of the \\( N \\) true pairs and lowers it for the \\( N^{2}-N \\) incorrect pairings, which forces the two encoders to place semantically matching content in the same region of the embedding space.\n\nWhy this learns aligned representations has a theoretical answer: maximization of the symmetric InfoNCE loss is achieved when the similarity between two features equals the pointwise mutual information of the two data points up to a constant, so the embedding geometry encodes how much information the modalities share.[5](https://ar5iv.labs.arxiv.org/html/2404.19228) Under linear representation settings, each gradient-descent step on a general class of multimodal contrastive losses, including the CLIP and ALIGN losses, is equivalent to performing SVD on a contrastive cross-covariance matrix.[6](https://ar5iv.labs.arxiv.org/html/2302.06232) The same analysis shows the method can recover core features at a parametric rate as long as observed pairs contain a non-ignorable portion of ground-truth pairs, which explains its robustness to noisy, wrongly matched pairs.[6](https://ar5iv.labs.arxiv.org/html/2302.06232)\n\n## How it is done\n\nA practitioner runs the following steps:\n\n1. **Collect paired data at scale.** CLIP's WIT dataset was built by searching the web for image-text pairs whose text contains one of 500,000 queries, approximately class-balanced by capping each query at 20,000 pairs.[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf) ALIGN instead used noisy image alt-text at scale.[2](https://proceedings.mlr.press/v139/jia21b/jia21b.pdf)\n2. **Choose encoders.** CLIP used a ResNet or Vision Transformer for images and a 63M-parameter 12-layer 512-wide [Transformer](https://www.edgepedia.org/transformer) with 8 attention heads over byte pair encoding for text.[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf)\n3. **Set the temperature.** The temperature \\( \\tau \\) is a learned log-parameterized scalar initialized to the equivalent of 0.07 and clipped to prevent scaling logits by more than 100, which CLIP's authors found necessary to prevent training instability.[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf)\n4. **Use a very large batch.** CLIP trained with a minibatch of 32,768.[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf) SigLIP's authors pushed batch size up to one million and found benefits quickly diminish, with 32k being sufficient.[3](https://openaccess.thecvf.com/content/ICCV2023/papers/Zhai_Sigmoid_Loss_for_Language_Image_Pre-Training_ICCV_2023_paper.pdf) When memory limits batch size, GradCache decouples backpropagation between the contrastive loss and the encoder to enable larger batches.[7](https://arxiv.org/abs/2410.05160)\n5. **Evaluate zero-shot.** Dataset class names are embedded with the text encoder, temperature-scaled cosine similarities are softmax-normalized, and the result is a multinomial logistic regression classifier with L2-normalized inputs and weights.[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf)\n\nCLIP itself removed the non-linear projection head and the text transformation function used by ConVIRT, keeping a random square crop as the only data augmentation.[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf)\n\n## Origin\n\nThe batch construction and objective trace to [deep metric learning](https://www.edgepedia.org/deep-metric-learning): the multi-class N-pair loss (Kihyuk Sohn, 2016, NeurIPS), the InfoNCE loss, and adapted for contrastive text-image learning in medical imaging by ConVIRT (Yuhao Zhang and colleagues, 2020, arXiv).[8](https://doi.org/10.48550/arxiv.2010.00747) Unimodal contrastive learning supplied the remaining machinery through SimCLR (Ting Chen, Simon Kornblith, Mohammad Norouzi, and [Geoffrey Hinton](https://www.edgepedia.org/geoffrey-hinton), 2020, arXiv).[9](https://doi.org/10.48550/arxiv.2002.05709) The ConVIRT paper states that since its original release in 2020 it directly inspired CLIP ([Alec Radford](https://www.edgepedia.org/alec-radford) and colleagues, 2021, arXiv)[10](https://doi.org/10.48550/arxiv.2103.00020) and ALIGN (Chao Jia and colleagues, 2021, arXiv)[11](https://doi.org/10.48550/arxiv.2102.05918), and CLIP is described as a simplified version of ConVIRT trained from scratch.[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf) Some later papers cite ConVIRT as Zhang et al., 2022 rather than 2020; the 2020 release date is the one the ConVIRT paper itself gives.\n\n## Variants\n\n- **ALIGN** uses a dual-encoder architecture, an [EfficientNet](https://www.edgepedia.org/efficientnet) image encoder with global pooling and a BERT text encoder using the [CLS] embedding, trained with the normalized softmax loss on noisy alt-text, treating matched pairs as positives and all other random pairs in the batch as negatives.[2](https://proceedings.mlr.press/v139/jia21b/jia21b.pdf)\n- **SigLIP** replaces the softmax-normalized contrastive loss with a pairwise sigmoid loss that operates solely on image-text pairs and requires no global normalization over the batch.[3](https://openaccess.thecvf.com/content/ICCV2023/papers/Zhai_Sigmoid_Loss_for_Language_Image_Pre-Training_ICCV_2023_paper.pdf)\n- **SigLIP 2** extends the objective with captioning-based pretraining, self-supervised losses, and online data curation, outperforming SigLIP at all four released model scales.[12](https://arxiv.org/pdf/2502.14786)\n- **ImageBind** learns a joint embedding across six modalities, images, text, audio, depth, thermal, and IMU data, using only image-paired data rather than all modality pairs, with an InfoNCE-style loss over positive and negative pairs.[13](https://doi.org/10.48550/arxiv.2305.05665)\n- **MedCLIP** decouples images and texts so usable training data scales combinatorially, and replaces InfoNCE with a semantic matching loss based on medical knowledge to eliminate false negatives.[14](https://doi.org/10.48550/arxiv.2210.10163)\n- **UniCLIP** uses an image encoder that produces an augmentation-agnostic representation and a projection head that outputs an augmentation-aware embedding.[15](https://papers.nips.cc/paper_files/paper/2022/file/072fd0525592b43da661e254bbaadc27-Paper-Conference.pdf)\n- **VLM2Vec** departs from separate encoders: it uses deep integration of vision and language features within a transformer and processes any combination of images and text with task instructions.[7](https://arxiv.org/abs/2410.05160)\n\n## Applications\n\nThe trained encoders are used directly for zero-shot classification, where class names become text anchors[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf), and for cross-modal retrieval: ALIGN outperforms the previous state of the art by over 7% in most zero-shot and fine-tuned R@1 metrics on Flickr30K and MSCOCO.[2](https://proceedings.mlr.press/v139/jia21b/jia21b.pdf) Embeddings also condition generative models; [ImageBind](https://www.edgepedia.org/imagebind) demonstrates audio-to-image generation by using its audio embeddings with a pre-trained DALLE-2 decoder designed to work with CLIP text embeddings.[13](https://doi.org/10.48550/arxiv.2305.05665) In medical imaging, ConVIRT pretraining needs only 10% as much labeled data as an ImageNet-initialized counterpart to reach better or comparable performance on all 4 medical classification tasks evaluated[16](https://proceedings.mlr.press/v182/zhang22a/zhang22a.pdf), and MedCLIP with 20K pre-training samples outperforms a prior method trained on roughly 200K.[14](https://doi.org/10.48550/arxiv.2210.10163) VLM2Vec's MMEB benchmark covers classification, visual question answering, multimodal retrieval, and visual grounding across 36 datasets.[7](https://arxiv.org/abs/2410.05160)\n\n## Limitations and alternatives\n\nThe best-documented failure mode is the modality gap: in models such as CLIP, images and text are embedded at arm's length in the shared space.[4](https://proceedings.neurips.cc/paper_files/paper/2022/file/702f4db7543a7432431df588d57bc7c9-Paper-Conference.pdf) Two accounts of its cause coexist. One attributes the gap to a combination of model initialization, where different random initializations create different embedding cones, and contrastive optimization, and finds that at CLIP's learned final temperature of 1/100 the default gap of 0.82 achieves the global minimum of the loss.[4](https://proceedings.neurips.cc/paper_files/paper/2022/file/702f4db7543a7432431df588d57bc7c9-Paper-Conference.pdf) A later study identifies an information imbalance between images and captions, images carrying more information than captions, as the driving factor behind both the modality gap and object bias.[17](https://proceedings.iclr.cc/paper_files/paper/2025/file/4572bc2f514e627914cbe60d0398a2d1-Paper-Conference.pdf)\n\nThe gap may not be a defect. Increasing it can improve zero-shot classification and fairness performance[4](https://proceedings.neurips.cc/paper_files/paper/2022/file/702f4db7543a7432431df588d57bc7c9-Paper-Conference.pdf), and embedding changes that increase the gap increase the entropy of the logits, giving per-sample control over uncertainty that a global temperature cannot provide.[17](https://proceedings.iclr.cc/paper_files/paper/2025/file/4572bc2f514e627914cbe60d0398a2d1-Paper-Conference.pdf) In off-the-shelf [vision-language models](https://www.edgepedia.org/vision-language-model) the gap's influence on performance is typically overshadowed by other factors, though closing it can lead to improvements.[17](https://proceedings.iclr.cc/paper_files/paper/2025/file/4572bc2f514e627914cbe60d0398a2d1-Paper-Conference.pdf)\n\nA structural limitation is the pairwise objective itself: with three or more modalities, pairwise CLIP fails to capture joint information. Symile, a contrastive approach capturing higher-order (total correlation) information between any number of modalities, retains a large advantage over pairwise CLIP on zero-shot retrieval even when each modality is independently missing with probability 0.5.[18](https://proceedings.neurips.cc/paper_files/paper/2024/file/6828259348d99d5e8994028bfdf15d09-Paper-Conference.pdf) On dataset-size thresholds, published comparisons are indirect: CLIP trained on 400M pairs while publicly available medical images and reports are orders of magnitude fewer[14](https://doi.org/10.48550/arxiv.2210.10163), and swapping a predictive bag-of-words objective for a contrastive one gave a further 4x efficiency improvement in zero-shot transfer to [ImageNet](https://www.edgepedia.org/imagenet)[1](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf), but no head-to-head benchmark has been published quantifying the point at which contrastive alignment beats supervised pretraining.\n\n## References\n\n1. [Learning Transferable Visual Models From Natural Language Supervision (CLIP)](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf)\n2. [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (ALIGN)](https://proceedings.mlr.press/v139/jia21b/jia21b.pdf)\n3. [Sigmoid Loss for Language Image Pre-Training (SigLIP)](https://openaccess.thecvf.com/content/ICCV2023/papers/Zhai_Sigmoid_Loss_for_Language_Image_Pre-Training_ICCV_2023_paper.pdf)\n4. [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://proceedings.neurips.cc/paper_files/paper/2022/file/702f4db7543a7432431df588d57bc7c9-Paper-Conference.pdf)\n5. [Understanding multimodal contrastive learning through pointwise mutual information](https://ar5iv.labs.arxiv.org/html/2404.19228)\n6. [Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data](https://ar5iv.labs.arxiv.org/html/2302.06232)\n7. [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://arxiv.org/abs/2410.05160)\n8. [Zhang, Yuhao and colleagues (2020). Contrastive Learning of Medical Visual Representations from Paired Images and Text. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2010.00747)\n9. [Chen, Ting and colleagues (2020). A Simple Framework for Contrastive Learning of Visual Representations. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2002.05709)\n10. [Radford, Alec and colleagues (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2103.00020)\n11. [Jia, Chao and colleagues (2021). Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2102.05918)\n12. [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://arxiv.org/pdf/2502.14786)\n13. [Girdhar, Rohit and colleagues (2023). ImageBind: One Embedding Space To Bind Them All. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2305.05665)\n14. [Wang, Zifeng and colleagues (2022). MedCLIP: Contrastive Learning from Unpaired Medical Images and Text. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2210.10163)\n15. [UniCLIP: Unified Framework for Contrastive Language–Image Pre-training](https://papers.nips.cc/paper_files/paper/2022/file/072fd0525592b43da661e254bbaadc27-Paper-Conference.pdf)\n16. [Contrastive Learning of Medical Visual Representations from Paired Images and Text (ConVIRT)](https://proceedings.mlr.press/v182/zhang22a/zhang22a.pdf)\n17. [Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models](https://proceedings.iclr.cc/paper_files/paper/2025/file/4572bc2f514e627914cbe60d0398a2d1-Paper-Conference.pdf)\n18. [Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities](https://proceedings.neurips.cc/paper_files/paper/2024/file/6828259348d99d5e8994028bfdf15d09-Paper-Conference.pdf)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Artificial intelligence and data › Machine learning and neural computation › Machine learning methods › Supervised, unsupervised, and semi-supervised learning*\n\n*Initially written Sep 29, 2026 · Reviewed: Sep 30, 2026 · Edited: Oct 15, 2026 · Last review: Sep 30, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/multimodal-contrastive-learning", "markdown_url": "https://www.edgepedia.org/multimodal-contrastive-learning.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Multimodal contrastive learning\", Edgepedia (EdgeChat), https://www.edgepedia.org/multimodal-contrastive-learning. Edgepedia Community License 1.0.", "credit_md": "\"[Multimodal contrastive learning](https://www.edgepedia.org/multimodal-contrastive-learning)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/multimodal-contrastive-learning](https://www.edgepedia.org/multimodal-contrastive-learning). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Multimodal contrastive learning\", Edgepedia (EdgeChat), https://www.edgepedia.org/multimodal-contrastive-learning. Edgepedia Community License 1.0.", "speakable": "Multimodal contrastive learning is a machine learning approach that aligns embeddings of paired data, such as images and text, prominent through CLIP and ALIGN in 2021." }