# Florence-2 Florence-2 is a small, open vision foundation model from Microsoft that handles captioning, object detection, visual grounding, referring expression segmentation, and related vision-language tasks in a single model, by treating each task as a text prompt followed by a generated text response. The paper describing it was posted in November 2023 (arXiv 2311.06242), the work was peer-reviewed and published at CVPR 2024, and the model checkpoints were released on [Hugging Face](https://www.edgepedia.org/hugging-face) on 16 June 2024 under the MIT license.[1](https://ar5iv.labs.arxiv.org/html/2311.06242) • [2](https://openaccess.thecvf.com/content/CVPR2024/papers/Xiao_Florence-2_Advancing_a_Unified_Representation_for_a_Variety_of_Vision_CVPR_2024_paper.pdf) • [3](https://huggingface.co/microsoft/Florence-2-base) • [4](https://huggingface.co/docs/transformers/model_doc/florence2) "Unified" here means that one model covers tasks normally split across separate systems. Microsoft describes it as a prompt-based model that takes text prompts as task instructions and produces results in text form, whether captioning, object detection, grounding, or segmentation.[5](https://www.microsoft.com/en-us/research/publication/florence-2-advancing-a-unified-representation-for-a-variety-of-vision-tasks/) It comes in two small variants: Florence-2-B with 232 million parameters and Florence-2-L with 771 million parameters.[1](https://ar5iv.labs.arxiv.org/html/2311.06242) | Key fact | Detail | |---|---| | Maker and release | Microsoft; paper November 2023, CVPR 2024, Hugging Face release 16 June 2024[1](https://ar5iv.labs.arxiv.org/html/2311.06242) • [2](https://openaccess.thecvf.com/content/CVPR2024/papers/Xiao_Florence-2_Advancing_a_Unified_Representation_for_a_Variety_of_Vision_CVPR_2024_paper.pdf) • [4](https://huggingface.co/docs/transformers/model_doc/florence2) | | Sizes | Florence-2-B: 232M parameters; Florence-2-L: 771M parameters[1](https://ar5iv.labs.arxiv.org/html/2311.06242) | | Tasks in one model | Captioning, object detection, grounding, referring segmentation, via text prompts[5](https://www.microsoft.com/en-us/research/publication/florence-2-advancing-a-unified-representation-for-a-variety-of-vision-tasks/) | | Training data | FLD-5B: 5.4 billion annotations on 126 million images, built by automated annotation and model refinement[5](https://www.microsoft.com/en-us/research/publication/florence-2-advancing-a-unified-representation-for-a-variety-of-vision-tasks/) | | Headline vendor result | 135.6 CIDEr zero-shot COCO captioning for the large model, versus 84.3 for the 80B Flamingo[1](https://ar5iv.labs.arxiv.org/html/2311.06242) | | License | MIT, four public checkpoints (base/large, each with a fine-tuned -ft variant)[3](https://huggingface.co/microsoft/Florence-2-base) | | Transformers support | Added to Hugging Face Transformers on 20 August 2025[4](https://huggingface.co/docs/transformers/model_doc/florence2) | ## Architecture and training The published design combines a DaViT vision encoder, which converts images into visual token embeddings, with a transformer-based multi-modal encoder-decoder initialized from UniCL and BART respectively for the two components.[1](https://ar5iv.labs.arxiv.org/html/2311.06242) • [2](https://openaccess.thecvf.com/content/CVPR2024/papers/Xiao_Florence-2_Advancing_a_Unified_Representation_for_a_Variety_of_Vision_CVPR_2024_paper.pdf) Image token embeddings are concatenated with text embeddings and the encoder-decoder generates the response.[2](https://openaccess.thecvf.com/content/CVPR2024/papers/Xiao_Florence-2_Advancing_a_Unified_Representation_for_a_Variety_of_Vision_CVPR_2024_paper.pdf) Every task is a translation problem. Given an input image and a task-specific prompt, the model generates the corresponding output, trained with a standard language-modeling objective.[1](https://ar5iv.labs.arxiv.org/html/2311.06242) Regions and boxes are expressed as text using 1,000 quantized coordinate bins, with region formats tailored to each task's requirements, so detection and grounding need no task-specific output heads.[1](https://ar5iv.labs.arxiv.org/html/2311.06242) Training data is the FLD-5B dataset, which Microsoft co-developed for this model: 5.4 billion visual annotations covering 126 million images, produced with an iterative strategy of automated image annotation and model refinement rather than large-scale human labeling.[5](https://www.microsoft.com/en-us/research/publication/florence-2-advancing-a-unified-representation-for-a-variety-of-vision-tasks/) • [4](https://huggingface.co/docs/transformers/model_doc/florence2) The published schedule used mini-batches of 2,048 (base) and 3,072 (large) at 384×384 resolution until 3 billion effective training samples, followed by high-resolution tuning at 768×768.[1](https://ar5iv.labs.arxiv.org/html/2311.06242) ## Benchmark results (vendor-reported) Microsoft reported the quantitative results below in the paper and on the model card. | Benchmark | Florence-2-base (0.23B) | Florence-2-large (0.77B) | |---|---|---| | COCO captioning, zero-shot CIDEr | 133.0 | 135.6 | | NoCaps CIDEr | 118.7 | 120.8 | | TextCaps CIDEr | 70.1 | 72.8 | | Flickr30k test Recall@1 | 83.6 | 84.4 | | COCO detection val2017 mAP | 34.7 | 37.5 | | RefCOCO val accuracy | 53.9 | 56.3 | | RefCOCO+ val accuracy | 51.5 | 53.6 | | RefCOCOg val accuracy | 66.3 | 68.0 | | RefCOCO referring segmentation mIoU | 34.6 | 35.8 | Sources: the paper and the Hugging Face model card.[1](https://ar5iv.labs.arxiv.org/html/2311.06242) • [3](https://huggingface.co/microsoft/Florence-2-base) One discrepancy is unresolved between the two vendor sources. The paper reports zero-shot COCO detection mAP of 19.7 (base) and 22.6 (large), while the model card table lists 34.7 and 37.5 for COCO detection val2017 without labeling the evaluation condition.[1](https://ar5iv.labs.arxiv.org/html/2311.06242) • [3](https://huggingface.co/microsoft/Florence-2-base) The sources do not reconcile the two figures, so readers should treat the detection numbers as condition-dependent and check which setting a comparison uses. ## How it compares with larger and specialist models The paper's comparisons, again vendor-reported, are against much larger general models and against task specialists. On zero-shot COCO captioning, Florence-2-L scored 135.6 CIDEr against 84.3 for the 80B-parameter Flamingo, using less than 1% of Flamingo's parameters.[1](https://ar5iv.labs.arxiv.org/html/2311.06242) On grounding, zero-shot Florence-2-L improved Flickr30k Recall@1 by 5.7 points over the 1.6B Kosmos-2 (84.4 versus 78.7) and improved RefCOCO, RefCOCO+, and RefCOCOg by roughly 4%, 8%, and 8% absolute; the 35.8 mIoU referring segmentation result is a capability the authors say prior foundation models did not support.[1](https://ar5iv.labs.arxiv.org/html/2311.06242) Against specialists, the CVPR paper reports that after fine-tuning on public human-annotated data the compact Florence-2 competes with larger specialist models and set new state-of-the-art results on RefCOCO/+/g.[2](https://openaccess.thecvf.com/content/CVPR2024/papers/Xiao_Florence-2_Advancing_a_Unified_Representation_for_a_Variety_of_Vision_CVPR_2024_paper.pdf) Specifically, Florence-2-L outperformed PolyFormer, a specialist referring-expression model, by 3.0 Accuracy@0.5 on RefCOCO REC and 3.54 mIoU on RES.[1](https://ar5iv.labs.arxiv.org/html/2311.06242) ## Licensing, availability and ecosystem Four checkpoints are published on Hugging Face under the MIT license: Florence-2-base (0.23B) and Florence-2-large (0.77B), each pretrained on FLD-5B, plus fine-tuned -ft variants of each trained on a collection of downstream tasks.[3](https://huggingface.co/microsoft/Florence-2-base) The model was released on Hugging Face on 16 June 2024 and added to the [Hugging Face Transformers](https://www.edgepedia.org/hugging-face-transformers) library on 20 August 2025, which made it usable through the standard Transformers API.[4](https://huggingface.co/docs/transformers/model_doc/florence2) ## References 1. [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (arXiv 2311.06242)](https://ar5iv.labs.arxiv.org/html/2311.06242) 2. [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (CVPR 2024)](https://openaccess.thecvf.com/content/CVPR2024/papers/Xiao_Florence-2_Advancing_a_Unified_Representation_for_a_Variety_of_Vision_CVPR_2024_paper.pdf) 3. [microsoft/Florence-2-base model card (Hugging Face)](https://huggingface.co/microsoft/Florence-2-base) 4. [Florence-2 in Hugging Face Transformers documentation](https://huggingface.co/docs/transformers/model_doc/florence2) 5. [Florence-2 publication page, Microsoft Research](https://www.microsoft.com/en-us/research/publication/florence-2-advancing-a-unified-representation-for-a-variety-of-vision-tasks/) --- *Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Model families and named models › Multimodal, vision, and world models* *Initially written Sep 17, 2026 · Edited: Oct 11, 2026; Oct 15, 2026* *Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.* License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license