microsoft/florence-2-base-ft

Florence-2 multi-task vision model (caption / detection / OCR). Accepts exactly one image_url; no text-only, video, or document_url input. Default method: caption. The caption and ocr methods emit a Markdown string; the region methods emit a region container.

Output by method

Method	Payload kind	`content.object`	Payload
`caption` (default)	markdown	none	Short image caption
`detailed_caption`	markdown	none	Longer caption
`more_detailed_caption`	markdown	none	Longest caption
`ocr`	markdown	none	All text on the image
`od`	json	`florence_2.od.regions`	Region items with `bbox_xywh` and `label`
`dense_region_caption`	json	`florence_2.dense_region_caption.regions`	Region items with `bbox_xywh` and `label`
`region_proposal`	json	`florence_2.region_proposal.regions`	Region items with `bbox_xywh`, no `label`
`ocr_with_region`	json	`florence_2.ocr_with_region.regions`	Region items with `bbox_xywh` and `label`

Each method maps to one Florence-2 task token. The task token itself (<CAPTION>, <OD>, …) is an implementation detail and is not part of the reply: a caption is the string, and a region method is the container. Boxes are normalized 0-1 against the image at precision decimals (default 4), and a region method with no hits returns "items": [].

Request

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key="<VLMRUN_API_KEY>",
)

response = client.chat.completions.create(
    model="microsoft/florence-2-base-ft",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"
                    },
                },
            ],
        }
    ],
    extra_body={"method": "caption"},
)

print(response.choices[0].message.content)

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "microsoft/florence-2-base-ft",
    "method": "caption",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "image_url",
            "image_url": {
              "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"
            }
          }
        ]
      }
    ]
  }'

Response

Text mode (caption)
JSON mode (caption)
JSON mode (od)

The caption string alone, with no wrapper:

A cat sitting on a windowsill.

One image entry; content is the string:

{
  "model": "microsoft/Florence-2-base-ft",
  "method": "caption",
  "data": [
    {
      "object": "image",
      "image_hash": "sha256:...",
      "image_width": 640,
      "image_height": 480,
      "content": "A cat sitting on a windowsill."
    }
  ]
}

For a detection method (method="od"), content is the region container:

{
  "model": "microsoft/Florence-2-base-ft",
  "method": "od",
  "data": [
    {
      "object": "image",
      "image_hash": "sha256:...",
      "image_width": 640,
      "image_height": 480,
      "content": {
        "object": "florence_2.od.regions",
        "items": [
          {"bbox_xywh": [0.0531, 0.0417, 0.275, 0.3333], "label": "cat"},
          {"bbox_xywh": [0.0781, 0.125, 0.1094, 0.1458], "label": "dog"}
        ]
      }
    }
  ]
}

Overview

Guides

API Reference

Misc

microsoft/florence-2-base-ft

Output by method

Request

Response

​Output by method

​Request

​Response

Output by method

Request

Response