Methods & Response Format

Two request fields decide what a VLM Run Gateway model returns:

method selects what the model computes (ocr, detect, parse_layout, chat, …).
response_format selects how that result is serialized (a text rendering, or a JSON object).

They are orthogonal: any method combines with any format, and the two renderings carry the same information. This page is the canonical reference for both. For copy-pasteable per-model examples, see Models.

The `method` field

Most models expose one or more method values. Pass method (and optional method_params) at the top level of the request body, or via extra_body in the OpenAI Python SDK. If you omit method, the model’s default is applied: markdown on zai-org/glm-ocr, deepseek-ai/deepseek-ocr-2, rednote-hilab/dots.mocr, baidu/unlimited-ocr, and paddlepaddle/paddleocr-vl-1.6, ocr on paddleocr/pp-ocrv6, caption on microsoft/florence-2-base-ft, pose on usyd-community/vitpose-plus-large, and chat on the chat VLMs.

response = client.chat.completions.create(
    model="paddleocr/pp-ocrv6",
    messages=[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg"}}]}],
    extra_body={"method": "ocr"},
)

const response = await client.chat.completions.create({
  model: "paddleocr/pp-ocrv6",
  messages: [{ role: "user", content: [{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg" } }] }],
  method: "ocr",
});

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg \
  -m paddleocr/pp-ocrv6 --method ocr

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "paddleocr/pp-ocrv6",
    "method": "ocr",
    "messages": [{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg"}}]}]
  }'

Every model’s live methods, default_method, and extra_body_help are also available on GET /v1/openai/models. A method a model does not advertise is a 400.

The `response_format` field

response_format mirrors the OpenAI object, so existing clients work unchanged:

`response_format`	Reply
omitted or `{"type":"text"}`	The text rendering.
`{"type":"json_object"}`	One JSON object.

A JSON response_format is always served non-streamed: the stream flag is ignored, because a single valid JSON object cannot be assembled from SSE deltas. In text mode, stream: true is honored and the stream is byte-identical to the non-streaming reply: the <document> open tag, then one chunk per <page> block, then the close tag. See Streaming.

The `precision` field

precision (int, range 1-8, default 4) sets the number of decimal places on normalized coordinates (bbox_xywh / poly_xy / point_xy) and score. It applies to both formats and is ignored for markdown payloads. The response shape is unchanged.

Text mode

A text reply is a sequence of top-level blocks, one per input medium. Each block body is either a json block or a markdown block, and nothing else. Json is the default. A method renders a markdown block only when it emits genuinely free-form text: markdown, chat, caption, text, free_ocr, and the ocr of rednote-hilab/dots.mocr and deepseek-ai/deepseek-ocr-2. Which one applies is a property of the (model, method) pair, and it is echoed on the wire as format=. See Methods by model for the mapping.

Document input

A PDF is one <document> block wrapping one <page> block per rasterized page.

<document file_name="invoice.pdf" file_hash="sha256:1b7f04c3…" file_bytes="48213" mimetype="application/pdf" npages="3" dpi="150">
<page id="0" format="json" width="1275" height="1650">
{"object": "doc.page.blocks", "items": [{"block_id": 0, "bbox_xywh": [0.0332, 0.0138, 0.1719, 0.0331], "text": "Invoice", "score": 0.998}]}
</page>
<page id="1" format="markdown" width="1275" height="1650">
## Line items

| SKU | Qty |
| --- | --- |
</page>
<page id="2" format="json" width="1275" height="1650" status="error"/>
</document>

Element	Attributes
`<document>`	`file_name`?, `file_hash`?, `file_bytes`?, `mimetype`, `npages`, `dpi`, `language`?, in that order. `file_name` is omitted for `data:` URI uploads. `language` is a comma-separated list of ISO-639 codes in confidence order, and it is present only when the read reports one.
`<page>`	`id` (zero-based), `format` (`json` or `markdown`), `width`, `height`, `status`?.

Each attribute is the JSON field of the same record with the record prefix dropped, because the tag already names the record: <page id> is page_id, <document npages> is document_npages. A failed page is self-closing, with status="error" and no body, so page numbering stays intact. format is the same on every page of a request, and it is emitted on every page.

Single image input

The block alone, with no wrapper:

{"object": "doc.ocr.lines", "items": [{"bbox_xywh": [0.0332, 0.0138, 0.1719, 0.0331], "text": "Invoice", "score": 0.998}]}

or, for a markdown-kind method:

# Annual Report 2024

## Overview

This is the one form that carries no image_hash / image_width / image_height. Use JSON mode when you need that metadata.

One input per structured request

A structured request carries exactly one medium: one image, one PDF, or one video. A second image_url part is a 400 capability error naming the model’s max_images, and mixing document_url with image_url or video_url is refused the same way. Only the chat VLMs take several images in one message, and they are not enveloped at all.

Chat VLMs pass through

The chat VLMs (qwen/qwen3.5-0.8b and the rest of the chat family) return their reply verbatim in every case: text, one image, or several. No wrapper, no tags, byte-identical to the model’s own output. They do not accept document_url: a PDF request is a 400 capability error.

JSON mode

With response_format={"type":"json_object"} the reply is one flat object: model, method, the input’s own details, then the payload. There is no data list to index, because a structured request carries exactly one input. The chat VLMs are the one exemption; see Chat VLMs are not enveloped. model and method name what produced the payload, and both keys are JSON-mode only. The remaining top-level keys are the input’s: image_* for an image, video_* for a clip, file_* (the source file’s identity) plus document_* for a PDF. A reply carries one of those sets, never two. An image or clip then carries content; a PDF carries pages.

Image reply
Video reply
Document reply

{
  "model": "paddleocr/pp-ocrv6",
  "method": "ocr",
  "image_hash": "sha256:…",
  "image_width": 1024,
  "image_height": 1448,
  "content": { "object": "doc.ocr.lines", "items": [ <line>, ... ] }
}

The hash and dimensions are omitted only when the image cannot be decoded. The payload type lives on the content container’s own object.

{
  "model": "usyd-community/vitpose-plus-large",
  "method": "pose",
  "video_hash": "sha256:…",
  "video_width": 1280,
  "video_height": 720,
  "video_fps": 23.976,
  "video_nframes": 6059,
  "video_duration": 252.711,
  "content": { "object": "vid.pose.kpts", "items": [ <item>, ... ], "frames": [ <frame>, ... ] }
}

video_fps, video_nframes and video_duration describe the source clip, not the frames that were sampled. What was sampled is the frames table on the container. See Video payloads.

{
  "model": "zai-org/GLM-OCR",
  "method": "markdown",
  "file_name": "invoice.pdf",
  "file_hash": "sha256:1b7f04c3…",
  "file_bytes": 48213,
  "document_mimetype": "application/pdf",
  "document_npages": 3,
  "document_dpi": 150,
  "pages": [
    {
      "object": "doc.page",
      "page_id": 0,
      "page_width": 1275,
      "page_height": 1650,
      "content": { "object": "doc.page.blocks", "items": [ <block>, ... ] }
    },
    {
      "object": "doc.page",
      "page_id": 1,
      "page_width": 1275,
      "page_height": 1650,
      "status": "error"
    }
  ]
}

A page record carries object, page_id, page_width, page_height, content, and optional status. A failed page carries "status": "error" and no content, so page numbering stays intact. file_name is omitted for inline (data:) uploads, document_npages counts the pages actually read after any document_pages selection, and the optional document_language is document-level.

`content` per medium

Medium	`content`
Document page	Always `{"object": "doc.page.blocks", "items": [ <block>, ... ]}`, whatever method read the page.
Image, json kind	`{"object": "img." \| "doc." \| "world.*", "items": [ <item>, ... ]}`. `items` is `[]` when nothing is found.
Video, json kind	`{"object": "vid.*", "items": [ ... ], "frames": [ ... ]}`. See Video payloads.
Image or video, markdown kind	The string itself, not wrapped.

A json payload is never a bare array, and a markdown payload is never wrapped. An image content has exactly one shape per (model, method).

{
  "model": "paddleocr/pp-ocrv6",
  "method": "ocr",
  "image_hash": "sha256:…",
  "image_width": 1024,
  "image_height": 1448,
  "content": {
    "object": "doc.ocr.lines",
    "items": [
      {
        "bbox_xywh": [0.0332, 0.0138, 0.1719, 0.0331],
        "text": "Invoice",
        "score": 0.998
      }
    ]
  }
}

{
  "model": "zai-org/GLM-OCR",
  "method": "markdown",
  "image_hash": "sha256:…",
  "image_width": 1700,
  "image_height": 2200,
  "content": "# Annual Report 2024\n\n## Overview"
}

{
  "object": "doc.page.blocks",
  "items": [
    {
      "block_id": 0,
      "text": "# Annual Report 2024\n\n## Overview"
    }
  ]
}

The item record

One entry of an image or video content.items, that is, one thing the model found. items is a flat list: one row per thing, never nested per frame or per instance.

Key	Type	Notes
`bbox_xywh`	`[x, y, w, h]`	The single universal box: normalized 0-1, `precision` dp. There is no `bbox`, no `bbox_norm`, and no pixel or xyxy variant.
`poly_xy`	`[[x, y], ...]`	Normalized outline, when the model emits one (`paddleocr/pp-ocrv6`).
`polys_xy`	`[[[x, y], ...], ...]`	Mask outline rings, on `facebook/sam3.1` with `method_params.polygons: true`. Holes wind opposite to their outer ring.
`label`	`string`	Class, prompt match, or layout category. A layout category is folded to lower `snake_case`, so `doc_title` and `section_header` read the same whichever layout model produced them. A prompt or open-vocabulary label is verbatim.
`text`	`string`	Recognized text.
`score`	`float`	Confidence 0-1.
`area`	`float`	Mask pixels as a fraction of the frame, 0-1. A cheap scalar to filter on without decoding the mask.
`kpts_xy`	`[[x, y], ...]`	Normalized joints. Names are on the container as `kpts_labels`. A joint may fall outside 0-1.
`kpts_score`	`[float]`	One confidence per joint, in `kpts_xy` order, as the model reports it.
`kpts_xyz`	`[[x, y, z], ...]`	Joints in metric 3D, on `world.*` payloads only.
`instance_id`	`int`	1-255. This instance’s pixel value in the container’s label map. Image payloads only.
`frame_id`	`int`	Source frame number. Video payloads only.
`track_id`	`int`	Identity held across frames, 1-255, and the pixel value in each frame’s label map. Video payloads only.
`order`	`int`	A detector’s own reading order, unedited. May start at 1.

Unset optional keys are omitted. Which keys a given (model, method) emits is fixed, and an unknown key is a validation error, not a silently ignored extra.

Segmentation masks are a label map

A *.segment.masks payload carries no per-item mask. Every instance lives in one 8-bit grayscale PNG on the container, where a pixel’s value is the item’s instance_id (image) or track_id (video). Pixel 0 is background. Ids therefore start at 1 and run to 255, which is also the ceiling on instances in one map. To cut out instance n, test pixel == n; to get the foreground of the whole frame, test pixel != 0.

"mask": {
  "format": "png",
  "height": 425,
  "width": 640,
  "data": "data:image/png;base64,iVBORw0KGgo…"
}

One decode gives every instance, and area, bbox_xywh and score sit beside it so a filter never has to decode at all. On a video the map is per frame, at frames[].mask. method_params.mask_format: "none" drops it and keeps area.

Video payloads

A vid.* payload is its img.* counterpart plus time. Items carry frame_id and track_id, and the container carries a frames table:

"frames": [ { "frame_id": 0, "frame_ts": 0.0 }, { "frame_id": 24, "frame_ts": 1.001 } ]

frames lists every sampled frame in frame_id order, including frames where nothing was found, so a gap in the results is distinguishable from a frame that was never read. frame_id is the only field an item shares with a frame record, and items is ordered by (frame_id, track_id) with untracked rows last in a frame.

The document block record

One entry of doc.page.blocks.items. Every key is optional, and a model fills the subset it can, so one type covers every document method.

Key	Type	Notes
`block_id`	`int`	The block’s position on the page, zero-based like `page_id`.
`bbox_xywh`	`[x, y, w, h]`	Normalized 0-1 box. Absent on a whole-page read.
`poly_xy`	`[[x, y], ...]`	Normalized polygon, from a quad or polygon detector.
`label`	`string`	Layout category, folded to lower `snake_case`.
`text`	`string`	Recognized text. Absent when the block was not read, for example a `picture` region.
`text_format`	`"text"`, `"markdown"`, `"html"`, `"latex"`	Format of `text`. Omitted means `markdown`.
`score`	`float`	Detector confidence 0-1.
`parent_id`	`int`	The `block_id` of the block that contains this one.
`child_ids`	`[int]`	The `block_id` of every block inside this one.
`attributes`	`object`	Metadata from a customized pipeline. Absent on the stock pipeline.

A whole-page read is one block that carries text and no geometry:

{
  "object": "doc.page.blocks",
  "items": [
    { "block_id": 0, "text": "# Annual Report 2024\n\n## Overview" }
  ]
}

A layout read is one block per region:

{
  "object": "doc.page.blocks",
  "items": [
    {
      "block_id": 0,
      "bbox_xywh": [0.08, 0.0485, 0.84, 0.0364],
      "label": "title",
      "text": "Annual Report 2024",
      "score": 0.98
    }
  ]
}

Content object tags

Tags read <domain>.<task>.<unit>: what the payload describes, the operation that produced it, and what one entry in items is. The domain is doc for page and text structure, img for something located in the image plane, and world for 3D. Two models running the same task return the same tag, so a parser keys on the tag rather than on the model it asked for. Aliases resolve first, so pp-ocrv6 and paddleocr/pp-ocrv6 share one contract.

Model	Method	`content.object`
`paddleocr/pp-ocrv6`	`ocr`	`doc.ocr.lines`
`paddleocr/pp-ocrv6`	`detect`	`doc.ocr.bboxes`
`rednote-hilab/dots.mocr`	`parse_layout`	`doc.page.blocks`
`rednote-hilab/dots.mocr`	`parse_layout_only`	`doc.layout.bboxes`
`deepseek-ai/deepseek-ocr-2`	`grounding_ocr`	`doc.ocr.lines`
`microsoft/florence-2-base-ft`	`ocr_with_region`	`doc.ocr.lines`
`microsoft/florence-2-base-ft`	`od`, `dense_region_caption`, `region_proposal`	`img.detect.bboxes`
`facebook/sam3.1`	`segment`, `segment_box`	`img.segment.masks`
`facebook/sam3.1`	`track` (video)	`vid.segment.masks`
`usyd-community/vitpose-plus-large`	`pose` (image)	`img.pose.kpts`
`usyd-community/vitpose-plus-large`	`pose` (video)	`vid.pose.kpts`
`geopavlakos/hamer`	`pose`	`world.pose.kpts`

On a document page the container tag is always doc.page.blocks, because the payload is a property of the medium. A markdown-kind method on an image has no container and therefore no tag.

Chat VLMs are not enveloped

A chat VLM reply is passed through in JSON mode too: the body is the model’s own JSON, with no data wrapper and no object tag.

{ "animal": "golden retriever", "setting": "beach at sunset" }

This is what OpenAI’s response_format={"type":"json_object"} means: the model emits the JSON.

Text and JSON modes carry the same data

A text rendering decodes into exactly the JSON-mode payload: parse a <page> body as JSON when format="json", and take it verbatim when format="markdown". One type therefore validates both renderings, and you can switch formats without a second parser. The single exception is the bare single-image form, which carries no image metadata: decoding recovers content but not image_hash, image_width, or image_height.

Methods by model

Output kind is a function of the model and the method. Default method in bold. Imgs is the maximum images per request, and Doc marks the models that accept document_url PDFs.

Model	Methods to output kind	Imgs	Doc
`qwen/qwen3.5-0.8b`	`chat` to passthrough	64	–
`qwen/qwen3.8-27b`	`chat` to passthrough	64	–
`microsoft/florence-2-base-ft`	`caption`, `detailed_caption`, `more_detailed_caption`, `ocr` to markdown · `od`, `dense_region_caption`, `region_proposal`, `ocr_with_region` to json	1	–
`paddleocr/pp-ocrv6`	`ocr`, `detect` to json · `text` to markdown	1	✓
`deepseek-ai/deepseek-ocr-2`	`markdown`, `ocr`, `free_ocr` to markdown · `grounding_ocr` to json	1	✓
`zai-org/glm-ocr`	`markdown`, `ocr` to markdown	1	✓
`rednote-hilab/dots.mocr`	`parse_layout`, `parse_layout_only` to json · `ocr`, `markdown` to markdown	1	✓
`paddlepaddle/paddleocr-vl-1.6`	`markdown`, `ocr`, `table`, `formula`, `chart` to markdown	1	✓
`baidu/unlimited-ocr`	`markdown`, `multi_page` to markdown	32	✓
`usyd-community/vitpose-plus-large`	`pose` to json	1	–
`geopavlakos/hamer`	`pose` to json	1	–
`facebook/sam3.1`	`segment`, `segment_box`, `track` to json	1	–

The kind above is the image kind, which is also the page format in text mode. In JSON mode a document page is always doc.page.blocks.

Method Parameters

method_params is an optional object passed alongside method at the top level of the request body, or via extra_body in the OpenAI Python SDK. Keys are model- and method-specific; the table below lists every key.

response = client.chat.completions.create(
    model="paddleocr/pp-ocrv6",
    messages=[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg"}}]}],
    extra_body={
        "method": "ocr",
        "method_params": {"lang": "en", "score_threshold": 0.5},
    },
)

const response = await client.chat.completions.create({
  model: "paddleocr/pp-ocrv6",
  messages: [{ role: "user", content: [{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg" } }] }],
  method: "ocr",
  method_params: { lang: "en", score_threshold: 0.5 },
});

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg \
  -m paddleocr/pp-ocrv6 --method ocr \
  --method-params '{"lang": "en", "score_threshold": 0.5}'

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "paddleocr/pp-ocrv6",
    "method": "ocr",
    "method_params": {
      "lang": "en",
      "score_threshold": 0.5
    },
    "messages": [{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg"}}]}]
  }'

Method-specific keys

Key	Model	Method(s)	Description
`lang`	`paddleocr/pp-ocrv6`	`ocr`, `detect`	Recognition language (`en`, `ch`, `japan`, …). Default `en`.
`score_threshold`	`paddleocr/pp-ocrv6`	`ocr`	Drop text regions below this recognition confidence (0.0-1.0). Default `0.5`.
`prompt`	`deepseek-ai/deepseek-ocr-2`	`grounding_ocr`	Full grounding prompt, with the phrase to locate wrapped in the model’s ref markers. See Grounding for the exact syntax.
`window_size`	`baidu/unlimited-ocr`	`multi_page`	Pages read per model call as a sliding window over the document. Default `8`.
`prompt`	`facebook/sam3.1`	`segment`, `track`	What to segment or track, in plain words (`"person"`, `"cat"`). Returned verbatim as each item’s `label`.
`bbox_xywh`	`facebook/sam3.1`	`segment_box`	Seed box as `[x, y, w, h]`, normalized 0-1, in place of a text prompt.
`mask_format`	`facebook/sam3.1`	`segment`, `segment_box`, `track`	`png` (default) or `none`. `none` drops the label map and keeps `area`.
`polygons`	`facebook/sam3.1`	`segment`, `segment_box`	`true` adds `polys_xy` outline rings to each item. Default `false`.
`body_conf`	`geopavlakos/hamer`	`pose`	Minimum confidence for the body detection that locates the hands. Default `0.5`.
`hand_conf`	`geopavlakos/hamer`	`pose`	Minimum confidence for a hand to be reported. Default `0.5`.
`rescale_factor`	`geopavlakos/hamer`	`pose`	How far the detected hand box is expanded before the mesh is fit. Default `2.0`.
`include_mesh`	`geopavlakos/hamer`	`pose`	`true` returns the full MANO mesh beside the keypoints. Default `false`.

Where video sampling lives depends on the model. video_fps at the top level of the request body is the decoder cadence for a model that reads a video_url natively, such as usyd-community/vitpose-plus-large. facebook/sam3.1 track reads its own sampling out of method_params instead; see the model page.

Models

Per-model dropdowns with request and response examples for every method.

Multimodal Inputs

Content part types and document input limits.

Document OCR

End-to-end recipe from model selection to response parsing.

Chat Completions

Full request and response schema.

Get Started

OpenAI-Compatible API

TypeSafe-Compatible API

Guides

API Reference

Tools

Platform

Methods & Response Format

The `method` field

The `response_format` field

The `precision` field

Text mode

Document input

Single image input

One input per structured request

Chat VLMs pass through

JSON mode

`content` per medium

The item record

Segmentation masks are a label map

Video payloads

The document block record

Content object tags

Chat VLMs are not enveloped

Text and JSON modes carry the same data

Methods by model

Method Parameters

Method-specific keys

Models

Multimodal Inputs

Document OCR

Chat Completions

​The method field

​The response_format field

​The precision field

​Text mode

​Document input

​Single image input

​One input per structured request

​Chat VLMs pass through

​JSON mode

​content per medium

​The item record

​Segmentation masks are a label map

​Video payloads

​The document block record

​Content object tags

​Chat VLMs are not enveloped

​Text and JSON modes carry the same data

​Methods by model

​Method Parameters

​Method-specific keys

​Related

Models

Multimodal Inputs

Document OCR

Chat Completions

The `method` field

The `response_format` field

The `precision` field

Text mode

Document input

Single image input

One input per structured request

Chat VLMs pass through

JSON mode

`content` per medium

The item record

Segmentation masks are a label map

Video payloads

The document block record

Content object tags

Chat VLMs are not enveloped

Text and JSON modes carry the same data

Methods by model

Method Parameters

Method-specific keys

Related