chat - VLM Run

vlmrun gw chat runs an OCR or VLM model over one or more documents, images, or videos. Inputs can be local paths, remote URLs, or a mix of both. See gateway for shared options and configuration.

# Parse a hosted document (PDF -> text/markdown)
vlmrun gw chat https://example.com/report.pdf -m zai-org/glm-ocr

# Parse a local document
vlmrun gw chat document.pdf -m zai-org/glm-ocr

# Several inputs at once, mixing local paths and URLs
vlmrun gw chat doc1.pdf https://example.com/doc2.pdf -m paddleocr/pp-ocrv6

# Prompt a model that supports text input
vlmrun gw chat image.jpg -p "describe this image" -m qwen/qwen3.5-0.8b

# Text-only, on a chat model that accepts it
vlmrun gw chat -p "What is OCR?" -m qwen/qwen3.5-0.8b

A run prints the resolved inputs, then the reply under a Response rule, with the metering signals on the closing rule:

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf \
  -m paddleocr/pp-ocrv6 \
  -e document_dpi=72

Processing 1 input(s) (paddleocr/pp-ocrv6)
https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf

Response
<document file_name="tsla-8k.pdf" file_hash="sha256:1b7f04c3…" file_bytes="48213" mimetype="application/pdf" npages="5" dpi="72">
<page id="0" format="json" width="612" height="792">
{"object": "doc.page.blocks", "items": [{"block_id": 0, "bbox_xywh": [0.3987, 0.0896, 0.1993, 0.0164], "poly_xy": [[0.3987, 0.0896], ...], "text": "UNITED STATES", "score": 0.9998}, ...]}
</page>
... one <page> block per page ...
</document>

paddleocr/pp-ocrv6 · P:1280 / C:1171 / T:2451 toks · 1060 toks/s · 5 pages · 4.53 pages/s · 1s · $0.000247

bbox_xywh and poly_xy are both normalized to the page, not pixels. Methods that return prose rather than regions (markdown on zai-org/glm-ocr, say) put the text straight in the <page> body with format="markdown". The footer reads model, prompt / completion / total tokens, throughput, pages, pages per second, latency, and cost. Segments drop out when the gateway does not report them: tokens and throughput without usage, pages for non-paginated replies, cost when none is returned. Token counts, pages and cost are reproducible for a given document and DPI; throughput and latency vary. See Pricing and Metering for the same signals.

Methods

Each model exposes one or more methods (ocr, detect, markdown) with a default; vlmrun gw models <model> lists them. --method and --method-params map to the gateway’s top-level method and method_params fields.

vlmrun gw chat img.jpg -m pp-ocrv6 --method detect

vlmrun gw chat img.jpg -m pp-ocrv6 --method ocr \
    --method-params '{"lang": "en", "score_threshold": 0.5}'

# Document markdown with DeepSeek OCR 2
vlmrun gw chat page.pdf -m deepseek-ocr-2 --method markdown

Video input

Video files and video URLs are sent as video_url content parts. Control frame sampling with the gateway’s video_fps and video_max_frames fields, passed through -e:

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4 \
  -p "Summarize what happens in this video." \
  -m qwen/qwen3.5-0.8b \
  -e video_fps=1.0 \
  -e video_max_frames=8

See Video Inputs for the full set of sampling knobs.

Extra completion kwargs

Forward extra chat.completions.create() kwargs as repeatable key=value pairs (values are JSON-parsed). Known keys go through as real kwargs; unknown ones route via extra_body to become top-level gateway request fields.

vlmrun gw chat document.pdf -m zai-org/glm-ocr -e temperature=0 -e max_tokens=4096

Streaming

gw chat streams by default: tokens print live, one chunk per page for multi-page documents. -ns/--no-stream waits for the full reply and renders it in a bordered panel instead.

JSON output vs JSON mode

Two different flags contain the word “json”, and they act on two different layers:

Flag	Layer	Effect
`-j, --json`	The CLI’s own output	Replaces the human-readable output with a JSON envelope on stdout
`--json-mode`	The gateway request	Sends `response_format: {"type": "json_object"}`, asking the model to return one JSON object instead of the `<document>` envelope

-j replaces that output with an envelope of model, content, latency_s, and usage, plus pages and pages_per_sec when the reply is paginated OCR output:

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf \
  -m paddleocr/pp-ocrv6 -e document_dpi=72 -j

{
  "model": "paddleocr/pp-ocrv6",
  "content": "<document file_name=\"tsla-8k.pdf\" ... </document>",
  "latency_s": 0.9522805213928223,
  "usage": { "prompt_tokens": 1280, "completion_tokens": 1171, "total_tokens": 2451, "prompt_tokens_details": {...}, "cost": 0.000247 },
  "pages": 5,
  "pages_per_sec": 5.25
}

usage is the gateway’s object passed through untouched: standard OpenAI breakdown fields (often null for OCR models) plus the VLM Run cost extension. Read usage.cost rather than recomputing from tokens. Adding --json-mode gives JSON mode plus machine-readable metering, but the result is double-encoded: content is a JSON string holding a JSON document, so it needs a second parse (jq -r .content | jq .). --response-format is the general form: text, json_object (alias json), or a full object like '{"type":"json_schema", ...}'. --json-mode is exactly --response-format json_object, and the two are mutually exclusive. json_schema is a per-model capability (capabilities.supports_json_schema): chat and frontier models constrain generation to it, while OCR and detection models return a 400 capability_violation. Use json_object there.

Options

Option	Short	Description
`INPUTS...`		Input file path(s) or http(s) URL(s): image, document, or video. Repeatable
`--model`	`-m`	Gateway model id, full `<org>/<name>` or alias (required)
`--prompt`	`-p`	Text prompt (only for models that support text input)
`--method`	`-M`	Model method, e.g. `ocr`, `detect`, `markdown`. Defaults to the model’s `default_method`
`--method-params`		JSON object of method arguments, e.g. `'{"lang": "en"}'`
`--json-mode`		Enable JSON mode (`response_format json_object`). Mutually exclusive with `--response-format`
`--response-format`		Constrain the output: `text`, `json_object`, or a JSON object like `'{"type":"json_schema",...}'`
`--extra`	`-e`	Extra `create()` kwarg as `key=value` (repeatable)
`--no-stream`	`-ns`	Disable streaming
`--json`	`-j`	Output raw JSON
`--timeout`		Request timeout in seconds

​Methods

​Video input

​Extra completion kwargs

​Streaming

​JSON output vs JSON mode

​Options