Models

The VLM Run Gateway serves open-weight vision and document models through a single OpenAI-compatible API. Each model declares the input types it accepts (text, image_url, video_url, document_url) and the operations it supports via the method field. Available means that the model is in the served catalog. An individual request can still be aborted with a 504 inference timeout. Use the standalone reference for any available model to see its accepted inputs, per-method output shapes, request, and response examples. The full card for each model, with price, context, and examples, is on the website at vlm.run/gateway/models.

​

Output at a glance

Every model returns one of two payload kinds, json or markdown, and response_format decides how that payload is serialized. This holds across every model and method on this page.

• Text mode (response_format omitted, or {"type":"text"}): one block per input medium. A document is a <document> / <page> block; a single image is the bare payload.
• JSON mode ({"type":"json_object"}): one JSON object, {"model", "method", "data"}, with one self-describing entry per input medium.

The chat VLMs are the exemption: they pass their own output through verbatim in both modes. Every other model shares one envelope, whatever the method. See Methods & Response Format for the full contract, the region schema, and the precision knob.

​

Chat and Visual Question Answering (VQA)

VQA models accept text plus up to 64 images, or one video, in the same message. They return the model’s reply verbatim, with no envelope.

​

Image Understanding

Single-image multi-task models that caption, detect, and OCR from one image_url. They do not accept document_url or video.

​

Document and Image OCR

These models accept a PDF via document_url or an image via image_url. The Gateway rasterizes each PDF page and runs per-page inference, returning one entry per input document. For images, the Gateway runs inference on the entire image. For consistency, structured output is not streamed token-by-token. For documents, OCR methods usually return the same <document> / <page> blocks in text mode. format="markdown" on the page marks a Markdown body (markdown, text, free_ocr, and the ocr of dots.mocr and deepseek-ocr-2), and format="json" marks a region payload (detect, grounding_ocr, parse_layout, parse_layout_only). baidu/unlimited-ocr shares that envelope and writes its own layout tags inside each page’s text, so a page reads title [230, 89, 768, 129]UNITED STATES…. See unlimited-ocr for that markup and Text mode for the shared contract. PaddleOCR-VL 1.6 is the only model in the catalog that also reads a page’s structured content: its table method returns OTSL structure tokens, formula returns LaTeX, and chart returns a described series as a Markdown table, in the same envelope as ocr.

​

Pose Estimation

Keypoint models over one image_url, and for ViTPose one video_url. ViTPose returns 17 normalized COCO keypoints per person in the image plane (img.pose.kpts). HaMeR finds hands instead, and returns 21 MANO keypoints per hand in both the image plane and 3D (world.pose.kpts).

​

Segmentation and Tracking

Promptable segmentation over one image_url, and multi-object tracking through one video_url. A text prompt names what to segment. Each instance returns a normalized bounding box, a score and its area, and every instance’s pixels arrive together in one label map.

​

Embeddings and Transcription

These models appear on GET /v1/openai/models with a task field other than chat. They use separate OpenAI-compatible endpoints, not chat completions. See Embeddings and Audio Transcriptions for full request schemas.

​

Model Aliases

Many models accept multiple request IDs:

1. Preferred: lowercase <org>/<slug> (listed on /models)
2. Short: slug only (e.g. pp-ocrv6, glm-ocr, dots.mocr, deepseek-ocr-2, paddleocr-vl-1.6, unlimited-ocr)
3. Hugging Face: upstream repo id (e.g. zai-org/GLM-OCR)

​

Frontier models

The Gateway also routes to frontier vision AI models via upstream providers, so you can use the same API key and baseURL to access them as well. These models are passthrough VLMs supporting the same methods and method_params reported by their providers. Prices are the provider’s listed prices in USD per 1M tokens. Which vendor serves a given model can change, so read provider from the catalog at call time rather than pinning it. See all 8 frontier models →

​

Typed decisions

google/diffusiongemma-26b-a4b-it is served on two surfaces. On chat completions it behaves like the other chat models above. On the TypeSafe-compatible route it answers typed questions with a calibrated probability per label, read rather than generated, so an answer can never be off-schema. qwen/qwen3.8-27b and qwen/qwen3.5-0.8b answer the same decisions on that route too, as does google/gemma-4-26b-a4b-it. There is no catch-all alias, so a request names the model it wants. See System One and TypeSafe SDK Compatibility.

​

Next steps