MCP Server

The VLM Run Gateway exposes a Model Context Protocol (MCP) server so any MCP-aware agent framework (Pydantic AI, LangChain, Mastra, OpenAI Agents SDK, Claude Code) can read documents, images, audio, and video through the same gateway pipeline as the REST API. It is a FastMCP server served over Streamable HTTP, and it re-enters the same ingress as chat/completions, so authentication, billing, and metering behave identically.

Property	Value
Endpoint	`POST https://gateway.vlm.run/mcp`
Transport	Streamable HTTP (FastMCP)
Auth	`Authorization: Bearer <VLMRUN_API_KEY>`, or omit the header for anonymous access
Tools	`read_document`, `read_image`, `read_audio`, `read_video`, `list_models`

Authentication follows the same tiers as the REST API. See Authentication and Rate Limits.

Quickstart

Start here to see the server answer before you wire an agent to it.

Add the server

claude mcp add --transport http vlmrun https://gateway.vlm.run/mcp \
  --header "Authorization: Bearer $VLMRUN_API_KEY"

Any MCP client works the same way: one URL, one bearer header. To click through the tools instead, run npx @modelcontextprotocol/inspector, set the transport to Streamable HTTP, and point it at the same URL.

Ask for a document

Read https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf
and tell me the date of the earliest event reported.

The agent picks read_document on its own. You write no extraction schema and no glue code.

Read what came back

<document file_name="tsla-8k.pdf" mimetype="application/pdf" num_pages="5" dpi="72">
<page page_index="0" format="markdown" page_width="612" page_height="792">
UNITED STATES
SECURITIES AND EXCHANGE COMMISSION
...
Date of report (Date of earliest event reported): October 2, 2024
</page>
</document>

Every read_* tool answers with text like this, plus the typed ReadResult that carries the model that ran and its cost.

Connect an agent

Every MCP-aware client connects the same way: point it at https://gateway.vlm.run/mcp with a bearer token, and the four read tools plus list_models show up alongside the agent’s other tools. Set your key once:

export VLMRUN_API_KEY="your-api-key"

Pydantic AI
LangChain
Mastra
OpenAI Agents SDK
Claude Code

import os

from pydantic_ai import Agent
from pydantic_ai.capabilities import MCP

agent = Agent(
    "anthropic:claude-sonnet-5",
    capabilities=[
        MCP(
            "https://gateway.vlm.run/mcp",
            headers={"Authorization": f"Bearer {os.environ['VLMRUN_API_KEY']}"},
        )
    ],
)

result = agent.run_sync(
    "Read https://example.com/invoice.pdf and tell me the total amount due."
)
print(result.output)

native=False (the default) runs the MCP client locally: Pydantic AI connects to the gateway and calls the tools. native=True advertises the URL to the model provider (OpenAI Responses, Anthropic, xAI) so the provider connects to the server directly.

import os

from langchain_mcp_adapters.client import MultiServerMCPClient
from langgraph.prebuilt import create_react_agent

client = MultiServerMCPClient(
    {
        "vlmrun": {
            "url": "https://gateway.vlm.run/mcp",
            "transport": "streamable_http",
            "headers": {"Authorization": f"Bearer {os.environ['VLMRUN_API_KEY']}"},
        }
    }
)

tools = await client.get_tools()
agent = create_react_agent("anthropic:claude-sonnet-5", tools)

Install the adapter with pip install langchain-mcp-adapters langgraph.

import { MCPClient } from "@mastra/mcp";

const mcp = new MCPClient({
  servers: {
    vlmrun: {
      url: new URL("https://gateway.vlm.run/mcp"),
      requestInit: {
        headers: { Authorization: `Bearer ${process.env.VLMRUN_API_KEY}` },
      },
    },
  },
});

const tools = await mcp.getTools();

Pass tools to any Mastra Agent. Install with npm install @mastra/mcp.

import os

from agents import Agent, Runner
from agents.mcp import MCPServerStreamableHttp

async with MCPServerStreamableHttp(
    params={
        "url": "https://gateway.vlm.run/mcp",
        "headers": {"Authorization": f"Bearer {os.environ['VLMRUN_API_KEY']}"},
    }
) as server:
    agent = Agent(
        name="Assistant",
        mcp_servers=[server],
    )
    result = await Runner.run(agent, "Summarize the attached PDF.")
    print(result.final_output)

claude mcp add --transport http vlmrun https://gateway.vlm.run/mcp \
  --header "Authorization: Bearer $VLMRUN_API_KEY"

The read_document, read_image, read_audio, read_video, and list_models tools are now available in your Claude Code session.

Once connected, the agent calls the read tools on its own: it picks read_document, receives the Markdown OCR plus the ReadResult payload (including the model that ran and its cost), and reasons over the text to answer, no extraction schema or glue code required. The initialize handshake returns server instructions that name each tool and its use, and most clients inject them as the system message. When your client does that, keep your own steering in the user turn, because a second system message breaks strict OpenAI-compatible endpoints.

Tools

The server exposes one read tool per modality plus a model-discovery tool. Every read_* tool takes a url, returns the extracted text, and accepts a model plus per-modality knobs.

Tool	Reads	Key arguments
`read_document`	PDFs and images	`url`, `model`, `dpi`, `pages`, `method`, `method_params`, `json_mode`
`read_image`	One still image	`url`, `model`, `prompt`, `json_mode`
`read_audio`	Speech audio	`url`, `model`, `language`, `json_mode`
`read_video`	Video	`url`, `model`, `prompt`, `fps`, `max_frames`, `encoder`, `encoder_params`, `json_mode`
`list_models`	The live model catalog	`modality`

read_document and read_image split by intent, not by file type. Both accept a JPEG or a PNG. Use read_document for the text printed on a page, which it OCRs into Markdown with headings, lists, and tables preserved. Use read_image for what a picture depicts, such as a photo, a chart, or a screenshot. url is always an http(s) URL, a data: URI, or a bare base64 string. It is never a local filesystem path. Host the file or inline it as base64 before calling a read tool. Every tool takes json_mode. Left false, a tool returns the native form as a plain string (Markdown for documents, text for audio and video); true returns the structured envelope, parsed under data.

Call list_models to pick a model the gateway actually serves. A model id passed to the wrong tool, or one that is not currently served, is rejected.

Pick a method by what you need

Every read_document method returns a different thing from the same page. list_models tags each one with a capability, so you can choose without reading a model card. Ask for the capability, then use the method that carries it.

Capability	What you get	Needs `json_mode`
`document_markdown`	Text with the structure kept: headings, lists, tables.	No
`text_extraction`	The text, as a plain string.	No
`layout_regions`	Labelled areas, such as `title`, `table`, `figure`.	Yes
`reading_order`	The regions in the order a person reads them.	Yes
`text_citations`	Each piece of text with the box it came from.	Yes
`text_highlighting`	Boxes to draw over the page.	Yes

For example, rednote-hilab/dots.mocr carries document_markdown on markdown and text_citations on parse_layout, and paddleocr/pp-ocrv6 carries text_citations on ocr and text_extraction on text.

`read_document`

OCR a PDF or image into clean Markdown (or structured JSON). Use it for invoices, forms, reports, slide decks, screenshots, and scanned pages.

Argument	Type	Default	Description
`url`	`string`	required	The PDF or image.
`model`	enum	`rednote-hilab/dots.mocr`	OCR model to run. One of `rednote-hilab/dots.mocr`, `deepseek-ai/deepseek-ocr-2`, `zai-org/glm-ocr`, `paddleocr/pp-ocrv6`.
`dpi`	`integer`	`72`	Rasterization DPI (72-400) per PDF page before OCR. Higher is sharper on small text, but slower.
`method`	`string`	model default	Backend method run on each page: `markdown` on the generative OCR models, `ocr` on `paddleocr/pp-ocrv6`. A method the model does not advertise is rejected before the call. See Pick a method.
`method_params`	`object`	unset	Keyword arguments for `method`, forwarded as-is.
`pages`	`list[int \| tuple[int,int]]`	every page (max 128)	0-indexed page selection. Each entry in the list is a page index or a `[start, stop]` half-open range (`start` inclusive, `stop` exclusive); negative indices count from the end. Only the selected pages are rasterized and read, and they are re-numbered from 0 in the response. Indices past the end are ignored.
`json_mode`	`boolean`	`false`	`false` returns the `<document>` / `<page>` text blocks; `true` returns the `{"model", "method", "data"}` response object of per-page records.

Page Selection

[0, 2, 4] reads pages 0, 2, and 4
[[0, 3]] reads pages 0-2
[[0, 3], [5, 7], -1] reads pages 0, 1, 2, 5, 6 and the last page

One call reads at most 128 pages. For a longer document, call the tool once per range and combine the results. The error names the exact ranges to use.

JSON Mode

json_mode: false
json_mode: true

Left unset, json_mode returns the same text blocks as the REST API: one <document> block per PDF, wrapping one <page> block per page, in reading order.

<document file_name="invoice.pdf" mimetype="application/pdf" num_pages="2" dpi="72">
<page page_index="0" format="markdown" page_width="612" page_height="792">
# ACME Corp, Invoice #4471

| Item | Qty | Price |
| ---- | --- | ------ |
| Widget | 3 | $12.00 |
</page>
<page page_index="1" format="markdown" page_width="612" page_height="792">
## Terms

Net 30. Thank you for your business.
</page>
</document>

A page that failed OCR is self-closing, with status="error" and no body. A single image returns the Markdown alone, with no wrapper.

json_mode: true returns the same pages as records in data, which is how you get boxes out of a region method:

[
  {
    "object": "document",
    "file_name": "invoice.pdf",
    "mimetype": "application/pdf",
    "num_pages": 2,
    "dpi": 72,
    "pages": [
      {
        "object": "document.page",
        "page_index": 0,
        "page_width": 612,
        "page_height": 792,
        "content": {
          "object": "document.page.blocks",
          "items": [
            {
              "index": 0,
              "bbox_xywh": [0.3987, 0.0896, 0.1993, 0.0164],
              "text": "ACME Corp",
              "score": 0.9998
            }
          ]
        }
      }
    ]
  }
]

Both shapes are documented under Text mode and JSON mode.

`read_image`

Send one still image to a vision-language model and get its answer back. Use it for photos, charts, diagrams, product shots, and UI screenshots, where the value is in what the picture depicts.

Argument	Type	Default	Description
`url`	`string`	required	The image.
`model`	enum	`qwen/qwen3.5-0.8b`	Vision-language model that looks at the image.
`prompt`	`string`	”Describe this image in detail.”	The question or the instruction, for example “What is the total on this receipt?”.
`json_mode`	`boolean`	`false`	`false` returns the answer as a string; `true` asks the model itself for JSON, which a chat VLM returns unenveloped. See Chat VLMs are not enveloped.

The read_image menu carries no OCR model on purpose. An image whose value is its printed text belongs to read_document, which accepts an image URL and returns the Markdown alone, with no wrapper, for a single image.

`read_audio`

Transcribe a speech audio file (wav, mp3, m4a, flac, ogg, …) into text.

Argument	Type	Default	Description
`url`	`string`	required	The audio file.
`model`	enum	`nvidia/parakeet-tdt-0.6b-v3`	Speech-to-text model to run.
`language`	`string`	auto-detect	ISO-639-1 language hint (`en`, `es`, …). Supply it when you already know the spoken language, to skip detection and improve accuracy.
`json_mode`	`boolean`	`false`	`false` returns the plain transcript; `true` returns the full transcription object (text plus metadata).

`read_video`

Describe or transcribe a video. Frames are sampled and encoded into image(s) so image-capable models can read them, then a vision-language model produces the text.

Argument	Type	Default	Description
`url`	`string`	required	The video file.
`model`	enum	`qwen/qwen3.5-0.8b`	Vision-language model that watches the sampled frames.
`prompt`	`string`	”Transcribe and describe the content of this video in detail.”	What to produce from the video. Steer it with, for example, “List the on-screen text” or “Summarize what happens in 3 bullets”.
`fps`	`number`	uniform sampling bounded by `max_frames`	Frames sampled per second of video. Higher captures more motion at higher cost.
`max_frames`	`integer`	model default	Upper bound on the number of frames sampled.
`encoder`	`mosaic` \| `frames` \| `keyframes`	`mosaic`	How the video becomes images. `mosaic` tiles sampled keyframes into grid images and is robust for every model; `frames` sends sampled frames as separate images; `keyframes` uses the video’s I-frames.
`encoder_params`	`object`	encoder defaults	Parameters for `encoder`, e.g. `{"tile_cols": 4, "tile_rows": 4, "num_frames": 128}` for `mosaic`, or `{"fps": 1.0}` for `frames`.
`json_mode`	`boolean`	`false`	`false` returns the description as a string; `true` asks the model itself for JSON, which a chat VLM returns unenveloped.

On the REST API, encoder and encoder_params map to video_encoder and video_encoder_params.

`list_models`

List the models the gateway currently serves and which read tool each one fits. Call it before passing a non-default model.

Argument	Type	Default	Description
`modality`	`document` \| `image` \| `audio` \| `video`	all	Restrict the listing to one tool’s models.

Each entry is {id, modality, tool, tools, aliases, methods, default_method, method_details}. Pass id verbatim as the model argument to the matching tool. tools lists every read tool the model fits, because one model can serve several: a Qwen vision model fits read_image and read_video. modality and tool name the primary one, and a modality filter matches on any fitting tool. method_details is the per-method answer to “what do I get back”: one record per method, with its capabilities, default: true on the model’s default, and requires_json_mode: true on every method that carries more than plain text_extraction.

{
  "id": "paddleocr/pp-ocrv6",
  "modality": "document",
  "tool": "read_document",
  "tools": ["read_document", "read_image"],
  "aliases": ["pp-ocrv6"],
  "methods": ["ocr", "detect", "text"],
  "default_method": "ocr",
  "method_details": [
    {
      "name": "ocr",
      "capabilities": ["text_extraction", "text_citations", "text_highlighting"],
      "default": true,
      "requires_json_mode": true
    },
    { "name": "detect", "capabilities": ["text_highlighting"], "requires_json_mode": true },
    { "name": "text", "capabilities": ["text_extraction"] }
  ]
}

Return shape (`ReadResult`)

Each read_* tool returns the clean text as the tool’s content and a typed structured payload, so an agent gets usable text directly while still being able to read which model actually ran and what it cost:

Field	Description
`text`	The extracted text (Markdown, transcript, or description). Empty when `json_mode` is `true`.
`data`	The structured records, when `json_mode` is `true`. Null otherwise.
`model`	The model the gateway actually ran.
`cost`	The metered cost of the call in US dollars, when the gateway reports it.

Very large output is trimmed before it reaches the agent so one document cannot flood the context: past roughly 60,000 characters the text is cut and a [truncated: …] note is appended telling the agent to narrow the request (for example, fewer pages) or process in sections.

Errors

An error names the next step, so the agent recovers on its own. A method the chosen model does not advertise is refused before the call, and the message lists the methods it does advertise:

model 'paddleocr/pp-ocrv6' does not support method 'markdown'. It advertises:
detect, ocr, text. Retry with one of those, or omit `method` for the default.
`list_models` reports the methods of every served model.

Symptom	Cause	Fix
`400 Bad Request: Missing session ID`	The client dropped the `mcp-session-id` header from `initialize`.	Send the header the handshake returned on every later request.
`Session not found`	The session expired, or the request reached a different gateway replica.	Run `initialize` again and use the new session id.
`Failed to fetch document URL: HTTP Error 404`	The gateway cannot reach `url`.	Host the file publicly, or pass it as a `data:` URI or base64.
`does not support method '…'`	The method belongs to another model.	Use a method the message names, or omit `method`.
Boxes or labels are missing from the answer	The payload only arrives under `json_mode`.	Set `json_mode: true`, then read `data`.
`[truncated: …]` at the end of the text	The output passed the inline cap.	Narrow `pages`, or read the document in sections.

Call it without a framework

Three requests reach a tool over plain HTTP. initialize returns the mcp-session-id header, notifications/initialized opens the session, and every later request carries the header. Send Accept: application/json, text/event-stream, because the reply is a Server-Sent Events stream.

MCP=https://gateway.vlm.run/mcp
HDRS=(-H "Authorization: Bearer $VLMRUN_API_KEY"
      -H "Content-Type: application/json"
      -H "Accept: application/json, text/event-stream")

curl -sD /tmp/h -o /dev/null -X POST $MCP "${HDRS[@]}" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{
       "protocolVersion":"2025-06-18","capabilities":{},
       "clientInfo":{"name":"curl","version":"1"}}}'

SESSION=$(grep -i '^mcp-session-id:' /tmp/h | tr -d '\r' | cut -d' ' -f2)

curl -s -o /dev/null -X POST $MCP "${HDRS[@]}" -H "mcp-session-id: $SESSION" \
  -d '{"jsonrpc":"2.0","method":"notifications/initialized"}'

curl -sN -X POST $MCP "${HDRS[@]}" -H "mcp-session-id: $SESSION" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{
       "name":"list_models","arguments":{"modality":"document"}}}'

Chat Completions API

The REST surface behind the same gateway pipeline.

List models

Query the live model catalog and capabilities.

Authentication

Bearer tokens, tiers, and anonymous access.

Multimodal Inputs

Content part types, document limits, and video knobs.

Overview

Guides

API Reference

Misc

Quickstart

Connect an agent

Tools

Pick a method by what you need

`read_document`

Page Selection

JSON Mode

`read_image`

`read_audio`

`read_video`

`list_models`

Return shape (`ReadResult`)

Errors

Call it without a framework

Chat Completions API

List models

Authentication

Multimodal Inputs

​Quickstart

​Connect an agent

​Tools

​Pick a method by what you need

​read_document

​Page Selection

​JSON Mode

​read_image

​read_audio

​read_video

​list_models

​Return shape (ReadResult)

​Errors

​Call it without a framework

​Related

Chat Completions API

List models

Authentication

Multimodal Inputs

Quickstart

Connect an agent

Tools

Pick a method by what you need

`read_document`

Page Selection

JSON Mode

`read_image`

`read_audio`

`read_video`

`list_models`

Return shape (`ReadResult`)

Errors

Call it without a framework

Related