MCP Server

The VLM Run Gateway exposes a Model Context Protocol (MCP) server, so any MCP-aware agent (Pydantic AI, LangChain, Mastra, OpenAI Agents SDK, Claude Code) can read documents, images, audio, and video through the same pipeline as the REST API. It is a FastMCP server over Streamable HTTP. Calls re-enter the same ingress as chat/completions, so authentication, billing, and metering behave identically.

Property	Value
Endpoint	`POST https://gateway.vlm.run/mcp`
Transport	Streamable HTTP (FastMCP)
Auth	`Authorization: Bearer <VLMRUN_API_KEY>`, or omit the header for anonymous access
Tools	`read_document`, `read_image`, `read_audio`, `read_video`, `list_models`

Authentication follows the same tiers as the REST API. See Authentication and Rate Limits.

What is MCP?

MCP is an open standard for connecting AI applications to external systems: data sources, tools, and workflows. A server publishes a typed catalog of its tools and their arguments, and can declare the shape of what each one returns, so a client discovers them at run time instead of carrying a hand-written integration per capability. Clients as different as Claude, ChatGPT, VS Code, and Cursor all speak it, so one server reaches all of them. Here the catalog is five tools. Point any MCP client at https://gateway.vlm.run/mcp and those five sit alongside the agent’s own. The model hands one of them a URL, gets text back, and reasons over it. You write no extraction schema and no glue code.

When to use MCP

Use the MCP server when you want an agent to read a document, an image, or a video, then answer questions about it or act on the text it returns. It is the recommended way to reach the Gateway: the agent discovers the read tools, picks the one a task calls for, and invokes it with a URL, so nothing in your code has to know which capability a given file needs. It fits wherever an agent is already in the loop:

Answering questions about a scanned invoice or contract, where read_document returns Markdown with the tables intact and pages narrows a long file to the part that matters.
Reading a chart, a diagram, or a UI screenshot with read_image, where the answer is what the picture shows rather than the text printed on it.
Summarizing a screen recording, or pulling its on-screen text, with read_video.
Checking list_models first, so the agent runs a model the Gateway serves today instead of one pinned in your code.

The tools appear natively inside any MCP-aware framework, so the same server serves an exploratory chat session and a scripted agent run alike.

Quickstart

Start here to inspect the server’s response before you wire an agent to it.

Add the server

claude mcp add --transport http vlmrun https://gateway.vlm.run/mcp \
  --header "Authorization: Bearer $VLMRUN_API_KEY"

To click through the tools instead of driving them from an agent, run npx @modelcontextprotocol/inspector, set the transport to Streamable HTTP, and point it at the same URL.

Ask the agent to parse a document

Read https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf
and tell me the date of the earliest event reported.

The agent picks read_document on its own.

Inspect the response

<document file_name="tsla-8k.pdf" mimetype="application/pdf" num_pages="5" dpi="72">
<page page_index="0" format="markdown" page_width="612" page_height="792">
UNITED STATES
SECURITIES AND EXCHANGE COMMISSION
...
Date of report (Date of earliest event reported): October 2, 2024
</page>
</document>

Every read_* tool answers with text like this, plus the typed ReadResult that carries the model that ran and its cost.

Connect an agent

Every MCP-aware client connects the same way: one URL, one bearer header. Set your key once:

export VLMRUN_API_KEY="your-api-key"

{
  "mcpServers": {
    "gateway": {
      "type": "http",
      "url": "https://gateway.vlm.run/mcp",
      "headers": {
        "Authorization": "Bearer <VLMRUN_API_KEY>"
      }
    }
  }
}

import os

from pydantic_ai import Agent
from pydantic_ai.capabilities import MCP

agent = Agent(
    "anthropic:claude-sonnet-5",
    capabilities=[
        MCP(
            "https://gateway.vlm.run/mcp",
            headers={"Authorization": f"Bearer {os.environ['VLMRUN_API_KEY']}"},
        )
    ],
)

result = agent.run_sync(
    "Read https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/sample-invoice.pdf and tell me the total amount due."
)
print(result.output)

native=False (the default) runs the MCP client locally: Pydantic AI connects to the gateway and calls the tools. native=True advertises the URL to the model provider (OpenAI Responses, Anthropic, xAI) so the provider connects to the server directly.

import os

from langchain_mcp_adapters.client import MultiServerMCPClient
from langgraph.prebuilt import create_react_agent

client = MultiServerMCPClient(
    {
        "vlmrun": {
            "url": "https://gateway.vlm.run/mcp",
            "transport": "streamable_http",
            "headers": {"Authorization": f"Bearer {os.environ['VLMRUN_API_KEY']}"},
        }
    }
)

tools = await client.get_tools()
agent = create_react_agent("anthropic:claude-sonnet-5", tools)

Install the adapter with pip install langchain-mcp-adapters langgraph.

import { MCPClient } from "@mastra/mcp";

const mcp = new MCPClient({
  servers: {
    vlmrun: {
      url: new URL("https://gateway.vlm.run/mcp"),
      requestInit: {
        headers: { Authorization: `Bearer ${process.env.VLMRUN_API_KEY}` },
      },
    },
  },
});

const tools = await mcp.getTools();

Pass tools to any Mastra Agent. Install with npm install @mastra/mcp.

import os

from agents import Agent, Runner
from agents.mcp import MCPServerStreamableHttp

async with MCPServerStreamableHttp(
    params={
        "url": "https://gateway.vlm.run/mcp",
        "headers": {"Authorization": f"Bearer {os.environ['VLMRUN_API_KEY']}"},
    }
) as server:
    agent = Agent(
        name="Assistant",
        mcp_servers=[server],
    )
    result = await Runner.run(agent, "Summarize the attached PDF.")
    print(result.final_output)

claude mcp add --transport http vlmrun https://gateway.vlm.run/mcp \
  --header "Authorization: Bearer $VLMRUN_API_KEY"

The read_document, read_image, read_audio, read_video, and list_models tools are now available in your Claude Code session.

The initialize handshake returns server instructions that name each tool and its use, and most clients inject them as the system message. When your client does that, keep your own steering in the user turn, because a second system message breaks strict OpenAI-compatible endpoints.

Tools

One read tool per modality, plus a model-discovery tool. Every read_* tool takes a url, returns the extracted text, and carries a model argument plus per-modality knobs.

Tool	Reads	Key arguments
`read_document`	PDFs and images	`url`, `model`, `dpi`, `pages`, `method`, `method_params`, `json_mode`
`read_image`	One still image	`url`, `model`, `prompt`, `json_mode`
`read_audio`	Speech audio	`url`, `model`, `language`, `json_mode`
`read_video`	Video	`url`, `model`, `prompt`, `fps`, `max_frames`, `encoder`, `encoder_params`, `json_mode`
`list_models`	The live model catalog	`modality`

read_document and read_image split by intent, not by file type. Both accept a JPEG or a PNG. Use read_document for the text printed on a page, which it OCRs into Markdown with headings, lists, and tables preserved. Use read_image for what a picture depicts, such as a photo, a chart, or a screenshot. url is always an http(s) URL, a data: URI, or a bare base64 string. It is never a local filesystem path. Host the file or inline it as base64 before calling a read tool. Every read_* tool takes json_mode. Left false, it returns the native form as a plain string (Markdown for documents, text for audio and video); true returns the structured envelope, parsed under data.

Only read_document offers a choice of model today. read_image, read_audio, and read_video each pin a single model, so their model argument accepts one value. Call list_models to see what the gateway serves: a model id passed to the wrong tool, or one that is not currently served, is rejected.

Pick a method by what you need

Every read_document method returns a different thing from the same page. list_models tags each one with a capability, so you can choose without reading a model card. Ask for the capability, then use the method that carries it.

Capability	What you get	Needs `json_mode`
`document_markdown`	Text with the structure kept: headings, lists, tables.	No
`text_extraction`	The text, as a plain string.	No
`layout_regions`	Labelled areas, such as `title`, `table`, `figure`.	Yes
`reading_order`	The regions in the order a person reads them.	Yes
`text_citations`	Each piece of text with the box it came from.	Yes
`text_highlighting`	Boxes to draw over the page.	Yes

For example, rednote-hilab/dots.mocr carries document_markdown on markdown and text_citations on parse_layout, and paddleocr/pp-ocrv6 carries text_citations on ocr and text_extraction on text.

`read_document`

OCR a PDF or image into clean Markdown (or structured JSON). Use it for invoices, forms, reports, slide decks, screenshots, and scanned pages.

Argument	Type	Default	Description
`url`	`string`	required	The PDF or image.
`model`	enum	`rednote-hilab/dots.mocr`	OCR model to run. The tool advertises `rednote-hilab/dots.mocr`, `deepseek-ai/deepseek-ocr-2`, `zai-org/glm-ocr`, and `paddleocr/pp-ocrv6`.
`dpi`	`integer`	`72`	Rasterization DPI (72-400) per PDF page before OCR. Higher is sharper on small text, but slower.
`method`	`string`	model default	Backend method run on each page: `markdown` on the generative OCR models, `ocr` on `paddleocr/pp-ocrv6`. A method the model does not advertise is rejected before the call. See Pick a method.
`method_params`	`object`	unset	Keyword arguments for `method`, forwarded as-is.
`pages`	`list[int \| tuple[int,int]]`	every page (max 128)	0-indexed page selection. Each entry in the list is a page index or a `[start, stop]` half-open range (`start` inclusive, `stop` exclusive); negative indices count from the end. Only the selected pages are rasterized and read, and they are re-numbered from 0 in the response, where `num_pages` counts the pages you selected rather than the pages in the file. Indices past the end are ignored.
`json_mode`	`boolean`	`false`	`false` returns the `<document>` / `<page>` text blocks; `true` returns the `{"model", "method", "data"}` response object of per-page records.

Page Selection

[0, 2, 4] reads pages 0, 2, and 4
[[0, 3]] reads pages 0-2
[[0, 3], [5, 7], -1] reads pages 0, 1, 2, 5, 6 and the last page

One call reads at most 128 pages. For a longer document, call the tool once per range and combine the results. The error names the exact ranges to use.

JSON Mode

json_mode: false
json_mode: true

Left unset, the tool returns the same text blocks as the REST API: one <document> block per PDF, wrapping one <page> block per page, in reading order.

<document file_name="invoice.pdf" mimetype="application/pdf" num_pages="2" dpi="72">
<page page_index="0" format="markdown" page_width="612" page_height="792">
# ACME Corp, Invoice #4471

| Item | Qty | Price |
| ---- | --- | ------ |
| Widget | 3 | $12.00 |
</page>
<page page_index="1" format="markdown" page_width="612" page_height="792">
## Terms

Net 30. Thank you for your business.
</page>
</document>

A page that failed OCR is self-closing, with status="error" and no body. A single image returns the Markdown alone, with no wrapper.

json_mode: true returns the same pages as records in data, which is how you get boxes out of a region method:

[
  {
    "object": "document",
    "file_name": "invoice.pdf",
    "mimetype": "application/pdf",
    "num_pages": 2,
    "dpi": 72,
    "pages": [
      {
        "object": "document.page",
        "page_index": 0,
        "page_width": 612,
        "page_height": 792,
        "content": {
          "object": "document.page.blocks",
          "items": [
            {
              "index": 0,
              "bbox_xywh": [0.3987, 0.0896, 0.1993, 0.0164],
              "text": "ACME Corp",
              "score": 0.9998
            }
          ]
        }
      }
    ]
  }
]

Both shapes are documented under Text mode and JSON mode.

`read_image`

Send one still image to a vision-language model and get its answer back. Use it for photos, charts, diagrams, product shots, and UI screenshots, where the value is in what the picture depicts.

Argument	Type	Default	Description
`url`	`string`	required	The image.
`model`	`string`	`qwen/qwen3.5-0.8b`	Vision-language model that looks at the image.
`prompt`	`string`	”Describe this image in detail.”	The question or the instruction, for example “What is the total on this receipt?”.
`json_mode`	`boolean`	`false`	`false` returns the answer as a string; `true` asks the model itself for JSON, which a chat VLM returns unenveloped. See Chat VLMs are not enveloped.

The read_image menu carries no OCR model on purpose. An image whose value is its printed text belongs to read_document, which accepts an image URL and returns the Markdown alone, with no wrapper, for a single image.

`read_audio`

Transcribe a speech audio file (wav, mp3, m4a, flac, ogg, …) into text.

Argument	Type	Default	Description
`url`	`string`	required	The audio file.
`model`	`string`	`nvidia/parakeet-tdt-0.6b-v3`	Speech-to-text model to run.
`language`	`string`	auto-detect	ISO-639-1 language hint (`en`, `es`, …). Supply it when you already know the spoken language, to skip detection and improve accuracy.
`json_mode`	`boolean`	`false`	`false` returns the plain transcript; `true` returns the full transcription object (text plus metadata).

`read_video`

Describe or transcribe a video. Frames are sampled and encoded into image(s) so image-capable models can read them, then a vision-language model produces the text.

Argument	Type	Default	Description
`url`	`string`	required	The video file.
`model`	`string`	`qwen/qwen3.5-0.8b`	Vision-language model that watches the sampled frames.
`prompt`	`string`	”Transcribe and describe the content of this video in detail.”	What to produce from the video. Steer it with, for example, “List the on-screen text” or “Summarize what happens in 3 bullets”.
`fps`	`number`	uniform sampling bounded by `max_frames`	Frames sampled per second of video. Higher captures more motion at higher cost.
`max_frames`	`integer`	model default	Upper bound on the number of frames sampled.
`encoder`	`mosaic` \| `frames` \| `keyframes`	`mosaic`	How the video becomes images. `mosaic` tiles sampled keyframes into grid images and is robust for every model; `frames` sends sampled frames as separate images; `keyframes` uses the video’s I-frames.
`encoder_params`	`object`	encoder defaults	Parameters for `encoder`, e.g. `{"tile_cols": 4, "tile_rows": 4, "num_frames": 128}` for `mosaic`, or `{"fps": 1.0}` for `frames`.
`json_mode`	`boolean`	`false`	`false` returns the description as a string; `true` asks the model itself for JSON, which a chat VLM returns unenveloped.

On the REST API, encoder and encoder_params map to video_encoder and video_encoder_params.

`list_models`

List the models the gateway currently serves and which read tool each one fits. Call it before passing a non-default model.

Argument	Type	Default	Description
`modality`	`document` \| `image` \| `audio` \| `video`	all	Restrict the listing to one tool’s models.

Each entry is {id, modality, tool, tools, aliases, methods, default_method, method_details}. Pass id verbatim as the model argument to the matching tool. tools lists every read tool the model fits, because one model can serve several: a Qwen vision model fits read_image and read_video. modality and tool name the primary one, and a modality filter matches on any fitting tool. Fitting a tool is not the same as being selectable through it: outside read_document, each tool’s model argument accepts a single id, so treat tools as what the gateway can route rather than as a menu. method_details is the per-method answer to “what do I get back”: one record per method, with its capabilities, default: true on the model’s default, and requires_json_mode: true when the method’s full payload only arrives parsed under json_mode. A method that carries document_markdown or text_extraction still returns that text as a plain string.

{
  "id": "paddleocr/pp-ocrv6",
  "modality": "document",
  "tool": "read_document",
  "tools": ["read_document", "read_image"],
  "aliases": ["pp-ocrv6"],
  "methods": ["ocr", "detect", "text"],
  "default_method": "ocr",
  "method_details": [
    {
      "name": "ocr",
      "capabilities": [
        "text_extraction",
        "text_citations",
        "text_highlighting"
      ],
      "default": true,
      "requires_json_mode": true
    },
    {
      "name": "detect",
      "capabilities": ["text_highlighting"],
      "requires_json_mode": true
    },
    { "name": "text", "capabilities": ["text_extraction"] }
  ]
}

Return shape (`ReadResult`)

Each read_* tool returns the clean text as the tool’s content and a typed structured payload, so an agent gets usable text directly while still being able to read which model actually ran and what it cost:

Field	Description
`text`	The extracted text (Markdown, transcript, or description). Empty when `json_mode` is `true`.
`data`	The structured records, when `json_mode` is `true`. Null otherwise.
`model`	The model the gateway actually ran.
`cost`	The metered cost of the call in US dollars, when the gateway reports it.

Very large output is trimmed before it reaches the agent so one document cannot flood the context: past roughly 60,000 characters the text is cut and a [truncated: …] note is appended telling the agent to narrow the request (for example, fewer pages) or process in sections.

Errors

An error names the next step, so the agent recovers on its own. A method the chosen model does not advertise is refused before the call, and the message lists the methods it does advertise:

model 'paddleocr/pp-ocrv6' does not support method 'markdown'. It advertises:
detect, ocr, text. Retry with one of those, or omit `method` for the default.
`list_models` reports the methods of every served model.

Symptom	Cause	Fix
`400 Bad Request: Missing session ID`	The client dropped the `mcp-session-id` header from `initialize`.	Send the header the handshake returned on every later request.
`Session not found`	The session expired, or the request reached a different gateway replica.	Run `initialize` again and use the new session id.
`Failed to fetch document URL: HTTP Error 404`	The gateway cannot reach `url`.	Host the file publicly, or pass it as a `data:` URI or base64.
`does not support method '…'`	The method belongs to another model.	Use a method the message names, or omit `method`.
Boxes or labels are missing from the answer	The payload only arrives under `json_mode`.	Set `json_mode: true`, then read `data`.
`[truncated: …]` at the end of the text	The output passed the inline cap.	Narrow `pages`, or read the document in sections.

Call it without a framework

Three requests reach a tool over plain HTTP. initialize returns the mcp-session-id header, notifications/initialized completes the handshake, and every later request carries the header. Send Accept: application/json, text/event-stream, because the reply is a Server-Sent Events stream.

MCP=https://gateway.vlm.run/mcp
HDRS=(-H "Authorization: Bearer $VLMRUN_API_KEY"
      -H "Content-Type: application/json"
      -H "Accept: application/json, text/event-stream")

curl -sD /tmp/h -o /dev/null -X POST $MCP "${HDRS[@]}" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{
       "protocolVersion":"2025-06-18","capabilities":{},
       "clientInfo":{"name":"curl","version":"1"}}}'

SESSION=$(grep -i '^mcp-session-id:' /tmp/h | tr -d '\r' | cut -d' ' -f2)

curl -s -o /dev/null -X POST $MCP "${HDRS[@]}" -H "mcp-session-id: $SESSION" \
  -d '{"jsonrpc":"2.0","method":"notifications/initialized"}'

curl -sN -X POST $MCP "${HDRS[@]}" -H "mcp-session-id: $SESSION" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{
       "name":"list_models","arguments":{"modality":"document"}}}'

Chat Completions API

The REST surface behind the same gateway pipeline.

List models

Query the live model catalog and capabilities.

Authentication

Bearer tokens, tiers, and anonymous access.

Multimodal Inputs

Content part types, document limits, and video knobs.

Overview

Guides

API Reference

Misc

What is MCP?

When to use MCP

Quickstart

Connect an agent

Tools

Pick a method by what you need

`read_document`

Page Selection

JSON Mode

`read_image`

`read_audio`

`read_video`

`list_models`

Return shape (`ReadResult`)

Errors

Call it without a framework

Chat Completions API

List models

Authentication

Multimodal Inputs

​What is MCP?

​When to use MCP

​Quickstart

​Connect an agent

​Tools

​Pick a method by what you need

​read_document

​Page Selection

​JSON Mode

​read_image

​read_audio

​read_video

​list_models

​Return shape (ReadResult)

​Errors

​Call it without a framework

​Related

Chat Completions API

List models

Authentication

Multimodal Inputs

What is MCP?

When to use MCP

Quickstart

Connect an agent

Tools

Pick a method by what you need

`read_document`

Page Selection

JSON Mode

`read_image`

`read_audio`

`read_video`

`list_models`

Return shape (`ReadResult`)

Errors

Call it without a framework

Related