MCP Server

The VLM Run Gateway exposes a Model Context Protocol (MCP) server so any MCP-aware agent framework (Pydantic AI, LangChain, Mastra, OpenAI Agents SDK, Claude Code) can read documents, audio, and video through the same gateway pipeline as the REST API. It is a FastMCP server served over Streamable HTTP, and it re-enters the same ingress as chat/completions, so authentication, billing, and metering behave identically.

Property	Value
Endpoint	`POST https://gateway.vlm.run/mcp`
Transport	Streamable HTTP (FastMCP)
Auth	`Authorization: Bearer <VLMRUN_API_KEY>`

Authentication follows the same tiers as the REST API. See Authentication and Rate Limits.

Tools

The server exposes one read tool per modality plus a model-discovery tool. Every read_* tool takes a url and returns the extracted text.

Tool	Signature	Returns
`read_document`	`read_document(url, model?, dpi?, pages?, response_format?)`	Markdown OCR
`read_audio`	`read_audio(url, model?, language?, response_format?)`	Transcript
`read_video`	`read_video(url, model?, prompt?, fps?, max_frames?, encoder?, encoder_params?, response_format?)`	Text description / transcript
`list_models`	`list_models(modality?)`	Models the gateway serves, and which read tool each fits

`url`

url is always an http(s) URL, a data: URI, or a bare base64 string. It is never a local filesystem path. Host the file or inline it as base64 before calling a read tool.

`response_format`

response_format selects how the result is serialized:

The native form (markdown for documents, text for audio and video) returns a plain string.
json returns the structured envelope.

Return shape (`ReadResult`)

Each read_* tool returns the clean text as the tool’s content and a typed structured payload, so an agent gets usable text directly while still being able to read which model actually ran and what it cost:

Field	Description
`text`	The extracted text (markdown / transcript / description).
`data`	The structured envelope (present for `json` output).
`model`	The model the gateway actually ran.
`cost`	The metered cost of the call.

Errors

A rejected or unsupported model returns a clean, actionable message plus a next step (for example, “call the list_models tool”) rather than a raw error envelope, so the agent can recover on its own.

Page selection with `pages`

read_document accepts a pages argument to read only specific PDF pages instead of the whole document, saving cost and latency.

Pages are 0-indexed. Pass a list whose items are single page indices and/or [start, stop] half-open ranges (numpy-style: start inclusive, stop exclusive). Negative indices count from the end.
Only the selected pages are rasterized and read. They are re-numbered from 0 in the response.

`pages`	Reads
`[0, 2, 4]`	pages 0, 2, 4
`[[0, 3]]`	pages 0, 1, 2
`[[0, 3], [5, 7]]`	pages 0, 1, 2, 5, 6
`[-1]`	the last page

Page selection is specific to the read_document MCP tool. The REST chat/completions API reads every page (up to document_max_pages).

Video encoding with `encoder`

read_video encodes a video into image(s) before dispatch so image-only models can read it. Control this with encoder and encoder_params:

encoder defaults to mosaic (sampled keyframes tiled into a single grid image). frames and keyframes are also available.
encoder_params tunes the encoder (for example tile_cols, tile_rows, num_frames).

On the REST API these map to video_encoder and video_encoder_params.

Connect an agent

Every MCP-aware client connects the same way: point it at https://gateway.vlm.run/mcp with a bearer token, and the four tools above show up alongside the agent’s other tools. Set your key once:

export VLMRUN_API_KEY="your-api-key"

Pydantic AI
LangChain
Mastra
OpenAI Agents SDK
Claude Code

from pydantic_ai import Agent
from pydantic_ai.capabilities import MCP

agent = Agent(
    "anthropic:claude-sonnet-5",
    capabilities=[
        MCP(
            "https://gateway.vlm.run/mcp",
            headers={"Authorization": f"Bearer {KEY}"},
        )
    ],
)

native=False (the default) runs the MCP client locally: Pydantic AI connects to the gateway and calls the tools. native=True advertises the URL to the model provider (OpenAI Responses, Anthropic, xAI) so the provider connects to the server directly.

from langchain_mcp_adapters.client import MultiServerMCPClient
from langgraph.prebuilt import create_react_agent

client = MultiServerMCPClient(
    {
        "vlmrun": {
            "url": "https://gateway.vlm.run/mcp",
            "transport": "streamable_http",
            "headers": {"Authorization": f"Bearer {KEY}"},
        }
    }
)

tools = await client.get_tools()
agent = create_react_agent("anthropic:claude-sonnet-5", tools)

Install the adapter with pip install langchain-mcp-adapters langgraph.

import { MCPClient } from "@mastra/mcp";

const mcp = new MCPClient({
  servers: {
    vlmrun: {
      url: new URL("https://gateway.vlm.run/mcp"),
      requestInit: {
        headers: { Authorization: `Bearer ${process.env.VLMRUN_API_KEY}` },
      },
    },
  },
});

const tools = await mcp.getTools();

Pass tools to any Mastra Agent. Install with npm install @mastra/mcp.

from agents import Agent, Runner
from agents.mcp import MCPServerStreamableHttp

async with MCPServerStreamableHttp(
    params={
        "url": "https://gateway.vlm.run/mcp",
        "headers": {"Authorization": f"Bearer {KEY}"},
    }
) as server:
    agent = Agent(
        name="Assistant",
        mcp_servers=[server],
    )
    result = await Runner.run(agent, "Summarize the attached PDF.")
    print(result.final_output)

claude mcp add --transport http vlmrun https://gateway.vlm.run/mcp \
  --header "Authorization: Bearer $VLMRUN_API_KEY"

The read_document, read_audio, read_video, and list_models tools are now available in your Claude Code session.

End-to-end example

Once connected, the agent calls the read tools on its own. Ask a question about a document and it will fetch, OCR, and answer, all through the gateway:

from pydantic_ai import Agent
from pydantic_ai.capabilities import MCP

agent = Agent(
    "anthropic:claude-sonnet-5",
    capabilities=[
        MCP(
            "https://gateway.vlm.run/mcp",
            headers={"Authorization": f"Bearer {KEY}"},
        )
    ],
)

result = agent.run_sync(
    "Read https://example.com/invoice.pdf and tell me the total amount due."
)
print(result.output)

The agent picks read_document, receives the markdown OCR plus the ReadResult payload (including the model that ran and its cost), and reasons over the text to answer, no extraction schema or glue code required.

Explore the server

Point the MCP Inspector at the gateway to list tools and call them interactively before wiring up an agent:

npx @modelcontextprotocol/inspector

Set the transport to Streamable HTTP, the URL to https://gateway.vlm.run/mcp, and add an Authorization: Bearer <VLMRUN_API_KEY> header. Start with list_models to see what the gateway currently serves and which read tool each model fits.

Chat Completions API

The REST surface behind the same gateway pipeline.

List models

Query the live model catalog and capabilities.

Authentication

Bearer tokens, tiers, and anonymous access.

Multimodal Inputs

Content part types, document limits, and video knobs.

Overview

Guides

API Reference

Misc

Tools

`url`

`response_format`

Return shape (`ReadResult`)

Errors

Page selection with `pages`

Video encoding with `encoder`

Connect an agent

End-to-end example

Explore the server

Chat Completions API

List models

Authentication

Multimodal Inputs

​Tools

​url

​response_format

​Return shape (ReadResult)

​Errors

​Page selection with pages

​Video encoding with encoder

​Connect an agent

​End-to-end example

​Explore the server

​Related

Chat Completions API

List models

Authentication

Multimodal Inputs

Tools

`url`

`response_format`

Return shape (`ReadResult`)

Errors

Page selection with `pages`

Video encoding with `encoder`

Connect an agent

End-to-end example

Explore the server

Related