Quickstart

Install the CLI, point any OpenAI SDK at https://gateway.vlm.run/v1/openai, or add the vlmrun-gw agent skill. This page covers the three most common request shapes: visual Q&A, video Q&A, and document OCR. See Introduction for the rationale behind the VLM Run Gateway.

Prerequisites

pip install openai

npm install openai

pip install -U vlmrun

export VLMRUN_API_KEY="your-api-key"

Base URL: https://gateway.vlm.run/v1/openai. Use Authorization: Bearer <VLMRUN_API_KEY>, or api_key="vlmrun" for anonymous access. See Authentication and Rate Limits.

Coding agents

Skills-aware coding agents can drive the Gateway through the vlmrun-gw skill: model catalog, vlmrun gw commands, request knobs, response shapes, and errors. The canonical skill file lives at vlm.run/gateway/SKILL.md. Install it with the skills CLI:

npx skills add vlm-run/skills@vlmrun-gw

Use -g to install under your user skills directory instead of the current repo. To limit the install to one agent, pass -a with its id (for example -a claude-code for Claude Code). To try the skill once without installing, send the following to your agent:

Read https://vlm.run/gateway/SKILL.md, then OCR
https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf
with glm-ocr and tell me the earliest event date in the filing.

After the skill is installed, a natural-language ask is enough. The agent picks vlmrun gw chat, the OpenAI SDK, or VLMRun().gateway as the skill describes:

Convert invoice.pdf to markdown with zai-org/glm-ocr and save the output next to the PDF.

For MCP read tools inside an agent framework, see MCP Server.

Visual Q&A

Send a question with an optional image. See Models for the rest of the catalog.

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key=os.environ["VLMRUN_API_KEY"],
)

response = client.chat.completions.create(
    model="qwen/qwen3.5-0.8b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is happening in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg",
                    },
                },
            ],
        }
    ],
)

print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.vlm.run/v1/openai",
  apiKey: process.env.VLMRUN_API_KEY,
});

const response = await client.chat.completions.create({
  model: "qwen/qwen3.5-0.8b",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "What is happening in this image?" },
        {
          type: "image_url",
          image_url: {
            url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg",
          },
        },
      ],
    },
  ],
});

console.log(response.choices[0].message.content);

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg \
  -p "What is happening in this image?" \
  -m qwen/qwen3.5-0.8b

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.5-0.8b",
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "What is happening in this image?" },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg"
            }
          }
        ]
      }
    ]
  }'

Chat models return their reply verbatim, with no envelope around it.

Video Q&A

Send a question with a hosted video URL. Tune frame sampling with video_fps and video_max_frames. See Video Inputs for all knobs.

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key=os.environ["VLMRUN_API_KEY"],
)

response = client.chat.completions.create(
    model="qwen/qwen3.5-0.8b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Summarize what happens in this video."},
                {
                    "type": "video_url",
                    "video_url": {
                        "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4",
                    },
                },
            ],
        }
    ],
    extra_body={"video_fps": 1.0, "video_max_frames": 8},
)

print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.vlm.run/v1/openai",
  apiKey: process.env.VLMRUN_API_KEY,
});

const response = await client.chat.completions.create({
  model: "qwen/qwen3.5-0.8b",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Summarize what happens in this video." },
        {
          type: "video_url",
          video_url: {
            url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4",
          },
        },
      ],
    },
  ],
  video_fps: 1.0,
  video_max_frames: 8,
});

console.log(response.choices[0].message.content);

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4 \
  -p "Summarize what happens in this video." \
  -m qwen/qwen3.5-0.8b \
  -e video_fps=1.0 \
  -e video_max_frames=8

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.5-0.8b",
    "video_fps": 1.0,
    "video_max_frames": 8,
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "Summarize what happens in this video." },
          {
            "type": "video_url",
            "video_url": {
              "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"
            }
          }
        ]
      }
    ]
  }'

Document and Image OCR

Pass a PDF as a document_url content part, or an image as image_url. Start with paddlepaddle/paddleocr-vl-1.6 for general-purpose OCR, and set method to pick the page read (ocr for plain text, markdown for Markdown).

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key=os.environ["VLMRUN_API_KEY"],
)

response = client.chat.completions.create(
    model="paddlepaddle/paddleocr-vl-1.6",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "document_url",
                    "document_url": {
                        "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
                    },
                }
            ],
        }
    ],
    extra_body={
        "method": "ocr",
        "document_dpi": 96,
    },
)

print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.vlm.run/v1/openai",
  apiKey: process.env.VLMRUN_API_KEY,
});

const response = await client.chat.completions.create({
  model: "paddlepaddle/paddleocr-vl-1.6",
  messages: [
    {
      role: "user",
      content: [
        {
          type: "document_url",
          document_url: {
            url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf",
          },
        },
      ],
    },
  ],
  method: "ocr",
  document_dpi: 96,
});

console.log(response.choices[0].message.content);

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf \
  -m paddlepaddle/paddleocr-vl-1.6 \
  --method ocr \
  -e document_dpi=96

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "paddlepaddle/paddleocr-vl-1.6",
    "method": "ocr",
    "document_dpi": 96,
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "document_url",
            "document_url": {
              "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
            }
          }
        ]
      }
    ]
  }'

The reply is one <document> block wrapping one <page> block per page. Each page declares its payload kind in format:

<document file_name="tsla-8k.pdf" file_hash="sha256:1b7f04c3…" file_bytes="48213" mimetype="application/pdf" npages="5" dpi="96">
<page id="0" format="markdown" width="816" height="1056">
UNITED STATES
SECURITIES AND EXCHANGE COMMISSION
WASHINGTON, DC 20549

FORM 8-K
</page>
</document>

Set response_format={"type": "json_object"} (CLI: --json-mode) to get one JSON object instead. See Methods & Response Format for the contract and Document OCR for the rest of the pipeline.

Next steps

The same models are available from the Python SDK via client.gateway. See vlmrun gw for gw models, gw embed, and gw transcribe.

Document OCR

Request knobs, page blocks, and streaming for PDFs.

Models

Catalog, capabilities, and model selection.

Chat Completions API

Full request parameters and response schema.

MCP Server

Give any MCP-aware agent read tools for documents, audio, and video.

Get Started

OpenAI-Compatible API

TypeSafe-Compatible API

Guides

API Reference

Tools

Platform

Prerequisites

Coding agents

Visual Q&A

Video Q&A

Document and Image OCR

Next steps

Document OCR

Models

Chat Completions API

MCP Server

​Prerequisites

​Coding agents

​Visual Q&A

​Video Q&A

​Document and Image OCR

​Next steps

Document OCR

Models

Chat Completions API

MCP Server

Prerequisites

Coding agents

Visual Q&A

Video Q&A

Document and Image OCR

Next steps