Quickstart

Install the CLI, point any OpenAI SDK at https://gateway.vlm.run/v1/openai, or add the vlmrun-gw skill.

Prerequisites

pip install openai

npm install openai

pip install -U vlmrun

export VLMRUN_API_KEY="your-api-key"

The base URL is https://gateway.vlm.run/v1/openai. Send Authorization: Bearer <VLMRUN_API_KEY>, or omit the header for anonymous access. For details, see Authentication and Rate Limits.

Visual Q&A
Video Q&A
Document OCR

Send a question with an optional image. To see which models you can call, browse the model catalog.

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key="<VLMRUN_API_KEY>",
)

response = client.chat.completions.create(
    model="qwen/qwen3.5-0.8b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is happening in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg",
                    },
                },
            ],
        }
    ],
)

print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.vlm.run/v1/openai",
  apiKey: process.env.VLMRUN_API_KEY,
});

const response = await client.chat.completions.create({
  model: "qwen/qwen3.5-0.8b",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "What is happening in this image?" },
        {
          type: "image_url",
          image_url: {
            url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg",
          },
        },
      ],
    },
  ],
});

console.log(response.choices[0].message.content);

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg \
  -p "What is happening in this image?" \
  -m qwen/qwen3.5-0.8b

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.5-0.8b",
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "What is happening in this image?" },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg"
            }
          }
        ]
      }
    ]
  }'

Chat models return the reply verbatim, with no envelope.

Send a question with a hosted video URL. video_fps and video_max_frames set frame sampling. For every video option, see Video Inputs.

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key="<VLMRUN_API_KEY>",
)

response = client.chat.completions.create(
    model="qwen/qwen3.5-0.8b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Summarize what happens in this video."},
                {
                    "type": "video_url",
                    "video_url": {
                        "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4",
                    },
                },
            ],
        }
    ],
    extra_body={"video_fps": 1.0, "video_max_frames": 8},
)

print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.vlm.run/v1/openai",
  apiKey: process.env.VLMRUN_API_KEY,
});

const response = await client.chat.completions.create({
  model: "qwen/qwen3.5-0.8b",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Summarize what happens in this video." },
        {
          type: "video_url",
          video_url: {
            url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4",
          },
        },
      ],
    },
  ],
  video_fps: 1.0,
  video_max_frames: 8,
});

console.log(response.choices[0].message.content);

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4 \
  -p "Summarize what happens in this video." \
  -m qwen/qwen3.5-0.8b \
  -e video_fps=1.0 \
  -e video_max_frames=8

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.5-0.8b",
    "video_fps": 1.0,
    "video_max_frames": 8,
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "Summarize what happens in this video." },
          {
            "type": "video_url",
            "video_url": {
              "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"
            }
          }
        ]
      }
    ]
  }'

PDF: document_url.
Image: image_url.
Start with paddlepaddle/paddleocr-vl-1.6 for general-purpose OCR.
method: ocr (plain text) or markdown.

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key="<VLMRUN_API_KEY>",
)

response = client.chat.completions.create(
    model="paddlepaddle/paddleocr-vl-1.6",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "document_url",
                    "document_url": {
                        "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
                    },
                }
            ],
        }
    ],
    extra_body={
        "method": "ocr",
        "document_dpi": 96,
    },
)

print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.vlm.run/v1/openai",
  apiKey: process.env.VLMRUN_API_KEY,
});

const response = await client.chat.completions.create({
  model: "paddlepaddle/paddleocr-vl-1.6",
  messages: [
    {
      role: "user",
      content: [
        {
          type: "document_url",
          document_url: {
            url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf",
          },
        },
      ],
    },
  ],
  method: "ocr",
  document_dpi: 96,
});

console.log(response.choices[0].message.content);

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf \
  -m paddlepaddle/paddleocr-vl-1.6 \
  --method ocr \
  -e document_dpi=96

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "paddlepaddle/paddleocr-vl-1.6",
    "method": "ocr",
    "document_dpi": 96,
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "document_url",
            "document_url": {
              "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
            }
          }
        ]
      }
    ]
  }'

The reply is one <document> block. Each page is one <page> block. format is the payload kind:

<document file_name="tsla-8k.pdf" file_hash="sha256:1b7f04c3…" file_bytes="48213" mimetype="application/pdf" npages="5" dpi="96">
<page id="0" format="markdown" width="816" height="1056">
UNITED STATES
SECURITIES AND EXCHANGE COMMISSION
WASHINGTON, DC 20549

FORM 8-K
</page>
</document>

response_format={"type": "json_object"} returns one JSON object. The CLI equivalent is --json-mode. For the response contract, see Response Formats. To build a full document pipeline, see Document OCR.

Use with a coding agent

Install the vlmrun-gw skill so your agent calls the Gateway for you.

npx skills add vlm-run/skills@vlmrun-gw

Or point it at skill.md for one session.

https://vlm.run/gateway/skill.md

Next steps

You can call the same models from the Python SDK and Node.js SDK through client.gateway. From the command line, use vlmrun gw, which includes gw models, gw embed, and gw transcribe.

Document OCR

Request knobs, page blocks, and streaming for PDFs.

Models

Browse the catalog and each model’s capabilities.

Chat Completions API

Request parameters and response schema.

Node.js SDK

client.gateway on the vlmrun package.

MCP Server

Read tools for documents, audio, and video.

Overview

Platform

OpenAI Compatibility

TypeSafe Compatibility

MCP

Misc

Prerequisites

Use with a coding agent

Next steps

Document OCR

Models

Chat Completions API

Node.js SDK

MCP Server

​Prerequisites

​Use with a coding agent

​Next steps

Document OCR

Models

Chat Completions API

Node.js SDK

MCP Server

Prerequisites

Use with a coding agent

Next steps