Quickstart

Install the CLI, or point any OpenAI SDK at https://gateway.vlm.run/v1/openai. This page covers the three most common request shapes: visual Q&A, video Q&A, and document OCR. See Introduction for the rationale behind the VLM Run Gateway.

Prerequisites

pip install vlmrun

pip install openai

npm install openai

export VLMRUN_API_KEY="your-api-key"

Base URL: https://gateway.vlm.run/v1/openai. Use Authorization: Bearer <VLMRUN_API_KEY>, or api_key="vlmrun" for anonymous access. See Authentication and Rate Limits.

Visual Q&A

Send a question with an optional image. See Models for the rest of the catalog.

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg \
  -p "What is happening in this image?" \
  -m qwen/qwen3.5-0.8b

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key=os.environ["VLMRUN_API_KEY"],
)

response = client.chat.completions.create(
    model="qwen/qwen3.5-0.8b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is happening in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg",
                    },
                },
            ],
        }
    ],
)

print(response.choices[0].message.content)

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.5-0.8b",
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "What is happening in this image?" },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg"
            }
          }
        ]
      }
    ]
  }'

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.vlm.run/v1/openai",
  apiKey: process.env.VLMRUN_API_KEY,
});

const response = await client.chat.completions.create({
  model: "qwen/qwen3.5-0.8b",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "What is happening in this image?" },
        {
          type: "image_url",
          image_url: {
            url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg",
          },
        },
      ],
    },
  ],
});

console.log(response.choices[0].message.content);

Chat models return their reply verbatim, with no envelope around it.

Video Q&A

Send a question with a hosted video URL. Tune frame sampling with video_fps and video_max_frames. See Video Inputs for all knobs.

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4 \
  -p "Summarize what happens in this video." \
  -m qwen/qwen3.5-0.8b \
  -e video_fps=1.0 \
  -e video_max_frames=8

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key=os.environ["VLMRUN_API_KEY"],
)

response = client.chat.completions.create(
    model="qwen/qwen3.5-0.8b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Summarize what happens in this video."},
                {
                    "type": "video_url",
                    "video_url": {
                        "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4",
                    },
                },
            ],
        }
    ],
    extra_body={"video_fps": 1.0, "video_max_frames": 8},
)

print(response.choices[0].message.content)

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.5-0.8b",
    "video_fps": 1.0,
    "video_max_frames": 8,
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "Summarize what happens in this video." },
          {
            "type": "video_url",
            "video_url": {
              "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"
            }
          }
        ]
      }
    ]
  }'

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.vlm.run/v1/openai",
  apiKey: process.env.VLMRUN_API_KEY,
});

const response = await client.chat.completions.create({
  model: "qwen/qwen3.5-0.8b",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Summarize what happens in this video." },
        {
          type: "video_url",
          video_url: {
            url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4",
          },
        },
      ],
    },
  ],
  video_fps: 1.0,
  video_max_frames: 8,
});

console.log(response.choices[0].message.content);

Document and Image OCR

Pass a PDF as a document_url content part, or an image as image_url. Start with paddleocr/pp-ocrv6 for general-purpose OCR (ocr is the default method).

vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf \
  -m paddleocr/pp-ocrv6 \
  -e document_dpi=96

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key=os.environ["VLMRUN_API_KEY"],
)

response = client.chat.completions.create(
    model="paddleocr/pp-ocrv6",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "document_url",
                    "document_url": {
                        "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
                    },
                }
            ],
        }
    ],
    extra_body={
        "method": "ocr",
        "document_dpi": 96,
    },
)

print(response.choices[0].message.content)

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "paddleocr/pp-ocrv6",
    "method": "ocr",
    "document_dpi": 96,
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "document_url",
            "document_url": {
              "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
            }
          }
        ]
      }
    ]
  }'

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.vlm.run/v1/openai",
  apiKey: process.env.VLMRUN_API_KEY,
});

const response = await client.chat.completions.create({
  model: "paddleocr/pp-ocrv6",
  messages: [
    {
      role: "user",
      content: [
        {
          type: "document_url",
          document_url: {
            url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf",
          },
        },
      ],
    },
  ],
  method: "ocr",
  document_dpi: 96,
});

console.log(response.choices[0].message.content);

The reply is one <document> block wrapping one <page> block per page. Each page’s body is document.page.blocks:

<document file_name="tsla-8k.pdf" mimetype="application/pdf" num_pages="5" dpi="96">
<page page_index="0" format="json" page_width="612" page_height="792">
{"object": "document.page.blocks", "items": [{"index": 0, "bbox_xywh": [0.3987, 0.0896, 0.1993, 0.0164], "text": "UNITED STATES", "score": 0.9998}]}
</page>
</document>

Set response_format={"type": "json_object"} (CLI: --json-mode) to get one JSON object instead. See Methods & Response Format for the contract and Flexible Document OCR for the rest of the pipeline.

Next steps

The same models are available from the Python SDK via client.gateway. See vlmrun gw for gw models, gw embed, and gw transcribe.

Flexible Document OCR

Request knobs, page blocks, and streaming for PDFs.

Models

Catalog, capabilities, and model selection.

Chat Completions API

Full request parameters and response schema.

MCP Server

Give any MCP-aware agent read tools for documents, audio, and video.

Overview

Guides

API Reference

Misc

Prerequisites

Visual Q&A

Video Q&A

Document and Image OCR

Next steps

Flexible Document OCR

Models

Chat Completions API

MCP Server

​Prerequisites

​Visual Q&A

​Video Q&A

​Document and Image OCR

​Next steps

Flexible Document OCR

Models

Chat Completions API

MCP Server

Prerequisites

Visual Q&A

Video Q&A

Document and Image OCR

Next steps