client.gateway

client.gateway is an OpenAI-compatible surface for third-party OCR and vision-language models. Auth uses the same VLMRUN_API_KEY as the rest of the SDK. The OpenAI SDK base URL is https://gateway.vlm.run/v1/openai. client.gateway.completions, client.gateway.embeddings, and client.gateway.transcriptions are standard OpenAI resource objects.

The gateway passes requests through to the underlying model. Most models, especially OCR models, do not accept text-only input. You generally need at least one image_url or document_url content part. Agent calls use client.agent.

Installation

The OpenAI SDK is an optional peer dependency.

npm install vlmrun openai

Chat Completions

Pass PDFs as document_url content parts. Pass images as image_url. method and document_dpi travel with the standard OpenAI fields.

import { VlmRun } from "vlmrun";

const client = new VlmRun({ apiKey: "<VLMRUN_API_KEY>" });

const response = await client.gateway.completions.create({
  model: "paddleocr/pp-ocrv6",
  messages: [
    {
      role: "user",
      content: [
        {
          type: "document_url",
          document_url: {
            url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf",
          },
        },
      ],
    },
  ],
  method: "ocr",
  document_dpi: 96,
});

console.log(response.choices[0].message.content);

Visual Q&A

Models that accept text, such as qwen/qwen3.5-0.8b, take a text content part with an image.

const response = await client.gateway.completions.create({
  model: "qwen/qwen3.5-0.8b",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "What is happening in this image?" },
        { type: "image_url", image_url: { url: "https://example.com/image.jpg" } },
      ],
    },
  ],
});

console.log(response.choices[0].message.content);

Streaming

Document (PDF) requests stream one SSE chunk per page.

const stream = await client.gateway.completions.create({
  model: "zai-org/glm-ocr",
  messages: [
    {
      role: "user",
      content: [
        { type: "document_url", document_url: { url: "https://example.com/doc.pdf" } },
      ],
    },
  ],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

Embeddings

Models accept text, images, or video.

Multimodal input nests content parts one level deeper than plain text. input is a list. Each item is a string or a list of content parts.

const response = await client.gateway.embeddings.create({
  model: "qwen/qwen3-vl-embedding-2b",
  input: [[{ type: "image_url", image_url: { url: "data:image/jpeg;base64,..." } }]],
});

console.log(response.data[0].embedding.length);

Transcriptions

Models accept audio, including a video’s audio track.

import fs from "fs";

const response = await client.gateway.transcriptions.create({
  model: "nvidia/parakeet-tdt-0.6b-v3",
  file: fs.createReadStream("clip.mp3"),
});

console.log(response.text);

Listing Models

Each model adds input and output pricing, modality support, and supported methods beyond the OpenAI fields.

const models = await client.gateway.models();
models.forEach((model) => console.log(model.id));

System One

client.gateway.systemone answers typed questions about text, JSON, images and PDFs on POST /typesafe/v1/systemone, returning a calibrated probability for every answer. It uses the client’s API key and the gateway base URL, so no extra package or configuration is needed.

const result = await client.gateway.systemone.decide({
  state: "Invoice #44 was charged twice, I need this fixed today",
  questions: [
    { id: "is_urgent", type: "noul", instructions: "Is this time-sensitive?" },
    { id: "department", type: "choice", options: ["billing", "technical", "sales"] },
    { id: "mood", type: "score", levels: ["calm", "frustrated", "very angry"] },
  ],
});

result.nouls.is_urgent.noul;       // 0.91
result.choices.department.choice;  // "billing"
result.scores.mood.score;          // 1.4
result.usage?.input_tokens;

Questions are a list with an id on each, or a mapping of id to question. options (choice) and levels (score) are the friendly spellings of the wire’s criteria, and a noul’s criteria accepts yes/no for true/false. A malformed spec throws InputError before anything is sent.

Media

Attach up to 8 images (local paths, http(s) URLs or data URLs) and one PDF. Images are always inlined as base64 data URLs and checked by magic bytes (JPEG, PNG, WebP, GIF; 5 MB each). Remote images are fetched by the SDK, which refuses hosts that resolve to private or loopback addresses unless VLMRUN_ALLOW_PRIVATE_URLS=1 is set. A remote PDF is passed by URL for the gateway to fetch.

const result = await client.gateway.systemone.decide({
  state: "Answer about the attached page.",
  questions: {
    kind: { type: "choice", options: ["invoice", "receipt", "contract", "other"] },
    has_total: { type: "noul", instructions: "Is a total amount due visible?" },
  },
  images: ["invoice-page.jpg"],
  // or: document: "invoice.pdf",
  detail: "high",
});

Option	Description
`model`	Model override (default `google/diffusiongemma-26b-a4b-it`, or `TYPESAFE_DEFAULT_MODEL`)
`images` / `document` / `text`	Media and extra text parts, read after the state
`detail`	Vision budget: `auto`, `low` or `high`
`steps` / `samples`	Denoise steps per read (1-8) and noise draws to average (1-32)
`reasoningEffort`	`none` … `high`; generative engines only
`timeoutMs`	Per-call timeout in milliseconds
`extraBody` / `extraHeaders`	Extra body fields (merged last) and request headers

buildRequest() returns the exact body decide() would send, without sending it. models() lists the models the route serves.

Streams

For a series of inputs (video frames, a camera, a queue) with the same questions, open a stream. map() overlaps up to concurrency reads and yields results in input order; send() reads one input at a time.

const stream = await client.gateway.systemone.stream({
  questions: [{ id: "is_open", type: "noul" }],
  state: "Is the door open?",
  concurrency: 4,
});

try {
  for await (const decision of stream.map(["frame-0.jpg", "frame-1.jpg"])) {
    console.log(decision.index, decision.response.nouls.is_open.noul);
  }
} finally {
  await stream.close();
}

Inputs may be { frame, index, timestampS } objects to keep a timeline. Streams run over HTTP by default. Pass transport: "ws" to pipeline the reads over one WebSocket session; this needs the optional ws package (npm install ws).

Health Check

health() returns true when the gateway is reachable and authenticated.

if (await client.gateway.health()) {
  console.log("Gateway is healthy");
}

Configuration

The default gateway URL is https://gateway.vlm.run/v1. Override it with VLMRUN_GATEWAY_URL.

export VLMRUN_GATEWAY_URL="https://gateway.vlm.run/v1"

System One derives its /typesafe root from the gateway URL (https://gateway.vlm.run/typesafe); set TYPESAFE_BASE_URL to point it elsewhere. Multi-page PDF OCR often exceeds the client timeout default of 120s. The gateway timeout floor is 600s. An explicit timeout other than the default is used as-is.

SDK Reference

Attribute / Method	Description
`client.gateway.completions`	OpenAI-compatible chat completions interface
`client.gateway.embeddings`	OpenAI-compatible embeddings interface
`client.gateway.transcriptions`	OpenAI-compatible audio transcriptions interface
`client.gateway.models()`	List models available on the gateway
`client.gateway.systemone`	System One typed decisions (`decide`, `buildRequest`, `stream`, `models`)
`client.gateway.health()`	Resolves to `true` if the gateway is reachable and authenticated
`client.gateway.baseUrl`	Gateway base URL (without trailing slash)

Gateway documentation lists the model catalog. Methods. Multimodal input. Python equivalent.

Overview

SDKs

CLI

CLI - Gateway

CLI - Orion

Installation

Chat Completions

Visual Q&A

Streaming

Embeddings

Transcriptions

Listing Models

System One

Media

Streams

Health Check

Configuration

SDK Reference

​Installation

​Chat Completions

​Visual Q&A

​Streaming

​Embeddings

​Transcriptions

​Listing Models

​System One

​Media

​Streams

​Health Check

​Configuration

​SDK Reference

Installation

Chat Completions

Visual Q&A

Streaming

Embeddings

Transcriptions

Listing Models

System One

Media

Streams

Health Check

Configuration

SDK Reference