from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="<VLMRUN_API_KEY>",
)
response = client.chat.completions.create(
model="qwen/qwen3.5-0.8b",
max_tokens=60,
messages=[{"role": "user", "content": [
{"type": "text", "text": "What is happening in this image?"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"}},
]}],
)
print(response.choices[0].message.content)
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const response = await client.chat.completions.create({
model: "qwen/qwen3.5-0.8b",
max_tokens: 60,
messages: [{ role: "user", content: [
{ type: "text", text: "What is happening in this image?" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg" } },
] }],
});
console.log(response.choices[0].message.content);
curl https://gateway.vlm.run/v1/openai/chat/completions \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen3.5-0.8b",
"max_tokens": 60,
"messages": [{"role": "user", "content": [
{"type": "text", "text": "What is happening in this image?"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"}}
]}]
}'
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg \
-m qwen/qwen3.5-0.8b -p "What is happening in this image?" \
-e max_tokens=60
{
"id": "chatcmpl-efdaf1b758544decb5906225e66",
"object": "chat.completion",
"created": 1791250196,
"model": "qwen/qwen3.5-0.8b",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "This image features a vintage blue Volkswagen Beetle parked in front of a rustic house. The car, painted in a light teal color, captures the nostalgic charm of classic car models. Its iconic dome-shaped roof and round taillights add to its historical appeal. Surrounding it is an older charm shaped and"
},
"finish_reason": "length"
}
],
"usage": {
"prompt_tokens": 129,
"completion_tokens": 60,
"total_tokens": 189,
"cost": 0.001
}
}
{
"detail": [
{
"type": "missing",
"loc": ["body", "messages"],
"msg": "Field required",
"input": {
"model": "qwen/qwen3.5-0.8b",
"max_tokens": 60
}
}
]
}
OpenAI Compatible API
Chat Completions
OpenAI-compatible OCR, VQA, and document inference
POST
/
v1
/
openai
/
chat
/
completions
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="<VLMRUN_API_KEY>",
)
response = client.chat.completions.create(
model="qwen/qwen3.5-0.8b",
max_tokens=60,
messages=[{"role": "user", "content": [
{"type": "text", "text": "What is happening in this image?"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"}},
]}],
)
print(response.choices[0].message.content)
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const response = await client.chat.completions.create({
model: "qwen/qwen3.5-0.8b",
max_tokens: 60,
messages: [{ role: "user", content: [
{ type: "text", text: "What is happening in this image?" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg" } },
] }],
});
console.log(response.choices[0].message.content);
curl https://gateway.vlm.run/v1/openai/chat/completions \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen3.5-0.8b",
"max_tokens": 60,
"messages": [{"role": "user", "content": [
{"type": "text", "text": "What is happening in this image?"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"}}
]}]
}'
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg \
-m qwen/qwen3.5-0.8b -p "What is happening in this image?" \
-e max_tokens=60
{
"id": "chatcmpl-efdaf1b758544decb5906225e66",
"object": "chat.completion",
"created": 1791250196,
"model": "qwen/qwen3.5-0.8b",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "This image features a vintage blue Volkswagen Beetle parked in front of a rustic house. The car, painted in a light teal color, captures the nostalgic charm of classic car models. Its iconic dome-shaped roof and round taillights add to its historical appeal. Surrounding it is an older charm shaped and"
},
"finish_reason": "length"
}
],
"usage": {
"prompt_tokens": 129,
"completion_tokens": 60,
"total_tokens": 189,
"cost": 0.001
}
}
{
"detail": [
{
"type": "missing",
"loc": ["body", "messages"],
"msg": "Field required",
"input": {
"model": "qwen/qwen3.5-0.8b",
"max_tokens": 60
}
}
]
}
Authentication is optional. See Authentication for tiers and Rate Limits for per-tier quotas.
Request
| Part | Behavior | See |
|---|---|---|
model | A full id, a short form, or a Hugging Face id. | Addressing Models |
| Sampling fields | Only the fields in supported_parameters. Any other is a 400 capability_violation, and param names it. | Supported parameters |
| Content parts | text, image_url, document_url, video_url. | Supported Inputs |
response_format | Omit for text mode. json_object for JSON mode. json_schema on chat and provider models. | Response Formats |
| Gateway fields | method, method_params, precision, document_*, image_resolution, and video_*. Send them at the top level, or in extra_body on the OpenAI Python SDK. | Extra Kwargs |
Response
An OpenAI chat completion, plususage.cost in USD. The content depends on response_format. See Response Formats and Response Types.
Streaming
Setstream: true in text mode. The streamed text is byte-identical to the non-streaming reply, and a document request emits <document>, one <page> per page, then </document>.
- Streams deltas: chat VLMs, and single-image OCR methods that return the model’s text unchanged.
- Buffered: every other reply is built in full, then re-chunked as
chat.completion.chunkSSE. That includes rewritten text,paddleocr/pp-ocrv6, structured envelopes, and JSON modes. There is no time-to-first-token benefit. - Usage: the last chunk carries aggregated
usage, includingusage.cost. It arrives whether or not you passstream_options.include_usage.
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="<VLMRUN_API_KEY>",
)
stream = client.chat.completions.create(
model="qwen/qwen3.5-0.8b",
stream=True,
stream_options={"include_usage": True},
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What is happening in this image?"},
{
"type": "image_url",
"image_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"
},
},
],
}
],
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="")
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const stream = await client.chat.completions.create({
model: "qwen/qwen3.5-0.8b",
stream: true,
stream_options: { include_usage: true },
messages: [
{
role: "user",
content: [
{ type: "text", text: "What is happening in this image?" },
{
type: "image_url",
image_url: {
url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg",
},
},
],
},
],
});
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta?.content;
if (delta) process.stdout.write(delta);
}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg \
-p "What is happening in this image?" \
-m qwen/qwen3.5-0.8b
curl https://gateway.vlm.run/v1/openai/chat/completions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen3.5-0.8b",
"stream": true,
"stream_options": { "include_usage": true },
"messages": [
{
"role": "user",
"content": [
{ "type": "text", "text": "What is happening in this image?" },
{
"type": "image_url",
"image_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"
}
}
]
}
]
}'
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="<VLMRUN_API_KEY>",
)
response = client.chat.completions.create(
model="qwen/qwen3.5-0.8b",
max_tokens=60,
messages=[{"role": "user", "content": [
{"type": "text", "text": "What is happening in this image?"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"}},
]}],
)
print(response.choices[0].message.content)
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const response = await client.chat.completions.create({
model: "qwen/qwen3.5-0.8b",
max_tokens: 60,
messages: [{ role: "user", content: [
{ type: "text", text: "What is happening in this image?" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg" } },
] }],
});
console.log(response.choices[0].message.content);
curl https://gateway.vlm.run/v1/openai/chat/completions \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen3.5-0.8b",
"max_tokens": 60,
"messages": [{"role": "user", "content": [
{"type": "text", "text": "What is happening in this image?"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"}}
]}]
}'
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg \
-m qwen/qwen3.5-0.8b -p "What is happening in this image?" \
-e max_tokens=60
{
"id": "chatcmpl-efdaf1b758544decb5906225e66",
"object": "chat.completion",
"created": 1791250196,
"model": "qwen/qwen3.5-0.8b",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "This image features a vintage blue Volkswagen Beetle parked in front of a rustic house. The car, painted in a light teal color, captures the nostalgic charm of classic car models. Its iconic dome-shaped roof and round taillights add to its historical appeal. Surrounding it is an older charm shaped and"
},
"finish_reason": "length"
}
],
"usage": {
"prompt_tokens": 129,
"completion_tokens": 60,
"total_tokens": 189,
"cost": 0.001
}
}
{
"detail": [
{
"type": "missing",
"loc": ["body", "messages"],
"msg": "Field required",
"input": {
"model": "qwen/qwen3.5-0.8b",
"max_tokens": 60
}
}
]
}
Errors and limits
Every response includes anx-request-id header. A sanitized 500 body repeats it as error.request_id. See Error Codes and Rate Limits.
Related
List models
Query the live model catalog and capabilities.
Response Formats
Text, JSON, and JSON schema modes.
Supported Inputs
Content part types, document limits, and format tradeoffs.
Document OCR
Request knobs, page blocks, and streaming for PDFs.
Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Body
application/json
OpenAI-compatible chat completion request.
Available options:
none, minimal, low, medium, high Allowed value:
"native"Available options:
224x224, 336x336, 384x384, 448x448, 512x512, 768x768 Available options:
256x192, 320x240, 448x336, 512x384, 640x480 Required range:
x >= 1Requested serialization of the assistant reply.
Mirrors the OpenAI response_format object so existing client code
keeps working:
{"type": "text"}(the default) returns the reply verbatim.{"type": "json_object"}returns the reply as a single valid JSON object string: structured results (detections, OCR regions, document pages) are emitted as parseable JSON under a top-leveldatakey.{"type": "json_schema"}constrains generation to the supplied schema, on the models that declare :attr:ModelCapabilities.supports_json_schema— the chat VLMs served by vLLM and the routed provider models. Anything else (the OCR and detection models, whose reply is an envelope this gateway builds rather than text the model is free to shape) rejects it with a 400 capability violation instead of accepting it and ignoring it.
Required range:
1 <= x <= 8Response
Successful Response
OpenAI-compatible chat completion response.
Beyond the OpenAI shape we surface three extra fields so clients can audit which model actually produced the output:
served_model_id— the canonical id of the deployment that ran the request (e.g."rednote-hilab/dots.mocr"), even when the caller used a short alias. Mirrors the"id"echoed inGET /v1/openai/models.backend— static label naming the inference runtime that served the request ("vllm"/"transformers"/"llama.cpp"/"paddleocr"/ ...). Lets clients see the runtime without opening a separate /models RPC.provider— who ran the hardware. Absent for models served on VLM Run's own GPUs; otherwise the upstream that served the request ("google-vertex","huggingface", ...). Reported per response rather than read from the catalog because a request can be routed to a different upstream than the last one — the same model id may be served by more than one, and they do not always agree on capabilities or price.
The OpenAI-standard model field still echoes the canonical
id (matching OpenAI's own behaviour where a generic alias like
gpt-4 resolves to the dated snapshot gpt-4-0613 in the
response).
Hide child attributes
Hide child attributes
Available options:
stop, length Allowed value:
"chat.completion"Hide child attributes
Hide child attributes