Realtime Frames

WS /v1/ws reads every frame you send with one model and method, and returns the model’s reply as a result event. The reply is what Chat Completions returns for the same image. You name the model and method once, in session.create, then send each frame as one binary message of raw image bytes. The session reads one frame at a time by default. A frame that arrives while a read is outstanding is dropped with reason: "busy", and a frame that arrives faster than the model’s max_fps is dropped with reason: "rate". The lifecycle, close codes, and models are on Realtime.

Example

pip install websockets   # Python
npm install ws           # Node.js

import asyncio
import json
import urllib.request

import websockets

URL = "wss://gateway.vlm.run/v1/ws"
IMAGE_URL = "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg"


async def main(frames: list[bytes]) -> None:
    async with websockets.connect(
        URL, additional_headers={"Authorization": "Bearer <VLMRUN_API_KEY>"}
    ) as ws:
        await ws.send(json.dumps({
            "type": "session.create",
            "model": "usyd-community/vitpose-plus-large",
            "method": "pose",
        }))
        created = json.loads(await ws.recv())
        if created["type"] == "error":
            raise RuntimeError(created["message"])
        print(created["session_id"], created["max_fps"], created["max_inflight"])

        async def send_frames() -> None:
            for frame in frames:          # raw JPEG/PNG/WebP/GIF bytes, no base64
                await ws.send(frame)
                await asyncio.sleep(1 / created["max_fps"])

        sender = asyncio.create_task(send_frames())
        for _ in frames:                  # one result, frame.dropped, or error per frame
            event = json.loads(await ws.recv())
            if event["type"] == "result":
                pose = json.loads(event["response"]["choices"][0]["message"]["content"])
                print(event["frame"], len(pose["items"]), "people", round(event["latency_ms"]), "ms")
            elif event["type"] == "frame.dropped":
                print(event["frame"], "dropped:", event["reason"])
            elif event["type"] == "error":
                print(event.get("frame"), event["error_type"], event["message"])
        await sender

        await ws.send(json.dumps({"type": "session.close"}))
        print(json.loads(await ws.recv()))   # session.stats


image = urllib.request.urlopen(IMAGE_URL).read()
asyncio.run(main([image] * 20))

import WebSocket from "ws";

const URL = "wss://gateway.vlm.run/v1/ws";
const IMAGE_URL =
  "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg";

const image = Buffer.from(await (await fetch(IMAGE_URL)).arrayBuffer());
const frames = Array.from({ length: 20 }, () => image);   // raw bytes, no base64

const ws = new WebSocket(URL, {
  headers: { Authorization: `Bearer ${process.env.VLMRUN_API_KEY}` },
});

let answered = 0;

ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "session.create",
    model: "usyd-community/vitpose-plus-large",
    method: "pose",
  }));
});

ws.on("message", async (data) => {
  const event = JSON.parse(data.toString());

  switch (event.type) {
    case "session.created":
      console.log(event.session_id, event.max_fps, event.max_inflight);
      for (const frame of frames) {
        ws.send(frame);
        await new Promise((resolve) => setTimeout(resolve, 1000 / event.max_fps));
      }
      return;
    case "result": {
      const pose = JSON.parse(event.response.choices[0].message.content);
      console.log(event.frame, pose.items.length, "people", Math.round(event.latency_ms), "ms");
      break;
    }
    case "frame.dropped":
      console.log(event.frame, "dropped:", event.reason);
      break;
    case "error":
      console.log(event.frame, event.error_type, event.message);
      break;
    case "session.stats":
      console.log(event);
      ws.close(1000);
      return;
  }
  if (++answered === frames.length) ws.send(JSON.stringify({ type: "session.close" }));
});

ws.on("close", (code, reason) => console.log("closed", code, reason.toString()));

The sample opens a pose session on usyd-community/vitpose-plus-large, sends one photo 20 times at the model’s max_fps, and prints each result, frame.dropped, and error. It then sends session.close and prints session.stats.

The result event

{
  "type": "result",
  "frame": 1,
  "latency_ms": 371.0251930169761,
  "response": {
    "id": "chatcmpl-9a23c119537542379570a50c64a",
    "object": "chat.completion",
    "created": 1793493630,
    "model": "usyd-community/vitpose-plus-large",
    "served_model_id": "usyd-community/vitpose-plus-large",
    "backend": "vitpose-plus-large",
    "provider": null,
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "{\"object\": \"img.pose.kpts\", \"items\": [{\"bbox_xywh\": [0.2313, 0.2849, 0.1079, 0.3001], \"label\": \"person\", \"kpts_xy\": [[0.2832, 0.3212], [0.2876, 0.3154], ...], \"kpts_score\": [0.9728, 0.9839, ...]}, ...], \"kpts_labels\": [\"nose\", \"left_eye\", ...]}"
        },
        "finish_reason": "stop"
      }
    ],
    "usage": {
      "prompt_tokens": 0,
      "completion_tokens": 0,
      "total_tokens": 0,
      "prompt_tokens_details": { "cached_tokens": 0 },
      "completion_tokens_details": null,
      "cost": null
    }
  }
}

content is the pose method’s output as a JSON string: an img.pose.kpts container with one item per person. response.usage.cost is null. The socket does not bill per frame: it bills the session by the second, as session.stats.cost.

Session stats

{
  "type": "session.stats",
  "session_id": "fe8d2c58a6e2",
  "id": "ws-b675e9281d2341399b24fb9803329534",
  "reads": 5,
  "dropped": 15,
  "errors": 0,
  "cost": 0.001527,
  "p50_ms": 249.01388795115054,
  "p95_ms": 371.0251930169761,
  "fps": 2.727938173146203
}

Of the 20 frames, 5 were read and 15 arrived while a read was outstanding. The session was open about 1.8 seconds, which is $0.0015 at $3 / hour. The frame count does not change the price. Read the row back with GET /v1/completions/{id}.

Pick the model and method

session.create takes the same method and method_params as a Chat Completions request. See Methods. prompt is optional text that goes with every frame.

{ "type": "session.create", "model": "geopavlakos/hamer", "method": "pose" }

geopavlakos/hamer returns one world.pose.kpts item per hand, with 2D and 3D joints. A model without realtime support is refused at session.create. The models that stream on this socket are listed in Supported models.

Errors

An error event with a frame applies to that frame only, and the session stays open. A frame that is not JPEG, PNG, WebP, or GIF, or is larger than max_frame_bytes, returns invalid_request_error. A model failure returns overloaded_error or api_error with retry_after. An error at open, or when the session reaches its spend ceiling, ends the session, and a close follows. The error types are in the reference. What to do after each close code is in Close codes.

Limits and cost

A session lasts up to 10 minutes, closes after 30 seconds idle, and stops at $5 of spend. Frames are capped at 2 MiB. Sessions per plan are on Rate Limits, and the price of each model on the socket is in Supported models.

Next steps

WS /v1/ws reference

Every client and server event, with its fields.

Response Types

The keypoint payload each result carries.

Realtime System One

Typed decisions per frame instead of a model reply.

Rate Limits

Sessions per plan and per-session bounds.

Overview

Platform

OpenAI Compatibility

TypeSafe Compatibility

Realtime

MCP

Misc

Realtime Frames

Example

The result event

Session stats

Read more of the frames

Pick the model and method

Errors

Limits and cost

Next steps

WS /v1/ws reference

Response Types

Realtime System One

Rate Limits

​Example

​The result event

​Session stats

​Read more of the frames

​Pick the model and method

​Errors

​Limits and cost

​Next steps

WS /v1/ws reference

Response Types

Realtime System One

Rate Limits

Example

The result event

Session stats

Read more of the frames

Pick the model and method

Errors

Limits and cost

Next steps