Realtime System One

WS /v1/ws/systemone answers one set of System One questions for every frame you send. A reply is one decision. It carries the same answers and usage as a posted read. The lifecycle, close codes, and models are on Realtime.

	Posted read	Streamed read
Route	`POST /typesafe/v1/systemone`	`WS /v1/ws/systemone`
Media	Content parts, base64 in JSON	One binary message per frame, raw bytes
Questions	Sent every request	Sent once, at `session.create`
Schema work	Per request	Once per session

WS /typesafe/ws is a deprecated alias of this socket, with the same protocol and bounds. The vlmrun SDK’s transport="ws" and the CLI’s --ws still connect to it. New raw-socket clients use /v1/ws/systemone.

Example

pip install "vlmrun[typesafe,video,ws]"

VLMRun(...) defaults to https://gateway.vlm.run/v1. Set VLMRUN_GATEWAY_URL for another gateway. The SDK derives the socket URL from it: https://gw.internal/v1 becomes wss://gw.internal/typesafe/ws.

Do not pass the Gateway URL as VLMRun(base_url=...). That argument is the platform API. The client health-checks it on construction. A Gateway URL there fails before a read is sent. Use VLMRUN_GATEWAY_URL for this route.

from vlmrun.client import VLMRun
from vlmrun.common.video import VideoReader

client = VLMRun(api_key="<VLMRUN_API_KEY>")

QUESTIONS = [{"id": "has_people", "type": "noul", "instructions": "Are people visible?"}]

with client.gateway.systemone.stream(
    questions=QUESTIONS,
    state="Answer about the frame.",
    model="google/diffusiongemma-26b-a4b-it",
    detail="low",
    concurrency=8,          # requested as the session's max_inflight
    transport="ws",         # one socket; "http" sends a request per frame
) as stream, VideoReader("clip.mp4") as video:
    for decision in stream.map(video.frames(fps=1)):
        print(decision.timestamp_s, decision.response.nouls["has_people"].noul)

vlmrun gw s1 clip.mp4 --fps 1 --ws -c 8 \
  -m google/diffusiongemma-26b-a4b-it --detail low \
  --noul has_people="Are people visible?"

import asyncio, json, pathlib
import websockets

URL = "wss://gateway.vlm.run/v1/ws/systemone"

async def main(frames: list[bytes]) -> None:
    async with websockets.connect(
        URL, additional_headers={"Authorization": "Bearer <VLMRUN_API_KEY>"}
    ) as ws:
        await ws.send(json.dumps({
            "type": "session.create",
            "model": "google/diffusiongemma-26b-a4b-it",
            "state": "Answer about the frame.",
            "questions": {
                "has_people": {"type": "noul", "instructions": "Are people visible?"}
            },
            "detail": "low",
        }))
        created = json.loads(await ws.recv())
        print(created["session_id"], created["max_frame_bytes"])

        for frame in frames:          # raw JPEG/PNG/WebP/GIF bytes, no base64
            await ws.send(frame)

        seen = 0
        while seen < len(frames):
            event = json.loads(await ws.recv())
            if event["type"] == "decision":
                print(event["frame"], event["answers"]["has_people"]["noul"])
                seen += 1
            elif event["type"] == "frame.dropped":
                seen += 1             # arrived with every read slot busy
            elif event["type"] == "error":
                raise RuntimeError(event["message"])

        await ws.send(json.dumps({"type": "session.close"}))
        print(json.loads(await ws.recv()))   # session.stats

asyncio.run(main([pathlib.Path("frame1.jpg").read_bytes()]))

import WebSocket from "ws";
import { readFileSync } from "node:fs";

const URL = "wss://gateway.vlm.run/v1/ws/systemone";
const frames = [readFileSync("frame1.jpg")];   // raw bytes, no base64

const ws = new WebSocket(URL, {
  headers: { Authorization: `Bearer ${process.env.VLMRUN_API_KEY}` },
});

let seen = 0;

ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "session.create",
    model: "google/diffusiongemma-26b-a4b-it",
    state: "Answer about the frame.",
    questions: {
      has_people: { type: "noul", instructions: "Are people visible?" },
    },
    detail: "low",
  }));
});

ws.on("message", (data) => {
  const event = JSON.parse(data.toString());

  if (event.type === "session.created") {
    console.log(event.session_id, event.max_frame_bytes);
    for (const frame of frames) ws.send(frame);
    return;
  }
  if (event.type === "decision") {
    console.log(event.frame, event.answers.has_people.noul);
    seen++;
  } else if (event.type === "frame.dropped") {
    seen++;                                    // every read slot was busy
  } else if (event.type === "error") {
    throw new Error(event.message);
  } else if (event.type === "session.stats") {
    console.log(event);
    ws.close();
    return;
  }

  if (seen === frames.length) ws.send(JSON.stringify({ type: "session.close" }));
});

Each sample opens a session with one noul question, sends frames, and prints the has_people probability per frame. A FrameDecision from the SDK carries index, timestamp_s, and response. Results stay in input order. The raw samples print session.stats after session.close.

Why the socket

The saving is round trips. It grows with frame count. 60 frames from a 60 s clip:

Reads in flight	HTTP	WebSocket
4	2458 ms	1525 ms
8	1969 ms	904 ms

Tokens and cost match HTTP. Ask for the concurrency you want: a session that omits max_inflight reads one frame at a time.

Decision

{
  "type": "decision",
  "frame": 1,
  "answers": { "has_people": { "type": "noul", "noul": 2.8643205662214584e-05 } },
  "usage": {
    "input_tokens": 147,
    "output_tokens": 0,
    "input_tokens_details": { "cached_tokens": 32, "image_tokens": 70, "text_tokens": 77 },
    "reads": 1,
    "cost": 0.001
  },
  "latency_ms": 40.682217018911615,
  "id": null
}

The schema is prefix-cached across frames. Most of each prompt returns as cached_tokens. Those tokens bill at the cache rate. Every decision bills at least $0.001.

Messages

After session.create, each frame is one binary message. A read fires every stride frames over the newest window frames. Two more client events change a running session: state re-pins the state later frames read against, and decide runs one read with its own body. Every event and field is in the WS /v1/ws/systemone reference.

Backpressure

Backpressure drops. It never queues. A frame that arrives while max_inflight reads are running is dropped and reported on frame.dropped. Answering it would answer for a frame you have moved past. A decide message is never dropped.

Raise stride before max_inflight. Reading fewer frames costs less. Consecutive frames rarely change a decision.

Limits and billing

Each decision is one billing row, priced per token as a posted read, never by connection time. Skipped, dropped, and failed frames, and idle time, bill nothing. A session lasts up to 10 minutes, closes after 30 seconds idle, and stops at $5 of spend. Sessions per plan are on Rate Limits.

System One

Question types, inputs, confidence, and cost levers.

WS /v1/ws/systemone reference

Every client and server event, with its fields.

System One API

The HTTP route for the same questions. The socket reuses its answers and usage shapes.

Realtime Frames

A model reply per frame, for pose models.

Overview

Platform

OpenAI Compatibility

TypeSafe Compatibility

Realtime

MCP

Misc

Realtime System One

Example

Why the socket

Messages

Backpressure

Limits and billing

System One

WS /v1/ws/systemone reference

System One API

Realtime Frames

​Example

​Why the socket

​Messages

​Backpressure

​Limits and billing

​Related

System One

WS /v1/ws/systemone reference

System One API

Realtime Frames

Example

Why the socket

Messages

Backpressure

Limits and billing

Related