> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime System One

> WS /v1/ws/systemone: typed System One decisions per frame, over one WebSocket

`WS /v1/ws/systemone` answers one set of [System One](https://docs.vlm.run/gateway/system-one.md) questions for every frame you send. A reply is one `decision`. It carries the same `answers` and `usage` as a posted read. The lifecycle, close codes, and models are on [Realtime](https://docs.vlm.run/gateway/realtime.md).

| | Posted read | Streamed read |
| - | - | - |
| Route | `POST /typesafe/v1/systemone` | `WS /v1/ws/systemone` |
| Media | Content parts, base64 in JSON | One binary message per frame, raw bytes |
| Questions | Sent every request | Sent once, at `session.create` |
| Schema work | Per request | Once per session |

> [!NOTE]
>
> `WS /typesafe/ws` is a deprecated alias of this socket, with the same protocol and bounds. The `vlmrun` SDK's `transport="ws"` and the CLI's `--ws` still connect to it. New raw-socket clients use `/v1/ws/systemone`.

## Example

```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
pip install "vlmrun[typesafe,video,ws]"
```

`VLMRun(...)` defaults to `https://gateway.vlm.run/v1`. Set `VLMRUN_GATEWAY_URL` for another gateway. The SDK derives the socket URL from it: `https://gw.internal/v1` becomes `wss://gw.internal/typesafe/ws`.

> [!WARNING]
>
> Do not pass the Gateway URL as `VLMRun(base_url=...)`. That argument is the platform API. The client health-checks it on construction. A Gateway URL there fails before a read is sent. Use `VLMRUN_GATEWAY_URL` for this route.

```python Python [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.common.video import VideoReader

client = VLMRun(api_key="<VLMRUN_API_KEY>")

QUESTIONS = [{"id": "has_people", "type": "noul", "instructions": "Are people visible?"}]

with client.gateway.systemone.stream(
    questions=QUESTIONS,
    state="Answer about the frame.",
    model="google/diffusiongemma-26b-a4b-it",
    detail="low",
    concurrency=8,          # requested as the session's max_inflight
    transport="ws",         # one socket; "http" sends a request per frame
) as stream, VideoReader("clip.mp4") as video:
    for decision in stream.map(video.frames(fps=1)):
        print(decision.timestamp_s, decision.response.nouls["has_people"].noul)
```

```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw s1 clip.mp4 --fps 1 --ws -c 8 \
  -m google/diffusiongemma-26b-a4b-it --detail low \
  --noul has_people="Are people visible?"
```

```python Python (websockets) [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import asyncio, json, pathlib
import websockets

URL = "wss://gateway.vlm.run/v1/ws/systemone"

async def main(frames: list[bytes]) -> None:
    async with websockets.connect(
        URL, additional_headers={"Authorization": "Bearer <VLMRUN_API_KEY>"}
    ) as ws:
        await ws.send(json.dumps({
            "type": "session.create",
            "model": "google/diffusiongemma-26b-a4b-it",
            "state": "Answer about the frame.",
            "questions": {
                "has_people": {"type": "noul", "instructions": "Are people visible?"}
            },
            "detail": "low",
        }))
        created = json.loads(await ws.recv())
        print(created["session_id"], created["max_frame_bytes"])

        for frame in frames:          # raw JPEG/PNG/WebP/GIF bytes, no base64
            await ws.send(frame)

        seen = 0
        while seen < len(frames):
            event = json.loads(await ws.recv())
            if event["type"] == "decision":
                print(event["frame"], event["answers"]["has_people"]["noul"])
                seen += 1
            elif event["type"] == "frame.dropped":
                seen += 1             # arrived with every read slot busy
            elif event["type"] == "error":
                raise RuntimeError(event["message"])

        await ws.send(json.dumps({"type": "session.close"}))
        print(json.loads(await ws.recv()))   # session.stats

asyncio.run(main([pathlib.Path("frame1.jpg").read_bytes()]))
```

```typescript Node.js [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import WebSocket from "ws";
import { readFileSync } from "node:fs";

const URL = "wss://gateway.vlm.run/v1/ws/systemone";
const frames = [readFileSync("frame1.jpg")];   // raw bytes, no base64

const ws = new WebSocket(URL, {
  headers: { Authorization: `Bearer ${process.env.VLMRUN_API_KEY}` },
});

let seen = 0;

ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "session.create",
    model: "google/diffusiongemma-26b-a4b-it",
    state: "Answer about the frame.",
    questions: {
      has_people: { type: "noul", instructions: "Are people visible?" },
    },
    detail: "low",
  }));
});

ws.on("message", (data) => {
  const event = JSON.parse(data.toString());

  if (event.type === "session.created") {
    console.log(event.session_id, event.max_frame_bytes);
    for (const frame of frames) ws.send(frame);
    return;
  }
  if (event.type === "decision") {
    console.log(event.frame, event.answers.has_people.noul);
    seen++;
  } else if (event.type === "frame.dropped") {
    seen++;                                    // every read slot was busy
  } else if (event.type === "error") {
    throw new Error(event.message);
  } else if (event.type === "session.stats") {
    console.log(event);
    ws.close();
    return;
  }

  if (seen === frames.length) ws.send(JSON.stringify({ type: "session.close" }));
});
```

Each sample opens a session with one `noul` question, sends frames, and prints the `has_people` probability per frame. A `FrameDecision` from the SDK carries `index`, `timestamp_s`, and `response`. Results stay in input order. The raw samples print `session.stats` after `session.close`.

### Why the socket

The saving is round trips. It grows with frame count. 60 frames from a 60 s clip:

| Reads in flight | HTTP | WebSocket |
| - | - | - |
| 4 | 2458 ms | 1525 ms |
| 8 | 1969 ms | **904 ms** |

Tokens and cost match HTTP. Ask for the concurrency you want: a session that omits `max_inflight` reads one frame at a time.

```json Decision theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
  "type": "decision",
  "frame": 1,
  "answers": { "has_people": { "type": "noul", "noul": 2.8643205662214584e-05 } },
  "usage": {
    "input_tokens": 147,
    "output_tokens": 0,
    "input_tokens_details": { "cached_tokens": 32, "image_tokens": 70, "text_tokens": 77 },
    "reads": 1,
    "cost": 0.001
  },
  "latency_ms": 40.682217018911615,
  "id": null
}
```

The schema is prefix-cached across frames. Most of each prompt returns as `cached_tokens`. Those tokens bill at the cache rate. Every decision bills at least `$0.001`.

## Messages

After `session.create`, each frame is one binary message. A read fires every `stride` frames over the newest `window` frames. Two more client events change a running session: `state` re-pins the state later frames read against, and `decide` runs one read with its own body. Every event and field is in the [`WS /v1/ws/systemone` reference](https://docs.vlm.run/gateway/api-reference/ws-systemone).

## Backpressure

> [!WARNING]
>
> **Backpressure drops. It never queues.** A frame that arrives while `max_inflight` reads are running is dropped and reported on `frame.dropped`. Answering it would answer for a frame you have moved past. A `decide` message is never dropped.

Raise `stride` before `max_inflight`. Reading fewer frames costs less. Consecutive frames rarely change a decision.

## Limits and billing

Each decision is one billing row, priced per token as a posted read, never by connection time. Skipped, dropped, and failed frames, and idle time, bill nothing. A session lasts up to 10 minutes, closes after 30 seconds idle, and stops at \$5 of spend. Sessions per plan are on [Rate Limits](https://docs.vlm.run/gateway/rate-limits.md#websocket-sessions).

## Related

> [card] [System One](https://docs.vlm.run/gateway/system-one.md)
>
> Question types, inputs, confidence, and cost levers.

> [card] [WS /v1/ws/systemone reference](https://docs.vlm.run/gateway/api-reference/ws-systemone.md)
>
> Every client and server event, with its fields.

> [card] [System One API](https://docs.vlm.run/gateway/api-reference/post-systemone.md)
>
> The HTTP route for the same questions. The socket reuses its `answers` and `usage` shapes.

> [card] [Realtime Frames](https://docs.vlm.run/gateway/realtime-frames.md)
>
> A model reply per frame, for pose models.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.