> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime Frames

> WS /v1/ws: send frames to a model, get its reply per frame

`WS /v1/ws` reads every frame you send with one model and method, and returns the model's reply as a `result` event. The reply is what [Chat Completions](https://docs.vlm.run/gateway/api-reference/post-chat-completions.md) returns for the same image. You name the model and method once, in `session.create`, then send each frame as one binary message of raw image bytes.

The session reads one frame at a time by default. A frame that arrives while a read is outstanding is dropped with `reason: "busy"`, and a frame that arrives faster than the model's `max_fps` is dropped with `reason: "rate"`. The lifecycle, close codes, and models are on [Realtime](https://docs.vlm.run/gateway/realtime.md).

## Example

```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
pip install websockets   # Python
npm install ws           # Node.js
```

```python Python [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import asyncio
import json
import urllib.request

import websockets

URL = "wss://gateway.vlm.run/v1/ws"
IMAGE_URL = "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg"


async def main(frames: list[bytes]) -> None:
    async with websockets.connect(
        URL, additional_headers={"Authorization": "Bearer <VLMRUN_API_KEY>"}
    ) as ws:
        await ws.send(json.dumps({
            "type": "session.create",
            "model": "usyd-community/vitpose-plus-large",
            "method": "pose",
        }))
        created = json.loads(await ws.recv())
        if created["type"] == "error":
            raise RuntimeError(created["message"])
        print(created["session_id"], created["max_fps"], created["max_inflight"])

        async def send_frames() -> None:
            for frame in frames:          # raw JPEG/PNG/WebP/GIF bytes, no base64
                await ws.send(frame)
                await asyncio.sleep(1 / created["max_fps"])

        sender = asyncio.create_task(send_frames())
        for _ in frames:                  # one result, frame.dropped, or error per frame
            event = json.loads(await ws.recv())
            if event["type"] == "result":
                pose = json.loads(event["response"]["choices"][0]["message"]["content"])
                print(event["frame"], len(pose["items"]), "people", round(event["latency_ms"]), "ms")
            elif event["type"] == "frame.dropped":
                print(event["frame"], "dropped:", event["reason"])
            elif event["type"] == "error":
                print(event.get("frame"), event["error_type"], event["message"])
        await sender

        await ws.send(json.dumps({"type": "session.close"}))
        print(json.loads(await ws.recv()))   # session.stats


image = urllib.request.urlopen(IMAGE_URL).read()
asyncio.run(main([image] * 20))
```

```typescript Node.js [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import WebSocket from "ws";

const URL = "wss://gateway.vlm.run/v1/ws";
const IMAGE_URL =
  "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg";

const image = Buffer.from(await (await fetch(IMAGE_URL)).arrayBuffer());
const frames = Array.from({ length: 20 }, () => image);   // raw bytes, no base64

const ws = new WebSocket(URL, {
  headers: { Authorization: `Bearer ${process.env.VLMRUN_API_KEY}` },
});

let answered = 0;

ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "session.create",
    model: "usyd-community/vitpose-plus-large",
    method: "pose",
  }));
});

ws.on("message", async (data) => {
  const event = JSON.parse(data.toString());

  switch (event.type) {
    case "session.created":
      console.log(event.session_id, event.max_fps, event.max_inflight);
      for (const frame of frames) {
        ws.send(frame);
        await new Promise((resolve) => setTimeout(resolve, 1000 / event.max_fps));
      }
      return;
    case "result": {
      const pose = JSON.parse(event.response.choices[0].message.content);
      console.log(event.frame, pose.items.length, "people", Math.round(event.latency_ms), "ms");
      break;
    }
    case "frame.dropped":
      console.log(event.frame, "dropped:", event.reason);
      break;
    case "error":
      console.log(event.frame, event.error_type, event.message);
      break;
    case "session.stats":
      console.log(event);
      ws.close(1000);
      return;
  }
  if (++answered === frames.length) ws.send(JSON.stringify({ type: "session.close" }));
});

ws.on("close", (code, reason) => console.log("closed", code, reason.toString()));
```

The sample opens a `pose` session on `usyd-community/vitpose-plus-large`, sends one photo 20 times at the model's `max_fps`, and prints each `result`, `frame.dropped`, and `error`. It then sends `session.close` and prints `session.stats`.

## The result event

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
  "type": "result",
  "frame": 1,
  "latency_ms": 371.0251930169761,
  "response": {
    "id": "chatcmpl-9a23c119537542379570a50c64a",
    "object": "chat.completion",
    "created": 1793493630,
    "model": "usyd-community/vitpose-plus-large",
    "served_model_id": "usyd-community/vitpose-plus-large",
    "backend": "vitpose-plus-large",
    "provider": null,
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "{\"object\": \"img.pose.kpts\", \"items\": [{\"bbox_xywh\": [0.2313, 0.2849, 0.1079, 0.3001], \"label\": \"person\", \"kpts_xy\": [[0.2832, 0.3212], [0.2876, 0.3154], ...], \"kpts_score\": [0.9728, 0.9839, ...]}, ...], \"kpts_labels\": [\"nose\", \"left_eye\", ...]}"
        },
        "finish_reason": "stop"
      }
    ],
    "usage": {
      "prompt_tokens": 0,
      "completion_tokens": 0,
      "total_tokens": 0,
      "prompt_tokens_details": { "cached_tokens": 0 },
      "completion_tokens_details": null,
      "cost": null
    }
  }
}
```

`content` is the `pose` method's output as a JSON string: an [`img.pose.kpts`](https://docs.vlm.run/gateway/response-types#img-pose-kpts) container with one item per person. `response.usage.cost` is `null`. The socket does not bill per frame: it bills the session by the second, as `session.stats.cost`.

## Session stats

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
  "type": "session.stats",
  "session_id": "fe8d2c58a6e2",
  "id": "ws-b675e9281d2341399b24fb9803329534",
  "reads": 5,
  "dropped": 15,
  "errors": 0,
  "cost": 0.001527,
  "p50_ms": 249.01388795115054,
  "p95_ms": 371.0251930169761,
  "fps": 2.727938173146203
}
```

Of the 20 frames, 5 were read and 15 arrived while a read was outstanding. The session was open about 1.8 seconds, which is $0.0015 at $3 / hour. The frame count does not change the price. Read the row back with [`GET /v1/completions/{id}`](https://docs.vlm.run/gateway/api-reference/get-completion).

## Read more of the frames

> [!WARNING]
>
> **Backpressure drops. It never queues.** A queued frame would answer for a moment you have already moved past. Each dropped frame is reported, so you can count what you skipped.

* **Raise `max_inflight`.** Set it in `session.create`, up to 8. The session keeps that many reads outstanding, and a slot frees when its read returns.
* **Pace at `max_fps`.** `session.created` returns the model's rate. Sending faster only adds `rate` drops.
* **Close the session when you stop.** A session bills every second it is open, with or without frames. Send `session.close` instead of leaving it to the 30 second idle close.

## Pick the model and method

`session.create` takes the same `method` and `method_params` as a Chat Completions request. See [Methods](https://docs.vlm.run/gateway/methods.md). `prompt` is optional text that goes with every frame.

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{ "type": "session.create", "model": "geopavlakos/hamer", "method": "pose" }
```

`geopavlakos/hamer` returns one [`world.pose.kpts`](https://docs.vlm.run/gateway/response-types#world-pose-kpts) item per hand, with 2D and 3D joints. A model without realtime support is refused at `session.create`. The models that stream on this socket are listed in [Supported models](https://docs.vlm.run/gateway/realtime.md#supported-models).

## Errors

An `error` event with a `frame` applies to that frame only, and the session stays open. A frame that is not JPEG, PNG, WebP, or GIF, or is larger than `max_frame_bytes`, returns `invalid_request_error`. A model failure returns `overloaded_error` or `api_error` with `retry_after`.

An `error` at open, or when the session reaches its spend ceiling, ends the session, and a close follows. The error types are in the [reference](https://docs.vlm.run/gateway/api-reference/ws-realtime.md#errors). What to do after each close code is in [Close codes](https://docs.vlm.run/gateway/realtime.md#close-codes).

## Limits and cost

A session lasts up to 10 minutes, closes after 30 seconds idle, and stops at \$5 of spend. Frames are capped at 2 MiB. Sessions per plan are on [Rate Limits](https://docs.vlm.run/gateway/rate-limits.md#websocket-sessions), and the price of each model on the socket is in [Supported models](https://docs.vlm.run/gateway/realtime.md#supported-models).

## Next steps

> [card] [WS /v1/ws reference](https://docs.vlm.run/gateway/api-reference/ws-realtime.md)
>
> Every client and server event, with its fields.

> [card] [Response Types](https://docs.vlm.run/gateway/response-types.md#img-pose-kpts)
>
> The keypoint payload each `result` carries.

> [card] [Realtime System One](https://docs.vlm.run/gateway/realtime-systemone.md)
>
> Typed decisions per frame instead of a model reply.

> [card] [Rate Limits](https://docs.vlm.run/gateway/rate-limits.md#websocket-sessions)
>
> Sessions per plan and per-session bounds.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.