> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.

# Video over WebSocket

> Stream the frames of a video to a model over one socket and read a reply per frame

Send every frame of a video to a model over one WebSocket and read one reply per frame. This guide counts the people in each frame of a 20 s clip with `usyd-community/vitpose-plus-large`. A session bills every second it is open, at \$3 / hour, whatever the frame count.

## 1. Pick the socket

| You want | Socket | Reply per frame |
| - | - | - |
| Boxes, keypoints, masks, or text lines | `wss://gateway.vlm.run/v1/ws` | The model's reply, as [Chat Completions](https://docs.vlm.run/gateway/api-reference/post-chat-completions.md) returns it |
| A yes/no, a label, or a score you can threshold | `wss://gateway.vlm.run/v1/ws/systemone` | A System One decision. See [System One Decisions](https://docs.vlm.run/gateway/guides/systemone-decisions.md). |

This guide uses `/v1/ws`. The models it serves, and their methods, are in [Supported models](https://docs.vlm.run/gateway/realtime.md#supported-models).

## 2. Cut the video into frames

Each frame is one binary message of raw JPEG, PNG, WebP, or GIF bytes, at most 2 MiB. Cut the video at the rate you want to read, then send the files as they are.

```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -sO https://storage.googleapis.com/vlm-data-public-prod/web/videos/video-trim-20s.mp4
mkdir -p frames
ffmpeg -i video-trim-20s.mp4 -vf "fps=4,scale=640:-2" -q:v 3 frames/%04d.jpg
```

That writes 80 JPEG frames of 640 x 360 pixels, about 30 KB each. The frame count does not change the price. A lower rate leaves more read slots free for the frames you send.

## 3. Stream the frames

```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
pip install websockets   # Python
npm install ws           # Node.js
```

```python Python [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import asyncio
import json
import pathlib

import websockets

URL = "wss://gateway.vlm.run/v1/ws"
FPS = 4

frames = [p.read_bytes() for p in sorted(pathlib.Path("frames").glob("*.jpg"))]


async def main() -> None:
    async with websockets.connect(
        URL, additional_headers={"Authorization": "Bearer <VLMRUN_API_KEY>"}
    ) as ws:
        await ws.send(json.dumps({
            "type": "session.create",
            "model": "usyd-community/vitpose-plus-large",
            "method": "pose",
            "max_inflight": 4,
        }))
        created = json.loads(await ws.recv())
        if created["type"] == "error":
            raise RuntimeError(created["message"])

        async def send_frames() -> None:
            for frame in frames:
                await ws.send(frame)
                await asyncio.sleep(1 / FPS)

        sender = asyncio.create_task(send_frames())
        people = {}
        for _ in frames:
            event = json.loads(await ws.recv())
            if event["type"] == "result":
                pose = json.loads(event["response"]["choices"][0]["message"]["content"])
                people[event["frame"]] = len(pose["items"])
            elif event["type"] == "frame.dropped":
                print("dropped", event["frame"], event["reason"])
            elif event["type"] == "error":
                print("error", event.get("frame"), event["error_type"], event["message"])
        await sender

        await ws.send(json.dumps({"type": "session.close"}))
        stats = json.loads(await ws.recv())

    for frame, count in sorted(people.items()):
        print(f"{(frame - 1) / FPS:5.2f} s  people={count}")
    print(stats)


asyncio.run(main())
```

```typescript Node.js [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { readdirSync, readFileSync } from "node:fs";
import WebSocket from "ws";

const URL = "wss://gateway.vlm.run/v1/ws";
const FPS = 4;

const frames = readdirSync("frames")
  .filter((name) => name.endsWith(".jpg"))
  .sort()
  .map((name) => readFileSync(`frames/${name}`));

const ws = new WebSocket(URL, {
  headers: { Authorization: `Bearer ${process.env.VLMRUN_API_KEY}` },
});

const people = new Map();
let answered = 0;

ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "session.create",
    model: "usyd-community/vitpose-plus-large",
    method: "pose",
    max_inflight: 4,
  }));
});

ws.on("message", async (data) => {
  const event = JSON.parse(data.toString());

  switch (event.type) {
    case "session.created":
      for (const frame of frames) {
        ws.send(frame);
        await new Promise((resolve) => setTimeout(resolve, 1000 / FPS));
      }
      return;
    case "result": {
      const pose = JSON.parse(event.response.choices[0].message.content);
      people.set(event.frame, pose.items.length);
      break;
    }
    case "frame.dropped":
      console.log("dropped", event.frame, event.reason);
      break;
    case "error":
      if (event.frame == null) throw new Error(event.message);
      console.log("error", event.frame, event.error_type, event.message);
      break;
    case "session.stats":
      for (const [frame, count] of [...people].sort((a, b) => a[0] - b[0])) {
        console.log(`${((frame - 1) / FPS).toFixed(2).padStart(5)} s  people=${count}`);
      }
      console.log(event);
      ws.close(1000);
      return;
  }
  if (++answered === frames.length) ws.send(JSON.stringify({ type: "session.close" }));
});
```

The sample opens one `pose` session, sends the 80 frames at 4 per second, and keeps the person count of each `result`. Every frame gets exactly one event back: a `result`, a `frame.dropped`, or an `error`. Frames are numbered from 1, in the order you send them.

```text Output theme={"theme":{"light":"github-light","dark":"dark-plus"}}
 0.00 s  people=3
 0.25 s  people=3
 0.50 s  people=5
...
10.75 s  people=1
11.00 s  people=6
11.25 s  people=10
11.50 s  people=9
11.75 s  people=9
12.00 s  people=11
12.25 s  people=9
12.50 s  people=10
12.75 s  people=2
...
19.75 s  people=1
{'type': 'session.stats', 'session_id': '...', 'id': 'ws-...', 'reads': 80, 'dropped': 0, 'errors': 0, 'cost': 0.016875, 'p50_ms': 33.84, 'p95_ms': 86.36, 'fps': 3.95}
```

The count peaks at 11 people at 12.00 s, inside the bakery. All 80 frames were read, with a median read of 34 ms. The session was open about 20 s, which bills $0.0169 at $3 / hour.

## 4. Read the result event

`content` is the method's output as a JSON string, the same string Chat Completions returns for one image. For `pose`, it is an [`img.pose.kpts`](https://docs.vlm.run/gateway/response-types#img-pose-kpts) container with one item per person: a box, 17 keypoints, and a score per keypoint.

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
  "type": "result",
  "frame": 1,
  "latency_ms": 98.5,
  "response": {
    "model": "usyd-community/vitpose-plus-large",
    "choices": [
      {
        "message": {
          "role": "assistant",
          "content": "{\"object\": \"img.pose.kpts\", \"items\": [{\"bbox_xywh\": [0.2307, 0.086, 0.6071, 0.9053], \"label\": \"person\", \"kpts_xy\": [[0.4686, 0.2665], ...], \"kpts_score\": [0.9659, ...]}], \"kpts_labels\": [\"nose\", \"left_eye\", ...]}"
        }
      }
    ],
    "usage": { "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0, "cost": null }
  }
}
```

Coordinates are fractions of the frame width and height, from 0 to 1, so they do not depend on the frame size you send. `usage.cost` is `null` on every frame. The session bills as a whole, and `session.stats.cost` is its total. Every field of every event is in the [`WS /v1/ws` reference](https://docs.vlm.run/gateway/api-reference/ws-realtime).

To read other output, change `model` and `method` in `session.create`:

| Task | `session.create` | `content` per frame |
| - | - | - |
| Body pose | `{"model": "usyd-community/vitpose-plus-large", "method": "pose"}` | [`img.pose.kpts`](https://docs.vlm.run/gateway/response-types#img-pose-kpts) |
| Hand pose | `{"model": "geopavlakos/hamer", "method": "pose"}` | [`world.pose.kpts`](https://docs.vlm.run/gateway/response-types#world-pose-kpts) |
| Masks by prompt | `{"model": "facebook/sam3.1", "method": "segment", "method_params": {"prompt": "person"}}` | [`img.segment.masks`](https://docs.vlm.run/gateway/response-types#img-segment-masks) |
| Text lines | `{"model": "paddleocr/pp-ocrv6", "method": "ocr"}` | [`doc.ocr.lines`](https://docs.vlm.run/gateway/response-types#doc-ocr-lines) |

## 5. Keep up with the stream

The session drops a frame it cannot read now. It never queues one, because a queued frame answers for a moment the video has moved past. A dropped frame gets a `frame.dropped` event with a `reason`, so you can count what you skipped.

| `reason` | Cause | Fix |
| - | - | - |
| `busy` | `max_inflight` reads were outstanding when the frame arrived | Raise `max_inflight`, up to 8, or send fewer frames |
| `rate` | The frame came sooner than the model's `max_fps` allows | Pace frames at `session.created.max_fps` or below |

A slow read shows up as drops, not as a backlog. In one run of this sample, one read took 12.5 s: 12 of the 80 frames were dropped, and the other 68 were read.

## 6. Close the session

Send `session.close` when the last reply arrives. Reads in flight finish, then the Gateway sends `session.stats` and waits up to 5 seconds for you to close the socket. A session left open bills its idle seconds until the 30 second idle close.

A session lasts up to 10 minutes and stops at \$5 of spend. For a longer video, open a new session per 10 minutes of stream. Sessions per plan are on [Rate Limits](https://docs.vlm.run/gateway/rate-limits.md#websocket-sessions), and the close codes are on [Realtime](https://docs.vlm.run/gateway/realtime.md#close-codes).

## Related

> [card] [System One Decisions](https://docs.vlm.run/gateway/guides/systemone-decisions.md)
>
> Ask the same questions of every frame and act on the probability.

> [card] [Realtime](https://docs.vlm.run/gateway/realtime.md)
>
> Session lifecycle, close codes, supported models, and billing.

> [card] [WS /v1/ws reference](https://docs.vlm.run/gateway/api-reference/ws-realtime.md)
>
> Every client and server event, with its fields.

> [card] [Response Types](https://docs.vlm.run/gateway/response-types.md)
>
> The payload each method returns.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.