> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime Frames

> WS /v1/ws: a model reply per frame, over one WebSocket

Streams frames to one model and method and returns the model's reply per frame. For a walkthrough and samples, see [Realtime Frames](https://docs.vlm.run/gateway/realtime-frames.md). For the lifecycle, close codes, and models, see [Realtime](https://docs.vlm.run/gateway/realtime.md). The socket is not in the OpenAPI schema, because the schema does not describe WebSocket routes.

## Connection

| | |
| - | - |
| URL | `wss://gateway.vlm.run/v1/ws` |
| Auth | `Authorization: Bearer <VLMRUN_API_KEY>` on the upgrade request |
| Anonymous | No key, or `Bearer vlmrun`: `401` at the upgrade, with `{"detail": "WebSocket sessions need an API key."}` |
| Events | JSON text messages, each with a `type` |
| Frames | Binary messages |
| First event | `session.create`, within 30 seconds of the upgrade |

## Client events

### session.create

Opens the session. The Gateway checks the model, method, access, credit, and capacity once, here. An unknown field is refused.

| Field | Type | Required | Description |
| - | - | - | - |
| `type` | `"session.create"` | Yes | |
| `model` | `string` | Yes | A model id or alias with realtime support. See [Supported models](https://docs.vlm.run/gateway/realtime.md#supported-models). |
| `method` | `string` | No | A method the model lists, for example `pose`. Default: the model's default method. See [Methods](https://docs.vlm.run/gateway/methods.md). |
| `method_params` | `object` | No | Parameters for the method, as on Chat Completions. |
| `prompt` | `string` | No | Text sent with every frame. Default: none. |
| `max_inflight` | `integer` | No | Reads outstanding at once, 1 to 8. Default `1`. |

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{ "type": "session.create", "model": "usyd-community/vitpose-plus-large", "method": "pose" }
```

### Frame (binary message)

One frame: raw JPEG, PNG, WebP, or GIF bytes, with no base64 and no JSON wrapper. A frame is at most `max_frame_bytes`. Frames are numbered from 1 in the order they arrive, dropped frames included. Each frame gets exactly one event back with its number: `result`, `frame.dropped`, or `error`.

### session.close

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{ "type": "session.close" }
```

Reads in flight finish and return their events. The server then sends `session.stats` and closes with `1000`. Any other event after `session.create` returns `invalid_request_error`, and the session stays open.

## Server events

### session.created

| Field | Type | Description |
| - | - | - |
| `session_id` | `string` | The session's id. |
| `model` | `string` | The canonical model id, also when you sent an alias. |
| `method` | `string` \| `null` | The method you sent. `null` means the model's default method. |
| `max_fps` | `number` | Reads per second the model allows, on average. A frame sooner than that is dropped with `rate`. |
| `max_inflight` | `integer` | Reads outstanding at once. |
| `max_frame_bytes` | `integer` | The largest frame, in bytes: `2097152` (2 MiB). |
| `max_cost` | `number` | The session's spend ceiling, in USD. |
| `expires_in` | `number` | Seconds the session may last, from the upgrade. |

### result

| Field | Type | Description |
| - | - | - |
| `frame` | `integer` | The frame this reply is for. |
| `latency_ms` | `number` | Time the read took, in milliseconds. |
| `response` | `object` | What [`POST /v1/openai/chat/completions`](https://docs.vlm.run/gateway/api-reference/post-chat-completions) returns for this image. `choices[0].message.content` holds the method's output as a JSON string. |

`response.usage.cost` is `null`. The socket does not bill per frame: the session bills by the second, as `session.stats.cost`.

### frame.dropped

| Field | Type | Description |
| - | - | - |
| `frame` | `integer` | The frame that was not read. |
| `inflight` | `integer` | Reads outstanding when it arrived. |
| `reason` | `"rate"` \| `"busy"` | `rate`: it arrived sooner than `max_fps` allows after the last read. `busy`: every read slot was taken. |

A dropped frame is not read. It does not change the session's price.

### error

| Field | Type | Description |
| - | - | - |
| `error_type` | `string` | One of the [error types](#errors). |
| `message` | `string` | What went wrong. |
| `frame` | `integer` \| `null` | The frame the error is for. `null` for an error about the session. |
| `retry_after` | `integer` \| `null` | Seconds to wait before you retry, when set. |

An `error` with a `frame` applies to that frame, and the session stays open. So does an `error` for a bad event after `session.create`, which has no `frame`. An `error` at open, or when the session reaches its spend ceiling, is followed by a close.

### session.stats

Sent after your `session.close`. A session that the Gateway closes for idle, lifetime, or spend ends without it.

| Field | Type | Description |
| - | - | - |
| `session_id` | `string` | The session's id. |
| `id` | `string` | The session's completion id. Read the billing row with [`GET /v1/completions/{id}`](https://docs.vlm.run/gateway/api-reference/get-completion). |
| `reads` | `integer` | Frames read. |
| `dropped` | `integer` | Frames dropped, for `rate` or `busy`. |
| `errors` | `integer` | Frames and events that returned an `error`. |
| `cost` | `number` | What the session bills, in USD: its open seconds at the model's hourly rate, at least \$0.001. |
| `p50_ms` | `number` | Median read latency, in milliseconds. |
| `p95_ms` | `number` | 95th percentile read latency, in milliseconds. |
| `fps` | `number` | Frames read per second, over the session. |

## Errors

| `error_type` | When | Close |
| - | - | - |
| `invalid_request_error` | A bad event or frame: not JSON, an unknown field, a method the model does not list, `max_inflight` over 8, a second `session.create`, or a frame that is not an image or is over `max_frame_bytes`. | `1008` when `session.create` is bad. None for a later event or frame. |
| `invalid_request_error` | The model has no realtime support. | `1008` |
| `not_found_error` | The model is unknown. | `1008` |
| `permission_error` | The key has no access to the model. | `1008` |
| `billing_error` | The organization is out of credit. | `1008` |
| `rate_limit_error` | Your plan's sessions are all open. | `1013` |
| `rate_limit_error` | Your plan's daily sessions are spent. Has `retry_after`. | `1008` |
| `rate_limit_error` | The session reached `max_cost`. | `1000`, reason `spent` |
| `capacity_error` | A session cap is full. The message is `WebSocket sessions are at capacity; retry shortly.` or names the model. | `1013` |
| `model_provisioning_error` | The model has no running replica. The Gateway starts one. Has `retry_after`. | `1013` |
| `overloaded_error` | One frame's read hit a busy model. Has `retry_after`. | None |
| `api_error` | One frame's read failed. Has `retry_after`. | None |

The Gateway also closes with `1000` and reason `idle` after 30 seconds with no frame and no read outstanding, and with reason `expired` at `expires_in`. What to do after each close code is in [Close codes](https://docs.vlm.run/gateway/realtime.md#close-codes). Session limits are in [Rate Limits](https://docs.vlm.run/gateway/rate-limits.md#websocket-sessions).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.