Realtime Frames

Streams frames to one model and method and returns the model’s reply per frame. For a walkthrough and samples, see Realtime Frames. For the lifecycle, close codes, and models, see Realtime. The socket is not in the OpenAPI schema, because the schema does not describe WebSocket routes.

Connection


URL	`wss://gateway.vlm.run/v1/ws`
Auth	`Authorization: Bearer <VLMRUN_API_KEY>` on the upgrade request
Anonymous	No key, or `Bearer vlmrun`: `401` at the upgrade, with `{"detail": "WebSocket sessions need an API key."}`
Events	JSON text messages, each with a `type`
Frames	Binary messages
First event	`session.create`, within 30 seconds of the upgrade

Client events

session.create

Opens the session. The Gateway checks the model, method, access, credit, and capacity once, here. An unknown field is refused.

Field	Type	Required	Description
`type`	`"session.create"`	Yes
`model`	`string`	Yes	A model id or alias with realtime support. See Supported models.
`method`	`string`	No	A method the model lists, for example `pose`. Default: the model’s default method. See Methods.
`method_params`	`object`	No	Parameters for the method, as on Chat Completions.
`prompt`	`string`	No	Text sent with every frame. Default: none.
`max_inflight`	`integer`	No	Reads outstanding at once, 1 to 8. Default `1`.

{ "type": "session.create", "model": "usyd-community/vitpose-plus-large", "method": "pose" }

Frame (binary message)

One frame: raw JPEG, PNG, WebP, or GIF bytes, with no base64 and no JSON wrapper. A frame is at most max_frame_bytes. Frames are numbered from 1 in the order they arrive, dropped frames included. Each frame gets exactly one event back with its number: result, frame.dropped, or error.

session.close

{ "type": "session.close" }

Reads in flight finish and return their events. The server then sends session.stats and closes with 1000. Any other event after session.create returns invalid_request_error, and the session stays open.

Server events

session.created

Field	Type	Description
`session_id`	`string`	The session’s id.
`model`	`string`	The canonical model id, also when you sent an alias.
`method`	`string` \| `null`	The method you sent. `null` means the model’s default method.
`max_fps`	`number`	Reads per second the model allows, on average. A frame sooner than that is dropped with `rate`.
`max_inflight`	`integer`	Reads outstanding at once.
`max_frame_bytes`	`integer`	The largest frame, in bytes: `2097152` (2 MiB).
`max_cost`	`number`	The session’s spend ceiling, in USD.
`expires_in`	`number`	Seconds the session may last, from the upgrade.

result

Field	Type	Description
`frame`	`integer`	The frame this reply is for.
`latency_ms`	`number`	Time the read took, in milliseconds.
`response`	`object`	What `POST /v1/openai/chat/completions` returns for this image. `choices[0].message.content` holds the method’s output as a JSON string.

response.usage.cost is null. The socket does not bill per frame: the session bills by the second, as session.stats.cost.

frame.dropped

Field	Type	Description
`frame`	`integer`	The frame that was not read.
`inflight`	`integer`	Reads outstanding when it arrived.
`reason`	`"rate"` \| `"busy"`	`rate`: it arrived sooner than `max_fps` allows after the last read. `busy`: every read slot was taken.

A dropped frame is not read. It does not change the session’s price.

error

Field	Type	Description
`error_type`	`string`	One of the error types.
`message`	`string`	What went wrong.
`frame`	`integer` \| `null`	The frame the error is for. `null` for an error about the session.
`retry_after`	`integer` \| `null`	Seconds to wait before you retry, when set.

An error with a frame applies to that frame, and the session stays open. So does an error for a bad event after session.create, which has no frame. An error at open, or when the session reaches its spend ceiling, is followed by a close.

session.stats

Sent after your session.close. A session that the Gateway closes for idle, lifetime, or spend ends without it.

Field	Type	Description
`session_id`	`string`	The session’s id.
`id`	`string`	The session’s completion id. Read the billing row with `GET /v1/completions/{id}`.
`reads`	`integer`	Frames read.
`dropped`	`integer`	Frames dropped, for `rate` or `busy`.
`errors`	`integer`	Frames and events that returned an `error`.
`cost`	`number`	What the session bills, in USD: its open seconds at the model’s hourly rate, at least $0.001.
`p50_ms`	`number`	Median read latency, in milliseconds.
`p95_ms`	`number`	95th percentile read latency, in milliseconds.
`fps`	`number`	Frames read per second, over the session.

Errors

`error_type`	When	Close
`invalid_request_error`	A bad event or frame: not JSON, an unknown field, a method the model does not list, `max_inflight` over 8, a second `session.create`, or a frame that is not an image or is over `max_frame_bytes`.	`1008` when `session.create` is bad. None for a later event or frame.
`invalid_request_error`	The model has no realtime support.	`1008`
`not_found_error`	The model is unknown.	`1008`
`permission_error`	The key has no access to the model.	`1008`
`billing_error`	The organization is out of credit.	`1008`
`rate_limit_error`	Your plan’s sessions are all open.	`1013`
`rate_limit_error`	Your plan’s daily sessions are spent. Has `retry_after`.	`1008`
`rate_limit_error`	The session reached `max_cost`.	`1000`, reason `spent`
`capacity_error`	A session cap is full. The message is `WebSocket sessions are at capacity; retry shortly.` or names the model.	`1013`
`model_provisioning_error`	The model has no running replica. The Gateway starts one. Has `retry_after`.	`1013`
`overloaded_error`	One frame’s read hit a busy model. Has `retry_after`.	None
`api_error`	One frame’s read failed. Has `retry_after`.	None

The Gateway also closes with 1000 and reason idle after 30 seconds with no frame and no read outstanding, and with reason expired at expires_in. What to do after each close code is in Close codes. Session limits are in Rate Limits.

OpenAI Compatible API

TypeSafe Compatible API

Realtime API

Realtime Frames

Connection

Client events

session.create

Frame (binary message)

session.close

Server events

session.created

result

frame.dropped

error

session.stats

Errors

​Connection

​Client events

​session.create

​Frame (binary message)

​session.close

​Server events

​session.created

​result

​frame.dropped

​error

​session.stats

​Errors

Connection

Client events

session.create

Frame (binary message)

session.close

Server events

session.created

result

frame.dropped

error

session.stats

Errors