Skip to main content
Ask one set of System One questions of every frame of a video, over one WebSocket. Each read returns a decision with a calibrated probability per answer, so you act on a threshold instead of parsing text. This guide finds the moments a shop front is on screen in a 20 s clip, for $0.0008 in total.

1. Write the questions

A session asks the same questions of every frame. Send them once, in session.create. This guide asks two:
storefront is a noul: it returns the probability that the answer is yes, from 0 to 1. scene is a choice: it returns one label from your set, a probability per label, and a confidence. Every question type is in System One.

2. Choose how often to read

You send every frame, and the session decides which frames fire a read. A read fires every stride frames, over the newest window frames. Frames between reads bill nothing. Raise stride before max_inflight. Consecutive frames rarely change a decision, and fewer reads cost less.

3. Stream the clip

Cut the clip into JPEG frames at 4 per second, as in Video over WebSocket:
The sample sends the 80 frames at 4 per second. With stride: 4, frames 4, 8, 12, and so on fire a read, so the session returns 20 decisions, one per second of video.
Output
storefront rises above 0.8 from 6.75 s to 8.75 s, the street shot of the bakery. The median read took 63 ms. The 20 decisions billed $0.00082 together.

4. Read a decision

frame is the frame that fired the read. answers and usage have the same shape as a POST /typesafe/v1/systemone reply, and usage.cost is what this decision bills. A choice always names a label, so read confidence before you trust it: at 4.75 s above, scene is street with a confidence of 0.00, which is a guess.

5. Turn probabilities into moments

Threshold each noul and merge the consecutive reads above it into spans:
A span ends at the first read below the threshold, so it is accurate to one stride, here 1 s. Lower stride for tighter edges. A threshold is model-specific, so tune it on a labelled sample of your own video, and tune it again when you change models.

6. Ask on demand with decide

A decide event runs one read against the session’s questions, with its own content and, optionally, its own state. It is never dropped, and the decision echoes its id. Use it for a frame you want read for certain, such as a still a user flags.
A decide reads only its own content. It does not read the frames you streamed. A decide with no content answers from the state text alone, with 0 image tokens. Its images read at the session’s detail.
To change what later frames read against, without a read, send a state event: {"type": "state", "state": "Answer about the kitchen camera."}. It gets no reply.

7. Cost and limits

Each decision is one billing row, priced per token like a posted read. Frames that do not fire a read, dropped frames, failed reads, and idle time bill nothing. Two runs of this clip billed $0.00082 and $0.00024 for 20 decisions each. The difference is the prefix cache: 1,920 and 3,840 of the 4,000 input tokens were cached, and cached tokens bill at the cache rate. A session lasts up to 10 minutes, closes after 30 seconds idle, and stops at $5 of spend. The first read of a session can take longer than the rest: in one run it took 3.5 s, and 2 frames were dropped as busy while it ran. Sessions per plan are on Rate Limits.

Video over WebSocket

A model reply per frame: pose, masks, or text lines.

Realtime System One

The socket, its SDK and CLI clients, and HTTP vs WebSocket latency.

WS /v1/ws/systemone reference

Every client and server event, with its fields.

Document Classification

The same questions over a single image or PDF.