> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.

# System One Decisions

> Ask the same typed questions of every frame of a video and act on the probabilities

Ask one set of [System One](https://docs.vlm.run/gateway/system-one.md) questions of every frame of a video, over one WebSocket. Each read returns a `decision` with a calibrated probability per answer, so you act on a threshold instead of parsing text. This guide finds the moments a shop front is on screen in a 20 s clip, for \$0.0008 in total.

## 1. Write the questions

A session asks the same questions of every frame. Send them once, in `session.create`. This guide asks two:

```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
QUESTIONS = {
    "storefront": {"type": "noul", "instructions": "Is a shop front visible from the street?"},
    "scene": {
        "type": "choice",
        "instructions": "Where is the camera?",
        "criteria": {
            "street": None,
            "shop_counter": None,
            "kitchen": None,
            "portrait": "People posing for the camera",
            "other": None,
        },
    },
}
```

`storefront` is a `noul`: it returns the probability that the answer is yes, from 0 to 1. `scene` is a `choice`: it returns one label from your set, a probability per label, and a `confidence`. Every question type is in [System One](https://docs.vlm.run/gateway/system-one.md).

| Do | Why |
| - | - |
| Ask about one frame, not the video | A read sees the newest `window` frames only, 1 by default |
| Include `other` in a `choice` | A read must pick one of your labels, so a frame that fits none becomes a confident wrong answer |
| Keep `detail` at `low` | 70 vision tokens per frame instead of 280. Scene-level questions rarely need small print. |

## 2. Choose how often to read

You send every frame, and the session decides which frames fire a read. A read fires every `stride` frames, over the newest `window` frames. Frames between reads bill nothing.

| `session.create` | Reads, at 4 frames sent per second | Use it when |
| - | - | - |
| `{"stride": 1}` | 4 per second | The answer can change within a quarter second |
| `{"stride": 4}` | 1 per second | Scene-level questions, as in this guide |
| `{"stride": 4, "window": 4}` | 1 per second, each over the last 4 frames | The question is about motion, such as "Is the door opening?" |

Raise `stride` before `max_inflight`. Consecutive frames rarely change a decision, and fewer reads cost less.

## 3. Stream the clip

Cut the clip into JPEG frames at 4 per second, as in [Video over WebSocket](https://docs.vlm.run/gateway/guides/websocket-video.md#2-cut-the-video-into-frames):

```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -sO https://storage.googleapis.com/vlm-data-public-prod/web/videos/video-trim-20s.mp4
mkdir -p frames
ffmpeg -i video-trim-20s.mp4 -vf "fps=4,scale=640:-2" -q:v 3 frames/%04d.jpg
```

```python Python [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import asyncio
import json
import pathlib

import websockets

URL = "wss://gateway.vlm.run/v1/ws/systemone"
FPS = 4
STRIDE = 4

frames = [p.read_bytes() for p in sorted(pathlib.Path("frames").glob("*.jpg"))]

QUESTIONS = {
    "storefront": {"type": "noul", "instructions": "Is a shop front visible from the street?"},
    "scene": {
        "type": "choice",
        "instructions": "Where is the camera?",
        "criteria": {
            "street": None,
            "shop_counter": None,
            "kitchen": None,
            "portrait": "People posing for the camera",
            "other": None,
        },
    },
}


async def main() -> None:
    async with websockets.connect(
        URL, additional_headers={"Authorization": "Bearer <VLMRUN_API_KEY>"}
    ) as ws:
        await ws.send(json.dumps({
            "type": "session.create",
            "model": "google/diffusiongemma-26b-a4b-it",
            "state": "Answer about the video frame.",
            "questions": QUESTIONS,
            "detail": "low",
            "stride": STRIDE,
            "max_inflight": 2,
        }))
        created = json.loads(await ws.recv())
        if created["type"] == "error":
            raise RuntimeError(created["message"])

        async def send_frames() -> None:
            for frame in frames:
                await ws.send(frame)
                await asyncio.sleep(1 / FPS)

        sender = asyncio.create_task(send_frames())
        decisions = []
        for _ in range(len(frames) // STRIDE):
            event = json.loads(await ws.recv())
            if event["type"] == "decision":
                decisions.append(event)
            elif event["type"] == "frame.dropped":
                print("dropped", event["frame"])
            elif event["type"] == "error":
                print("error", event.get("frame"), event["error_type"], event["message"])
        await sender

        await ws.send(json.dumps({"type": "session.close"}))
        stats = json.loads(await ws.recv())

    for d in decisions:
        scene = d["answers"]["scene"]
        storefront = d["answers"]["storefront"]["noul"]
        print(f"{(d['frame'] - 1) / FPS:5.2f} s  storefront={storefront:.2f}  scene={scene['choice']} ({scene['confidence']:.2f})")
    print(stats)


asyncio.run(main())
```

```typescript Node.js [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { readdirSync, readFileSync } from "node:fs";
import WebSocket from "ws";

const URL = "wss://gateway.vlm.run/v1/ws/systemone";
const FPS = 4;
const STRIDE = 4;

const frames = readdirSync("frames")
  .filter((name) => name.endsWith(".jpg"))
  .sort()
  .map((name) => readFileSync(`frames/${name}`));

const QUESTIONS = {
  storefront: { type: "noul", instructions: "Is a shop front visible from the street?" },
  scene: {
    type: "choice",
    instructions: "Where is the camera?",
    criteria: {
      street: null,
      shop_counter: null,
      kitchen: null,
      portrait: "People posing for the camera",
      other: null,
    },
  },
};

const ws = new WebSocket(URL, {
  headers: { Authorization: `Bearer ${process.env.VLMRUN_API_KEY}` },
});

const decisions = [];
let answered = 0;

ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "session.create",
    model: "google/diffusiongemma-26b-a4b-it",
    state: "Answer about the video frame.",
    questions: QUESTIONS,
    detail: "low",
    stride: STRIDE,
    max_inflight: 2,
  }));
});

ws.on("message", async (data) => {
  const event = JSON.parse(data.toString());

  switch (event.type) {
    case "session.created":
      for (const frame of frames) {
        ws.send(frame);
        await new Promise((resolve) => setTimeout(resolve, 1000 / FPS));
      }
      return;
    case "decision":
      decisions.push(event);
      break;
    case "frame.dropped":
      console.log("dropped", event.frame);
      break;
    case "error":
      if (event.frame == null) throw new Error(event.message);
      console.log("error", event.frame, event.error_type, event.message);
      break;
    case "session.stats":
      for (const d of decisions) {
        const { scene, storefront } = d.answers;
        const t = ((d.frame - 1) / FPS).toFixed(2).padStart(5);
        console.log(`${t} s  storefront=${storefront.noul.toFixed(2)}  scene=${scene.choice} (${scene.confidence.toFixed(2)})`);
      }
      console.log(event);
      ws.close(1000);
      return;
  }
  if (++answered === Math.floor(frames.length / STRIDE)) ws.send(JSON.stringify({ type: "session.close" }));
});
```

The sample sends the 80 frames at 4 per second. With `stride: 4`, frames 4, 8, 12, and so on fire a read, so the session returns 20 decisions, one per second of video.

```text Output theme={"theme":{"light":"github-light","dark":"dark-plus"}}
 0.75 s  storefront=0.07  scene=shop_counter (1.00)
 1.75 s  storefront=0.12  scene=shop_counter (0.98)
 2.75 s  storefront=0.04  scene=shop_counter (0.94)
 3.75 s  storefront=0.02  scene=shop_counter (0.81)
 4.75 s  storefront=0.00  scene=street (0.00)
 5.75 s  storefront=0.11  scene=street (0.66)
 6.75 s  storefront=0.87  scene=street (0.87)
 7.75 s  storefront=0.97  scene=street (0.99)
 8.75 s  storefront=0.95  scene=street (0.98)
 9.75 s  storefront=0.03  scene=portrait (0.13)
...
19.75 s  storefront=0.24  scene=shop_counter (0.96)
{'type': 'session.stats', 'session_id': '...', 'frames': 80, 'decisions': 20, 'dropped': 0, 'errors': 0, 'reads': 20, 'input_tokens': 4000, 'cached_tokens': 1920, 'cost': 0.00082, 'p50_ms': 62.65, 'p95_ms': 64.9, 'fps': 3.91}
```

`storefront` rises above 0.8 from 6.75 s to 8.75 s, the street shot of the bakery. The median read took 63 ms. The 20 decisions billed \$0.00082 together.

## 4. Read a decision

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
  "type": "decision",
  "frame": 4,
  "answers": {
    "storefront": { "type": "noul", "noul": 0.5184 },
    "scene": {
      "type": "choice",
      "choice": "shop_counter",
      "probabilities": { "street": 0.00003, "shop_counter": 0.99988, "kitchen": 0.00003, "portrait": 0.00003, "other": 0.00003 },
      "confidence": 0.9991
    }
  },
  "usage": {
    "input_tokens": 200,
    "output_tokens": 0,
    "input_tokens_details": { "cached_tokens": 32, "image_tokens": 70, "text_tokens": 130 },
    "reads": 1,
    "cost": 0.00006
  },
  "latency_ms": 3507.6,
  "id": null
}
```

`frame` is the frame that fired the read. `answers` and `usage` have the same shape as a [`POST /typesafe/v1/systemone`](https://docs.vlm.run/gateway/api-reference/post-systemone) reply, and `usage.cost` is what this decision bills. A `choice` always names a label, so read `confidence` before you trust it: at 4.75 s above, `scene` is `street` with a confidence of 0.00, which is a guess.

## 5. Turn probabilities into moments

Threshold each `noul` and merge the consecutive reads above it into spans:

```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
def moments(decisions, question, threshold=0.8):
    spans, start = [], None
    for d in decisions:
        t = (d["frame"] - 1) / FPS
        hit = d["answers"][question]["noul"] >= threshold
        if hit and start is None:
            start = t
        elif not hit and start is not None:
            spans.append((start, t))
            start = None
    if start is not None:
        spans.append((start, t + STRIDE / FPS))
    return spans


print(moments(decisions, "storefront"))   # [(6.75, 9.75)]
```

A span ends at the first read below the threshold, so it is accurate to one `stride`, here 1 s. Lower `stride` for tighter edges. A threshold is model-specific, so tune it on a labelled sample of your own video, and tune it again when you change models.

## 6. Ask on demand with `decide`

A `decide` event runs one read against the session's questions, with its own `content` and, optionally, its own `state`. It is never dropped, and the `decision` echoes its `id`. Use it for a frame you want read for certain, such as a still a user flags.

```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import base64, json, pathlib

still = base64.b64encode(pathlib.Path("still.jpg").read_bytes()).decode()

await ws.send(json.dumps({
    "type": "decide",
    "id": "still-8s",
    "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{still}"}}
    ],
}))
```

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
  "type": "decision",
  "frame": 0,
  "answers": {
    "storefront": { "type": "noul", "noul": 0.8711 },
    "scene": {
      "type": "choice",
      "choice": "street",
      "probabilities": { "street": 0.99973, "shop_counter": 0.00007, "kitchen": 0.00007, "portrait": 0.00007, "other": 0.00007 },
      "confidence": 0.9982
    }
  },
  "usage": {
    "input_tokens": 200,
    "output_tokens": 0,
    "input_tokens_details": { "cached_tokens": 96, "image_tokens": 70, "text_tokens": 130 },
    "reads": 1,
    "cost": 0.000041
  },
  "latency_ms": 69.7,
  "id": "still-8s"
}
```

> [!WARNING]
>
> A `decide` reads only its own `content`. It does not read the frames you streamed. A `decide` with no `content` answers from the state text alone, with 0 image tokens. Its images read at the session's `detail`.

To change what later frames read against, without a read, send a `state` event: `{"type": "state", "state": "Answer about the kitchen camera."}`. It gets no reply.

## 7. Cost and limits

Each decision is one billing row, priced per token like a posted read. Frames that do not fire a read, dropped frames, failed reads, and idle time bill nothing. Two runs of this clip billed $0.00082 and $0.00024 for 20 decisions each. The difference is the prefix cache: 1,920 and 3,840 of the 4,000 input tokens were cached, and cached tokens bill at the cache rate.

A session lasts up to 10 minutes, closes after 30 seconds idle, and stops at \$5 of spend. The first read of a session can take longer than the rest: in one run it took 3.5 s, and 2 frames were dropped as `busy` while it ran. Sessions per plan are on [Rate Limits](https://docs.vlm.run/gateway/rate-limits.md#websocket-sessions).

## Related

> [card] [Video over WebSocket](https://docs.vlm.run/gateway/guides/websocket-video.md)
>
> A model reply per frame: pose, masks, or text lines.

> [card] [Realtime System One](https://docs.vlm.run/gateway/realtime-systemone.md)
>
> The socket, its SDK and CLI clients, and HTTP vs WebSocket latency.

> [card] [WS /v1/ws/systemone reference](https://docs.vlm.run/gateway/api-reference/ws-systemone.md)
>
> Every client and server event, with its fields.

> [card] [Document Classification](https://docs.vlm.run/gateway/guides/document-classification.md)
>
> The same questions over a single image or PDF.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.