> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.

# Masks per Frame

> Segment every frame of a video by a text prompt and draw the masks on it

Segment every frame of a video by a text prompt, over one WebSocket, with `facebook/sam3.1`. Each reply carries a box and a score per object, plus one PNG label map for the whole frame. This guide masks every person in a 20 s clip and writes the result as a video, for \$0.017.

![Three video frames with each person tinted in a different color](https://mintcdn.com/autonomiai/PRWL1LD_l1Le04c7/images/gateway-live-masks.jpg?fit=max\&auto=format\&n=PRWL1LD_l1Le04c7\&q=85\&s=731dfc33ec16480f9d55750970e655d2)

Frames 1, 29, and 49 of the output. Each color is one instance in the label map.

## 1. Choose the prompt

`session.create` takes the same `method` and `method_params` as a [Chat Completions](https://docs.vlm.run/gateway/api-reference/post-chat-completions.md) request. The prompt is fixed for the session.

| Method | `method_params` | Segments |
| - | - | - |
| `segment` | `{"prompt": "person"}` | Every object that matches the prompt |
| `segment_box` | `{"bbox_xywh": [0.4, 0.3, 0.2, 0.5]}` | The object inside one box, as fractions of the frame |

Use a short noun phrase, such as `"person"`, `"car"`, or `"cardboard box"`. To segment a different thing, open a new session. `track` reads a whole video in one request, so it does not run on the socket. See [Methods](https://docs.vlm.run/gateway/methods.md).

## 2. Stream and draw

Cut the clip into frames as in [Video over WebSocket](https://docs.vlm.run/gateway/guides/websocket-video.md#2-cut-the-video-into-frames), then run:

```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
pip install websockets numpy pillow
```

```python Python [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import asyncio
import base64
import io
import json
import pathlib

import numpy as np
import websockets
from PIL import Image

URL = "wss://gateway.vlm.run/v1/ws"
FPS = 4
COLORS = np.array([[255, 80, 80], [80, 200, 120], [80, 140, 255], [255, 200, 60], [200, 100, 255]])

paths = sorted(pathlib.Path("frames").glob("*.jpg"))
out = pathlib.Path("masked")
out.mkdir(exist_ok=True)


def overlay(frame_path: pathlib.Path, content: dict) -> Image.Image:
    frame = np.asarray(Image.open(frame_path).convert("RGB"), dtype=np.float32)
    png = base64.b64decode(content["mask"]["data"].split(",", 1)[1])
    labels = np.asarray(Image.open(io.BytesIO(png))).astype(int)
    tint = COLORS[labels % len(COLORS)]
    hit = (labels > 0)[..., None]
    blended = np.where(hit, 0.5 * frame + 0.5 * tint, frame)
    return Image.fromarray(blended.astype(np.uint8))


async def main() -> None:
    async with websockets.connect(
        URL,
        additional_headers={"Authorization": "Bearer <VLMRUN_API_KEY>"},
        max_size=None,
    ) as ws:
        await ws.send(json.dumps({
            "type": "session.create",
            "model": "facebook/sam3.1",
            "method": "segment",
            "method_params": {"prompt": "person"},
            "max_inflight": 4,
        }))
        created = json.loads(await ws.recv())
        if created["type"] == "error":
            raise RuntimeError(created["message"])

        async def send_frames() -> None:
            for path in paths:
                await ws.send(path.read_bytes())
                await asyncio.sleep(1 / FPS)

        sender = asyncio.create_task(send_frames())
        for _ in paths:
            event = json.loads(await ws.recv())
            if event["type"] != "result":
                print(event)
                continue
            content = json.loads(event["response"]["choices"][0]["message"]["content"])
            overlay(paths[event["frame"] - 1], content).save(out / f"{event['frame']:04d}.jpg")
            print(event["frame"], [f"{i['label']} {i['score']:.2f}" for i in content["items"]])
        await sender

        await ws.send(json.dumps({"type": "session.close"}))
        print(json.loads(await ws.recv()))


asyncio.run(main())
```

The sample sends 80 frames at 4 per second and writes each masked frame to `masked/`. It prints the label and score of every instance:

```text Output theme={"theme":{"light":"github-light","dark":"dark-plus"}}
1 ['person 0.90', 'person 0.83', 'person 0.82', 'person 0.67', 'person 0.64', 'person 0.51']
2 ['person 0.88', 'person 0.86', 'person 0.84', 'person 0.74', 'person 0.73']
...
49 ['person 0.96', 'person 0.96', 'person 0.95', 'person 0.94', 'person 0.92', 'person 0.91', 'person 0.88', 'person 0.86', 'person 0.86', 'person 0.71', 'person 0.70', 'person 0.59']
...
{'type': 'session.stats', 'session_id': '...', 'id': 'ws-...', 'reads': 80, 'dropped': 0, 'errors': 0, 'cost': 0.016894, 'p50_ms': 65.9, 'p95_ms': 118.58, 'fps': 3.95}
```

All 80 frames were read, with a median read of 66 ms. Join the frames into a video with `ffmpeg`:

```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
ffmpeg -framerate 4 -i masked/%04d.jpg -c:v libx264 -pix_fmt yuv420p masked.mp4
```

`websockets.connect` takes `max_size=None`, which lifts the library's 1 MiB cap on incoming messages. Each reply carries a PNG at the full frame size, so a large frame makes a large reply.

## 3. Read the label map

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
  "object": "img.segment.masks",
  "items": [
    {"bbox_xywh": [0.0, 0.3722, 0.1516, 0.6278], "label": "person", "score": 0.8984, "area": 0.0618, "instance_id": 1},
    {"bbox_xywh": [0.2984, 0.0028, 0.2969, 0.3306], "label": "person", "score": 0.8281, "area": 0.0563, "instance_id": 2},
    {"bbox_xywh": [0.5453, 0.0028, 0.4391, 0.2917], "label": "person", "score": 0.8242, "area": 0.0496, "instance_id": 3}
  ],
  "mask": {"format": "png", "height": 360, "width": 640, "data": "data:image/png;base64,iVBORw0KGgo..."}
}
```

`mask` is one PNG at the frame's size. A pixel's value is the `instance_id` of the object it belongs to, and `0` is background. To get one object's mask, compare the label map with its id: `labels == item["instance_id"]`. `area` is that mask's share of the frame, from 0 to 1. The full schema is in [`img.segment.masks`](https://docs.vlm.run/gateway/response-types#img-segment-masks).

A frame with no match returns `"items": []` and a label map of zeros, so the overlay code needs no special case.

> [!WARNING]
>
> `instance_id` is per frame. It numbers the objects in one reply and does not follow an object from frame to frame. To follow one person across frames, match boxes between consecutive frames, for example by their overlap.

## 4. Cost and pace

A session bills every second it is open, at $3 / hour, whatever the number of frames or objects. The 20 s clip billed $0.0169. `sam3.1` allows up to 20 frames per second, and `session.created.max_fps` returns the rate. Pacing, drops, and `max_inflight` work as on any frames session. See [Keep up with the stream](https://docs.vlm.run/gateway/guides/websocket-video.md#5-keep-up-with-the-stream).

## Related

> [card] [Webcam from the Browser](https://docs.vlm.run/gateway/guides/live-webcam.md)
>
> Run the same session on a user's camera.

> [card] [Hand Tracking](https://docs.vlm.run/gateway/guides/live-hands.md)
>
> 2D and 3D hand joints per frame.

> [card] [Response Types](https://docs.vlm.run/gateway/response-types.md#img-segment-masks)
>
> The `img.segment.masks` schema in JSON, Pydantic, and Zod.

> [card] [Realtime](https://docs.vlm.run/gateway/realtime.md)
>
> Session lifecycle, close codes, and billing.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.