Skip to main content
Segment every frame of a video by a text prompt, over one WebSocket, with facebook/sam3.1. Each reply carries a box and a score per object, plus one PNG label map for the whole frame. This guide masks every person in a 20 s clip and writes the result as a video, for $0.017.
Three video frames with each person tinted in a different color

Frames 1, 29, and 49 of the output. Each color is one instance in the label map.

1. Choose the prompt

session.create takes the same method and method_params as a Chat Completions request. The prompt is fixed for the session. Use a short noun phrase, such as "person", "car", or "cardboard box". To segment a different thing, open a new session. track reads a whole video in one request, so it does not run on the socket. See Methods.

2. Stream and draw

Cut the clip into frames as in Video over WebSocket, then run:
Python
The sample sends 80 frames at 4 per second and writes each masked frame to masked/. It prints the label and score of every instance:
Output
All 80 frames were read, with a median read of 66 ms. Join the frames into a video with ffmpeg:
websockets.connect takes max_size=None, which lifts the library’s 1 MiB cap on incoming messages. Each reply carries a PNG at the full frame size, so a large frame makes a large reply.

3. Read the label map

mask is one PNG at the frame’s size. A pixel’s value is the instance_id of the object it belongs to, and 0 is background. To get one object’s mask, compare the label map with its id: labels == item["instance_id"]. area is that mask’s share of the frame, from 0 to 1. The full schema is in img.segment.masks. A frame with no match returns "items": [] and a label map of zeros, so the overlay code needs no special case.
instance_id is per frame. It numbers the objects in one reply and does not follow an object from frame to frame. To follow one person across frames, match boxes between consecutive frames, for example by their overlap.

4. Cost and pace

A session bills every second it is open, at $3 / hour, whatever the number of frames or objects. The 20 s clip billed $0.0169. sam3.1 allows up to 20 frames per second, and session.created.max_fps returns the rate. Pacing, drops, and max_inflight work as on any frames session. See Keep up with the stream.

Webcam from the Browser

Run the same session on a user’s camera.

Hand Tracking

2D and 3D hand joints per frame.

Response Types

The img.segment.masks schema in JSON, Pydantic, and Zod.

Realtime

Session lifecycle, close codes, and billing.