facebook/sam3.1. Each reply carries a box and a score per object, plus one PNG label map for the whole frame. This guide masks every person in a 20 s clip and writes the result as a video, for $0.017.

Frames 1, 29, and 49 of the output. Each color is one instance in the label map.
1. Choose the prompt
session.create takes the same method and method_params as a Chat Completions request. The prompt is fixed for the session.
Use a short noun phrase, such as
"person", "car", or "cardboard box". To segment a different thing, open a new session. track reads a whole video in one request, so it does not run on the socket. See Methods.
2. Stream and draw
Cut the clip into frames as in Video over WebSocket, then run:Python
masked/. It prints the label and score of every instance:
Output
ffmpeg:
websockets.connect takes max_size=None, which lifts the library’s 1 MiB cap on incoming messages. Each reply carries a PNG at the full frame size, so a large frame makes a large reply.
3. Read the label map
mask is one PNG at the frame’s size. A pixel’s value is the instance_id of the object it belongs to, and 0 is background. To get one object’s mask, compare the label map with its id: labels == item["instance_id"]. area is that mask’s share of the frame, from 0 to 1. The full schema is in img.segment.masks.
A frame with no match returns "items": [] and a label map of zeros, so the overlay code needs no special case.
4. Cost and pace
A session bills every second it is open, at $3 / hour, whatever the number of frames or objects. The 20 s clip billed $0.0169.sam3.1 allows up to 20 frames per second, and session.created.max_fps returns the rate. Pacing, drops, and max_inflight work as on any frames session. See Keep up with the stream.
Related
Webcam from the Browser
Run the same session on a user’s camera.
Hand Tracking
2D and 3D hand joints per frame.
Response Types
The
img.segment.masks schema in JSON, Pydantic, and Zod.Realtime
Session lifecycle, close codes, and billing.