facebook/sam3.1

SAM 3.1 Object Multiplex: promptable segmentation and multi-object video tracking (848M). Accepts one image_url or one video_url; no text-only input and no document_url. A text prompt names what to segment, so segment on an image and track on a video both take method_params.prompt. segment is the default method. Each instance returns a normalized bounding box, a score, its area, and an instance_id. The pixels are not on the item: every instance shares one label map on the container.

Output by method

Method	Payload kind	`content.object`	Payload
`segment` (default)	json	`img.segment.masks`	One item per instance matching `method_params.prompt`, plus the container’s label map
`segment_box`	json	`img.segment.masks`	The same shape, seeded from `method_params.bbox_xywh` instead of a text prompt. Items carry no `label`.
`track` (video)	json	`vid.segment.masks`	One item per instance per sampled frame, carrying `frame_id` and `track_id`, with a label map on each entry of `frames`

Method parameters

Parameter	Default	Description
`prompt`	none	What to segment or track, in plain words. Returned verbatim as each item’s `label`.
`bbox_xywh`	none	`segment_box` only: the seed box as `[x, y, w, h]`, normalized 0-1.
`mask_format`	`png`	`none` drops the label map and keeps `area`.
`polygons`	`false`	`true` adds `polys_xy` outline rings to each item.
`video_fps`	none	`track` only: sample the clip at this rate, in frames per second. Must be greater than 0.
`video_skip_frames`	`1`	`track` only: sample every Nth frame. Cannot be combined with `video_fps`.
`video_max_frames`	`128`	`track` only: cap on sampled frames, 1 to 128. Above 128 is a `400`; lower `video_fps` to cover a longer clip.

Unlike the models that decode a video_url natively, facebook/sam3.1 reads its sampling out of method_params, not from a top-level video_fps.

Request

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key="<VLMRUN_API_KEY>",
)

response = client.chat.completions.create(
    model="facebook/sam3.1",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "http://images.cocodataset.org/val2017/000000000785.jpg"
                    },
                },
            ],
        }
    ],
    extra_body={"method": "segment", "method_params": {"prompt": "person"}},
)

print(response.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://gateway.vlm.run/v1/openai",
  apiKey: process.env.VLMRUN_API_KEY,
});

const response = await client.chat.completions.create({
  model: "facebook/sam3.1",
  messages: [
    {
      role: "user",
      content: [
        {
          type: "image_url",
          image_url: {
            url: "http://images.cocodataset.org/val2017/000000000785.jpg",
          },
        },
      ],
    },
  ],
  method: "segment",
  method_params: { prompt: "person" },
});

console.log(response.choices[0].message.content);

vlmrun gw chat http://images.cocodataset.org/val2017/000000000785.jpg \
  -m facebook/sam3.1 --method segment --method-params prompt=person

curl https://gateway.vlm.run/v1/openai/chat/completions \
  -X POST \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "facebook/sam3.1",
    "method": "segment",
    "method_params": { "prompt": "person" },
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "image_url",
            "image_url": {
              "url": "http://images.cocodataset.org/val2017/000000000785.jpg"
            }
          }
        ]
      }
    ]
  }'

Response

Text mode
JSON mode
Tracking a video

A single image returns the json block alone, with no wrapper:

{
  "object": "img.segment.masks",
  "items": [
    {
      "bbox_xywh": [0.4391, 0.1035, 0.3344, 0.8118],
      "label": "person",
      "score": 0.9688,
      "area": 0.0992,
      "instance_id": 1
    }
  ],
  "mask": {
    "format": "png",
    "height": 425,
    "width": 640,
    "data": "data:image/png;base64,iVBORw0KGgo…"
  }
}

The flat reply, with the image’s details beside the payload:

{
  "model": "facebook/sam3.1",
  "method": "segment",
  "image_hash": "sha256:1433c58c...",
  "image_width": 640,
  "image_height": 425,
  "content": {
    "object": "img.segment.masks",
    "items": [
      {
        "bbox_xywh": [0.4391, 0.1035, 0.3344, 0.8118],
        "label": "person",
        "score": 0.9688,
        "area": 0.0992,
        "instance_id": 1
      }
    ],
    "mask": {
      "format": "png",
      "height": 425,
      "width": 640,
      "data": "data:image/png;base64,iVBORw0KGgo…"
    }
  }
}

track returns the vid counterpart: items gain frame_id and track_id, and each sampled frame carries its own label map.

{
  "model": "facebook/sam3.1",
  "method": "track",
  "video_hash": "sha256:5c9e5eb1...",
  "video_width": 1280,
  "video_height": 720,
  "video_fps": 23.976,
  "video_nframes": 6059,
  "video_duration": 252.711,
  "content": {
    "object": "vid.segment.masks",
    "items": [
      {
        "bbox_xywh": [0.2297, 0.2604, 0.2133, 0.7396],
        "label": "person",
        "score": 0.9922,
        "area": 0.2751,
        "frame_id": 168,
        "track_id": 1
      }
    ],
    "frames": [
      {
        "frame_id": 0,
        "frame_ts": 0.0,
        "mask": {
          "format": "png",
          "height": 720,
          "width": 1280,
          "data": "data:image/png;base64,iVBORw0KGgo…"
        }
      }
    ]
  }
}

bbox_xywh and area are normalized against the source frame. The pixels live in the label map: one 8-bit grayscale PNG where a pixel’s value is the item’s instance_id on an image, or its track_id on a video. 0 is background, so ids start at 1 and run to 255, which is also the most instances one map can carry. Test pixel == n to cut out instance n, or pixel != 0 for the whole foreground. One decode gives every instance, and area and score sit on the items so a filter never has to decode at all. Where two instances overlap the pixel goes to the higher score, so each item’s area, box and outline describe what is actually visible in the map. video_fps, video_nframes and video_duration describe the source clip. frames is what was actually sampled, listed in frame_id order and including frames where nothing was found. A prompt that matches nothing is not an error. The call succeeds with an empty items array.

Get Started

OpenAI-Compatible API

Guides

API Reference

Tools

Platform

facebook/sam3.1

Output by method

Method parameters

Request

Response

​Output by method

​Method parameters

​Request

​Response

Output by method

Method parameters

Request

Response