> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.

# facebook/sam3.1

> Promptable segmentation masks on an image, and multi-object tracking through a video.

SAM 3.1 Object Multiplex: promptable segmentation and multi-object video
tracking (848M). Accepts one `image_url` or one `video_url`; no text-only input
and no `document_url`. A text prompt names what to segment, so `segment` on an
image and `track` on a video both take `method_params.prompt`.

`segment` is the default method. Each instance returns a normalized bounding
box, a `score`, its `area`, and an `instance_id`. The pixels are not on the
item: every instance shares one [label map](/gateway/methods#label-map) on the
container.

## Output by method

| Method              | Payload kind | `content.object`    | Payload                                                                                                                 |
| ------------------- | ------------ | ------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `segment` (default) | json         | `img.segment.masks` | One item per instance matching `method_params.prompt`, plus the container's label map                                   |
| `segment_box`       | json         | `img.segment.masks` | The same shape, seeded from `method_params.bbox_xywh` instead of a text prompt. Items carry no `label`.                 |
| `track` (video)     | json         | `vid.segment.masks` | One item per instance per sampled frame, carrying `frame_id` and `track_id`, with a label map on each entry of `frames` |

## Method parameters

| Parameter           | Default | Description                                                                                                    |
| ------------------- | ------: | -------------------------------------------------------------------------------------------------------------- |
| `prompt`            |    none | What to segment or track, in plain words. Returned verbatim as each item's `label`.                            |
| `bbox_xywh`         |    none | `segment_box` only: the seed box as `[x, y, w, h]`, normalized 0-1.                                            |
| `mask_format`       |   `png` | `none` drops the label map and keeps `area`.                                                                   |
| `polygons`          | `false` | `true` adds `polys_xy` outline rings to each item.                                                             |
| `video_fps`         |    none | `track` only: sample the clip at this rate, in frames per second. Must be greater than 0.                      |
| `video_skip_frames` |     `1` | `track` only: sample every Nth frame. Cannot be combined with `video_fps`.                                     |
| `video_max_frames`  |   `128` | `track` only: cap on sampled frames, 1 to 128. Above 128 is a `400`; lower `video_fps` to cover a longer clip. |

Unlike the models that decode a `video_url` natively, `facebook/sam3.1` reads
its sampling out of `method_params`, not from a top-level `video_fps`.

## Request

<CodeGroup>
  ```python Python [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  from openai import OpenAI

  client = OpenAI(
      base_url="https://gateway.vlm.run/v1/openai",
      api_key="<VLMRUN_API_KEY>",
  )

  response = client.chat.completions.create(
      model="facebook/sam3.1",
      messages=[
          {
              "role": "user",
              "content": [
                  {
                      "type": "image_url",
                      "image_url": {
                          "url": "http://images.cocodataset.org/val2017/000000000785.jpg"
                      },
                  },
              ],
          }
      ],
      extra_body={"method": "segment", "method_params": {"prompt": "person"}},
  )

  print(response.choices[0].message.content)
  ```

  ```typescript Node.js [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import OpenAI from "openai";

  const client = new OpenAI({
    baseURL: "https://gateway.vlm.run/v1/openai",
    apiKey: process.env.VLMRUN_API_KEY,
  });

  const response = await client.chat.completions.create({
    model: "facebook/sam3.1",
    messages: [
      {
        role: "user",
        content: [
          {
            type: "image_url",
            image_url: {
              url: "http://images.cocodataset.org/val2017/000000000785.jpg",
            },
          },
        ],
      },
    ],
    method: "segment",
    method_params: { prompt: "person" },
  });

  console.log(response.choices[0].message.content);
  ```

  ```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  vlmrun gw chat http://images.cocodataset.org/val2017/000000000785.jpg \
    -m facebook/sam3.1 --method segment --method-params prompt=person
  ```

  ```bash cURL [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  curl https://gateway.vlm.run/v1/openai/chat/completions \
    -X POST \
    -H "Authorization: Bearer $VLMRUN_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "facebook/sam3.1",
      "method": "segment",
      "method_params": { "prompt": "person" },
      "messages": [
        {
          "role": "user",
          "content": [
            {
              "type": "image_url",
              "image_url": {
                "url": "http://images.cocodataset.org/val2017/000000000785.jpg"
              }
            }
          ]
        }
      ]
    }'
  ```
</CodeGroup>

## Response

<Tabs>
  <Tab title="Text mode">
    A single image returns the json block alone, with no wrapper:

    ```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
    {
      "object": "img.segment.masks",
      "items": [
        {
          "bbox_xywh": [0.4391, 0.1035, 0.3344, 0.8118],
          "label": "person",
          "score": 0.9688,
          "area": 0.0992,
          "instance_id": 1
        }
      ],
      "mask": {
        "format": "png",
        "height": 425,
        "width": 640,
        "data": "data:image/png;base64,iVBORw0KGgo…"
      }
    }
    ```
  </Tab>

  <Tab title="JSON mode">
    The flat reply, with the image's details beside the payload:

    ```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
    {
      "model": "facebook/sam3.1",
      "method": "segment",
      "image_hash": "sha256:1433c58c...",
      "image_width": 640,
      "image_height": 425,
      "content": {
        "object": "img.segment.masks",
        "items": [
          {
            "bbox_xywh": [0.4391, 0.1035, 0.3344, 0.8118],
            "label": "person",
            "score": 0.9688,
            "area": 0.0992,
            "instance_id": 1
          }
        ],
        "mask": {
          "format": "png",
          "height": 425,
          "width": 640,
          "data": "data:image/png;base64,iVBORw0KGgo…"
        }
      }
    }
    ```
  </Tab>

  <Tab title="Tracking a video">
    `track` returns the `vid` counterpart: items gain `frame_id` and `track_id`,
    and each sampled frame carries its own label map.

    ```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
    {
      "model": "facebook/sam3.1",
      "method": "track",
      "video_hash": "sha256:5c9e5eb1...",
      "video_width": 1280,
      "video_height": 720,
      "video_fps": 23.976,
      "video_nframes": 6059,
      "video_duration": 252.711,
      "content": {
        "object": "vid.segment.masks",
        "items": [
          {
            "bbox_xywh": [0.2297, 0.2604, 0.2133, 0.7396],
            "label": "person",
            "score": 0.9922,
            "area": 0.2751,
            "frame_id": 168,
            "track_id": 1
          }
        ],
        "frames": [
          {
            "frame_id": 0,
            "frame_ts": 0.0,
            "mask": {
              "format": "png",
              "height": 720,
              "width": 1280,
              "data": "data:image/png;base64,iVBORw0KGgo…"
            }
          }
        ]
      }
    }
    ```
  </Tab>
</Tabs>

`bbox_xywh` and `area` are normalized against the source frame. The pixels live
in the label map: one 8-bit grayscale PNG where a pixel's value is the item's
`instance_id` on an image, or its `track_id` on a video.

**`0` is background**, so ids start at **1** and run to **255**, which is also
the most instances one map can carry. Test `pixel == n` to cut out instance `n`,
or `pixel != 0` for the whole foreground. One decode gives every instance, and
`area` and `score` sit on the items so a filter never has to decode at all.

Where two instances overlap the pixel goes to the higher `score`, so each item's
`area`, box and outline describe what is actually visible in the map.

`video_fps`, `video_nframes` and `video_duration` describe the source clip.
`frames` is what was actually sampled, listed in `frame_id` order and including
frames where nothing was found.

A prompt that matches nothing is not an error. The call succeeds with an empty
`items` array.