Voxel51 FiftyOne

Overview

The VLM Run plugin for FiftyOne extracts structured data from images, documents, and videos in FiftyOne.

Installation

fiftyone plugins download \
    https://github.com/vlm-run/vlmrun-voxel51-plugin

fiftyone plugins requirements @vlm-run/vlmrun-voxel51-plugin --install

Plugin management is covered in the FiftyOne Plugins documentation.

Configuration

export VLMRUN_API_KEY="your-api-key-here"

Get an API key from vlm.run. You can also pass the key when you run an operator in the FiftyOne App.

Getting Started

Launch the FiftyOne App with your dataset:

import fiftyone as fo
import fiftyone.zoo as foz

# Load a sample dataset
dataset = foz.load_zoo_dataset("quickstart", max_samples=10)
session = fo.launch_app(dataset)

Press ` or click the Browse operations action to open the Operators list.
Select a VLM Run operator.

Run the operator on selected samples or the whole dataset. Set the domain options on the operator. Enable visual grounding when you need boxes on extracted fields.

Available Operators

Operator	Domain	Result
Object detection	`image.object-detection`	Detections with normalized bounding boxes, confidence scores, and labels
Person detection	`image.person-detection`	Person detections, including crowds and occlusions
Document analysis	`document.markdown`	Text from PDFs and images, with spatial coordinates, structure (headers, paragraphs, sections), tables and figures with boxes, and reading order
Invoice parsing	`document.invoice`	Totals, line items, vendor, dates, and payment terms. Optional visual grounding per field
Layout detection	`document.layout-detection`	Text regions, columns, headers, footers, tables, figures, and a bounding box for each element

Layout detection example showing structural elements.

Video Transcription

Mode	Result
`transcription`	Audio-to-text with timestamps
`comprehensive`	Audio, visual content, and activities
`objects`	Objects across frames
`scenes`	Scene classification and changes
`activities`	Activity and action recognition

Each mode returns temporal information. Modes can be combined.

Visual Grounding

Enabled grounding returns normalized xywh boxes. Each value is from 0 to 1.

Field	Value
`x`	Horizontal position of the top-left corner
`y`	Vertical position of the top-left corner
`w`	Width
`h`	Height

Execution Modes

Mode	Behavior
Immediate	Process in the FiftyOne App. Default.
Delegated	Queue for background processing. Requires orchestrator setup.

Supported Formats

Kind	Formats
Images	JPEG, PNG, BMP, TIFF, and other common formats
Documents	PDF files and document images
Videos	MP4, AVI, MOV, MKV, WEBM, FLV, WMV, M4V

Community

Discord or support@vlm.run.

Overview

Orion

Input & Outputs

Skills

Platform

Observability

Frameworks

Integrations

Voxel51 FiftyOne

Overview

Installation

Configuration

Getting Started

Available Operators

Video Transcription

Visual Grounding

Execution Modes

Supported Formats

Community

Learn More

​Overview

​Installation

​Configuration

​Getting Started

​Available Operators

​Video Transcription

​Visual Grounding

​Execution Modes

​Supported Formats

​Community

​Learn More

Overview

Installation

Configuration

Getting Started

Available Operators

Video Transcription

Visual Grounding

Execution Modes

Supported Formats

Community

Learn More