.domain-column {
    width: 240px;
    display: inline-block;
  }

.function-column {
  width: 240px;
  display: inline-block;
}

.allowed-inputs-column {
  width: 160px;
  display: inline-block;
}

.description-column {
  width: 100%;
  display: inline-block;
}

.tag {
  padding: 2px 8px;
  border-radius: 12px;
  font-size: 0.9em;
  font-weight: 500;
  display: inline-block;
  margin-right: 8px;
  margin-bottom: 4px;
}

.tag-document {
  background: #FFF3E0;
  color: #E65100;
}

.tag-image {
  background: #E6FFDD;
  color: #438F2A;
}

.tag-video {
  background: #EAE0FF;
  color: #784BCE;
}

.tag-audio {
  background: #DEE9FF;
  color: #4478D7;
}

.tag-url {
  background: #E3F2FD;
  color: #1976D2;
}

.tag-ref {
  background: #F3E5F5;
  color: #7B1FA2;
}




.card {
  background-color: #f9f9f9;
}

.card-demo {
    color: #4478D7;
    background-color: #dae6ff;
}


I/O Tools

Image Tools

Document Tools

Video Tools

Hub Tools

Available Tools

Supported Inputs

Complete reference of all available VLM Run MCP tools for visual AI processing.

MCP Tools Reference

VLM Run

Extract JSON from images, videos, and documents with a unified API.

Introduction

Extract JSON from images, videos, and documents with type-safety.

Structured Responses

Pre-built schemas and domain definitions for common data extraction tasks.

Supported Domains

Define custom schemas for visual extraction purposes.

Custom Schemas

Ground extracted data with location (bounding box) coordinates and confidence scores.

Visual Grounding

Ground extracted data with start/end times for audio/video segments and speaker identification.

Temporal Grounding

Support for long-output contexts for domains like audio/video transcription, exceeding 8K token limits.

Long-context Outputs

Query a subset of schema fields to improve efficiency for querying and document ETL.

GraphQL

Extract structured data from long documents and reports.

Parsing Documents

Learn how to classify documents into categories like invoices, bank statements, and utility bills.

Classifying Documents

Learn how to classify images into categories like animals, landscapes, and objects using AI.

Classifying Images

Learn how to generate captions, tags and descriptions for images.

Cataloging Images

Learn how to transcribe and analyze hours-long video content using our Video Transcription API.

Transcribing Video

Learn how to transcribe and analyze long-form audio.

Transcribing Audio

Improve model performance through feedback collection and fine-tuning.

Providing Feedback

Rate limits to consider when using the API.

Rate Limits

List of error codes that you may encounter when using the API

Error Codes

Changelog

Teach your AI agents to See, Act and Automate – with VLM Run MCP

VLM Run MCP Server

Connect to the remote VLM Run MCP Server and start building agentic workflows with visual AI in minutes.

MCP Quickstart

Health

Get models

List domains

Get the JSON schema for a given domain (document.invoice, document.receipt, etc).

Get schema for domain

Generate structured prediction for the given image.

Image -> JSON

Generate structured prediction for the given document.

Doc -> JSON

Generate structured prediction for the given document using a custom agent.

Doc Agent -> JSON

Generate structured prediction for the given audio file.

Audio -> JSON

Generate structured prediction for the given video file.

Video -> JSON

Get Predictions by ID

Get all predictions uploaded by the user with pagination.

Get Predictions

Upload File

Get File by ID

Get all files uploaded by the user with pagination.

List Files

Create chat completion

List all models

Submit feedback for a prediction by its ID.

Submit Feedback

Get all feedbacks for a specific request by its ID.

Get Feedback

How to get started with the VLM Run Python SDK

Getting Started

Core concepts and components of the VLM Run Python SDK

SDK Overview

Detailed guide to the VLM Run Python SDK client

Client Reference

client.files

client.models

client.hub

client.image

client.document

client.audio

client.video

Manage predictions with the VLM Run Python SDK

client.predictions

Complete guide to the VLM Run Command Line Interface

CLI Reference

Learn how to install and use the VLM Run Node.js SDK

VLM Run Node.js SDK Client Configuration and Usage

client

Learn how to upload and manage files with the VLM Run Node.js SDK

Learn how to work with models in the VLM Run Node.js SDK

Learn how to process images with the VLM Run Node.js SDK

Learn how to process documents with the VLM Run Node.js SDK

Learn how to process audio files with the VLM Run Node.js SDK

Extract JSON from images, videos, and documents with custom schemas.

Run VLM-1 with the OpenAI Python SDK with just 2 lines of code change.

OpenAI Compatibility

Run `vlm-1` with the Instructor Python SDK with minimal code changes.

Instructor Compatibility

The VLM Run Hub is a collection of pre-defined domains and schemas for structured data extraction.

Tool	Description
`put_image_url`	Load images from URLs into the system for processing by other tools.
`put_file_url`	Load files from URLs into the system for processing by other tools.
`preview_object_ref`	Generate preview URLs for images, files, and other objects stored in the system.

Tool	Description
`detect_faces`	Detect faces in images and return facial features, landmarks, and confidence scores.
`detect_objects`	Detect and classify objects in images with bounding boxes and labels.
`detect_texts`	Extract text from images using OCR, returning detected text with bounding boxes and confidence scores.
`detect_logos`	Detect brand logos and trademarks in images with confidence scores and bounding box locations.
`detect_barcodes`	Detect and decode barcodes and QR codes in images, returning data, type, and location.
`parse_image`	Parse the image into a structured format.
`rotate_image`	Rotate images by specified angle in degrees (counter-clockwise). Supports 90°, 180°, -90° rotations.
`crop_image`	Crop images to specified bounding box coordinates for precise region extraction.

Tool	Description
`document_images`	Extract pages from documents as images with configurable offset and limit for batch processing.
`detect_document_layout`	Extract document layout including tables, figures, paragraphs, headers, and other structural elements.
`detect_document_image_layout`	Extract image layout including tables, figures, paragraphs, headers, and other structural elements.
`parse_document`	Parse documents into structured format using specific domain schemas (e.g., invoices, receipts).
`parse_document_image`	Parse single document images into structured format using domain-specific schemas.

Tool	Description
`search_video`	Search a video for specific content and return relevant scenes, frames, or timestamps based on your query.
`transcribe_video`	Transcribe entire videos with detailed audio transcript and visual scene breakdown for ~30 second segments.
`add_video_captions`	Automatically add subtitles or captions to video clips with customizable colors and positioning.

Tool	Description
`get_hub_info`	Get hub version information and additional details about the domain catalog.
`list_hub_domains`	List all available domains for extraction, with optional filtering by category or name.
`get_hub_schema`	Get the complete JSON schema for any domain to understand the expected output structure.

Format Type	Supported Extensions
Image	`.jpg`, `.jpeg`, `.png`, `.webp`, `.bmp`, `.tiff`, `.tif`, `.heic`
Document	`.pdf`
Video	`.mp4`, `.avi`, `.mov`, `.webm`
Audio	`.mp3`, `.wav`, `.m4a`, `.aac`, `.flac`

Get Started

​Available Tools

​I/O Tools

​Image Tools

​Document Tools

​Video Tools

​Hub Tools

​Supported Inputs