",
"inputs": {
"file": {
"type": "input_file",
"file_id": "dbb28d43-d741-4e0c-b25b-04ddc69b3197"
},
"supporting_document": {
"type": "file_url",
"file_url": { "url": "https://example.com/referral.pdf" }
},
"reference_image": {
"type": "image_url",
"image_url": { "url": "https://example.com/layout.png", "detail": "high" }
},
"demo_video": {
"type": "video_url",
"video_url": { "url": "https://example.com/clip.mp4" }
},
"voicemail": {
"type": "audio_url",
"audio_url": { "url": "https://example.com/voicemail.mp3" }
},
"instruction": {
"type": "text",
"text": "Schedule the patient and confirm insurance eligibility."
},
"email_details": "Hi,
Please see the attached order form for Oscar Bhujel. Kindly let us know once the appointment is scheduled.
Thank you,
Camielle Jane Lim
",
"metadata": {
"received_at": "2026-04-20T16:30:00Z",
"priority": "normal",
"source": "gmail"
}
},
"batch": true
}
```
### Minimal payload shapes
```json Document (PDF, Word, etc.) via URL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"name": ":",
"inputs": {
"file": { "type": "file_url", "file_url": { "url": "https://example.com/invoice.pdf" } }
}
}
```
```json Document via uploaded file ID theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"name": ":",
"inputs": {
"file": { "type": "input_file", "file_id": "file_abc123" }
}
}
```
```json Image + text instruction theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"name": ":",
"inputs": {
"image": { "type": "image_url", "image_url": { "url": "https://example.com/photo.jpg", "detail": "high" } },
"instruction": { "type": "text", "text": "Describe the product in the image." }
}
}
```
```json Video + reference image theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"name": ":",
"inputs": {
"video": { "type": "video_url", "video_url": { "url": "https://example.com/clip.mp4" } },
"reference": { "type": "image_url", "image_url": { "url": "https://example.com/style.jpg" } }
}
}
```
```json Audio transcription theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"name": ":",
"inputs": {
"audio": { "type": "audio_url", "audio_url": { "url": "https://example.com/meeting.mp3" } }
}
}
```
```json Uploaded file + raw string / JSON context theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"name": ":",
"inputs": {
"file": { "type": "input_file", "file_id": "dbb28d43-d741-4e0c-b25b-04ddc69b3197" },
"email_details": "Please see the attached order form. Let us know once scheduled.
",
"metadata": { "received_at": "2026-04-20T16:30:00Z", "source": "gmail" }
}
}
```
Set `config.service_tier` to control both **billing** and **request routing** — mirroring OpenAI's `service_tier` and Vertex AI's Gemini Flex/Priority offering:
* `standard` / `default` *(default)* — baseline rates and latency.
* `flex` — **0.5×** cost (50% off), higher latency. Best for batch / background workloads.
* `priority` — **1.8×** cost, lowest latency. Best for latency-sensitive, user-facing workflows.
Omitting the field (or passing `"auto"` or `null`) resolves to `standard`. See the [pricing guide](/pricing#service-tiers) for full details.
```python Python (file URL) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
!pip install vlmrun
from pathlib import Path
from pydantic import BaseModel, Field
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionResponse, AgentExecutionConfig
from vlmrun.types import MessageContent, FileUrl
# Define a Pydantic model for the execution inputs
class ExecutionInputs(BaseModel):
file: MessageContent = Field(..., description="The file to extract data from")
# Define a Pydantic model for the response
class Invoice(BaseModel):
invoice_id: str = Field(..., description="The ID of the invoice")
total_amount: float = Field(..., description="The total amount of the invoice")
client = VLMRun(api_key="")
# Upload the file to the object store
file = client.files.upload(file=Path("test.pdf"))
# Execute the agent (by name and version)
response: AgentExecutionResponse = client.agent.execute(
name=":",
inputs=ExecutionInputs(
file=MessageContent(type="file_url", file_url=FileUrl(url=file.public_url))
),
batch=True,
)
# Execute the agent (by inline prompt)
response: AgentExecutionResponse = client.agent.execute(
inputs=ExecutionInputs(
file=MessageContent(type="file_url", file_url=FileUrl(url=file.public_url))
),
config=AgentExecutionConfig(
prompt="Extract the invoice_id and total amount from the invoice.",
response_model=Invoice,
),
batch=True,
)
```
```python Python (image + text) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pydantic import BaseModel, Field
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionConfig, AgentExecutionResponse
from vlmrun.types import MessageContent, ImageUrl
class ExecutionInputs(BaseModel):
image: MessageContent = Field(..., description="Image to analyze")
instruction: MessageContent = Field(..., description="What to extract or do")
client = VLMRun(api_key="")
response: AgentExecutionResponse = client.agent.execute(
name=":",
inputs=ExecutionInputs(
image=MessageContent(
type="image_url",
image_url=ImageUrl(url="https://example.com/photo.jpg", detail="high"),
),
instruction=MessageContent(
type="text",
text="Describe the product in the image.",
),
),
batch=True,
)
```
```python Python (uploaded file) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from pydantic import BaseModel, Field
from vlmrun.client import VLMRun
from vlmrun.types import MessageContent
class ExecutionInputs(BaseModel):
file: MessageContent = Field(..., description="Uploaded file")
client = VLMRun(api_key="")
file = client.files.upload(file=Path("invoice.pdf"))
response = client.agent.execute(
name=":",
inputs=ExecutionInputs(
file=MessageContent(type="input_file", file_id=file.id),
),
batch=True,
)
```
```javascript Node.js (file URL) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
npm install vlmrun zod
import { VlmRun } from "vlmrun";
const client = new VlmRun({
baseUrl: "https://api.vlm.run/v1",
apiKey: "",
});
// Execute the agent with a file URL
const response = await client.agent.execute({
name: ":",
inputs: {
file: {
type: "file_url",
file_url: { url: "https://example.com/invoice.pdf" },
},
},
batch: true,
});
```
```javascript Node.js (image + text) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
baseUrl: "https://api.vlm.run/v1",
apiKey: "",
});
const response = await client.agent.execute({
name: ":",
inputs: {
image: {
type: "image_url",
image_url: { url: "https://example.com/photo.jpg", detail: "high" },
},
instruction: {
type: "text",
text: "Describe the product in the image.",
},
},
batch: true,
});
```
```bash cURL (document URL) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/agent/execute \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": ":",
"inputs": {
"file": {
"type": "file_url",
"file_url": { "url": "https://example.com/invoice.pdf" }
}
},
"batch": true
}'
```
```bash cURL (uploaded file) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/agent/execute \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": ":",
"inputs": {
"file": { "type": "input_file", "file_id": "file_abc123" }
},
"batch": true
}'
```
```bash cURL (multi-modal: image + text) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/agent/execute \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": ":",
"inputs": {
"image": {
"type": "image_url",
"image_url": { "url": "https://example.com/photo.jpg", "detail": "high" }
},
"instruction": {
"type": "text",
"text": "Describe the product in the image."
}
},
"batch": true
}'
```
## OpenAPI
````yaml POST /v1/agent/execute
openapi: 3.1.0
info:
title: VLM Run Unified Server
description: Unified server for VLM Run Agent and API
termsOfService: https://vlm.run/terms-of-service
contact:
name: VLM Run Support Team
url: https://vlm.run/
email: support@vlm.run
version: 2026-05-19.0
servers: []
security: []
paths:
/v1/agent/execute:
post:
tags:
- agent
summary: ' Execute Agent'
operationId: _execute_agent_v1_agent_execute_post
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/AgentExecutionRequest'
required: true
responses:
'200':
description: Successful Response
content:
application/json:
schema:
$ref: '#/components/schemas/AgentExecutionResponse'
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
security:
- HTTPBearer: []
components:
schemas:
AgentExecutionRequest:
properties:
metadata:
$ref: '#/components/schemas/RequestMetadata'
description: Optional metadata to pass to the model.
config:
$ref: '#/components/schemas/AgentExecutionConfig'
description: The configuration for the agent execution request.
id:
type: string
title: Id
description: Unique identifier of the request.
created_at:
type: string
format: date-time
title: Created At
description: Date and time when the request was created (in UTC timezone)
callback_url:
anyOf:
- type: string
minLength: 1
format: uri
- type: 'null'
title: Callback Url
description: The URL to call when the request is completed.
model:
anyOf:
- type: string
enum:
- vlmrun-orion-1
- vlmrun-orion-1:lite
- vlmrun-orion-1:auto
- vlmrun-orion-1:fast
- vlmrun-orion-1:pro
- vlmrun-orion-2
- vlmrun-orion-2:lite
- vlmrun-orion-2:auto
- vlmrun-orion-2:qwen3.6-35b-a3b
- vlmrun-orion-2:gemma4-26b-a4b
- vlmrun-orion-2:kimi-2.6
- vlmrun-orion-2:gpt-5.5
- vlmrun-orion-2:opus-4.8
- vlmrun-orion-2:gemini-flash-3.5
- vlmrun-orion-2:fast
- vlmrun-orion-2:pro
- type: 'null'
title: Model
description: >-
VLM Run Agent model to use for execution. When omitted, the skill's
vlmrun.yaml model is used; otherwise the agent default.
name:
anyOf:
- type: string
- type: 'null'
title: Name
description: >-
Name of the agent. If not provided, we use the prompt to identify
the unique agent.
batch:
type: boolean
title: Batch
description: Whether to process the document in batch mode (async).
default: true
inputs:
anyOf:
- $ref: '#/components/schemas/AgentExecutionInputs'
- type: 'null'
description: The inputs to the agent.
toolsets:
anyOf:
- items:
$ref: '#/components/schemas/AgentToolset'
type: array
- type: 'null'
title: Toolsets
description: >-
List of tool categories to enable for this agent execution.
Available categories: core, image, image-gen, world_gen, viz,
document, video, web. When specified, only tools from these
categories will be available. If None, defaults to 'core' tools
only.
models:
anyOf:
- items:
$ref: '#/components/schemas/AgentModel'
type: array
- type: 'null'
title: Models
description: >-
List of model-specific tool providers to enable for this execution.
Available models: depth-anything-3, google-gemini-3-analysis,
google-gemini-3-image, google-gemini-robotics-er, google-veo-3.1,
meta-sam2, meta-sam3, meta-sam3d, microsoft-omniparser-v2,
nvidia-cosmos-reason-2-8b, qwen-qwen3-vl-8b, vlm-dots-ocr. Multiple
models can be selected — their tools are merged.
type: object
title: AgentExecutionRequest
description: Request to execute an agent.
AgentExecutionResponse:
properties:
usage:
$ref: '#/components/schemas/CreditUsageResponse'
description: The usage metrics for the request.
id:
type: string
title: Id
description: Unique identifier of the agent execution response.
name:
type: string
title: Name
description: Name of the agent
response:
anyOf:
- {}
- type: 'null'
title: Response
description: The response from the model.
status:
type: string
enum:
- pending
- enqueued
- running
- completed
- failed
- paused
title: Status
description: The status of the job.
default: pending
created_at:
type: string
format: date-time
title: Created At
description: Date and time when the execution was created (in UTC timezone)
completed_at:
anyOf:
- type: string
format: date-time
- type: 'null'
title: Completed At
description: Date and time when the execution was completed (in UTC timezone)
type: object
required:
- name
title: AgentExecutionResponse
description: Response to the agent execution request.
HTTPValidationError:
properties:
detail:
items:
$ref: '#/components/schemas/ValidationError'
type: array
title: Detail
type: object
title: HTTPValidationError
RequestMetadata:
properties:
environment:
type: string
enum:
- dev
- staging
- prod
title: Environment
description: The environment where the request was made.
default: dev
session_id:
anyOf:
- type: string
- type: 'null'
title: Session Id
description: The session ID of the request
allow_logging:
type: boolean
title: Allow Logging
description: Whether to enable logs for this request.
default: true
allow_training:
type: boolean
title: Allow Training
description: Whether the file can be used for training
default: true
allow_retention:
type: boolean
title: Allow Retention
description: Whether to allow retention of the data
default: true
extra:
additionalProperties: true
type: object
title: Extra
description: Extra metadata for the request (e.g. `dataset_id`, `subset_id`).
type: object
title: RequestMetadata
description: >-
Metadata for the request.
Typically captured in {"vlmrun": {"metadata": {"environment":
, ...}}.
AgentExecutionConfig:
properties:
prompt:
anyOf:
- type: string
- type: 'null'
title: Prompt
description: The prompt to guide the execution of the agent.
json_schema:
anyOf:
- additionalProperties: true
type: object
- type: 'null'
title: Json Schema
description: The JSON schema to the agent
tools_type:
anyOf:
- type: string
enum:
- document
- image
- video
- multimodal
- type: 'null'
title: Tools Type
description: The type of tools to use for the agent
skills:
anyOf:
- items:
$ref: '#/components/schemas/AgentSkill'
type: array
- type: 'null'
title: Skills
description: >-
List of agent skills to enable for this execution. Skills provide
domain-specific expertise and capabilities.
selected_tools:
anyOf:
- items:
type: string
type: array
- type: 'null'
title: Selected Tools
description: >-
List of tool names to use for this agent execution. If provided,
only these tools will be loaded. Tool names should match function
names exactly.
use_context_cache:
type: boolean
title: Use Context Cache
description: >-
Reuse cached representations of large document/video inputs across
calls in the same session to reduce input-token cost and latency.
default: true
service_tier:
anyOf:
- type: string
enum:
- auto
- default
- standard
- flex
- priority
- type: 'null'
title: Service Tier
description: >-
Delivery tier for the agent run. `auto`/`default`/`None` resolves to
`standard` (baseline 1.0× billing); `flex` is 0.5× billing with
higher latency, and `priority` is 1.8× billing with a premium
latency SLO.
type: object
title: AgentExecutionConfig
description: Configuration for the agent execution request.
AgentExecutionInputs:
properties: {}
additionalProperties: true
type: object
title: AgentExecutionInputs
description: |-
Base class for agent execution inputs.
Users can subclass this to create typed input models for their agents.
For example:
class MyAgentInputs(AgentExecutionInputs):
image: MessageContent
prompt: str
request = AgentExecutionRequest(
name="my-agent",
inputs=MyAgentInputs(image=..., prompt=...)
)
AgentToolset:
type: string
enum:
- core
- code-execution
- document
- image
- image-gen
- video
- viz
- web
- world-gen
title: AgentToolset
description: |-
Available toolsets for agent tool selection.
Each toolset represents a category of related tools that can be enabled
together for an agent execution.
AgentModel:
type: string
enum:
- google-gemini-3-image
- google-gemini-3-analysis
- google-gemini-robotics-er
- google-veo-3.1
- microsoft-omniparser-v2
- qwen-qwen3-vl-8b
- meta-sam2
- meta-sam3
- meta-sam3d
- depth-anything-3
- vlm-dots-ocr
- nvidia-cosmos-reason-2-8b
title: AgentModel
description: |-
Available models for agent tool selection.
Each model represents a specialized capability backed by a specific
model deployment. Multiple models can be selected simultaneously —
pass a list and the tools are merged and deduplicated.
Usage in vlmrun.yaml::
model: vlmrun-orion-1:auto
toolsets:
- core
- image
models:
- nvidia-cosmos-reason-2-8b
- meta-sam3
CreditUsageResponse:
properties:
elements_processed:
anyOf:
- type: integer
- type: 'null'
title: Elements Processed
description: Number of elements processed.
element_type:
anyOf:
- type: string
enum:
- image
- page
- video
- audio
- type: 'null'
title: Element Type
description: The type of element processed (e.g. image, page, video, audio).
credits_used:
anyOf:
- type: integer
- type: 'null'
title: Credits Used
description: Amount of total credits used.
steps:
anyOf:
- type: integer
- type: 'null'
title: Steps
description: Number of steps processed, in case of agentic execution.
message:
anyOf:
- type: string
- type: 'null'
title: Message
description: The message from the credit usage job.
duration_seconds:
type: integer
title: Duration Seconds
description: Duration of the request in seconds.
default: 0
service_tier:
anyOf:
- type: string
- type: 'null'
title: Service Tier
description: Delivery tier (standard, priority, flex).
mode_multiplier:
anyOf:
- type: number
- type: 'null'
title: Mode Multiplier
description: Pricing multiplier applied to standard cost.
standard_cost_dollars:
anyOf:
- type: number
- type: 'null'
title: Standard Cost Dollars
description: >-
Pre-multiplier customer cost in USD, derived from tokens and unit
rates.
cost_dollars:
anyOf:
- type: number
- type: 'null'
title: Cost Dollars
description: Effective customer cost in USD after the service-tier multiplier.
savings_dollars:
anyOf:
- type: number
- type: 'null'
title: Savings Dollars
description: >-
Discount in USD when using flex (standard_cost_dollars -
cost_dollars).
type: object
title: CreditUsageResponse
description: Response model for credit usage metrics.
ValidationError:
properties:
loc:
items:
anyOf:
- type: string
- type: integer
type: array
title: Location
msg:
type: string
title: Message
type:
type: string
title: Error Type
input:
title: Input
ctx:
type: object
title: Context
type: object
required:
- loc
- msg
- type
title: ValidationError
AgentSkill:
properties:
type:
type: string
title: Type
description: >-
The type of the skill. Use 'skill_reference' for DB-stored skills
referenced by id/name. Use 'inline' to provide the skill as a
base64-encoded zip bundle.
default: skill_reference
skill_id:
anyOf:
- type: string
- type: 'null'
title: Skill Id
description: >-
The unique identifier of the skill — a UUID or a name string (e.g.,
'pillow', 'batch-processing').
skill_name:
anyOf:
- type: string
- type: 'null'
title: Skill Name
description: >-
Human-readable skill name for lookup (e.g., 'invoice-extraction').
Alternative to skill_id. Deprecated in favour of skill_id.
skill_version:
anyOf:
- type: integer
- type: string
title: Skill Version
description: The version of the skill — an integer (e.g. 2) or 'latest'.
default: latest
version:
anyOf:
- type: integer
- type: string
- type: 'null'
title: Version
description: 'DEPRECATED: Use ''skill_version'' instead. The version of the skill.'
name:
anyOf:
- type: string
- type: 'null'
title: Name
description: >-
Human-readable name for the inline skill (used for discovery and
logging).
description:
anyOf:
- type: string
- type: 'null'
title: Description
description: Short description of what the inline skill does.
source:
anyOf:
- $ref: '#/components/schemas/InlineSkillSource'
- type: 'null'
description: >-
Source payload for inline skills. Contains the base64-encoded zip
bundle with type, media_type, and data fields.
bundle:
anyOf:
- type: string
- type: 'null'
title: Bundle
description: >-
DEPRECATED: Use 'source.data' instead. Base64-encoded zip bundle
containing the skill files (inline skills only).
type: object
title: AgentSkill
description: >-
A modular capability that extends the agent's functionality.
Agent Skills are reusable, filesystem-based resources that provide the
agent
with domain-specific expertise: workflows, context, and best practices.
Each skill packages instructions, metadata, and optional resources
(scripts,
templates, snippets) that the agent uses automatically when relevant.
Two modes are supported:
1. **Referenced skills** (``type="skill_reference"``) – Provide
``skill_id``
(UUID or name) and optionally ``skill_version`` (integer or ``"latest"``).
.. code-block:: json
{"type": "skill_reference", "skill_id": "pillow", "skill_version": "latest"}
2. **Inline skills** (``type="inline"``) – Supply ``name``,
``description``,
and a ``source`` object containing the base64-encoded zip bundle. The zip
must contain exactly one ``SKILL.md`` file. No database lookup is required.
.. code-block:: json
{
"type": "inline",
"name": "csv-insights",
"description": "Summarize CSV files.",
"source": {
"type": "base64",
"media_type": "application/zip",
"data": ""
}
}
Legacy format with flat ``bundle`` field is also accepted for backward
compatibility.
InlineSkillSource:
properties:
type:
type: string
const: base64
title: Type
description: >-
Encoding type for the inline skill data. Currently only 'base64' is
supported.
default: base64
media_type:
type: string
title: Media Type
description: MIME type of the skill bundle. Must be 'application/zip'.
default: application/zip
data:
type: string
title: Data
description: Base64-encoded zip bundle containing the skill files.
type: object
required:
- data
title: InlineSkillSource
description: |-
Source payload for an inline skill bundle.
Follows the OpenAI inline skill format::
{
"type": "base64",
"media_type": "application/zip",
"data": ""
}
securitySchemes:
HTTPBearer:
type: http
scheme: bearer
````