Introduction

​

What is Orion?

• Sees: reads pixels with dedicated vision models, including detection, segmentation, and specialized document OCR VLMs. It does not guess from a caption.
• Reasons: decides which tools to run and in what order.
• Acts: runs the tools and hands back something you can use: validated JSON, a cropped or redacted image, a generated video.

• vlmrun-orion-1: our first visual agent, released in November 2025. It calls one tool at a time.
• vlmrun-orion-2: our second generation. It writes and runs code, so one program runs a whole pipeline. It is faster, cheaper, and more capable, with high reliability and repeatability.

​

What Orion does

• Documents: layout detection, visual grounding, and multi-page analysis.
• Images: caption and tag, detection, segmentation, pointing, and UI parsing.
• Video: caption and tag and video tools.
• Create and edit: generate and edit images, crop, rotate, upscale, and colorize, and generate video.
• Structured output: JSON that validates against your schema, and artifacts for generated files.

​

Agents Supported

• vlmrun-orion-2: writes and runs code in a sandbox.
• vlmrun-orion-1: calls tools with structured tool calls.

• fast: speed and cost. Lighter models for short, simple workflows.
• auto: the default. Picks the model, tool, and thinking budget from the task.
• pro: maximum quality. A high thinking budget for long pipelines and tool trajectories.

​

How it works

1. You ask. Send a request in natural language with your media. See inputs.
2. Orion plans. The agent picks the tools and the order.
3. Orion runs the tools. OCR, detection, segmentation, and the rest run as tools and return exact results.
4. You get the result. Text, validated JSON, or generated files.

from vlmrun.client import VLMRun

client = VLMRun(api_key="<VLMRUN_API_KEY>")

response = client.agent.completions.create(
    model="vlmrun-orion-2:auto",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "Detect the car and describe its color."},
        {"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"}},
    ]}],
)
print(response.choices[0].message.content)

​

Next

• Code Execution: when to use Orion-2, and how it runs.
• Chat: try an agent in the browser.
• SDK: call an agent from code.