> ## Documentation Index > Fetch the complete documentation index at: https://docs.vlm.run/llms.txt > Use this file to discover all available pages before exploring further. # Pydantic AI Compatibility > Use Orion-2 with Pydantic AI for document, image, and video work with hybrid client tools. Orion-2 works with [Pydantic AI](https://ai.pydantic.dev/) as the model behind an `Agent`. Point `OpenAIChatModel` at the VLM Run OpenAI-compatible endpoint, enable hybrid tool execution, and Pydantic AI keeps running your local tools while Orion-2 does the document, image, and video work in its [code-execution sandbox](/agents/code-execution). ## What works With Pydantic AI as the orchestrator and Orion-2 as the model, the following are supported: | Capability | Who runs it | Example | | -------------------- | --------------------- | --------------------------------------------------------------- | | Document extraction | Orion-2 | Invoice fields, filing summary, page classification | | Video understanding | Orion-2 | Timestamped scene summary, per-segment object counts | | Image pipelines | Orion-2 | Detect, crop, annotate, and count in one `execute_code` program | | Client / local tools | Pydantic AI | ERP lookups, database queries, internal APIs | | Streaming | Pydantic AI + Orion-2 | `agent.iter(...)` against Orion-2 | Hybrid mode is required so Pydantic AI executes client tools locally while Orion-2 keeps server tools such as `execute_code`. ## Prerequisites * A VLM Run API key from the [API Keys](https://app.vlm.run/dashboard/settings/api-keys) page * Python with `pydantic-ai` ```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}} pip install "pydantic-ai>=1.100" export VLMRUN_API_KEY="" ``` ## Configure Pydantic AI with Orion-2 Use the OpenAI-compatible chat endpoint, an Orion-2 model id, and a model profile tuned for the VLM Run wire format: ```python theme={"theme":{"light":"github-light","dark":"dark-plus"}} import os from pydantic_ai.models.openai import OpenAIChatModel from pydantic_ai.profiles.openai import OpenAIModelProfile from pydantic_ai.providers.openai import OpenAIProvider model = OpenAIChatModel( "vlmrun-orion-2:auto", provider=OpenAIProvider( base_url="https://api.vlm.run/v1/openai", api_key=os.environ["VLMRUN_API_KEY"], ), profile=OpenAIModelProfile( openai_supports_strict_tool_definition=False, openai_unsupported_model_settings=("max_completion_tokens",), openai_chat_supports_document_input=True, ), ) ``` | Profile setting | Why it is needed | | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | | `openai_supports_strict_tool_definition=False` | VLM Run serves tools through an OpenAI-compatible proxy, which does not require strict tool schemas | | `openai_unsupported_model_settings=("max_completion_tokens",)` | Keeps Pydantic AI from sending a parameter Orion-2 does not accept | | `openai_chat_supports_document_input=True` | Allows document inputs on chat completions instead of rejecting them client-side | ### Map media inputs to VLM Run content parts Pydantic AI's stock Chat Completions mapping sends `DocumentUrl` as an OpenAI `file` part and raises `NotImplementedError` for `VideoUrl`. VLM Run expects `file_url` and `video_url` parts, as described in [Multi-modal Inputs](/agents/inputs), so override those two mappers once at setup. `ImageUrl` needs no patch: it already maps to `image_url`. ```python theme={"theme":{"light":"github-light","dark":"dark-plus"}} from pydantic_ai import DocumentUrl, VideoUrl def use_vlmrun_media_parts(model: OpenAIChatModel) -> None: """Send Pydantic AI media inputs as VLM Run content parts.""" async def _document(item: DocumentUrl): return {"type": "file_url", "file_url": {"url": item.url, "detail": "auto"}} async def _video(item: VideoUrl): return {"type": "video_url", "video_url": {"url": item.url, "detail": "auto"}} model._map_document_url_item = _document model._map_video_url_item = _video use_vlmrun_media_parts(model) ``` Without this, a document prompt never becomes a sandbox document ref, so `execute_code` has nothing to read. The overrides replace internal Pydantic AI mappers, so pin the version you tested against. ### Define the agent and its local tools Keep client tools for the systems only your process can reach. Anything that requires looking at pixels or pages belongs to Orion-2: ```python theme={"theme":{"light":"github-light","dark":"dark-plus"}} from pydantic_ai import Agent, RunContext agent = Agent( model, system_prompt=( "You are a document and video analyst. Use the server execute_code tool " "for anything involving the attached document, image, or video, and report " "only values it returns. Call lookup_purchase_order exactly once only when " "the user explicitly asks you to reconcile an invoice against the ERP. If " "its result is already in the conversation, answer from that result and do " "not call it again. Never invent field values." ), ) @agent.tool def lookup_purchase_order(ctx: RunContext[None], invoice_number: str) -> dict: """Return the purchase order your ERP has on file for an invoice number.""" # Replace with a real query against your system of record. return {"invoice_number": invoice_number, "po_number": "PO-4417", "total": 4560.00} ``` Use `model: "vlmrun-orion-2:auto"` for the default Orion-2 tier. Other tiers and pinned variants are listed in [Code Execution](/agents/code-execution#model-variants). ## Hybrid tool execution Orion-2 needs two extra body fields on every request. Pass them through `model_settings`: ```python theme={"theme":{"light":"github-light","dark":"dark-plus"}} import uuid def hybrid_run_kwargs() -> dict: return { "model_settings": { "extra_body": { "session_id": str(uuid.uuid4()), "tool_execution": "hybrid", } } } ``` | Mode | Behavior | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `tool_execution: "hybrid"` | Orion-2 runs server tools such as `execute_code` in its own agent loop and pauses client tool calls back to Pydantic AI | | Without hybrid | Tools pass straight through to the model, and client tool round-trips may not complete | Send a fresh `session_id` (UUID) per turn so Orion-2 can keep workspace and artifact context. See [Artifacts](/agents/artifacts) for retrieving generated files. ## Scenarios Each scenario uses the same agent, the same hybrid contract, and a public VLM Run sample file. ### 1. Invoice extraction from a PDF Orion-2 pulls the document into its sandbox and extracts fields with code. Pydantic AI stays the orchestrator and runs no local tools. ```python theme={"theme":{"light":"github-light","dark":"dark-plus"}} INVOICE_PDF = ( "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/" "document.invoice/wordpress-pdf-invoice-plugin-sample.pdf" ) result = agent.run_sync( [ "Extract the vendor, invoice number, invoice date, and total from this " "invoice. Use execute_code with the attached document ref.", DocumentUrl(INVOICE_PDF), ], **hybrid_run_kwargs(), ) print(result.output) ``` **Expected:** the four invoice fields, sourced from a single `execute_code` pass on the server, with no client tool calls. ### 2. Video understanding Video works the same way once `VideoUrl` maps to a `video_url` part. Prefer streaming for video: these runs often take one to two minutes, and some VLM Run keys only allow `stream=True`. Use the `run_streamed` helper from the [Streaming](#streaming) section below. ```python theme={"theme":{"light":"github-light","dark":"dark-plus"}} import asyncio BAKERY_VIDEO = ( "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/" "video.transcription/bakery.mp4" ) output = asyncio.run( run_streamed( agent, [ "Summarize this video segment by segment. For each segment, give the start " "and end timestamp and one sentence describing what happens.", VideoUrl(BAKERY_VIDEO), ], hybrid_run_kwargs(), ) ) ``` **Expected:** a timestamped segment summary produced by `execute_code`, with no client tool calls. For longer footage, ask for a fixed segment length so the agent captions in even windows. ### 3. Extraction reconciled against your system of record This is where hybrid earns its place. Orion-2 reads the invoice, Pydantic AI runs the ERP lookup in your process, and neither side needs the other's credentials. ```python theme={"theme":{"light":"github-light","dark":"dark-plus"}} result = agent.run_sync( [ "Extract the invoice number and total from this invoice with execute_code. " "Then call lookup_purchase_order exactly once with that invoice number. " "Compare the two totals and say whether they match. Do not call the tool again.", DocumentUrl(INVOICE_PDF), ], **hybrid_run_kwargs(), ) ``` **Expected:** one `execute_code` pass on the server, one `lookup_purchase_order` call inside your process, and a match or mismatch verdict that cites both totals. ### 4. Image pipeline with an annotated result Images need no patch, so an `ImageUrl` prompt goes straight through as an `image_url` part. ```python theme={"theme":{"light":"github-light","dark":"dark-plus"}} from pydantic_ai import ImageUrl DONUTS_IMAGE = ( "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/" "image.object-detection/donuts.png" ) result = agent.run_sync( [ "Detect every donut in this image, draw bounding boxes, and return the " "count along with the annotated image.", ImageUrl(DONUTS_IMAGE), ], **hybrid_run_kwargs(), ) ``` **Expected:** a count plus an image ref for the annotated output. Fetch the image with the `session_id` you sent, as shown in [Artifacts](/agents/artifacts). ## Streaming Some VLM Run keys only permit chat completions with `stream=True`, and `run_sync` issues non-streaming requests. Use `agent.iter` to stream every model request in the run, including the tool-call turn and the final answer. Cap the request budget with `UsageLimits` so a loose client-tool instruction cannot loop until Pydantic AI aborts: ```python theme={"theme":{"light":"github-light","dark":"dark-plus"}} import asyncio from pydantic_ai.usage import UsageLimits async def run_streamed(agent: Agent, prompt, run_kwargs: dict) -> str: async with agent.iter( prompt, usage_limits=UsageLimits(request_limit=20), **run_kwargs, ) as run: async for node in run: if Agent.is_model_request_node(node): async with node.stream(run.ctx) as request_stream: async for _ in request_stream: pass return run.result.output output = asyncio.run( run_streamed( agent, ["Summarize this invoice.", DocumentUrl(INVOICE_PDF)], hybrid_run_kwargs(), ) ) ``` ## Related Orion-2 sandbox, libraries, and model variants The same hybrid contract from a TypeScript agent Document, image, and video content parts Retrieve annotated images, clips, and generated files