Skip to main content
Foundation vision models support chat over visual inputs, but automation needs reliable, machine-validated output. Agent chat completions let you define the expected structure up front and get consistently formatted JSON back – either loosely with json_object or strictly via json_schema.

​
Structured Responses with Orion

Here’s an example of using the agent chat completions endpoint to extract typed JSON directly from user prompts and files. 
import datetime
from pydantic import BaseModel, Field
from vlmrun.client import VLMRun

client = VLMRun(
    base_url="https://agent.vlm.run/v1", api_key="<VLMRUN_API_KEY>",
)

class Invoice(BaseModel):
    invoice_number: str = Field(description="The number of the invoice")
    invoice_date: datetime.date = Field(description="The date of the invoice")
    total_amount: float = Field(description="The total amount of the invoice")
    vendor_name: str = Field(description="The name of the vendor")

# Ask the agent for structured output using a strict JSON Schema
response = client.agent.completions.create(
    ... # Same as above
    response_format={"type": "json_schema", "json_schema": Invoice.model_json_schema()},
)

# Validate and print the response
invoice = Invoice.model_validate_json(response.choices[0].message.content)
print(invoice)
>>> Invoice(invoice_number="INV-2024-001", date="2024-09-15", total_amount=1250.00, vendor_name="Acme Corporation")

​
Structured Response Formats

Invoice(
  invoice_number="INV-2024-001",
  invoice_date=datetime.date(2024, 9, 15),
  total_amount=1250.00,
  vendor_name="Acme Corporation"
)

​
Response Format Types

Similar to OpenAI’s json_object and json_schema response formats, you can use the json_object or json_schema response format types to extract structured JSON from the agent’s response.
TypeDescription
json_objectValid JSON object without specific schema
json_schemaStrict JSON conforming to provided schema

​
Best Practices

  • Clear system prompts: Define role and output format in the system message
  • Prefer schemas for automation: Use json_schema for guaranteed structure
  • Control randomness: Use lower temperature (0.0–0.3) for deterministic outputs
  • Validate responses: Parse/validate JSON and handle errors gracefully
  • Keep history concise: Shorter message histories improve latency and reliability