Custom Schemas

vlm-1 accepts a custom schema in addition to pre-defined domains. Define fields, types, and validation rules with Pydantic or Zod. Pass the schema as json_schema on GenerationConfig. The response matches that schema.

Schemas include visual grounding and confidence scores.

1. Defining a Custom Schema

from typing import Literal
from pydantic import BaseModel, Field

class ImagePrediction(BaseModel):
    label: Literal["tv", "document", "other"] = Field(..., title="Class label for the image.")
    caption: str = Field(..., title="Caption for the image.")

2. Extracting Structured JSON from Images with a Custom Schema

from PIL import Image
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, GenerationConfig

# Initialize the client
client = VLMRun(api_key="your-api-key")

# Process the image with the custom schema
image = Image.open("path/to/image.jpg")
prediction: PredictionResponse = client.image.generate(
  images=[image],
  domain="image.caption",
  config=GenerationConfig(
    json_schema=ImagePrediction.model_json_schema()
  )
)
response_dict = prediction.response.model_dump()

3. Response Validation

try:
    image_prediction: ImagePrediction = ImagePrediction.model_validate(response_dict)
    print(f"Image classified as: {image_prediction.label}")
    print(f"Caption: {image_prediction.caption}")
except ValidationError as e:
    print(f"Validation error: {e}")

Schema best practices

See schema best practices.

Try our Image -> JSON API today

Open Image -> JSON on VLM Run. Sign up on the platform.

Overview

SDKs

CLI

CLI - Gateway

CLI - Orion

1. Defining a Custom Schema

2. Extracting Structured JSON from Images with a Custom Schema

3. Response Validation

Schema best practices

Try our Image -> JSON API today

​1. Defining a Custom Schema

​2. Extracting Structured JSON from Images with a Custom Schema

​3. Response Validation

​Schema best practices

​Try our Image -> JSON API today

1. Defining a Custom Schema

2. Extracting Structured JSON from Images with a Custom Schema

3. Response Validation

Schema best practices

Try our Image -> JSON API today