In addition to the pre-defined domains, vlm-1 also supports custom schemas that allows you to define your own schema for a specific domain or use-case. This feature allows you to extract structured data that conforms to your specific needs and requirements, while still leveraging all the vision-based reasoning capabilities of vlm-1 (see Capabilities section for more details).

​

What are Custom Schemas?

Custom schemas define the structure and validation rules for the data you want to extract from visual content. For example, you can use Pydantic or Zod models to specify exactly which fields you need, their types, and validation rules - passing the schema to the API will ensure our VLM will extract the data in exactly the format you defined.

​

Benefits of Custom Schemas

• Type Safety: Enforce proper data types and validation rules
• Flexibility: Extract only the data you need in the format you prefer
• Integration: Seamlessly connect with your existing data models and systems
• Customization: Create domain-specific extraction rules tailored to your use case

​

1. Defining a Custom Schema

VLM Run has first-class support for Pydantic and Zod, which allows you to define your schema using rich, strongly-typed Pydantic models. Here’s an example of a custom schema for classifying and captioning images:

from typing import Literal
from pydantic import BaseModel, Field

class ImagePrediction(BaseModel):
    label: Literal["tv", "document", "other"] = Field(..., title="Class label for the image.")
    caption: str = Field(..., title="Caption for the image.")

​

2. Extracting Structured JSON from Images with a Custom Schema

Once you have defined your custom schema, you can use it with the VLM Run API to extract structured data that conforms to this schema. The extracted data will be validated against the schema you defined.

Here’s how to use a custom schema with the Python SDK:

from PIL import Image
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, GenerationConfig

# Initialize the client
client = VLMRun(api_key="your-api-key")

# Process the image with the custom schema
image = Image.open("path/to/image.jpg")
prediction: PredictionResponse = client.image.generate(
  images=[image],
  domain="image.caption",
  config=GenerationConfig(
    json_schema=ImagePrediction.model_json_schema()
  )
)
response_dict = prediction.response.model_dump()

​

3. Response Validation

Since we’ve defined the schema using Pydantic, you can validate and use the extracted data as a strongly-typed object:

try:
    image_prediction: ImagePrediction = ImagePrediction.model_validate(response_dict)
    print(f"Image classified as: {image_prediction.label}")
    print(f"Caption: {image_prediction.caption}")
except ValidationError as e:
    print(f"Validation error: {e}")

​

Want to build your own schema?

If you’re interested in building your own schema for a specific domain or use-case, take a look at our schema best practices guide.

​

Try our Image -> JSON API today

Head over to our Image -> JSON to start building your own document processing pipeline with VLM Run. Sign-up for access on our platform.