> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom Schemas

> Define custom schemas for visual extraction purposes.

In addition to the [pre-defined domains](/capabilities/structured-responses), **`vlm-1`** also supports custom schemas that allows you to define your own schema for a specific domain or use-case. This feature allows you to extract structured data that conforms to your specific needs and requirements, while still leveraging all the vision-based reasoning capabilities of **`vlm-1`** (see [Capabilities section](/capabilities/) for more details).

## What are Custom Schemas?

Custom schemas define the structure and validation rules for the data you want to extract from visual content. For example, you can use [Pydantic](https://docs.pydantic.dev/latest/) or [Zod](https://zod.dev/) models to specify exactly which fields you need, their types, and validation rules - passing the schema to the API will ensure our VLM will extract the data in exactly the format you defined.

## Benefits of Custom Schemas

* **Type Safety**: Enforce proper data types and validation rules
* **Flexibility**: Extract only the data you need in the format you prefer
* **Integration**: Seamlessly connect with your existing data models and systems
* **Customization**: Create domain-specific extraction rules tailored to your use case

<Tip>
  Type-based data validation for LLMs have been popularized by tools like [Instructor](https://github.com/jxnl/instructor) and [LangChain](https://python.langchain.com/docs/concepts/structured_outputs/), however we take it a step further by instrumenting new capabilities on top of your schemas such as [visual grounding](/capabilities/visual-grounding), confidence scores, [GQL querying](/capabilities/graphql) and much more.
</Tip>

## 1. Defining a Custom Schema

VLM Run has first-class support for [Pydantic](https://docs.pydantic.dev/latest/) and [Zod](https://zod.dev/), which allows you to define your schema using rich, strongly-typed Pydantic models. Here's an example of a custom schema for classifying and captioning images:

```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from typing import Literal
from pydantic import BaseModel, Field

class ImagePrediction(BaseModel):
    label: Literal["tv", "document", "other"] = Field(..., title="Class label for the image.")
    caption: str = Field(..., title="Caption for the image.")
```

## 2. Extracting Structured JSON from Images with a Custom Schema

Once you have defined your custom schema, you can use it with the VLM Run API to extract structured data that conforms to this schema. The extracted data will be validated against the schema you defined.

Here's how to use a custom schema with the Python SDK:

<CodeGroup>
  ```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  from PIL import Image
  from vlmrun.client import VLMRun
  from vlmrun.client.types import PredictionResponse, GenerationConfig

  # Initialize the client
  client = VLMRun(api_key="your-api-key")

  # Process the image with the custom schema
  image = Image.open("path/to/image.jpg")
  prediction: PredictionResponse = client.image.generate(
    images=[image],
    domain="image.caption",
    config=GenerationConfig(
      json_schema=ImagePrediction.model_json_schema()
    )
  )
  response_dict = prediction.response.model_dump()
  ```
</CodeGroup>

## 3. Response Validation

Since we've defined the schema using Pydantic, you can validate and use the extracted data as a strongly-typed object:

```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
try:
    image_prediction: ImagePrediction = ImagePrediction.model_validate(response_dict)
    print(f"Image classified as: {image_prediction.label}")
    print(f"Caption: {image_prediction.caption}")
except ValidationError as e:
    print(f"Validation error: {e}")
```

## Want to build your own schema?

If you're interested in building your own schema for a specific domain or use-case, take a look at our [schema best practices](/guides/schema/schema-best-practices) guide.

## Try our Image -> JSON API today

Head over to our [Image -> JSON](/api-reference/v1/post-image-generate) to start building your own document processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).