Cataloging Images

vlm-1 returns captions, tags, and descriptions as structured fields from one image.

Cataloging Product Images

The example uses ashraq/fashion-product-images-small.

Preview of the 'fashion-product-images-small' dataset from Huggingface.

1. Define a custom schema for cataloging

vlm-1 fills each field from the image.

from typing import Literal
from pydantic import BaseModel, Field

class ProductCatalog(BaseModel):
    description: str = Field(..., description="A 2-sentence general visual description of the product embedded as an image.")
    category: str = Field(..., description="One or two-word category of the product (i.e, Apparel, Accessories, Footwear etc).")
    season: Literal["Fall", "Spring", "Summer", "Winter"] = Field(..., description="The season the product is intended for.")
    gender: Literal["Men", "Women", "Kids"] = Field(..., description="Gender or audience the product is intended for.")

2. Extract cataloging information from images

The API validates the response against the schema. Use the REST API, or the OpenAI Python SDK with the OpenAI-Compatible API.

from PIL import Image
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, GenerationConfig

# Initialize the client
client = VLMRun(api_key="<VLMRUN_API_KEY>")

# Load the first image from the dataset and encode it as base64
ds = load_dataset("ashraq/fashion-product-images-small", split="train[:1%]")
image: Image.Image = next(ds["image"])

# Predict the product catalog information from the image
response: PredictionResponse = client.image.generate(
  images=[image],
  domain="retail.product-catalog",
  config=GenerationConfig(
    json_schema=ProductCatalog.model_json_schema(),
  )
)
response_dict = response.model_dump()

Example Product Cataloging Prediction

Sample output for the first image, a navy plaid shirt.

{
  "description": "A casual, button-up plaid shirt with short sleeves in a light fabric. The shirt features a combination of blue and white colors in a checkered pattern.",
  "category": "Apparel",
  "season": "Summer",
  "gender": "Men"
}

Field	Task
`description`	Captioning
`category`	Classification
`season`	Classification. `Literal`: Fall, Spring, Summer, Winter.
`gender`	Classification. `Literal`: Men, Women, Kids.

Cataloging larger image catalogs

Use the same call for multiple images. Store the JSON in MongoDB, Elasticsearch, or SQL.

`vlm-1` predictions for the fashion dataset.

Fine-tuning for custom cataloging

Fine-tune on your own data. Follow the fine-tuning guides.

This feature is currently only available for our enterprise-tier customers. If you are interested in using this feature, please contact us.

Try our Image -> JSON API today

Open Image -> JSON on VLM Run. Sign up on the platform.

Gateway

Documents

Images

Video and audio

Cataloging Images

Cataloging Product Images

1. Define a custom schema for cataloging

2. Extract cataloging information from images

Example Product Cataloging Prediction

Cataloging larger image catalogs

Fine-tuning for custom cataloging

Try our Image -> JSON API today

​Cataloging Product Images

​1. Define a custom schema for cataloging

​2. Extract cataloging information from images

​Example Product Cataloging Prediction

​Cataloging larger image catalogs

​Fine-tuning for custom cataloging

​Try our Image -> JSON API today

Cataloging Product Images

1. Define a custom schema for cataloging

2. Extract cataloging information from images

Example Product Cataloging Prediction

Cataloging larger image catalogs

Fine-tuning for custom cataloging

Try our Image -> JSON API today