Skip to main content
Generate comprehensive, contextual captions for images using state-of-the-art vision-language models. Perfect for accessibility, content management, and automated image analysis workflows.
Image captioning example showing detailed scene description

​
Example Response

This is an example of the response from the Chat Completions API example (using the image shown above): 
A classic, light turquoise Volkswagen Beetle with chrome accents is parked on a cobblestone street, set against a warm yellow stucco wall with rustic brown wooden doors and windows.
Tags: car, volkswagen, beetle, street, cobblestone, wooden, doors, windows

{
  "caption": "A classic, light turquoise Volkswagen Beetle with chrome accents is parked on a cobblestone street, set against a warm yellow stucco wall with rustic brown wooden doors and windows.",
  "tags": ["car", "volkswagen", "beetle", "street", "cobblestone", "wooden", "doors", "windows"]
}

​
Usage Example

For best results, we recommend using the Structured Outputs API to get responses in a structured and validated data format.
from vlmrun.client import VLMRun

# Initialize the VLMRun client
client = VLMRun(api_key="<VLMRUN_API_KEY>")

# Caption the image
response = client.agent.completions.create(
    model="vlmrun-orion-1:auto",
    messages=[
        {"role": "user",
        "content": [
          {"type": "text", "text": "Generate a detailed caption for this image"},
          {"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto"}}
        ]
      }
    ],
)

# Print the response
print(response.choices[0].message.content)
# >> "A classic, light turquoise Volkswagen Beetle..."

from vlmrun.client import VLMRun
from pydantic import BaseModel, Field

# Define the response schema
class ImageCaption(BaseModel):
    caption: str = Field(..., description="Detailed caption of the scene")
    tags: list[str] = Field(..., description="Tags that describe the image")

# Initialize the VLMRun client
client = VLMRun(api_key="<VLMRUN_API_KEY>")

# Caption the image with structured output
response = client.agent.completions.create(
    model="vlmrun-orion-1:auto",
    messages=[
        {"role": "user",
        "content": [
          {"type": "text", "text": "Generate a detailed caption for this image"},
          {"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto"}}
        ]
      }
    ],
    response_format={"type": "json_schema", "schema": ImageCaption.model_json_schema()}
)

# Validate the response
result = ImageCaption.model_validate_json(response.choices[0].message.content)
# >>> ImageCaption(caption="...", tags=[...])

import { VlmRun } from "vlmrun";

// Initialize the VLMRun client
const client = new VlmRun({
  apiKey: "<VLMRUN_API_KEY>",
  baseURL: "https://api.vlm.run/v1"
});

// Caption the image
const response = await client.agent.completions.create({
  model: "vlmrun-orion-1:auto",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Generate a detailed caption for this image" },
        { type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
      ]
    }
  ]
});

console.log(response.choices[0].message.content);

import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";

// Define the response schema with Zod
const ImageCaptionSchema = z.object({
  caption: z.string().describe("Detailed caption of the scene"),
  tags: z.array(z.string()).describe("Tags that describe the image")
});

// Initialize the VLMRun client
const client = new VlmRun({
  apiKey: "<VLMRUN_API_KEY>",
  baseURL: "https://api.vlm.run/v1"
});

// Caption the image with structured output
const response = await client.agent.completions.create({
  model: "vlmrun-orion-1:auto",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Generate a detailed caption for this image" },
        { type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
      ]
    }
  ],
  response_format: {
    type: "json_schema",
    schema: zodToJsonSchema(ImageCaptionSchema)
  }
});

const result = ImageCaptionSchema.parse(JSON.parse(response.choices[0].message.content));

​
FAQ

You can ask simply ask for a more detailed caption by providing a more detailed prompt. In most cases, you can provide the number of words you want the caption to be, and the model will generate a more detailed caption.
  • Common Objects: person, car, truck, bus, bicycle, motorcycle
  • Scenes: street, building, park, forest, beach, etc.
  • Time-of-Day: morning, afternoon, evening, night
  • Weather: sunny, cloudy, rainy, snowing, etc.
The tags come in the format of a list of strings.