Example image to be captioned.
Example Response
This is an example of the response from theChat Completions API example (using the image shown above):
A classic, light turquoise Volkswagen Beetle with chrome accents is parked on a cobblestone street, set against a warm yellow stucco wall with rustic brown wooden doors and windows.
Tags: car, volkswagen, beetle, street, cobblestone, wooden, doors, windows
{
"caption": "A classic, light turquoise Volkswagen Beetle with chrome accents is parked on a cobblestone street, set against a warm yellow stucco wall with rustic brown wooden doors and windows.",
"tags": ["car", "volkswagen", "beetle", "street", "cobblestone", "wooden", "doors", "windows"]
}
Usage Example
For best results, we recommend using the Structured Outputs API to get responses in a structured and validated data format.
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="<VLMRUN_API_KEY>")
# Caption the image
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{"role": "user",
"content": [
{"type": "text", "text": "Generate a detailed caption for this image"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto"}}
]
}
],
)
# Print the response
print(response.choices[0].message.content)
# >> "A classic, light turquoise Volkswagen Beetle..."
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class ImageCaption(BaseModel):
caption: str = Field(..., description="Detailed caption of the scene")
tags: list[str] = Field(..., description="Tags that describe the image")
# Initialize the VLMRun client
client = VLMRun(api_key="<VLMRUN_API_KEY>")
# Caption the image with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{"role": "user",
"content": [
{"type": "text", "text": "Generate a detailed caption for this image"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": ImageCaption.model_json_schema()}
)
# Validate the response
result = ImageCaption.model_validate_json(response.choices[0].message.content)
# >>> ImageCaption(caption="...", tags=[...])
import { VlmRun } from "vlmrun";
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "<VLMRUN_API_KEY>",
baseURL: "https://api.vlm.run/v1"
});
// Caption the image
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Generate a detailed caption for this image" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const ImageCaptionSchema = z.object({
caption: z.string().describe("Detailed caption of the scene"),
tags: z.array(z.string()).describe("Tags that describe the image")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "<VLMRUN_API_KEY>",
baseURL: "https://api.vlm.run/v1"
});
// Caption the image with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Generate a detailed caption for this image" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(ImageCaptionSchema)
}
});
const result = ImageCaptionSchema.parse(JSON.parse(response.choices[0].message.content));
FAQ
How do I ask the model for more detailed captions?
How do I ask the model for more detailed captions?
You can ask simply ask for a more detailed caption by providing a more detailed prompt. In most cases, you can provide the number of words you want the caption to be, and the model will generate a more detailed caption.
What tags are supported?
What tags are supported?
- Common Objects: person, car, truck, bus, bicycle, motorcycle
- Scenes: street, building, park, forest, beach, etc.
- Time-of-Day: morning, afternoon, evening, night
- Weather: sunny, cloudy, rainy, snowing, etc.
What format do the tags come in?
What format do the tags come in?
The tags come in the format of a list of strings.