> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.
# Caption & Tag
> Generate detailed captions and tags for videos using advanced vision models.
Generate comprehensive, contextual captions for videos using state-of-the-art vision-language models. Perfect for accessibility, content management, and automated video analysis workflows.
## Example Response
This is an example of the response from the `Chat Completions API` example (using the video shown above):
```mdx Chat Completions wrap expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Topic: The story of a multi-generational family bakery, its history, its destruction by fire, and the determination to rebuild and adapt the business for a new era.
Summary: The video chronicles the history of the Jenny Lee Bakery, a beloved institution in McKees Rocks, Pennsylvania, run by the Baker family for generations. It details the bakery's founding in 1941, its role in the community, and the passion for baking passed down through generations. The story takes a tragic turn with a devastating fire and a recession, leading to the closure and demolition of the bakery. However, the narrative concludes with the current generation, Scott Baker, deciding to rebuild the business with a modern, wholesale-focused approach.
Chapters (mm:ss format):
00:00 - 00:15: Scott Baker introduces himself and his family's deep-rooted connection to the McKees Rocks community through the Jenny Lee Bakery, which his grandfather opened in 1941.
00:15 - 00:31: A long-time employee and customer, Donna, shares fond memories of visiting the bakery for treats after church and later working there herself.
00:31 - 00:48: The video shows the transition to the next generation, with Scott's father, Bernie, taking over. Scott recalls his own childhood experiences working in the bakery and developing a love for the family business.
00:48 - 01:14: The narrative shifts to a tragic event, as Donna recounts learning that the bakery was on fire on Thanksgiving, a moment that cost her her job. Newspaper headlines confirm the devastating blaze.
01:14 - 01:42: Scott and his father, Bernie, recall the despair of seeing their life's work destroyed by the fire. The combination of the fire and the subsequent recession led to the difficult decision to close the bakery, which was later demolished.
01:42 - 02:08: Feeling burnt out, Scott was advised by his father to pursue a different career. However, Scott felt that baking was in his blood and was determined to revive the family business in McKees Rocks.
02:08 - 02:23: After researching the modern market and realizing the decline of traditional retail bakeries, Scott devises a new plan. He decides to adapt by creating a wholesale bakery to supply baked goods to other stores.
```
```json Structured Outputs expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"topic": "The story of a multi-generational family bakery, its history, its destruction by fire, and the determination to rebuild and adapt the business for a new era.",
"summary": "The video chronicles the history of the Jenny Lee Bakery, a beloved institution in McKees Rocks, Pennsylvania, run by the Baker family for generations. It details the bakery's founding in 1941, its role in the community, and the passion for baking passed down through generations. The story takes a tragic turn with a devastating fire and a recession, leading to the closure and demolition of the bakery. However, the narrative concludes with the current generation, Scott Baker, deciding to rebuild the business with a modern, wholesale-focused approach.",
"chapters": [
{
"start_time": "00:00:00",
"end_time": "00:00:15",
"description": "Scott Baker introduces himself and his family's deep-rooted connection to the McKees Rocks community through the Jenny Lee Bakery, which his grandfather opened in 1941."
},
{
"start_time": "00:00:15",
"end_time": "00:00:31",
"description": "A long-time employee and customer, Donna, shares fond memories of visiting the bakery for treats after church and later working there herself."
}
]
}
```
## Usage Example
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Caption the video
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Parse this video"},
{"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
]
}
],
)
print(response.choices[0].message.content)
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class ParsedVideoChapter(BaseModel):
start_time: str = Field(..., description="Start time in HH:MM:SS format")
end_time: str = Field(..., description="End time in HH:MM:SS format")
description: str = Field(..., description="Description of the chapter")
class ParsedVideoResponse(BaseModel):
topic: str = Field(..., description="Topic of the video")
summary: str = Field(..., description="Summary of the video content")
chapters: list[ParsedVideoChapter] = Field(..., description="Video chapters")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Parse the video with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Parse this video"},
{"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
]
}
],
response_format={"type": "json_schema", "schema": ParsedVideoResponse.model_json_schema()}
)
# Validate the response
result = ParsedVideoResponse.model_validate_json(response.choices[0].message.content)
# >>> ParsedVideoResponse(topic="...", summary="...", chapters=[...])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Parse this video" },
{ type: "video_url", video_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const ParsedVideoResponseSchema = z.object({
topic: z.string().describe("Topic of the video"),
summary: z.string().describe("Summary of the video content"),
chapters: z.array(z.object({
start_time: z.string().describe("Start time in HH:MM:SS format"),
end_time: z.string().describe("End time in HH:MM:SS format"),
description: z.string().describe("Description of the chapter")
})).describe("Video chapters")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Parse the video with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Parse this video" },
{ type: "video_url", video_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(ParsedVideoResponseSchema)
}
});
const result = ParsedVideoResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
You can ask simply ask for a more detailed caption by providing a more detailed prompt. In most cases, you can provide the number of words you want the caption to be, and the model will generate a more detailed caption.
* **Content Types**: presentation, tutorial, interview, documentary, news
* **Scenes**: office, outdoor, studio, classroom, conference room
* **People**: presenter, audience, speaker, interviewer
* **Objects**: whiteboard, charts, graphs, computer, microphone
The video segments come in the format of a list of dictionaries with start time, end time, and description fields.
Yes, the structured output includes segments with timestamps that break down the video into different parts with descriptions for each segment.