# FAQ
Source: https://docs.vlm.run/FAQ
Frequently Asked Questions.
We are working on a fine-tuning feature that will be available soon. [Reach out](mailto:support@vlm.run) to us if you have a specific use-case in mind.
While our API is currently hosted in the cloud, we do offer private deployment options (on-premises or in-VPC deployments). Reach out to us at [support](mailto:support@vlm.run) to discuss your requirements.
Yes, you can use the API for commercial purposes. We will soon offer a free tier for testing and development, and currently offer paid plans for production use.
We currently do not support research use-cases just yet. That said, if you are interested in partnering with us to learn more about VLM-1 and improve our models, we would love to hear from you.
# Multi-modal Artifacts
Source: https://docs.vlm.run/agents/artifacts
Retrieve generated images, videos, audio, and documents from agent responses
Artifacts are binary objects generated during agent interactions, such as images, videos, audio files, and documents. When agents perform operations like image generation, face blurring, video trimming, or document processing, the results are stored as artifacts that can be retrieved using object references.
## Object References
Agent responses return object references (refs) instead of raw binary data. Each reference is a string identifier that follows a specific format: a 3-5 letter type prefix followed by an underscore and a 6-digit hexadecimal string (e.g., `img_a1b2c3`).
| Artifact Type | Prefix | Reference Type | Python Return Type |
| -------------- | -------- | -------------- | ------------------------- |
| Image | `img_` | `ImageRef` | `PIL.Image.Image` |
| Video | `vid_` | `VideoRef` | `Path` (mp4) |
| Audio | `aud_` | `AudioRef` | `Path` (mp3) |
| Document | `doc_` | `DocumentRef` | `Path` (pdf) |
| Reconstruction | `recon_` | `ReconRef` | `Path` (spz) |
| URL | `url_` | `UrlRef` | `Path` (any of the above) |
| Array | `arr_` | `ArrayRef` | `np.ndarray` |
Import reference types from the SDK:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.types import ImageRef, VideoRef, AudioRef, DocumentRef, ReconRef, UrlRef
```
## Retrieving an Artifact
### In a Chat Completion
To retrieve a chat completion artifact, use the `session_id` from the chat response and the `object_id` (returned as a Ref type) from the JSON result.
```python Python [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from pydantic import BaseModel, Field
from PIL import Image
from vlmrun.client import VLMRun
from vlmrun.types import ImageRef
client = VLMRun(api_key="")
# Define a response model with an ImageRef field
class BlurredImageResponse(BaseModel):
image: ImageRef = Field(..., description="The blurred image")
# Make a chat completion request
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Blur all the faces in this image"},
{"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
]
}
],
response_format={
"type": "json_schema",
"schema": BlurredImageResponse.model_json_schema()
}
)
# Parse the response
result = BlurredImageResponse.model_validate_json(response.choices[0].message.content)
# Retrieve the artifact using session_id and object_id
blurred_image: Image.Image = client.artifacts.get(
session_id=response.session_id,
object_id=result.image.id
)
# Display or save the image
blurred_image.save("blurred_output.jpg")
```
```typescript TypeScript [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Coming soon...
```
## Common Use Cases
Generate multi-modal artifacts such as images and videos.
Generate multiple images of a scene (e.g. virtual try-on, video thumbnails, etc.).
Redact sensitive information from documents, and return the processed document as a PDF.
Generate 3D models from images or videos, and return ply/spz files.
## Best Practices
When working with artifacts, keep these guidelines in mind:
* For large artifacts like videos, the Python and Node SDKs download files to disk rather than loading them into memory. This prevents memory issues when working with large files. Always check the file size before loading video content into memory.
* Use structured response models with appropriate `Ref` types (`ImageRef`, `VideoRef`, etc.) to ensure type safety and enable IDE autocompletion. The Python and Node SDKs will automatically handle the conversion to the appropriate Python type when retrieving artifacts.
View the complete API reference for artifact retrieval
# Layout Detection
Source: https://docs.vlm.run/agents/capabilities/document/layout-understanding
Identify and analyze document structure with visual result previews showing highlighted extractions and field overlays
## Usage Example
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
The following examples can detect headers, paragraphs, tables, lists, figures, and other document elements. The response schema includes bounding boxes, reading order and more.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Analyze document layout
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Analyze the document layout and identify all elements with bounding boxes"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.layout/qwen-25-vl-tech-report.jpg", "detail": "auto"}}
]
}
]
)
print(response.choices[0].message.content)
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class LayoutElement(BaseModel):
type: str = Field(..., description="Type of layout element")
xywh: tuple[float, float, float, float] = Field(..., description="Bounding box (x, y, w, h)")
class LayoutResponse(BaseModel):
elements: list[LayoutElement] = Field(..., description="List of layout elements")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Analyze document layout with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Analyze the document layout and identify all elements with bounding boxes"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.layout/qwen-25-vl-tech-report.jpg", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": LayoutResponse.model_json_schema()},
)
# Validate the response
result = LayoutResponse.model_validate_json(response.choices[0].message.content)
# >>> LayoutResponse(elements=[LayoutElement(type="caption", xywh=(...)), ...])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Analyze the document layout and identify all elements with bounding boxes" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.layout/qwen-25-vl-tech-report.jpg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const LayoutResponseSchema = z.object({
elements: z.array(z.object({
type: z.string().describe("Type of layout element"),
xywh: z.array(z.number()).describe("Bounding box (x, y, w, h)")
})).describe("List of layout elements")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Analyze document layout with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Analyze the document layout and identify all elements with bounding boxes" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.layout/qwen-25-vl-tech-report.jpg", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(LayoutResponseSchema)
}
});
const result = LayoutResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
* **Headers**: H1-H6 level headers with hierarchical structure
* **Paragraphs**: Body text blocks with proper text flow
* **Titles**: Main title of the document
* **Tables**: Structured data with row/column detection
* **Figures**: Images, charts, diagrams, and visual elements
* **Lists**: Bulleted and numbered list structures
* **Captions**: Figure and table captions with associations
* **Footnotes**: Footnotes with references and content
* **Formulas**: Mathematical formulas and equations
* **Pictures**: Images and visual elements
* **Section Headers**: Section headers and titles
The bounding boxes come in the format of `xywh`, where `x` and `y` are the top-left corner coordinates, and `w` and `h` are the width and height of the bounding box. All values are in pixels relative to the document image.
The reading order indicates the sequence in which elements should be read, following the natural document flow from top to bottom and left to right. This is useful for accessibility and content extraction.
Yes, the layout detection can process multi-page documents. Each page is analyzed separately, and the results include page-specific bounding boxes and reading orders.
# Multi-Page Analysis
Source: https://docs.vlm.run/agents/capabilities/document/multi-page-analysis
Process and analyze documents across multiple pages with context preservation and cross-document correlation
Process and analyze documents across multiple pages with context preservation and cross-document correlation. Perfect for medical record processing, legal document review, multi-document workflows, and comprehensive document analysis that requires understanding relationships between different document types.
## Usage Example
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
The following examples can analyze and triage multiple pages or documents, identify cross-document relationships, extract consistent information across pages, and provide comprehensive analysis with context preservation. The response schema includes page summaries, cross-document connections, and thematic analysis.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Analyze multi-page medical documents
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Analyze this multi-page medical document set. Extract patient referral page, medical insurance card and identification form in 3 separate fields in JSON format."},
{"type": "file_url", "file_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.agent/multi-document-input-example.pdf", "detail": "auto"}}
]
}
]
)
print(response.choices[0].message.content)
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
from typing import Literal
# Define the response schema
class DocumentPage(BaseModel):
page_id: int = Field(..., description="Page number (0-indexed)")
document_type: Literal["referral", "insurance-card", "identification"]
class MultiPageResponse(BaseModel):
pages: list[DocumentPage] = Field(..., description="List of document pages")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Analyze multi-page document with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Analyze this multi-page medical document set. Extract patient referral page, medical insurance card and identification form in 3 separate fields in JSON format."},
{"type": "file_url", "file_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.agent/multi-document-input-example.pdf", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": MultiPageResponse.model_json_schema()},
)
# Validate the response
result = MultiPageResponse.model_validate_json(response.choices[0].message.content)
# >>> MultiPageResponse(pages=[DocumentPage(page_id=0, document_type="referral"), ...])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Analyze this multi-page medical document set. Extract patient referral page, medical insurance card and identification form in 3 separate fields in JSON format." },
{ type: "file_url", file_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.agent/multi-document-input-example.pdf", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const MultiPageResponseSchema = z.object({
pages: z.array(z.object({
page_id: z.number().int().describe("Page number (0-indexed)"),
document_type: z.enum(["referral", "insurance-card", "identification"])
})).describe("List of document pages")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Analyze multi-page document with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Analyze this multi-page medical document set. Extract patient referral page, medical insurance card and identification form in 3 separate fields in JSON format." },
{ type: "file_url", file_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.agent/multi-document-input-example.pdf", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(MultiPageResponseSchema)
}
});
const result = MultiPageResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
* **Cross-Document Correlation**: Identify relationships between different document types
* **Data Consistency Checking**: Verify data matches across pages and documents
* **Theme Analysis**: Track recurring themes and topics across pages
* **Reference Tracking**: Follow references and citations across pages
* **Content Flow Analysis**: Understand how content flows between pages
* **Medical Records**: Referral forms, insurance cards, ID forms, lab reports
* **Legal Documents**: Contracts, amendments, exhibits, supporting documents
* **Financial Documents**: Invoices, receipts, statements, tax forms
* **Academic Papers**: Research papers, appendices, references, figures
* **Business Reports**: Executive summaries, detailed sections, appendices
The system identifies connections between different pages/documents by:
* **Data Matching**: Finding identical or similar values across documents
* **Reference Tracking**: Following explicit references between pages
* **Contextual Analysis**: Understanding semantic relationships
* **Confidence Scoring**: Providing reliability scores for each connection
The confidence score is a value between 0 and 1 that indicates the reliability of cross-document connections. Higher scores indicate more reliable matches and relationships between pages.
Yes, multi-page analysis can process different document types within a single PDF or across multiple uploaded documents, identifying relationships and correlations between them.
# Visual Grounding
Source: https://docs.vlm.run/agents/capabilities/document/visual-grounding
Connect text elements with their visual locations in documents for precise content understanding
Connect text elements with their visual locations in documents for precise content understanding. Perfect for interactive document analysis, content verification, automated form filling, and document comparison workflows.
Document Form
Driver's License
TV News Broadcast Text
## Usage Example
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
The following examples can map text elements to their visual locations, detect spatial relationships, and identify cross-references in documents. The response schema includes bounding boxes, confidence scores, and relationship types.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Perform visual grounding
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Localize all the speaker names in the TV news broadcast text and visualize them on the image. Only provide one bounding box for each speaker name."},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/media.tv-news/finance_bb_3_speakers.jpg", "detail": "auto"}}
]
}
]
)
print(response.choices[0].message.content)
# >>> {"elements": [{"content": "HAIDI STROUD-WATTS", "xywh": [0.428, 0.217, 0.128, 0.286]}, ...]}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class GroundingWithText(BaseModel):
content: str = Field(..., description="The text content")
xywh: tuple[float, float, float, float] = Field(..., description="Bounding box (x, y, w, h)")
class GroundingResponse(BaseModel):
elements: list[GroundingWithText] = Field(..., description="Text to visual mappings")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Perform visual grounding with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Localize all the speaker names in the TV news broadcast text and visualize them on the image. Only provide one bounding box for each speaker name."},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/media.tv-news/finance_bb_3_speakers.jpg", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": GroundingResponse.model_json_schema()},
)
# Validate the response
result = GroundingResponse.model_validate_json(response.choices[0].message.content)
# >>> GroundingResponse(elements=[GroundingWithText(content="...", xywh=(...)), ...])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Localize all the speaker names in the TV news broadcast text and visualize them on the image. Only provide one bounding box for each speaker name." },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/media.tv-news/finance_bb_3_speakers.jpg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const GroundingResponseSchema = z.object({
elements: z.array(z.object({
content: z.string().describe("The text content"),
xywh: z.array(z.number()).describe("Bounding box (x, y, w, h)")
})).describe("Text to visual mappings")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Perform visual grounding with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Localize all the speaker names in the TV news broadcast text and visualize them on the image. Only provide one bounding box for each speaker name." },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/media.tv-news/finance_bb_3_speakers.jpg", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(GroundingResponseSchema)
}
});
const result = GroundingResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
* **Form Fields**: Connect labels with input fields, checkboxes, and buttons
* **Data Fields**: Map data labels with their corresponding values
* **Interactive Elements**: Link text instructions with clickable elements
* **Validation Rules**: Connect validation text with form fields
* **Cross-References**: Map text mentions with figures, tables, and sections
The bounding boxes come in the format of `xywh`, where `x` and `y` are the top-left corner coordinates, and `w` and `h` are the width and height of the bounding box. All values are in pixels relative to the document image.
* **Label-Field Pairs**: Identify which labels belong to which fields
* **Hierarchical Structure**: Understand parent-child relationships
* **Proximity Analysis**: Determine related elements based on spatial proximity
* **Alignment Patterns**: Detect aligned elements and groups
The confidence score is a value between 0 and 1 that indicates the confidence of the text-visual mapping. Higher scores indicate more reliable connections.
Yes, visual grounding can process multi-page documents. Each page is analyzed separately, and the results include page-specific mappings and relationships.
# Caption & Tag
Source: https://docs.vlm.run/agents/capabilities/image/captioning
Generate detailed captions and tags for images using advanced vision models.
Generate comprehensive, contextual captions for images using state-of-the-art vision-language models. Perfect for accessibility, content management, and automated image analysis workflows.
## Example Response
This is an example of the response from the `Chat Completions API` example (using the image shown above):
```mdx Chat Completions wrap theme={"theme":{"light":"github-light","dark":"dark-plus"}}
A classic, light turquoise Volkswagen Beetle with chrome accents is parked on a cobblestone street, set against a warm yellow stucco wall with rustic brown wooden doors and windows.
Tags: car, volkswagen, beetle, street, cobblestone, wooden, doors, windows
```
```json Structured Outputs wrap theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"caption": "A classic, light turquoise Volkswagen Beetle with chrome accents is parked on a cobblestone street, set against a warm yellow stucco wall with rustic brown wooden doors and windows.",
"tags": ["car", "volkswagen", "beetle", "street", "cobblestone", "wooden", "doors", "windows"]
}
```
## Usage Example
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Caption the image
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{"role": "user",
"content": [
{"type": "text", "text": "Generate a detailed caption for this image"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto"}}
]
}
],
)
# Print the response
print(response.choices[0].message.content)
# >> "A classic, light turquoise Volkswagen Beetle..."
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class ImageCaption(BaseModel):
caption: str = Field(..., description="Detailed caption of the scene")
tags: list[str] = Field(..., description="Tags that describe the image")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Caption the image with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{"role": "user",
"content": [
{"type": "text", "text": "Generate a detailed caption for this image"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": ImageCaption.model_json_schema()}
)
# Validate the response
result = ImageCaption.model_validate_json(response.choices[0].message.content)
# >>> ImageCaption(caption="...", tags=[...])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Caption the image
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Generate a detailed caption for this image" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const ImageCaptionSchema = z.object({
caption: z.string().describe("Detailed caption of the scene"),
tags: z.array(z.string()).describe("Tags that describe the image")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Caption the image with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Generate a detailed caption for this image" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(ImageCaptionSchema)
}
});
const result = ImageCaptionSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
You can ask simply ask for a more detailed caption by providing a more detailed prompt. In most cases, you can provide the number of words you want the caption to be, and the model will generate a more detailed caption.
* **Common Objects**: person, car, truck, bus, bicycle, motorcycle
* **Scenes**: street, building, park, forest, beach, etc.
* **Time-of-Day**: morning, afternoon, evening, night
* **Weather**: sunny, cloudy, rainy, snowing, etc.
The tags come in the format of a list of strings.
# Detection
Source: https://docs.vlm.run/agents/capabilities/image/detection
Detect and locate objects, faces or people in images with bounding boxes and confidence scores
Detect objects, people, or other entities in images with precise bounding boxes and confidence scores. Ideal for inventory management, quality control, security applications, and automated visual inspection.
Persons
Faces
## Usage Example
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
The following examples can also be used for face or person detection. The response schema is identical to the object detection example.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Execute object detection
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{"role": "user", "content": [
{"type": "text", "text": "Detect all objects in this image"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/donuts.png", "detail": "auto"}}
]},
],
)
print(response.choices[0].message.content)
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class Detection(BaseModel):
label: str = Field(..., description="Name of the detected object")
xywh: tuple[float, float, float, float] = Field(..., description="Bounding box (x, y, width, height)")
class Detections(BaseModel):
detections: list[Detection] = Field(..., description="List of detections")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Execute object detection with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{"role": "user", "content": [
{"type": "text", "text": "Detect all objects in this image"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/donuts.png", "detail": "auto"}}
]},
],
response_format={"type": "json_schema", "schema": Detections.model_json_schema()},
)
# Validate the response
result = Detections.model_validate_json(response.choices[0].message.content)
# >>> Detections(detections=[Detection(label="donut", xywh=(...)), ...])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Detect all objects in this image" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/donuts.png", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const DetectionsSchema = z.object({
detections: z.array(z.object({
label: z.string().describe("Name of the detected object"),
xywh: z.array(z.number()).describe("Bounding box (x, y, width, height)")
})).describe("List of detections")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Execute object detection with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Detect all objects in this image" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/donuts.png", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(DetectionsSchema)
}
});
const result = DetectionsSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
Orion supports open-world (aka open-set) detection. This means it is not limited to a predefined list of categories; it can detect, locate, and ground virtually any object described in natural language, including rare items, specific parts of objects, and complex visual relations.
The bounding boxes come in the format of normalized `xywh`, where `x` and `y` are the top-left corner of the bounding box, and `w` and `h` are the width and height of the bounding box. All values are between 0 and 1, and normalized by the image size. `x` and `w` are normalized by the image width, and `y` and `h` are normalized by the image height.
The confidence score is a value between 0 and 1 that indicates the confidence of the detection.
# Generate & Edit
Source: https://docs.vlm.run/agents/capabilities/image/generation
Generate and edit images from text prompts, sketches, or existing images with creative control
Generate and edit images from text prompts, sketches, or existing images with creative control. Perfect for creative content generation, marketing and advertising, product visualization, and artistic expression.
Image-to-Image
Image-Inpainting
Style Transfer
## Example Usage
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
### Text-to-Image
Generate images from text descriptions with creative control over style, composition, and details.
Generate an image of a cat flying through the sky and clouds, with the background of a green city with river
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Generate the image
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": "Generate an image of a modern building as a cyberpunk-style futuristic structure with neon lights and holographic elements"
}
]
)
print(response.choices[0].message.content)
# >>> {"url": "https://.../image.jpg"}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class ImageGenerationResponse(BaseModel):
url: str = Field(..., description="The URL of the generated image")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Generate the image with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": "Generate an image of a modern building as a cyberpunk-style futuristic structure with neon lights and holographic elements"
}
],
response_format={"type": "json_schema", "schema": ImageGenerationResponse.model_json_schema()}
)
# Validate the response
result = ImageGenerationResponse.model_validate_json(response.choices[0].message.content)
# >>> ImageGenerationResponse(url="https://.../image.jpg")
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: "Generate an image of a modern building as a cyberpunk-style futuristic structure with neon lights and holographic elements"
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const ImageGenerationResponseSchema = z.object({
url: z.string().describe("The URL of the generated image")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Generate the image with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: "Generate an image of a modern building as a cyberpunk-style futuristic structure with neon lights and holographic elements"
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(ImageGenerationResponseSchema)
}
});
const result = ImageGenerationResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
### Image-to-Image
Transform existing images by applying new styles, enhancing details, or changing specific elements while preserving the original structure.
Reference Dog Image
Dog flying through space
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Generate the image
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Transform the image of my dog on the right into a flying dog with superman cape, with majestic background"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/dog-cat.jpg", "detail": "auto"}}
]
}
]
)
print(response.choices[0].message.content)
# >>> {"url": "https://.../image.jpg"}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class ImageGenerationResponse(BaseModel):
url: str = Field(..., description="The URL of the generated image")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Transform the image with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Transform the image of my dog on the right into a flying dog with superman cape, with majestic background"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/dog-cat.jpg", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": ImageGenerationResponse.model_json_schema()}
)
# Validate the response
result = ImageGenerationResponse.model_validate_json(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Transform the image of my dog on the right into a flying dog with superman cape, with majestic background" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/dog-cat.jpg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const ImageGenerationResponseSchema = z.object({
url: z.string().describe("The URL of the generated image")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Transform the image with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Transform the image of my dog on the right into a flying dog with superman cape, with majestic background" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/dog-cat.jpg", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(ImageGenerationResponseSchema)
}
});
const result = ImageGenerationResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
### Image Inpainting
Fill in missing areas, remove unwanted objects, or add new elements to existing images seamlessly.
Reference Dog Image
Reference Car Image
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Generate the image
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Given two images: the first with a dog in front holding a yellow ball, inpaint her driving the car shown in the second image."},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/dogs.jpg", "detail": "auto"}},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto"}}
]
}
]
)
print(response.choices[0].message.content)
# >>> {"url": "https://.../image.jpg"}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class ImageGenerationResponse(BaseModel):
url: str = Field(..., description="The URL of the generated image")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Inpaint the image with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Given two images: the first with a dog in front holding a yellow ball, inpaint her driving the car shown in the second image."},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/dogs.jpg", "detail": "auto"}},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": ImageGenerationResponse.model_json_schema()}
)
# Validate the response
result = ImageGenerationResponse.model_validate_json(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Given two images: the first with a dog in front holding a yellow ball, inpaint her driving the car shown in the second image." },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/dogs.jpg", detail: "auto" } },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const ImageGenerationResponseSchema = z.object({
url: z.string().describe("The URL of the generated image")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Inpaint the image with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Given two images: the first with a dog in front holding a yellow ball, inpaint her driving the car shown in the second image." },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/dogs.jpg", detail: "auto" } },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(ImageGenerationResponseSchema)
}
});
const result = ImageGenerationResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
### Style Transfer
Apply artistic styles from reference images to transform the visual appearance while preserving content. In the example below, we apply the Van Gogh's "Starry Night" painting style to a photo of a city skyline at night.
Reference Van Gogh Image
Reference City Image
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Generate the image
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Given two images: the first with a Van Gogh painting, and the second with a city skyline at night, apply the Van Gogh painting style to the city skyline."},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/starry-night.jpg", "detail": "auto"}},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg", "detail": "auto"}}
]
}
]
)
print(response.choices[0].message.content)
# >>> {"url": "https://.../image.jpg"}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class ImageGenerationResponse(BaseModel):
url: str = Field(..., description="The URL of the generated image")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Apply style transfer with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Given two images: the first with a Van Gogh painting, and the second with a city skyline at night, apply the Van Gogh painting style to the city skyline."},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/starry-night.jpg", "detail": "auto"}},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": ImageGenerationResponse.model_json_schema()}
)
# Validate the response
result = ImageGenerationResponse.model_validate_json(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Given two images: the first with a Van Gogh painting, and the second with a city skyline at night, apply the Van Gogh painting style to the city skyline." },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/starry-night.jpg", detail: "auto" } },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const ImageGenerationResponseSchema = z.object({
url: z.string().describe("The URL of the generated image")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Apply style transfer with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Given two images: the first with a Van Gogh painting, and the second with a city skyline at night, apply the Van Gogh painting style to the city skyline." },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/starry-night.jpg", detail: "auto" } },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(ImageGenerationResponseSchema)
}
});
const result = ImageGenerationResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
* **Photorealistic**: Ultra-realistic images with fine details
* **High Detail**: Ultra-realistic images with fine details
* **Natural Lighting**: Realistic lighting and shadows
* **Professional Quality**: Suitable for commercial and professional use
* **Multiple Subjects**: People, objects, landscapes, architecture
* **Be Specific**: Include details about style, composition, lighting, and mood
* **Use Keywords**: Include relevant art terms, techniques, and descriptors
* **Reference Styles**: Mention specific artists, art movements, or visual styles
* **Technical Details**: Specify resolution, quality, and output format needs
# Pointing
Source: https://docs.vlm.run/agents/capabilities/image/pointing
Detect and predict key anatomical points and structural features in images with sub-pixel accuracy
Detect and localize keypoints of objects, people or faces in images with precise coordinate mapping. Ideal for counting, localization and salience detection.
Object Localization
Person Localization
Face Localization
## Usage Example
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Predict the keypoints in the image
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Point to all the cars and doors in this image"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto"}}
]
}
],
)
print(response.choices[0].message.content)
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class KeyPoint(BaseModel):
xy: tuple[float, float] = Field(..., description="Normalized keypoint coordinates [x, y]")
label: str = Field(..., description="Label of the keypoint")
class Keypoints(BaseModel):
keypoints: list[KeyPoint] = Field(..., description="List of keypoints")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Predict the keypoints with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Point to all the cars and doors in this image"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": Keypoints.model_json_schema()},
)
# Validate the response
result = Keypoints.model_validate_json(response.choices[0].message.content)
# >>> Keypoints(keypoints=[KeyPoint(xy=(0.5, 0.5), label='car'), ...])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Point to all the cars and doors in this image" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const KeypointsSchema = z.object({
keypoints: z.array(z.object({
xy: z.array(z.number()).describe("Normalized keypoint coordinates [x, y]"),
label: z.string().describe("Label of the keypoint")
})).describe("List of keypoints")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Predict the keypoints with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Point to all the cars and doors in this image" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(KeypointsSchema)
}
});
const result = KeypointsSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
The keypoints come in the format of a list of objects with their keypoints. The keypoints are in the format of normalized `xy`, where `x` and `y` are the top-left corner of the keypoint. All values are between 0 and 1, and normalized by the image size. `x` and `y` are normalized by the image width and height respectively.
You can extract the following tags for each keypoint:
* **Object Name**: The name of the object that the keypoint belongs to. For example, "car", "door", "person", "face", etc.
* **Confidence Score**: The confidence score of the keypoint detection.
# Segmentation
Source: https://docs.vlm.run/agents/capabilities/image/segmentation
Create precise pixel-level segmentation masks for objects, regions, and features in images
Create precise pixel-level segmentation masks for objects, regions, and features in images. Perfect for medical imaging, autonomous driving, photo editing, and augmented reality applications.
Object Segmentation
Face Segmentation
## Usage Example
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Segment objects in the image
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Segment all the cars in this image"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/nascar.jpg", "detail": "auto"}}
]
}
],
)
print(response.choices[0].message.content)
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class SegmentationMask(BaseModel):
label: str = Field(..., description="Name of the segmented object")
mask_url: str = Field(..., description="Pre-signed URL to the PNG mask")
class Segmentations(BaseModel):
segmentations: list[SegmentationMask] = Field(..., description="List of segmentations")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Segment objects with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Segment all the cars in this image"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/nascar.jpg", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": Segmentations.model_json_schema()},
)
# Validate the response
result = Segmentations.model_validate_json(response.choices[0].message.content)
# >>> Segmentations(segmentations=[SegmentationMask(label="car", mask_url="https://..."), ...])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Segment all the cars in this image" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/nascar.jpg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const SegmentationsSchema = z.object({
segmentations: z.array(z.object({
label: z.string().describe("Name of the segmented object"),
mask_url: z.string().describe("Pre-signed URL to the PNG mask")
})).describe("List of segmentations")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Segment objects with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Segment all the cars in this image" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/nascar.jpg", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(SegmentationsSchema)
}
});
const result = SegmentationsSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
* **Instance Segmentation**: Segment individual objects with unique masks
* **Semantic Segmentation**: Classify pixels by category or class
* **Panoptic Segmentation**: Combine instance and semantic segmentation
The segmentation masks come in the format of a list of objects with their masks. The masks are in PNG format that can be retrieved as a pre-signed URL, per object instance.
Orion supports promptable segmentation, where you can describe in natural language the object(s) you want to segment (e.g., "segment the writing-related items in this image"). For best results, clearly describe the object(s) of interest. Below are some common examples of classes that can be segmented:
Common Objects
Binary or grayscale images where each pixel value represents a segment ID
Compatible with most image editing software
Small file size for simple segmentations
Other Formats (Coming soon!): JSON Polygons and COCO Format
# Image Tools
Source: https://docs.vlm.run/agents/capabilities/image/tools
Image tools for cropping, rotating, enhancing, and transforming images
VLM Run's Orion agents can leverage various image-editing tools such as cropping, rotating, super-resolution, and de-oldify. These tools are designed to help you enhance image quality, extract specific regions, correct orientation, and restore historical photos with modern AI techniques.
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
### 1. Image Cropping
Extract specific regions or focus on particular subjects within an image.
```mdx wrap theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Crop the clock to tell the time more clearly.
```
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Crop image to focus on main subject
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Crop the clock to tell the time more clearly"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/clock.jpg", "detail": "auto"}}
]
}
]
)
print(response.choices[0].message.content)
# >>> {"url": "https://.../cropped.jpg", "label": "clock", "xywh": [0.2, 0.2, 0.6, 0.6]}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class ImageCropResponse(BaseModel):
url: str = Field(..., description="URL of the cropped image")
label: str = Field(..., description="Object label")
xywh: tuple[float, float, float, float] = Field(..., description="Bounding box (x, y, w, h)")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Crop image with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Crop the clock to tell the time more clearly"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/clock.jpg", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": ImageCropResponse.model_json_schema()}
)
# Validate the response
result = ImageCropResponse.model_validate_json(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Crop the clock to tell the time more clearly" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/clock.jpg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const ImageCropResponseSchema = z.object({
url: z.string().describe("URL of the cropped image"),
label: z.string().describe("Object label"),
xywh: z.array(z.number()).describe("Bounding box (x, y, w, h)")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Crop image with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Crop the clock to tell the time more clearly" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/clock.jpg", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(ImageCropResponseSchema)
}
});
const result = ImageCropResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
While the demonstration uses a single crop, we also support cropping multiple regions at once.
### 2. Image Rotation
Correct image orientation or apply creative rotations for better composition.
```mdx wrap theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Rotate the image 90 degrees clockwise to correct the orientation.
```
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Rotate image to correct orientation
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Rotate this image 90 degrees clockwise"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/cats.jpg"}}
]
}
]
)
print(response.choices[0].message.content)
# >>> {"url": "https://.../rotated.jpg", "angle": 90}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class ImageRotationResponse(BaseModel):
url: str = Field(..., description="URL of the rotated image")
angle: int = Field(..., description="Rotation angle (0, 90, 180, 270) degrees clockwise")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Rotate image with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Rotate this image 90 degrees clockwise"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/cats.jpg"}}
]
}
],
response_format={"type": "json_schema", "schema": ImageRotationResponse.model_json_schema()}
)
# Validate the response
result = ImageRotationResponse.model_validate_json(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Rotate this image 90 degrees clockwise" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/cats.jpg" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const ImageRotationResponseSchema = z.object({
url: z.string().describe("URL of the rotated image"),
angle: z.number().int().describe("Rotation angle (0, 90, 180, 270) degrees clockwise")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Rotate image with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Rotate this image 90 degrees clockwise" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/cats.jpg" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(ImageRotationResponseSchema)
}
});
const result = ImageRotationResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
### 3. Super-Resolution Enhancement
Upscale images while maintaining quality and adding realistic details.
```mdx wrap theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Enhance this image using super-resolution to increase its resolution while preserving quality.
```
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Apply super-resolution enhancement
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Enhance this image using super-resolution to increase its resolution while preserving quality"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/vegetables-lo.jpg"}}
]
}
]
)
print(response.choices[0].message.content)
# >>> {"url": "https://.../enhanced.jpg"}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class SuperResolutionResponse(BaseModel):
url: str = Field(..., description="URL of the enhanced image")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Enhance image with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Enhance this image using super-resolution to increase its resolution while preserving quality"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/vegetables-lo.jpg"}}
]
}
],
response_format={"type": "json_schema", "schema": SuperResolutionResponse.model_json_schema()}
)
# Validate the response
result = SuperResolutionResponse.model_validate_json(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Enhance this image using super-resolution to increase its resolution while preserving quality" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/vegetables-lo.jpg" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const SuperResolutionResponseSchema = z.object({
url: z.string().describe("URL of the enhanced image")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Enhance image with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Enhance this image using super-resolution to increase its resolution while preserving quality" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/vegetables-lo.jpg" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(SuperResolutionResponseSchema)
}
});
const result = SuperResolutionResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
### 4. De-Oldify (Colorization)
Transform black and white or sepia images into vibrant color photos using AI.
```mdx wrap theme={"theme":{"light":"github-light","dark":"dark-plus"}}
De-oldify this image so that it's colorized and upsampled.
```
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Colorize black and white image
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "De-oldify this image so that it's colorized and upsampled"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg"}}
]
}
]
)
print(response.choices[0].message.content)
# >>> {"url": "https://.../colorized.jpg"}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class DeOldifyResponse(BaseModel):
url: str = Field(..., description="URL of the colorized image")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Colorize image with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "De-oldify this image so that it's colorized and upsampled"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg"}}
]
}
],
response_format={"type": "json_schema", "schema": DeOldifyResponse.model_json_schema()}
)
# Validate the response
result = DeOldifyResponse.model_validate_json(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "De-oldify this image so that it's colorized and upsampled" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const DeOldifyResponseSchema = z.object({
url: z.string().describe("URL of the colorized image")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Colorize image with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "De-oldify this image so that it's colorized and upsampled" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.agent/lunch-skyscraper.jpg" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(DeOldifyResponseSchema)
}
});
const result = DeOldifyResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
* **JPEG/JPG**: Most common format with excellent compatibility
* **PNG**: Lossless format with transparency support
* **TIFF**: High-quality format for professional editing
* **WebP**: Modern format with superior compression
* **BMP**: Uncompressed bitmap format
* **Quality Preservation**: Maintains original image quality in all transformations
* **Rule of Thirds**: Align subjects with intersection points for better composition
* **Aspect Ratio**: Maintain consistent aspect ratios for professional results
* **Subject Focus**: Keep the main subject centered or following composition rules
* **Background Removal**: Remove distracting elements while preserving context
* **AI-Powered**: Uses advanced neural networks for realistic detail generation
* **Multiple Scales**: Supports 2x, 4x, and 8x upscaling with quality preservation
* **Detail Enhancement**: Intelligently adds realistic textures and patterns
* **Quality Metrics**: Provides confidence scores for enhancement quality
* **Historical Accuracy**: Uses context-aware AI to suggest period-appropriate colors
* **Natural Colors**: Generates realistic skin tones, clothing, and environmental colors
* **Confidence Scoring**: Provides confidence levels for color accuracy
* **Region Analysis**: Identifies and colors different regions with appropriate palettes
# UI Parsing
Source: https://docs.vlm.run/agents/capabilities/image/ui-parsing
Analyze and understand user interface elements in screenshots and application images
Analyze and understand user interface elements in screenshots and application images. Perfect for automated testing, design system validation, accessibility auditing, and mobile app analysis.
UI VQA & Grounding
Web Interface
## Usage Example
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Parse UI elements in the image
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Analyze all UI elements in this mobile app screenshot"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/web.ui-automation/win11.jpeg", "detail": "auto"}}
]
}
],
)
print(response.choices[0].message.content)
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class UIElement(BaseModel):
type: str = Field(..., description="Type of UI element")
text: str | None = Field(None, description="Text content of the element")
interactive: bool = Field(..., description="Whether the element is interactive")
xywh: tuple[float, float, float, float] = Field(..., description="Bounding box coordinates")
class UIResponse(BaseModel):
elements: list[UIElement] = Field(..., description="List of detected UI elements")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Parse UI elements with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Analyze all UI elements in this mobile app screenshot"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/web.ui-automation/win11.jpeg", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": UIResponse.model_json_schema()},
)
# Validate the response
result = UIResponse.model_validate_json(response.choices[0].message.content)
# >>> UIResponse(elements=[UIElement(type="button", text="Sign In", ...), ...])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Analyze all UI elements in this mobile app screenshot" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/web.ui-automation/win11.jpeg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const UIResponseSchema = z.object({
elements: z.array(z.object({
type: z.string().describe("Type of UI element"),
text: z.string().nullable().describe("Text content of the element"),
interactive: z.boolean().describe("Whether the element is interactive"),
xywh: z.array(z.number()).describe("Bounding box coordinates")
})).describe("List of detected UI elements")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Parse UI elements with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Analyze all UI elements in this mobile app screenshot" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/web.ui-automation/win11.jpeg", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(UIResponseSchema)
}
});
const result = UIResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
UI Parsing is the process of analyzing UI elements in screenshots and application images to identify UI elements, buttons, and interactive components for automated testing.
UI VQA & Grounding is the process of asking specific questions about the UI elements in screenshots and application images to identify UI elements, buttons, and interactive components for automated testing. This is different from UI parsing, where all UI elements are returned. In most cases, you should use UI VQA & Grounding to get more accurate results.
# Caption & Tag
Source: https://docs.vlm.run/agents/capabilities/video/captioning
Generate detailed captions and tags for videos using advanced vision models.
Generate comprehensive, contextual captions for videos using state-of-the-art vision-language models. Perfect for accessibility, content management, and automated video analysis workflows.
## Example Response
This is an example of the response from the `Chat Completions API` example (using the video shown above):
```mdx Chat Completions wrap expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Topic: The story of a multi-generational family bakery, its history, its destruction by fire, and the determination to rebuild and adapt the business for a new era.
Summary: The video chronicles the history of the Jenny Lee Bakery, a beloved institution in McKees Rocks, Pennsylvania, run by the Baker family for generations. It details the bakery's founding in 1941, its role in the community, and the passion for baking passed down through generations. The story takes a tragic turn with a devastating fire and a recession, leading to the closure and demolition of the bakery. However, the narrative concludes with the current generation, Scott Baker, deciding to rebuild the business with a modern, wholesale-focused approach.
Chapters (mm:ss format):
00:00 - 00:15: Scott Baker introduces himself and his family's deep-rooted connection to the McKees Rocks community through the Jenny Lee Bakery, which his grandfather opened in 1941.
00:15 - 00:31: A long-time employee and customer, Donna, shares fond memories of visiting the bakery for treats after church and later working there herself.
00:31 - 00:48: The video shows the transition to the next generation, with Scott's father, Bernie, taking over. Scott recalls his own childhood experiences working in the bakery and developing a love for the family business.
00:48 - 01:14: The narrative shifts to a tragic event, as Donna recounts learning that the bakery was on fire on Thanksgiving, a moment that cost her her job. Newspaper headlines confirm the devastating blaze.
01:14 - 01:42: Scott and his father, Bernie, recall the despair of seeing their life's work destroyed by the fire. The combination of the fire and the subsequent recession led to the difficult decision to close the bakery, which was later demolished.
01:42 - 02:08: Feeling burnt out, Scott was advised by his father to pursue a different career. However, Scott felt that baking was in his blood and was determined to revive the family business in McKees Rocks.
02:08 - 02:23: After researching the modern market and realizing the decline of traditional retail bakeries, Scott devises a new plan. He decides to adapt by creating a wholesale bakery to supply baked goods to other stores.
```
```json Structured Outputs expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"topic": "The story of a multi-generational family bakery, its history, its destruction by fire, and the determination to rebuild and adapt the business for a new era.",
"summary": "The video chronicles the history of the Jenny Lee Bakery, a beloved institution in McKees Rocks, Pennsylvania, run by the Baker family for generations. It details the bakery's founding in 1941, its role in the community, and the passion for baking passed down through generations. The story takes a tragic turn with a devastating fire and a recession, leading to the closure and demolition of the bakery. However, the narrative concludes with the current generation, Scott Baker, deciding to rebuild the business with a modern, wholesale-focused approach.",
"chapters": [
{
"start_time": "00:00:00",
"end_time": "00:00:15",
"description": "Scott Baker introduces himself and his family's deep-rooted connection to the McKees Rocks community through the Jenny Lee Bakery, which his grandfather opened in 1941."
},
{
"start_time": "00:00:15",
"end_time": "00:00:31",
"description": "A long-time employee and customer, Donna, shares fond memories of visiting the bakery for treats after church and later working there herself."
}
]
}
```
## Usage Example
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Caption the video
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Parse this video"},
{"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
]
}
],
)
print(response.choices[0].message.content)
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class ParsedVideoChapter(BaseModel):
start_time: str = Field(..., description="Start time in HH:MM:SS format")
end_time: str = Field(..., description="End time in HH:MM:SS format")
description: str = Field(..., description="Description of the chapter")
class ParsedVideoResponse(BaseModel):
topic: str = Field(..., description="Topic of the video")
summary: str = Field(..., description="Summary of the video content")
chapters: list[ParsedVideoChapter] = Field(..., description="Video chapters")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Parse the video with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Parse this video"},
{"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
]
}
],
response_format={"type": "json_schema", "schema": ParsedVideoResponse.model_json_schema()}
)
# Validate the response
result = ParsedVideoResponse.model_validate_json(response.choices[0].message.content)
# >>> ParsedVideoResponse(topic="...", summary="...", chapters=[...])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Parse this video" },
{ type: "video_url", video_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const ParsedVideoResponseSchema = z.object({
topic: z.string().describe("Topic of the video"),
summary: z.string().describe("Summary of the video content"),
chapters: z.array(z.object({
start_time: z.string().describe("Start time in HH:MM:SS format"),
end_time: z.string().describe("End time in HH:MM:SS format"),
description: z.string().describe("Description of the chapter")
})).describe("Video chapters")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Parse the video with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Parse this video" },
{ type: "video_url", video_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(ParsedVideoResponseSchema)
}
});
const result = ParsedVideoResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
You can ask simply ask for a more detailed caption by providing a more detailed prompt. In most cases, you can provide the number of words you want the caption to be, and the model will generate a more detailed caption.
* **Content Types**: presentation, tutorial, interview, documentary, news
* **Scenes**: office, outdoor, studio, classroom, conference room
* **People**: presenter, audience, speaker, interviewer
* **Objects**: whiteboard, charts, graphs, computer, microphone
The video segments come in the format of a list of dictionaries with start time, end time, and description fields.
Yes, the structured output includes segments with timestamps that break down the video into different parts with descriptions for each segment.
# Generate & Edit
Source: https://docs.vlm.run/agents/capabilities/video/generation
Create and edit videos with AI-powered tools for content creation and manipulation
Create and edit videos with AI-powered tools for content creation and manipulation. Perfect for content marketing, educational video creation, entertainment production, and training content generation.
## Example Usage
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
### Text-to-Video
Generate videos from text descriptions with creative control over style, composition, and motion.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Generate the video
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": "Generate a video of a serene mountain landscape with flowing clouds, cinematic wide shot, golden hour lighting, 4K, 20 seconds"
}
]
)
print(response.choices[0].message.content)
# >>> {"url": "https://.../video.mp4"}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class VideoGenerationResponse(BaseModel):
url: str = Field(..., description="The URL of the generated video")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Generate the video with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": "Generate a video of a serene mountain landscape with flowing clouds, cinematic wide shot, golden hour lighting, 4K, 20 seconds"
}
],
response_format={"type": "json_schema", "schema": VideoGenerationResponse.model_json_schema()}
)
# Validate the response
result = VideoGenerationResponse.model_validate_json(response.choices[0].message.content)
# >>> VideoGenerationResponse(url="https://.../video.mp4")
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: "Generate a video of a serene mountain landscape with flowing clouds, cinematic wide shot, golden hour lighting, 4K, 20 seconds"
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const VideoGenerationResponseSchema = z.object({
url: z.string().describe("The URL of the generated video")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Generate the video with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: "Generate a video of a serene mountain landscape with flowing clouds, cinematic wide shot, golden hour lighting, 4K, 20 seconds"
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(VideoGenerationResponseSchema)
}
});
const result = VideoGenerationResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
* **Cinematic**: High-end film quality with dramatic lighting and professional camera work
* **Realistic**: Ultra-realistic videos with natural motion and fine details
* **Artistic**: Watercolor, oil painting, sketch, and digital art styles
* **Creative**: Cyberpunk, fantasy, abstract, and minimalist designs
* **Scene Description**: Clearly describe the setting, characters, and actions
* **Camera Movement**: Specify camera angles, movements, and framing
* **Lighting & Mood**: Include lighting conditions and emotional tone
* **Technical Details**: Specify resolution, duration, and quality requirements
* **Motion Consistency**: Describe desired motion patterns and temporal flow
* **Resolutions**: Up to 1080p (1920x1080) for high-quality output
* **Frame Rates**: 24fps
* **Formats**: MP4
* **Duration**: Typically 30-60 seconds for optimal quality and processing time
Experience video generation with live examples in our interactive chat interface
# Video Tools
Source: https://docs.vlm.run/agents/capabilities/video/tools
Video tools for trimming, sampling, and extracting segments from videos
VLM Run's Orion agents can leverage various video-editing tools such as trimming, sampling, and extracting segments from videos. These tools are designed to help you extract key moments from videos, trim videos to specific segments, and sample frames from videos for analysis.
## Example Usage
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
### 1. Video Frame Sampling
Extract frames at regular intervals or specific timestamps for analysis.
```mdx wrap theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Extract at least 3 frames from the video for thumbnail generation.
```
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Extract keyframes for thumbnails
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract keyframes from this video for thumbnail generation, sampling every 5 seconds"},
{"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
]
}
]
)
print(response.choices[0].message.content)
# >>> {"frames": [{"url": "https://.../frame-1.jpg", "timestamp": "00:00:00.000"}, ...]}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class VideoFrame(BaseModel):
url: str = Field(..., description="The URL of the extracted frame")
timestamp: str = Field(..., description="Timestamp in HH:MM:SS.MS format")
class VideoSamplingResponse(BaseModel):
frames: list[VideoFrame] = Field(..., description="List of extracted frames")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Extract keyframes with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract keyframes from this video for thumbnail generation, sampling every 5 seconds"},
{"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
]
}
],
response_format={"type": "json_schema", "schema": VideoSamplingResponse.model_json_schema()}
)
# Validate the response
result = VideoSamplingResponse.model_validate_json(response.choices[0].message.content)
# >>> VideoSamplingResponse(frames=[VideoFrame(url="...", timestamp="..."), ...])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Extract keyframes from this video for thumbnail generation, sampling every 5 seconds" },
{ type: "video_url", video_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const VideoSamplingResponseSchema = z.object({
frames: z.array(z.object({
url: z.string().describe("The URL of the extracted frame"),
timestamp: z.string().describe("Timestamp in HH:MM:SS.MS format")
})).describe("List of extracted frames")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Extract keyframes with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Extract keyframes from this video for thumbnail generation, sampling every 5 seconds" },
{ type: "video_url", video_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(VideoSamplingResponseSchema)
}
});
const result = VideoSamplingResponseSchema.parse(JSON.parse(response.choices[0].message.content));
console.log(result);
// >>> { frames: [{ url: "https://.../frame-1.jpg", timestamp: "00:00:00.000" }, ...] }
```
***
### 2. Video Highlight Extraction
Our video agents can extract the best moments from a video, focusing on scoring plays and key actions.
```mdx wrap theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Extract the 3 best moments from this video, including the start and end times of each moment.
```
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Extract multiple segments
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract the 3 best moments from this video, including the start and end times of each moment."},
{"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
]
}
]
)
print(response.choices[0].message.content)
# >>> {"segments": [...]}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class HighlightVideo(BaseModel):
start_time: str = Field(..., description="Start time in HH:MM:SS.MS format")
end_time: str = Field(..., description="End time in HH:MM:SS.MS format")
url: str = Field(..., description="URL of the extracted segment")
class HighlightExtractionResponse(BaseModel):
segments: list[HighlightVideo] = Field(..., description="List of extracted segments")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Extract highlights with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract the 3 best moments from this video, including the start and end times of each moment."},
{"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
]
}
],
response_format={"type": "json_schema", "schema": HighlightExtractionResponse.model_json_schema()}
)
# Validate the response
result = HighlightExtractionResponse.model_validate_json(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Extract the 3 best moments from this video, including the start and end times of each moment." },
{ type: "video_url", video_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const HighlightExtractionResponseSchema = z.object({
segments: z.array(z.object({
start_time: z.string().describe("Start time in HH:MM:SS.MS format"),
end_time: z.string().describe("End time in HH:MM:SS.MS format"),
url: z.string().describe("URL of the extracted segment")
})).describe("List of extracted segments")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Extract highlights with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Extract the 3 best moments from this video, including the start and end times of each moment." },
{ type: "video_url", video_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(HighlightExtractionResponseSchema)
}
});
const result = HighlightExtractionResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
### 3. Time-Based Trimming
Extract specific segments from videos with precise start and end timestamps.
```mdx wrap theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Trim the video from 10 seconds to 30 seconds
```
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Trim video
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Trim the video from 10 seconds to 30 seconds"},
{"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
]
}
]
)
print(response.choices[0].message.content)
# >>> {"start_time": "00:00:10.000", "end_time": "00:00:30.000", "url": "https://.../trimmed.mp4"}
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class VideoResponse(BaseModel):
start_time: str = Field(..., description="Start time in HH:MM:SS.MS format")
end_time: str = Field(..., description="End time in HH:MM:SS.MS format")
url: str = Field(..., description="URL of the trimmed video")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Trim video with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Trim the video from 10 seconds to 30 seconds"},
{"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
]
}
],
response_format={"type": "json_schema", "schema": VideoResponse.model_json_schema()}
)
# Validate the response
result = VideoResponse.model_validate_json(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Trim the video from 10 seconds to 30 seconds" },
{ type: "video_url", video_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const VideoResponseSchema = z.object({
start_time: z.string().describe("Start time in HH:MM:SS.MS format"),
end_time: z.string().describe("End time in HH:MM:SS.MS format"),
url: z.string().describe("URL of the trimmed video")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Trim video with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Trim the video from 10 seconds to 30 seconds" },
{ type: "video_url", video_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(VideoResponseSchema)
}
});
const result = VideoResponseSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
* **MP4**: Most common format with excellent compatibility
* **MOV**: Apple QuickTime format
* **AVI**: Windows video format
* **MKV**: Matroska video format
* **WebM**: Web-optimized format
* **Quality Preservation**: Maintains original video quality in trimmed segments
* **Uniform Sampling**: Extract frames at regular intervals (e.g., every 1-5 seconds)
* **Keyframe Sampling**: Extract only keyframes for efficient analysis
* **Scene-Based**: Sample based on scene changes for better content analysis
* **Quality Balance**: Choose appropriate sampling rate based on analysis needs
* **Millisecond Precision**: Cut videos to exact time ranges with millisecond accuracy
* **Keyframe Alignment**: Align cuts to nearest keyframes for clean edits
* **Smart Boundaries**: Automatically detect optimal cut points
* **Quality Preservation**: Maintain video quality without re-encoding when possible
Experience video trimming and frame sampling with live examples in our interactive chat interface
# Chat with Orion
Source: https://docs.vlm.run/agents/chat
Bridging computer-vision tools to AI agents through language.
Navigate over to [Chat](https://chat.vlm.run/?utm_source=docs\&utm_medium=link\&utm_campaign=chat) to interact with our visual AI models in real-time and explore their capabilities.
Our [chat playground](https://chat.vlm.run/?utm_source=docs\&utm_medium=link\&utm_campaign=chat) provides an interactive interface for exploring visual AI capabilities of **VLM Run's Orion** family of visual agents through natural conversation. Upload images, documents, or videos and engage in dynamic conversations that leverage our advanced vision-language models for comprehensive analysis and structured data extraction.
Our [chat](https://chat.vlm.run/?utm_source=docs\&utm_medium=link\&utm_campaign=chat) is built on the same powerful foundation as our [Structured Responses API](/capabilities/structured-responses), but provides an intuitive conversational interface that makes visual AI accessible to everyone - from developers to business users.
We are working on adding more capabilities to the chat, and you can expect to see more features and capabilities added in the coming months.
## Capability Showcase
Explore the full range of visual AI capabilities through interactive examples:
Analyze images with detailed captions, object detection, segmentation, and visual question answering. Perfect for content moderation, accessibility, and automated image analysis.
Upload invoices, receipts, contracts, or any document and extract structured data automatically. The playground supports all our [pre-built document domains](/hub) including invoices, receipts, forms, and more.
Edit and generate images, and videos from text prompts or images. Get creative with your videos and images.
Engage in rich conversations that combine text, images, and structured data. Ask follow-up questions, request modifications, and explore different analysis approaches.
## Supported Content Types
The Chat Playground supports a wide range of visual content:
* **Images**: JPG, PNG, WebP (up to 10MB)
* **Documents**: PDF (up to 25MB, 30 pages max)
* **Videos**: MP4 (up to 100MB)
* **Audio**: MP3 (up to 25MB)
## Try our Chat Playground today
Head over to our [Chat Playground](https://chat.vlm.run) to start building your own visual AI workflows with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Code Execution
Source: https://docs.vlm.run/agents/code-execution
Orion-2 code-execution sandbox for composable visual pipelines
Orion-2 agents (`vlmrun-orion-2`) write and execute Python code in a secure,
sandboxed environment. Instead of invoking tools one at a time (Orion-1),
Orion-2 composes CV operations into multi-step pipelines — detect, crop,
annotate, measure, and transform — all within a single `execute_code` call.
## When to Use Orion-2
| Scenario | Recommended |
| ------------------------------------------------- | ----------- |
| Multi-step pipelines (detect → crop → annotate) | Orion-2 |
| Custom data transformations with numpy/matplotlib | Orion-2 |
| Iterative code refinement across turns | Orion-2 |
| Skill-based extraction with programmatic logic | Orion-2 |
## How It Works
Orion-2 is a visual agent harness: a planner and a code runtime wrapped around a vision-language model. It accepts text, images, video, and documents, compiles each request into an executable program, and dispatches visual tools and code execution from a single harness.
1. **Prompt → Spec**: An ambiguous request is compiled into an exact, executable program written in a visual DSL that reads like idiomatic Python.
2. **Execution**: The program runs in a sandboxed runtime with async-native parallelism — independent operations dispatch concurrently via `asyncio`, with no per-step model round-trips.
3. **Self-correction**: Execution results return to the harness, which repairs and re-executes until the program runs to completion.
Read the full [Orion-2 blog post](https://vlm.run/blog/orion-2) for architecture details, benchmarks, and live examples. To run Orion-2 from a Mastra agent with hybrid client and server tools, see [Mastra Compatibility](/agents/integrations/integrations-mastra).
### Orion-1 vs. Orion-2
The difference is clearest on a concrete task. Consider a virtual try-on that composes detection, cropping, and image generation across two input images.
**Orion-1** — sequential tool-calling, one LLM round-trip per tool:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Tools are called sequentially, with LLM reasoning at each step
boxes = tool_call("detect", image, target="person") # call 1
person = tool_call("crop", image, xywh=[0.22, 0.35, 0.04, 0.15]) # call 2
garment = tool_call("detect", dress_img, target="garment") # call 3
garment = tool_call("crop", dress_img, xywh=[0.33, 0.41, 0.05, 0.13]) # call 4
result = tool_call("generate", person, garment) # call 5
```
**Orion-2** — code-mode, one program with parallel dispatch:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import asyncio
async def process(ctx, person_image, dress_img):
vlmrun = ctx.import_lib("vlmrun")
def crop(img, d):
bx, by, bw, bh = d["xywh"]; W, H = img.width, img.height
return img.crop(int(by * H), int((by + bh) * H), int(bx * W), int((bx + bw) * W))
# Detect person and garment in parallel
p_det, g_det = await asyncio.gather(
vlmrun.image.detect(person_image, "person"),
vlmrun.image.detect(dress_img, "garment"),
)
person_crop = crop(person_image, p_det["detections"][0])
garment_crop = crop(dress_img, g_det["detections"][0])
# Composite the try-on
(composite,) = await vlmrun.image.generate(
"virtual try-on", images=[person_crop, garment_crop]
)
return {"composite": composite}
```
## Available Libraries
Inside the sandbox, the agent accesses libraries through `ctx.import_lib(...)`:
| Library | Import | Capabilities |
| ---------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------ |
| OpenCV | `ctx.import_lib("cv2")` | Classical CV operations, drawing, color conversion |
| NumPy | `ctx.import_lib("numpy")` | Array operations, math, linear algebra |
| Matplotlib | `ctx.import_lib("matplotlib")` | Plotting, charts, visualization |
| VLM Run | `ctx.import_lib("vlmrun")` | Detection, OCR, captioning, segmentation, generation, video, documents, [LLM text extraction](#vlmrun-llm-extract) |
| FFmpeg | `ctx.import_lib("ffmpeg")` | Video processing, frame extraction, transcoding |
Standard library modules (`json`, `math`, `re`, `pathlib`, `asyncio`, etc.) are
available via normal `import` statements.
### VLM Run Proxy API
The `vlmrun` proxy provides access to the full suite of CV capabilities:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Image operations
caption = await vlmrun.image.caption(img, "describe this image")
dets = await vlmrun.image.detect(img, "cars")
segments = await vlmrun.image.segment(img, "person")
points = await vlmrun.image.point(img, "eyes")
ocr = await vlmrun.image.ocr(img)
(gen,) = await vlmrun.image.generate("a sunset over mountains")
recon = await vlmrun.image.reconstruct_3d(img, mask_img, objects)
# Document operations
n = await vlmrun.document.length(doc_path)
pages = await vlmrun.document.get_pages(doc_path, offset=0, limit=3)
page_img = await vlmrun.document.get_page(doc_path, index=0)
# Video operations
report = await vlmrun.video.caption(vid_path, segment_duration=60.0)
video_paths = await vlmrun.video.generate("a timelapse of clouds", resolution="720p")
result = await vlmrun.video.segment(vid_path, prompts=["person", "car"])
# File I/O within the sandbox
content = vlmrun.io.read_file("data.json")
vlmrun.io.write_file("output.csv", csv_content)
path = await vlmrun.io.download("https://example.com/file.pdf")
# LLM-powered text extraction
result = await vlmrun.llm.extract(text, json_schema=schema)
result = await vlmrun.llm.extract(text) # free-form JSON when no schema
```
### vlmrun.llm.extract
Runs a text-only LLM extraction inside the sandbox. With a `json_schema`, it returns a
validated `dict` matching the schema; without one, it returns free-form JSON parsed from
the model response. When a skill has a `schema.json` file, the pipeline can read and pass
it directly:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Inside execute_code
schema = json.loads(vlmrun.io.read_file("skills/my-skill/schema.json"))
result = await vlmrun.llm.extract(raw_text, json_schema=schema)
```
## Example: Chat Completion with Orion-2
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.agent.completions.create(
model="vlmrun-orion-2:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Detect all cars in this image, draw bounding boxes, and count them."},
{"type": "image_url", "image_url": {"url": "https://example.com/parking-lot.jpg"}}
]
}
],
)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
baseURL: "https://api.vlm.run/v1",
apiKey: "",
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-2:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Detect all cars in this image, draw bounding boxes, and count them." },
{ type: "image_url", image_url: { url: "https://example.com/parking-lot.jpg" } }
]
}
],
});
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/openai/chat/completions \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{
"model": "vlmrun-orion-2:auto",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Detect all cars in this image, draw bounding boxes, and count them."},
{"type": "image_url", "image_url": {"url": "https://example.com/parking-lot.jpg"}}
]
}
]
}'
```
The agent will automatically write and execute code like:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
async def process(ctx, img):
cv2 = ctx.import_lib("cv2")
vlmrun = ctx.import_lib("vlmrun")
dets = await vlmrun.image.detect(img, "cars")
W, H = img.width, img.height
for d in dets["detections"]:
bx, by, bw, bh = d["xywh"]
x, y, w, h = int(bx * W), int(by * H), int(bw * W), int(bh * H)
cv2.rectangle(img, (x, y), (x + w, y + h), (0, 255, 0), 2)
return {"count": len(dets["detections"]), "annotated_image": img}
```
## Skills with Orion-2
When skills are attached to an Orion-2 request, the skill workspace is materialized
into the session directory at `/skills//`. The agent can
read skill resources (SKILL.md, schemas, templates) directly using `vlmrun.io.read_file`
or `cv2.imread` — no special API calls needed.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Inside execute_code, the agent can read skill resources:
skill_instructions = await vlmrun.io.read_file("skills/invoice-extraction/SKILL.md")
schema = await vlmrun.io.read_file("skills/invoice-extraction/schema.json")
```
Skills work with both Orion-1 and Orion-2. Orion-1 injects skill instructions into the system prompt, while Orion-2 materializes skill files into the workspace for programmatic access.
## Program Execution
When an Orion-2 skill has been run at least once, the agent's authored `pipeline.py` is
cached inside the skill bundle. On subsequent executions, the platform can run that
pipeline *directly* through the code-execution sandbox, bypassing the LLM agent loop
entirely. This is called **program** execution: the cached pipeline is the compiled
program that you built once and now just run.
### How it works
1. **First run (authoring)**: The agent plans and writes `pipeline.py`. The code is
persisted into the skill's stored bundle for reuse.
2. **Subsequent runs (replay)**: The cached `pipeline.py` executes directly via
`CodeExecutionRunner`. If execution fails or no cached pipeline exists, the system
falls back to the full agent loop automatically.
### Controlling execution mode
Use `mode` in your execution config. It accepts `program` (default) or `agent`:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionConfig
client = VLMRun(api_key="")
# Default: run the cached pipeline.py as a fixed program when available
response = client.agent.execute(
inputs={"file": "https://example.com/document.pdf"},
model="vlmrun-orion-2:pro",
config=AgentExecutionConfig(
mode="program", # default
skills=[{"skill_id": "my-skill-id"}],
),
)
# Force full agent loop (e.g. for authoring or debugging)
response = client.agent.execute(
inputs={"file": "https://example.com/document.pdf"},
model="vlmrun-orion-2:pro",
config=AgentExecutionConfig(
mode="agent",
skills=[{"skill_id": "my-skill-id"}],
),
)
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Program (default)
curl -X POST https://api.vlm.run/v1/agent/execute \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "vlmrun-orion-2:pro",
"inputs": { "file": { "type": "file_url", "file_url": { "url": "https://example.com/document.pdf" } } },
"config": {
"mode": "program",
"skills": [{ "skill_id": "my-skill-id" }]
}
}'
# Force agent loop
curl -X POST https://api.vlm.run/v1/agent/execute \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "vlmrun-orion-2:pro",
"inputs": { "file": { "type": "file_url", "file_url": { "url": "https://example.com/document.pdf" } } },
"config": {
"mode": "agent",
"skills": [{ "skill_id": "my-skill-id" }]
}
}'
```
The response includes `execution_mode` indicating which path was taken:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"execution_id": "exec_abc123",
"status": "completed",
"execution_mode": "program",
"response": { ... }
}
```
| `execution_mode` | Meaning |
| ---------------- | -------------------------------------------------------- |
| `program` | Ran cached `pipeline.py` directly (no LLM orchestration) |
| `agent` | Ran the full LLM agent loop |
Program execution preserves structured output validation, grounding metadata, and billing accuracy. Billing for program runs captures sandbox and tool costs only: zero LLM orchestration tokens.
### Performance
Program execution can be an order of magnitude faster than the full agent loop. In
testing, a medical-referral document extraction skill completed in \~13s in program mode vs.
\~160s with full agent orchestration.
## Security
The code execution sandbox enforces strict security boundaries:
* **Import restrictions**: Only allowlisted libraries (`cv2`, `numpy`, `matplotlib`, `vlmrun`, `ffmpeg`) via `ctx.import_lib()`, plus Python stdlib. Dangerous modules (`os`, `io`, `shutil`, `importlib`) are blocked at AST parse time.
* **Workspace confinement**: All file operations are restricted to the session workspace. Symlink traversal and absolute path escapes are rejected.
* **Introspection blocking**: Builtins like `eval`, `exec`, `compile`, `getattr`, and `__import__` are blocked to prevent sandbox escape.
## Model Variants
Orion-2 is model-agnostic — the same harness and runtime work with any multimodal model that has strong code generation. The default `vlmrun-orion-2:auto` routes each request to the best backbone for the job.
| Model ID | Description |
| -------------------------------- | ------------------------------------------------------------------ |
| `vlmrun-orion-2:fast` | Optimized for speed and cost-efficiency |
| `vlmrun-orion-2:auto` | Automatically routes to the best backend for each task (default) |
| `vlmrun-orion-2:pro` | Most capable tier for complex multi-step workflows |
| `vlmrun-orion-2:qwen3.6-35b-a3b` | Open-weight Qwen 3.6 35B — strong at code generation and reasoning |
| `vlmrun-orion-2:gemma4-26b-a4b` | Open-weight Gemma 4 26B — strong at localization and spatial tasks |
| `vlmrun-orion-2:kimi-2.6` | Kimi 2.6 — strong at multi-turn dialogue and long-context tasks |
| `vlmrun-orion-2:gpt-5.5` | GPT-5.5 — strong at instruction following and structured output |
| `vlmrun-orion-2:claude-opus-4.8` | Claude Opus 4.8 — strong at nuanced reasoning and analysis |
# Multi-modal Inputs
Source: https://docs.vlm.run/agents/inputs
Encode images, videos, documents, and other media in a consistent format for agent execution and chat completions
Multi-modal inputs allow you to pass various types of content—text, images, videos, audio files, and documents—to agents in a consistent, type-safe format. The `MessageContent` type provides a unified interface for encoding different media types, whether you're executing agents or using chat completions.
## MessageContent Overview
`MessageContent` is a Pydantic model that encapsulates different types of input content with validation. It supports six content types:
| Type | Description | Use Case |
| ------------ | ------------------------------- | ----------------------------------------------------- |
| `text` | Plain text content | Instructions, questions, or text-based prompts |
| `image_url` | Image from a public URL | Images hosted on the web or cloud storage |
| `video_url` | Video from a public URL | Videos hosted on the web or cloud storage |
| `audio_url` | Audio from a public URL | Audio files hosted on the web or cloud storage |
| `file_url` | Generic file from a public URL | Documents (PDFs, Word docs, etc.) or other file types |
| `input_file` | File uploaded via the Files API | Files uploaded to VLM Run storage using file IDs |
Import `MessageContent` and related types from the SDK:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.types import MessageContent, ImageUrl, VideoUrl, AudioUrl, FileUrl
```
```javascript Node theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { MessageContent, ImageUrl, VideoUrl, AudioUrl, FileUrl } from '@vlmrun/sdk';
```
## Input Types
### Text Input
Use `text` for plain text instructions, questions, or prompts:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.types import MessageContent
# Simple text input
text_content = MessageContent(type="text", text="Analyze this image and describe what you see")
```
```javascript Node theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { MessageContent } from '@vlmrun/sdk';
// Simple text input
const textContent = new MessageContent({ type: 'text', text: 'Analyze this image and describe what you see' });
```
### Image Input
Use `image_url` for images accessible via HTTP/HTTPS URLs. The `ImageUrl` type supports an optional `detail` parameter to control image processing quality:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.types import MessageContent, ImageUrl
# Image with default detail level (auto)
image_content = MessageContent(
type="image_url",
image_url=ImageUrl(url="https://example.com/photo.jpg")
)
# Image with high detail for better quality processing
image_content = MessageContent(
type="image_url",
image_url=ImageUrl(url="https://example.com/photo.jpg", detail="high")
)
```
```javascript Node theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { MessageContent, ImageUrl } from '@vlmrun/sdk';
// Image with default detail level (auto)
const imageContent = new MessageContent({ type: 'image_url', image_url: new ImageUrl({ url: 'https://example.com/photo.jpg' }) });
// Image with high detail for better quality processing
const imageContent = new MessageContent({ type: 'image_url', image_url: new ImageUrl({ url: 'https://example.com/photo.jpg', detail: 'high' }) });
```
```bash Bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg", "detail": "high"}}
```
The `detail` parameter accepts:
* `"auto"` (default): Automatically determines the appropriate detail level
* `"low"`: Lower resolution, faster processing
* `"high"`: Higher resolution, more detailed analysis
### Video Input
Use `video_url` for videos accessible via HTTP/HTTPS URLs:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.types import MessageContent, VideoUrl
video_content = MessageContent(
type="video_url",
video_url=VideoUrl(url="https://example.com/video.mp4")
)
```
```javascript Node theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { MessageContent, VideoUrl } from '@vlmrun/sdk';
// Video with default detail level (auto)
const videoContent = new MessageContent({ type: 'video_url', video_url: new VideoUrl({ url: 'https://example.com/video.mp4' }) });
```
```bash Bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}}
```
### Audio Input
Use `audio_url` for audio files accessible via HTTP/HTTPS URLs:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.types import MessageContent, AudioUrl
audio_content = MessageContent(
type="audio_url",
audio_url=AudioUrl(url="https://example.com/audio.mp3")
)
```
```javascript Node theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { MessageContent, AudioUrl } from '@vlmrun/sdk';
// Audio with default detail level (auto)
const audioContent = new MessageContent({ type: 'audio_url', audio_url: new AudioUrl({ url: 'https://example.com/audio.mp3' }) });
```
```bash Bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{"type": "audio_url", "audio_url": {"url": "https://example.com/audio.mp3"}}
```
### Document / File Input (URL)
Use `file_url` for documents and other file types accessible via HTTP/HTTPS URLs:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.types import MessageContent, FileUrl
document_content = MessageContent(
type="file_url",
file_url=FileUrl(url="https://example.com/document.pdf")
)
```
```javascript Node theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { MessageContent, FileUrl } from '@vlmrun/sdk';
// Document with default detail level (auto)
const documentContent = new MessageContent({ type: 'file_url', file_url: new FileUrl({ url: 'https://example.com/document.pdf' }) });
```
```bash Bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{"type": "file_url", "file_url": {"url": "https://example.com/document.pdf"}}
```
### Document / File Input (Upload)
Use `input_file` with a file ID for files uploaded via the Files API. This is the recommended approach for files you want to manage through VLM Run's file storage:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.types import MessageContent
from vlmrun.client import VLMRun
from pathlib import Path
client = VLMRun(api_key="")
# Step 1: Upload the file
file_response = client.files.upload(file=Path("local_image.jpg"))
# Step 2: Use the file ID in MessageContent
file_content = MessageContent(
type="input_file",
file_id=file_response.id
)
```
```javascript Node theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { MessageContent } from '@vlmrun/sdk';
// File with default detail level (auto)
const fileContent = new MessageContent({ type: 'input_file', file_id: '' });
```
```bash Bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{"type": "input_file", "file_id": ""}
```
When using `input_file`, you can provide either `file_id` (from Files API upload) or `file_url` (presigned URL or public URL). The SDK automatically handles file retrieval and processing.
## Using Multi-modal Inputs
Agents can accept multiple inputs of different types. Define each input as a separate field in your input model:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pydantic import BaseModel, Field
from vlmrun.types import MessageContent, ImageUrl, FileUrl
class MultiModalInputs(BaseModel):
image: MessageContent = Field(..., description="The reference image")
document: MessageContent = Field(..., description="The document to process")
instruction: MessageContent = Field(..., description="Processing instructions")
inputs = MultiModalInputs(
image=MessageContent(
type="image_url",
image_url=ImageUrl(url="https://example.com/reference.jpg")
),
document=MessageContent(
type="file_url",
file_url=FileUrl(url="https://example.com/document.pdf")
),
instruction=MessageContent(
type="text",
text="Extract information matching the reference image format"
)
)
```
### In an Agent Execution
When executing agents, define typed and compound input models using `MessageContent` for type safety and validation:
```python Python [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pydantic import BaseModel, Field
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionConfig, AgentExecutionResponse, ImageUrl
from vlmrun.types import MessageContent, ImageRef
client = VLMRun(api_key="")
# Define typed inputs using MessageContent
class ExecutionInputs(BaseModel):
image: MessageContent = Field(..., description="The input image to process")
ref_image: MessageContent = Field(..., description="The reference style image to use for style transfer")
class ImageResponse(BaseModel):
image: ImageRef = Field(..., description="The stylized output image")
# Execute agent with image URL
execution: AgentExecutionResponse = client.agent.execute(
name="image/blur-image",
inputs=ExecutionInputs(
image=MessageContent(
type="image_url",
image_url=ImageUrl(url="https://example.com/photo.jpg")
),
reference=MessageContent(
type="image_url",
image_url=ImageUrl(url="https://example.com/style.jpg")
)
),
config=AgentExecutionConfig(
prompt="Blur all faces in the image",
response_model=ImageResponse
)
)
```
```python Python - Uploaded File [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from pydantic import BaseModel, Field
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionConfig, AgentExecutionResponse
from vlmrun.types import MessageContent, ImageRef
client = VLMRun(api_key="")
class ExecutionInputs(BaseModel):
image: MessageContent = Field(..., description="The input image to process")
ref_image: MessageContent = Field(..., description="The reference style image to use for style transfer")
class ImageResponse(BaseModel):
image: ImageRef = Field(..., description="The stylized output image")
# Upload file first
file_response = client.files.upload(file=Path("local_image.jpg"))
ref_file_response = client.files.upload(file=Path("local_reference.jpg"))
# Execute agent with uploaded file
execution: AgentExecutionResponse = client.agent.execute(
name="test-image-captioner",
inputs=ExecutionInputs(
image=MessageContent(type="input_file", file_id=file_response.id),
ref_image=MessageContent(type="input_file", file_id=ref_file_response.id)
),
config=AgentExecutionConfig(
prompt="Describe the image in detail",
response_model=ImageResponse
)
)
```
```python Python - Document Processing [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from typing import Literal
from pydantic import BaseModel, Field
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionConfig, AgentExecutionResponse
from vlmrun.types import MessageContent, FileUrl
client = VLMRun(api_key="")
class ExecutionInputs(BaseModel):
url: MessageContent = Field(..., description="The document URL to process")
class DocumentResponse(BaseModel):
reasoning: str = Field(..., description="The reasoning for the classification")
classification: Literal["referral_letter", "physician_order", "insurance_authorization", "patient_consent_form", "miscellaneous"] = Field(..., description="The classification of the document")
prompt = (
"Classify each page of the document into one of these categories:\n"
"- Referral Letter\n"
"- Physician Order\n"
"- Insurance Authorization\n"
"- Patient Consent Form\n"
"- Miscellaneous\n\n"
"Provide a brief reason for your classification."
)
execution: AgentExecutionResponse = client.agent.execute(
inputs=ExecutionInputs(
url=MessageContent(
type="file_url",
file_url=FileUrl(url="https://example.com/document.pdf")
)
),
config=AgentExecutionConfig(prompt=prompt, response_model=DocumentResponse)
)
```
### In a Chat Completion
For chat completions, use arrays of content objects in the OpenAI-compatible format. Each message can contain multiple content items:
```python Python - Text and Image [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Blur all the faces in this image"},
{"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg", "detail": "auto"}}
]
}
]
)
```
```python Python - Video Processing [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Trim this video to the first 10 seconds"},
{"type": "video_url", "video_url": {"url": "https://example.com/video.mp4", "detail": "auto"}}
]
}
],
response_format={
"type": "json_schema",
"schema": {
"type": "object",
"properties": {
"video": {"type": "object"}
}
}
}
)
```
```python Python - Multiple Images [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Compare these two images and describe the differences"},
{"type": "image_url", "image_url": {"url": "https://example.com/image1.jpg"}},
{"type": "image_url", "image_url": {"url": "https://example.com/image2.jpg"}}
]
}
]
)
```
```python Python - File Input [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.types import MessageContent, FileUrl
client = VLMRun(api_key="")
# Upload file first
file_response = client.files.upload(file=Path("local_document.pdf"))
# Execute agent with uploaded file
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract information from the document"},
{"type": "input_file", "file_id": file_response.id}
]
}
],
response_format={"type": "json_schema", "schema": {
"type": "object",
"properties": {
"reasoning": {"type": "string"},
"classification": {"type": "string"}
},
"required": ["reasoning", "classification"]
}}
)
```
## Toolset Selection
The `toolsets` parameter allows you to explicitly specify which tool categories the agent should use for processing your request. This gives you fine-grained control over the agent's capabilities and can improve performance by limiting the tools to only those needed for your task.
### Available Tool Categories
| Category | Description |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `core` | Essential tools for analyzing images, extracting content from documents, and processing video - the fundamental capabilities for most tasks |
| `image` | Comprehensive image understanding including object detection, text recognition, UI element detection, segmentation, and visual quality assessment |
| `image-gen` | Create new images from text descriptions, transform existing images, and apply visual effects like blurring regions |
| `world_gen` | Generate 3D models from images, including object-level reconstruction and full scene reconstruction |
| `viz` | Annotate images with bounding boxes, keypoints, and segmentation masks for visual output |
| `document` | Process documents with layout detection, text extraction, content parsing, and structured data extraction |
| `video` | Video processing capabilities including frame sampling, trimming, segmentation, and video generation |
| `web` | Search the web for information to augment agent responses with real-time data |
### Usage Examples
```python Python - Chat Completions theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
# Use specific toolsets for image analysis
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Detect all objects in this image and draw bounding boxes"},
{"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
]
}
],
toolsets=["image", "viz"]
)
```
```python Python - Agent Execution theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionConfig
client = VLMRun(api_key="")
# Execute agent with document processing tools
response = client.agent.execute(
name="document-processor",
inputs={"url": "https://example.com/document.pdf"},
toolsets=["document", "core"],
config=AgentExecutionConfig(prompt="Extract all tables from this document")
)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Use specific toolsets for video processing
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Trim this video to the first 30 seconds" },
{ type: "video_url", video_url: { url: "https://example.com/video.mp4" } }
]
}
],
toolsets: ["video", "core"]
});
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/openai/chat/completions \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{
"model": "vlmrun-orion-1:auto",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Generate a new image based on this description"},
{"type": "text", "text": "A serene mountain landscape at sunset"}
]
}
],
"toolsets": ["image-gen"]
}'
```
When you know exactly which capabilities your task requires, specifying toolsets can improve response time and reduce costs by avoiding unnecessary tool routing overhead.
## Best Practices
When working with multi-modal inputs, follow these guidelines:
* **Use `input_file` for production**: Upload files via the Files API and use `file_id` for better security, access control, and file management. URL-based inputs are convenient for development and testing.
* **Specify image detail levels**: Use `detail="high"` for images requiring fine-grained analysis (e.g., medical imaging, document OCR). Use `detail="low"` for faster processing when high detail isn't needed.
* **Validate URLs before use**: Ensure all URLs are publicly accessible and use HTTPS when possible. The SDK validates URL format but cannot verify accessibility.
* **Use typed input models**: Define Pydantic models for agent execution inputs to leverage type checking, IDE autocompletion, and automatic validation.
* **Handle large files appropriately**: For large videos or documents, prefer uploading via the Files API rather than using public URLs, as the Files API provides better error handling and progress tracking.
* **Combine text with media**: Always include text instructions alongside media inputs to provide context and specify the desired operation.
For chat completions, you can mix text and media in a single message's content array. This allows you to provide both instructions and the media to process in one request.
### URL Validation
All URL-based input types (`image_url`, `video_url`, `audio_url`, `file_url`) require valid HTTP or HTTPS URLs. The SDK automatically validates URLs:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.types import ImageUrl
# Valid - HTTP URL
image_url = ImageUrl(url="http://example.com/image.jpg")
# Valid - HTTPS URL
image_url = ImageUrl(url="https://example.com/image.jpg")
# Invalid - Will raise ValueError
image_url = ImageUrl(url="file:///local/path/image.jpg") # ❌ Not HTTP/HTTPS
```
## Common Use Cases
Process images with text instructions for classification, object detection, or transformation.
Extract structured data from PDFs, Word documents, and other file formats.
Analyze video content for transcription, scene detection, or frame extraction.
Combine multiple input types (text, images, documents) for complex processing workflows.
## Related Documentation
Learn how to retrieve generated artifacts from agent responses
Execute agents with multi-modal inputs and retrieve structured results
Create reusable agents that accept multi-modal inputs
# Instructor Compatibility
Source: https://docs.vlm.run/agents/integrations/integrations-instructor
Run VLM Run Agents with the Instructor Python SDK with minimal code changes.
With our new [OpenAI-compatible API](/api-reference/v1/post-chat-completions), you can use the popular [Instructor](https://github.com/instructor-ai/instructor) library to interact with VLM Run. This allows developers to switch between OpenAI, Instructor and VLM Run APIs with minimal changes.
## Configure the OpenAI-compatible Instructor Client with VLM Run
Since Instructor is compatible with the OpenAI API, you can use the same configuration methods as described in the [OpenAI Compatibility](/agents/integrations/integrations-openai-compatibility#configure-the-default-endpoint-and-api-key) page.
Here's an example of how to configure the Instructor client to work with VLM Run:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import instructor
import openai
# Configure the OpenAI client
client = openai.OpenAI(
base_url="https://api.vlm.run/v1/openai",
api_key=""
)
# Configure the Instructor client
inst_client = instructor.from_openai(
client, mode=instructor.Mode.MD_JSON
)
```
## 1. Chat Completion with Instructor
Now that you have configured the Instructor client, you can use the `inst_client.chat.completions.create` method to interact with VLM Run.
Below is an example of how to use the Instructor client to create a chat completion:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pydantic import BaseModel, Field
# Let's define a simplified Pydantic model to represent the outputs. Let's say we want to generate an image of big ben at a distance, and crop into the clock face.
class AgentResponse(BaseModel):
image_url: str = Field(description="The pre-signed URL of the image generated.")
clock_image_url: str = Field(description="The pre-signed URL of the image of the clock face cropped from the generated image.")
crop_xywh: tuple[float, float, float, float] = Field(description="The (x, y, width, height) of the clock face cropped from the generated image.")
# Now we can use the Instructor client to create a chat completion
response = inst_client.chat.completions.create(
model="vlmrun-orion-1:auto",
max_retries=0,
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Generate an image of big ben at a distance. Crop into the clock face and provide a close up of the clock face."},
],
}
],
response_model=AgentResponse,
)
logger.debug(f"type={type(response)}, response={response}")
```
## 2. Guided Chat Completion with Instructor
The example above is an unconstrained chat completion request, relying solely on the JSON schema provided to the VLM Run Agents as an instruction. In order to enforce the JSON schema, you can use the `json_schema` extra body parameter to guide the VLM Run Agents model to extract structured data from the image.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Adapt the above example with an additional "json_schema" parameter
# to enforce guided decoding of the JSON in the schema.
response = inst_client.chat.completions.create(
... # same as above
response_model=AgentResponse,
response_format={"type": "json_schema", "json_schema": AgentResponse.model_json_schema()},
)
```
This will ensure that the VLM Run model generates the image and crops the clock face in the specified JSON format.
# Mastra Compatibility
Source: https://docs.vlm.run/agents/integrations/integrations-mastra
Use Orion-2 with Mastra for document, image, audio, and video work with hybrid client tools.
Point a [Mastra](https://mastra.ai/) `Agent` at the VLM Run OpenAI-compatible chat endpoint. Mastra runs your local tools; Orion-2 runs document, image, audio, and video work in its [code-execution sandbox](/agents/code-execution).
## Quick start
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
npm install @mastra/core @ai-sdk/openai-compatible zod
export VLMRUN_API_KEY=""
```
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { Agent } from "@mastra/core/agent";
import { createTool } from "@mastra/core/tools";
import { z } from "zod";
const vlmrun = createOpenAICompatible({
name: "vlmrun",
baseURL: "https://api.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY!,
});
const lookupPurchaseOrder = createTool({
id: "lookup_purchase_order",
description: "Return the purchase order your ERP has on file for an invoice number",
inputSchema: z.object({ invoiceNumber: z.string() }),
execute: async ({ context }) => ({
invoiceNumber: context.invoiceNumber,
poNumber: "PO-4417",
total: 4560.0,
}),
});
const agent = new Agent({
id: "orion2-mastra-agent",
name: "Orion-2 Mastra Agent",
model: vlmrun.chatModel("vlmrun-orion-2:auto"),
tools: { lookupPurchaseOrder },
instructions:
"Use execute_code for attached documents, images, audio, or video. " +
"Call lookup_purchase_order only when asked to reconcile an invoice. " +
"Never invent field values.",
});
const runOptions = () => ({
providerOptions: {
vlmrun: {
session_id: crypto.randomUUID(),
tool_execution: "hybrid",
},
},
});
```
| Setting | Why |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `tool_execution: "hybrid"` | Orion-2 runs `execute_code`; client tools pause back to Mastra |
| `session_id` | Fresh UUID per turn for workspace / [artifacts](/agents/artifacts) |
| URLs in the prompt | AI SDK OpenAI-compatible only forwards image file parts; put PDF / audio / video URLs in the prompt text so Orion-2 can download them |
`providerOptions.vlmrun` is merged into the chat-completions body (provider name must match `createOpenAICompatible({ name: "vlmrun" })`).
Use `vlmrun-orion-2:auto` for the default tier. Other variants are listed in [Code Execution](/agents/code-execution#model-variants). Get an API key from [API Keys](https://app.vlm.run/dashboard/settings/api-keys).
## Examples
Public sample files — same agent and `runOptions()` for each. Prefer `agent.stream` (some VLM Run keys only allow `stream=True`).
### Document
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const INVOICE =
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/wordpress-pdf-invoice-plugin-sample.pdf";
const out = await agent.stream(
`Extract vendor, invoice number, date, and total from ${INVOICE}.`,
runOptions(),
);
console.log(await out.text);
```
### Image
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const DONUTS =
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.object-detection/donuts.png";
const out = await agent.stream(
`Count the donuts and briefly describe the image: ${DONUTS}`,
runOptions(),
);
console.log(await out.text);
```
### Audio
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const AUDIO =
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/audio.transcription-summary/two_minute_rules.mp3";
const out = await agent.stream(
`Summarize this audio in 2 sentences: ${AUDIO}`,
runOptions(),
);
console.log(await out.text);
```
### Video
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const VIDEO =
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4";
const out = await agent.stream(
`Describe this video in 2 sentences: ${VIDEO}`,
runOptions(),
);
console.log(await out.text);
```
### Hybrid: extract + local ERP tool
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const out = await agent.stream(
`Extract the invoice number and total from ${INVOICE} with execute_code. ` +
"Then call lookup_purchase_order once and say if the totals match.",
runOptions(),
);
console.log(await out.text);
```
## What runs where
| Capability | Who runs it |
| -------------------------------- | --------------------------------------- |
| Document / image / audio / video | Orion-2 (`execute_code`) |
| ERP / DB / internal APIs | Your Mastra `createTool` |
| Streaming | `agent.stream(...)` (see video example) |
## Related
Orion-2 sandbox, libraries, and model variants
The same hybrid contract from a Python agent
Document, image, audio, and video content parts
Retrieve annotated images, clips, and generated files
# OpenAI Compatibility
Source: https://docs.vlm.run/agents/integrations/integrations-openai-compatibility
Run VLM Run Agents with the OpenAI Python SDK with just 2 lines of code change.
With our OpenAI-compatible API, you can use the [OpenAI Python SDK](https://github.com/openai/openai-python) to interact with VLM Run Agents. This allows developers to trivially switch between OpenAI and VLM Run APIs without having to change any code.
Our VLM Agents are fully compatible with the OpenAI API. Notably, our API also supports a whole range of features with multi-modal data types that OpenAI currently does not support. Our OpenAI-Compatible endpoint is available at `https://api.vlm.run/v1/openai`.
In order to use the VLM Run Agents API, you simply need to override the default endpoint and API key when using the OpenAI Python SDK. The API key can be found in the [API Keys](https://app.vlm.run/dashboard/settings/api-keys) page of the VLM Run dashboard.
## OpenAI Client Configuration
Override the default endpoint and API key by initializing the OpenAI client with the following configuration:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import openai
client = openai.OpenAI(
base_url="https://api.vlm.run/v1/openai",
api_key=""
)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { OpenAI } from "openai";
const client = new OpenAI({
baseURL: "https://api.vlm.run/v1/openai",
apiKey: ""
});
```
Alternatively, you can also set the following environment variables to achieve the same effect:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
export OPENAI_API_BASE="https://api.vlm.run/v1/openai"
export OPENAI_API_KEY=""
```
### Usage 1: Basic Chat Completion
Once you have set the endpoint and API key, you can use the OpenAI Python SDK as you normally would. Note that the only change required to the `client.chat.completions.create` method is the `extra_body` field that allows you to specify the `domain` and additional request `metadata`.
For example:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import openai
# Initialize the OpenAI client
client = openai.OpenAI(
base_url="https://api.vlm.run/v1/openai", api_key=""
)
# Example: Chat completion with an image input
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": [
{"type": "text", "text": "What's in this image?"},
{"type": "image_url", "image_url": {"url": "", "detail": "auto"}},
]}
]
# Perform chat completion
chat_completion = client.chat.completions.create(
model="vlmrun-orion-1:auto",
messages=messages,
temperature=0,
extra_body={"session_id": ""}, # optional session id for persistence
)
print(chat_completion.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { OpenAI } from "openai";
// Initialize the OpenAI client
const client = new OpenAI({
baseURL: "https://api.vlm.run/v1/openai",
apiKey: ""
});
// Example: Chat completion with an image input
const messages = [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: [
{ type: "text", text: "What's in this image?" },
{ type: "image_url", image_url: { url: "", detail: "auto" } }
]}
];
// Perform chat completion
const chatCompletion = await client.chat.completions.create({
model: "vlmrun-orion-1:auto",
messages: messages,
temperature: 0,
extra_body: { session_id: "" }
});
console.log(chatCompletion.choices[0].message.content);
```
### Usage 2: Chat Completion with Structured Outputs
```python Python expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import openai
from pydantic import BaseModel, Field
class ImageCaption(BaseModel):
caption: str = Field(..., description="Detailed caption of the scene")
tags: list[str] = Field(..., description="Tags that describe the image")
# Initialize the OpenAI client
client = openai.OpenAI(
base_url="https://api.vlm.run/v1/openai", api_key=""
)
# Example: Chat completion with an image input
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": [
{"type": "text", "text": "What's in this image?"},
{"type": "image_url", "image_url": {"url": "", "detail": "auto"}},
]}
]
# Perform chat completion (with JSON Schema)
chat_completion = client.chat.completions.create(
model="vlmrun-orion-1:auto",
messages=messages,
response_format={"type": "json_schema", "schema": ImageCaption.model_json_schema()},
)
print(chat_completion.choices[0].message.content)
# >> {"caption": "...", "tags": [...]}
print(ImageCaption.model_validate_json(chat_completion.choices[0].message.content))
# >> ImageCaption(caption="...", tags=[...])
```
```typescript Node.js expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { OpenAI } from "openai";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the schema with Zod
const ImageCaptionSchema = z.object({
caption: z.string().describe("Detailed caption of the scene"),
tags: z.array(z.string()).describe("Tags that describe the image")
});
// Initialize the OpenAI client
const client = new OpenAI({
baseURL: "https://api.vlm.run/v1/openai",
apiKey: ""
});
// Example: Chat completion with an image input
const messages = [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: [
{ type: "text", text: "What's in this image?" },
{ type: "image_url", image_url: { url: "", detail: "auto" } }
]}
];
// Perform chat completion (with JSON Schema)
const chatCompletion = await client.chat.completions.create({
model: "vlmrun-orion-1:auto",
messages: messages,
response_format: {
type: "json_schema",
schema: zodToJsonSchema(ImageCaptionSchema)
}
});
// Validate the response with Zod
const result = ImageCaptionSchema.parse(JSON.parse(chatCompletion.choices[0].message.content));
console.log(result);
// >> { caption: "...", tags: [...] }
```
```python Instructor theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import instructor
from openai import OpenAI
from pydantic import BaseModel, Field
class ImageCaption(BaseModel):
caption: str = Field(..., description="Detailed caption of the scene")
tags: list[str] = Field(..., description="Tags that describe the image")
# Initialize the Instructor client (OpenAI)
client = instructor.from_openai(OpenAI(
base_url="https://api.vlm.run/v1/openai", api_key=""
))
# Define the messages
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": [
{"type": "text", "text": "What's in this image?"},
{"type": "image_url", "image_url": {"url": "", "detail": "auto"}}
]}
]
response = client.chat.completions.create(
model="vlmrun-orion-1:auto",
messages=messages,
response_model=ImageCaption,
)
print(response)
# >>> ImageCaption(caption="...", tags=[...])
```
### Usage 3: Basic Chat Completion with Streaming
```python Python expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import openai
# Initialize the OpenAI client
client = openai.OpenAI(
base_url="https://api.vlm.run/v1/openai", api_key=""
)
# Define the messages
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": [
{"type": "text", "text": "What's in this image?"},
{"type": "image_url", "image_url": {"url": "https://example.com/image.jpg", "detail": "auto"}}
]}
]
# Perform chat completion (with streaming)
chat_completion = client.chat.completions.create(
model="vlmrun-orion-1:auto",
messages=messages,
temperature=0,
stream=True,
)
for chunk in chat_completion:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
```
```typescript Node.js expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { OpenAI } from "openai";
// Initialize the OpenAI client
const client = new OpenAI({
baseURL: "https://api.vlm.run/v1/openai",
apiKey: ""
});
// Define the messages
const messages = [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: [
{ type: "text", text: "What's in this image?" },
{ type: "image_url", image_url: { url: "https://example.com/image.jpg", detail: "auto" } }
]}
];
// Perform chat completion (with streaming)
const stream = await client.chat.completions.create({
model: "vlmrun-orion-1:auto",
messages: messages,
temperature: 0,
stream: true
});
for await (const chunk of stream) {
if (chunk.choices[0]?.delta?.content) {
process.stdout.write(chunk.choices[0].delta.content);
}
}
```
### Usage 4: Mixed-Modality Inputs
```python Python expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
import openai
from vlmrun.client import VLMRun
# Initialize the VLM Run client for file uploads
vlmrun_client = VLMRun(api_key="")
# Upload an image and document to the VLM Run Agents API
file1 = vlmrun_client.files.upload(file=Path("image.jpg"), purpose="assistants")
file2 = vlmrun_client.files.upload(file=Path("document.pdf"), purpose="assistants")
# Initialize the OpenAI client
client = openai.OpenAI(
base_url="https://api.vlm.run/v1/openai", api_key=""
)
# Perform chat completion (with mixed-modality inputs)
chat_completion = client.chat.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{"role": "user", "content": [
{"type": "text", "text": "What's in this image and document?"},
{"type": "input_file", "file_id": file1.id},
{"type": "input_file", "file_id": file2.id},
]}
],
)
```
```typescript Node.js expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { OpenAI } from "openai";
import fs from "fs";
// Initialize the OpenAI client
const client = new OpenAI({
baseURL: "https://api.vlm.run/v1/openai",
apiKey: ""
});
// Upload an image and document to the VLM Run Agents API
const file1 = await client.files.create({
file: fs.createReadStream("image.jpg"),
purpose: "assistants"
});
const file2 = await client.files.create({
file: fs.createReadStream("document.pdf"),
purpose: "assistants"
});
// Perform chat completion (with mixed-modality inputs)
const chatCompletion = await client.chat.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{ role: "user", content: [
{ type: "text", text: "What's in this image and document?" },
{ type: "input_file", file_id: file1.id },
{ type: "input_file", file_id: file2.id }
]}
]
});
```
## Extra Body
The `extra_body` field allows you to specify additional request metadata that is used by the VLM Run Agents API (outside of the OpenAI Python SDK), as indicated by the `vlmrun` field. This metadata is used to specify other request metadata such as `allow_training`, `environment` etc.
For example, the following code specifies the request `metadata`:
```python Python expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import openai
client = openai.OpenAI(
base_url="https://api.vlm.run/v1/openai", api_key=""
)
chat_completion = client.chat.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{ role: "user", content: "What's the capital of France?" }
],
temperature=0,
extra_body={
"vlmrun": {
"metadata": {
"environment": "dev",
"allow_retention": False,
},
}
}
)
```
```typescript Node.js expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { OpenAI } from "openai";
const client = new OpenAI({
baseURL: "https://api.vlm.run/v1/openai",
apiKey: ""
});
const chatCompletion = await client.chat.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{ role: "user", content: "What's the capital of France?" }
],
temperature: 0,
extra_body: {
vlmrun: {
metadata: {
environment: "dev",
allow_retention: false
}
}
}
});
```
## Request Metadata
For more details on the request metadata, please refer to the [Request Metadata](/api-reference/v1/post-agent-execute#body-metadata) section of the API reference.
Currently, the VLM Run Agents API supports submitting request metadata along with the chat completions request via the `extra_body` keyword argument. For example, the VLM Run Agents API accepts the following request metadata:
The VLM Agents API supports the following request `vlmrun.metadata` fields.
* `environment` (`dev`, `staging`, `prod`): This property specifies the environment in which the request is being made. This can be useful for tracking requests across different environments. By default, this property is set to `prod`.
* `session_id`: This property is a string UUID for the session, which can be used to track requests across different sessions. This property is required and must be a valid UUID (36 characters long).
* `allow_training`: This property flags the request as a potential candidate for our training dataset. If set to `true`, the request may be used for training our base models. If set to `false`, the request will be used for inference only. By default, this property is set to `true`.
* `allow_retention`: This property flags the request as a potential candidate for our retention dataset. If set to `true`, the request may be used for retention of the data. If set to `false`, the request will be used for inference only. By default, this property is set to `true`.
* `allow_logging`: This property flags the request as a potential candidate for our logging dataset. If set to `true`, the request may be used for logging of the data. If set to `false`, the request will be used for inference only. By default, this property is set to `true`.
* `extra`: This property is a dictionary of extra metadata that can be used to track the request.
```python Python expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import openai
client = openai.OpenAI(
base_url="https://api.vlm.run/v1/openai", api_key=""
)
chat_completion = client.chat.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{ role: "user", content: "What's the capital of France?" }
],
temperature=0,
extra_body={
"vlmrun": {
"domain": "...",
"metadata": {
"environment": "dev",
"session_id": "...",
"allow_training": False,
}
}
}
)
```
```typescript Node.js expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { OpenAI } from "openai";
const client = new OpenAI({
baseURL: "https://api.vlm.run/v1/openai",
apiKey: ""
});
const chatCompletion = await client.chat.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{ role: "user", content: "What's the capital of France?" }
],
temperature: 0,
extra_body: {
vlmrun: {
domain: "...",
metadata: {
environment: "dev",
session_id: "...",
allow_training: false
}
}
}
});
```
## Token Usage
The OpenAI Python SDK provides usage statistics for your account on every API call. This can be useful for monitoring your usage and costs when using the API. We refer the user to the [VLM Run Pricing](https://vlm.run/#pricing) page for more information on pricing and usage.
## Compatibility Differences
Unlike the OpenAI API, the VLM Run Agents API adds support to the following fields:
* messages can now contain `input_file` objects `{"type": "input_file", "file_id": ""}` where `file_id` is the id of the file uploaded to the VLM Run Agents API. These are especially useful for processing large files such as videos, images, etc.
* `max_tokens`: The `max_tokens` field in `chat.completions.create` is currently not respected by our server. This means that in case the token outputs exceed the limit, the server will still return the full output.
* `logprobs`, `logit_bias`, `top_logprobs`, `presence_penalty`, `frequency_penalty`, `n`, `stream`, `stop`: These fields are not currently supported by the VLM Run Agents API. We will be adding support for these features in the near future.
# Pydantic AI Compatibility
Source: https://docs.vlm.run/agents/integrations/integrations-pydantic-ai
Use Orion-2 with Pydantic AI for document, image, audio, and video work with hybrid client tools.
Point a [Pydantic AI](https://ai.pydantic.dev/) `Agent` at the VLM Run OpenAI-compatible chat endpoint. Pydantic AI runs your local tools; Orion-2 runs document, image, audio, and video work in its [code-execution sandbox](/agents/code-execution).
## Quick start
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
pip install "pydantic-ai>=1.100"
export VLMRUN_API_KEY=""
```
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import os
import uuid
from pydantic_ai import Agent, AudioUrl, DocumentUrl, ImageUrl, RunContext, VideoUrl
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.profiles.openai import OpenAIModelProfile
from pydantic_ai.providers.openai import OpenAIProvider
model = OpenAIChatModel(
"vlmrun-orion-2:auto",
provider=OpenAIProvider(
base_url="https://api.vlm.run/v1/openai",
api_key=os.environ["VLMRUN_API_KEY"],
),
profile=OpenAIModelProfile(
openai_supports_strict_tool_definition=False,
openai_unsupported_model_settings=("max_completion_tokens",),
openai_chat_supports_document_input=True,
openai_chat_supports_file_urls=True,
openai_chat_audio_input_encoding="uri",
),
)
# Stock OpenAIChatModel rejects VideoUrl; map it to VLM's video_url part.
async def _map_video_url(item: VideoUrl):
return {"type": "video_url", "video_url": {"url": item.url}}
model._map_video_url_item = _map_video_url
agent = Agent(
model,
system_prompt=(
"Use the server execute_code tool for attached documents, images, "
"audio, or video. Call lookup_purchase_order only when asked to "
"reconcile an invoice. Never invent field values."
),
)
@agent.tool
def lookup_purchase_order(ctx: RunContext[None], invoice_number: str) -> dict:
"""Return the purchase order your ERP has on file for an invoice number."""
return {"invoice_number": invoice_number, "po_number": "PO-4417", "total": 4560.00}
def run_kwargs() -> dict:
return {
"model_settings": {
"extra_body": {
"session_id": str(uuid.uuid4()),
"tool_execution": "hybrid",
}
}
}
```
| Setting | Why |
| ---------------------------------------- | ------------------------------------------------------------------------------ |
| `tool_execution: "hybrid"` | Orion-2 runs `execute_code`; client tools pause back to Pydantic AI |
| `session_id` | Fresh UUID per turn for workspace / [artifacts](/agents/artifacts) |
| `openai_chat_supports_file_urls=True` | Lets `DocumentUrl` pass as a URL (normalized to `file_url` server-side) |
| `openai_chat_audio_input_encoding="uri"` | Lets `AudioUrl` encode cleanly for chat completions |
| `_map_video_url_item` | Only media patch needed — stock Pydantic AI has no Chat Completions video part |
`ImageUrl` and `DocumentUrl` need no mapper overrides. `AudioUrl` works with the profile above. Only `VideoUrl` needs the one-line patch.
Use `vlmrun-orion-2:auto` for the default tier. Other variants are listed in [Code Execution](/agents/code-execution#model-variants). Get an API key from [API Keys](https://app.vlm.run/dashboard/settings/api-keys).
## Examples
Public sample files — same agent and `run_kwargs()` for each.
### Document
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
INVOICE = (
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/"
"document.invoice/wordpress-pdf-invoice-plugin-sample.pdf"
)
result = agent.run_sync(
["Extract vendor, invoice number, date, and total.", DocumentUrl(INVOICE)],
**run_kwargs(),
)
print(result.output)
```
### Image
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
DONUTS = (
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/"
"image.object-detection/donuts.png"
)
result = agent.run_sync(
["Count the donuts and briefly describe the image.", ImageUrl(DONUTS)],
**run_kwargs(),
)
print(result.output)
```
### Audio
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
AUDIO = (
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/"
"audio.transcription-summary/two_minute_rules.mp3"
)
result = agent.run_sync(
["Summarize this audio in 2 sentences.", AudioUrl(AUDIO)],
**run_kwargs(),
)
print(result.output)
```
### Video
Prefer streaming for video (often 1–2 minutes; some keys require `stream=True`):
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import asyncio
from pydantic_ai.usage import UsageLimits
VIDEO = (
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/"
"video.transcription/bakery.mp4"
)
async def run_streamed(prompt) -> str:
async with agent.iter(
prompt,
usage_limits=UsageLimits(request_limit=20),
**run_kwargs(),
) as run:
async for node in run:
if Agent.is_model_request_node(node):
async with node.stream(run.ctx) as stream:
async for _ in stream:
pass
return run.result.output
print(asyncio.run(run_streamed(["Describe this video in 2 sentences.", VideoUrl(VIDEO)])))
```
### Hybrid: extract + local ERP tool
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
result = agent.run_sync(
[
"Extract the invoice number and total with execute_code. "
"Then call lookup_purchase_order once and say if the totals match.",
DocumentUrl(INVOICE),
],
**run_kwargs(),
)
print(result.output)
```
## What runs where
| Capability | Who runs it |
| -------------------------------- | ------------------------------------- |
| Document / image / audio / video | Orion-2 (`execute_code`) |
| ERP / DB / internal APIs | Your Pydantic AI `@agent.tool` |
| Streaming | `agent.iter(...)` (see video example) |
## Related
Orion-2 sandbox, libraries, and model variants
The same hybrid contract from a TypeScript agent
Document, image, audio, and video content parts
Retrieve annotated images, clips, and generated files
# Introduction
Source: https://docs.vlm.run/agents/introduction
Introducing VLM Run Orion – the first visual agent that sees, reasons, and acts.
Today's frontier Vision-Language Models like GPT, Claude, and Gemini can describe images and answer questions, but they operate as monolithic inference engines. They generate descriptive outputs but cannot *act* on visual data with the precision, determinism, or compositional control required for production-grade workflows.
[Orion](https://vlm.run/orion) introduces a new paradigm for agentic visual reasoning and execution. Unlike monolithic VLMs, Orion orchestrates specialized computer vision tools – OCR, detection, segmentation, keypoint localization, diffusion, and geometric analysis – to execute complex multi-step visual workflows from natural language instructions. This marks the transition from passive visual understanding to **autonomous, tool-augmented visual intelligence** that bridges neural perception with symbolic execution.
Looking to chat with VLM Run's Orion agents? Visit [chat.vlm.run](https://chat.vlm.run).
Read more about [VLM Run Orion](https://vlm.run/orion) in our [technical whitepaper](https://vlm.run/orion/whitepaper).
## Agents Supported
Orion is available in two families, **Orion-1** (tool-calling agents) and **Orion-2** (code-execution agents), each with `fast`, `auto`, and `pro` tiers.
### Orion-1: Tool-Calling Agents
Orion-1 agents orchestrate specialized CV tools (OCR, detection, segmentation, etc.) via structured tool calls. Each tool invocation is a discrete API call managed by the agent.
Our fast visual agent for simple multi-modal workflows. Optimized for speed and quick responses.
Automatically selects the best model, tool and thinking budget based on your task complexity. Balanced performance and capability.
Our most capable visual agent for complex, multi-step workflows. Handles long tool-trajectories and advanced reasoning with a high thinking budget.
### Orion-2: Code-Execution Agents
Orion-2 agents write and execute Python code in a secure sandbox, composing CV operations programmatically. This enables multi-step pipelines, iterative refinement, and complex data transformations within a single turn. See the [Code Execution](/agents/code-execution) guide for details.
Fast code-execution agent for quick pipelines. Uses lightweight models for rapid iteration.
Automatically routes to the best backend model (Qwen, Gemma, or frontier models) based on task complexity. Default tier for Orion-2.
Most capable code-execution agent for complex multi-step pipelines with extended reasoning budgets.
Orion-2 also supports pinned backend variants for advanced use cases: `vlmrun-orion-2:qwen3.6-35b-a3b`, `vlmrun-orion-2:gemma4-26b-a4b`, `vlmrun-orion-2:kimi-2.6`, `vlmrun-orion-2:gpt-5.5`, and `vlmrun-orion-2:claude-opus-4.8`.
## What makes VLM Run Agents unique?
Here are some key features of VLM Run Agents that set it apart from other AI agent platforms:
Execute complex multi-step visual workflows with adaptive context management across extended conversations.
Comprehensive suite of specialized tools across document, image, video, and multimodal processing—composable into multi-stage pipelines.
Use our OpenAI Chat Completions endpoint to interact with VLM Run's Orion agents with just 2 lines of code change.
Our agents are SOC2-Type 2 and HIPAA-compliant, production-ready with automatic validation, with support for full traceability and auditability.
## How is VLM Run's Orion different from frontier models?
Unlike monolithic Vision-Language Models (VLMs like GPT-5, Claude 4.5, and Gemini 2.5), VLM Run's Orion family of visual agents delivers comprehensive capabilities across all modalities and tasks. The table below highlights key differences that matter for building production-grade visual workflows:
Task
VLM Run Orion
OpenAI GPT-5
Google Gemini 2.5
Anthropic Claude Sonnet 4.5
Alibaba Qwen3-VL 235B-A22B
Image / Video
Understanding
✓
⚠
✓
⚠
✓
Reasoning
✓
✗
✗
✗
✓
Structured Outputs
✓
✓
✓
✓
✓
Multi-modal Tool-Calling
✓
✗
✗
✗
⚠
Specialized Skills
✓
✗
⚠
⚠
✗
Document
Understanding
✓
✓
✓
✓
✓
Reasoning
✓
✓
✓
✓
✗
Structured Outputs
✓
✓
✓
✓
✓
Multi-modal Tool-Calling
✓
⚠
⚠
⚠
✗
Specialized Skills
✓
✓
⚠
✓
✗
In the table above, we refer to **Specialized Skills** as tasks such as object localization, segmentation, image-generation / editing, or geometric tools typically found in specialized computer vision applications.
**Key advantages for developers:**
* **Mixed-modality Reasoning**: Only VLM Run's Orion agents provide full reasoning across images, documents, and video - critical for building multi-step visual workflows.
* **Multi-modal Tool-Calling**: With unique tool-calling support for images, videos and documents, VLM Run's Orion agents enable multi-modal reasoning and execution that other models cannot perform.
* **Production-Ready Structured Outputs**: Consistent structured output support across all modalities with automatic validation and retry logic
## Let's get started!
Below you'll find the API reference and code samples so you can start building intelligent agents for your use case.
Sign up for an API key on our [platform](https://app.vlm.run/?utm_source=docs\&utm_medium=link\&utm_campaign=chat), then check out some of our [cookbooks](https://github.com/autonomi-ai/vlm-cookbook) to learn how to use VLM Run Agents to build sophisticated visual AI workflows.
Chat with our visual agent direcly in your browser.
See the complete catalog of visual AI capabilities and tools.
Enough talk, show me the code.
Various cookbooks showcasing VLM Run Agents in action.
# Structured Responses
Source: https://docs.vlm.run/agents/structured-responses
Agents that reliably return JSON via chat completions – with schema validation.
Foundation vision models support chat over visual inputs, but automation needs reliable, machine-validated output. Agent chat completions let you define the expected structure up front and get consistently formatted JSON back – either loosely with `json_object` or strictly via `json_schema`.
Using the OpenAI SDK? See [OpenAI Compatibility](/agents/integrations/integrations-openai-compatibility).
## Extract Structured JSON with VLM Run's Orion Agents
Here's an example of using the agent chat completions endpoint to extract typed JSON directly from user prompts and files.
```python Python (json_object) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Ask the agent for structured output using a loose JSON object
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract invoice number, dates, totals, and vendor in JSON."},
{"type": "image_url", "image_url": {"url": "https://example.com/invoice.jpg"}}
]
}
],
response_format={"type": "json_object"},
)
print(response.choices[0].message.content)
>>> '{"invoice_number":"INV-2024-001","invoice_date":"2024-09-15","total_amount":1250.00,"vendor_name":"Acme Corporation"}'
```
```python Python (json_schema) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
## Structured Response Formats
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Ask the agent for structured output using a strict JSON Schema
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract invoice number, dates, totals, and vendor in JSON."},
{"type": "image_url", "image_url": {"url": "https://example.com/invoice.jpg"}}
]
}
],
response_format={"type": "json_schema", "json_schema": Invoice.model_json_schema()},
)
# Validate the response
invoice = Invoice.model_validate_json(response.choices[0].message.content)
print(invoice)
```
```typescript Node.js (json_object) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Ask the agent for structured output using a loose JSON object
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Extract invoice number, dates, totals, and vendor in JSON." },
{ type: "image_url", image_url: { url: "https://example.com/invoice.jpg" } }
]
}
],
response_format: { type: "json_object" }
});
```
```typescript Node.js (json_schema) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the schema with Zod
const InvoiceSchema = z.object({
invoice_number: z.string().describe("The number of the invoice"),
date: z.string().describe("The date of the invoice"),
total_amount: z.number().describe("The total amount of the invoice"),
vendor_name: z.string().describe("The name of the vendor")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Ask the agent for structured output using a strict JSON Schema
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Extract invoice number, dates, totals, and vendor in JSON." },
{ type: "image_url", image_url: { url: "https://example.com/invoice.jpg" } }
]
}
],
response_format: { type: "json_schema", schema: zodToJsonSchema(InvoiceSchema) }
});
// Validate the response
const invoice = InvoiceSchema.parse(JSON.parse(response.choices[0].message.content));
console.log(invoice);
```
## JSON Response
```json JSON theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"invoice_number": "INV-2024-001",
"invoice_date": "2024-09-15T00:00:00",
"total_amount": 1250.00,
"vendor_name": "Acme Corporation"
}
```
## Response Format Types
| Type | Description |
| ------------- | ----------------------------------------- |
| `json_object` | Valid JSON object without specific schema |
| `json_schema` | Strict JSON conforming to provided schema |
# Overview
Source: https://docs.vlm.run/api-reference/index
The **VLM Run API** is a unified platform for production-ready multimodal AI. Use it to extract structured data from documents, images, videos, and audio — or run complex multi-step workflows with visual agents.
* **Base URL**: `https://api.vlm.run/v1`
* **Authentication**: `Authorization: Bearer `
* **Models Supported**:
* Requests: `vlm-1`
* Agent Executions / Chat Completions: `vlmrun-orion-1:auto`, `vlmrun-orion-1:fast`, `vlmrun-orion-1:pro`
See [Ways to Use VLM Run](/ways-to-use-vlm-run) for a side-by-side comparison of Requests, Executions, Chat Completions, and the Chat UI.
Access your [API keys](https://app.vlm.run/dashboard/settings/api_keys) in our dashboard.
## Structured Extraction
Use the [Generate](/api-reference/v1/post-image-generate) endpoints to extract structured JSON from images, documents, audio, and video.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse
# Initialize the client
client = VLMRun(api_key="")
# Document -> JSON
response: PredictionResponse = client.document.generate(
file=Path("path/to/document.pdf"),
model="vlm-1",
domain="document.invoice",
)
```
```javascript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VLMRun } from "vlmrun";
// Initialize the client
const client = new VLMRun({
apiKey: "",
});
// Upload a document
const file = await client.files.upload({
filePath: "path/to/invoice.pdf",
});
// Process a document using file ID
const response = await client.document.generate({
fileId: file.id,
model: "vlm-1",
domain: "document.markdown",
});
console.log(response);
```
## Chat Completions & Agent Executions
Use the [Chat Completions](/api-reference/v1/post-chat-completions) endpoint for interactive multi-modal conversations, or the [Agent Executions](/api-reference/v1/get-agent-executions) endpoint for batch execution workflows.
```python Python / VLMRun SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLM Run client
client = VLMRun(api_key="")
# Create a chat completion
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What do you see in this image?" },
{"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
]
}
],
max_tokens=1000
)
```
```typescript TypeScript / VLMRun SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
// Initialize the VLM Run client
const client = new VlmRun({
apiKey: ""
});
// Create a chat completion
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "What do you see in this image?" },
{ type: "image_url", image_url: { url: "https://example.com/image.jpg" } }
]
}
],
max_tokens: 1000
});
console.log(response.choices[0].message.content);
```
```curl cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/openai/chat/completions \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "vlmrun-orion-1:auto",
"messages": [
{
"role": "user",
"content": [
{ "type": "text", "text": "What do you see in this image?" },
{ "type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
]
}
],
"max_tokens": 1000
}'
```
# Delete File
Source: https://docs.vlm.run/api-reference/v1/files/delete-file
https://api.vlm.run/openapi.json DELETE /v1/files/{file_id}
Delete a file by ID. Only available for Pro and Enterprise users.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
!pip install vlmrun
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
# Delete a file by ID
response = client.files.delete(file_id="")
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
// Delete a file by ID
await client.files.delete("");
```
# Get File by ID
Source: https://docs.vlm.run/api-reference/v1/files/get-files-by-id
https://api.vlm.run/openapi.json GET /v1/files/{file_id}
Get a file by ID.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
!pip install vlmrun
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
# Get the file by ID
response = client.files.get(file_id="file_123")
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
// Get the file by ID
const response = await client.files.get("file_123");
console.log(response);
```
# List Files
Source: https://docs.vlm.run/api-reference/v1/files/get-files-list
https://api.vlm.run/openapi.json GET /v1/files
Get all files uploaded by the user with pagination.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
!pip install vlmrun
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
# Get the list of files
response = client.files.list(limit=10)
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
// Get the list of files
const response = await client.files.list({ limit: 10 });
console.log(response);
```
# Upload File
Source: https://docs.vlm.run/api-reference/v1/files/post-file-upload
https://api.vlm.run/openapi.json POST /v1/files
Upload a file.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
!pip install vlmrun
from vlmrun.client import VLMRun
from pathlib import Path
client = VLMRun(api_key="")
# Upload the file to the object store
response = client.files.upload(file=Path("test.pdf"))
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
// Upload the file to the object store
const response = await client.files.upload({ filePath: "test.pdf" });
console.log(response);
```
# Get Artifact
Source: https://docs.vlm.run/api-reference/v1/get-artifact-by-id
https://api.vlm.run/openapi.json GET /v1/artifacts
Retrieve an artifact by session ID or execution ID.
Retrieve the raw content of an artifact using either a `session_id` (for chat completions) or an `execution_id` (for agent executions), along with the `object_id` from the response. Artifacts are binary objects generated during agent interactions, such as images, videos, audio files, and documents.
For a comprehensive guide on working with artifacts, including usage patterns and examples, see the [Artifacts Guide](/agents/artifacts).
## Query Parameters
| Parameter | Type | Required | Description |
| -------------- | ------ | -------- | ------------------------------------------------------------------------------- |
| `object_id` | string | Yes | Object ID for the artifact (format: `_<6-hex-chars>`, e.g., `img_a1b2c3`) |
| `session_id` | string | No | Session ID from chat completions (mutually exclusive with `execution_id`) |
| `execution_id` | string | No | Execution ID from agent executions (mutually exclusive with `session_id`) |
Either `session_id` or `execution_id` must be provided, but not both. Use `session_id` for artifacts from chat completions and `execution_id` for artifacts from agent executions.
## Object Reference Format
Object references follow the format: `_<6-digit-hex-string>` (e.g., `img_a1b2c3`).
| Artifact Type | Prefix | Reference Type | Python Return Type |
| -------------- | -------- | -------------- | ------------------------- |
| Image | `img_` | `ImageRef` | `PIL.Image.Image` |
| Video | `vid_` | `VideoRef` | `Path` (mp4) |
| Audio | `aud_` | `AudioRef` | `Path` (mp3) |
| Document | `doc_` | `DocumentRef` | `Path` (pdf) |
| Reconstruction | `recon_` | `ReconRef` | `Path` (spz) |
| URL | `url_` | `UrlRef` | `Path` (any of the above) |
| Array | `arr_` | `ArrayRef` | `np.ndarray` |
The Python SDK provides convenience methods that automatically convert artifacts to the appropriate Python types.
## Response
Returns the raw binary content of the artifact with the appropriate content type based on the artifact type.
## Get Artifact by Session ID
Use `session_id` to retrieve artifacts from chat completion responses.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from PIL import Image
from pydantic import BaseModel, Field
from vlmrun.client import VLMRun
from vlmrun.types import ImageRef
client = VLMRun(api_key="")
# Define a response model with an ImageRef field
class BlurredImageResponse(BaseModel):
image: ImageRef = Field(..., description="The blurred image")
# Make a chat completion request that generates an image artifact
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Blur all the faces in this image"},
{"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
]
}
],
response_format={"type": "json_schema", "schema": BlurredImageResponse.model_json_schema()},
)
# Parse the response to get the artifact reference
result = BlurredImageResponse.model_validate_json(response.choices[0].message.content)
# Retrieve the artifact using session_id and object_id
image: Image.Image = client.artifacts.get(
session_id=response.session_id,
object_id=result.image.id,
)
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
baseURL: "https://api.vlm.run/v1",
apiKey: ""
});
// Retrieve the artifact using session_id and object_id
const artifact = await client.artifacts.get({
sessionId: "",
objectId: "img_a1b2c3"
});
console.log(artifact);
```
## Get Artifact by Execution ID
Use `execution_id` to retrieve artifacts from agent execution responses.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from PIL import Image
from pydantic import BaseModel, Field
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionConfig, AgentExecutionResponse, ImageUrl
from vlmrun.types import ImageRef, MessageContent
client = VLMRun(api_key="")
# Define typed inputs and response model
class ExecutionInputs(BaseModel):
image: MessageContent = Field(..., description="The input image")
class ImageResponse(BaseModel):
image: ImageRef = Field(..., description="The processed image")
# Execute an agent
execution: AgentExecutionResponse = client.agent.execute(
name="image/blur-image",
inputs=ExecutionInputs(
image=MessageContent(type="image_url", image_url=ImageUrl(url="https://example.com/photo.jpg"))
),
config=AgentExecutionConfig(
prompt="Blur the entire image",
response_model=ImageResponse
)
)
# Wait for completion
execution = client.executions.wait(execution.id, timeout=180)
# Parse the response and retrieve the artifact using execution_id
result = ImageResponse.model_validate(execution.response)
image: Image.Image = client.artifacts.get(
execution_id=execution.id,
object_id=result.image.id,
)
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
baseURL: "https://api.vlm.run/v1",
apiKey: ""
});
// Retrieve the artifact using execution_id and object_id
const artifact = await client.artifacts.get({
executionId: "",
objectId: "img_a1b2c3"
});
console.log(artifact);
```
# List models
Source: https://docs.vlm.run/api-reference/v1/get-models
https://api.vlm.run/openapi.json GET /v1/models
Get the list of supported models.
Get a list of schemas currently supported by the API.
These are available for use with the `generate` endpoint.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
!pip install vlmrun
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.models.list()
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
const response = await client.models.list();
console.log(response);
```
# Health
Source: https://docs.vlm.run/api-reference/v1/health
https://api.vlm.run/openapi.json GET /v1/health
Health check endpoint.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
!pip install vlmrun
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.healthcheck()
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
const response = await client.models.list();
// Health check can be verified by successful API call
console.log("API is healthy");
```
```curl cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl --request GET \
--url https://api.vlm.run/v1/health
```
# List domains
Source: https://docs.vlm.run/api-reference/v1/hub/get-domains
https://api.vlm.run/openapi.json GET /v1/hub/domains
Get the list of supported domains.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
!pip install vlmrun
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.hub.list_domains()
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
const response = await client.hub.listDomains();
console.log(response);
```
## Description
This endpoint returns the list of supported domains / schemas in the [VLM Run Hub](https://github.com/vlm-run/vlmrun-hub).
# Audio → JSON
Source: https://docs.vlm.run/api-reference/v1/post-audio-generate
https://api.vlm.run/openapi.json POST /v1/audio/generate
Generate structured prediction for the given audio file.
For all supported `audio` domains, see the [Hub Catalog](/hub#audio-domains).
```python Python (with domain) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.audio.generate(
file=Path(".mp3"),
domain="audio.transcription",
batch=True
)
```
```python Python (with skill) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig, AgentSkill
client = VLMRun(api_key="")
response = client.audio.generate(
file=Path(".mp3"),
domain="audio.transcription",
batch=True,
config=GenerationConfig(
skills=[AgentSkill(skill_name="")]
)
)
```
```typescript Node.js SDK (with domain) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({apiKey: ""});
const fileResponse = await client.files.upload(
filePath: ".mp3"
);
const response = await client.audio.generate({
fileId: fileResponse.id,
domain: "audio.transcription",
});
```
```typescript Node.js SDK (with skill) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({apiKey: ""});
const fileResponse = await client.files.upload(
filePath: ".mp3"
);
const response = await client.audio.generate({
fileId: fileResponse.id,
batch: true,
config: {
skills: [{ skillName: "" }],
},
});
```
### Example Output
```json Example Audio Transcription [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"metadata": {
"duration": 146.94
},
"segments": [
{
"start_time": 0,
"end_time": 24.88,
"content": " After reading tons of productivity books, I came across so many rules, like the two-year rule, the five-minute rule, the five-second rule. No, not that five second rule. The problem is that these rules were meant for companies or entrepreneurs, but I was able to adapt them to my studies during med school and drastically cut down to my procrastination. So I'm going to share with you two different two minute rules for the next two minutes. The first two minute rule comes from"
},
{
"start_time": 24.88,
"end_time": 45.5,
"content": " getting things done by David Allen. He says if it takes two minutes to do, get it done right now. For example, if I need to take out the trash today, it takes two minutes to do. So if I'm thinking about it now, might as well just do it now. Instead of writing it down on a to-do list or probably forgetting about it or having to come back to it later, which takes more than two minutes. That's how I see it."
},
{
"start_time": 45.5,
"end_time": 67.86,
"content": " So here's a list of things that might take two minutes throughout the day, like organizing your desk or watering your plants or clipping those nasty nails. I just do it when I notice it, but these little things start to add up, so this rule biases my brain towards taking action and away from procrastination. The second two-minute rule comes from atomic habits by James Clear. He says, when you're trying to do something you don't really want to do, simplify the"
},
{
"start_time": 67.86,
"end_time": 91.27,
"content": " task down to two minutes or less. So doing your entire reading assignment becomes just reading one paragraph or memorizing the entire periodic table becomes memorizing just 10 flashcards. Now, some of you might think, yeah, this is just a Jedi mind trick. Like, why would I fall for it? How is this at all sustainable? And to that, he says, when you're starting out, limit yourself to only two minutes."
},
{
"start_time": 91.27,
"end_time": 117.33,
"content": " So back in med school, I wanted to build a habit of studying for one hour every day before dinner. So I tried this trick, but I limited myself to just two minutes. I'd sit down, open my laptop, study for two minutes, and then close my laptop and went to do something else. It seems unproductive at first, right? It seems stupid. But staying consistent with this two-minute routine day after day meant that I was becoming the type of person who studies daily."
},
{
"start_time": 117.33,
"end_time": 137.99,
"content": " I was mastering the habit of just showing up because a habit needs to be established before it can be expanded upon. If I can't become a person who studies for just two minutes a day, I'd never be able to become the person that studies for an hour a day. You've got to start somewhere, but starting small is easier. There's a lot of other useful tips from books."
},
{
"start_time": 138.15,
"end_time": 146.94,
"content": " I cover more here in this video on three books and three minutes. Check it out. And if you guys like these types of videos, let me know in the comments below. I'll see you there. Bye."
}
]
}
```
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"usage": {
"elements_processed": 123,
"element_type": "image",
"credits_used": 123
},
"id": "",
"created_at": "2023-11-07T05:31:56Z",
"completed_at": "2023-11-07T05:31:56Z",
"response": "",
"status": "enqueued"
}
```
# Doc → JSON
Source: https://docs.vlm.run/api-reference/v1/post-document-generate
https://api.vlm.run/openapi.json POST /v1/document/generate
Generate structured prediction for the given document.
For all supported `document` domains, see the [Hub Catalog](/hub).
```python Python (with domain) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.document.generate(
file=Path(".pdf"),
domain=""
)
```
```python Python (with skill) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig, AgentSkill
client = VLMRun(api_key="")
response = client.document.generate(
file=Path(".pdf"),
config=GenerationConfig(
skills=[AgentSkill(skill_name="", version="latest")]
)
)
```
```typescript Node.js SDK (with domain) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({apiKey: ""});
const fileResponse = await client.files.upload(
filePath: ".pdf"
);
const response = await client.document.generate({
fileId: fileResponse.id,
domain: "",
});
```
```typescript Node.js SDK (with skill) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({apiKey: ""});
const fileResponse = await client.files.upload(
filePath: ".pdf"
);
const response = await client.document.generate({
fileId: fileResponse.id,
config: {
skills: [{ skillName: "", version: "latest" }],
},
});
```
# Image → JSON
Source: https://docs.vlm.run/api-reference/v1/post-image-generate
https://api.vlm.run/openapi.json POST /v1/image/generate
Generate structured prediction for the given image.
For all supported `image` domains, see the [Hub Catalog](/hub#image-domains).
```python Python (with domain) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from PIL import Image
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.image.generate(
images=[Image.open(".jpg")],
domain=""
)
```
```python Python (with skill) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from PIL import Image
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig, AgentSkill
client = VLMRun(api_key="")
response = client.image.generate(
images=[Image.open(".jpg")],
domain="",
config=GenerationConfig(
skills=[AgentSkill(skill_name="")]
)
)
```
```typescript Node.js SDK (with domain) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({apiKey: ""});
const fileResponse = await client.files.upload(
filePath: ".jpg"
);
const response = await client.image.generate({
fileId: fileResponse.id,
domain: "",
});
```
```typescript Node.js SDK (with skill) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({apiKey: ""});
const fileResponse = await client.files.upload(
filePath: ".jpg"
);
const response = await client.image.generate({
fileId: fileResponse.id,
config: {
skills: [{ skillName: "" }],
},
});
```
# Get Schema
Source: https://docs.vlm.run/api-reference/v1/post-schema
https://api.vlm.run/openapi.json POST /v1/schema/{domain}
# Submit Feedback
Source: https://docs.vlm.run/api-reference/v1/post-submit-feedback
https://api.vlm.run/openapi.json POST /v1/feedback/submit
Submit feedback for a request, execution, or chat by its ID.
Submit feedback for a prediction to help improve model performance through fine-tuning.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.feedback.submit(
request_id="",
response={"name": "John Doe", "date_of_birth": "1955-01-01", "email": "john@doe.com"},
notes="Excellent prediction quality"
)
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({apiKey: ""});
const response = await client.feedback.submit({
requestId: "",
response: {"name": "John Doe", "date_of_birth": "1955-01-01", "email": "john@doe.com"},
notes: "Excellent prediction quality"
});
console.log(response);
```
# Video → JSON
Source: https://docs.vlm.run/api-reference/v1/post-video-generate
https://api.vlm.run/openapi.json POST /v1/video/generate
Generate structured prediction for the given video file.
For all supported `video` domains, see the [Hub Catalog](/hub#video-domains).
```python Python (with domain) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.video.generate(
file=Path(".mp4"),
domain="video.transcription",
batch=True
)
```
```python Python (with skill) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig, AgentSkill
client = VLMRun(api_key="")
response = client.video.generate(
file=Path(".mp4"),
batch=True,
config=GenerationConfig(
skills=[AgentSkill(skill_name="")]
)
)
```
```typescript Node.js SDK (with domain) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({apiKey: ""});
const fileResponse = await client.files.upload(
filePath: ".mp4"
);
const response = await client.video.generate({
fileId: fileResponse.id,
domain: "video.transcription",
batch: true,
});
```
```typescript Node.js SDK (with skill) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({apiKey: ""});
const fileResponse = await client.files.upload(
filePath: ".mp4"
);
const response = await client.video.generate({
fileId: fileResponse.id,
config: {
skills: [{ skillName: "" }],
},
});
```
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"usage": {
"elements_processed": 123,
"element_type": "image",
"credits_used": 123
},
"id": "",
"created_at": "2023-11-07T05:31:56Z",
"completed_at": "2023-11-07T05:31:56Z",
"response": "",
"status": "enqueued"
}
```
Try our Colab Cookbook example for long-form video transcription.
### Example Output
```json Example Video Transcription [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"metadata": {
"language": null,
"content": null,
"topics": null,
"duration": 488.56
},
"segments": [
{
"start_time": 0,
"end_time": 25.8,
"audio": {
"content": " Like the only way to find these opportunities to learn about them is to find weirdos on the internet that are also into this thing. Yes. And they're figuring it out too. And you can kind of compare notes. Yes. And this is how new industries are created. Literally. By weirdos on the internet. Like literally. Literally. This is Dalton, plus Michael, and today we're going to talk about why AI is going to create more successful founders in the world."
},
"video": {
"content": "Two men are engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks. The man on the right, dressed in a blue shirt, listens attentively and occasionally responds with hand gestures. They appear to be in a professional setting, possibly an office or conference room, with large windows in the background allowing natural light to fill the space."
}
},
{
"start_time": 25.8,
"end_time": 51.71,
"audio": {
"content": " It's interesting, as we've gotten older, we kind of see a new set of tools come into the market and then an explosion in the number of founders who can now create value. And we've seen this before, right? Like, what was the first time you saw this? I certainly noticed when the internet was new, people that knew how to build websites were suddenly able to make lots of money from"
},
"video": {
"content": "The video features two individuals engaged in a conversation at a table. The person on the left, wearing a light gray shirt, is facing the person on the right, who is dressed in a blue jacket over a black shirt. The background is minimalistic, with a plain wall and a window allowing natural light to enter. The text overlay on the left side of the screen reads \"AI Will Create More Successful Founders\" and \"Founder Explosion.\" On the right side, there is a list titled \"Founder Explosion\" with various items such as \"On The Cusp,\" \"Cost Of Business,\" \"Get In Early,\" \"Whatnot,\" \"Endless Opportunity,\" and \"Internet Weirdos.\" The conversation appears to be focused on the impact of artificial intelligence on business and entrepreneurship."
}
},
{
"start_time": 51.71,
"end_time": 71.89,
"audio": {
"content": " the skill. And it was like really basic stuff. High school kids were making tons of money. Yep. I remember people that could just figure out how to sell stuff on eBay, where you would go buy something cheap but then listed on eBay and arbitrage. Yep. Basically, you would see people that kind of understood the new tooling that came out and would like do a hustle and make ungodly amounts of money."
},
"video": {
"content": "The video features two men engaged in a conversation in an office setting. The man on the left, wearing a light gray button-up shirt, is actively speaking and gesturing with his hands, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively with his arms crossed. The background includes a large window with blinds partially drawn, allowing natural light to filter into the room. The conversation appears to be focused on business-related topics, as indicated by the text on the right side of the screen, which lists various themes such as 'On The Cusp,' 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' and 'Internet Weirdos.' The overall atmosphere suggests a professional discussion."
}
},
{
"start_time": 72.01,
"end_time": 92.67,
"audio": {
"content": " Yeah. And it was just because they understood the new tools. And I already wasn't even a hustle. Like it was a good business. Like it was, they saw that tools enabled new businesses. You know, we saw this, you know, tail end of the open source world where like we could build all of Justin TV with free software. Yep."
},
"video": {
"content": "The video features two men engaged in a conversation in an office setting. The man on the left, wearing a light gray shirt, is gesturing animatedly with his hands as he speaks, indicating an active discussion. The man on the right, dressed in a blue jacket over a black shirt, listens attentively with his hands clasped together on the table. The background includes a window with blinds partially open, allowing natural light to filter into the room. On the right side of the screen, there is a vertical list titled \"Founder Explosion\" with various topics such as \"On The Cusp,\" \"Cost Of Business,\" \"Get In Early,\" \"Whatnot,\" \"Endless Opportunity,\" and \"Internet Weirdos.\""
}
},
{
"start_time": 92.67,
"end_time": 112.85,
"audio": {
"content": " And then we were there in the beginning of cloud compute where we didn't have to rack servers anymore. Any kid could sign up for an Amazon account, put a couple bucks down, and get access to a server. And so what's interesting is that we might, I think we feel pretty good about saying this, we might be on"
},
"video": {
"content": "The video features a conversation between two men seated at a table in a modern office setting. The man on the right, wearing a blue shirt and glasses, is speaking animatedly, gesturing with his hands as he discusses various topics related to entrepreneurship and business. The man on the left, dressed in a light-colored shirt, listens attentively, occasionally nodding and responding. The background includes a white wall and a window, suggesting a professional environment. On the right side of the screen, there is a list of topics being discussed, such as 'Founder Explosion,' 'On The Cusp,' 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' and 'Internet Weirdos.'"
}
},
{
"start_time": 112.85,
"end_time": 135.69,
"audio": {
"content": " the cusp of the next one of these. And that means there are maybe a whole bunch of new opportunities for successful businesses to be created. Yeah, starting now. Yeah, I mean, here's another metaphor. When the iPhone came out, who would have thought that Flappy Bird would have been created? And I think I read that that guy made like 20 million in cash."
},
"video": {
"content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is speaking animatedly, gesturing with his hands as he talks. The man on the right, dressed in a blue jacket over a black shirt, listens attentively, occasionally nodding and responding. The setting appears to be an office or meeting room with large windows in the background, allowing natural light to fill the space. The overall atmosphere suggests a professional discussion or interview."
}
},
{
"start_time": 135.87,
"end_time": 159.75,
"audio": {
"content": " Boom. In like two months and then shut it down. And so if you watch, okay, iPhone, Steve Jobs on stage, some guy in Southeast Asia building Flappy Bird. That's like wild. Never would have guessed. And so, again, to be very direct, what we're arguing is that when brand new technologies come out that are powerful, the people that are on the cusp of understanding them and that quickly"
},
"video": {
"content": "Two men are engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing animatedly with his hands as he speaks. The man on the right, dressed in a blue jacket, listens attentively, occasionally nodding and smiling. The background features a large window with a view of a body of water, suggesting an indoor setting with natural light."
}
},
{
"start_time": 159.75,
"end_time": 180.77,
"audio": {
"content": " build businesses or build useful things using those tools have a very unique view of creating businesses and wealth. And again, to be on the nose for AI, it seems like you can do things that would require way more headcount than you would otherwise. Yes. And so, you know, we're not even saying we know the ideas."
},
"video": {
"content": "The video features two men engaged in a conversation in an indoor setting. The man on the left, wearing a light gray button-up shirt, is actively gesturing with his hands as he speaks, indicating an animated discussion. The man on the right, dressed in a dark blue shirt, listens attentively, occasionally nodding and responding. The background includes a window with blinds, suggesting a modern office or studio environment. The video also displays a sidebar with various topics such as 'On The Cusp,' 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'New Is The Time,' which likely relate to the conversation's themes."
}
},
{
"start_time": 180.97,
"end_time": 201.72,
"audio": {
"content": " No. We're just saying if you're watching this and you're interested in being a founder or maybe not working at a company. Yeah. And you just pay attention to every new thing that comes out and try to find these opportunities or, I don't't know arbitrage is the right word, but no, just you know, new opportunities. New opportunities using these cutting edge tools"
},
"video": {
"content": "The video depicts a conversation between two men seated at a table in an office setting. The man on the left, wearing a light gray shirt, is gesturing animatedly with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively with his hands clasped together. The background features large windows with a view of a cityscape, and the room has a modern, minimalist design with white walls and a light-colored floor. The conversation appears to be focused and engaged, with both individuals actively participating in the dialogue."
}
},
{
"start_time": 201.72,
"end_time": 221.8,
"audio": {
"content": " and you're on the bleeding edge, you're not competing with anyone. No. It's green field. I think what's cool is any time one of these technologies shifts happens, the cost of starting a business, some set of businesses, reduces by up to like 10x. Yep. And so suddenly, businesses that either wouldn't have made sense"
},
"video": {
"content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively. The setting appears to be an office or meeting room with large windows in the background, allowing natural light to fill the space. The conversation seems to revolve around business topics, as indicated by the text on the right side of the screen, which includes phrases like 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'Now Is The Time.' The overall atmosphere suggests a professional discussion."
}
},
{
"start_time": 221.8,
"end_time": 244.96,
"audio": {
"content": " or certainly a normal person couldn't just stand up and do, right? Like can you imagine just, oh, it's pre online selling in eBay. All you have to do is rent a storefront and run a store, right? Like that's cheap, right? Like, absolutely not. Or like pre-Ari-NB. Like, all you have to do is just like buy a house and set up your own air bed and breakfast"
},
"video": {
"content": "The video features two men engaged in a conversation at a table in an office setting. The man on the left, wearing a light gray shirt, listens attentively while the man on the right, dressed in a blue jacket over a black shirt, gestures animatedly as he speaks. The background includes large windows with a view of a body of water, suggesting a modern and open environment. The conversation appears to be focused on business topics, as indicated by the text on the right side of the screen, which lists various themes such as 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'Now Is The Time.' The overall atmosphere is professional and collaborative."
}
},
{
"start_time": 244.96,
"end_time": 265.28,
"audio": {
"content": " bed and breakfast thing or even by hotel yeah that's crazy crazy. Whereas like Airbnb can rent a room. And think about it. Your own place. If you saw Airbnb early and you just decided to be a host and be like, oh, I should like do this as a business. You could do it pretty well. You could do it pretty well. Yeah. When Shopify was a brand new thing, like all of these platforms exactly the people that were the first to recognize that these were"
},
"video": {
"content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively. The background is minimalistic, with a plain wall and a window that lets in natural light. On the right side of the screen, there is a list of topics or themes, including \"Cost Of Business,\" \"Get In Early,\" \"Whatnot,\" \"Endless Opportunity,\" \"Internet Weirdos,\" and \"Now Is The Time.\" The overall setting appears to be a casual interview or discussion."
}
},
{
"start_time": 265.84,
"end_time": 286.22,
"audio": {
"content": " gave them leverage yes those entrepreneurial-minded people did really well. Yes. And so I think what's so cool is that what we're saying is like if you're ambitious and you're paying attention, you might not ever need to work at a big company. You might not ever need to have a boss."
},
"video": {
"content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing animatedly with his hands as he speaks. The man on the right, dressed in a blue jacket, listens attentively, occasionally responding with his own gestures. The setting appears to be an office or meeting room with large windows in the background, allowing natural light to fill the space. The conversation seems to revolve around business or startup topics, as indicated by the text on the right side of the screen, which includes phrases like 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'Now Is The Time.' The overall atmosphere suggests a professional and collaborative discussion."
}
},
{
"start_time": 286.22,
"end_time": 307.96,
"audio": {
"content": " Like you can be in control of your own destiny. And these moments don't happen every week. No. Like, we wish we did. It would be the investor. But like, when they do, the people who move. I mean, this is a very specific example. Yeah. But whatnot, the online live shopping thing, I still talk to the founders a lot."
},
"video": {
"content": "Two men are sitting at a table in a modern office setting, engaged in a conversation. The man on the left is wearing a light gray button-up shirt and has short, curly hair. He appears to be listening attentively to the man on the right, who is bald, wearing glasses, and dressed in a blue jacket over a black shirt. The man on the right is gesturing with his hands as he speaks, indicating an animated discussion. The background features large windows with a view of a cityscape, suggesting an urban environment. On the right side of the screen, there is a list of topics or themes related to business and entrepreneurship, such as 'Founder Explosion,' 'On The Cusp,' 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'Now Is The Time.'"
}
},
{
"start_time": 308.22,
"end_time": 328,
"audio": {
"content": " And they have, I think, like, high school-aged kids selling stuff on their... Making real money, right? And making just, again, I don't even want to say the numbers. Yeah. But they figured out the format. They understand how to use whatnot. They built a user base there. Yeah. And they're basically... They're making enough money to set themselves up for their entire life."
},
"video": {
"content": "A man with curly hair and a light gray shirt is speaking animatedly to another man who has a shaved head and glasses. The man with curly hair uses expressive hand gestures as he talks, while the other man listens attentively. The background features a window with blinds, and there is a menu or list of topics on the right side of the screen, including \"Founder Explosion,\" \"On The Cusp,\" \"Cost Of Business,\" \"Get In Early,\" \"Whatnot,\" \"Endless Opportunity,\" \"Internet Weirdos,\" and \"Now Is The Time.\""
}
},
{
"start_time": 328.12,
"end_time": 351.76,
"audio": {
"content": " Yeah. By just seeing this new platform, figuring it out, and then making a bet on it. Yes. I mean, this happened with Twitch. Happens with Twitch. Whole new industry, basically. Yeah. No, I think that what's cool is that we're also talking about every scale, right? We're talking about things that can be venture backed, maybe billion dollar companies one day. But we're also talking about things that can just"
},
"video": {
"content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively with his hands clasped together. The background includes a large window with a view of a body of water, suggesting an indoor setting with natural light. The conversation appears to be casual and focused, with both individuals actively participating in the dialogue."
}
},
{
"start_time": 351.76,
"end_time": 372.66,
"audio": {
"content": " set you up so that you can pay your rent and live a good life. Yep. The opportunities are across the entire spectrum. And I think that's what's really cool about new technology. Like when there's a real technology shift, it affects businesses across the board. We're not just talking about companies that YCP even fun. I think that the last point I'd want to make"
},
"video": {
"content": "A man in a blue shirt is seated at a table, engaged in a conversation with another person whose back is facing the camera. The man in the blue shirt is gesturing with his hands as he speaks, indicating an animated discussion. The setting appears to be an office or a professional environment, with a window and some furniture visible in the background. The overall atmosphere suggests a serious and focused conversation."
}
},
{
"start_time": 372.66,
"end_time": 398.57,
"audio": {
"content": " on this front is that you don't get this opportunity if you're just thinking, you gotta actually do. Well, and they won't teach you this in schools. Schools teach you stuff for 10 or 20 years ago. So the other thing that I've noticed in these trends is that when you are part of the history being made and you're this early on the cutting edge of a new tech coming out, you can't expect your university or your teachers or your peers people in your community"
},
"video": {
"content": "Two men are engaged in a conversation at a table in an office setting. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively. The background features large windows with a view of a cityscape, and the room has a modern, minimalist design with white walls and a light-colored floor. The conversation appears to be focused and serious, with both individuals maintaining eye contact and using expressive hand movements."
}
},
{
"start_time": 398.57,
"end_time": 419.33,
"audio": {
"content": " or your peers to teach you about it. It's only basically weirdos on the internet. Yes. Like the only way to find these opportunities to learn about them is to find weirdos on the internet that are also into this thing. Yes. And they're figuring it out too. that are also into this thing. And they're figuring it out too. And you can kind of compare notes. Yes. And this is how new industries are created. Literally. By weirdos on the internet. Like literally. Literally. By weirdos on the internet. Like literally. Literally, there's like some subreddit with a bunch of weirdos."
},
"video": {
"content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is animatedly gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket, listens attentively with his hands clasped together. The setting appears to be a modern office or conference room with large windows in the background, allowing natural light to fill the space. The conversation seems to be casual and friendly, with both individuals appearing relaxed and engaged."
}
},
{
"start_time": 419.33,
"end_time": 447.2,
"audio": {
"content": " And like someday from now, you know, 10 years from now, there'll be an entire industry of people that learned about this thing in some subred somewhere there. Yeah, no, I totally agree. So hey, the big takeaway is if you've been wrestling your lawyers, if you thought, oh, this isn't the time to start a new business. Maybe you should reconsider. Yeah. This is a very interesting time. I think the final argument is there's a good case where a smaller percentage of the population will need to get jobs,"
},
"video": {
"content": "The video features two men engaged in a conversation in an office setting. The man on the left, wearing a light gray shirt, is animatedly speaking and using hand gestures to emphasize his points. He appears to be explaining something with enthusiasm. The man on the right, dressed in a blue jacket over a black shirt, listens attentively, occasionally nodding and responding. The background includes a window with blinds, suggesting a modern office environment. The conversation seems to revolve around business or technology topics, as indicated by the text overlays such as 'Cost Of Business' and 'Endless Opportunity.'"
}
},
{
"start_time": 447.26,
"end_time": 469.08,
"audio": {
"content": " and more people will be able to use tools like this to be self-employed in some way. I don't think there's any, I think all the structural changes imply that more folks will just use their highly leveraged selves using all these tools to run businesses, then have to go get a job. Yeah. Right? And I think that story isn't told, right? I think the story is always this kind of depressing story of like,"
},
"video": {
"content": "The video features two men engaged in a conversation in an indoor setting. The man on the left, wearing a light gray button-up shirt, is actively gesturing with his hands as he speaks, indicating an animated discussion. The man on the right, dressed in a blue jacket over a black shirt, listens attentively, occasionally nodding and responding. The background includes a window with blinds, suggesting a modern office or conference room environment. The overall atmosphere appears to be professional and focused."
}
},
{
"start_time": 469.08,
"end_time": 488.56,
"audio": {
"content": " oh, maybe you won't need it, you won't be needed anymore, as opposed to here's a set of tools. You could do things that people couldn't think of doing affordably before. Like you could be your own boss. You don't even need to be inside of a company to create value. Yeah. So anyways, hopefully that's inspiring. Good shot. Thanks. good shot thanks"
},
"video": {
"content": "Two men are engaged in a conversation at a table in an office setting. The man on the left, wearing a light gray shirt, listens attentively while the man on the right, dressed in a blue jacket over a black shirt, speaks animatedly. He uses hand gestures to emphasize his points, occasionally clasping his hands together on the table. The background features large windows with a view of a cloudy sky, and the room is well-lit with natural light."
}
}
]
}
```
# Get Prediction by ID
Source: https://docs.vlm.run/api-reference/v1/predictions/get-predictions-by-id
https://api.vlm.run/openapi.json GET /v1/predictions/{id}
Get prediction JSON by request ID.
Get the predictions for a given prediction ID.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
!pip install vlmrun
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse
client = VLMRun(api_key="")
response: PredictionResponse = client.predictions.get("")
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
const response = await client.predictions.get("");
console.log(response);
```
# Get Predictions
Source: https://docs.vlm.run/api-reference/v1/predictions/get-predictions-list
https://api.vlm.run/openapi.json GET /v1/predictions
Get all predictions uploaded by the user with pagination.
Get the list of predictions for the current user.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
!pip install vlmrun
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse
client = VLMRun(api_key="")
response: list[PredictionResponse] = client.predictions.list()
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
const response = await client.predictions.list();
console.log(response);
```
# Custom Schemas
Source: https://docs.vlm.run/capabilities/custom-schemas
Define custom schemas for visual extraction purposes.
In addition to the [pre-defined domains](/capabilities/structured-responses), **`vlm-1`** also supports custom schemas that allows you to define your own schema for a specific domain or use-case. This feature allows you to extract structured data that conforms to your specific needs and requirements, while still leveraging all the vision-based reasoning capabilities of **`vlm-1`** (see [Capabilities section](/capabilities/) for more details).
## What are Custom Schemas?
Custom schemas define the structure and validation rules for the data you want to extract from visual content. For example, you can use [Pydantic](https://docs.pydantic.dev/latest/) or [Zod](https://zod.dev/) models to specify exactly which fields you need, their types, and validation rules - passing the schema to the API will ensure our VLM will extract the data in exactly the format you defined.
## Benefits of Custom Schemas
* **Type Safety**: Enforce proper data types and validation rules
* **Flexibility**: Extract only the data you need in the format you prefer
* **Integration**: Seamlessly connect with your existing data models and systems
* **Customization**: Create domain-specific extraction rules tailored to your use case
Type-based data validation for LLMs have been popularized by tools like [Instructor](https://github.com/jxnl/instructor) and [LangChain](https://python.langchain.com/docs/concepts/structured_outputs/), however we take it a step further by instrumenting new capabilities on top of your schemas such as [visual grounding](/capabilities/visual-grounding), confidence scores, [GQL querying](/capabilities/graphql) and much more.
## 1. Defining a Custom Schema
VLM Run has first-class support for [Pydantic](https://docs.pydantic.dev/latest/) and [Zod](https://zod.dev/), which allows you to define your schema using rich, strongly-typed Pydantic models. Here's an example of a custom schema for classifying and captioning images:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from typing import Literal
from pydantic import BaseModel, Field
class ImagePrediction(BaseModel):
label: Literal["tv", "document", "other"] = Field(..., title="Class label for the image.")
caption: str = Field(..., title="Caption for the image.")
```
## 2. Extracting Structured JSON from Images with a Custom Schema
Once you have defined your custom schema, you can use it with the VLM Run API to extract structured data that conforms to this schema. The extracted data will be validated against the schema you defined.
Here's how to use a custom schema with the Python SDK:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from PIL import Image
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, GenerationConfig
# Initialize the client
client = VLMRun(api_key="your-api-key")
# Process the image with the custom schema
image = Image.open("path/to/image.jpg")
prediction: PredictionResponse = client.image.generate(
images=[image],
domain="image.caption",
config=GenerationConfig(
json_schema=ImagePrediction.model_json_schema()
)
)
response_dict = prediction.response.model_dump()
```
## 3. Response Validation
Since we've defined the schema using Pydantic, you can validate and use the extracted data as a strongly-typed object:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
try:
image_prediction: ImagePrediction = ImagePrediction.model_validate(response_dict)
print(f"Image classified as: {image_prediction.label}")
print(f"Caption: {image_prediction.caption}")
except ValidationError as e:
print(f"Validation error: {e}")
```
## Want to build your own schema?
If you're interested in building your own schema for a specific domain or use-case, take a look at our [schema best practices](/guides/schema/schema-best-practices) guide.
## Try our Image -> JSON API today
Head over to our [Image -> JSON](/api-reference/v1/post-image-generate) to start building your own document processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# GraphQL
Source: https://docs.vlm.run/capabilities/graphql
Query a subset of schema fields to improve efficiency for querying and document ETL.
One of the most powerful features of Vision Language Models is their ability to reason about complex queries, and answer with the relevant data. Unlike traditional OCR-based methods, where every single character is extracted and processed, VLMs admit a much more powerful query mechanism, which is the basis of our GraphQL-based query mechanism.
## Why GraphQL?
First, if you are not familiar with GraphQL, it is a query language for APIs that allows you to specify exactly which fields you want to extract from a given schema. Instead of always receiving the full JSON response, and post-processing to extract the specific fields you need, you can simply request only the specific data points relevant to your application. There is a similar and direct analog to querying LLMs today, where you can specify the exact fields you want to extract from the LLM's response in a structured JSON response.
## Querying VLMs with GraphQL
We simply take this one step further, and enable this same query mechanism for Vision Language Models (VLMs). VLM Run's GraphQL capability enables you to extract only the specific fields you need from complex schemas, improving efficiency for querying and document ETL processes. This powerful feature allows you to precisely control what data is extracted, minimizing server-side processing overhead of extracting unncessary details, and simultaneously reducing the amount of data transferred over the network, providing a much more efficient and scalable way to extract data (i.e. ETL) from complex unstructured data.
## Benefits of GraphQL
* **Improved Performance**: Extract only the data fields you need (unlike OCR-based methods), reducing server-side computational overhead.
* **Reduced Bandwidth**: Minimize network traffic by receiving smaller, targeted responses
* **Flexible Data Selection**: Dynamically adjust which fields to extract based on your needs
* **Hierarchical Queries**: Select nested fields with intuitive syntax
## A Concrete End-to-End Example
Let's say you have an invoice PDF document that contains a table of data with an extensive list of fields (such as line items, tax, total, etc.). You can see the official schema we use for invoices [here](https://github.com/vlm-run/vlmrun-hub/blob/main/vlmrun/hub/schemas/document/invoice.py).
However, for your use case, you only need to extract the most important fields such as:
* **Invoice Number**: The number of the invoice
* **Issue Date**: The date of the invoice
* **Due Date**: The due date of the invoice
* **Total Amount**: The total amount of the invoice
Since we have already defined the schema for invoices, you can simply use it as a reference and select the fields you specifically need in the following GraphQL query:
```graphql theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
invoice_number
issue_date
due_date
total_amount
}
```
Now that you have defined the GraphQL query, you can provide it via the `gql_stmt` parameter to the `GenerationConfig` object in the `generate` method.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig
client = VLMRun(api_key="...")
response = client.document.generate(
file=Path("path/to/invoice.pdf"),
domain="document.invoice",
config=GenerationConfig(
gql_stmt="{invoice_number issue_date due_date total_amount}"
)
)
```
### Extracting Nested Fields with GraphQL
GraphQL’s hierarchical query structure enables precise extraction of deeply nested fields from complex document schemas. For instance, consider a scenario where you require not only top-level invoice metadata—such as `invoice_number`, `issue_date`, and `due_date`—but also a specific nested attribute like the `postal_code` within the `customer_billing_address` object. This can be accomplished with a single, declarative GraphQL query:
```graphql theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
invoice_number
issue_date
due_date
customer_billing_address {
postal_code
}
}
```
This approach leverages GraphQL’s ability to traverse and select arbitrary subfields within a schema, ensuring that only the minimal, application-relevant data is extracted from the model’s output. The result is a significant reduction in both server-side post-processing and network payload size, which is especially impactful when dealing with high-throughput ETL pipelines or latency-sensitive applications.
By architecting your data extraction workflows around GQL queries, you can enforce strict data contracts, optimize resource utilization, and build robust, scalable systems on top of VLM Run’s document intelligence capabilities.
## Try our Document -> JSON API today
Head over to our [Document -> JSON](/api-reference/v1/post-document-generate) to start building your own document processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Long-context Outputs
Source: https://docs.vlm.run/capabilities/long-context-outputs
Support for long-output contexts for domains like audio/video transcription, exceeding 8K token limits.
Navigate over to the video-transcription playground in our [playground](https://app.vlm.run/playground/video.transcription) to see the long-context outputs in action.
VLM Run provides robust support for processing and extracting structured data from long-context inputs like audio and video files. This capability enables you to work with extended content that would exceed the token limits of many foundation models (typically around 8K tokens).
This feature is currently only available for our enterprise-tier customers. If you are interested in using this feature, please [contact us](mailto:support@vlm.run).
## What are Long-context Outputs?
Long-context outputs refer to VLM Run's ability to process and extract structured data from extended output content like:
* Transcripts of long-form audio or video content (12+ hours of audio or 4+ hours of video)
* Extracted structured data from multi-page documents (>128 pages)
* Extracted structured data from large collections of related images (>128 images)
This capability is essential for applications that deal with lengthy content such as podcast transcriptions, lecture recordings, interviews, meetings, and extended video analysis.
## Using Long-context Processing
You can process long-form audio and video content using the VLM Run API with batch processing enabled:
### Long-form Audio / Audio Processing (with batch processing)
For all long-form audio and video processing, we only support `batch=True` mode.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig
# Initialize the client
client = VLMRun(api_key="your-api-key")
# Submit a prediction request with `batch=True`
prediction: PredictionResponse = client.audio.generate(
file=Path("path/to/long_video.mp4"), # Can be up to 4 hours long
domain="video.transcription", # use `audio.transcription` for audio files
batch=True,
config=GenerationConfig(
max_tokens=65_536,
),
)
# You can manually get the prediction by it's ID and check the status of the prediction
# Response status can take the following values: "pending" | "running" | "completed" | "failed"
# while :
# prediction: PredictionResponse = client.predictions.get(id=prediction.id)
# ...
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// TODO: Add Node SDK example
```
For batch processing, you are provided with a prediction ID and can check the status of the prediction later. We provide a polling mechanism and some convenience functions to check the status of the prediction and wait for it to complete.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Wait for the prediction to complete (with a timeout of 600 seconds)
prediction: PredictionResponse = client.predictions.wait(id=prediction.id, timeout=600)
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// TODO: Add Node SDK example
```
## Domain-specific Schemas
VLM Run provides specialized schemas for different types of long-form content:
* **`audio.transcription`**: General-purpose audio transcription with speaker detection
* **`video.transcription`**: General-purpose video transcription with visual scene analysis
* **`video.transcription-summary`**: Summary of a video transcription with key points and speaker analysis
* **`video.conferencing-summary`**: Summary of a video conference with key points and speaker analysis
* **`video.tv-news-summary`**: Summary of a TV news broadcast with anchors, reporters, chyrons, and segments
* **`video.dashcam`**: Analysis of a dashcam video with scene analysis and spoken language detection
Refer to the [Hub Catalog](/hub) for more information on the schemas supported by VLM Run.
## Example: Transcription of a YC Podcast Episode
Here's an example of a long-context output for a YC episode on [How New Technology Creates New Businesses](https://www.youtube.com/watch?v=KxjPgGLVJSg). As you can see, the output is a list of temporal segments grounded with start and end times, both audio transcription and visual understanding of the content.
```json YC Podcast Transcription [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"metadata": {
"language": null,
"content": null,
"topics": null,
"duration": 488.56
},
"segments": [
{
"start_time": 0,
"end_time": 25.8,
"audio": {
"content": " Like the only way to find these opportunities to learn about them is to find weirdos on the internet that are also into this thing. Yes. And they're figuring it out too. And you can kind of compare notes. Yes. And this is how new industries are created. Literally. By weirdos on the internet. Like literally. Literally. This is Dalton, plus Michael, and today we're going to talk about why AI is going to create more successful founders in the world."
},
"video": {
"content": "Two men are engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks. The man on the right, dressed in a blue shirt, listens attentively and occasionally responds with hand gestures. They appear to be in a professional setting, possibly an office or conference room, with large windows in the background allowing natural light to fill the space."
}
},
{
"start_time": 25.8,
"end_time": 51.71,
"audio": {
"content": " It's interesting, as we've gotten older, we kind of see a new set of tools come into the market and then an explosion in the number of founders who can now create value. And we've seen this before, right? Like, what was the first time you saw this? I certainly noticed when the internet was new, people that knew how to build websites were suddenly able to make lots of money from"
},
"video": {
"content": "The video features two individuals engaged in a conversation at a table. The person on the left, wearing a light gray shirt, is facing the person on the right, who is dressed in a blue jacket over a black shirt. The background is minimalistic, with a plain wall and a window allowing natural light to enter. The text overlay on the left side of the screen reads \"AI Will Create More Successful Founders\" and \"Founder Explosion.\" On the right side, there is a list titled \"Founder Explosion\" with various items such as \"On The Cusp,\" \"Cost Of Business,\" \"Get In Early,\" \"Whatnot,\" \"Endless Opportunity,\" and \"Internet Weirdos.\" The conversation appears to be focused on the impact of artificial intelligence on business and entrepreneurship."
}
},
{
"start_time": 51.71,
"end_time": 71.89,
"audio": {
"content": " the skill. And it was like really basic stuff. High school kids were making tons of money. Yep. I remember people that could just figure out how to sell stuff on eBay, where you would go buy something cheap but then listed on eBay and arbitrage. Yep. Basically, you would see people that kind of understood the new tooling that came out and would like do a hustle and make ungodly amounts of money."
},
"video": {
"content": "The video features two men engaged in a conversation in an office setting. The man on the left, wearing a light gray button-up shirt, is actively speaking and gesturing with his hands, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively with his arms crossed. The background includes a large window with blinds partially drawn, allowing natural light to filter into the room. The conversation appears to be focused on business-related topics, as indicated by the text on the right side of the screen, which lists various themes such as 'On The Cusp,' 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' and 'Internet Weirdos.' The overall atmosphere suggests a professional discussion."
}
},
{
"start_time": 72.01,
"end_time": 92.67,
"audio": {
"content": " Yeah. And it was just because they understood the new tools. And I already wasn't even a hustle. Like it was a good business. Like it was, they saw that tools enabled new businesses. You know, we saw this, you know, tail end of the open source world where like we could build all of Justin TV with free software. Yep."
},
"video": {
"content": "The video features two men engaged in a conversation in an office setting. The man on the left, wearing a light gray shirt, is gesturing animatedly with his hands as he speaks, indicating an active discussion. The man on the right, dressed in a blue jacket over a black shirt, listens attentively with his hands clasped together on the table. The background includes a window with blinds partially open, allowing natural light to filter into the room. On the right side of the screen, there is a vertical list titled \"Founder Explosion\" with various topics such as \"On The Cusp,\" \"Cost Of Business,\" \"Get In Early,\" \"Whatnot,\" \"Endless Opportunity,\" and \"Internet Weirdos.\""
}
},
{
"start_time": 92.67,
"end_time": 112.85,
"audio": {
"content": " And then we were there in the beginning of cloud compute where we didn't have to rack servers anymore. Any kid could sign up for an Amazon account, put a couple bucks down, and get access to a server. And so what's interesting is that we might, I think we feel pretty good about saying this, we might be on"
},
"video": {
"content": "The video features a conversation between two men seated at a table in a modern office setting. The man on the right, wearing a blue shirt and glasses, is speaking animatedly, gesturing with his hands as he discusses various topics related to entrepreneurship and business. The man on the left, dressed in a light-colored shirt, listens attentively, occasionally nodding and responding. The background includes a white wall and a window, suggesting a professional environment. On the right side of the screen, there is a list of topics being discussed, such as 'Founder Explosion,' 'On The Cusp,' 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' and 'Internet Weirdos.'"
}
},
{
"start_time": 112.85,
"end_time": 135.69,
"audio": {
"content": " the cusp of the next one of these. And that means there are maybe a whole bunch of new opportunities for successful businesses to be created. Yeah, starting now. Yeah, I mean, here's another metaphor. When the iPhone came out, who would have thought that Flappy Bird would have been created? And I think I read that that guy made like 20 million in cash."
},
"video": {
"content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is speaking animatedly, gesturing with his hands as he talks. The man on the right, dressed in a blue jacket over a black shirt, listens attentively, occasionally nodding and responding. The setting appears to be an office or meeting room with large windows in the background, allowing natural light to fill the space. The overall atmosphere suggests a professional discussion or interview."
}
},
{
"start_time": 135.87,
"end_time": 159.75,
"audio": {
"content": " Boom. In like two months and then shut it down. And so if you watch, okay, iPhone, Steve Jobs on stage, some guy in Southeast Asia building Flappy Bird. That's like wild. Never would have guessed. And so, again, to be very direct, what we're arguing is that when brand new technologies come out that are powerful, the people that are on the cusp of understanding them and that quickly"
},
"video": {
"content": "Two men are engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing animatedly with his hands as he speaks. The man on the right, dressed in a blue jacket, listens attentively, occasionally nodding and smiling. The background features a large window with a view of a body of water, suggesting an indoor setting with natural light."
}
},
{
"start_time": 159.75,
"end_time": 180.77,
"audio": {
"content": " build businesses or build useful things using those tools have a very unique view of creating businesses and wealth. And again, to be on the nose for AI, it seems like you can do things that would require way more headcount than you would otherwise. Yes. And so, you know, we're not even saying we know the ideas."
},
"video": {
"content": "The video features two men engaged in a conversation in an indoor setting. The man on the left, wearing a light gray button-up shirt, is actively gesturing with his hands as he speaks, indicating an animated discussion. The man on the right, dressed in a dark blue shirt, listens attentively, occasionally nodding and responding. The background includes a window with blinds, suggesting a modern office or studio environment. The video also displays a sidebar with various topics such as 'On The Cusp,' 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'New Is The Time,' which likely relate to the conversation's themes."
}
},
{
"start_time": 180.97,
"end_time": 201.72,
"audio": {
"content": " No. We're just saying if you're watching this and you're interested in being a founder or maybe not working at a company. Yeah. And you just pay attention to every new thing that comes out and try to find these opportunities or, I don't't know arbitrage is the right word, but no, just you know, new opportunities. New opportunities using these cutting edge tools"
},
"video": {
"content": "The video depicts a conversation between two men seated at a table in an office setting. The man on the left, wearing a light gray shirt, is gesturing animatedly with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively with his hands clasped together. The background features large windows with a view of a cityscape, and the room has a modern, minimalist design with white walls and a light-colored floor. The conversation appears to be focused and engaged, with both individuals actively participating in the dialogue."
}
},
{
"start_time": 201.72,
"end_time": 221.8,
"audio": {
"content": " and you're on the bleeding edge, you're not competing with anyone. No. It's green field. I think what's cool is any time one of these technologies shifts happens, the cost of starting a business, some set of businesses, reduces by up to like 10x. Yep. And so suddenly, businesses that either wouldn't have made sense"
},
"video": {
"content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively. The setting appears to be an office or meeting room with large windows in the background, allowing natural light to fill the space. The conversation seems to revolve around business topics, as indicated by the text on the right side of the screen, which includes phrases like 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'Now Is The Time.' The overall atmosphere suggests a professional discussion."
}
},
{
"start_time": 221.8,
"end_time": 244.96,
"audio": {
"content": " or certainly a normal person couldn't just stand up and do, right? Like can you imagine just, oh, it's pre online selling in eBay. All you have to do is rent a storefront and run a store, right? Like that's cheap, right? Like, absolutely not. Or like pre-Ari-NB. Like, all you have to do is just like buy a house and set up your own air bed and breakfast"
},
"video": {
"content": "The video features two men engaged in a conversation at a table in an office setting. The man on the left, wearing a light gray shirt, listens attentively while the man on the right, dressed in a blue jacket over a black shirt, gestures animatedly as he speaks. The background includes large windows with a view of a body of water, suggesting a modern and open environment. The conversation appears to be focused on business topics, as indicated by the text on the right side of the screen, which lists various themes such as 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'Now Is The Time.' The overall atmosphere is professional and collaborative."
}
},
{
"start_time": 244.96,
"end_time": 265.28,
"audio": {
"content": " bed and breakfast thing or even by hotel yeah that's crazy crazy. Whereas like Airbnb can rent a room. And think about it. Your own place. If you saw Airbnb early and you just decided to be a host and be like, oh, I should like do this as a business. You could do it pretty well. You could do it pretty well. Yeah. When Shopify was a brand new thing, like all of these platforms exactly the people that were the first to recognize that these were"
},
"video": {
"content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively. The background is minimalistic, with a plain wall and a window that lets in natural light. On the right side of the screen, there is a list of topics or themes, including \"Cost Of Business,\" \"Get In Early,\" \"Whatnot,\" \"Endless Opportunity,\" \"Internet Weirdos,\" and \"Now Is The Time.\" The overall setting appears to be a casual interview or discussion."
}
},
{
"start_time": 265.84,
"end_time": 286.22,
"audio": {
"content": " gave them leverage yes those entrepreneurial-minded people did really well. Yes. And so I think what's so cool is that what we're saying is like if you're ambitious and you're paying attention, you might not ever need to work at a big company. You might not ever need to have a boss."
},
"video": {
"content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing animatedly with his hands as he speaks. The man on the right, dressed in a blue jacket, listens attentively, occasionally responding with his own gestures. The setting appears to be an office or meeting room with large windows in the background, allowing natural light to fill the space. The conversation seems to revolve around business or startup topics, as indicated by the text on the right side of the screen, which includes phrases like 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'Now Is The Time.' The overall atmosphere suggests a professional and collaborative discussion."
}
},
{
"start_time": 286.22,
"end_time": 307.96,
"audio": {
"content": " Like you can be in control of your own destiny. And these moments don't happen every week. No. Like, we wish we did. It would be the investor. But like, when they do, the people who move. I mean, this is a very specific example. Yeah. But whatnot, the online live shopping thing, I still talk to the founders a lot."
},
"video": {
"content": "Two men are sitting at a table in a modern office setting, engaged in a conversation. The man on the left is wearing a light gray button-up shirt and has short, curly hair. He appears to be listening attentively to the man on the right, who is bald, wearing glasses, and dressed in a blue jacket over a black shirt. The man on the right is gesturing with his hands as he speaks, indicating an animated discussion. The background features large windows with a view of a cityscape, suggesting an urban environment. On the right side of the screen, there is a list of topics or themes related to business and entrepreneurship, such as 'Founder Explosion,' 'On The Cusp,' 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'Now Is The Time.'"
}
},
{
"start_time": 308.22,
"end_time": 328,
"audio": {
"content": " And they have, I think, like, high school-aged kids selling stuff on their... Making real money, right? And making just, again, I don't even want to say the numbers. Yeah. But they figured out the format. They understand how to use whatnot. They built a user base there. Yeah. And they're basically... They're making enough money to set themselves up for their entire life."
},
"video": {
"content": "A man with curly hair and a light gray shirt is speaking animatedly to another man who has a shaved head and glasses. The man with curly hair uses expressive hand gestures as he talks, while the other man listens attentively. The background features a window with blinds, and there is a menu or list of topics on the right side of the screen, including \"Founder Explosion,\" \"On The Cusp,\" \"Cost Of Business,\" \"Get In Early,\" \"Whatnot,\" \"Endless Opportunity,\" \"Internet Weirdos,\" and \"Now Is The Time.\""
}
},
{
"start_time": 328.12,
"end_time": 351.76,
"audio": {
"content": " Yeah. By just seeing this new platform, figuring it out, and then making a bet on it. Yes. I mean, this happened with Twitch. Happens with Twitch. Whole new industry, basically. Yeah. No, I think that what's cool is that we're also talking about every scale, right? We're talking about things that can be venture backed, maybe billion dollar companies one day. But we're also talking about things that can just"
},
"video": {
"content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively with his hands clasped together. The background includes a large window with a view of a body of water, suggesting an indoor setting with natural light. The conversation appears to be casual and focused, with both individuals actively participating in the dialogue."
}
},
{
"start_time": 351.76,
"end_time": 372.66,
"audio": {
"content": " set you up so that you can pay your rent and live a good life. Yep. The opportunities are across the entire spectrum. And I think that's what's really cool about new technology. Like when there's a real technology shift, it affects businesses across the board. We're not just talking about companies that YCP even fun. I think that the last point I'd want to make"
},
"video": {
"content": "A man in a blue shirt is seated at a table, engaged in a conversation with another person whose back is facing the camera. The man in the blue shirt is gesturing with his hands as he speaks, indicating an animated discussion. The setting appears to be an office or a professional environment, with a window and some furniture visible in the background. The overall atmosphere suggests a serious and focused conversation."
}
},
{
"start_time": 372.66,
"end_time": 398.57,
"audio": {
"content": " on this front is that you don't get this opportunity if you're just thinking, you gotta actually do. Well, and they won't teach you this in schools. Schools teach you stuff for 10 or 20 years ago. So the other thing that I've noticed in these trends is that when you are part of the history being made and you're this early on the cutting edge of a new tech coming out, you can't expect your university or your teachers or your peers people in your community"
},
"video": {
"content": "Two men are engaged in a conversation at a table in an office setting. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively. The background features large windows with a view of a cityscape, and the room has a modern, minimalist design with white walls and a light-colored floor. The conversation appears to be focused and serious, with both individuals maintaining eye contact and using expressive hand movements."
}
},
{
"start_time": 398.57,
"end_time": 419.33,
"audio": {
"content": " or your peers to teach you about it. It's only basically weirdos on the internet. Yes. Like the only way to find these opportunities to learn about them is to find weirdos on the internet that are also into this thing. Yes. And they're figuring it out too. that are also into this thing. And they're figuring it out too. And you can kind of compare notes. Yes. And this is how new industries are created. Literally. By weirdos on the internet. Like literally. Literally. By weirdos on the internet. Like literally. Literally, there's like some subreddit with a bunch of weirdos."
},
"video": {
"content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is animatedly gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket, listens attentively with his hands clasped together. The setting appears to be a modern office or conference room with large windows in the background, allowing natural light to fill the space. The conversation seems to be casual and friendly, with both individuals appearing relaxed and engaged."
}
},
{
"start_time": 419.33,
"end_time": 447.2,
"audio": {
"content": " And like someday from now, you know, 10 years from now, there'll be an entire industry of people that learned about this thing in some subred somewhere there. Yeah, no, I totally agree. So hey, the big takeaway is if you've been wrestling your lawyers, if you thought, oh, this isn't the time to start a new business. Maybe you should reconsider. Yeah. This is a very interesting time. I think the final argument is there's a good case where a smaller percentage of the population will need to get jobs,"
},
"video": {
"content": "The video features two men engaged in a conversation in an office setting. The man on the left, wearing a light gray shirt, is animatedly speaking and using hand gestures to emphasize his points. He appears to be explaining something with enthusiasm. The man on the right, dressed in a blue jacket over a black shirt, listens attentively, occasionally nodding and responding. The background includes a window with blinds, suggesting a modern office environment. The conversation seems to revolve around business or technology topics, as indicated by the text overlays such as 'Cost Of Business' and 'Endless Opportunity.'"
}
},
{
"start_time": 447.26,
"end_time": 469.08,
"audio": {
"content": " and more people will be able to use tools like this to be self-employed in some way. I don't think there's any, I think all the structural changes imply that more folks will just use their highly leveraged selves using all these tools to run businesses, then have to go get a job. Yeah. Right? And I think that story isn't told, right? I think the story is always this kind of depressing story of like,"
},
"video": {
"content": "The video features two men engaged in a conversation in an indoor setting. The man on the left, wearing a light gray button-up shirt, is actively gesturing with his hands as he speaks, indicating an animated discussion. The man on the right, dressed in a blue jacket over a black shirt, listens attentively, occasionally nodding and responding. The background includes a window with blinds, suggesting a modern office or conference room environment. The overall atmosphere appears to be professional and focused."
}
},
{
"start_time": 469.08,
"end_time": 488.56,
"audio": {
"content": " oh, maybe you won't need it, you won't be needed anymore, as opposed to here's a set of tools. You could do things that people couldn't think of doing affordably before. Like you could be your own boss. You don't even need to be inside of a company to create value. Yeah. So anyways, hopefully that's inspiring. Good shot. Thanks. good shot thanks"
},
"video": {
"content": "Two men are engaged in a conversation at a table in an office setting. The man on the left, wearing a light gray shirt, listens attentively while the man on the right, dressed in a blue jacket over a black shirt, speaks animatedly. He uses hand gestures to emphasize his points, occasionally clasping his hands together on the table. The background features large windows with a view of a cloudy sky, and the room is well-lit with natural light."
}
}
]
}
```
## Use Cases
* **Content Search**: Make audio/video content searchable through transcription
* **Meeting Intelligence**: Extract action items and key points from meeting recordings
* **Media Monitoring**: Analyze news broadcasts and identify topics and speakers
* **Educational Content**: Structure course lectures with chapters and topics
* **Podcast Production**: Generate show notes, summaries, and topic timestamps
By leveraging VLM Run's long-context output capabilities, you can efficiently extract structured information from extended audio and video content that would otherwise exceed traditional token limits.
## Try our Video / Audio -> JSON API today
Head over to our [Video -> JSON](/api-reference/v1/post-video-generate) or [Audio -> JSON](/api-reference/v1/post-audio-generate) to start building your own video/audio processing pipelines with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Structured Responses
Source: https://docs.vlm.run/capabilities/structured-responses
Extract JSON from images, videos, and documents with type-safety.
Navigate over to the [hub](https://github.com/vlm-run/vlmrun-hub) to see the structured responses for various domains in action.
Foundation vision models like [OpenAI's **GPT4o**](https://openai.com/index/hello-gpt-4o/) and [Anthropic's **Claude Vision**](https://docs.anthropic.com/en/docs/build-with-claude/vision) support question answering over visual inputs, a.k.a. *chat with images*.
However, we believe chat is **NOT** the ideal interface for many software workflows, especially those that require automation. Instead, developers want **strongly-typed** and **validated outputs** that can be easily integrated into their existing software workflows.
Our internal VLMs are built on exactly this insight - instead of free-form text outputs,
we define our API in terms of fixed types for specific domains (e.g. PDF presentations, TV news, audio / video podcasts etc). The schemas defined can be arbitrarily nested, and can include lists, dictionaries, and other complex types that can richly capture the information contained in the input.
In other words, **`vlm-1`** is purpose-built for what is popularly known as [**JSON mode**](https://platform.openai.com/docs/guides/json-mode). This mode is particularly useful for developers who want to build automation workflows, data pipelines, or other software systems that require structured data as output.
## Extract Structured Data
With our [pre-defined domains](/hub.mdx), you can quickly extract structured data from images, videos, and other visual content in a single API call. The extracted data will be validated against the schema you defined, ensuring that it conforms to the expected structure and types.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse
# Initialize the client
client = VLMRun(api_key="")
# Process the file or image with the predefined schema
path: Path = Path("path/to/invoice.pdf")
prediction: PredictionResponse = client.document.generate(
file=path,
domain="document.invoice",
)
response_dict = prediction.response.model_dump()
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
// Initialize the client
const client = new VlmRun({ apiKey: '' });
// Process the file or image with the predefined schema
const fileResponse = await client.files.upload(filePath: "")
const prediction = await client.document.generate({
fileId: fileResponse.file_id,
domain: "document.invoice"
});
console.log(prediction);
```
## Illustrative Examples
Here is an example of the structured JSON output that `vlm-1` can extract from an invoice:
You should see a response like this:
```bash Response [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"invoice_id": "79BBD516-0005",
"period_start": null,
"period_end": null,
"invoice_issue_date": "2024-01-10",
"invoice_due_date": "2024-02-09",
"order_id": null,
"customer_id": null,
"issuer": "Typographic",
"issuer_address": {
"street": "1 Grand Canal St Lower",
"city": "Dublin",
"state": "Co. Dublin",
"postal_code": "D04 Y7R5",
"country": "Ireland"
},
"customer": "French Customer",
"customer_email": null,
"customer_phone": "+33 1 23 45 67 89",
"customer_billing_address": {
"street": "5 Avenue Anatole France",
"city": "Champ de Mars",
"state": "Paris",
"postal_code": "75007",
"country": "France"
},
"customer_shipping_address": null,
"items": [
{
"description": "Line Item 1",
"quantity": 1,
"currency": "EUR",
"unit_price": 10.0,
"total_price": 10.0
},
{
"description": "Line Item 2",
"quantity": 1,
"currency": "EUR",
"unit_price": 5.0,
"total_price": 5.0
}
],
"subtotal": 15.0,
"tax": 0.0,
"total": 15.0,
"currency": "EUR",
"notes": "[1] Tax to be paid on reverse charge basis",
"others": {
"due_amount": 15.0,
"vat_number": "FRAB123456789",
"support_email": "support@typographic.com",
"contact_phone": "+353123456789"
}
}
```
As you can see, `vlm-1` can extract a wide range of detailed information from the invoice, including vendor and customer details, line items, payment terms, and more. This structured data can be easily integrated into various financial systems, accounting software, or used for automated invoice processing.
## Custom Schemas
In addition to the [pre-defined schemas](/hub.mdx) we provide, **`vlm-1`** also supports custom schemas that allows you to define your own schema for a specific domain or use-case. This gives you the flexibility to extract structured data that conforms to your specific needs and requirements, while still leveraging all the vision-based reasoning capabilities of **`vlm-1`** (see [Capabilities section](/capabilities/) for more details). See the next section on [Custom Schemas](/capabilities/custom-schemas) for more details.
## Try our Document -> JSON API today
Head over to our [Document -> JSON](/api-reference/v1/post-document-generate) to start building your own document processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Temporal Grounding
Source: https://docs.vlm.run/capabilities/temporal-grounding
Ground extracted data with start/end times for audio/video segments and speaker identification.
Navigate over to the video-transcription playground in our [playground](https://app.vlm.run/playground/video.transcription) to see the temporal grounding in action.
Temporal grounding is a powerful capability of VLM Run that links extracted data to precise time segments within audio and video content. This feature is especially valuable for applications that need to process and analyze time-based media—such as podcasts, interviews, lectures, and meetings—by providing structured, timestamped insights.
Temporal grounding can be broadly categorized into two key functions:
1. **Time Segmentation**: Dividing content into meaningful segments, each with precise start and end timestamps.
2. **Content Localization**: Pinpointing exactly when and where specific information appears within the timeline.
***
## Using Temporal Grounding
Temporal grounding when processing audio/video content is enabled by default for all audio/video domains.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig
client = VLMRun(api_key="...")
response = client.video.generate(
file=Path("path/to/episode.mp4"),
domain="video.transcription",
)
```
## Understanding the Output
The response includes temporal information for each extracted segment, including start and end times, speaker identification, and confidence scores:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"metadata": {
"duration": 248.67 // total duration of the video in seconds
},
"segments": [
{
"start_time": 0, // start time of the first segment in seconds
"end_time": 21.33, // end time of the first segment in seconds
"audio": {
"content": " The Keys Rocks is a rough and tumble area just outside Pittsburgh. My name is Scott Baker. My family has been connected with this community for generations. In 1941, my grandfather opened his bakery here, and he called it Jenny Lee. We used to always go to Jenny Lee's after church. If you were good in church,"
},
"video": {
"content": "In this image we can see many buildings and trees. There is a bridge in the image. There are many towers in the background of the image and the sky is in white color."
}
},
{
"start_time": 21.33, // start time of the second segment in seconds
"end_time": 42.5, // end time of the second segment in seconds
"audio": {
"content": " oh, egg custard pot. Gooden Church. Oh, I ate custard pies, homemade bread that was still warm. Years later, I worked in the store. I did wedding cakes. My father, Bernie, took over after my grandfather retired. I am a baker. took over after my grandfather retired. I am a baker by name, baker by trade. I started coming in to the bakery with my dad when I was seven or eight years old."
},
"video": {
"content": "In this image we can see a man standing on the floor. We can also see a group of people standing beside a table containing some food items in a cupboard. On the backside we can a wall with some photo frames and a roof with some ceiling lights."
}
},
...
{
"start_time": 230.67, // start time of the last segment in seconds
"end_time": 248.67, // end time of the last segment in seconds
"audio": {
"content": " I made that. I'm proud to say that. You know, You know, You know, Thank you."
},
"video": {
"content": "In this image we can see a person holding a food item in his hand. In front of him there is a table. In the background of the image there are some objects."
}
}
]
}
```
If you want to test the grounding precision of our models, you can go to the [VLM Run Platform](https://app.vlm.run) and click on the `start_time` and `end_time` of any of the segments to skip to the corresponding audio/video segments.
## Use Cases
Temporal grounding enables numerous applications:
1. **Searchable Media Archives**: Create searchable indexes of audio and video content
2. **Meeting Summaries**: Generate timestamped summaries of meetings with speaker attribution
3. **Content Navigation**: Build interfaces that allow users to jump to specific topics or speakers
4. **Podcast Production**: Automatically generate show notes with timestamps and speaker labels
5. **Video Chapters**: Create chapter markers for long-form video content
6. **Interview Analysis**: Extract insights from interviews with accurate speaker attribution
7. **Compliance Monitoring**: Track who said what and when in regulated communications
By leveraging VLM Run's temporal grounding capabilities, you can extract rich, time-based structured data from audio and video content, enabling powerful applications that understand not just what was said, but who said it and when.
## Try our Video / Audio -> JSON API today
Head over to our [Video -> JSON](/api-reference/v1/post-video-generate) or [Audio -> JSON](/api-reference/v1/post-audio-generate) to start building your own video/audio processing pipelines with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Visual Grounding
Source: https://docs.vlm.run/capabilities/visual-grounding
Ground extracted data with location (bounding box) coordinates and confidence scores.
Navigate over to the [driver's license playground](https://app.vlm.run/playground/document.us-drivers-license) to see the visual grounding in action.
VLM Run's visual grounding capability connects extracted data to precise locations in your visual content. This feature maps structured data to specific coordinates within documents, images, or videos, providing spatial context for extracted information. With visual grounding, you can pinpoint exactly where data originated - for example, identifying which table in a document corresponds to a particular JSON object in your results.
## How Visual Grounding Works
When enabled, `vlm-1` provides the location information for each of the Pydantic fields in the JSON response, under the `metadata` key. The bounding box coordinates are represented in a normalized `xywh` format (see [Bounding Box](#2-bounding-box-bbox) below)
## Using Visual Grounding
You can enable visual grounding in by simply setting the `grounding` parameter to `True` in your `GenerationConfig`:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig, PredictionResponse
client = VLMRun(api_key="...")
prediction: PredictionResponse = client.image.generate(
file=Path("path/to/license.jpg"),
domain="document.us-drivers-license",
config=GenerationConfig(grounding=True),
)
print(prediction.response.model_dump_json(indent=2))
```
## Understanding the Output
For the purpose of this example, we have simplified the JSON response and metadata to only include the `customer_name` and `invoice_date` fields:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"license_number": "1234567",
"license_number_metadata": {
"confidence": "hi",
"bboxes": [
{
"xywh": [0.2, 0.15, 0.15, 0.05],
"page": 0
}
]
},
"license_expiration_date": "2014-01-05",
"license_expiration_date_metadata": {
"confidence": "hi",
"bboxes": [
{
"xywh": [0.8, 0.2, 0.15, 0.05],
"page": 0
}
]
}
...
}
```
### 1. Confidence Levels: `*_metadata.confidence`
Each grounded field includes a confidence score, which can be one of:
* `hi`: High confidence in the extraction accuracy
* `med`: Medium confidence, suggesting some uncertainty
* `low`: Low confidence, indicating potential inaccuracy
These confidence values help you assess the reliability of the extracted data and decide whether manual review might be needed. Only a *single* confidence value is returned for each field (unlike the bounding boxes below).
### 2. Bounding Box: `*_metadata.bboxes`
The bounding box coordinates are represented in a normalized `xywh` format, where each value is between 0 and 1, representing:
* `x`: horizontal position of the top-left corner (0 = left edge, 1 = right edge)
* `y`: vertical position of the top-left corner (0 = top edge, 1 = bottom edge)
* `w`: width of the box (0 = no width, 1 = full image/document width)
* `h`: height of the box (0 = no height, 1 = full image/document height)
## Visual Grounding on a Document
In the earlier example, we showed you how to ground the data fields in a single image. However, when working with documents, you may want to ground the data fields that may appear on multiple locations, across multiple pages. VLM Run's visual grounding capability extends to multi-page documents, allowing you to extract and localize data across entire document sets.
When processing multi-page documents:
* Instead of a single bounding box, each field `_metadata.bbox` may have multiple bounding boxes, along with the page number metadata under `_metadata.bboxes[].page`.
* Page numbers are included in the metadata for each grounded element. This allows you to navigate to the correct page when users interact with the data or are looking to cite the data.
Here's an example of visual grounding in a multi-page document:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"invoice_number": "INV-2023-0042",
"invoice_number_metadata": {
"confidence": "hi",
"bboxes": [
{
"xywh": [0.7, 0.1, 0.2, 0.05],
"page": 1
},
{
"xywh": [0.7, 0.1, 0.2, 0.05],
"page": 2 // invoice_number can typically be found on each page of the document
},
...
]
},
"total_amount": "1000",
"total_amount_metadata": {
"confidence": "hi",
"bboxes": [
{
"xywh": [0.2, 0.2, 0.3, 0.05],
"page": 0 // `total_amount` can typically be found on the first and last page of the document
},
{
"xywh": [0.6, 0.8, 0.3, 0.05],
"page": 6 // `total_amount` can typically be found on the first and last page of the document
}
]
}
}
```
## Use Cases
Visual grounding enables several powerful applications:
1. **Document verification**: Validate the location of key fields in identity documents (verification checks for KYC, AML, etc.)
2. **Data extraction audit**: Verify the source location of extracted information, typically for high-stakes or sensitive applications (finance, healthcare, etc.)
3. **Interactive annotation**: Build interfaces that highlight document regions as users interact with extracted data (e.g. highlight the bounding box around the `invoice_number` for back-office operations)
4. **Error correction**: Easily identify and fix extraction errors by referring to the original location of the data.
5. **Document comparison**: Compare the location of similar elements across different document versions
By combining structured data extraction with spatial localization, visual grounding provides a comprehensive solution for document processing tasks that require both the "what" and the "where" of information.
For a hands-on tutorial, check out our [Visual Grounding Notebook](https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/04_visual_grounding.ipynb) [](https://colab.research.google.com/github/vlm-run/vlmrun-cookbook/blob/main/notebooks/04_visual_grounding.ipynb).
## Try our Document -> JSON API today
Head over to our [Document -> JSON](/api-reference/v1/post-document-generate) to start building your own document processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Changelog
Source: https://docs.vlm.run/changelog
Changelog for VLM Run.
The following is a list of recent changes to the [VLM Run Platform](https://app.vlm.run) and [API](https://docs.vlm.run/api-reference/v1/introduction).
* [Orion 2 Release](https://www.vlm.run/blog/orion-2): Full launch of Orion 2, our most capable visual agent, now with code execution.
* [Model Selector](https://docs.vlm.run/agents/code-execution#model-variants): Choose which model runs inside the Orion 2 visual agent harness, from open-weight models like Gemma 4 to closed models like Opus 4.8.
* [Media Library](https://chat.vlm.run): Upload, browse, and chat with your own media and documents through a new library panel in chat.
* [Healthcare multi-document workflow cookbook](https://colab.research.google.com/github/vlm-run/vlmrun-cookbook/blob/main/notebooks/16_healthcare_multi_document_workflow.ipynb): Learn how to take multiple documents, classify them, and extract information from a specific document.
* New [Platform tab](/platform/overview): a single place to learn how the VLM Run Platform works.
* [Service Tiers](/pricing#service-tiers) (Standard/Flex/Priority): a new `service_tier` parameter is now supported end-to-end: from the backend, through the Python SDK's `GenerationConfig`, to full documentation with cURL and SDK examples.
* New [evaluations guide](/guides/evaluations): learn how to measure skill accuracy, collect feedback, run evaluations, and use results to optimize your skills over time.
* Improved video processing and transcription with longer, richer output.
* Process specific document pages with the new `page_indices` parameter in `GenerationConfig`.
* LaTeX rendering in chat messages for math and scientific content.
* Lossless PNG support for segmentation masks with color keys, unlocking more segmentation use cases.
* **Evaluations**: Measure workflow improvements using skills in the Evaluations tab.
* **Skills**:
* Support for uploading skills via zip files
* Refer to skills quickly in chat by typing `/` and selecting a skill you've used
* Python and Node SDKs support skills management, inline skills, and skill-driven extraction
* Improved chat completion latency
* Improved multimodal content handling
* New `/document/execute` endpoint for running arbitrary VLM DAGs over documents.
* Added support for `document.markdown` domain for extracting structured markdown from documents.
* Fixes for multi-page grounding support with custom JSON schemas in [Document -> JSON](/api-reference/v1/post-document-generate).
* Added support for `audio.transcription` domain for long-form audio transcription (upto 12 hours of audio).
* Added support for `video.transcription` domain for long-form video transcription (upto 3 hours of video).
* Added cookbook for [`video transcription`](https://github.com/vlm-run/vlmrun-cookbook?tab=readme-ov-file#-cookbook-notebooks).
# chat
Source: https://docs.vlm.run/cli/chat
Chat with Orion to process images, videos, and documents
The `vlmrun chat` command enables visual AI chat with Orion directly from your terminal. Process images, videos, and documents with natural language prompts.
## Basic Usage
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Describe an image
vlmrun chat "Describe this image" -i photo.jpg
# Analyze a document
vlmrun chat "Extract the key information" -i document.pdf
# Process a video
vlmrun chat "Summarize this video" -i video.mp4
# Compare multiple files
vlmrun chat "Compare these images" -i image1.jpg -i image2.jpg
```
## Prompt Sources
Prompts can be provided in three ways (in precedence order):
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# 1. Direct argument
vlmrun chat "Your prompt here" -i file.jpg
# 2. Using -p option (text, file path, or stdin)
vlmrun chat -p "Your prompt" -i file.jpg
vlmrun chat -p prompt.txt -i file.jpg
# 3. Piped stdin
echo "Describe this" | vlmrun chat - -i file.jpg
cat prompt.txt | vlmrun chat - -i file.jpg
```
## Using Skills
Pass a local skill directory with `-k` to apply skill instructions inline:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Use a local skill
vlmrun chat "Extract data from this invoice" -i invoice.pdf -k ./my-skill
# Multiple skills
vlmrun chat "Analyze this" -i photo.jpg -k ./skill-a -k ./skill-b
```
The `-k` flag sends the skill inline with the request (no server-side upload). To create a persistent server-side skill, use [`vlmrun skills upload`](/cli/skills).
## Stateful Sessions
Use `--session-id` to persist chat history across multiple calls:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Start a session
vlmrun chat "What's in this image?" -i photo.jpg --session-id my-session
# Continue the conversation
vlmrun chat "Now describe the colors" --session-id my-session
```
## Models
| Model | Description |
| --------------------- | ------------------ |
| `vlmrun-orion-1:fast` | Speed-optimized |
| `vlmrun-orion-1:auto` | Balanced (default) |
| `vlmrun-orion-1:pro` | Most capable |
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun chat "Describe this" -i photo.jpg -m vlmrun-orion-1:pro
```
## Output Formats
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Rich formatted output (default)
vlmrun chat "Describe this" -i photo.jpg
# JSON output for scripting
vlmrun chat "Describe this" -i photo.jpg --json
# Pipe JSON to jq
vlmrun chat "Describe this" -i photo.jpg --json | jq '.content'
# Disable streaming (wait for complete response)
vlmrun chat "Describe this" -i photo.jpg --no-stream
```
## Artifact Handling
When Orion generates artifacts (images, videos, etc.), they are automatically downloaded:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Default: saved to ~/.vlm/cache/artifacts//
vlmrun chat "Generate a variation of this image" -i photo.jpg
# Custom output directory
vlmrun chat "Generate a variation" -i photo.jpg -o ./output/
# Skip artifact download
vlmrun chat "Generate a variation" -i photo.jpg --no-download
```
## Supported File Types
| Category | Extensions |
| --------- | --------------------------------------------------------- |
| Images | `.jpg`, `.jpeg`, `.png`, `.gif`, `.webp`, `.bmp`, `.tiff` |
| Videos | `.mp4`, `.mov`, `.avi`, `.mkv`, `.webm` |
| Documents | `.pdf`, `.doc`, `.docx` |
| Audio | `.mp3`, `.wav`, `.m4a`, `.flac`, `.ogg` |
## Options Reference
| Option | Short | Description |
| --------------- | ----- | ---------------------------------------------- |
| `--prompt` | `-p` | Prompt: text string, file path, or `stdin` |
| `--input` | `-i` | Input file (repeatable) |
| `--skill` | `-k` | Path to a skill directory (repeatable) |
| `--output` | `-o` | Artifact output directory |
| `--model` | `-m` | Model variant (default: `vlmrun-orion-1:auto`) |
| `--json` | `-j` | Output JSON instead of formatted text |
| `--no-stream` | `-ns` | Disable streaming |
| `--no-download` | `-nd` | Skip artifact download |
| `--session-id` | `-s` | Session UUID for stateful conversations |
| `--base-url` | | API base URL override |
# config
Source: https://docs.vlm.run/cli/config
Manage CLI configuration
The `vlmrun config` command manages CLI configuration stored at `~/.vlmrun/config.toml`.
## Initialize
Create a default configuration file:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Create config file
vlmrun config init
# Overwrite existing config
vlmrun config init --force
```
| Option | Short | Description |
| --------- | ----- | ------------------------------ |
| `--force` | `-f` | Overwrite existing config file |
## Show
Display the current configuration (API key is masked):
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun config show
```
## Set
Set configuration values:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Set API key
vlmrun config set --api-key "your-api-key"
# Set base URL
vlmrun config set --base-url "https://api.vlm.run/v1"
```
| Option | Description |
| ------------ | -------------------- |
| `--api-key` | VLM Run API key |
| `--base-url` | VLM Run API base URL |
## Unset
Remove configuration values:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Remove API key
vlmrun config unset --api-key
# Remove base URL
vlmrun config unset --base-url
```
| Option | Description |
| ------------ | ------------------ |
| `--api-key` | Unset the API key |
| `--base-url` | Unset the base URL |
## Configuration Priority
Configuration values are resolved in the following order (highest priority first):
1. CLI flags (`--api-key`, `--base-url`)
2. Environment variables (`VLMRUN_API_KEY`, `VLMRUN_BASE_URL`)
3. Config file (`~/.vlmrun/config.toml`)
# execute
Source: https://docs.vlm.run/cli/execute
Execute agents with files, skills, and structured output
The `vlmrun execute` command submits agent executions via the `/v1/agent/execute` endpoint. Upload files, attach skills, and get structured JSON output.
## Basic Usage
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Execute a named agent with a file
vlmrun execute -n my-agent:v1 -i invoice.pdf
# Execute with a prompt and schema
vlmrun execute -p "Extract invoice fields" -i doc.pdf --schema schema.json
# Execute with multiple files and toolsets
vlmrun execute -n my-agent:v1 -i a.jpg -i b.pdf -t image -t document
```
## Agent Names
Use `:` format. If `--name` is omitted, the prompt is used to identify the agent.
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun execute -n invoice-extractor:v2 -i invoice.pdf
vlmrun execute -p "Describe this image" -i photo.jpg
```
## Using Skills
Attach skills inline from a local directory or reference server-side skills by ID. Only one of `--skill` or `--skill-id` may be provided.
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Inline skill from a local directory
vlmrun execute -n my-agent:v1 -i img.jpg --skill ./my-skill
# Server-side skill reference
vlmrun execute -n my-agent:v1 -i img.jpg --skill-id my-skill:latest
```
## Models
| Model | Description |
| --------------------- | --------------------- |
| `vlmrun-orion-1:lite` | Lightweight |
| `vlmrun-orion-1:fast` | Speed-optimized |
| `vlmrun-orion-1:auto` | Auto-select (default) |
| `vlmrun-orion-1:pro` | Most capable |
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun execute -n my-agent:v1 -i photo.jpg -m vlmrun-orion-1:pro
```
## Toolsets
Enable specific tool categories with `--toolset` (repeatable):
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun execute -n my-agent:v1 -i photo.jpg -t image -t web
```
Available toolsets: `core`, `image`, `image-gen`, `world-gen`, `viz`, `document`, `video`, `web`.
## Async Execution
By default the CLI waits for execution to complete. Use `--no-wait` to submit and return immediately:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Submit and return immediately
vlmrun execute -n my-agent:v1 -i photo.jpg --no-wait
# Check status later
vlmrun executions get
```
## Output Formats
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Rich formatted output (default)
vlmrun execute -n my-agent:v1 -i invoice.pdf
# JSON output for scripting
vlmrun execute -n my-agent:v1 -i invoice.pdf -f json
# Pipe JSON to jq
vlmrun execute -n my-agent:v1 -i invoice.pdf -f json | jq '.response'
```
## Prompt Sources
The prompt can be text or a path to a `.txt`/`.md` file:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Inline text
vlmrun execute -p "Extract all line items" -i invoice.pdf
# From a file
vlmrun execute -p instructions.md -i invoice.pdf
```
## Options Reference
| Option | Short | Description |
| ---------------------- | ----- | ---------------------------------------------------- |
| `--name` | `-n` | Agent name as `:` |
| `--prompt` | `-p` | Prompt text or path to a `.txt`/`.md` file |
| `--input` | `-i` | Input file (repeatable) |
| `--schema` | | Path to a JSON schema file for the response model |
| `--skill` | `-k` | Path to a local skill directory (repeatable) |
| `--skill-id` | | Server-side skill as `:` (repeatable) |
| `--toolset` | `-t` | Tool category to enable (repeatable) |
| `--model` | `-m` | Model variant (default: `vlmrun-orion-1:auto`) |
| `--wait` / `--no-wait` | | Wait for completion (default: `--wait`) |
| `--timeout` | | Timeout in seconds when waiting (default: `300`) |
| `--poll-interval` | | Seconds between status checks (default: `5`) |
| `--callback-url` | | Webhook URL called on completion |
| `--format` | `-f` | Output format (`json`) |
# executions
Source: https://docs.vlm.run/cli/executions
List and retrieve agent execution results
The `vlmrun executions` command lets you list and retrieve results from agent executions submitted via [`vlmrun execute`](/cli/execute) or the API.
## List Executions
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List recent executions (default: last 10)
vlmrun executions list
# Filter by status
vlmrun executions list --status completed
vlmrun executions list --status failed
# Filter by date range
vlmrun executions list --since 2026-03-01 --until 2026-03-30
# Pagination
vlmrun executions list --limit 25 --skip 10
# JSON output
vlmrun executions list -f json
```
| Option | Short | Description |
| ---------- | ----- | ----------------------------------------------------------------------------------- |
| `--limit` | | Max items to return (default: `10`) |
| `--skip` | | Number of items to skip (default: `0`) |
| `--status` | | Filter by status: `enqueued`, `pending`, `running`, `completed`, `failed`, `paused` |
| `--since` | | Show executions since date (`YYYY-MM-DD`) |
| `--until` | | Show executions until date (`YYYY-MM-DD`) |
| `--format` | `-f` | Output format (`json`) |
## Get Execution
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get execution details
vlmrun executions get
# Wait for an execution to complete
vlmrun executions get --wait
# Wait with custom timeout
vlmrun executions get --wait --timeout 120
# JSON output
vlmrun executions get -f json
```
| Argument/Option | Short | Description |
| ----------------- | ----- | ----------------------------------------------------- |
| `EXECUTION_ID` | | ID of the execution to retrieve (required) |
| `--wait` | | Wait for execution to complete (default: `--no-wait`) |
| `--timeout` | | Timeout in seconds when waiting (default: `300`) |
| `--poll-interval` | | Seconds between status checks (default: `5`) |
| `--format` | `-f` | Output format (`json`) |
# files
Source: https://docs.vlm.run/cli/files
Upload, list, retrieve, and delete files
The `vlmrun files` command manages files on the VLM Run platform.
## Upload
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun files upload photo.jpg
vlmrun files upload document.pdf --purpose vision
```
| Argument/Option | Description |
| --------------- | ------------------------------------------------------------------------------------------ |
| `FILE` | Path to the file to upload (required) |
| `--purpose` | File purpose: `datasets`, `fine-tune`, `assistants`, `vision`, etc. (default: `fine-tune`) |
## List
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun files list
```
## Get
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get file info
vlmrun files get
# Download to a specific path
vlmrun files get --output ./downloaded-file.jpg
```
| Argument/Option | Description |
| --------------- | ------------------------------------- |
| `FILE_ID` | ID of the file to retrieve (required) |
| `--output` | Output file path |
## Delete
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun files delete
```
| Argument/Option | Description |
| --------------- | ----------------------------------- |
| `FILE_ID` | ID of the file to delete (required) |
# gateway
Source: https://docs.vlm.run/cli/gateway
Run OpenAI-compatible OCR / VLM models on the VLM Run Gateway
The `vlmrun gateway` command (alias `vlmrun gw`) runs OpenAI-compatible OCR and
vision-language models on the [VLM Run Gateway](/gateway/introduction)
(`https://gateway.vlm.run/v1`), using the same `VLMRUN_API_KEY` as the rest of
the CLI. Unlike [`vlmrun chat`](/cli/chat), it is a raw passthrough: no Files
API upload, no Orion agent.
Every input is a **local file path** or an **http(s) URL**. URLs pass by
reference; local files are inlined as base64 `data:` URLs. The CLI picks the
content part from the media type:
| Media | Content part | Matches |
| -------- | -------------- | -------------------------------------------- |
| Document | `document_url` | `.pdf`, `.doc`, `.docx` |
| Image | `image_url` | any image type: `.jpg`, `.png`, `.webp`, ... |
| Video | `video_url` | any video type: `.mp4`, `.mov`, `.mkv`, ... |
| Other | `file_url` | everything else |
`gw chat` needs a file, a URL, or `-p/--prompt`. OCR models such as
`zai-org/glm-ocr` and `paddleocr/pp-ocrv6` reject text-only input, so use `-p`
alone only with chat models like `qwen/qwen3.5-0.8b`.
Model ids are the full `/` reported by `vlmrun gw models`. Short
aliases (for example `pp-ocrv6` for `paddleocr/pp-ocrv6`) also work.
## Health
Check gateway reachability. The command exits non-zero when the check fails.
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw health
```
## Models
List models, or detail one with runnable examples:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List every model with its methods
vlmrun gw models
# Methods, params, and example commands for one model
vlmrun gw models pp-ocrv6
# Raw JSON catalog
vlmrun gw models --json
```
The default method is marked `*`. `--json` returns the raw catalog records, and
adds a `commands` array on a detail view. See [Gateway Models](/gateway/models)
for the maintained catalog.
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
paddleocr/pp-ocrv6
task chat
methods ocr*, detect, text
inputs image_url, document_url, file_url, file
aliases pp-ocrv6
note document_url PDF (plain text per page)
examples
├── vlmrun gw chat doc.pdf -m paddleocr/pp-ocrv6 --method ocr
├── vlmrun gw chat doc.pdf -m paddleocr/pp-ocrv6 --method text
└── vlmrun gw chat doc.pdf -m paddleocr/pp-ocrv6 --method ocr --method-params '{"lang": "en", "score_threshold": 0.5}'
```
## Chat
Run an OCR / VLM model over one or more documents, images, or videos. Inputs can
be local paths, remote URLs, or a mix of both:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Parse a hosted document (PDF -> text/markdown)
vlmrun gw chat https://example.com/report.pdf -m zai-org/glm-ocr
# Parse a local document
vlmrun gw chat document.pdf -m zai-org/glm-ocr
# Several inputs at once, mixing local paths and URLs
vlmrun gw chat doc1.pdf https://example.com/doc2.pdf -m paddleocr/pp-ocrv6
# Prompt a model that supports text input
vlmrun gw chat image.jpg -p "describe this image" -m qwen/qwen3.5-0.8b
# Text-only, on a chat model that accepts it
vlmrun gw chat -p "What is OCR?" -m qwen/qwen3.5-0.8b
```
A run prints the resolved inputs, then the reply under a `Response` rule, with
the metering signals on the closing rule:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf \
-m paddleocr/pp-ocrv6 \
-e document_dpi=72
```
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Processing 1 input(s) (paddleocr/pp-ocrv6)
https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf
Response
{"object": "document.page.blocks", "items": [{"index": 0, "bbox_xywh": [0.3987, 0.0896, 0.1993, 0.0164], "poly_xy": [[0.3987, 0.0896], ...], "text": "UNITED STATES", "score": 0.9998}, ...]}
... one block per page ...
paddleocr/pp-ocrv6 · P:1280 / C:1171 / T:2451 toks · 1060 toks/s · 5 pages · 4.53 pages/s · 1s · $0.000247
```
`bbox_xywh` and `poly_xy` are both normalized to the page, not pixels. Methods
that return prose rather than regions (`markdown` on `zai-org/glm-ocr`, say) put
the text straight in the `` body with `format="markdown"`.
The footer reads model, prompt / completion / total tokens, throughput, pages,
pages per second, latency, and cost. Segments drop out when the gateway does not
report them: tokens and throughput without usage, pages for non-paginated
replies, cost when none is returned. Token counts, pages and cost are
reproducible for a given document and DPI; throughput and latency vary. See
[Pricing and Metering](/gateway/pricing) for the same signals.
### Methods
Each model exposes one or more methods (`ocr`, `detect`, `markdown`) with a
default; `vlmrun gw models ` lists them. `--method` and `--method-params`
map to the gateway's top-level `method` and `method_params` fields.
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat img.jpg -m pp-ocrv6 --method detect
vlmrun gw chat img.jpg -m pp-ocrv6 --method ocr \
--method-params '{"lang": "en", "score_threshold": 0.5}'
# Document markdown with DeepSeek OCR 2
vlmrun gw chat page.pdf -m deepseek-ocr-2 --method markdown
```
### Video input
Video files and video URLs are sent as `video_url` content parts. Control frame
sampling with the gateway's `video_fps` and `video_max_frames` fields, passed
through `-e`:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4 \
-p "Summarize what happens in this video." \
-m qwen/qwen3.5-0.8b \
-e video_fps=1.0 \
-e video_max_frames=8
```
See [Video Inputs](/gateway/multimodal-inputs#video-inputs) for the full set of
sampling knobs.
### Extra completion kwargs
Forward extra `chat.completions.create()` kwargs as repeatable `key=value` pairs
(values are JSON-parsed). Known keys go through as real kwargs; unknown ones
route via `extra_body` to become top-level gateway request fields.
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat document.pdf -m zai-org/glm-ocr -e temperature=0 -e max_tokens=4096
```
### Streaming
`gw chat` streams by default: tokens print live, one chunk per page for
multi-page documents. `-ns/--no-stream` waits for the full reply and renders it
in a bordered panel instead.
### JSON output vs JSON mode
Two different flags contain the word "json", and they act on two different
layers:
| Flag | Layer | Effect |
| ------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `-j, --json` | The CLI's own output | Replaces the human-readable output with a JSON envelope on stdout |
| `--json-mode` | The gateway request | Sends `response_format: {"type": "json_object"}`, asking the model to return one JSON object instead of the `` envelope |
`-j` replaces that output with an envelope of `model`, `content`, `latency_s`,
and `usage`, plus `pages` and `pages_per_sec` when the reply is paginated OCR
output:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf \
-m paddleocr/pp-ocrv6 -e document_dpi=72 -j
```
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"model": "paddleocr/pp-ocrv6",
"content": "",
"latency_s": 0.9522805213928223,
"usage": { "prompt_tokens": 1280, "completion_tokens": 1171, "total_tokens": 2451, "prompt_tokens_details": {...}, "cost": 0.000247 },
"pages": 5,
"pages_per_sec": 5.25
}
```
`usage` is the gateway's object passed through untouched: standard OpenAI
breakdown fields (often `null` for OCR models) plus the VLM Run `cost`
extension. Read `usage.cost` rather than recomputing from tokens.
Adding `--json-mode` gives JSON mode plus machine-readable metering, but the
result is **double-encoded**: `content` is a JSON *string* holding a JSON
document, so it needs a second parse (`jq -r .content | jq .`).
`--response-format` is the general form: `text`, `json_object` (alias `json`),
or a full object like `'{"type":"json_schema", ...}'`. `--json-mode` is exactly
`--response-format json_object`, and the two are mutually exclusive.
`json_schema` is a per-model capability (`capabilities.supports_json_schema`):
chat and routed models constrain generation to it, while OCR and detection
models return a 400 `capability_violation`. Use `json_object` there.
### Options
| Option | Short | Description |
| ------------------- | ----- | ------------------------------------------------------------------------------------------------- |
| `INPUTS...` | | Input file path(s) or http(s) URL(s): image, document, or video. Repeatable |
| `--model` | `-m` | Gateway model id, full `/` or alias (required) |
| `--prompt` | `-p` | Text prompt (only for models that support text input) |
| `--method` | `-M` | Model method, e.g. `ocr`, `detect`, `markdown`. Defaults to the model's `default_method` |
| `--method-params` | | JSON object of method arguments, e.g. `'{"lang": "en"}'` |
| `--json-mode` | | Enable JSON mode (`response_format json_object`). Mutually exclusive with `--response-format` |
| `--response-format` | | Constrain the output: `text`, `json_object`, or a JSON object like `'{"type":"json_schema",...}'` |
| `--extra` | `-e` | Extra `create()` kwarg as `key=value` (repeatable) |
| `--no-stream` | `-ns` | Disable streaming |
| `--json` | `-j` | Output raw JSON |
| `--timeout` | | Request timeout in seconds |
## Embed
Embed text, images, or video with a gateway embedding model. Each file and each
`-t/--text` becomes its own vector; `--join` combines them into one (at most one
file). File inputs must be images or video: use `-t` for text.
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Embed text
vlmrun gw embed -t "a blue parrot" -m qwen/qwen3-vl-embedding-2b
# Embed an image
vlmrun gw embed photo.jpg -m qwen/qwen3-vl-embedding-2b
# Embed an image and caption jointly
vlmrun gw embed photo.jpg -t "caption" --join -m qwen/qwen3-vl-embedding-2b
# Truncate vectors to a fixed number of dimensions
vlmrun gw embed -t "hi" -m qwen/qwen3-vl-embedding-2b --dimensions 64
# Output full vectors as JSON
vlmrun gw embed photo.jpg -m qwen/qwen3-vl-embedding-2b --json
```
Without `--json` the CLI prints one row per input with the vector's
dimensionality and first four components, under a
`model · T: tokens · s` footer.
`qwen/qwen3-vl-embedding-2b` returns 2048 dimensions by default;
`--dimensions` truncates that vector.
### Options
| Option | Short | Description |
| -------------- | ----- | ------------------------------------------------------ |
| `FILES...` | | Image/video file(s) to embed. Repeatable |
| `--model` | `-m` | Embedding model id (required) |
| `--text` | `-t` | Text to embed (repeatable) |
| `--join` | | Embed all inputs together as one vector (max one file) |
| `--dimensions` | `-d` | Truncate vectors to this many dimensions |
| `--json` | `-j` | Output raw JSON, including full vectors |
| `--timeout` | | Request timeout in seconds |
## Transcribe
Transcribe audio, or a video whose audio track is transcribed:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Transcribe a local file
vlmrun gw transcribe clip.mp3 -m nvidia/parakeet-tdt-0.6b-v3
# Transcribe a video's audio track
vlmrun gw transcribe clip.mp4 -m nvidia/parakeet-tdt-0.6b-v3
# Choose a response format (json, text, verbose_json, srt, vtt)
vlmrun gw transcribe clip.mp3 -m nvidia/parakeet-tdt-0.6b-v3 -f srt
# Provide a language hint
vlmrun gw transcribe clip.mp3 -m nvidia/parakeet-tdt-0.6b-v3 --language en
# Bias the transcript toward known proper nouns
vlmrun gw transcribe clip.mp3 -m nvidia/parakeet-tdt-0.6b-v3 -p "VLM Run, Orion"
# Transcribe a hosted URL instead of a local file
vlmrun gw transcribe --url https://example.com/a.mp3 -m nvidia/parakeet-tdt-0.6b-v3
```
Pass either a file or `--url`, not both. The transcript prints in a `Transcript`
panel with a `model · · s` footer.
### Options
| Option | Short | Description |
| ------------ | ----- | ------------------------------------------------------------------------------- |
| `FILE` | | Audio file, or a video whose audio track is transcribed |
| `--model` | `-m` | Transcription model id (required) |
| `--url` | | Hosted audio URL instead of a local file |
| `--format` | `-f` | Response format: `json`, `text`, `verbose_json`, `srt`, `vtt` (default: `json`) |
| `--language` | `-l` | ISO-639-1 language hint, e.g. `en` |
| `--prompt` | `-p` | Context to bias transcription (proper nouns) |
| `--json` | `-j` | Output raw JSON |
| `--timeout` | | Request timeout in seconds |
## Command Reference
| Command | Description |
| -------------------------------------- | -------------------------------------------------------------- |
| `vlmrun gw health` | Check gateway reachability (exits non-zero when unreachable) |
| `vlmrun gw models [MODEL]` | List models, or detail one model (`-j, --json` for raw output) |
| `vlmrun gw chat INPUTS... -m MODEL` | Run a model over one or more files or URLs |
| `vlmrun gw embed [FILES...] -m MODEL` | Embed text, images, or video |
| `vlmrun gw transcribe [FILE] -m MODEL` | Transcribe audio (or `--url`) |
## Configuration
| Variable | Purpose | Default |
| -------------------- | ------------------------------------------------------- | ---------------------------- |
| `VLMRUN_API_KEY` | Gateway authentication, shared with the rest of the CLI | none |
| `VLMRUN_GATEWAY_URL` | Gateway base URL | `https://gateway.vlm.run/v1` |
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
export VLMRUN_API_KEY="your-api-key"
export VLMRUN_GATEWAY_URL="https://gateway.vlm.run/v1"
```
The OpenAI-compatible surface is that base URL plus `/openai`, which is the
`base_url` to give an OpenAI SDK client directly. Gateway requests default to a
600 second timeout instead of the CLI's 120, since multi-page OCR runs long;
`--timeout` overrides it either way.
See the [Gateway documentation](/gateway/introduction) for the full model
catalog and [methods](/gateway/methods) reference, and
[`client.gateway`](/sdk-reference/components/gateway) for the Python SDK
equivalent.
# generate
Source: https://docs.vlm.run/cli/generate
Generate structured predictions from images and documents
The `vlmrun generate` command extracts structured data from files using pre-defined domains.
## Image Generation
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun generate image photo.jpg --domain document.invoice
```
| Argument/Option | Description |
| --------------- | ---------------------------------------------------------------- |
| `IMAGE` | Path to the input image (required) |
| `--domain` | Domain to use for generation, e.g. `document.invoice` (required) |
## Document Generation
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun generate document report.pdf --domain document.form
```
| Argument/Option | Description |
| --------------- | ------------------------------------------------------------- |
| `PATH` | Path to the document file (required) |
| `--domain` | Domain to use for generation, e.g. `document.form` (required) |
## Examples
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Extract invoice data from an image
vlmrun generate image invoice.jpg --domain document.invoice
# Parse a PDF form
vlmrun generate document form.pdf --domain document.form
```
For more flexible extraction with custom prompts and schemas, use [`vlmrun chat`](/cli/chat) with [skills](/cli/skills) instead of the `generate` command.
# Getting Started
Source: https://docs.vlm.run/cli/getting-started
Install and configure the VLM Run CLI
The VLM Run CLI (`vlmrun`) lets you interact with the VLM Run platform directly from your terminal: chat with Orion, generate structured predictions, manage files and skills, and more.
## Installation
The CLI ships with the VLM Run Python SDK. `typer`, `rich`, and `openai` are base dependencies, so a plain install gives you the full CLI, including [`vlmrun gw`](/cli/gateway):
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
pip install -U vlmrun
```
Verify the installation:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun --version
```
## Configuration
### Quick Setup
Initialize a config file and set your API key:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun config init
vlmrun config set --api-key "your-api-key"
```
Configuration is stored at `~/.vlmrun/config.toml`.
### Environment Variables
Alternatively, set your API key via environment variables:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
export VLMRUN_API_KEY="your-api-key"
export VLMRUN_BASE_URL="https://api.vlm.run/v1" # Optional
```
### Managing Config
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Show current configuration
vlmrun config show
# Set values
vlmrun config set --api-key "your-api-key"
vlmrun config set --base-url "https://api.vlm.run/v1"
# Unset values
vlmrun config unset --api-key
vlmrun config unset --base-url
# Re-initialize (overwrite existing)
vlmrun config init --force
```
## Global Options
All commands accept these global options:
| Option | Description |
| ----------------- | ----------------------------------- |
| `--api-key TEXT` | API key (overrides config/env) |
| `--base-url TEXT` | API base URL (overrides config/env) |
| `--debug` | Enable debug mode |
| `-v, --version` | Show version and exit |
| `--help` | Show help and exit |
## Commands
| Command | Description |
| --------------------------------- | -------------------------------------------------------------------------- |
| [`chat`](/cli/chat) | Chat with Orion to process images, videos, and documents |
| [`execute`](/cli/execute) | Execute agents with files, skills, and structured output |
| [`generate`](/cli/generate) | Generate structured predictions from files |
| [`executions`](/cli/executions) | List and retrieve agent execution results |
| [`predictions`](/cli/predictions) | List and retrieve prediction results |
| [`files`](/cli/files) | Upload, list, retrieve, and delete files |
| [`skills`](/cli/skills) | Create, list, lookup, update, and download skills |
| [`hub`](/cli/hub) | Browse available domains and JSON schemas |
| [`models`](/cli/models) | List supported models |
| [`gateway`](/cli/gateway) | Run OpenAI-compatible OCR / VLM models on the VLM Run Gateway (alias `gw`) |
| [`config`](/cli/config) | Manage CLI configuration |
## Shell Completion
Install tab completion for your shell:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Install completion
vlmrun --install-completion
# Show completion script (to customize)
vlmrun --show-completion
```
# hub
Source: https://docs.vlm.run/cli/hub
Browse available domains and JSON schemas
The `vlmrun hub` command lets you browse available domains and JSON schemas from the [VLM Run Hub](https://github.com/vlm-run/vlmrun-hub).
## List Domains
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List all domains
vlmrun hub list
# Filter by category
vlmrun hub list --domain document
vlmrun hub list --domain document.invoice
```
| Option | Description |
| ---------- | ------------------------------------------------------ |
| `--domain` | Filter domains (e.g. `document` or `document.invoice`) |
## Get Schema
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun hub schema document.invoice
```
| Argument | Description |
| -------- | ----------------------------------------------------- |
| `DOMAIN` | Domain identifier, e.g. `document.invoice` (required) |
## Hub Version
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun hub version
```
# models
Source: https://docs.vlm.run/cli/models
List supported models and domains
The `vlmrun models` command lists the models available on the VLM Run platform.
## List Models
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List all models
vlmrun models list
# Filter by domain
vlmrun models list --domain document
vlmrun models list --domain document.invoice
```
| Option | Description |
| ---------- | -------------------------------------------------------- |
| `--domain` | Filter by domain (e.g. `document` or `document.invoice`) |
Models are grouped by category (e.g. `document`, `image`, `video`, `audio`) and display the model name and domain.
# predictions
Source: https://docs.vlm.run/cli/predictions
List and retrieve prediction results
The `vlmrun predictions` command lets you list and retrieve prediction results.
## List Predictions
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List recent predictions (default: last 10)
vlmrun predictions list
# Filter by status
vlmrun predictions list --status completed
vlmrun predictions list --status failed
# Filter by date range
vlmrun predictions list --since 2026-03-01 --until 2026-03-30
# Pagination
vlmrun predictions list --limit 25 --skip 10
```
| Option | Description |
| ---------- | ----------------------------------------------------------------------------------- |
| `--limit` | Max items to return (default: `10`) |
| `--skip` | Number of items to skip (default: `0`) |
| `--status` | Filter by status: `enqueued`, `pending`, `running`, `completed`, `failed`, `paused` |
| `--since` | Show predictions since date (`YYYY-MM-DD`) |
| `--until` | Show predictions until date (`YYYY-MM-DD`) |
## Get Prediction
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get prediction details
vlmrun predictions get
# Wait for a prediction to complete (useful for batch jobs)
vlmrun predictions get --wait
# Wait with custom timeout
vlmrun predictions get --wait --timeout 120
```
| Argument/Option | Description |
| --------------- | ------------------------------------------------------ |
| `PREDICTION_ID` | ID of the prediction to retrieve (required) |
| `--wait` | Wait for prediction to complete (default: `--no-wait`) |
| `--timeout` | Timeout in seconds when waiting (default: `60`) |
# skills
Source: https://docs.vlm.run/cli/skills
Create, list, lookup, update, and download skills
The `vlmrun skills` command manages [skills](/skills/introduction) on the VLM Run platform.
## Upload
Zip and upload a local skill folder, then create the skill. The directory must contain a `SKILL.md` file — name and description are parsed from its YAML frontmatter automatically.
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload a skill folder
vlmrun skills upload ./my-skill
# Override name/description from frontmatter
vlmrun skills upload ./my-skill --name "invoice-extraction" --description "Extract invoice data"
```
| Argument/Option | Description |
| --------------------- | -------------------------------------------------- |
| `DIRECTORY` | Path to the skill folder (must contain `SKILL.md`) |
| `--name`, `-n` | Skill name (overrides frontmatter) |
| `--description`, `-d` | Skill description (overrides frontmatter) |
Archives are stored under `~/.vlmrun/skill_archives/`.
## Inline Skills via CLI
The `vlmrun chat` command uses inline skills by default when you pass a `--skill` directory. The CLI bundles the skill directory into a zip and sends it inline — no server upload required:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Use an inline skill (default behavior)
vlmrun chat "Extract data from this invoice" -i invoice.pdf --skill ./my-skill/
# To persist a skill on the server for reuse, upload it separately
vlmrun skills upload ./my-skill/
```
Inline skills via the CLI are ideal for local development and testing. For production use, upload the skill with `vlmrun skills upload` and reference it by name.
## Create
Create a skill from a prompt or pre-uploaded file:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# From a text prompt
vlmrun skills create --prompt "Extract invoice_id, date, and total_amount from invoices."
# From a prompt file
vlmrun skills create --prompt-file instructions.txt
# With a JSON schema
vlmrun skills create --prompt "Extract invoice data" --schema schema.json
# From a pre-uploaded file ID
vlmrun skills create --file-id --name "invoice-extraction"
```
| Option | Short | Description |
| --------------- | ----- | -------------------------------------- |
| `--prompt` | `-p` | Text prompt to auto-generate the skill |
| `--prompt-file` | `-f` | Read prompt from a file |
| `--schema` | | Path to a JSON schema file |
| `--file-id` | | Pre-uploaded skill zip file ID |
| `--name` | `-n` | Skill name (required for `--file-id`) |
| `--description` | `-d` | Skill description |
| `--json` | `-j` | Output raw JSON |
## List
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List skills (latest 25)
vlmrun skills list
# Show only latest version per name
vlmrun skills list --grouped
# Custom sort and limit
vlmrun skills list --limit 50 --order-by name --asc
# Output raw JSON
vlmrun skills list --json
```
| Option | Short | Description |
| ------------------ | ----- | ------------------------------------------------------------------------- |
| `--limit` | `-n` | Max items to return (default: `25`) |
| `--offset` | | Items to skip (default: `0`) |
| `--order-by` | | Sort field: `created_at`, `updated_at`, or `name` (default: `created_at`) |
| `--desc` / `--asc` | | Sort direction (default: `--desc`) |
| `--grouped` | `-g` | Show only latest version per skill name |
| `--json` | `-j` | Output raw JSON |
## Get
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get by name (latest version)
vlmrun skills get invoice-extraction
# Get a specific version
vlmrun skills get invoice-extraction --version 20260219-abc123
# Get by ID
vlmrun skills get
# Output raw JSON
vlmrun skills get invoice-extraction --json
```
| Argument/Option | Short | Description |
| --------------- | ----- | ----------------------------- |
| `NAME_OR_ID` | | Skill name or UUID (required) |
| `--version` | `-V` | Pin a specific version |
| `--json` | `-j` | Output raw JSON |
## Download
Download a skill zip and extract it locally:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Download to default location (~/.vlmrun/skills/)
vlmrun skills download invoice-extraction
# Download a specific version
vlmrun skills download invoice-extraction --version 20260219-abc123
# Download to a custom directory
vlmrun skills download invoice-extraction --output ./skills/
```
| Argument/Option | Short | Description |
| --------------- | ----- | ----------------------------------------------- |
| `NAME_OR_ID` | | Skill name or UUID (required) |
| `--version` | `-V` | Pin a specific version |
| `--output` | `-o` | Extract directory (default: `~/.vlmrun/skills`) |
# Error Codes
Source: https://docs.vlm.run/error-codes
List of error codes that you may encounter when using the API
Below is a list of error codes that you may encounter when using the API. If you encounter an error code that is not listed here, please reach out to us directly at [support](mailto:support@vlm.run).
| Error Code | Description | Solution |
| --------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| 400 - Bad Request | The request was invalid or cannot be processed | Check the request and try again |
| 401 - Unauthorized | The request requires user authentication | Check the API key `X-Api-Key` and try again |
| 402 - Payment Required | The account associated with the API key has consumed all available credits. | Adjust your billing settings or make a payment to resume service. |
| 403 - Forbidden | The server understood the request, but is refusing to fulfill it | Check the API key `X-Api-Key` and try again |
| 404 - Not Found | The requested resource could not be found | Check the request and try again |
| 429 - Too Many Requests | The user has sent too many requests in a given amount of time | Wait for a while and try again (see [Rate Limits](rate-limits.mdx)) |
| 500 - Internal Server Error | The server encountered an unexpected condition that prevented it from fulfilling the request | Contact [support](mailto:support@vlm.run) |
| 504 - Gateway Timeout | The server was acting as a gateway or proxy and did not receive a timely response from the upstream server | Wait for a while and try again |
# Health
Source: https://docs.vlm.run/gateway/api-reference/get-health
https://gateway.vlm.run/openapi.json GET /health
VLM Run Gateway liveness check
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import httpx
response = httpx.get("https://gateway.vlm.run/health")
print(response.status_code, response.json())
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const response = await fetch("https://gateway.vlm.run/health");
console.log(response.status, await response.json());
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw health
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/health
```
# Get Model
Source: https://docs.vlm.run/gateway/api-reference/get-model-by-id
https://gateway.vlm.run/openapi.json GET /v1/models/{model_id}
Retrieve a single VLM Run Gateway model and its capabilities
Send `Authorization: Bearer vlmrun` for anonymous access. See
[Rate Limits](/gateway/rate-limits) for per-tier quotas, which this route shares
with the inference routes. For the full catalog, use [List Models](/gateway/api-reference/get-models)
or see [Models](/gateway/models).
`model_id` is a path parameter, so any `/` in the id (e.g. `paddleocr/pp-ocrv6`)
must be percent-encoded as `%2F`. An unencoded slash is read as two path
segments and returns `404`.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import httpx
from urllib.parse import quote
model_id = quote("paddleocr/pp-ocrv6", safe="")
response = httpx.get(
f"https://gateway.vlm.run/v1/models/{model_id}",
headers={"Authorization": "Bearer "},
)
print(response.json())
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const modelId = encodeURIComponent("paddleocr/pp-ocrv6");
const response = await fetch(`https://gateway.vlm.run/v1/models/${modelId}`, {
headers: { Authorization: "Bearer " },
});
console.log(await response.json());
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw models pp-ocrv6
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl "https://gateway.vlm.run/v1/models/paddleocr%2Fpp-ocrv6" \
-H "Authorization: Bearer $VLMRUN_API_KEY"
```
## Response fields
This endpoint returns a flat, single-model detail object, richer than the
entries in [`GET /v1/openai/models`](/gateway/api-reference/get-models):
| Field | Description |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | Preferred public model id. |
| `name`, `description` | Human-readable catalog metadata. |
| `hf_model_id` | Upstream Hugging Face repo id, when the model is open-weight. |
| `task` | `chat`, `embed`, or `transcribe`. |
| `context_length` | Maximum context length, when defined. |
| `architecture` | `input_modalities`, `output_modalities`, and a summary `modality` string. |
| `capabilities` | Same shape as on [List Models](/gateway/api-reference/get-models#model-fields): max images/videos, accepted input types. |
| `aliases`, `methods` | Accepted request ids and supported `method` values. |
| `supported_parameters` | Request fields this model actually honors (a model may ignore fields like `temperature` or `top_p` that don't apply to it). |
| `pricing` | USD per 1M units: `prompt`, `completion`, `input_cache_read`, and `input_cache_write` per 1M tokens, and `image` per 1M image-token equivalents. See [Pricing](/gateway/pricing). |
`pricing` and `supported_parameters` are only available here, not on the
list endpoint. Fetch a single model's detail before you build cost
estimates or validate which request fields it will respect.
## Related
Full catalog in one call.
How to read the `pricing` object, and what each mode bills.
# List Models
Source: https://docs.vlm.run/gateway/api-reference/get-models
https://gateway.vlm.run/openapi.json GET /v1/openai/models
List available VLM Run Gateway models and their capabilities
Send `Authorization: Bearer vlmrun` for anonymous access. This route draws on
the same [rate-limit](/gateway/rate-limits) budget as chat completions. See
[Authentication](/gateway/authentication) for tier details.
For availability status and use-case guidance, see [Models](/gateway/models).
## Model fields
VLM Run Gateway model entries extend the OpenAI list-models shape with these fields:
| Field | Description |
| ----------------- | ---------------------------------------------------------------------------------------------------------------- |
| `id` | Preferred public model id (use this in `POST /chat/completions`). |
| `aliases` | All accepted request ids, including short and Hugging Face forms. |
| `methods` | Operations supported via the `method` request field. This list is authoritative: a method not listed is a `400`. |
| `default_method` | Applied when `method` is omitted. |
| `extra_body_help` | Example `method` / `method_params` payloads. |
| `capabilities` | Input limits and accepted content part types. |
| `task` | `chat`, `embed`, or `transcribe` (non-chat models use other endpoints). |
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="",
)
models = client.models.list()
for model in models.data:
caps = model.capabilities
print(f"{model.id}: {caps.supported_input_types}")
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const models = await client.models.list();
for (const model of models.data) {
console.log(model.id);
}
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw models
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/models \
-H "Authorization: Bearer $VLMRUN_API_KEY"
```
The OpenAI Python and Node.js SDKs may not surface VLM Run Gateway-specific
fields like `capabilities` or `methods` on the typed `Model` object. Parse
the raw response or use cURL when you need the full catalog metadata.
## Related
Full catalog with availability status and use-case guidance.
Send inference requests using model ids from this endpoint.
# Audio Transcriptions
Source: https://docs.vlm.run/gateway/api-reference/post-audio-transcriptions
https://gateway.vlm.run/openapi.json POST /v1/openai/audio/transcriptions
OpenAI-compatible speech-to-text transcription
Send `Authorization: Bearer vlmrun` for anonymous access. See
[Rate Limits](/gateway/rate-limits) for per-tier quotas, which this route shares
with chat completions. For available transcription models, see [Models](/gateway/models).
## Request parameters
The endpoint accepts a standard OpenAI multipart form:
| Field | Type | Default | Description |
| --------------------------- | ---------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `model` | `string` | required | Transcription model id, e.g. `nvidia/parakeet-tdt-0.6b-v3`. |
| `file` | file upload | optional | Audio bytes. Provide `file` or `url`. |
| `url` | `string` | optional | Hosted audio URL, as an alternative to a multipart `file` upload. |
| `language` | `string` | auto-detect | Language hint, e.g. `en`. The schema does not constrain the format; follows the OpenAI-compatible convention of an ISO-639-1 code. |
| `prompt` | `string` | `null` | Optional context to bias transcription (e.g. spelled-out proper nouns). |
| `response_format` | `string` | `json` | `json`, `verbose_json`, `text`, `srt`, or `vtt`, per the endpoint description. |
| `temperature` | `number` | `0.0` | Sampling temperature for the transcription model. |
| `timestamp_granularities[]` | array of strings | `null` | Accepted for OpenAI SDK compatibility; meaningful alongside `response_format: "verbose_json"` (`segment` and/or `word`). |
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="",
)
with open("sample.mp3", "rb") as audio_file:
response = client.audio.transcriptions.create(
model="",
file=audio_file,
)
print(response.text)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import fs from "fs";
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const response = await client.audio.transcriptions.create({
model: "",
file: fs.createReadStream("sample.mp3"),
});
console.log(response.text);
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw transcribe sample.mp3 -m
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/audio/transcriptions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-F "model=" \
-F "file=@sample.mp3"
```
## Hosted URL instead of a file upload
Pass `url` instead of `file` when the audio is already hosted, to avoid a
multipart upload:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import httpx
response = httpx.post(
"https://gateway.vlm.run/v1/openai/audio/transcriptions",
headers={"Authorization": "Bearer "},
data={
"model": "",
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/audio.transcription-summary/two_minute_rules.mp3",
},
)
print(response.json()["text"])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const form = new FormData();
form.set("model", "");
form.set(
"url",
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/audio.transcription-summary/two_minute_rules.mp3",
);
const response = await fetch(
"https://gateway.vlm.run/v1/openai/audio/transcriptions",
{
method: "POST",
headers: { Authorization: `Bearer ${process.env.VLMRUN_API_KEY}` },
body: form,
},
);
const { text } = await response.json();
console.log(text);
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw transcribe -m \
--url https://storage.googleapis.com/vlm-data-public-prod/hub/examples/audio.transcription-summary/two_minute_rules.mp3
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/audio/transcriptions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-F "model=" \
-F "url=https://storage.googleapis.com/vlm-data-public-prod/hub/examples/audio.transcription-summary/two_minute_rules.mp3"
```
## Related
Available transcription models.
Status codes and response bodies for failed requests.
# Chat Completions
Source: https://docs.vlm.run/gateway/api-reference/post-chat-completions
https://gateway.vlm.run/openapi.json POST /v1/openai/chat/completions
OpenAI-compatible chat completions for OCR, VQA, and document inference
Authentication is optional for the VLM Run Gateway (at the moment). See
[Authentication](/gateway/authentication) for tiers and
[Rate Limits](/gateway/rate-limits) for per-tier quotas.
## Models Supported
For available models and aliases, see [Models](/gateway/models).
Standard OpenAI sampling fields (`temperature`, `max_tokens`, `top_p`,
`frequency_penalty`, `presence_penalty`, `stop`, `n`) are accepted on every
request, but OCR and detection models generally ignore sampling params since
they are not free-form text generators. Check a model's
`supported_parameters` via [Get Model](/gateway/api-reference/get-model-by-id)
if a field you send does not appear to change the output.
`response_format` accepts `{"type":"json_object"}` for the JSON response object.
Omitting the field returns the text rendering. See
[Methods & Response Format](/gateway/methods#the-response_format-field).
## Gateway extensions
These fields are accepted at the top level of the request body (or via
`extra_body` in the OpenAI Python SDK):
| Field | Type | Default | Description |
| ---------------------- | --------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `method` | `string` | model default | Model-specific operation (`ocr`, `detect`, `markdown`, `text`, `parse_layout`, `chat`, …). It sets each page's payload kind: see [Document text output](#document-text-output). |
| `method_params` | `object` | `null` | Keyword arguments for the selected `method`. |
| `precision` | `integer` | `4` | Decimal places (1-8) on normalized `bbox_xywh` / `poly_xy` / `point_xy` and `score`. Ignored for markdown payloads. |
| `llm` | `string` | `null` | Optional LLM for post-processing structured model output. |
| `document_dpi` | `integer` | `72` | DPI when rasterizing PDF pages. |
| `document_max_pages` | `integer` | `128` | Maximum PDF pages per request. A longer PDF is a `400` naming the ranges that cover it. |
| `document_pages` | `array` | `null` (all) | Zero-indexed page indices and/or `[start, stop]` half-open ranges to read. Selected pages are re-numbered from 0. |
| `image_resolution` | `string` | `null` | Square resize preset (`224x224` … `768x768`). |
| `video_max_frames` | `integer` | `null` | Cap on frames sampled from `video_url` inputs. Used by `qwen/qwen3.5-0.8b` and `qwen/qwen3.8-27b`. |
| `video_fps` | `number` | `null` | Target frames per second to sample from `video_url` inputs. Used by `qwen/qwen3.5-0.8b` and `qwen/qwen3.8-27b`. |
| `video_resolution` | `string` | `null` | Resize preset for sampled video frames (`256x192` … `640x480`). Used by `qwen/qwen3.5-0.8b` and `qwen/qwen3.8-27b`. |
| `video_encoder` | `string` | `mosaic` | How a `video_url` is encoded into image(s) before dispatch (`mosaic`, `frames`, `keyframes`). See [Video encoding](#video-encoding). |
| `video_encoder_params` | `object` | `null` | Keyword arguments for `video_encoder` (e.g. `tile_cols`, `tile_rows`, `num_frames`). |
`video_max_frames`, `video_fps`, `video_resolution`, `video_encoder`, and
`video_encoder_params` apply when the request includes a `video_url` content
part on a video-capable model. See
[Video Inputs](/gateway/multimodal-inputs#video-inputs).
### Video encoding
When a request includes a `video_url`, the Gateway encodes it into image(s)
before dispatch so image-only models can read it. `video_encoder` selects the
strategy and defaults to **`mosaic`** (sampled keyframes tiled into a single grid
image); `frames` and `keyframes` are also available. Tune the chosen encoder with
`video_encoder_params` (for example `tile_cols`, `tile_rows`, `num_frames`).
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"model": "paddleocr/pp-ocrv6",
"video_encoder": "mosaic",
"video_encoder_params": { "tile_cols": 3, "tile_rows": 3, "num_frames": 9 },
"messages": [
{
"role": "user",
"content": [
{ "type": "text", "text": "What is happening in this video?" },
{
"type": "video_url",
"video_url": { "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4" }
}
]
}
]
}
```
An undecodable video on the chat-completions video path returns `400`
(`invalid_request_error`). See [Error Codes](/gateway/error-codes#invalid-video-400).
### Document text output
For `document_url` inputs, **text mode** (`response_format` omitted) always
returns one `` block per input PDF, wrapping one `` block per
rasterized page. `method` decides only each page's payload kind, declared on the
page as `format`:
| `method` | Page `format` | Page body |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | -------------------------------------------------------------------------------- |
| `markdown` (default on `zai-org/glm-ocr`, `deepseek-ai/deepseek-ocr-2`, `infly/infinity-parser2-flash`, `rednote-hilab/dots.mocr`, and `paddlepaddle/paddleocr-vl-1.6`), `text`, `ocr` on `rednote-hilab/dots.mocr` and `paddlepaddle/paddleocr-vl-1.6`, and `table`, `formula`, `chart` on `paddlepaddle/paddleocr-vl-1.6` | `markdown` | That page's text payload. |
| `ocr` on `paddleocr/pp-ocrv6`, `detect`, `parse_layout`, `parse_layout_only` | `json` | That page's block container, `{"object": "document.page.blocks", "items": […]}`. |
In JSON mode a page's `content` is `document.page.blocks` for **every** method: a
whole-page read is one block carrying `text`, and a region read is one block per
region. See [The document block record](/gateway/methods#document-block).
`` attributes are `file_name`?, `file_hash`?, `file_bytes`?,
`mimetype`, `num_pages`, `dpi`, `language`?;
`` attributes are `page_index`, `format`, `page_width`,
`page_height`, and `status`?. A page that failed keeps its slot as a self-closing
``. A single image returns the payload block alone, and a
multi-document request returns one `` block per input PDF in request
order.
**JSON mode is unaffected by `method`**: `response_format={"type":"json_object"}`
always returns `{"model": …, "method": …, "data": [{"object": "document", "pages": […]}]}`, with each page's `content` typed by the model and method. See
[JSON mode](/gateway/methods#json-mode) for full examples.
Each model exposes supported `method` values on
[`GET /v1/openai/models`](/gateway/api-reference/get-models). See
[Methods](/gateway/methods) for a per-model reference with examples.
## Content parts
Messages use the standard OpenAI multimodal shape. See
[Multimodal Inputs](/gateway/multimodal-inputs) for the full content part
reference (`text`, `image_url`, `document_url`) and
document-specific limits.
## Response extensions
Non-streaming responses follow the OpenAI chat completion shape with one
VLM Run Gateway extension:
| Field | Description |
| ------- | ---------------------------------------------------- |
| `usage` | Token counts; may include `usage.cost` for metering. |
## Streaming
Set `stream: true` on any **text-mode** chat request. The streamed result is
byte-identical to the non-streaming one. Document requests (`document_url`) emit SSE
chunks in document order: a `` open tag, one chunk per `` block in
ascending page order, then the matching `` close tag. Each ``
carries its own `format`, so a streamed page is self-describing. Other chat
requests (regular chat, single-image
OCR) buffer the full reply and re-chunk it into the same OpenAI
`chat.completion.chunk` contract (correct SSE, no time-to-first-token benefit yet).
Pass `stream_options={"include_usage": true}` to receive a terminal chunk carrying
aggregated `usage` (including `usage.cost`). A JSON `response_format` is always
served non-streamed. See [Flexible Document OCR](/gateway/guides/document-ocr#3-streaming-vs-non-streaming)
for the full SSE walkthrough.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="",
)
stream = client.chat.completions.create(
model="qwen/qwen3.5-0.8b",
stream=True,
stream_options={"include_usage": True},
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What is happening in this image?"},
{
"type": "image_url",
"image_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"
},
},
],
}
],
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="")
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const stream = await client.chat.completions.create({
model: "qwen/qwen3.5-0.8b",
stream: true,
stream_options: { include_usage: true },
messages: [
{
role: "user",
content: [
{ type: "text", text: "What is happening in this image?" },
{
type: "image_url",
image_url: {
url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg",
},
},
],
},
],
});
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta?.content;
if (delta) process.stdout.write(delta);
}
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg \
-p "What is happening in this image?" \
-m qwen/qwen3.5-0.8b
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/chat/completions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen3.5-0.8b",
"stream": true,
"stream_options": { "include_usage": true },
"messages": [
{
"role": "user",
"content": [
{ "type": "text", "text": "What is happening in this image?" },
{
"type": "image_url",
"image_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"
}
}
]
}
]
}'
```
### Document OCR
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="",
)
response = client.chat.completions.create(
model="paddlepaddle/paddleocr-vl-1.6",
messages=[
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
},
}
],
}
],
extra_body={
"method": "ocr",
"document_dpi": 72,
},
)
print(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const response = await client.chat.completions.create({
model: "paddlepaddle/paddleocr-vl-1.6",
messages: [
{
role: "user",
content: [
{
type: "document_url",
document_url: {
url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf",
},
},
],
},
],
method: "ocr",
document_dpi: 72,
});
console.log(response.choices[0].message.content);
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf \
-m paddlepaddle/paddleocr-vl-1.6 \
--method ocr \
-e document_dpi=72 \
--no-stream
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/chat/completions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "paddlepaddle/paddleocr-vl-1.6",
"method": "ocr",
"document_dpi": 72,
"messages": [
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
}
}
]
}
]
}'
```
### Stream a document page-by-page
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="",
)
stream = client.chat.completions.create(
model="paddlepaddle/paddleocr-vl-1.6",
stream=True,
messages=[
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf",
},
}
],
}
],
extra_body={
"method": "ocr",
"document_dpi": 72,
},
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="")
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const stream = await client.chat.completions.create({
model: "paddlepaddle/paddleocr-vl-1.6",
stream: true,
messages: [
{
role: "user",
content: [
{
type: "document_url",
document_url: {
url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf",
},
},
],
},
],
method: "ocr",
document_dpi: 72,
});
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta?.content;
if (delta) process.stdout.write(delta);
}
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf \
-m paddlepaddle/paddleocr-vl-1.6 \
--method ocr \
-e document_dpi=72
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/chat/completions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "paddlepaddle/paddleocr-vl-1.6",
"stream": true,
"method": "ocr",
"document_dpi": 72,
"messages": [
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
}
}
]
}
]
}'
```
## Errors and limits
* [Error codes](/gateway/error-codes): capability violations, invalid
documents, model not found, and sanitized 500 responses.
* [Rate limits](/gateway/rate-limits): 120 requests/min and 1000
requests/hr per IP (anonymous), or 240 requests/min and 10000
requests/hr per user (authenticated).
Every response includes an `x-request-id` header. You may send your own id on
the request; otherwise the VLM Run Gateway mints one. Sanitized `500` responses also
include the id as `error.request_id` in the JSON body.
## Related
Query the live model catalog and capabilities.
Full catalog with availability and use-case guidance.
First requests for VQA and document OCR.
Request knobs, page blocks, and streaming for PDFs.
Content part types, document limits, and format tradeoffs.
The same read pipeline as tools for MCP-aware agents.
# Embeddings
Source: https://docs.vlm.run/gateway/api-reference/post-embeddings
https://gateway.vlm.run/openapi.json POST /v1/openai/embeddings
OpenAI-compatible embeddings for text and image inputs
Send `Authorization: Bearer vlmrun` for anonymous access. See
[Rate Limits](/gateway/rate-limits) for per-tier quotas, which this route shares
with chat completions. For available embedding models, see [Models](/gateway/models).
## Request parameters
| Field | Type | Default | Description |
| ------------------------ | ----------------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------------- |
| `model` | `string` | required | Embedding model id, e.g. `qwen/qwen3-vl-embedding-2b`. |
| `input` | `string`, array of strings, or array of content parts | required | Text, or `text` / `image_url` / `video_url` content parts for multimodal embeddings. See below. |
| `encoding_format` | `string` | `float` | `float` or `base64`. |
| `dimensions` | `integer` | model default | Truncate the output vector to fewer dimensions, if the model supports it. |
| `truncate_prompt_tokens` | `integer` | `null` | Truncate long inputs to this many tokens instead of erroring. |
| `user` | `string` | `null` | Opaque end-user identifier, accepted for OpenAI SDK compatibility. |
## Text input
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="",
)
response = client.embeddings.create(
model="",
input="Extract a vector representation for this text.",
)
print(len(response.data[0].embedding))
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const response = await client.embeddings.create({
model: "",
input: "Extract a vector representation for this text.",
});
console.log(response.data[0].embedding.length);
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw embed -t "Extract a vector representation for this text." \
-m
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/embeddings \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "",
"input": "Extract a vector representation for this text."
}'
```
## Multimodal input
Vision-language embedding models such as `qwen/qwen3-vl-embedding-2b` also
accept `image_url` and `video_url` content parts in `input`, so you can embed
images or sampled video frames into the same vector space as text, for example
to build visual search or image-to-text retrieval. Confirm accepted input types
in a model's `capabilities.supported_input_types` on
[`GET /v1/openai/models`](/gateway/api-reference/get-models) before relying
on a modality, since not every embedding model on the catalog accepts images
or video.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="",
)
response = client.embeddings.create(
model="qwen/qwen3-vl-embedding-2b",
input=[
{
"type": "image_url",
"image_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"
},
}
],
)
print(len(response.data[0].embedding))
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const response = await client.embeddings.create({
model: "qwen/qwen3-vl-embedding-2b",
input: [
{
type: "image_url",
image_url: {
url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg",
},
},
],
});
console.log(response.data[0].embedding.length);
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw embed car.jpg -m qwen/qwen3-vl-embedding-2b
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/embeddings \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen3-vl-embedding-2b",
"input": [
{
"type": "image_url",
"image_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg"
}
}
]
}'
```
## Related
Available embedding models.
Content part types shared across chat and embeddings.
# Authentication
Source: https://docs.vlm.run/gateway/authentication
Authenticate requests to the VLM Run Gateway API
The VLM Run Gateway serves anonymous callers on a small free quota, keyed by
client IP. An API key raises the quota and attributes usage to your account.
See [Rate Limits](/gateway/rate-limits) for the per-tier limits.
The VLM Run Gateway accepts a standard `Authorization: Bearer ` header on
every endpoint. The header is optional: a call with no header is anonymous, so
`OpenAI(base_url=...)` works without an API key. `Bearer vlmrun` is the explicit
anonymous form for clients that need a non-empty key value.
## Tiers
| Tier | How to authenticate | Rate limit | Bucket key |
| ------------------ | --------------------------------------------------------------------------------------------- | ------------------------ | ------------ |
| Anonymous | Omit the header, or send `Bearer vlmrun` or an empty bearer value | 10/min · 30/hr · 100/day | Client IP |
| Authenticated | `Bearer ` from [app.vlm.run](https://app.vlm.run/dashboard/settings/api-keys) | 240/min · 10000/hr | Your account |
| First-party client | `Bearer vlmrun` plus a `vlmrun-` prefixed `User-Agent` | 20/sec | Client IP |
| M2M | An internal VLM Run service token | 20/sec | Client IP |
Authenticating raises your quota and changes attribution: anonymous requests are
capped per IP (so they can be shared with other users behind the same NAT or
proxy), while authenticated requests get the higher limit above, capped per
account.
## Tier resolution
The Gateway resolves your tier from the `Authorization` header in this order:
1. **No header** → anonymous, keyed by client IP. A deployment can set
`VLMRT_REQUIRE_AUTH_HEADER=1` to reject a missing header with `401`,
`{"detail": "Missing API Key"}`. The hosted Gateway does not set it.
2. **`Bearer vlmrun`, or an empty bearer value** → anonymous. `vlmrun` is the
documented sentinel, and the preferred form, because SDKs such as the OpenAI
Python client reject an empty `api_key` before the request leaves the
process.
3. **Valid VLM Run API key** → authenticated, as the owning user and
organization.
4. **Any other non-empty bearer token** → `403` (invalid API key). The
Gateway does not fall back to anonymous access for unrecognized tokens.
## Credits
An authenticated call is charged to the caller's organization. When that
organization has no credit left, the Gateway answers `402`:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"detail": "Access denied. Your credits are exhausted. Please move to Pay-As-You-Go to continue using the service."
}
```
The [routed models](/gateway/models/routed-models) carry the
`paid` access tier, so they need an organization with an active subscription or
a funded balance. Every GPU-served model is `public` and reachable anonymously.
Every endpoint documented under [API
Reference](/gateway/api-reference/post-chat-completions) accepts anonymous
traffic today: chat completions, models, embeddings, audio transcriptions, and
health. Authenticate with an API key for anything you plan to run in
production.
## Get an API key
1. Sign up at [app.vlm.run](https://app.vlm.run).
2. Copy your API key from
[Settings → API keys](https://app.vlm.run/dashboard/settings/api-keys).
3. Set it as an environment variable:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
export VLMRUN_API_KEY="your-api-key"
```
## Examples
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="", # omit, or use "vlmrun", for anonymous access
)
models = client.models.list()
for model in models.data:
print(model.id)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY, // or "vlmrun" for anonymous access
});
const models = await client.models.list();
for (const model of models.data) {
console.log(model.id);
}
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw models
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/models \
-H "Authorization: Bearer $VLMRUN_API_KEY"
```
An invalid (non-empty, unrecognized) bearer token returns `403` rather than
falling back to anonymous access. See [Error Codes](/gateway/error-codes) for
the response bodies.
## Related
Per-tier quotas and how to request a higher limit.
402, 403, and 429 response bodies.
# Error Codes
Source: https://docs.vlm.run/gateway/error-codes
HTTP errors returned by the VLM Run Gateway API
The VLM Run Gateway returns OpenAI-compatible error bodies for most failure modes.
Errors are wrapped in a `detail` object when returned by FastAPI:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"detail": {
"error": {
"message": "Human-readable description",
"type": "invalid_request_error",
"code": "capability_violation",
"param": "messages"
}
}
}
```
Rate limit errors use a top-level `error` object instead (see
[Rate Limits](/gateway/rate-limits)).
## HTTP status codes
| Status | When | `error.type` | `error.code` |
| ------ | ------------------------------------------------- | -------------------------------- | ---------------------- |
| `400` | Capability mismatch (wrong media type for model) | `invalid_request_error` | `capability_violation` |
| `400` | Invalid or unreadable PDF | `invalid_request_error` | `invalid_document` |
| `400` | Undecodable image or video | `invalid_request_error` | `invalid_media` |
| `402` | The organization's credits are exhausted | Plain detail | n/a |
| `403` | Invalid API key | Plain `"Invalid API Key"` detail | n/a |
| `404` | Unknown model, or unknown completion id | `model_not_found` | n/a |
| `422` | Request validation failure | FastAPI validation error | n/a |
| `429` | Rate limit exceeded, or the request queue is full | `rate_limit_exceeded` | n/a |
| `500` | Uncaught server error | `internal_server_error` | `internal_error` |
| `503` | The model deployment is not serving | `service_unavailable` | `model_unavailable` |
| `504` | Model did not respond within the dispatch window | `gateway_timeout` | `inference_timeout` |
## Common errors
### Capability violation (`400`)
Returned when the request content does not match the model's declared
capabilities. For example, sending a `document_url` to a model that only
accepts images, or sending multiple images to a single-image OCR model.
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"detail": {
"error": {
"message": "Model 'paddleocr/pp-ocrv6' does not accept document_url together with image_url parts.",
"type": "invalid_request_error",
"code": "capability_violation",
"param": "messages.content.document_url"
}
}
}
```
Check each model's capabilities on
[`GET /v1/openai/models`](/gateway/api-reference/get-models) or the
[Models](/gateway/models) page before sending a request.
### Invalid document (`400`)
Returned when a `document_url` points to a file that is not a
valid PDF, exceeds size limits, or cannot be decoded.
### Invalid media (`400`)
Returned when an `image_url` or a `video_url` points to a file the Gateway
cannot decode: an unsupported container, a corrupt stream, a URL that does not
resolve to media, or malformed base64. The Gateway returns a `400` with
`error.type = "invalid_request_error"` and `error.code = "invalid_media"` rather
than a `500`. `error.param` names the offending content part.
### Credits exhausted (`402`)
Returned when an authenticated caller's organization has no credit left.
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"detail": "Access denied. Your credits are exhausted. Please move to Pay-As-You-Go to continue using the service."
}
```
### Model not found (`404`)
Returned when the `model` field does not match any registered model or alias.
List available models with [`GET /v1/openai/models`](/gateway/api-reference/get-models).
### Rate limit exceeded (`429`)
Returned when a per-tier request quota is exhausted. Every public OpenAI route
shares one budget per caller. See [Rate Limits](/gateway/rate-limits) for bucket
sizes and tier attribution.
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"error": {
"type": "rate_limit_exceeded",
"message": "Rate limit exceeded. Get a free API key at https://app.vlm.run/dashboard/settings/api-keys for higher limits.",
"tier": "anonymous"
}
}
```
Anonymous callers receive the message above and an `X-VLMRun-Upgrade` header.
Authenticated callers receive `Rate limit exceeded: `. Every `429`
carries `Retry-After`, holding the longest exhausted window, so one wait of that
length is enough. Retry after it elapses.
A full request queue answers `429` too, with `Retry-After: 60`. The queue holds
128 waiting requests. That is capacity rather than quota, so the same call
succeeds on retry.
### Model unavailable (`503`)
Returned when the model's deployment is not serving traffic, for example during
a cold start. `Retry-After` carries the suggested wait, 30 seconds by default.
This is distinct from the [`504`](#inference-timeout-504), which means the model
accepted the request and then ran past the dispatch window.
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"detail": {
"error": {
"message": "Model 'zai-org/glm-ocr' is temporarily unavailable. Please retry shortly.",
"type": "service_unavailable",
"code": "model_unavailable"
}
}
}
```
### Internal server error (`500`)
Uncaught exceptions return a sanitized body with no stack traces or internal
paths. Every response includes an `x-request-id` header. On sanitized `500`
errors, the same id is also returned as `error.request_id` in the JSON body.
Include either value when contacting [support@vlm.run](mailto:support@vlm.run).
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"error": {
"message": "Internal server error. Please retry; if the problem persists, contact support with the request id.",
"type": "internal_server_error",
"code": "internal_error",
"request_id": "550e8400e29b41d4a716446655440000"
}
}
```
You may send an `x-request-id` request header to supply your own correlation
id; the Gateway echoes it on the response. When the header is omitted, the
Gateway mints a new id.
### Inference timeout (`504`)
Returned when the model does not return a response inside the Gateway's
dispatch window. The Gateway aborts the request after 270 seconds by default.
That window is configurable per deployment, is subject to change, and is not a
documented SLA.
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"detail": {
"error": {
"message": "Model did not respond within 270s and was aborted. Please retry shortly.",
"type": "gateway_timeout",
"code": "inference_timeout",
"param": null
}
}
}
```
The non-streaming path returns no partial result when the Gateway aborts the
request. A timeout can happen when a model deployment is not yet serving
traffic or is under heavy load.
To reduce the chance of a timeout:
* Retry after a short delay with exponential backoff.
* Keep your client-side timeout above the Gateway's dispatch window. An OpenAI
SDK client with a low `timeout=` value can fail earlier with its own error.
* Chunk the workload so each request carries a smaller payload.
* Use `stream=true` so tokens arrive as the model produces them.
Capture the `x-request-id` response header and include it when contacting
[support@vlm.run](mailto:support@vlm.run).
## Getting help
When reporting an issue, include:
* The `x-request-id` response header (or `error.request_id` on sanitized `500`
responses)
* The HTTP status code and response body
* The `model` value and content part types you sent
See also [Rate Limits](/gateway/rate-limits) and the
[Chat Completions API reference](/gateway/api-reference/post-chat-completions)
for retry-relevant request parameters.
# FAQ
Source: https://docs.vlm.run/gateway/faq
Frequently asked questions about the VLM Run Gateway
An OpenAI-compatible API for visual AI: OCR, VQA, document extraction, and text detection,
all behind one base URL and one `Authorization` header. See [Introduction](/gateway/introduction)
for the full rationale.
General-purpose routers are built for text and carry lots of LLMs, but only a small slice of
visual workloads. The VLM Run Gateway is built for OCR and VQA, and exposes each supported
model through the same `chat.completions` shape so switching models is a one-line change.
Yes. Pass a PDF as a `document_url` content part and the Gateway decodes it, fans out per-page
inference, and returns the assembled pages, concatenated or streamed, through the same chat
completions response. In text mode that is one `` block per PDF, wrapping one
`` block per page; `method` decides each page's body, and the page declares it in
`format`. Pass multiple `document_url` parts to get one block per input. See
[Flexible Document OCR](/gateway/guides/document-ocr) for the full walkthrough, including the
`method` and `document_dpi` knobs for tuning cost, latency, and accuracy per request.
Not on the Gateway. `response_format={"type":"json_object"}` returns the
Gateway's own response object.
For extraction against a schema you define, use the VLM Run API, which supports
[structured responses](/capabilities/structured-responses),
[custom schemas](/capabilities/custom-schemas), and the prebuilt schemas in the
[VLM Run Hub](/hub). See
[Methods & Response Format](/gateway/methods#the-response_format-field) for the
Gateway's full contract.
Yes, behind the same base URL and API key as chat completions. See
[Embeddings](/gateway/api-reference/post-embeddings) and
[Audio Transcriptions](/gateway/api-reference/post-audio-transcriptions)
for request schemas, or [Models](/gateway/models#embeddings-and-transcription)
for the models that back each endpoint.
Every model bills per token, at the rates published in the model catalog. Read
`usage.cost` on a chat completion for the exact metered charge of that request. See
[Pricing](/gateway/pricing) and [Rate Limits](/gateway/rate-limits).
Not to start. Every endpoint accepts anonymous access today, rate limited per IP. Authenticate
with an API key from [app.vlm.run](https://app.vlm.run) to get usage attributed to your account
instead of a shared IP bucket, which matters once you're building something you plan to run in
production. See [Authentication](/gateway/authentication).
The Gateway is in **alpha**: rate-limited, and the model catalog is intentionally small.
It already returns a live model catalog, per-request `usage.cost` for metering, `x-request-id`
for tracing, and sanitized error responses. Authenticate with an API key if you're building
something you plan to run in production, since alpha limits are subject to change.
Retry after a short delay with backoff. Batch or chunk the workload per request, keep your client timeout
above the Gateway window, or use `stream=true`. See
[Inference timeout (504)](/gateway/error-codes#inference-timeout-504) for the response
contract and more guidance.
Use `qwen/qwen3.5-0.8b` for VQA over images or video. It does not accept
`document_url`, so use an OCR model for PDFs.
Use `paddlepaddle/paddleocr-vl-1.6` for general OCR, or `paddleocr/pp-ocrv6`
with `method: "ocr"` or `method: "detect"` when you need per-line bounding
polygons. `zai-org/glm-ocr` serves `markdown`.
`deepseek-ai/deepseek-ocr-2` defaults to `markdown` and adds `grounding_ocr`
to locate a phrase on the page.
`rednote-hilab/dots.mocr` also defaults to `markdown` and adds
`parse_layout` for structured document layout, plus `parse_layout_only` and
`ocr`. For layout regions **without** text recognition, use
`rednote-hilab/dots.mocr` with `parse_layout_only`.
`paddlepaddle/paddleocr-vl-1.6` defaults to `markdown`, also serves `ocr`,
and adds `table`, `formula`, and `chart` when you need a page's tables,
equations, or charts as text.
For documents, treat model, method, and `document_dpi` as one tunable unit:
start with the cheapest combination that clears your accuracy bar on your own
data, and only pay for more where it actually moves the numbers.
## Related
Why the VLM Run Gateway exists and what it's built to help you do.
Full catalog with availability, methods, and accepted inputs.
Per-model method reference and request parameters.
First requests for VQA and document OCR.
# Flexible Document OCR
Source: https://docs.vlm.run/gateway/guides/document-ocr
End-to-end recipe for OCR and document extraction on the VLM Run Gateway
This guide walks through running a document (PDF) through the VLM Run Gateway,
from picking a model to handling the response. Document requests fan out one
page at a time, and each model applies its own default `method`.
For a quick copy-paste example, see [Quickstart](/gateway/quickstart#document-and-image-ocr).
For the cost/latency/accuracy rationale behind picking a model, method, and
DPI, see [Why the Gateway](/gateway/introduction#why-the-gateway).
## 1. Pick a model
Start with `paddlepaddle/paddleocr-vl-1.6` for general-purpose PDF OCR, and
switch only when you need a method it does not serve. Every model below defaults
to `markdown`, except `paddleocr/pp-ocrv6`, which defaults to `ocr`.
| Model | Reach for it when |
| -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| [`paddlepaddle/paddleocr-vl-1.6`](/gateway/models/paddlepaddle-paddleocr-vl-1.6) | General-purpose page reads (`markdown`, `ocr`), plus `table`, `formula`, and `chart` for structured content. |
| [`zai-org/glm-ocr`](/gateway/models/zai-org-glm-ocr) | Markdown-only page reads. |
| [`rednote-hilab/dots.mocr`](/gateway/models/rednote-hilab-dots-mocr) | Layout parsing (`parse_layout`), or layout boxes with no transcribed text (`parse_layout_only`). |
| [`deepseek-ai/deepseek-ocr-2`](/gateway/models/deepseek-ai-deepseek-ocr-2) | Locating a phrase on the page (`grounding_ocr`). |
| [`paddleocr/pp-ocrv6`](/gateway/models/paddleocr-pp-ocrv6) | Per-line geometry (`ocr`, `detect`), or plain text (`text`). |
| [`baidu/unlimited-ocr`](/gateway/models/baidu-unlimited-ocr) | Long documents read in sliding windows (`multi_page`). |
| [`infly/infinity-parser2-flash`](/gateway/models/infly-infinity-parser2-flash) | A compact Markdown reader. |
See [Models](/gateway/models#document-and-image-ocr) for the full catalog and
[Methods](/gateway/methods) for every method and its parameters.
## 2. Send the document
Pass the PDF as a `document_url` content part. Prefer a hosted URL over a
base64 data URI. See [Multimodal Inputs](/gateway/multimodal-inputs) for size
limits and [Method Parameters](/gateway/methods#method-parameters) for
`document_dpi` and other knobs.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key="",
)
response = client.chat.completions.create(
model="paddlepaddle/paddleocr-vl-1.6",
messages=[
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/sample-invoice.pdf"},
}
],
}
],
extra_body={"method": "ocr", "document_dpi": 72},
)
print(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const response = await client.chat.completions.create({
model: "paddlepaddle/paddleocr-vl-1.6",
messages: [
{
role: "user",
content: [
{
type: "document_url",
document_url: {
url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/sample-invoice.pdf",
},
},
],
},
],
method: "ocr",
document_dpi: 72,
});
console.log(response.choices[0].message.content);
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/sample-invoice.pdf \
-m paddlepaddle/paddleocr-vl-1.6 \
--method ocr \
-e document_dpi=72 \
--no-stream
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/chat/completions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "paddlepaddle/paddleocr-vl-1.6",
"method": "ocr",
"document_dpi": 72,
"messages": [
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/sample-invoice.pdf"
}
}
]
}
]
}'
```
### Request knobs
| Field | Default | Notes |
| -------------------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `document_dpi` | `72` | Rasterization DPI per page. See [Method Parameters](/gateway/methods#method-parameters). |
| `document_max_pages` | `128` | Pages one request may read. A longer PDF is a `400` naming the ranges that cover it. See [Multimodal Inputs](/gateway/multimodal-inputs#document-inputs). |
| `document_pages` | all | Which 0-indexed pages to read: page indices and/or `[start, stop]` half-open ranges. Only those pages are rasterized, and they are re-numbered from 0. |
| `method` | model default | Per-page backend operation. It sets each page's payload kind (`format="markdown"` or `format="json"`), not the page wrapper. See [Methods](/gateway/methods#methods-by-model). |
| `response_format` | omitted | Omitted (or `{"type":"text"}`) returns the `` / `` text blocks; `{"type":"json_object"}` returns the JSON object. See [Parse the response](#4-parse-the-response). |
| `precision` | `4` | Decimal places on normalized `bbox_xywh` / `poly_xy` / `point_xy` and `score`. Ignored for markdown payloads. |
| `stream` | `false` | When true, emits ordered SSE page blocks. Ignored when a JSON `response_format` is requested. See [Streaming vs non-streaming](#3-streaming-vs-non-streaming). |
### Document rasterization
When the request includes a `document_url` part, `document_dpi` controls the
rasterization DPI for each page before OCR. Set it at the top level or inside
`method_params` (either location works; if both are present, `method_params`
wins).
| DPI | Tradeoff |
| -------------- | ----------------------------------------------------------------------------------- |
| `72` (default) | Fast and cheap; fine for clean, standard-size text. |
| `150` | Good balance of legibility and per-page inference cost for denser pages. |
| `300`+ | Preserves fine print and small text, at higher inference cost and latency per page. |
Start at the default and only raise `document_dpi` if you see missed text on
dense or small-font pages. Page count and file-size limits are documented under
[Multimodal Inputs](/gateway/multimodal-inputs#document-inputs).
## 3. Streaming vs non-streaming
In **text mode**, `stream: true` is honored for every chat request, and the streamed
result is **byte-identical** to the non-streaming one. Document requests emit one SSE
chunk per completed page block, in ascending page order. Other chat requests (regular
chat, single-image OCR) buffer the full reply and re-chunk it into the same OpenAI SSE
contract (no time-to-first-token benefit yet). A JSON `response_format` is always
served non-streamed, since a single JSON object cannot be assembled from per-page
deltas.
| Mode | When to use |
| ------------------------------ | ------------------------------------------------------------------------------------- |
| **Non-streaming** (default) | Short documents, batch pipelines, or when you need the full result before continuing. |
| **Streaming** (`stream: true`) | Long documents where you want to render or process pages as they finish. |
Set `stream: true` to receive each page over SSE as it completes, in ascending
page order, instead of waiting for the entire document:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
stream = client.chat.completions.create(
model="paddlepaddle/paddleocr-vl-1.6",
stream=True,
messages=[
{
"role": "user",
"content": [
{"type": "document_url", "document_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"}}
],
}
],
extra_body={"method": "ocr", "document_dpi": 72},
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="")
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const stream = await client.chat.completions.create({
model: "paddlepaddle/paddleocr-vl-1.6",
stream: true,
messages: [
{
role: "user",
content: [
{
type: "document_url",
document_url: {
url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf",
},
},
],
},
],
method: "ocr",
document_dpi: 72,
});
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta?.content;
if (delta) process.stdout.write(delta);
}
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf \
-m paddlepaddle/paddleocr-vl-1.6 \
--method ocr \
-e document_dpi=72
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/chat/completions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "paddlepaddle/paddleocr-vl-1.6",
"stream": true,
"method": "ocr",
"document_dpi": 72,
"messages": [
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
}
}
]
}
]
}'
```
Each `delta.content` value is one component of the document structure: the
`` open tag, then one `` block per page, then the close tag. Each
`` carries its own `format` attribute, so a streamed page is
self-describing before the client has buffered the header.
## 4. Parse the response
`response_format` picks text mode or JSON mode, and `method` picks each page's
payload kind. Both modes carry the **same** per-page content; JSON mode is easiest
to consume programmatically. See
[Methods & Response Format](/gateway/methods#the-response_format-field) for the full
contract.
### Text mode (default)
Every document method returns the same wrapper: one `` element per input
PDF, carrying its metadata as attributes, wrapping one `` block per
rasterized page. `method` decides only the page body, and each page declares it in
`format`: `markdown` for a Markdown body (`markdown`, `text`, and `ocr` on
`paddlepaddle/paddleocr-vl-1.6` and `rednote-hilab/dots.mocr`) and `json` for a
region payload (`ocr` on `paddleocr/pp-ocrv6`, `detect`, `parse_layout`,
`parse_layout_only`).
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# ACME Corp, Invoice #4471
| Item | Qty | Price |
| ---- | --- | ------ |
| Widget | 3 | $12.00 |
```
* A failed page is **self-closing**, with `status="error"` and no body, so page numbering stays intact.
* A **single image** (not a PDF) returns the payload block alone, with no wrapper.
* Several images return one `` block each.
### JSON mode
JSON mode is **the same for every method**. Add
`response_format={"type":"json_object"}` to get one object, naming what produced
it. `data` holds **one document entry per input PDF**; each entry lists its pages,
and a failed page carries `"status": "error"` with no `content`:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"model": "paddlepaddle/paddleocr-vl-1.6",
"method": "ocr",
"data": [
{
"object": "document",
"file_name": "invoice.pdf",
"file_hash": "sha256:…",
"file_bytes": 120000,
"mimetype": "application/pdf",
"num_pages": 2,
"dpi": 72,
"pages": [
{
"object": "document.page",
"page_index": 0,
"page_width": 612,
"page_height": 792,
"content": {
"object": "document.page.blocks",
"items": [
{ "index": 0, "text": "ACME Corp\nInvoice #4471\n\nItem Qty Price\nWidget 3 $12.00" }
]
}
},
{
"object": "document.page",
"page_index": 1,
"page_width": 612,
"page_height": 792,
"status": "error"
}
]
}
]
}
```
Every page's `content` is `document.page.blocks`, whatever method read it. A
whole-page method (`markdown`, `text`, and `ocr` on
`paddlepaddle/paddleocr-vl-1.6` and `rednote-hilab/dots.mocr`) fills it with
**one** block that carries `text` and no geometry, as above. A region method
(`ocr` on `paddleocr/pp-ocrv6`, `detect`, `parse_layout`, `parse_layout_only`)
fills it with one block per region, each with `bbox_xywh` and, where the method
reads the region, `text`:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"object": "document.page.blocks",
"items": [
{ "index": 0, "bbox_xywh": [0.0332, 0.0138, 0.1719, 0.0331], "text": "Invoice", "score": 0.998 }
]
}
```
`items` is `[]` when nothing is found, and it is never a bare array. See
[The document block record](/gateway/methods#document-block) for every key, and
the per-model pages on [Models](/gateway/models#document-and-image-ocr) for
exact examples.
This is the **only** JSON shape the Gateway returns, so you cannot extract a
custom document schema here. For schema-driven extraction, use the VLM Run
API's [structured responses](/capabilities/structured-responses) and
[custom schemas](/capabilities/custom-schemas).
### Multiple documents in one request
When a request includes more than one `document_url` (or `file_url`) content
part, each PDF gets its **own** entry: they are never merged. In text mode that is
one `` block per input, concatenated in request order; in JSON mode
`data` is a list of document entries in the same order.
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Page one text
Section A
Section B
```
Page indices restart at 0 for each document, in both modes.
### Element attributes
| Element | Attribute | Description |
| ------------ | ---------------------------- | ------------------------------------------------------------------------------------------ |
| `` | `file_name` | Original file name of the source document. Omitted for `data:` URI inputs. |
| `` | `file_hash` | SHA-256 of the source bytes, for correlation. |
| `` | `file_bytes` | Size of the source document in bytes. |
| `` | `mimetype` | MIME type of the source (e.g. `application/pdf`). |
| `` | `num_pages` | Number of pages in the document. |
| `` | `dpi` | Rasterization DPI applied to every page. |
| `` | `language` | Comma-separated ISO-639 codes in confidence order. Present only when the read reports one. |
| `` | `page_index` | Zero-based page index within its parent document. |
| `` | `format` | Payload kind of this page's body: `markdown` or `json`. |
| `` | `page_width` / `page_height` | Rasterized page dimensions in pixels at the document's DPI. |
| `` | `status` | `error` on a page that failed. The element is then self-closing. |
The JSON-mode keys carry the same names, so the two modes map key for key.
### Consuming each shape
**Non-streaming:** the blocks are concatenated in order inside
`choices[0].message.content`.
**Streaming:** the VLM Run Gateway emits OpenAI-style SSE chunks in document order,
byte-identical to the non-streaming reply: the `` open tag, one chunk per
`` block, then the matching `` close tag. The next document
follows after.
To consume text mode, split on the `` / `` open and close tags,
then read each page body according to its `format`. To skip parsing entirely, ask
for JSON mode.
## 5. Track cost and handle errors
**Non-streaming:** read `response.usage.cost` for the metered charge of the
request, in USD.
**Streaming:** pass `stream_options={"include_usage": true}` so the terminal SSE
event carries `usage` (including `usage.cost`) on the chunk instead of
`delta.content`. Check for it before treating every chunk as page content:
```python Python [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
stream = client.chat.completions.create(
model="paddlepaddle/paddleocr-vl-1.6",
stream=True,
stream_options={"include_usage": True},
messages=[
{
"role": "user",
"content": [
{"type": "document_url", "document_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"}}
],
}
],
extra_body={"method": "ocr", "document_dpi": 72},
)
usage = None
pages: list[str] = []
for chunk in stream:
choice = chunk.choices[0]
if choice.delta.content:
pages.append(choice.delta.content)
if chunk.usage:
usage = chunk.usage
print(f"Cost: {usage.cost if usage else 'n/a'}")
```
```typescript Node.js [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const stream = await client.chat.completions.create({
model: "paddlepaddle/paddleocr-vl-1.6",
stream: true,
stream_options: { include_usage: true },
messages: [
{
role: "user",
content: [
{
type: "document_url",
document_url: {
url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf",
},
},
],
},
],
method: "ocr",
document_dpi: 72,
});
let usage;
const pages: string[] = [];
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta?.content;
if (delta) pages.push(delta);
if (chunk.usage) usage = chunk.usage;
}
console.log(`Cost: ${usage?.cost ?? "n/a"}`);
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf \
-m paddlepaddle/paddleocr-vl-1.6 \
--method ocr \
-e document_dpi=72 \
--json
```
```bash cURL [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/chat/completions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "paddlepaddle/paddleocr-vl-1.6",
"stream": true,
"stream_options": { "include_usage": true },
"method": "ocr",
"document_dpi": 72,
"messages": [
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
}
}
]
}
]
}'
```
For both modes:
* Retry `429`/`500` with backoff; do not retry `400` capability violations.
See [Error Codes](/gateway/error-codes).
* Keep the `x-request-id` response header if you need to contact support.
## Related
Content part types, document limits, and URL vs base64 tradeoffs.
Per-model `method` and `method_params` reference.
Status codes, error bodies, and retry guidance.
Document and image OCR model catalog.
# Introduction
Source: https://docs.vlm.run/gateway/introduction
One API for every visual model
The VLM Run Gateway is an **OpenAI-compatible** API for visual intelligence: OCR, VQA,
detection, embeddings, and transcription behind an interface you already know. Point
`base_url` at `https://gateway.vlm.run/v1/openai` and your existing SDK calls keep working.
## Why the Gateway
General-purpose model routers are built for text LLMs, so they cover only a thin slice of
visual workloads. The Gateway is built for vision models:
* **One API, many models.** Every model speaks OpenAI's own request shapes, on one
`base_url` and one API key. See [Models](/gateway/models).
* **Document routing built in.** Multi-page PDFs are rasterized and fanned out per page
for you, with no splitting or stitching.
* **Method and DPI routing.** `method` picks the operation (`ocr`, `detect`, `markdown`, …);
`document_dpi` tunes how much page detail the model sees.
* **Operational signals included.** A live
[model catalog](/gateway/api-reference/get-models), per-request `usage.cost`, and
`x-request-id` on every response.
* **Drop-in for agents.** The [MCP server](/gateway/mcp-server) exposes file-reading tools
to any MCP-aware framework.
Treat model, method, and DPI as one tunable unit: start with the cheapest combination that
clears your accuracy bar, and pay for more only where it moves the numbers.
## Next steps
Prerequisites, plus first VQA and document OCR requests.
Request knobs, page blocks, and streaming for PDFs.
Catalog, capabilities, and model selection.
Method and `method_params` reference.
Per-token rates, and per-request `usage.cost`.
Connect any MCP-aware agent (Pydantic AI, LangChain, Mastra, Claude Code).
# MCP Server
Source: https://docs.vlm.run/gateway/mcp-server
Connect any MCP-aware agent to the VLM Run Gateway over Streamable HTTP
The VLM Run Gateway exposes a **Model Context Protocol (MCP)** server, so any
MCP-aware agent (Pydantic AI, LangChain, Mastra, OpenAI Agents SDK, Claude Code)
can read documents, images, audio, and video through the same pipeline as the
REST API.
It is a [FastMCP](https://gofastmcp.com) server over **Streamable HTTP**,
served **statelessly**: the server issues no session id, every request carries
everything needed to serve it, and any replica answers any request. Calls
re-enter the same ingress as
[`chat/completions`](/gateway/api-reference/post-chat-completions), so
authentication, billing, and metering behave identically.
| Property | Value |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Server name | `vlmrun` |
| Endpoint | `POST https://gateway.vlm.run/mcp` |
| Transport | Streamable HTTP (FastMCP), stateless: no session id, no session affinity |
| Auth | `Authorization: Bearer `, or `Bearer vlmrun` for anonymous access |
| Tools | [`read_document`](#read_document), [`read_image`](#read_image), [`read_audio`](#read_audio), [`read_video`](#read_video), [`list_models`](#list_models), [`get_completion`](#get_completion) |
Authentication follows the same tiers as the REST API. See
[Authentication](/gateway/authentication) and [Rate Limits](/gateway/rate-limits).
## What is MCP?
[MCP](https://modelcontextprotocol.io) is an open standard for connecting AI
applications to external systems: data sources, tools, and workflows. A server
publishes a typed catalog of its tools and their arguments, and can declare the
shape of what each one returns, so a client discovers them at run time instead of
carrying a hand-written integration per capability. Clients as different as
Claude, ChatGPT, VS Code, and Cursor all speak it, so one server reaches all of
them.
Here the catalog is six [tools](#tools). Point any MCP client at
`https://gateway.vlm.run/mcp` and those six sit alongside the agent's own. The
model hands one of them a URL, gets text back, and reasons over it. You write no
extraction schema and no glue code.
## When to use MCP
Use the MCP server when you want an agent to read a document, an image, or a
video, then answer questions about it or act on the text it returns. It is
the recommended way to reach the Gateway: the agent discovers the read tools,
picks the one a task calls for, and invokes it with a URL, so nothing in your
code has to know which capability a given file needs.
It fits wherever an agent is already in the loop:
* Answering questions about a scanned invoice or contract, where
`read_document` returns Markdown with the tables intact and `pages` narrows a
long file to the part that matters.
* Reading a chart, a diagram, or a UI screenshot with `read_image`, where the
answer is what the picture shows rather than the text printed on it.
* Summarizing a screen recording, or pulling its on-screen text, with
`read_video`.
* Checking `list_models` first, so the agent runs a model the Gateway serves
today instead of one pinned in your code.
The tools appear natively inside any MCP-aware framework, so the same server
serves an exploratory chat session and a scripted agent run alike.
## Quickstart
Start here to inspect the server's response before you wire an agent to it.
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
claude mcp add --transport http vlmrun https://gateway.vlm.run/mcp \
--header "Authorization: Bearer $VLMRUN_API_KEY"
```
To click through the tools instead of driving them from an agent, run
`npx @modelcontextprotocol/inspector`, set the transport to
**Streamable HTTP**, and point it at the same URL.
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Read https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf
and tell me the date of the earliest event reported.
```
The agent picks `read_document` on its own.
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
UNITED STATES
SECURITIES AND EXCHANGE COMMISSION
...
Date of report (Date of earliest event reported): October 2, 2024
```
Every `read_*` tool answers with text like this, plus the typed
[`ReadResult`](#return-shape) that carries the model that ran and its cost.
## Connect an agent
Every MCP-aware client connects the same way: one URL, one bearer header. Set
your key once:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
export VLMRUN_API_KEY="your-api-key"
```
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"mcpServers": {
"vlmrun": {
"type": "http",
"url": "https://gateway.vlm.run/mcp",
"headers": {
"Authorization": "Bearer "
}
}
}
}
```
Claude Desktop cannot use this shape. See the
[Claude Desktop](#claude-desktop) tab.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import os
from pydantic_ai import Agent
from pydantic_ai.capabilities import MCP
agent = Agent(
"anthropic:claude-sonnet-5",
capabilities=[
MCP(
"https://gateway.vlm.run/mcp",
headers={"Authorization": f"Bearer {os.environ['VLMRUN_API_KEY']}"},
)
],
)
result = agent.run_sync(
"Read https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/sample-invoice.pdf and tell me the total amount due."
)
print(result.output)
```
`native=False` (the default) runs the MCP client locally: Pydantic AI
connects to the gateway and calls the tools. `native=True` advertises the
URL to the model provider (OpenAI Responses, Anthropic, xAI) so the provider
connects to the server directly.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import os
from langchain_mcp_adapters.client import MultiServerMCPClient
from langgraph.prebuilt import create_react_agent
client = MultiServerMCPClient(
{
"vlmrun": {
"url": "https://gateway.vlm.run/mcp",
"transport": "streamable_http",
"headers": {"Authorization": f"Bearer {os.environ['VLMRUN_API_KEY']}"},
}
}
)
tools = await client.get_tools()
agent = create_react_agent("anthropic:claude-sonnet-5", tools)
```
Install the adapter with `pip install langchain-mcp-adapters langgraph`.
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { MCPClient } from "@mastra/mcp";
const mcp = new MCPClient({
servers: {
vlmrun: {
url: new URL("https://gateway.vlm.run/mcp"),
requestInit: {
headers: { Authorization: `Bearer ${process.env.VLMRUN_API_KEY}` },
},
},
},
});
const tools = await mcp.getTools();
```
Pass `tools` to any Mastra `Agent`. Install with `npm install @mastra/mcp`.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import os
from agents import Agent, Runner
from agents.mcp import MCPServerStreamableHttp
async with MCPServerStreamableHttp(
params={
"url": "https://gateway.vlm.run/mcp",
"headers": {"Authorization": f"Bearer {os.environ['VLMRUN_API_KEY']}"},
}
) as server:
agent = Agent(
name="Assistant",
mcp_servers=[server],
)
result = await Runner.run(agent, "Summarize the attached PDF.")
print(result.final_output)
```
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
claude mcp add --transport http vlmrun https://gateway.vlm.run/mcp \
--header "Authorization: Bearer $VLMRUN_API_KEY"
```
The `read_document`, `read_image`, `read_audio`, `read_video`,
`list_models`, and `get_completion` tools are now available in your Claude
Code session.
Claude Desktop speaks MCP over stdio, and its connector UI cannot send the
`Authorization` header the gateway requires. So bridge the two with
[`mcp-remote`](https://www.npmjs.com/package/mcp-remote), configured by hand
in `claude_desktop_config.json` (macOS:
`~/Library/Application Support/Claude/claude_desktop_config.json`, Windows:
`%APPDATA%\Claude\claude_desktop_config.json`):
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"mcpServers": {
"vlmrun": {
"command": "/path/to/node/bin/npx",
"args": [
"-y",
"mcp-remote",
"https://gateway.vlm.run/mcp",
"--transport",
"http-only",
"--header",
"Authorization:${VLM_AUTH}"
],
"env": {
"PATH": "/path/to/node/bin:/usr/local/bin:/usr/bin:/bin",
"VLM_AUTH": "Bearer "
}
}
}
}
```
Replace `/path/to/node` with the directory holding your Node.js install, so
`/usr/local` if `npx` is at `/usr/local/bin/npx`. Claude Desktop does not
inherit your shell `PATH`, so `command`, `PATH`, and `VLM_AUTH` all have to
be set explicitly. Node.js 18 or newer is required.
The `initialize` handshake returns server instructions that name each tool and
its use, and most clients inject them as the system message. When your client
does that, keep your own steering in the user turn, because a second system
message breaks strict OpenAI-compatible endpoints.
## Tools
One read tool per modality, plus a model-discovery tool and a completion lookup.
Every `read_*` tool takes a `url`, returns the extracted text, and carries a
`model` argument plus per-modality knobs.
| Tool | Reads | Key arguments |
| ----------------------------------- | --------------------------- | --------------------------------------------------------------------------------------- |
| [`read_document`](#read_document) | PDFs and images | `url`, `model`, `dpi`, `pages`, `method`, `method_params`, `json_mode` |
| [`read_image`](#read_image) | One still image | `url`, `model`, `prompt`, `json_mode` |
| [`read_audio`](#read_audio) | Speech audio | `url`, `model`, `language`, `json_mode` |
| [`read_video`](#read_video) | Video | `url`, `model`, `prompt`, `fps`, `max_frames`, `encoder`, `encoder_params`, `json_mode` |
| [`list_models`](#list_models) | The read tools' model menus | `modality` |
| [`get_completion`](#get_completion) | Metadata for a prior read | `completion_id` |
`read_document` and `read_image` split by intent, not by file type. Both accept a
JPEG or a PNG. Use `read_document` for the text printed **on** a page, which it OCRs
into Markdown with headings, lists, and tables preserved. Use `read_image` for what a
picture **depicts**, such as a photo, a chart, or a screenshot.
`url` is always an `http(s)` URL, a `data:` URI, or a bare base64 string. It is
never a local filesystem path. Host the file or inline it as base64 before
calling a read tool.
Every `read_*` tool takes `json_mode`. Left `false`, it returns the native form
as a plain string (Markdown for documents, text for audio and video); `true`
returns the structured envelope, parsed under `data`.
Only `read_document` offers a choice of `model` today. `read_image`,
`read_audio`, and `read_video` each pin a single model, so their `model`
argument accepts one value. Call `list_models` to see what the tools accept: a
model id passed to the wrong tool, or one that is not currently served, is
rejected.
### Pick a method by what you need
Every `read_document` method returns a different thing from the same page.
`list_models` tags each one with a capability, so you can choose without reading
a model card. Ask for the capability, then use the method that carries it.
| Capability | What you get | Needs `json_mode` |
| ------------------- | ------------------------------------------------------ | ----------------- |
| `document_markdown` | Text with the structure kept: headings, lists, tables. | No |
| `text_extraction` | The text, as a plain string. | No |
| `layout_regions` | Labelled areas, such as `title`, `table`, `figure`. | Yes |
| `reading_order` | The regions in the order a person reads them. | Yes |
| `text_citations` | Each piece of text with the box it came from. | Yes |
| `text_highlighting` | Boxes to draw over the page. | Yes |
For example, `rednote-hilab/dots.mocr` carries `document_markdown` on `markdown`
and `text_citations` on `parse_layout`, and `paddleocr/pp-ocrv6` carries
`text_citations` on `ocr` and `text_extraction` on `text`.
### `read_document`
OCR a PDF or image into clean Markdown (or structured JSON). Use it for invoices,
forms, reports, slide decks, screenshots, and scanned pages.
| Argument | Type | Default | Description |
| --------------- | ----------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `url` | `string` | required | The PDF or image. |
| `model` | enum | `rednote-hilab/dots.mocr` | OCR model to run. The tool advertises `rednote-hilab/dots.mocr`, `infly/Infinity-Parser2-Flash`, `deepseek-ai/deepseek-ocr-2`, `zai-org/glm-ocr`, and `paddleocr/pp-ocrv6`. |
| `dpi` | `integer` | `96` | Rasterization DPI (72-400) per PDF page before OCR. Higher is sharper on small text, but slower. |
| `method` | `string` | model default | Backend method run on each page: `markdown` on the generative OCR models, `ocr` on `paddleocr/pp-ocrv6`. A method the model does not advertise is rejected before the call. See [Pick a method](#pick-a-method). |
| `method_params` | `object` | unset | Keyword arguments for `method`, forwarded as-is. |
| `pages` | `list[int \| tuple[int,int]]` | every page (max 128) | 0-indexed page selection. Each entry in the list is a page index or a `[start, stop]` half-open range (`start` inclusive, `stop` exclusive); negative indices count from the end. Only the selected pages are rasterized and read, and they are **re-numbered from 0** in the response, where `num_pages` counts the pages you selected rather than the pages in the file. Indices past the end are ignored. |
| `json_mode` | `boolean` | `false` | `false` returns the `` / `` text blocks; `true` returns the `{"model", "method", "data"}` response object of per-page records. |
#### Page Selection
* `[0, 2, 4]` reads pages 0, 2, and 4
* `[[0, 3]]` reads pages 0-2
* `[[0, 3], [5, 7], -1]` reads pages 0, 1, 2, 5, 6 and the last page
One call reads at most 128 pages. For a longer document, call the tool once
per range and combine the results. The error names the exact ranges to use.
#### JSON Mode
Left unset, the tool returns the same text blocks as the REST API: one
`` block per PDF, wrapping one `` block per page, in reading order.
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# ACME Corp, Invoice #4471
| Item | Qty | Price |
| ---- | --- | ------ |
| Widget | 3 | $12.00 |
## Terms
Net 30. Thank you for your business.
```
A page that failed OCR is self-closing, with `status="error"` and no body. A single
image returns the Markdown alone, with no wrapper.
`json_mode: true` returns the same pages as records in `data`, which is how you
get boxes out of a region method:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
[
{
"object": "document",
"file_name": "invoice.pdf",
"mimetype": "application/pdf",
"num_pages": 2,
"dpi": 96,
"pages": [
{
"object": "document.page",
"page_index": 0,
"page_width": 612,
"page_height": 792,
"content": {
"object": "document.page.blocks",
"items": [
{
"index": 0,
"bbox_xywh": [0.3987, 0.0896, 0.1993, 0.0164],
"text": "ACME Corp",
"score": 0.9998
}
]
}
}
]
}
]
```
Both shapes are documented under [Text mode](/gateway/methods#text-mode) and
[JSON mode](/gateway/methods#json-mode).
### `read_image`
Send one still image to a vision-language model and get its answer back. Use it for
photos, charts, diagrams, product shots, and UI screenshots, where the value is in
what the picture depicts.
| Argument | Type | Default | Description |
| ----------- | --------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url` | `string` | required | The image. |
| `model` | `string` | `qwen/qwen3.5-0.8b` | Vision-language model that looks at the image. |
| `prompt` | `string` | "Describe this image in detail." | The question or the instruction, for example "What is the total on this receipt?". |
| `json_mode` | `boolean` | `false` | `false` returns the answer as a string; `true` asks the model itself for JSON, which a chat VLM returns unenveloped. See [Chat VLMs are not enveloped](/gateway/methods#chat-vlms-are-not-enveloped). |
The `read_image` menu carries no OCR model on purpose. An image whose value is
its printed text belongs to `read_document`, which accepts an image URL and
returns the Markdown alone, with no wrapper, for a single image.
### `read_audio`
Transcribe a speech audio file (`wav`, `mp3`, `m4a`, `flac`, `ogg`, …) into text.
| Argument | Type | Default | Description |
| ----------- | --------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `url` | `string` | required | The audio file. |
| `model` | `string` | `nvidia/parakeet-tdt-0.6b-v3` | Speech-to-text model to run. |
| `language` | `string` | auto-detect | ISO-639-1 language hint (`en`, `es`, …). Supply it when you already know the spoken language, to skip detection and improve accuracy. |
| `json_mode` | `boolean` | `false` | `false` returns the plain transcript; `true` returns the full transcription object (text plus metadata). |
### `read_video`
Describe or transcribe a video. Frames are sampled and encoded into image(s) so
image-capable models can read them, then a vision-language model produces the text.
| Argument | Type | Default | Description |
| ---------------- | ----------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url` | `string` | required | The video file. |
| `model` | `string` | `qwen/qwen3.5-0.8b` | Vision-language model that watches the sampled frames. |
| `prompt` | `string` | "Transcribe and describe the content of this video in detail." | What to produce from the video. Steer it with, for example, "List the on-screen text" or "Summarize what happens in 3 bullets". |
| `fps` | `number` | uniform sampling bounded by `max_frames` | Frames sampled per second of video. Higher captures more motion at higher cost. |
| `max_frames` | `integer` | model default | Upper bound on the number of frames sampled. |
| `encoder` | `mosaic` \| `frames` \| `keyframes` | `mosaic` | How the video becomes images. `mosaic` tiles sampled keyframes into grid images and is robust for every model; `frames` sends sampled frames as separate images; `keyframes` uses the video's I-frames. |
| `encoder_params` | `object` | encoder defaults | Parameters for `encoder`, e.g. `{"tile_cols": 4, "tile_rows": 4, "num_frames": 128}` for `mosaic`, or `{"fps": 1.0}` for `frames`. |
| `json_mode` | `boolean` | `false` | `false` returns the description as a string; `true` asks the model itself for JSON, which a chat VLM returns unenveloped. |
On the REST API, `encoder` and `encoder_params` map to
[`video_encoder` and `video_encoder_params`](/gateway/api-reference/post-chat-completions#video-encoding).
### `list_models`
List the served models that appear on an MCP tool menu, and which read tool each
one fits. Call it before passing a non-default `model`. A model the gateway
serves over the REST API but that sits on no tool menu, for example
`microsoft/florence-2-base-ft`, is not listed here: query
[`GET /v1/openai/models`](/gateway/api-reference/get-models) for the full
catalog.
| Argument | Type | Default | Description |
| ---------- | ------------------------------------------- | ------- | ------------------------------------------ |
| `modality` | `document` \| `image` \| `audio` \| `video` | all | Restrict the listing to one tool's models. |
Each entry is `{id, modality, tool, tools, aliases, methods, default_method,
method_details}`. Pass `id` verbatim as the `model` argument to the matching
`tool`. `tools` lists the read-tool menus the id sits on, because one model can
sit on several: a Qwen vision model is selectable through `read_image` and
`read_video`. `modality` and `tool` name the primary one, and a `modality`
filter matches on any of them. `tools` is menu membership, never inferred from
the model's declared capabilities: every OCR model accepts a bare image, yet
only `read_document` takes one as its `model`, so an OCR model lists
`read_document` alone.
`method_details` is the per-method answer to "what do I get back": one record per
method, with its [capabilities](#pick-a-method), `default: true` on the model's
default, and `requires_json_mode: true` when the method's full payload only
arrives parsed under `json_mode`. A method that carries `document_markdown` or
`text_extraction` still returns that text as a plain string.
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"id": "paddleocr/pp-ocrv6",
"modality": "document",
"tool": "read_document",
"tools": ["read_document"],
"aliases": ["pp-ocrv6"],
"methods": ["ocr", "detect", "text"],
"default_method": "ocr",
"method_details": [
{
"name": "ocr",
"capabilities": [
"text_extraction",
"text_citations",
"text_highlighting"
],
"default": true,
"requires_json_mode": true
},
{
"name": "detect",
"capabilities": ["text_highlighting"],
"requires_json_mode": true
},
{ "name": "text", "capabilities": ["text_extraction"] }
]
}
```
### `get_completion`
Look up what a prior `read_*` call cost and how it was served. Pass the
`completion_id` from that call's [`ReadResult`](#return-shape).
| Argument | Type | Default | Description |
| --------------- | -------- | -------- | -------------------------------------- |
| `completion_id` | `string` | required | The id a prior `read_*` tool returned. |
The reply carries `id`, `model`, `served_model_id`, `backend`, `cost`, `usage`
(tokens, or transcription seconds), `status`, and `latency_ms`. Records are
scoped to the caller's bearer token, so a completion made under one key is not
readable under another, and an unknown id is a `404`. The lookup spends one
model call from your [rate-limit](/gateway/rate-limits) budget, like the REST
`GET /v1/completions/{completion_id}` route behind it.
### Return shape (`ReadResult`)
Each `read_*` tool returns the clean text as the tool's content **and** a typed
structured payload, so an agent gets usable text directly while still being able
to read which model actually ran and what it cost:
| Field | Description |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `text` | The extracted text (Markdown, transcript, or description). Empty when `json_mode` is `true`. |
| `data` | The structured records, when `json_mode` is `true`. Null otherwise. |
| `model` | The model the gateway actually ran. |
| `cost` | The metered cost of the call in US dollars, when the gateway reports it. |
| `completion_id` | The id of this call (`chatcmpl-…` or `transcription-…`). Pass it to [`get_completion`](#get_completion). |
| `quota` | The free-tier allowance left, for an anonymous caller. Null when a key was used, and appended to the visible text when present. |
Very large output is trimmed before it reaches the agent so one document cannot
flood the context: past roughly 60,000 characters the text is cut and a
`[truncated: …]` note is appended telling the agent to narrow the request (for
example, fewer `pages`) or process in sections.
### Errors
An error names the next step, so the agent recovers on its own. A method the
chosen model does not advertise is refused before the call, and the message lists
the methods it does advertise:
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
model 'paddleocr/pp-ocrv6' does not support method 'markdown'. It advertises:
detect, ocr, text. Retry with one of those, or omit `method` for the default.
`list_models` reports the methods of every served model.
```
| Symptom | Cause | Fix |
| ---------------------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `Missing API Key` in Claude Desktop | The Connectors UI or a `"type": "http"` block sends no `Authorization`. | Delete the connector and use the [Claude Desktop](#claude-desktop) `mcp-remote` config. |
| `Some MCP servers could not be loaded` | Claude Desktop rejected a `"type": "http"` entry in its config file. | Replace it with the [Claude Desktop](#claude-desktop) `mcp-remote` config. |
| `405 Method Not Allowed` on `GET /mcp` | The server is stateless and serves POST only. | Send every request as a POST. |
| `Failed to fetch document URL: HTTP Error 404` | The gateway cannot reach `url`. | Host the file publicly, or pass it as a `data:` URI or base64. |
| `does not support method '…'` | The method belongs to another model. | Use a method the message names, or omit `method`. |
| Boxes or labels are missing from the answer | The payload only arrives under `json_mode`. | Set `json_mode: true`, then read `data`. |
| `[truncated: …]` at the end of the text | The output passed the inline cap. | Narrow `pages`, or read the document in sections. |
| Tool call times out or returns `500` | Intermittent load on the gateway backend. | Retry the call. If it persists, narrow the request (fewer `pages`, lower `max_frames`, or a shorter clip). |
## Call it without a framework
The server is stateless, so every request stands alone: no session id is
issued, no header needs carrying between requests, and any replica answers any
request. A single POST reaches a tool. Send
`Accept: application/json, text/event-stream`, because the reply is a
Server-Sent Events stream. The endpoint answers POST only; `GET /mcp` is a
`405`.
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
MCP=https://gateway.vlm.run/mcp
HDRS=(-H "Authorization: Bearer $VLMRUN_API_KEY"
-H "Content-Type: application/json"
-H "Accept: application/json, text/event-stream")
curl -sN -X POST $MCP "${HDRS[@]}" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{
"name":"list_models","arguments":{"modality":"document"}}}'
```
`initialize` still answers, and still returns the server instructions, so run
it when you want the tool catalog and usage notes rather than to open a
session:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -sN -X POST $MCP "${HDRS[@]}" \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{
"protocolVersion":"2025-06-18","capabilities":{},
"clientInfo":{"name":"curl","version":"1"}}}'
```
## Related
The REST surface behind the same gateway pipeline.
Query the live model catalog and capabilities.
Bearer tokens, tiers, and anonymous access.
Content part types, document limits, and video knobs.
# Methods & Response Format
Source: https://docs.vlm.run/gateway/methods
How method and response_format shape every VLM Run Gateway reply
Two request fields decide what a VLM Run Gateway model returns:
* **`method`** selects *what* the model computes (`ocr`, `detect`, `parse_layout`, `chat`, ...).
* **`response_format`** selects *how* that result is serialized (a text rendering, or a JSON object).
They are **orthogonal**: any method combines with any format, and the two
renderings carry the same information. This page is the canonical reference for
both. For copy-pasteable per-model examples, see [Models](/gateway/models).
## The `method` field
Most models expose one or more `method` values. Pass `method` (and optional
`method_params`) at the top level of the request body, or via `extra_body` in the
OpenAI Python SDK. If you omit `method`, the model's default is applied:
`markdown` on `zai-org/glm-ocr`, `deepseek-ai/deepseek-ocr-2`,
`rednote-hilab/dots.mocr`, `infly/infinity-parser2-flash`, and
`paddlepaddle/paddleocr-vl-1.6`, `ocr` on `paddleocr/pp-ocrv6`, `caption` on
`microsoft/florence-2-base-ft`, `pose` on `usyd-community/vitpose-plus-large`,
and `chat` on the chat VLMs.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
response = client.chat.completions.create(
model="paddleocr/pp-ocrv6",
messages=[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg"}}]}],
extra_body={"method": "ocr"},
)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const response = await client.chat.completions.create({
model: "paddleocr/pp-ocrv6",
messages: [{ role: "user", content: [{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg" } }] }],
method: "ocr",
});
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg \
-m paddleocr/pp-ocrv6 --method ocr
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/chat/completions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "paddleocr/pp-ocrv6",
"method": "ocr",
"messages": [{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg"}}]}]
}'
```
Every model's live `methods`, `default_method`, and `extra_body_help` are also
available on [`GET /v1/openai/models`](/gateway/api-reference/get-models). A
method a model does not advertise is a `400`.
## The `response_format` field
`response_format` mirrors the OpenAI object, so existing clients work unchanged:
| `response_format` | Reply |
| ---------------------------- | --------------------------------- |
| omitted or `{"type":"text"}` | The [text rendering](#text-mode). |
| `{"type":"json_object"}` | One [JSON object](#json-mode). |
A JSON `response_format` is always served **non-streamed**: the `stream` flag is
ignored, because a single valid JSON object cannot be assembled from SSE deltas.
In **text mode**, `stream: true` is honored and the stream is byte-identical to
the non-streaming reply: the `` open tag, then one chunk per ``
block, then the close tag. See
[Streaming](/gateway/guides/document-ocr#3-streaming-vs-non-streaming).
### The `precision` field
`precision` (int, range 1-8, default `4`) sets the number of decimal places on
normalized coordinates (`bbox_xywh` / `poly_xy` / `point_xy`) and `score`. It
applies to both formats and is ignored for markdown payloads. The response shape
is unchanged.
## Text mode
A text reply is a sequence of **top-level blocks**, one per input medium. Each
block body is either a **json block** or a **markdown block**, and nothing else.
Json is the default. A method renders a markdown block only when it emits
genuinely free-form text: `markdown`, `chat`, `caption`, `text`, `free_ocr`, and
the `ocr` of `rednote-hilab/dots.mocr` and `deepseek-ai/deepseek-ocr-2`. Which
one applies is a property of the
`(model, method)` pair, and it is echoed on the wire as `format=`. See
[Methods by model](#methods-by-model) for the mapping.
### Document input
Each PDF is one `` block wrapping one `` block per rasterized
page. Multiple `document_url` parts yield one `` block each, never
merged.
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{"object": "document.page.blocks", "items": [{"index": 0, "bbox_xywh": [0.0332, 0.0138, 0.1719, 0.0331], "text": "Invoice", "score": 0.998}]}
## Line items
| SKU | Qty |
| --- | --- |
```
| Element | Attributes |
| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `` | `file_name`?, `file_hash`?, `file_bytes`?, `mimetype`, `num_pages`, `dpi`, `language`?, in that order. `file_name` is omitted for `data:` URI uploads. `language` is a comma-separated list of ISO-639 codes in confidence order, and it is present only when the read reports one. |
| `` | `page_index` (zero-based), `format` (`json` or `markdown`), `page_width`, `page_height`, `status`?. |
A failed page is **self-closing**, with `status="error"` and no body, so page
numbering stays intact. `format` is the same on every page of a request, and it is
emitted on every page.
### Single image input
The block alone, with no wrapper:
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{"object": "pp_ocrv6.ocr.regions", "items": [{"bbox_xywh": [0.0332, 0.0138, 0.1719, 0.0331], "text": "Invoice", "score": 0.998}]}
```
or, for a markdown-kind method:
```markdown theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Annual Report 2024
## Overview
```
This is the one form that carries no `image_hash` / `image_width` /
`image_height`. Use JSON mode when you need that metadata.
### Multi-image input
Repeated `` blocks, one per image:
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{"object": "pp_ocrv6.ocr.regions", "items": [{"bbox_xywh": [0.0332, 0.0138, 0.1719, 0.0331], "text": "Invoice", "score": 0.998}]}
{"object": "pp_ocrv6.ocr.regions", "items": [{"bbox_xywh": [0.0117, 0.0203, 0.2422, 0.0414], "text": "Receipt", "score": 0.994}]}
```
Structured models (OCR, detection, and layout: everything except the chat VLMs)
run inference **per image** and emit one block each, and usage is summed. Chat
VLMs reason over all the images together and emit a single block.
### Chat VLMs pass through
The chat VLMs (`qwen/qwen3.5-0.8b` and the rest of the `chat` family) return
their reply **verbatim** in every case: text, one image, or several. No wrapper,
no tags, byte-identical to the model's own output.
They do not accept `document_url`: a PDF request is a
[`400` capability error](/gateway/error-codes#capability-violation-400).
## JSON mode
With `response_format={"type":"json_object"}` the reply is **always** the same
envelope. The chat VLMs are the one exemption; see
[Chat VLMs are not enveloped](#chat-vlms-are-not-enveloped).
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{ "model": "paddleocr/pp-ocrv6", "method": "ocr", "data": [ , ... ] }
```
`model` and `method` name what produced the payload. Both keys are JSON-mode only.
`data` holds **one self-describing entry per input medium**. A single image or
PDF is a list of one; multiple images or documents (up to the model's
`max_images`) are a list of many. Clients iterate the same way every time.
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"object": "image",
"image_hash": "sha256:…",
"image_width": 1024,
"image_height": 1448,
"content": ""
}
```
`object` is the input medium: `image`, `video`, or `document`. It says which
fields to read; the payload *type* lives on the content container's own
`object`. The hash and dimensions are omitted only when the image cannot be
decoded.
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"object": "document",
"file_name": "invoice.pdf",
"file_hash": "sha256:…",
"file_bytes": 48213,
"mimetype": "application/pdf",
"num_pages": 3,
"dpi": 150,
"pages": [
{
"object": "document.page",
"page_index": 0,
"page_width": 1275,
"page_height": 1650,
"content": { "object": "document.page.blocks", "items": [ , ... ] }
},
{
"object": "document.page",
"page_index": 1,
"page_width": 1275,
"page_height": 1650,
"status": "error"
}
]
}
```
A page record carries `object`, `page_index`, `page_width`, `page_height`,
`content`, and optional `status`. A failed page carries `"status": "error"`
and no `content`. `file_name` is omitted for inline (`data:`) uploads, and
the optional `language` is document-level.
### `content` per medium
| Medium | `content` |
| -------------------- | ------------------------------------------------------------------------------------------------------------------- |
| Document page | Always `{"object": "document.page.blocks", "items": [ , ... ]}`, whatever method read the page. |
| Image, json kind | `{"object": "..regions", "items": [ , ... ]}`. `items` is `[]` when nothing is found. |
| Image, markdown kind | The **string** itself, not wrapped. |
A json payload is **never a bare array**, and a markdown payload is never
wrapped. An image `content` has exactly one shape per `(model, method)`.
```json Regions theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"model": "paddleocr/pp-ocrv6",
"method": "ocr",
"data": [
{
"object": "image",
"image_hash": "sha256:…",
"image_width": 1024,
"image_height": 1448,
"content": {
"object": "pp_ocrv6.ocr.regions",
"items": [
{
"bbox_xywh": [0.0332, 0.0138, 0.1719, 0.0331],
"text": "Invoice",
"score": 0.998
}
]
}
}
]
}
```
```json Markdown theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"model": "zai-org/GLM-OCR",
"method": "markdown",
"data": [
{
"object": "image",
"image_hash": "sha256:…",
"image_width": 1700,
"image_height": 2200,
"content": "# Annual Report 2024\n\n## Overview"
}
]
}
```
```json Document page theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"object": "document.page.blocks",
"items": [
{
"index": 0,
"text": "# Annual Report 2024\n\n## Overview"
}
]
}
```
### The region record
One entry of an image `content.items`, that is, what one detector found in one
pass.
| Key | Type | Notes |
| ----------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bbox_xywh` | `[x, y, w, h]` | The single universal box: normalized 0-1, `precision` dp. There is no `bbox`, no `bbox_norm`, and no pixel or xyxy variant. |
| `poly_xy` | `[[x, y], ...]` | Normalized polygon, only when the model emits one (`paddleocr/pp-ocrv6`). |
| `point_xy` | `[x, y]` | Normalized point, only when the method returns points rather than boxes. |
| `label` | `string` | Class or category. A layout category is folded to lower `snake_case`, so `doc_title` and `section_header` read the same whichever layout model produced them. |
| `text` | `string` | Recognized text. |
| `score` | `float` | Confidence 0-1. |
| `kpts_xy` | `[[x, y], ...]` | Normalized COCO joints, only on `usyd-community/vitpose-plus-large`. |
Unset optional keys are omitted. Which keys a given `(model, method)` emits is
fixed, and an unknown key is a **validation error**, not a silently ignored
extra.
### The document block record
One entry of `document.page.blocks.items`. Every key is optional, and a model
fills the subset it can, so one type covers every document method.
| Key | Type | Notes |
| ------------- | ------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `index` | `int` | The block's position on the page, zero-based like `page_index`. |
| `bbox_xywh` | `[x, y, w, h]` | Normalized 0-1 box. Absent on a whole-page read. |
| `poly_xy` | `[[x, y], ...]` | Normalized polygon, from a quad or polygon detector. |
| `label` | `string` | Layout category, folded to lower `snake_case`. |
| `text` | `string` | Recognized text. Absent when the block was **not read**, for example a `picture` region. |
| `text_format` | `"text"`, `"markdown"`, `"html"`, `"latex"` | Format of `text`. Omitted means `markdown`. |
| `score` | `float` | Detector confidence 0-1. |
| `parent` | `int` | The `index` of the block that contains this one. |
| `children` | `[int]` | The `index` of every block inside this one. |
| `attributes` | `object` | Metadata from a customized pipeline. Absent on the stock pipeline. |
A whole-page read is **one block** that carries `text` and no geometry:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"object": "document.page.blocks",
"items": [
{ "index": 0, "text": "# Annual Report 2024\n\n## Overview" }
]
}
```
A layout read is one block per region:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"object": "document.page.blocks",
"items": [
{
"index": 0,
"bbox_xywh": [0.08, 0.0485, 0.84, 0.0364],
"label": "title",
"text": "Annual Report 2024",
"score": 0.98
}
]
}
```
### Content object tags
On a document page the tag is always `document.page.blocks`, because the payload
is a property of the medium. On an image the tag is `..regions`,
where the group is the model's contract family. Aliases resolve first, so
`pp-ocrv6` and `paddleocr/pp-ocrv6` share one contract.
| Model | Method | `content.object` |
| ----------------------------------- | ------------------------------------------------------------------ | -------------------------------------- |
| `paddleocr/pp-ocrv6` | `ocr` | `pp_ocrv6.ocr.regions` |
| `paddleocr/pp-ocrv6` | `detect` | `pp_ocrv6.detect.regions` |
| `rednote-hilab/dots.mocr` | `parse_layout` | `dots_mocr.parse_layout.regions` |
| `rednote-hilab/dots.mocr` | `parse_layout_only` | `dots_mocr.parse_layout_only.regions` |
| `deepseek-ai/deepseek-ocr-2` | `grounding_ocr` | `deepseek_ocr_2.grounding_ocr.regions` |
| `microsoft/florence-2-base-ft` | `od`, `dense_region_caption`, `region_proposal`, `ocr_with_region` | `florence_2..regions` |
| `usyd-community/vitpose-plus-large` | `pose` | `vitpose_plus.pose.keypoints` |
A markdown-kind method on an image has no container and therefore no tag.
### Chat VLMs are not enveloped
A chat VLM reply is passed through in JSON mode too: the body is the **model's
own JSON**, with no `data` wrapper and no `object` tag.
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{ "animal": "golden retriever", "setting": "beach at sunset" }
```
This is what OpenAI's `response_format={"type":"json_object"}` means: the
*model* emits the JSON.
## Text and JSON modes carry the same data
A text rendering decodes into exactly the JSON-mode `data` list: parse the body
as JSON when `format="json"`, and take it verbatim when `format="markdown"`. One
type therefore validates both renderings, and you can switch formats without a
second parser.
The single exception is the bare single-image form, which carries no image
metadata: decoding recovers `content` but not `image_hash`, `image_width`, or
`image_height`.
## Methods by model
Output kind is a function of the model **and** the method. Default method in
**bold**. `Imgs` is the maximum images per request, and `Doc` marks the models
that accept `document_url` PDFs.
| Model | Methods to output kind | Imgs | Doc |
| ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ---- | --- |
| [`qwen/qwen3.5-0.8b`](/gateway/models/qwen-qwen3.5-0.8b) | **`chat`** to passthrough | 64 | – |
| [`qwen/qwen3.8-27b`](/gateway/models/qwen-qwen3.8-27b) | **`chat`** to passthrough | 64 | – |
| [`microsoft/florence-2-base-ft`](/gateway/models/microsoft-florence-2-base-ft) | **`caption`**, `detailed_caption`, `more_detailed_caption`, `ocr` to markdown · `od`, `dense_region_caption`, `region_proposal`, `ocr_with_region` to json | 1 | – |
| [`paddleocr/pp-ocrv6`](/gateway/models/paddleocr-pp-ocrv6) | **`ocr`**, `detect` to json · `text` to markdown | 8 | ✓ |
| [`deepseek-ai/deepseek-ocr-2`](/gateway/models/deepseek-ai-deepseek-ocr-2) | **`markdown`**, `ocr`, `free_ocr` to markdown · `grounding_ocr` to json | 1 | ✓ |
| [`zai-org/glm-ocr`](/gateway/models/zai-org-glm-ocr) | **`markdown`** to markdown | 1 | ✓ |
| [`rednote-hilab/dots.mocr`](/gateway/models/rednote-hilab-dots-mocr) | `parse_layout`, `parse_layout_only` to json · `ocr`, **`markdown`** to markdown | 1 | ✓ |
| [`infly/infinity-parser2-flash`](/gateway/models/infly-infinity-parser2-flash) | **`markdown`** to markdown | 1 | ✓ |
| [`paddlepaddle/paddleocr-vl-1.6`](/gateway/models/paddlepaddle-paddleocr-vl-1.6) | **`markdown`** to markdown (model-specific envelope) · `ocr`, `table`, `formula`, `chart` to markdown | 1 | ✓ |
| [`baidu/unlimited-ocr`](/gateway/models/baidu-unlimited-ocr) | **`markdown`**, `multi_page` to markdown (model-specific envelope) | 1 | ✓ |
| [`usyd-community/vitpose-plus-large`](/gateway/models/usyd-community-vitpose-plus-large) | **`pose`** to json | 1 | – |
The kind above is the **image** kind, which is also the page `format` in text
mode. In JSON mode a document page is always
[`document.page.blocks`](#document-block).
## Method Parameters
`method_params` is an optional object passed alongside `method` at the top level
of the request body, or via `extra_body` in the OpenAI Python SDK. Keys are model-
and method-specific; the table below lists every key.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
response = client.chat.completions.create(
model="paddleocr/pp-ocrv6",
messages=[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg"}}]}],
extra_body={
"method": "ocr",
"method_params": {"lang": "en", "score_threshold": 0.5},
},
)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const response = await client.chat.completions.create({
model: "paddleocr/pp-ocrv6",
messages: [{ role: "user", content: [{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg" } }] }],
method: "ocr",
method_params: { lang: "en", score_threshold: 0.5 },
});
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg \
-m paddleocr/pp-ocrv6 --method ocr \
--method-params '{"lang": "en", "score_threshold": 0.5}'
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/chat/completions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "paddleocr/pp-ocrv6",
"method": "ocr",
"method_params": {
"lang": "en",
"score_threshold": 0.5
},
"messages": [{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.receipt/playground/2.jpg"}}]}]
}'
```
### Method-specific keys
| Key | Model | Method(s) | Description |
| ----------------- | ---------------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `lang` | `paddleocr/pp-ocrv6` | `ocr`, `detect` | Recognition language (`en`, `ch`, `japan`, ...). Default `en`. |
| `score_threshold` | `paddleocr/pp-ocrv6` | `ocr` | Drop text regions below this recognition confidence (0.0-1.0). Default `0.5`. |
| `prompt` | `deepseek-ai/deepseek-ocr-2` | `grounding_ocr` | Full grounding prompt, with the phrase to locate wrapped in the model's ref markers. See [Grounding](/gateway/models/deepseek-ai-deepseek-ocr-2#grounding) for the exact syntax. |
| `window_size` | `baidu/unlimited-ocr` | `multi_page` | Pages read per model call as a sliding window over the document. Default `8`. |
## Related
Per-model dropdowns with request and response examples for every method.
Content part types and document input limits.
End-to-end recipe from model selection to response parsing.
Full request and response schema.
# Models
Source: https://docs.vlm.run/gateway/models
VLM Run Gateway model catalog for OCR, VQA, and document inference
The VLM Run Gateway serves open-weight vision and document models through a single
OpenAI-compatible API. Each model declares the input types it accepts
(`text`, `image_url`, `video_url`, `document_url`) and the operations it supports
via the `method` field.
**Available** means that the model is in the served catalog. An individual
request can still be aborted with a [504 inference timeout](/gateway/error-codes#inference-timeout-504).
Use the standalone reference for any available model to see its accepted inputs,
per-method output shapes, request, and response examples.
## Output at a glance
Every model returns one of two payload kinds, **json** or **markdown**, and
`response_format` decides how that payload is serialized. This holds across every
model and method on this page.
* **Text mode** (`response_format` omitted, or `{"type":"text"}`): one block per input medium. A document is a `` / `` block; a single image is the bare payload.
* **JSON mode** (`{"type":"json_object"}`): one JSON object, `{"model", "method", "data"}`, with one self-describing entry per input medium.
The chat VLMs are the exemption: they pass their own output through verbatim in
both modes. `baidu/unlimited-ocr` and `paddlepaddle/paddleocr-vl-1.6` keep the
block-per-medium shape but use model-specific attribute and key names. See
[Methods & Response Format](/gateway/methods) for the full contract, the region
schema, and the `precision` knob.
## Chat and Visual Question Answering (VQA)
VQA models accept text plus up to 64 images, or one video, in the same message.
They return the model's reply verbatim, with no envelope.
| Model | Status | Method | Accepted Inputs | Max images / videos |
| -------------------------------------------------------- | --------- | ------ | -------------------------------- | ------------------- |
| [`qwen/qwen3.5-0.8b`](/gateway/models/qwen-qwen3.5-0.8b) | Available | chat | `text`, `image_url`, `video_url` | 64 images, 1 video |
| [`qwen/qwen3.8-27b`](/gateway/models/qwen-qwen3.8-27b) | Available | chat | `text`, `image_url`, `video_url` | 64 images, 1 video |
## Image Understanding
Single-image multi-task models that caption, detect, and OCR from one `image_url`.
They do not accept `document_url` or video.
| Model | Status | Default method | Other methods | Accepted Inputs | Max images |
| ------------------------------------------------------------------------------ | --------- | -------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------- | ---------- |
| [`microsoft/florence-2-base-ft`](/gateway/models/microsoft-florence-2-base-ft) | Available | `caption` | `detailed_caption`, `more_detailed_caption`, `ocr`, `ocr_with_region`, `od`, `dense_region_caption`, `region_proposal` | `image_url` | 1 |
## Document and Image OCR
These models accept a PDF via `document_url` or an image via `image_url`. The
Gateway rasterizes each PDF page and runs per-page inference, returning one entry
per input document. For images, the Gateway runs inference on the entire image.
For consistency, structured output is not streamed token-by-token.
| Model | Status | Default method | Other methods | Accepted Inputs | Streaming |
| -------------------------------------------------------------------------------- | --------- | -------------- | ------------------------------------------ | --------------------------- | --------- |
| [`paddleocr/pp-ocrv6`](/gateway/models/paddleocr-pp-ocrv6) | Available | `ocr` | `detect`, `text` | `image_url`, `document_url` | Page-wise |
| [`deepseek-ai/deepseek-ocr-2`](/gateway/models/deepseek-ai-deepseek-ocr-2) | Available | `markdown` | `ocr`, `free_ocr`, `grounding_ocr` | `image_url`, `document_url` | Page-wise |
| [`zai-org/glm-ocr`](/gateway/models/zai-org-glm-ocr) | Available | `markdown` | none | `image_url`, `document_url` | Page-wise |
| [`rednote-hilab/dots.mocr`](/gateway/models/rednote-hilab-dots-mocr) | Available | `markdown` | `parse_layout`, `parse_layout_only`, `ocr` | `image_url`, `document_url` | Page-wise |
| [`infly/infinity-parser2-flash`](/gateway/models/infly-infinity-parser2-flash) | Available | `markdown` | none | `image_url`, `document_url` | Page-wise |
| [`paddlepaddle/paddleocr-vl-1.6`](/gateway/models/paddlepaddle-paddleocr-vl-1.6) | Available | `markdown` | `ocr`, `table`, `formula`, `chart` | `image_url`, `document_url` | Page-wise |
| [`baidu/unlimited-ocr`](/gateway/models/baidu-unlimited-ocr) | Available | `markdown` | `multi_page` | `image_url`, `document_url` | Page-wise |
For documents, OCR methods usually return the same `` / ``
blocks in text mode. `format="markdown"` on the page marks a Markdown body
(`markdown`, `text`, `free_ocr`, and the `ocr` of dots.mocr and
deepseek-ocr-2), and `format="json"` marks a region payload
(`detect`, `grounding_ocr`, `parse_layout`, `parse_layout_only`).
`baidu/unlimited-ocr` and `paddlepaddle/paddleocr-vl-1.6` use a model-specific
envelope; see [unlimited-ocr](/gateway/models/baidu-unlimited-ocr) and
[paddleocr-vl-1.6](/gateway/models/paddlepaddle-paddleocr-vl-1.6) for the exact
shapes, and [Text mode](/gateway/methods#text-mode) for the shared contract.
PaddleOCR-VL 1.6 is the only model in the catalog that also reads a page's
*structured* content: its `table` method returns OTSL structure tokens, `formula`
returns LaTeX, and `chart` returns a described series as a Markdown table, in
the same envelope as `ocr`.
## Pose Estimation
2D human pose estimation over one `image_url` or one `video_url`. Each detected
person returns a bounding box and 17 normalized COCO keypoints.
| Model | Status | Method | Accepted Inputs | Max images / videos |
| ---------------------------------------------------------------------------------------- | --------- | ------ | ------------------------ | ------------------- |
| [`usyd-community/vitpose-plus-large`](/gateway/models/usyd-community-vitpose-plus-large) | Available | `pose` | `image_url`, `video_url` | 1 image, 1 video |
## Embeddings and Transcription
These models appear on [`GET /v1/openai/models`](/gateway/api-reference/get-models)
with a `task` field other than `chat`. They use separate OpenAI-compatible
endpoints, not chat completions. See
[Embeddings](/gateway/api-reference/post-embeddings) and
[Audio Transcriptions](/gateway/api-reference/post-audio-transcriptions) for full
request schemas.
| Model | Status | Task | Endpoint |
| ---------------------------------------------------------------------------- | --------- | ------------ | ------------------------------------------------------------- |
| [`qwen/qwen3-vl-embedding-2b`](/gateway/models/qwen-qwen3-vl-embedding-2b) | Available | `embed` | `POST /v1/openai/embeddings` (text, `image_url`, `video_url`) |
| [`nvidia/parakeet-tdt-0.6b-v3`](/gateway/models/nvidia-parakeet-tdt-0.6b-v3) | Available | `transcribe` | `POST /v1/openai/audio/transcriptions` |
## Model Aliases
Many models accept multiple request IDs:
1. **Preferred:** lowercase `/` (listed on `/models`)
2. **Short:** slug only (e.g. `pp-ocrv6`, `glm-ocr`, `dots.mocr`, `deepseek-ocr-2`, `paddleocr-vl-1.6`, `infinity-parser2-flash`)
3. **Hugging Face:** upstream repo id (e.g. [`zai-org/GLM-OCR`](https://huggingface.co/zai-org/GLM-OCR))
## Routed models
The Gateway also routes to frontier vision AI models via upstream providers, so you can use the same API key and baseURL to access them as well.
These models are passthrough VLMs supporting the same `methods` and `method_params` reported by their providers. Prices are the provider's listed
prices in USD per 1M tokens.
| Model | Provider | Modalities | Context | Input | Output |
| ------------------------- | --------------- | ------------------ | ------: | -----: | ------: |
| `moonshotai/kimi-k3` | `fireworks` | text, image, video | 262K | \$3.00 | \$15.00 |
| `meta/muse-spark-1.2` | `meta` | text, image, video | 1M | \$1.25 | \$4.25 |
| `google/gemini-3.7-flash` | `google-vertex` | text, image, video | 1M | \$0.75 | \$3.75 |
| `minimax/minimax-m3` | `fireworks` | text, image, video | 1M | \$0.30 | \$1.20 |
[See all 9 routed models →](/gateway/models/routed-models)
## Next steps
The full response envelope, region schema, and `precision` reference.
End-to-end recipe from model selection to response parsing.
Full request parameters, streaming, and error handling.
Per-token rates and how to read them from the live catalog.
# Routed Models
Source: https://docs.vlm.run/gateway/models/routed-models
Frontier vision AI models the Gateway routes to via upstream providers.
The Gateway also routes to frontier vision AI models via upstream providers, so
you can use the same API key and baseURL to access them as well. These models
are passthrough VLMs supporting the same `methods` and `method_params` reported
by their providers. On
[`GET /v1/openai/models`](/gateway/api-reference/get-models) a routed model
carries a non-empty `provider` field.
Prices are the provider's listed prices in USD per 1M tokens (input / output).
| Model | Provider | Modalities | Context | Input | Output |
| ------------------------------------- | --------------- | ------------------ | ------: | -----: | ------: |
| `google/gemini-3.5-flash-lite` | `google-vertex` | text, image, video | 1M | \$0.30 | \$2.50 |
| `google/gemini-3.7-flash` | `google-vertex` | text, image, video | 1M | \$0.75 | \$3.75 |
| `google/gemini-robotics-er-2-preview` | `google-gemini` | text, image, video | 1M | \$0.30 | \$2.50 |
| `google/gemma-4-26b-a4b-it` | `google-vertex` | text, image | 131K | \$0.10 | \$0.30 |
| `google/gemma-4-31b-it` | `huggingface` | text, image, video | 131K | \$0.15 | \$0.40 |
| `meta/muse-glimmer-30b` | `fireworks` | text, image, video | 131K | \$0.35 | \$1.50 |
| `meta/muse-spark-1.2` | `meta` | text, image, video | 1M | \$1.25 | \$4.25 |
| `minimax/minimax-m3` | `fireworks` | text, image, video | 1M | \$0.30 | \$1.20 |
| `moonshotai/kimi-k3` | `fireworks` | text, image, video | 262K | \$3.00 | \$15.00 |
Prices in USD per 1M tokens
Every routed model accepts a JSON Schema `response_format`, except
`meta/muse-glimmer-30b`, which accepts `json_object` only. See
[Methods & Response Format](/gateway/methods).
[`qwen/qwen3.8-27b`](/gateway/models/qwen-qwen3.8-27b) is not listed here. It runs
on VLM Run GPUs, so it carries no `provider`, and it is documented with the
Gateway [VQA models](/gateway/models#visual-question-answering-vqa).
## Next steps
The full Gateway model catalog.
Full request parameters, streaming, and error handling.
# Multimodal Inputs
Source: https://docs.vlm.run/gateway/multimodal-inputs
Content part types, per-modality limits, and format tradeoffs
Content parts are supplied under `messages[].content`, following the same
shape OpenAI's SDKs already use. Each part is one of:
| Part | Accepts | Notes |
| -------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `text` | Plain text | Prompt or question text. |
| `image_url` | `url` or base64 data URI | Single image per request for OCR models; VQA models accept multiple |
| `document_url` | `url` or base64 data URI | PDF; enters the page-wise document pipeline |
| `file_url` | Same as `document_url` | Accepted as an alias of `document_url` for PDF inputs. |
| `video_url` | `url` | One video per request on `qwen/qwen3.5-0.8b` and `qwen/qwen3.8-27b`. Tune frame sampling with `video_fps`, `video_max_frames`, and `video_resolution`. See [Video Inputs](#video-inputs). |
The VLM Run Gateway validates content parts against each model's capabilities before
inference. Mismatches return `400` with
[`capability_violation`](/gateway/error-codes#capability-violation-400). Check
a model's accepted input types on [Models](/gateway/models) or
[`GET /v1/openai/models`](/gateway/api-reference/get-models).
## Image Inputs
Most OCR and vision models accept a single `image_url` part per request. VQA
models such as `qwen/qwen3.5-0.8b` and `qwen/qwen3.8-27b` accept multiple images in the same
message, for side-by-side comparison or multi-image context.
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"type": "image_url",
"image_url": { "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto" }
}
```
`image_url.detail` (`auto` / `low` / `high`) is accepted for OpenAI SDK
compatibility. The Gateway's own documented cost/quality knob for images is
`image_resolution`, below; prefer it when you want a specific resize
behavior.
Use `image_resolution` to resize an image before inference when a model's
default resolution costs more than the task needs. See
[Gateway extensions](/gateway/api-reference/post-chat-completions#gateway-extensions)
for the full parameter reference.
## Video Inputs
`qwen/qwen3.5-0.8b` and `qwen/qwen3.8-27b` accept one `video_url` part per request alongside an
optional text prompt. The Gateway samples frames from the video before
inference. Control how many frames reach the model with these top-level
fields (or via `extra_body` in the OpenAI Python SDK):
| Field | Type | Description |
| ---------------------- | --------- | -------------------------------------------------------------------------------------------------- |
| `video_fps` | `number` | Target frames per second to sample from the source video. |
| `video_max_frames` | `integer` | Hard cap on frames passed to the model. |
| `video_resolution` | `string` | Resize preset for sampled frames: `256x192`, `320x240`, `448x336`, `512x384`, `640x480`. |
| `video_encoder` | `string` | How the video is encoded into image(s) before dispatch: `mosaic` (default), `frames`, `keyframes`. |
| `video_encoder_params` | `object` | Keyword arguments for the encoder (e.g. `tile_cols`, `tile_rows`, `num_frames`). |
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"type": "video_url",
"video_url": { "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4" }
}
```
Start with a low `video_max_frames` (for example `8`) and tune `video_fps`
before raising either knob. More frames increase latency and cost.
Image-only models can still read video: the Gateway encodes sampled frames into
one or more images before dispatch. `video_encoder` picks the strategy
(`mosaic`, the default, tiles keyframes into a single grid image; `frames` and
`keyframes` are also available) and `video_encoder_params` tunes it. See
[Video encoding](/gateway/api-reference/post-chat-completions#video-encoding).
An undecodable video returns `400` with `error.type = "invalid_request_error"`.
See [Error Codes](/gateway/error-codes#invalid-video-400).
## Document Inputs
Document (PDF) requests are decoded and rasterized at the ingress before
per-page inference.
### Limits
| Limit | Value | Notes |
| --------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| File size | 256 MB | Enforced on decoded bytes, for both `document_url` URLs and base64 data URIs. |
| Pages | 128 (`document_max_pages`) | Requests over the cap are rejected before inference, and the error names the page ranges that cover the document. Read a longer PDF section by section with `document_pages`. |
Requests over either limit return `400` with `error.code = "invalid_document"`.
See [Error Codes](/gateway/error-codes#invalid-document-400).
### URL vs. base64
| Input | Use when |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `document_url` with a hosted URL | Default choice; keeps the request body small and fast to transmit. |
| Base64 data URI | Only when the document isn't already hosted somewhere reachable. Inflates the request body by \~33% and increases request latency for large files. |
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"type": "document_url",
"document_url": { "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf" }
}
```
Prefer a URL over base64 whenever possible, especially for documents over a
few megabytes.
For `document_dpi` and other method-specific knobs, see
[Method Parameters](/gateway/methods#method-parameters).
### Scanned vs. native PDFs
The VLM Run Gateway rasterizes every page and runs OCR regardless of whether the
source PDF has a text layer. Scanned (image-only) and native (text-layer)
PDFs are handled the same way, both go through the OCR model rather than a
text-extraction shortcut. If you already have a reliable text layer and
don't need OCR, extracting it client-side before calling the Gateway will be
faster and cheaper.
### VQA on PDFs
The chat VLMs, including `qwen/qwen3.5-0.8b` and `qwen/qwen3.8-27b`, do not accept `document_url`. A PDF
request is a
[`400` capability error](/gateway/error-codes#capability-violation-400).
For PDFs, use a document model from
[Models](/gateway/models#document-and-image-ocr). To ask free-form questions about
a PDF, read it first with an OCR model, then send the extracted text to the chat
model.
## Related
End-to-end recipe from model selection to response parsing.
Receive pages as they finish instead of waiting for the whole document.
Which input types each model accepts.
Full request and response schema.
# Pricing
Source: https://docs.vlm.run/gateway/pricing
Per-token rates for VLM Run Gateway models, metered per request.
The VLM Run Gateway meters every request. Every served model bills **per token**.
The rates here come straight from the live model catalog
([`GET /v1/models/{model_id}`](/gateway/api-reference/get-model-by-id)), so they
always match what you are actually charged. Two signals let you wire up cost
tracking today:
* **Per-model rate card.** Each model exposes a `pricing` object with
USD-per-1M rates for `prompt`, `completion`, `input_cache_read`,
`input_cache_write`, and `image`.
* **Per-request cost.** `usage.cost` on a chat completion, streaming or not, is
the actual metered charge for that request, in USD rounded to six decimals,
the resolution the ledger bills at. Embeddings and transcriptions report token
counts only, so multiply them by the rate below.
The rates below are the currently published rates and may change. Always read
`pricing` from the model catalog (or `usage.cost` on a chat completion) as the
source of truth.
## How billing works
* Token rates are **USD per 1M tokens**. **Input** is prompt tokens, **Cached
input** is `input_cache_read`, and **Output** is completion tokens. A request
pays `prompt` on the uncached prompt tokens, `input_cache_read` on the cached
ones, and `completion` on the output.
* **Documents bill per page.** The Gateway rasterizes a PDF and runs one
inference per page. It sums the per-page token counts, so the charge is linear
in page count.
* `input_cache_write` is `$0.00` across all models today. The `image` rate is
published for reference. No served model bills on it today, because image and
audio inputs already count as prompt tokens.
* Neither `method` (`ocr`, `markdown`, `detect`, `parse_layout`, `chat`) nor
`response_format` (text or JSON) changes the rate. You pay the rate of
whichever model serves the request.
* Non-generative vision models emit no text. They report a flat prompt-token
count per image instead, so the token rate gives a fixed price per image.
### What 1M tokens buys
A 1M-token budget covers roughly the following:
* **\~3,000 image captions** at \~200 input + 130 output tokens / image
* Markdown from a **\~2K-page slide deck with figures** at \~500 output tokens / page
* **\~4,000 visual questions** at \~200 input + 50 output tokens / question
* **\~2,000 text passages** embedded at \~512 tokens / passage
* Markdown from **\~1K pages of legal docs with dense text** at \~1,000 output tokens / page
### Chat and document models
| Model | Input | Cached input | Output |
| ------------------------------- | -----: | -----------: | -----: |
| `paddleocr/pp-ocrv6` | \$0.01 | \$0.01 | \$0.20 |
| `qwen/qwen3.5-0.8b` | \$0.08 | \$0.02 | \$0.15 |
| `microsoft/florence-2-base-ft` | \$0.10 | \$0.03 | \$0.30 |
| `zai-org/glm-ocr` | \$0.10 | \$0.02 | \$0.20 |
| `paddlepaddle/paddleocr-vl-1.6` | \$0.15 | \$0.15 | \$0.35 |
| `rednote-hilab/dots.mocr` | \$0.20 | \$0.03 | \$0.40 |
| `infly/infinity-parser2-flash` | \$0.25 | \$0.03 | \$0.45 |
| `baidu/unlimited-ocr` | \$0.25 | \$0.03 | \$0.55 |
| `deepseek-ai/deepseek-ocr-2` | \$0.25 | \$0.25 | \$0.80 |
| `qwen/qwen3.8-27b` | \$0.35 | \$0.085 | \$2.55 |
Prices in USD per 1M tokens
### Pose estimation
`usyd-community/vitpose-plus-large` emits no text. It reports a flat 256 prompt
tokens per image and no output tokens, so one image costs about \$0.001, or
about 1,000 images per dollar.
| Model | Input | Output |
| ----------------------------------- | -----: | -----: |
| `usyd-community/vitpose-plus-large` | \$3.90 | \$0.00 |
Prices in USD per 1M tokens
### Routed models
The [routed models](/gateway/models/routed-models) also bill per token, at the
provider's listed rates. They carry the `paid` access tier, so an organization
needs an active subscription or a funded balance to call them.
| Model | Input | Cached input | Output |
| ------------------------------------- | -----: | -----------: | ------: |
| `google/gemma-4-26b-a4b-it` | \$0.10 | \$0.00 | \$0.30 |
| `google/gemma-4-31b-it` | \$0.15 | \$0.05 | \$0.40 |
| `google/gemini-3.5-flash-lite` | \$0.30 | \$0.03 | \$2.50 |
| `google/gemini-robotics-er-2-preview` | \$0.30 | \$0.03 | \$2.50 |
| `minimax/minimax-m3` | \$0.30 | \$0.06 | \$1.20 |
| `meta/muse-glimmer-30b` | \$0.35 | \$0.04 | \$1.50 |
| `google/gemini-3.7-flash` | \$0.75 | \$0.075 | \$3.75 |
| `meta/muse-spark-1.2` | \$1.25 | \$0.15 | \$4.25 |
| `moonshotai/kimi-k3` | \$3.00 | \$0.30 | \$15.00 |
Prices in USD per 1M tokens
### Embeddings
Text, image, and video inputs all bill as prompt tokens. There is no output
charge.
| Model | Input |
| ---------------------------- | ------: |
| `qwen/qwen3-vl-embedding-2b` | \$0.013 |
Prices in USD per 1M tokens
### Transcription
Transcription bills the audio it ingests as prompt tokens. There is no separate
output charge.
| Model | Audio input |
| ----------------------------- | ----------: |
| `nvidia/parakeet-tdt-0.6b-v3` | \$6.00 |
Prices in USD per 1M tokens
### How to get model pricing
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import httpx
from urllib.parse import quote
model_id = quote("paddlepaddle/paddleocr-vl-1.6", safe="")
response = httpx.get(
f"https://gateway.vlm.run/v1/models/{model_id}",
headers={"Authorization": "Bearer "},
)
print(response.json()["pricing"])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const modelId = encodeURIComponent("paddlepaddle/paddleocr-vl-1.6");
const response = await fetch(`https://gateway.vlm.run/v1/models/${modelId}`, {
headers: { Authorization: `Bearer ${process.env.VLMRUN_API_KEY}` },
});
const { pricing } = await response.json();
console.log(pricing);
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl "https://gateway.vlm.run/v1/models/paddlepaddle%2Fpaddleocr-vl-1.6" \
-H "Authorization: Bearer $VLMRUN_API_KEY"
```
Full response schema, including `pricing`.
Per-tier request quotas.
# Quickstart
Source: https://docs.vlm.run/gateway/quickstart
Prerequisites and first requests for VQA, video Q&A, and document OCR
Install the CLI, or point any OpenAI SDK at
`https://gateway.vlm.run/v1/openai`. This page covers the three most common
request shapes: visual Q\&A, video Q\&A, and document OCR. See
[Introduction](/gateway/introduction) for the rationale behind the VLM Run Gateway.
## Prerequisites
```bash Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
pip install openai
```
```bash Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
npm install openai
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
pip install -U vlmrun
```
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
export VLMRUN_API_KEY="your-api-key"
```
Base URL: `https://gateway.vlm.run/v1/openai`. Use
`Authorization: Bearer `, or `api_key="vlmrun"` for anonymous
access. See
[Authentication](/gateway/authentication) and
[Rate Limits](/gateway/rate-limits).
## Visual Q\&A
Send a question with an optional image. See
[Models](/gateway/models#visual-question-answering-vqa) for the rest of the catalog.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key=os.environ["VLMRUN_API_KEY"],
)
response = client.chat.completions.create(
model="qwen/qwen3.5-0.8b",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What is happening in this image?"},
{
"type": "image_url",
"image_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg",
},
},
],
}
],
)
print(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const response = await client.chat.completions.create({
model: "qwen/qwen3.5-0.8b",
messages: [
{
role: "user",
content: [
{ type: "text", text: "What is happening in this image?" },
{
type: "image_url",
image_url: {
url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg",
},
},
],
},
],
});
console.log(response.choices[0].message.content);
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg \
-p "What is happening in this image?" \
-m qwen/qwen3.5-0.8b
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/chat/completions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen3.5-0.8b",
"messages": [
{
"role": "user",
"content": [
{ "type": "text", "text": "What is happening in this image?" },
{
"type": "image_url",
"image_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.generation/sf-golden-gate.jpg"
}
}
]
}
]
}'
```
Chat models return their reply verbatim, with no envelope around it.
## Video Q\&A
Send a question with a hosted video URL. Tune frame sampling with
`video_fps` and `video_max_frames`. See
[Video Inputs](/gateway/multimodal-inputs#video-inputs) for all knobs.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key=os.environ["VLMRUN_API_KEY"],
)
response = client.chat.completions.create(
model="qwen/qwen3.5-0.8b",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Summarize what happens in this video."},
{
"type": "video_url",
"video_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4",
},
},
],
}
],
extra_body={"video_fps": 1.0, "video_max_frames": 8},
)
print(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const response = await client.chat.completions.create({
model: "qwen/qwen3.5-0.8b",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Summarize what happens in this video." },
{
type: "video_url",
video_url: {
url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4",
},
},
],
},
],
video_fps: 1.0,
video_max_frames: 8,
});
console.log(response.choices[0].message.content);
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4 \
-p "Summarize what happens in this video." \
-m qwen/qwen3.5-0.8b \
-e video_fps=1.0 \
-e video_max_frames=8
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/chat/completions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen3.5-0.8b",
"video_fps": 1.0,
"video_max_frames": 8,
"messages": [
{
"role": "user",
"content": [
{ "type": "text", "text": "Summarize what happens in this video." },
{
"type": "video_url",
"video_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"
}
}
]
}
]
}'
```
## Document and Image OCR
Pass a PDF as a `document_url` content part, or an image as `image_url`. Start
with `paddlepaddle/paddleocr-vl-1.6` for general-purpose OCR, and set `method`
to pick the page read (`ocr` for plain text, `markdown` for Markdown).
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import os
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.vlm.run/v1/openai",
api_key=os.environ["VLMRUN_API_KEY"],
)
response = client.chat.completions.create(
model="paddlepaddle/paddleocr-vl-1.6",
messages=[
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
},
}
],
}
],
extra_body={
"method": "ocr",
"document_dpi": 96,
},
)
print(response.choices[0].message.content)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.vlm.run/v1/openai",
apiKey: process.env.VLMRUN_API_KEY,
});
const response = await client.chat.completions.create({
model: "paddlepaddle/paddleocr-vl-1.6",
messages: [
{
role: "user",
content: [
{
type: "document_url",
document_url: {
url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf",
},
},
],
},
],
method: "ocr",
document_dpi: 96,
});
console.log(response.choices[0].message.content);
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun gw chat https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf \
-m paddlepaddle/paddleocr-vl-1.6 \
--method ocr \
-e document_dpi=96
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl https://gateway.vlm.run/v1/openai/chat/completions \
-X POST \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "paddlepaddle/paddleocr-vl-1.6",
"method": "ocr",
"document_dpi": 96,
"messages": [
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
}
}
]
}
]
}'
```
The reply is one `` block wrapping one `` block per page. Each
page declares its payload kind in `format`:
```text theme={"theme":{"light":"github-light","dark":"dark-plus"}}
UNITED STATES
SECURITIES AND EXCHANGE COMMISSION
WASHINGTON, DC 20549
FORM 8-K
```
Set `response_format={"type": "json_object"}` (CLI: `--json-mode`) to get one
JSON object instead. See
[Methods & Response Format](/gateway/methods) for the contract and
[Flexible Document OCR](/gateway/guides/document-ocr) for the rest of the pipeline.
## Next steps
The same models are available from the
[Python SDK](/sdk-reference/components/gateway) via `client.gateway`. See
[`vlmrun gw`](/cli/gateway) for `gw models`, `gw embed`, and `gw transcribe`.
Request knobs, page blocks, and streaming for PDFs.
Catalog, capabilities, and model selection.
Full request parameters and response schema.
Give any MCP-aware agent read tools for documents, audio, and video.
# Rate Limits
Source: https://docs.vlm.run/gateway/rate-limits
Rate limits for the VLM Run Gateway API
Every public OpenAI route shares **one** per-caller budget: chat completions,
audio transcriptions, embeddings, the model catalog routes, and
`GET /v1/completions/{completion_id}`. A catalog listing and an inference call
draw on the same counter. When a bucket fills, the API returns
`429 Too Many Requests`. See [Authentication](/gateway/authentication) for how
tiers are determined.
## Limits by Tier
| Tier | How it is identified | Rate limit | Bucket key |
| ---------------------- | -------------------------------------------------------------------- | ---------------------- | ---------- |
| **Anonymous** | No `Authorization` header, `Bearer vlmrun`, or an empty bearer value | 10/min, 30/hr, 100/day | Client IP |
| **Authenticated** | Valid VLM Run API key (`Authorization: Bearer `) | 240/min, 10000/hr | User ID |
| **First-party client** | `Bearer vlmrun` plus a `vlmrun-` prefixed `User-Agent` | 20/sec | Client IP |
| **M2M** | An internal VLM Run service token | 20/sec | Client IP |
The first-party bucket is for VLM Run tooling, such as `vlmrun-cli/1.2`. It
needs both signals. A `vlmrun-` prefixed `User-Agent` without the `vlmrun`
bearer token stays on the anonymous bucket.
The windows **stack**: an anonymous request has to fit all three of 10/min,
30/hr, and 100/day, and the first one to fill returns the `429`. Attribution
differs too: anonymous requests are capped per client IP, so callers behind one
NAT or proxy share a bucket, while an API key is capped per user account. See
[Authentication](/gateway/authentication#tier-resolution) for how the Gateway
resolves tiers from the `Authorization` header.
Sign up at
[app.vlm.run](https://app.vlm.run/dashboard/settings/api-keys) for a free API
key. An API key lifts you from 10 requests a minute to 240, attributes usage
to your account, and stops you sharing a bucket with other callers on the
same IP.
## Response headers
Anonymous responses carry advisory quota headers, so a client can pace itself
before it is refused:
| Header | Meaning |
| ----------------------- | --------------------------------------------------- |
| `X-RateLimit-Limit` | The size of the window that was evaluated. |
| `X-RateLimit-Remaining` | Requests left in that window. |
| `X-VLMRun-Upgrade` | The prompt to get a free API key for higher limits. |
| `Retry-After` | On a `429`, the seconds to wait. |
`Retry-After` reports the **longest** exhausted window, not the one that
tripped, so a client that waits that long does not immediately hit a second
`429`.
A `429` body names the tier that was limited:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"error": {
"type": "rate_limit_exceeded",
"message": "Rate limit exceeded. Get a free API key at https://app.vlm.run/dashboard/settings/api-keys for higher limits.",
"tier": "anonymous"
}
}
```
## Quota is not queue backpressure
A `429` also arrives when the model's request queue is full, which is
capacity, not quota: the queue holds 128 waiting requests, and past that the
Gateway refuses the call. Retry it. A model whose deployment is cold or not up
answers `503` with `code: "model_unavailable"` and `Retry-After: 30`, and a call
that runs past the inference timeout (270 seconds by default) answers `504`.
See [Error Codes](/gateway/error-codes).
## Tips
* Back off and retry after `Retry-After` seconds when you receive a `429`.
* Prefer URL-based document inputs over large base64 payloads to keep request
latency predictable under load.
* For production document workloads, use an authenticated API key so limits
apply to your account rather than a shared IP bucket.
* Read a long PDF in page ranges rather than in one call, so a single request
does not sit in the queue for minutes.
Counters live in the serving process unless the deployment sets
`VLMRT_RATE_LIMIT_STORAGE_URI` to shared storage. Without shared storage,
a caller's effective limit multiplies by the number of replicas, so treat the
published numbers as the guaranteed floor.
If you need higher limits, contact [support@vlm.run](mailto:support@vlm.run?subject=Gateway%20rate%20limit%20increase).
# Transcribing Audio
Source: https://docs.vlm.run/guides/audio-ai/guide-audio-transcription
Learn how to transcribe and analyze long-form audio.
While traditional speech-to-text models focus solely on transcription, `vlm-1` can simultaneously generate rich, structured insights from audio content. This includes transcription, chapter segmentation, topic extraction, entity recognition, and sentiment analysis - all in a single API call. These capabilities are particularly valuable for podcast analysis, interview processing, and content management systems. In this guide, we'll walk you through how to use the `audio.transcription` domain to transcribe and analyze long-form audio content.
In subsequent guides, we'll cover more advanced capabilities like topic extraction, entity recognition, and sentiment analysis.
Navigate over to the audio-transcription playground in our [playground](https://app.vlm.run/playground/audio.transcription) to see the audio transcription in action.
## Analyzing Podcast Episodes
Let's look at a podcast analysis example to see how `vlm-1` can be used to extract structured insights from audio content. In this example, we'll use `vlm-1` to transcribe a **5-hour long** Lex Fridman podcast, generating segmented chapters with start and end timestamps, and corresponding full transcript that can be used for content organization and discovery.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, GenerationConfig
# Initialize the client
client = VLMRun(api_key="")
# Submit the audio file for transcription
prediction: PredictionResponse = client.audio.generate(
file=Path("path/to/podcast_episode.mp3"),
domain="audio.transcription",
batch=True,
)
print(prediction.response.model_dump())
# Wait for the prediction to complete (with a timeout of 600 seconds)
prediction: PredictionResponse = client.predictions.wait(id=prediction.id, timeout=600)
print(prediction.response.model_dump())
```
Let's look at the few lines of code to transcribe the audio file with `vlm-1`:
## Understanding the Output
Here's an example of the output in JSON format, for the entire 5 hour podcast:
```json Example Audio Transcription [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"metadata": {
"language": null,
"content": null,
"topics": null,
"duration": 18375.22
},
"segments": [
{
"start_time": 0,
"end_time": 20.83,
"content": " The following is a conversation with Dylan Patel and Nathan Lampert. Dylan runs semi-analysis, a well-respected research and analysis company that specializes in semiconductors, GPUs, CPUs, and AI hardware in general. Nathan is a research scientist at the Allen Institute for AI,"
},
{
"start_time": 20.83,
"end_time": 47.04,
"content": " and is the author of the amazing blog on AI called Interconnects. They are both highly respected, read, and listened to by the experts, researchers, and engineers in the field of AI. And personally, I'm just a fan of the two of them. So I use the deep seek moment that shook the AI world a bit as an opportunity to sit down with them and lay it all"
},
{
"start_time": 47.04,
"end_time": 70.17,
"content": " out from DeepSeek, Open AI, Google XAI, Metanthropic to Nvidia and DSMC, and to US-Ch., China, Taiwan relations, and everything else that is happening at the cutting edge of AI. This conversation is a deep dive into many critical aspects of the AI industry. While it does get super technical,"
},
{
"start_time": 70.75,
"end_time": 91.67,
"content": " we try to make sure that it's still accessible to folks outside of the AI field by defining terms, stating important concepts explicitly, spelling out acronyms, and in general, always moving across the several layers of abstraction and levels of detail. There is a lot of hype in the media about what AI is and isn't."
},
{
"start_time": 92.53,
"end_time": 113.23,
"content": " The purpose of this podcast, in part, is to cut through the hype, through the bullshit, and the low-resolution analysis, and to discuss in detail how stuff works and what the implications are. Let me also, if I may, comment on the new OpenA.O3 mini reasoning model, the release of which we were"
},
{
"start_time": 113.23,
"end_time": 134.71,
"content": " anticipating during the conversation, and it did indeed come out right after. Its capabilities and costs are on par with our expectations, as we stated. OpenAI 03 Mini is indeed a great model, but it should be stated that Deep Seekar 1 has similar performance on benchmarks. It's still cheaper and it"
},
{
"start_time": 134.71,
"end_time": 161.17,
"content": " reveals its chain of thought reasoning, which O3Mini does not. It only shows a summary of the reasoning. Plus R1 is open weight. And O3 Mini is not. By the way, I got a chance to play with 03 Mini. And anecdotal vibe check-wise, I felt that 03 Mini, specifically O3 Mini High, is better than R1."
},
{
"start_time": 161.96,
"end_time": 182.44,
"content": " Still, for me personally, I find that Claude Sonna 3-5 is the best model for programming, except for tricky cases where I will use O1 Pro to brainstorm. Either way, many more better AI models will come, including reasoning models, both from American and Chinese companies. They will continue to shift the cost curve."
},
{
"start_time": 183.42,
"end_time": 205.54,
"content": " But the quote, deep seek moment is indeed real. I think it will still be remembered five years from now as a pivotal event in tech history, due in part to the geopolitical implications, but for other reasons too, as we discuss in detail from many perspectives in this conversation. This is the Lex Friedman podcast. To support it, please check out"
},
{
"start_time": 205.54,
"end_time": 227.12,
"content": " our sponsors in the description. And now, dear friends, here's Dylan Patel and Nathan Lambert. A lot of people are curious to understand China's deep seek AI models. So let's lay it out. Nathan, can you describe what deep seek V3 and deep seek r1r how they work how they're trained let's look at the big picture"
},
{
"start_time": 227.12,
"end_time": 249.79,
"content": " and then we'll zoom in on the details. Yeah, so DeepSeek v3 is a new mixture of experts transformer language model from Deepseek, who is based in China. They have some new specifics in the model that we'll get into. Largely, this is a open weight model, and it's a instruction model like what you would use in chat GPT."
},
{
"start_time": 250.83,
"end_time": 270.39,
"content": " They also release what is called the base model, which is before these techniques of post-training. Most people use instruction models today, and those are what served in all sorts of applications. This was released on, I believe, December 26th or that week. And then weeks later on January 20th,"
},
{
"start_time": 270.39,
"end_time": 293.66,
"content": " DeepSeek released DeepSeek R1, which is a reasoning model, which really accelerated a lot of this discussion. This reasoning model has a lot of overlapping training steps to deep seek v3. And it's confusing that you have a base model called V3, that you do something to get a chat model, and then you do some different things to get a reasoning model."
},
{
"start_time": 293.66,
"end_time": 314.76,
"content": " I think a lot of the AI industry is going through this challenge of communications right now, where Open AI makes fun of their own naming schemes. They have GPT40. They have GPT4O. They have OpenAI01. And there's a lot of types of models. So we're going to break down what each of them are. There's a lot of technical specifics on training and go through them high level to specific and kind of go through each of them."
},
{
"start_time": 314.76,
"end_time": 346,
"content": " There's so many places we can go here, but maybe let's go to open weights first. What does it mean for model to be open weights, and what are the different flavors of open source in general? Yeah, so this discussion has been going on for a long time in AI. It became more important since ChatG chat GPT or more focal since chat to BT at the end of 2022. Open weights is the accepted term for when model weights of a language model are available on the internet for people to download. Those weights can have different licenses, which is effectively the terms by which you can use the model."
},
{
"start_time": 346,
"end_time": 371.66,
"content": " There are licenses that come from history and open source software. There are licenses that are designed by companies specifically. All of Lama, Deepseek, Kwan, Mistral, these popular names in open weight models have some of their own licenses. It's complicated because not all the same models have the same terms. The big debate is on what makes a model open weight."
},
{
"start_time": 371.66,
"end_time": 391.94,
"content": " It's like, why are we saying this term? It's kind of a mouthful. It sounds close to open source, but it's not of a mouthful it sounds close to open source but it's not the same there's still a lot of debate on the definition and soul of open source AI. Open source software has a rich history on freedom to modify, freedom to take on your own, freedom for many restrictions on how you would use the software and what"
},
{
"start_time": 391.94,
"end_time": 416.79,
"content": " that means for AI is still being defined. So for what I do, I work at the Allen Institute for AI. We're a nonprofit. We want to make AI open for everybody, and we try to lead on what we think is truly open source. There's not full agreement in the community, but for us, that means releasing the training data, releasing the training code, and then also having open weights like this. And we'll get into the details of the models"
},
{
"start_time": 416.79,
"end_time": 443.92,
"content": " and again and again as we try to get deeper into how the models were trained, we will say things like the data processing, data filtering, data quality is the number one determinant of the model quality, and then a lot of the training code is the determinant on how long it takes to train and how faster experimentation is. So without fully open source models where you have access to this data, it is hard to know, or it's harder to replicate."
},
{
"start_time": 443.92,
"end_time": 466.18,
"content": " So we'll get into cost numbers for deep seek v3 on mostly GPU hours and how much you could pay to rent those yourselves, but without the data, the replication cost is going to be far, far higher. And same goes for the code. We should also say that this is probably one of the more open models out of the frontier models. So like in this full spectrum"
},
{
"start_time": 466.18,
"end_time": 487.12,
"content": " where probably the fullest open source, like you said, open source, like you said, open code, open data, open weights. This is not open code. This is probably not open data. And this is open weights and the licensing is MIT license or it's I mean there's"
},
{
"start_time": 487.12,
"end_time": 509.66,
"content": " some nuance in the different models, but it's towards the free, in terms of the open source movement, these are the kind of the good guys. Yeah, Deepseek is doing fantastic work for disseminating understanding of AI. Their papers are extremely detailed in what they do. And for other teams around the world, they're very actionable in terms of improving your own training techniques."
},
{
"start_time": 509.66,
"end_time": 537.27,
"content": " And we'll talk about licenses more. The DeepSeek R1 model has a very permissive license. It's called the MIT license. That effectively means there's no downstream restrictions on commercial use. There's no use case restrictions. You can use the outputs from the models to create synthetic data. And this is all fantastic. I think the closest peer is something like Lama, where you have the weights and you have a technical report and the technical report"
},
{
"start_time": 537.27,
"end_time": 557.45,
"content": " is very good for Lama. One of the most read PDFs of the year last year is the Lama 3 paper, but in some ways it's slightly less actionable. It has less details on the training specifics, less plots, and so on. And the Lama 3 license is more restrictive than MIT. And then between the Deep Sea custom license and the llama license. We could get into this whole rabbit hole."
},
{
"start_time": 557.45,
"end_time": 582.5,
"content": " I think we'll make sure we want to go down the license rabbit hole before we do specifics. Yeah, and I mean, so it should be stated that one of the implications that Deep Seek puts pressure on Lama and everybody else on Open AI to push towards open source. And that's the other side of open source that you mentioned is how much is published in detail about it. So how open are you with the sort of the insights behind the code."
},
{
"start_time": 582.92,
"end_time": 603.38,
"content": " So, like, how good is the technical reports? Are they hand wavy, or is there actual details in there? And that's one of the things that DeepSeek did well is they published a lot of the details. Yeah, especially in the DeepSeek V3, which is their pre-training paper, they were very clear that they are doing interventions on the technical stack that go at many different levels."
},
{
"start_time": 603.56,
"end_time": 625,
"content": " For example, to get highly efficient training, they're making modifications at or below the Kuda layer for Nvidia chips. I have never worked there myself and there are a few people in the world that do that very well and some of them are at deep seek and these types of people are at deep seek and leading American frontier labs, but there are not many places."
},
{
"start_time": 625,
"end_time": 647.42,
"content": " To help people understand the other implication of open weights. Just, you know, there's a topic will return to often here. So there's a fear that China, the nation, might have interest in stealing American data, violating privacy of American citizens. What can we say about open weights"
},
{
"start_time": 647.42,
"end_time": 669.36,
"content": " to help us understand what the weights are able to do in terms of stealing people's data. Yeah. So these weights that you can download from Hugging Face or other platforms are very big matrices of numbers. You can download them to a computer in your own house that has no internet, and you can run this model and you're totally control of your data."
},
{
"start_time": 669.36,
"end_time": 692.34,
"content": " That is something that is different than how a lot of language model usage is actually done today, which is mostly through APIs, where you send your prompt to GPUs run by certain companies. And these companies will have different distributions and policies on how your data is stored, if it is used to train future models, where it is stored, if it is encrypted, and so on. So the open weights are you have your fate of data in your own hands and"
},
{
"start_time": 692.34,
"end_time": 712.76,
"content": " that is something that is deeply connected to the soul of open source. So it's not the model that steals your data. It's Clover's hosting the model, which could be China, if you're using the deep seek app, or it could be perplexity. You're trusting them with your data or Open AI, you trust them with your data. And some of these are American companies,"
},
{
"start_time": 712.76,
"end_time": 738.37,
"content": " some of these are Chinese companies, but the model itself is not doing the stealing. It's the host. All right. So back to the basics. What's the difference between DeepSeek V3 and DeepSeek R1? Can we try to like lay out the confusion potential? Yes. So for one, I have very understanding of many people being confused by these two model names. So I would say the best way to"
},
{
"start_time": 738.37,
"end_time": 759.55,
"content": " think about this is that when training a language model, you have what is called pre-training, which is when you're predicting the large amounts of mostly internet text. You're trying to predict the next token. And what to know about these new deep seek models is that they do this internet large-scale pre-training wants to get what is called deep seek v3 base. This is a base model."
},
{
"start_time": 760.15,
"end_time": 780.21,
"content": " It's just going to finish your sentences for you. It's going to be harder to work with than chatypT. And then what DeepSeek did is they've done two different post-training regimes to make the models have specific desirable behaviors. So what is the more normal model in terms of the last few years of AI, an instruct"
},
{
"start_time": 780.21,
"end_time": 803,
"content": " model, a chat model, a quote-unquote aligned model, a help-pocket model, a chat model, a quote unquote aligned model, a helpful model. There are many ways to describe this is more standard post-training. So this is things like instruction tuning, reinforce learning from human feedback. We'll get into some of these words. And this is what they did to create the deep seek V3 model. This was the first model to be released and it is very high performance."
},
{
"start_time": 803,
"end_time": 824.99,
"content": " It's competitive with GPD4, Lama 405B, so on. And then when this release was happening, we don't know their exact timeline or soon after. They were finishing the training of a different training process from the same next token prediction based model that I talked about, which is when this new reasoning training that people have heard about comes in"
},
{
"start_time": 824.99,
"end_time": 846.17,
"content": " in order to create the model that is called DeepSeek R1. The R through this conversation is good for grounding for reasoning, and the name is also similar to Open AIs 01, which is the other reasoning model that people have heard about. And we'll have to break down the training for R1 in more detail because for one, we have a paper detailing it, but also it is a far newer set of"
},
{
"start_time": 846.17,
"end_time": 871.66,
"content": " techniques for the AI community. So it's a much more rapidly evolving area of research. Maybe we should also say the big two categories of training of pre-training and post-training, these umbrella terms that people use. So what is pre-training and what is post-training? And what are the different flavors of things underneath post-training umbrella? Yeah, so pre-training, I'm using some of the same words that really get the message across"
},
{
"start_time": 871.66,
"end_time": 892.86,
"content": " is you're doing what is called auto-regressive prediction to predict the next token in a series of documents. This is done over standard practice is trillions of tokens, So this is a ton of data that is mostly scraped from the web. And some of DeepSeaks earlier papers, they talk about their training data being distilled for math."
},
{
"start_time": 892.86,
"end_time": 917.57,
"content": " I shouldn't use this word yet, but taken from Common Crawl, and that's a public access that anyone listening to this could go download data from the Common Crawl website. This is a crawler that is maintained publicly. Yes, other tech companies eventually shift to their own crawler, and DeepSeak likely has done this as well as most frontier labs do. But this sort of data is something that people can get started with and you're just predicting text in a series of documents."
},
{
"start_time": 918.29,
"end_time": 944.99,
"content": " This can be scaled to be very efficient. And there's a lot of numbers that are thrown around in AI training, like how many floating point operations or flops are used and then you can also look at how many hours of these GPUs that are used. And it's largely one loss function taken to a very large amount of compute usage. You set up relay efficient systems."
},
{
"start_time": 944.99,
"end_time": 965.99,
"content": " And then at the end of that, you have this base model, and pre-training is where there is a lot more of complexity in terms of how the process is emerging or evolving and the different types of training losses that you will use. I think this is a lot of techniques grounded in the natural language processing literature."
},
{
"start_time": 965.99,
"end_time": 987.43,
"content": " The oldest technique, which is still used today, is something called instruction tuning or also known as supervised fine tuning. These acronyms will be IFT or SFFT that people really go back and forth throughout them and I will probably do the same, which is where you add this formatting to the model where it knows to take a question that is like, explain"
},
{
"start_time": 987.43,
"end_time": 1012.48,
"content": " the history of the Roman Empire to me. And or something you'll sort of question you'll see on Reddit or Stack Overflow, and then the model will respond in a information dense but presentable manner. The core of that formatting is in this instruction tuning phase. And then there's two other categories of loss functions that are being used today. One I will classify as preference fine tuning. Preference fine tuning is a generalized term for"
},
{
"start_time": 1012.48,
"end_time": 1033.66,
"content": " what came out of reinforcement learning from human feedback, which is RLHF. This reinforcement learning from human feedback is credited as the technique that helped chat GPT break through. It is a technique to make the responses that are nicely formatted like these Reddit answers more in tune with what a human would like to read."
},
{
"start_time": 1033.66,
"end_time": 1054.92,
"content": " This is done by collecting Paralyzed preferences from actual humans out in the world to start, and now AIs are also labeling this data and we'll get into those tradeoffs. And you have this kind of contrastive loss function between a good answer and a bad answer. And the model learns to pick up these trends. There's different implementation ways. You have things called reward models."
},
{
"start_time": 1055.07,
"end_time": 1079.25,
"content": " You could have direct alignment algorithms. There's a lot of really specific things you can do, but all of this is about fineuning to human preferences. And the final stage is much newer and we'll link to what is done in R1 and these reasoning models is, I think OpenAI's name for this. They had this new API in the fall, which they called the reinforcement fine-tuning API. This is the idea that you use the techniques"
},
{
"start_time": 1079.25,
"end_time": 1099.33,
"content": " of reinforcement learning, which is a whole framework of AI. There's a deep literature here. To summarize, it's often known as trial and error learning, or the subfield of AI where you're trying to make sequential decisions in a certain potentially unpotentially noisy environment. There's a lot of ways we could go down that. But fine-tuning"
},
{
"start_time": 1099.33,
"end_time": 1122.21,
"content": " language models where they can generate an answer and then you check to see if the answer matches the true solution. For math or code, you have an exactly correct answer for math. You can have unit tests for code. And what we're doing is we are checking the language models work, and we're giving it multiple opportunities on the same questions to see if it is right. And if you keep doing this, the models can learn to improve in verifiable domains"
},
{
"start_time": 1122.21,
"end_time": 1145.71,
"content": " to a great extent. It works really well. It's a newer technique in the academic literature. It's been used at Frontier labs in the U.S. that don't share every detail for multiple years. So this is the idea of using reinforcement learning with language models, and it has been taking off, especially in this deep seek moment. And we should say that there's a lot of exciting stuff going on on the, again, across the stack,"
},
{
"start_time": 1145.79,
"end_time": 1165.81,
"content": " but the post-training probably this year's going to be a lot of interesting developments in the post-training. We'll talk about it. I almost forgot to talk about the difference between DeepSeek V3 and r1 on the user experience side so forget the technical stuff forget all that just people that don't know anything about AI, they show up. Like, what's the actual experience?"
},
{
"start_time": 1165.97,
"end_time": 1186.67,
"content": " What's the use case for each one when they actually like type and talk to it what is each good at and that kind of thing so let's start with deep seek v3 again it's what more people would have tried something like it. You ask it a question. It'll start generating tokens very fast, and those tokens will look like a very human legible answer. It'll be some sort of markdown list."
},
{
"start_time": 1186.87,
"end_time": 1206.99,
"content": " It might have formatting to help you draw to the core details in the answer, and it'll generate tens to hundreds of tokens. A token is normally a It tends to hundreds of tokens. A token is normally a word for common words or a subword part in a longer word. And it'll look like a very high quality Reddit or Stack Overflow answer."
},
{
"start_time": 1206.99,
"end_time": 1227.47,
"content": " These models are really getting good at doing these across a wide variety of domains. Even things that if you're an expert, things that are close to the fringe of knowledge, they will still be fairly good at. Cutting edge AI topics that I do research on, these models are capable for study aid and they're regularly updated."
},
{
"start_time": 1228.66,
"end_time": 1248.66,
"content": " Where this changes is with the DeepSeek R1, what is called these reasoning models, is when you see tokens coming from these models to start, it will be a large chain of thought process. We'll get back to chain of thought in a second, which looks like a lot of tokens where the model is explaining the problem. The model will often break down the problem and be like,"
},
{
"start_time": 1248.66,
"end_time": 1271.28,
"content": " okay, they ask me for this. Let's break down the problem. I'm going to need to do this. And you'll see all of this generating from the model. It'll come very fast in most user experiences. These APIs are very fast, so you'll see a lot of tokens. A lot of words show up really fast. It'll keep flowing on the screen, and this is all the reasoning process, and then eventually the model will change its tone in R1 and it'll write the answer, where it summarizes its"
},
{
"start_time": 1271.28,
"end_time": 1293.66,
"content": " reasoning process and writes a similar answer to the first types of model. But in DeepSeek's case, which is part of why this was so popular, even outside the AI community is that you can see how the language model is breaking down problems. And then you get this answer on a technical side. They train the model to do this specifically, where they have a section, which is reasoning,"
},
{
"start_time": 1293.66,
"end_time": 1316.01,
"content": " and then it generates a special token, which is probably hidden from the user most of the time, which says, okay, I'm starting to answer. So the model is trained to do this two-stage process on its own. If you use a similar model in, say, OpenAI's user interface is trying to summarize this process for you nicely by kind of showing the sections that the model is doing, and it'll kind of click through."
},
{
"start_time": 1316.01,
"end_time": 1340.52,
"content": " It'll say breaking down the problem, making X calculation, cleaning the result, and then the answer will come for something like Open AI. Maybe it's useful here to go through like an example of a deep seek R1 reasoning. Yeah, so if you're looking at the screen here, what you'll see is a screenshot of the deep seek chat app and at the top is thought for 151 7 seconds with the drop down arrow"
},
{
"start_time": 1340.52,
"end_time": 1364.71,
"content": " underneath that if we were in an app that we were running, the drop-down arrow would have the reasoning. So in this case, the specific question, which, you know, I'm philosophically slash pothead inclined. So this is asking deep, deep see car one for one truly novel insight about humans. And it reveals the reasoning and basically the truly novel aspect"
},
{
"start_time": 1364.71,
"end_time": 1386.13,
"content": " is what's pushing the reasoning to constantly sort of the model asking itself is this truly novel. So it's actually challenging itself to be more novel, more counterintuitive, more less cringe, I suppose. So some of the reasoning says, this is just snapshots. Alternatively, humans have a unique meta-emotion where they feel emotions about their own emotions,"
},
{
"start_time": 1386.13,
"end_time": 1406.39,
"content": " e.g. feeling guilty about being angry. This, of course, of emotional layering creates complex motivational drives that don't exist in other animals. The insight is that human emotions are nested. So it's like, it's reasoning through how humans feel emotions. It's reasoning about meta emotions. You're gonna have pages and pages of this. It's almost too much to actually read, but it's nice to skim as it's coming."
},
{
"start_time": 1406.39,
"end_time": 1428.09,
"content": " It's a stream of con- it's a James Joyce like stream of consciousness. And then it goes, wait, the user wants something that's not seen anywhere else. Let me dig deeper. And consider the human ability to hold contradictory beliefs simultaneously. Cognitive dissonance is known, but perhaps the function is to allow flexible adaptation, so on and so forth. I mean, that"
},
{
"start_time": 1428.09,
"end_time": 1454.26,
"content": " really captures the public imagination that holy shit, this isn't, I mean, intelligence slash almost like an inkling of sentience, because like you're thinking through yourself reflecting, you're deliberating. And the final result of that after 157 seconds is humans instinctively converts selfish desires into cooperative systems by collectively"
},
{
"start_time": 1454.26,
"end_time": 1484.97,
"content": " pretending abstract rules, money, laws, rights, are real. These shared hallucinations act as, quote, games, competition is secretly redirected to benefit the group, turning conflict into society's fuel. Pretty profound. I mean, you know. There's a potential digression, but a lot of people have found that these reasoning models can sometimes produce much more eloquent text. That is a, at least interesting example, I think, depending on how open-minded you are,"
},
{
"start_time": 1484.97,
"end_time": 1506.21,
"content": " you find language models interesting or not, and there's a spectrum there. Well, I mean, some of the, we'll talk about different benchmarks and songs, but some is just a vibe. Like that in itself is a, let's say, quote, fire tweet. Yeah. If I, if I, if I, I, I'm trying to produce something, something that were people like, oh shit. Okay, so that's Chanathaw. We'll probably return to it more."
},
{
"start_time": 1507.61,
"end_time": 1527.29,
"content": " How were they able to achieve such low cost on the training and the inference? Maybe you could talk the training first. Yeah. So there's two main techniques that they implemented that are probably the majority of their efficiency. And then there's a lot of implementation details that maybe we'll gloss over or get into later"
},
{
"start_time": 1527.29,
"end_time": 1551.66,
"content": " that sort of contribute to it. But those two main things are, one is they went to a mixture of experts model, which we'll define in a second. And then the other thing is that they invented this new technique called MLA, latent attention. Both of these are big deals. A mixture of experts is something that's been in the literature for a handful of years. And Open AI with GPT4 was the first one to productize a mixture of experts model."
},
{
"start_time": 1551.66,
"end_time": 1573.7,
"content": " And what this means is when you look at the common models around that most people have been able to interact with that are open, think Lama. Lama is a dense model, i.e. every single parameter or neuron is activated as you're going through the model for every single token you generate. Right. Now, with a mixture of experts model, you don't do that, right?"
},
{
"start_time": 1573.7,
"end_time": 1594.92,
"content": " How does the human actually work right? It's like, oh, well, my visual cortex is active when I'm thinking about, you know, vision tasks and like, you know, other things, right? My amygdala is when I'm scared, right? These different aspects of your brain are focused on different things. A mixture of experts' model attempts to approximate this to some extent. It's nowhere close to what a brain architecture is, but different portions of the model activate."
},
{
"start_time": 1605.47,
"end_time": 1616.36,
"content": " You'll have a set number of experts in the model and a set number that are activated each time, and this dramatically reduces both your training and inference costs. Because now you're, you know, if you think about the parameter count as the sort of total embedding space for all of this knowledge that you're compressing down during training. When you're embedding this data in,"
},
{
"start_time": 1616.36,
"end_time": 1638.84,
"content": " instead of having to activate every single parameter, every single time you're training or running inference, now you can just activate a subset. And the model will learn which expert to route to for different tasks. And so this is a humongous innovation in terms of, hey, I can continue to grow the total embedding space of parameters. And so deep-seek's model is, you know, 600-something billion parameters, right? Relative to"
},
{
"start_time": 1638.84,
"end_time": 1663.53,
"content": " Lama 405B, it's four or five billion parameters, right? Lama to Lama, relative to Lama 70B, it's 70 billion parameters, right? So this model technically has more embedding space for information, right, to compress all of the world's knowledge that's on the internet down, but at the same time, it is only activating around 37 billion of the parameters. So only 37 billion of these parameters actually need to be computed every single time you're training data or inferencing data out of it."
},
{
"start_time": 1664.03,
"end_time": 1686.49,
"content": " And so versus, versus again, a Lama model, 70 billion parameters must be activated or 405 billion parameters must be activated. So you've dramatically reduced your compute cost when you're doing training and inference with this mixture of experts architecture. So we break down where it actually applies and go into the transformer. Is that useful? Let's go. Let's go into the transformer. So the transformer is a thing that is talked about a lot,"
},
{
"start_time": 1686.49,
"end_time": 1713.8,
"content": " and we will not cover every detail. Essentially, the transformer is built on repeated blocks of this attention mechanism, and then a traditional dense, fully connected, multi-layer perceptron, whatever word you want to use for your normal neural network and you alternate these blocks, there's other details, and where a mixture of experts is applied is that this dense model. The dense model holds most of the weights if you count them in a transformer model."
},
{
"start_time": 1713.8,
"end_time": 1734.5,
"content": " So you can get really big gains from those mixture of experts on parameter efficiency at training and inference because you get this efficiency by not activating all of these parameters. We should also say that a transformer is a giant neural network. Yeah. And then there's four is a giant neural network. Yeah. And then there's for 15 years now, there's what's called the Deep Learning Revolution."
},
{
"start_time": 1737.12,
"end_time": 1755.98,
"content": " Network's gotten larger and larger in a certain point. The scaling laws appeared where people realized... This is a scaling law shirt, by the way. Representing scaling laws, where it became more and more formalized that bigger is better across multiple dimensions of what bigger means so and but these are all sort of neural networks we're talking about,"
},
{
"start_time": 1756.42,
"end_time": 1776.32,
"content": " and we're talking about different architectures to construct these neural networks such that the training and the inference on them is super efficient. Yeah. Every different type of model has a different scaling law for it, which is effectively for how much compute you put in, the architecture will get to different levels of performance at test tasks."
},
{
"start_time": 1776.32,
"end_time": 1797.48,
"content": " And mixture of experts is one of the ones at training time, even if you don't consider the inference benefits, which are also big. At training time, your efficiency with your GPUs is dramatically improved by using this architecture if it is well implemented. So you can get effectively the same performance model and evaluation scores with numbers like 30% less compute."
},
{
"start_time": 1800.58,
"end_time": 1825.35,
"content": " I think there's going to be a wide variation depending on your implementation details and stuff. But it is just important to realize that this type of technical innovation is something that gives huge gains. And I expect most companies that are serving their models to move to this mixture of experts implementation. Historically, the reason why not everyone might do it is because it's an implementation complexity, especially when doing these big models. So this is one of the things that Deep Seek gets credit for is they do this extremely"
},
{
"start_time": 1825.35,
"end_time": 1853.16,
"content": " well. They do mixture of experts extremely well. This architecture for what is called deep seek MOE, MOE is the shortened version of mixture of experts, is multiple papers old. This part of their training infrastructure is not new to these models. Alone, and same goes for what Dylan mentioned with multi-head latent attention. It's all about reducing memory usage during inference and same things during training by using some fancy low-rank approximation math."
},
{
"start_time": 1853.16,
"end_time": 1878.44,
"content": " If you get into the details with this latent attention, it's one of those things I look at. It's like, OK, they're doing really complex implementations, because there's other parts of language models such as embeddings that are used to extend the context length. The common one that DeepSeek used is rotary, positional, and penings, which is called rope. And if you want to use rope with a normal MOE. It's kind of a sequential thing. You take these, you take two of the"
},
{
"start_time": 1878.44,
"end_time": 1903.49,
"content": " attention matrices and you rotate them by a complex value rotation, which is a matrix multiplication. With deep seek's MLA, with this new attention architecture, they need to do some clever things because they're not set up the same and it just makes the implementation complexity much higher. So they're managing all of these things. And these are probably the sort of things that Open AI, these closed labs are doing. We don't know if they're doing the exact same techniques, but they actually shared them"
},
{
"start_time": 1903.49,
"end_time": 1924.99,
"content": " with the world, which is really nice to feel like this is the cutting edge of efficient language model training. And some of this requires low-level engineering. Just is a giant mess and trickery. So as I understand, that went below Kuda. So they go super low programming of GPUs. Effectively, Nvidia builds this library called nickel, right?"
},
{
"start_time": 1924.99,
"end_time": 1945.63,
"content": " In which, you know, when you're training a model, you have all these communications between every single layer of the model, and you may have over 100 layers. What does the nickel stand for? It's NCCL. Invidia Communications Collective's Library. Nice. And so when you're training a model, right, you're training a model right you're going to have all these all reduces and all gathers right"
},
{
"start_time": 1945.63,
"end_time": 1967.43,
"content": " between each layer between the multi-lier perceptron or feed forward network and the attention mechanism, you'll have, you'll have basically the model synchronize, right? Or you'll have all the, you'll have all reducer and all gather. And this is a communication between all the GPUs in the network, whether it's in training or inference. So, Nvidia has a standard library. This is one of the reasons why it's really difficult to use"
},
{
"start_time": 1967.43,
"end_time": 1990.67,
"content": " anyone else's hardware for training is because no one's really built a standard communications library. And in Nvidia's done this at a sort of a higher level, right? A deep seek because they have certain limitations around the GPUs that they have access to, the interconnects are limited to some extent by the restrictions of the GPUs that were shipped into China legally, not the ones that are smuggled, but legally shipped in that they used to train this"
},
{
"start_time": 1990.67,
"end_time": 2010.73,
"content": " model, they had to figure out how to get efficiencies, right? And one of those things is that instead of just calling the Nvidia library nickel, right? They instead created their their they scheduled their own communications which which the last some of the labs do right um emetta talked about in Lama 3 how they made their own custom version of nickel."
},
{
"start_time": 2011.19,
"end_time": 2032.57,
"content": " They didn't talk about the implementation details. This is some of what they did. Probably not as well as, maybe not as well as DeepSeek, because Deep Seek, you know, necessity is the mother of innovation. And they had to do this. Whereas in the case, you know, open AI has people that do this sort of stuff, anthropic, et cetera. But, you know, DeepSeek certainly did it publicly, and they may have done it even better because they were"
},
{
"start_time": 2032.57,
"end_time": 2055.74,
"content": " gimped on a certain aspect of the chips that they have access to. And so they scheduled communications, you know, by scheduling specific SMs, SMs you could think of as like the core on a GPU, right? So there's hundreds of cores, or there's, you know, a bit over 100 cores, SMs on a GPU, and they were specifically scheduling, hey, which ones are running the model, which ones are doing all reduce,"
},
{
"start_time": 2055.82,
"end_time": 2075.8,
"content": " which one are doing all gather, right? And they would flip back and forth between them, and this requires extremely low- level programming. This is what Nickel does automatically, or other Nvidia libraries handle this automatically, usually. Yeah, exactly. And so technically they're using, you know, PtX, which is like sort of like, you could think of it as like an assembly type language. It's not exactly that or instruction set, right?"
},
{
"start_time": 2075.86,
"end_time": 2098.2,
"content": " Like coding directly to assembly or instruction set. It's not exactly that, but that's still part of technically CUDA, but it's like, do I want to write in Python, you know, PiTorch equivalent and call Invidia libraries, do you want to go down to the C level, right? Or you know, encode even lower level or do I want to go all the way down to the assembly or ISO level? And there are cases where you go all the way down there at the very big labs, but most companies"
},
{
"start_time": 2098.2,
"end_time": 2118.66,
"content": " just do not do that, right? Because it's a waste of time and the efficiency gains you get are not worth it. But deep seek's implementation is so complex, right? Especially with their mixture of experts, right? People have done mixture of experts, but they're generally 8, 16 experts, right? And they activate too. So, you know, one of the words, like, you like to use as like sparsity factor, right,"
},
{
"start_time": 2118.66,
"end_time": 2139.36,
"content": " or usage, right? So you might have four, you know, one fourth of your model, activate, right? And that's what mistral's mixtral model, right? Their model that really catapulted them to like oh my god they're really really good um open ai has also had models that are MOE and and so have all the other labs that are MOE, and so have all the other labs that are major closed. But what DeepSeek did that"
},
{
"start_time": 2139.36,
"end_time": 2159.68,
"content": " maybe only the leading labs have only just started recently doing is have such a high sparsity factor, right? It's not one fourth of the model, right? Two out of eight experts activating every time you go through the model, it's eight out of 256. And there's different implementations for mixture of experts where you can have some of these experts that are always activated, which this just"
},
{
"start_time": 2159.68,
"end_time": 2181.48,
"content": " looks like a small neural network and then all the tokens go through that and then they also go through some that are selected by this routing mechanism. And one of the innovations in DeepSeaks architecture is that they changed the routing mechanism in mixture of expert models. There's something called an auxiliary loss, which effectively means during training,"
},
{
"start_time": 2181.48,
"end_time": 2204.15,
"content": " you want to make sure that all of these experts are used across the tasks that the model sees. Why there can be failures and mixture of experts is that when you're doing this training, the one objective is token prediction accuracy. And if you just let turning go with a mixture of expert model on your own, it can be that the model learns to only use a subset of the experts."
},
{
"start_time": 2204.73,
"end_time": 2224.99,
"content": " And in the MOE literature, there's something called the auxiliary loss, which helps balance them. But if you think about the loss functions of deep learning, this even connects to the bitter lesson, is that you want to have the minimum inductive bias in your model to let the model learn maximally. And this auxiliary loss, this balancing across experts,"
},
{
"start_time": 2224.99,
"end_time": 2247.99,
"content": " could be seen as intention with the prediction accuracy of the tokens. So we don't know the exact extent that the deep seek M-OE change, which is instead of doing an auxiliary loss, they have an extra parameter in their routing, which after the batches, they update this parameter to make sure that the next batches all have a similar use of experts. And this type of change can be big, it can be small, but they add up over time."
},
{
"start_time": 2247.99,
"end_time": 2269.65,
"content": " And this is the sort of thing that just points small, but they add up over time. And this is the sort of thing that just points to them innovating. And I'm sure all the labs that are training big MEOs are looking at this sort of things, which is getting away from the auxiliary loss. Some of them might already use it. But you just keep you keep accumulating gains. And we'll talk about the philosophy of training and how you organize these organizations. And a lot of it is just compounding small improvements over time in your data,"
},
{
"start_time": 2269.65,
"end_time": 2291.85,
"content": " in your architecture, and your post-training, and how they integrate with each other. DeepSeek does the same thing, and some of them are shared or a lot we have to take them on face value that they share their most important details i mean the architecture and the weights are out there, so we're seeing what they're doing. And it adds up. Going back to sort of the like efficiency and complexity point, right? It's 32 versus a four, right, for like Mixed"
},
{
"start_time": 2291.85,
"end_time": 2312.81,
"content": " Strahl and other MOE models that have been publicly released. So this ratio is extremely high. And sort of what Nathan was getting at there was, when you have such a different level of sparsity, You can't just have every GPU have the entire model, right? The model's too big, there's too much complexity there. So you have to split up the model with different types of parallelism, right? And so you might have different experts"
},
{
"start_time": 2312.81,
"end_time": 2333.17,
"content": " on different GPU nodes. But now what happens when this set of data that you get, all of it looks like this one way and all of it should route to one part of my model, right? So when all of it routes to one part of the model, then you can have this overloading of a certain set of the GPU resources"
},
{
"start_time": 2333.17,
"end_time": 2357.58,
"content": " or a certain set of the GPUs, and then the rest of the training network sits idle because all of the tokens are just routing to that. So this is the biggest complexity, one of the big complexities with running a very, you know, sparse mixture of experts model, i.e., you know, this 32 ratio versus this four ratio is that you end up with so many of the experts just sitting there idle. So how do I load balance between them? How do I schedule the communications"
},
{
"start_time": 2357.58,
"end_time": 2379.08,
"content": " between them? This is a lot of the like extremely low level detailed work that they figured out in the public first and potentially like second or third in the world and maybe even first in some cases. What lesson do you in the direction of the better lesson do you take from all of this? Where is this going to better lesson do you take from all of this? Is this going to be the direction where a lot of the gain is going to be,"
},
{
"start_time": 2379.18,
"end_time": 2400.32,
"content": " which is this kind of low level optimization, or is this a short-term thing where the biggest gains will be more on the algorithmic high-level side of post-training. Is this like a short-term leap because they've figured out like a hack because constraints, necessities the mother of invention, or is there still a lot of gains?"
},
{
"start_time": 2400.32,
"end_time": 2424,
"content": " I think we should summarize what the bitter lesson actually is about. Is that the bitter lesson, essentially, if you paraphrase it, is that the types of training that will win out in deep learning as we go are those methods that are which are scalable in learning and search is what it calls out. And this scale word gets a lot of attention in this."
},
{
"start_time": 2424.62,
"end_time": 2446.5,
"content": " The interpretation that I use is effectively to avoid adding in the human priors to your learning process. And if you read the original essay, this is what it talks about, is how researchers will try to come up with clever solutions to their specific problem that might get them small gains in the short term while"
},
{
"start_time": 2446.5,
"end_time": 2477.02,
"content": " simply enabling these deep learning systems to work efficiently and for these bigger problems in the long term might be more likely to scale and continue to drive success. And therefore, we were talking about relatively small implementation changes to the mixture of experts model. And therefore, it's like, okay, we will need a few more years to know if one of these are actually really crucial to the bitter lesson, but the bitter lesson is really this long-term arc of how simplicity can often win."
},
{
"start_time": 2477.32,
"end_time": 2499.38,
"content": " And there's a lot of sayings in the industry, like the models just want to learn. You have to give them the simple lost landscape where you put compute through the model and they will learn and getting barriers out of the way. That's where the power, something like nickel comes in, where standardized code that could be used by a lot of people to create simple innovations that can scale,"
},
{
"start_time": 2499.38,
"end_time": 2523.52,
"content": " which is why the hacks, I imagine that the code base for DeepSeek is probably a giant mess. I'm sure they have, DeepSeek definitely has code bases that are extremely messy where they're testing these new ideas. Multi-head latent attention. Probably could start in something like a Jupiter notebook or somebody tries something on a few GPUs and that is really messy. But the stuff that trains the deep seek V3 and deep seek R1,"
},
{
"start_time": 2523.52,
"end_time": 2545.82,
"content": " those libraries, if you were to present them to us, I would guess are extremely high quality code. So high quality readable code. Yeah. I think there is one aspect to note though, right? Is that there is the general general ability for that to transfer across different types of runs right you? You may make really, really high quality code for one specific model"
},
{
"start_time": 2545.82,
"end_time": 2566.68,
"content": " architecture at one size. And then that is not transferable to, hey, when I make this architecture tweak, everything's broken again, right? Like that's something that could be, you know, with their specific low level coding of like scheduling SMs is specific to this model architecture and size, right? And whereas like, Invidia's collectives library is more like,"
},
{
"start_time": 2566.68,
"end_time": 2586.74,
"content": " hey, it'll work for anything, right? You want to do an all reduce? Great. I don't care what your model architecture is. It'll work. And you're giving up a lot of performance when you do that in many cases, but it's worthwhile for them to do the specific optimization for the specific run, given the constraints that they have regarding compute. I wonder how stressful it is to like,"
},
{
"start_time": 2587.48,
"end_time": 2607.98,
"content": " you know, these frontier models, like initiate training, like to have the code to push the button that like you're now spending a large amount of money and time to train this like there must i mean there must be a lot of innovation on the debugging stage of like making sure there's no issues"
},
{
"start_time": 2607.98,
"end_time": 2631.05,
"content": " that you're monitoring and visualizing every aspect of the training, all that kind of stuff. When people are training, they have all these various dashboards, but like the most simple one is your loss, right? And it continues to go down, but in reality, especially with more complicated stuff like M-O-E, the biggest problem with it, or FP8 training, which is another innovation, you know, going to a lower precision number format, i.e. less accurate, is that you end up"
},
{
"start_time": 2631.05,
"end_time": 2657.4,
"content": " with lost bikes, right? And no one knows why the lost spike happened. And for a long of them, you do. Some of them are bad data. Can I give AI2's example of what blew up earlier models is a subreddit called Microwave Gang. We love to shout out the South. It's a real thing. You can pull up microwave gang. Essentially, it's a subreddit where everybody makes posts that are just the letter M so it's like mm so there's extremely long sequences of the letter M and then the comments are like beep beep because it's in the micro"
},
{
"start_time": 2657.4,
"end_time": 2678.38,
"content": " events. But if you pass this into a model that's trained to be a normal producing text, it's extremely high loss. Because normally you see an M, you don't predict M's for a long time. So like this is something that caused a loss spikes for us. But when you have much like this is old, this is not recent, and when you have more mature data systems, that's not the thing that causes the loss. And when you have more mature data systems, that's not the thing that causes the loss spike. And what Dylan is saying is true."
},
{
"start_time": 2678.38,
"end_time": 2699.26,
"content": " But it's like, it's levels to this sort of idea. With regards to the stress, right? These people are like, you know, you'll go out to dinner with like a friend that works at one of these labs. And they'll just be, like, looking at their phone every, like, 10 minutes. And they're not, like, you know, it's one thing if they're texting, but they're just like, like, is the loss? Yeah. Tocons per second."
},
{
"start_time": 2700.56,
"end_time": 2724.3,
"content": " Lost not blown up. They're just walking just watching this. And the heart rate goes up if there's a spike. And some level of spikes is normal, right? It'll recover and be back. Sometimes a lot of the old strategy was like, you just stop the run, restart from the old version, and then like change the data mix, and then it keeps going there are even different types of spikes so dirk grenovel has a theory a day i do that's like fast spikes and slow spikes, where there are sometimes where you're"
},
{
"start_time": 2724.3,
"end_time": 2750.35,
"content": " looking at the loss and there are other parameters, you can see it start to creep up and then blow up and that's really hard to recover from. So you have to go back much further. So you have the stressful period where it's like flat or might start going up and you're like, what do I do? Whereas there are also lost spikes that are, it looks good, and then there's one spiky data point. And what you could do is you just skip those. You see that there's a spike, You're like, okay, I can ignore this data, don't update the model, and do the next one, and it'll recover quickly. But these like untrickier implementations."
},
{
"start_time": 2750.45,
"end_time": 2776.08,
"content": " So as you get more complex in your architecture and you scale up to more GPUs, you have more potential for your loss blowing up. So it's like, there's a distribution. The whole idea of grocking also comes in, right? It's like just because it slowed down from improving and loss doesn't mean it's not learning because all of a sudden it could be like this and they could just spike down and loss again because it learned truly learned something right and it took some time for it to learn that it's not like a gradual process, right?"
},
{
"start_time": 2776.08,
"end_time": 2798.32,
"content": " And that's what humans are like, that's what models are like. So it's really a stressful task, as you mentioned. And the whole time the dollar count is going up. Every company has failed runs. You need failed runs to push the envelope on your infrastructure. So a lot of news cycles are made of X company had Y failed to run. Every company that's trying to push the frontier of AI has these."
},
{
"start_time": 2798.32,
"end_time": 2819.32,
"content": " Yes, it's noteworthy because it's a lot of money and it can be week to month setback, but it is part of the process. But how do you get, if you're deep seek, how do you get to a place where, holy shit, there's a successful combination of hyperparameters? A lot of small failed runs. So rapid, failed runs. And so rapid iteration through failed runs until"
},
{
"start_time": 2819.32,
"end_time": 2844.72,
"content": " and successful ones. And then you build a sum of intuition like this mixture of expert works and then this implementation of MLA works. Key hyperparameters, like learning rate and regulation and things like this. And you find the regime that works for your codebase. I've talking to people at Frontier Labs, there's a story that you can tell where training language models is kind of a path that you need to follow."
},
{
"start_time": 2844.9,
"end_time": 2866.38,
"content": " So you need to unlock the ability to train a certain type of model or a certain scale. And then your code base and your internal know-how of which hyperparameters work for it is kind of known. And you look at the deep seek papers and models, they've scaled up, they've added complexity, and it's just continuing to build the capabilities that they have. There's the concept of a YOLO run. So YOLO you only live once."
},
{
"start_time": 2866.38,
"end_time": 2888.93,
"content": " And what it is is like, you know, there's there's, there's all this experimentation you do at the small scale, right? Research ablations, right? Like you have your Jupyter notebook whether you're experimenting with mLA on like three GPUs or whatever um and you're doing all these different uh things like hey hey, do I do four expert, four active experts, 128 experts, do I arrange the experts this way? You know, all these different model architecture things,"
},
{
"start_time": 2888.93,
"end_time": 2911.21,
"content": " you're testing at a very small scale, right? Couple researchers, few GPUs, tens of GPUs, hundreds of GPUs, whatever it is. And then all of a sudden, you're like, okay, guys, no more, no more fucking around, right? No more screen around. Everyone take all the resources we have, let's pick what we think will work and just go for it, right? YOLO. And this is where that sort of stress comes in is like, well, I know it works here, but some things that work"
},
{
"start_time": 2911.21,
"end_time": 2936.11,
"content": " here don't work here and some things that work here don't work down here, right? In this terms of scale, right? So it's it's really truly a YOL run and and sort of like, there's this like discussion of like certain researchers just have like this methodical nature. Like they can find the whole search space and like figure out all the ablations of different research and really see what is best. And there's certain researchers who just kind of like, you know, have that innate gut instinct of like,"
},
{
"start_time": 2936.11,
"end_time": 2956.39,
"content": " this is the Yolo run. Like, you know, looking at the data, I think this is it. This is why you want to work in post-training because the GPU costs for training is lower, so you can make a higher percentage of your training runs, Yolo runs. Yeah. For now. Yeah, for now. For now. Yeah, for now. For now. So some of this is fundamentally luck still. Luck is skill, right? In many cases."
},
{
"start_time": 2956.71,
"end_time": 2979.31,
"content": " Yeah, I mean, it looks lucky, right? When you're... But the hill to climb, if you're out one of these labs, you have an evaluation, you're not crushing. There's a repeated playbook of how you improve things. There are localized improvements, which might be data improvements, and these add up into the whole model just being much better. And when you zoom in really closed, it can be really obvious that this model is just really bad at this thing and we can fix it and you just add these up."
},
{
"start_time": 2979.31,
"end_time": 2999.75,
"content": " So some of it feels like look, but on the ground, especially with these new reasoning models we're talking to, is just so many ways that we can poke around. And normally normally it's that some of them give big improvements. The search space is near infinite, right? And yet the amount of compute in time you have is very low and you have to hit release schedules."
},
{
"start_time": 2999.91,
"end_time": 3021.78,
"content": " You have to not get blown past by everyone. Otherwise, you know, what happened with deep seek, you know, crushing meta and mistral and coherent and all these guys, they moved too slow, right? They maybe were too methodical. I don't know, they didn't hit the Yolo run, whatever the reason was, maybe they weren't a skill. Whatever, you know, you can call it luck if you want, but at end of day, it's skill. So 2025 is the year of the YOLO run."
},
{
"start_time": 3021.78,
"end_time": 3051.33,
"content": " It seems like all the labs are like going in. I think it's even more impressive what opening AI did in 2022, right? At the time, no one believed in mixture of experts models, right, at Google, who had all the researchers. Open AI had such little compute and they devoted all of their compute for many months, all of it, 100% for many months, all of it, 100% for many months to GPT4 with a brand new architecture with no belief that, hey, let me spend a couple hundred million dollars, which is all of the money I have"
},
{
"start_time": 3051.33,
"end_time": 3072.89,
"content": " on this model, right? Now, now, you know, people are like, all these like training run failures that are in the media, right? It's like, okay, great, but like actually a lot, a huge chunk my GPs are doing inference. I still have a bunch doing research constantly. And yes, my biggest cluster is training, but like on this YOLO run, but like that YOLO run is much less risky than like"
},
{
"start_time": 3072.89,
"end_time": 3094.49,
"content": " what opening I did in 2022. Or maybe what Deep Seek did now, or, you know, like, sort of like, hey, we're just going to throw everything at it. The big winners throughout human history are the ones who are willing to do yellow at some point. Okay. What do we understand about the hardware it's been trained on, deep seek? Deepseek is very interesting. This is second to take us to zoom out"
},
{
"start_time": 3094.49,
"end_time": 3115.93,
"content": " out of who they are, first of all, right? High Flyer is a hedge fund that has historically done quantitative trading in China as well as elsewhere. And they have always had a significant number of GPUs, right? In the past, a lot of these high frequency trading algorithmic quant traders used FPGAs. But it shifted to GPUs definitely. And there's both, right? But GPUs especially in deep and high flyer,"
},
{
"start_time": 3115.93,
"end_time": 3139.31,
"content": " which is the hedge fund that owns deep seek and everyone who works for deep seek is part of high flyer to some extent, right? Same parent company, same owner, same CEO. They had all these resources and infrastructure for trading. And then they devoted a humongous portion of them to training models, both language models and otherwise, right? Because these these techniques were heavily AI influenced."
},
{
"start_time": 3140.31,
"end_time": 3165.64,
"content": " You know, more recently, people have, you know, realized, hey, trading with, you know, like even, even when you go back to like Renaissance and all these, all these like quantitative firms, natural language processing is the key to like trading really fast, right? Understanding a press release and making the right trade. And so DeepSeek has always been really good at this. And even as far back as 2021, they have press releases and papers saying, like, hey, we're the first company in"
},
{
"start_time": 3165.64,
"end_time": 3190.97,
"content": " China with an A100 cluster this large. It was 10,000 A100 GPs, right? This is in 2021. Now, this wasn't all for training, you know, large language models. This is mostly for training models for their quantitative aspects there are quantitative trading as well as you know a lot of that was natural language processing, to be clear. Right. And so this is the sort of history, right? So verifiable fact is that in 2021, they built the largest Chinese cluster, at least they claim it was the largest cluster in China,"
},
{
"start_time": 3190.97,
"end_time": 3213.49,
"content": " 10,000 GPUs. Before expert controls started. Yeah. It's like they've had a huge cluster before any conversation of export controls. So then you step it forward to like, what have they done over the last four years since then, right? Obviously, they've continued to operate the hedge fund, probably make tons of money. And the other thing is that they've leaned more and more and more into AI. The CEO, Leon Ching Feng, Leon."
},
{
"start_time": 3213.49,
"end_time": 3237.81,
"content": " You're not putting me on this. We discussed this. Leon Fing, right? The CEO, he owns maybe, Leon Fing, he owns maybe a little bit more than half the company, allegedly, right? Is an extremely like Elon Jensen kind of figure where he's just like involved in everything, right? And so over that time period, he's gotten really in-depth into AI. He actually has a bit of a like, if you see some of his statements, a bit of an"
},
{
"start_time": 3237.81,
"end_time": 3259.31,
"content": " EAC vibe almost, right? Total AGI vibes. And like, we need to do this. We need to do this. We need to make a new ecosystem of open AI. We need China to lead on this sort of ecosystem because historically the Western countries have led on software ecosystems and straight up acknowledges like in order to do this, we need to do something different."
},
{
"start_time": 3259.31,
"end_time": 3284.12,
"content": " Deep Seek is his way of doing this. Some of the translated interviews with him are fantastic. So he has done interviews? Yeah. You think you would do a Western interview or no? Or is there controls on the channel? There hasn't been one yet, but I would try it. I just got a Chinese translator, so it's great. This is this is all push. So fascinating figure, engineer, pushing full on into AI, leveraging the success from the high frequency trading."
},
{
"start_time": 3284.12,
"end_time": 3306.94,
"content": " Very direct quotes. We will not switch to closed source when asked about this stuff. He's very long-term motivated in how the ecosystem of AI should work. And from a Chinese perspective, he wants a Chinese company, a Chinese company to build this vision. And so this is sort of like the quote unquote visionary behind the company, right?"
},
{
"start_time": 3306.98,
"end_time": 3327.48,
"content": " This hedge fund still exists, right? This quantitative firm and so deep seek is the sort of at you know solely he got turned to this full view of like AI, everything about this, right? But at some point, it slowly maneuvered and he made deep seek. And deep seek has done multiple models since then. They've acquired more and more GPUs. They share infrastructure with the fund, right?"
},
{
"start_time": 3327.98,
"end_time": 3348.14,
"content": " And so, you know, there is right and so you know there is no exact number of public GPU resources that they have but besides this 10,000 GPUs that they bought in 2021, right, and they were fantastically profitable, right? And then this paper claims they did only 2,000 H800 GPs, which are a restricted GPU that was previously allowed in China, but no longer allowed,"
},
{
"start_time": 3348.14,
"end_time": 3368.26,
"content": " and there's a new version, but no longer allowed, and there's a new version. But it's basically NVIDIA's H-100 for China, right? And there's some restrictions on it specifically around the communications, sort of speed, the interconnect speed, right? Which is why they had to do this crazy SM, you know, scheduling stuff, right? So going back to that, right? It looks like, this is obviously not true in terms of their total GPU account."
},
{
"start_time": 3368.26,
"end_time": 3388.85,
"content": " Obvious available GPUs, but for this training run, you think 2000 is the correct number or no? So this is where it takes, you know, a significant amount of sort of like zoning in, right? Like what do you call your training run, right? Do you count all of the research and ablations that you ran, right? Picking all this stuff because yes, you can do a YOLO run, but at some level,"
},
{
"start_time": 3388.85,
"end_time": 3408.89,
"content": " you have to do the test at the small scale, and then you have to do some test at medium scale before you go to a large scale. Accepted practice is that for any given model that is a notable advancement, you're going to do 2 to 4x compute of the full training run in experiments alone. So a lot of this compute that's being scaled up, is probably used in large part at this time for research."
},
{
"start_time": 3408.89,
"end_time": 3439.23,
"content": " Yeah, and research will, you know, research begets the new ideas that let you get huge efficiency. Research gets you 01. Like research gets you 01. Like research gets you breakthroughs and you need to bet on it. So some of the pricing strategy they will discuss has the research baked into the price? So the numbers that DeepSeek specifically said publicly, right, are just the 10,000 GPUs in 2021, and then 2,000 GPs for only the pre-training for V3. They did not discuss cost on R1. They did not discuss cost on all the other RL for the instruct model that they made,"
},
{
"start_time": 3439.55,
"end_time": 3460.69,
"content": " right? They only discussed the pre-training for the base model, and they did not discuss anything on research and ablations. And they do not talk about any of the resources that are shared in terms of, hey, the fund is using all these GPUs, right? And we know that they're very profitable and that 10,000 GPUs in, in 2021 so so the some of the research that we've found is that we actually"
},
{
"start_time": 3460.69,
"end_time": 3483.64,
"content": " believe they have closer to 50,000 GPUs. We as semi-analysis. So we should say that you're sort of one of the world experts in figuring out what everybody's doing in terms of the semiconductor, in terms of cluster build-outs, in terms of like who's doing what in terms of training runs. So yeah, so that's the we. Okay, go ahead. Yeah, sorry. We believe they actually have something closer to 50,000 GPs, right?"
},
{
"start_time": 3483.64,
"end_time": 3505.52,
"content": " Now, this is split across many tasks, right? Again, the fund. Research and ablations. For ballpark, how much would open AI or Anthropic had? I think the clearest example we have, because meta is also open, they talk about order of 60K to 100K, H-100K-Equivalent GPUs in their training clusters. Right, so like Lama 3, they said, in their training clusters. Right. So like Lama 3, they trained on 16,000 H-100s, right?"
},
{
"start_time": 3508.22,
"end_time": 3526.98,
"content": " But the company of META last year publicly disclosed, they bought like 400 something thousand GPS. Yeah. Right. So of course, tiny percentage on the training. Again, like most of it is like serving me the best Instagram reels, right? Um, or whatever, right? I mean, we could get into a cost of like, what is the cost of ownership for a 2000 GPU cluster, 10,000. There's just different sizes of companies that can afford these things."
},
{
"start_time": 3526.98,
"end_time": 3547.92,
"content": " And DeepSeek is reasonably big. Their compute allocation compared to is one of the top few in the world. It's not open AI, entropic, et cetera, but they have a lot of compute. Can you, in general, actually just zoom out and also talk about the Hopper architecture, the Nvidia Hopper GPU architecture and the difference between H-100 and H-800,"
},
{
"start_time": 3548.14,
"end_time": 3567.98,
"content": " like you mentioned, the interconnects. Yeah, so there's, you know, Amper was the A-100, and then H-100, Hopper, right? People are used them synonymously in the US because really there's just H100 and now there's H 200, right? But same thing. Mostly. In China, they've had two, there have been different salvos of export restrictions. So initially, the US government limited on a two-factor scale, right?"
},
{
"start_time": 3567.98,
"end_time": 3591.64,
"content": " Which is chip interconnect versus flops, right? So any chip that had interconnects above a certain level and flops above a certain floating point operations above a certain level was restricted. Later, the government realized that this was a flaw in the restriction, and they cut it down to just floating point operations. And so H-800 had high flops, low communication."
},
{
"start_time": 3591.64,
"end_time": 3619.55,
"content": " Exactly. So the H-800 was the same performance as H-100 on flops, right? But it didn't have, it just had the interconnect bandwidth cut. DeepSeek knew how to utilize this, you know, hey, even though we're cut back on the interconnect, we can do all this fancy stuff to figure out how to use the GPU fully anyways, right? And so that was back in October 2022, but later in 2020, end of 20203 implemented in 2024, the US government banned the H800, right?"
},
{
"start_time": 3620.11,
"end_time": 3639.63,
"content": " And so by the way, this H800800 cluster, these 2,000 GPUs, was not even purchased in 2024, right? It was purchased in late 2023. And they're just getting the model out now, right? Because it takes a lot of research, et cetera. H-800 was banned and now there's a new chip called the H-20. The H-20 is cut back on only flops, but the interconnect bandwidth is the same."
},
{
"start_time": 3639.63,
"end_time": 3661.31,
"content": " And in fact, in some ways, it's better than the H-100 because has better memory bandwidth and memory capacity. So there are, you know, Nvidia is working within the constraints of what the government sets and then builds the best possible GP for China. Can we take this actual tangent and we'll return back to the hardware is the philosophy, the motivation, the case for export controls. What is"
},
{
"start_time": 3661.31,
"end_time": 3682.36,
"content": " it? Dari Amadegh just published a blog post about expert controls. The case he makes is that if AI becomes super powerful and he says by 26 we'll have AGI or super powerful AI and that's going to give a significant, whoever builds that will have a significant military advantage. And so, because the United States is a democracy"
},
{
"start_time": 3682.36,
"end_time": 3704.8,
"content": " and as he says China is authoritarian or has authoritarian elements, you want a unipolar world where the super powerful military because of the AI is one that's a democracy. It's a much more complicated world geopolitically when you have two superpowers with super powerful AI"
},
{
"start_time": 3704.8,
"end_time": 3728.3,
"content": " and one is authoritarian. So that's the case he makes. And so we want to, the United States wants to use export controls to slow down to make sure that China can't do these gigantic training runs that will be presumably required to build AGI. This is very abstract. I think this can be the goal of how some people describe export controls,"
},
{
"start_time": 3728.3,
"end_time": 3752.64,
"content": " is this super powerful AI. There's, and you touched on the training run idea, there's not many worlds where China cannot train AI models. Export controls are kneecapping the amount of compute or the density of compute that China can have. And if you think about the AI ecosystem right now as all of these AI companies, revenue numbers are up into the right,"
},
{
"start_time": 3752.64,
"end_time": 3776.91,
"content": " the AI usage is just continuing to grow, more GPUs are going to inference. A large part of export controls, if they work, is just that the amount of AI that can be run in China is going to be much lower. So on the training side, deep seek V3 is a great example, which you have a very focused team that can still get to the frontier of AI. This 2000 GPUs is not that hard to get, all considering in the world."
},
{
"start_time": 3776.91,
"end_time": 3800.09,
"content": " They're still going to have those GPUs. They're still going to be able to train models. But if there's going to be a huge market for AI, if you have strong export controls and you want to have 100,000 GPUs just serving the equivalent of chat GPT clusters, with good export controls, it also just makes it so that AI can be used much less. And I think that is a much easier goal to achieve than trying to debate on what AGI is."
},
{
"start_time": 3800.09,
"end_time": 3821.21,
"content": " And if you have these extremely intelligent autonomous AIs and data centers, like those are the things that could be running in these GPU clusters in the United States, but not in China. To some extent, training a model does effectively nothing, right? Like they have a model. The thing that Dario is sort of speaking to is the implementation of that model once trained"
},
{
"start_time": 3821.21,
"end_time": 3844.24,
"content": " to then create huge economic growth, huge increases in military capabilities, huge increases in productivity of people, betterment of lives, whatever you want to direct super powerful AI towards, you can. But that requires significant amounts of compute, right? And so the U.S. government has effectively said, and forever, right, like training will always be a portion of the total compute."
},
{
"start_time": 3844.72,
"end_time": 3864.9,
"content": " You know, we mentioned META 400,000 GPUs, only 16,000 made Lama. So the percentage that meta is dedicating to inference, now this might be for recommendation systems that are trying to hack our mind into spending more time and watching more ads, or if it's for a super powerful AI that's doing productive things, doesn't matter about the exact use that our economic system decides."
},
{
"start_time": 3865.2,
"end_time": 3886.78,
"content": " It's that that can be delivered in whatever way we want. Whereas with China, right, you know, you're, you know, expert restrictions, great. You're never going to be able to cut everything off, right? And that's, that's like, I think that's quite well understood by the U.S. government is that you can't cut everything off. And they'll make their own chips. And they're trying to make their own chips. And they're trying to make their own chips. They'll be worse than ours. But the whole point is to just keep a gap, right?"
},
{
"start_time": 3886.78,
"end_time": 3908.48,
"content": " And therefore, at some point as the AI, you know, in a world where two, three percent economic growth, this is really dumb, by the way, right? To cut off, you know, high tech and not make money off of it but in a world where super powerful AI comes about and then starts creating significant changes in society, which is what all the AI leaders and big tech companies believe. I think super powerful AI is going to change society massively."
},
{
"start_time": 3912.46,
"end_time": 3930.7,
"content": " And therefore, this compounding effect of the difference in compute is really important. There's some sci-fi out there where, like, AI is like measured in the power of in like how much power is delivered to compute right or how much is being you know that's sort of a way of thinking about what's the economic output is just how much power are you directing towards that AI. Should we talk about reasoning models with this as a way that this might be actionable as"
},
{
"start_time": 3930.7,
"end_time": 3952.64,
"content": " something that people can actually see? So the reasoning models that are coming out with R1 and 01, they're designed to use more compute. There's a lot of buzzy words in the AI community about this, test time compute, inference time compute, whatever. But Dylan has good research on this. You can get to the specific numbers on the ratio of a new training model, you can look at things about the amount of compute use at training and amount of compute use at inference."
},
{
"start_time": 3952.64,
"end_time": 3974.08,
"content": " These reasoning models are making inference way more important to doing complex tasks. In the fall, in December, their Open AI announced this O3 model. There's another thing in AI when things move fast. We get both announcements and releases. Announcements are essentially blog posts where you pat yourself on the back and you say you did things and releases or run the models out there, the papers out there, etc. So OpenAI has announced"
},
{
"start_time": 3974.08,
"end_time": 3994.54,
"content": " 03 and we can check if 03 Mini is out as of recording potentially, but that doesn't really change the point, which is that the breakthrough result was something called ARC-AGI task, which is the abstract reasoning corpus, a task for artificial general intelligence. Francois Chile is the guy who's been, it's a multi-year-old paper."
},
{
"start_time": 3994.66,
"end_time": 4015.31,
"content": " It's a multi-year-old paper. It's a brilliant benchmark. And the number for OpenAI 03 to solve this was that it used as some sort of number of samples in the API. The API has like thinking effort and number of samples. They used a thousand samples to solve this task and it comes out to be like five to $20 per question, which you're putting in effectively a math puzzle"
},
{
"start_time": 4015.31,
"end_time": 4037.23,
"content": " and then it takes orders of dollars to answer one question. And this is a lot of compute. If those are going to take off in the U.S. OpenAI needs a ton of GPUs on inference to capture this. They have this OpenAI chat GPT pro subscription, which is $200 a month which sam said they're losing money on which means that people are burning a lot of GPUs on inference and i've signed up with it i I've played with it, I don't think I'm a power user,"
},
{
"start_time": 4037.39,
"end_time": 4057.47,
"content": " but I use it. And it's like, that is the thing that a Chinese company with mediumly strong expert controls, there will always be loopholes, might not be able to do it all. And if that, the main result for O3 is also a spectacular coding performance, and if that feeds back into AI companies being able to experiment better."
},
{
"start_time": 4057.47,
"end_time": 4080.31,
"content": " So presumably the idea is for an AGI, a much larger fraction of the compute would be used for this test time compute, for the reasoning. For the AGI goes into a room and thinks about how to take over the world and come back in 2.7 hours. This is what- And that it's to take a lot of computers. This is what people, like, CEO or leaders of open AI and"
},
{
"start_time": 4080.31,
"end_time": 4105.01,
"content": " anthropic talk about is like autonomous AI models, which is you give them a task and they work on it in the background. I think my personal definition of AGI is much simpler. I think language models are a form of AGI and all of this super powerful stuff is a next step that's great if we get these tools, but a language model has so much value in so many domains. It is a general intelligence to me. But this next step of agentic things where they're independent and they"
},
{
"start_time": 4105.01,
"end_time": 4129.14,
"content": " can do tasks that aren't in the training data is what the few-year outlook that these AI companies are driving for. I think the terminology here, that Dario uses a super powerful AI. So I agree with you on the AGI. I think we already have something like that's exceptionally impressive that Alan Turing would for sure say is AGI, but he's referring more to something once in possession of,"
},
{
"start_time": 4129.76,
"end_time": 4153.34,
"content": " then you would have a significant military and geopolitical advantage over other nations. So it's not just like you can ask it how to cook an omelet. And he has a much more positive view and his essay, Machines of Love and Grace. I read into this. I don't have enough background in physical sciences to gauge exactly how confident I am and if AI can revolutionaries biology, I'm safe saying that AI is going to"
},
{
"start_time": 4153.34,
"end_time": 4174.4,
"content": " accelerate the progress of any computational science. So we're doing a depth-first search here on topics, taking tangent of a tangent, so let's continue on that depth first search. You said that you're both feeling the AGI. So what's your timeline? Dario's 2026 for the super powerful AI that's"
},
{
"start_time": 4174.4,
"end_time": 4197.31,
"content": " you know, that's basically agentic to a degree where it's a real security threat, that level of AGI. What's your timeline? I don't like to attribute specific abilities, because predicting specific abilities and when is very hard. I think mostly if you're going to say that I'm feeling the AGI is that I expect continued rapid surprising progress over the next few years."
},
{
"start_time": 4197.31,
"end_time": 4217.51,
"content": " So something like R1 is less surprising to me from DeepSeek because I expect there to be new paradigms where substantial progress can be made. I think DeepSeek R1 is so unsteadling because we're kind of on this path with chat GPT. It's like, it's getting better, it's getting better, it's getting better. And then we have a new direction for, for changing the models, and we took one step like this and we like took a step up."
},
{
"start_time": 4217.51,
"end_time": 4237.77,
"content": " So it looks like a really fast slope and then we're going to just take more steps. So like it's just really unsettling when you have these big steps. And I expect that to keep happening. I see I've tried opening eye operator, I've tried Claude computer use. They're not there yet. I understand the idea, but it's just so hard to predict what is the breakthrough that will make something like that work."
},
{
"start_time": 4237.77,
"end_time": 4258.89,
"content": " And I think it's more likely that we have breakthroughs that work and things that we don't know what they're going to do. So, like, everyone wants agents. Dario has very eloquent way of describing this. And I just think that it's like, way of describing this. And I just think that it's like, there's going to be more than that. So I could just expect these things to come. I'm going to have to try to pin you down to a date on the AGI timeline."
},
{
"start_time": 4261.65,
"end_time": 4279.47,
"content": " Like the nuclear weapon moment. So moment where on the geopolitical stage, there's a real, like, you know, because we're talking about export controls. When do you think, just even throw out a date, when do you think that would be like for me it's probably after 2030 so i'm not as"
},
{
"start_time": 4279.47,
"end_time": 4301.77,
"content": " what i would say so define that right to me, it kind of almost has already happened, right? You look at elections in India and Pakistan, people get AI voice calls and think they're talking to the politician, right? The AI diffusion rules, which was enacted in the last couple of weeks of the Biden admin, and looks like the Trump admin will keep and potentially even strengthen, limit cloud computing and GPU sales to countries"
},
{
"start_time": 4301.77,
"end_time": 4323.41,
"content": " that are not even related to China. It's like this is Portugal and all these like normal companies are on the you need approval from the US list like yeah Portugal and like you know like all these countries that are allies right Singapore right like they they freaking have F-35s and we don't let them buy GPUs like this is this to me is already to the scale of like, you know. Well, that just means that"
},
{
"start_time": 4323.41,
"end_time": 4345.71,
"content": " the U.S. military is really nervous about this new technology. That doesn't mean the technology is already there. So, like, they might be just very cautious about this thing that they don't quite understand. But that's a really good point so the the robocalls swarms of semi-intelligent bots could be a weapon, could be doing a lot of social engineering. I mean,"
},
{
"start_time": 4345.71,
"end_time": 4366.85,
"content": " there's tons of talk about, you know, from the 2016 elections like Cambridge Analytica and all this stuff, Russian influence. I mean, every country in the world is pushing stuff onto the internet and has narratives they want, right? Like that's every, every like technically competent, whether it's Russia, China, US, Israel, et cetera, right? You know, people are pushing viewpoints onto the internet en masse. And language models crash the cost"
},
{
"start_time": 4366.85,
"end_time": 4387.95,
"content": " of like very intelligent sounding language. There's some research that shows that the distribution is actually the limiting factor. So language models haven't yet made misinformation particularly change the equation there. The internet is still ongoing. I think there's a blog AI snake oil and some of my friends at Princeton that write on this stuff. So there is research. It's like it's a default that"
},
{
"start_time": 4387.95,
"end_time": 4410.64,
"content": " everyone assumes. And I would have thought the same thing, is that misinformation isn't yet far worse with language models, I think. In terms of internet posts and things that people have been measuring, it hasn't been a exponential increase or something extremely measurable. And things you're talking about with like voice calls and stuff like that, it could be in modalities that are harder to measure. So it's something that it's too soon to tell in terms of..."
},
{
"start_time": 4410.64,
"end_time": 4433.32,
"content": " I think that's like political instability via the web is very, it's monitored by a lot of researchers to see what's happening. I think that you're asking about the AGI thing. If you make me give a year, I would be like, okay, I have AI CEOs saying this. They've been saying two years for a while. I think that there people like Dario,"
},
{
"start_time": 4433.96,
"end_time": 4455.87,
"content": " Anthropic, the CEO, had thought about this so deeply. I need to take their words seriously, but also understand that they have different incentives. So I would be like add a few years to that, which is how you get something similar to 2030 or a little after 2030. I think to some extent we have or a little after 2030. I think to some extent we have capabilities that hit a certain point where any one person could say, okay, if I can leverage those capabilities"
},
{
"start_time": 4455.87,
"end_time": 4476.43,
"content": " for X amount of time, this is AGI, right? Call it 27, 28. But then the cost of actually operating that capability. Yeah, this is going to be my point. So, so extreme that no one can actually deploy it at scale and mass to actually completely revolutionize the economy on a click on a snap of a finger. So I don't think it will be like a snap of the finger moment. It's a physical constraint."
},
{
"start_time": 4476.59,
"end_time": 4498.31,
"content": " Rather, it'll be a, you know, oh, the capabilities are here, but I can't deploy it everywhere, right? And so one, one simple example going back sort of to 2023 was when being with GPD 4 came out and everyone was freaking out about search right perplexity came out if you did the cost on like hey implementing GPT3 into every Google search. I was like, oh, okay, this is just like physically impossible to implement."
},
{
"start_time": 4498.31,
"end_time": 4519.37,
"content": " Right. And as we step forward to like going back to the test time compute thing, right? A query for, you know, you ask chat GPT a question, it costs sense, right? For their most capable model of chat, right, to get a query back. To solve an ARC AGI problem, though, cost five to 20 bucks right and this is this is an a only going up from"
},
{
"start_time": 4519.37,
"end_time": 4541.38,
"content": " there this is a thousand 10,000 X factor difference in cost to respond to a query versus do a task. And the task of our AGI is not like it's like it's it's simple to some extent um you know but it's also like what are the tasks that we want age okay aGI quote unquote what we have today can do arc a gai three years from now it can do much more complicated problems but the cost is going to be"
},
{
"start_time": 4541.38,
"end_time": 4565.73,
"content": " measured in thousands and thousands and hundreds of thousands of dollars of GPU time and there just won't be enough power GPUs, infrastructure to operate this and therefore shift everything in the world on the snap the finger. But at that moment, who gets to control and point the AGI at a task. And so this was in Dario's post that he's like, hey, China can effectively and more quickly than us point their AGI at"
},
{
"start_time": 4565.73,
"end_time": 4586.97,
"content": " military tasks, right? And they have been in many ways faster at adopting certain new technologies into into their military right especially with regards to drones, right? The US maybe has a longstanding, you know, large air sort of, you know, fighter jet type of thing, bombers. But when it comes to asymmetric arms, such as drones, they've completely leapfrogged"
},
{
"start_time": 4586.97,
"end_time": 4608.27,
"content": " the US and the west. And the fear that Dario is sort of pointing out there, I think, is that, yeah, great, we'll have AGI in the commercial sector, the U.S. military won't be able to implement it super fast. Chinese military could and they could direct all their resources to implementing it in the military and therefore solving, you know, military logistics or solving some, some other aspect of like disinformation"
},
{
"start_time": 4608.27,
"end_time": 4629.64,
"content": " for targeted certain set of people so they can flip a country's politics or something like that that is actually like catastrophic versus you know, the US just wants to, because it'll be more capitalistically allocated just towards whatever is the highest return of income, which might be like building, you know, factories better or whatever. So everything I've seen, people's intuition seems to fail on robotics."
},
{
"start_time": 4629.64,
"end_time": 4653.98,
"content": " So you have this kind of general optimism. I've seen this on self-driving cars. People think it's much easier problem than it is. Similar with drones. Here, I understand it a little bit less, but I've just seen the reality of the war in Ukraine and the usage of drones on both sides. And it seems that humans still far outperform any fully autonomous systems."
},
{
"start_time": 4653.98,
"end_time": 4674.52,
"content": " AI is an assistant, but humans drive. FPV drones where the humans control. Most of it just far, far, far outperforms AI system. So I think it's not obvious to me that we're going to have swarms of autonomous robots anytime soon in the military context. Maybe the fastest I can imagine is 2030,"
},
{
"start_time": 4674.82,
"end_time": 4703.51,
"content": " which is why I said 2030 for the super powerful AI. Whenever you have large scale swarms of robots doing military actions, that's when the world just starts to look different to me. So that's the thing I'm really worried about. But there could be cyber war, cyber war type of technologies that, uh, from social engineering to actually just swarms the robots that find attack vectors in our code bases and shut down power grids, that kind of stuff."
},
{
"start_time": 4704.63,
"end_time": 4724.61,
"content": " And it could be one of those things like on any given weekend or something, power goes out. Nobody knows why, and the world changes forever. Just power going out for two days in all of the United States. That will lead to murder, to chaos. But going back to export controls,"
},
{
"start_time": 4724.87,
"end_time": 4745.07,
"content": " do you see that as a useful way to control the balance of power geopolitically in the context of AI. And I think going back to my viewpoint is, if you believe we're in the sort of stage of economic growth and change that we've been in for the last 20 years,"
},
{
"start_time": 4745.07,
"end_time": 4766.37,
"content": " the export controls are absolutely guaranteeing that China will win long term, right? If you do not believe AI is going to make significant changes to society in the next 10 years or five years, right? Five, five year timelines are sort of what the more executives and such of AI companies and even big tech companies believe. But even 10-year timelines, you know, it's reasonable."
},
{
"start_time": 4766.37,
"end_time": 4786.69,
"content": " But once you get to, hey, these timelines are below that time period, then the only way to sort of like create a sizable advantage or disadvantage for America versus China is if you constrain compute because talent is not really something that's"
},
{
"start_time": 4786.69,
"end_time": 4807.97,
"content": " constraining, right? China arguably has more talent, right? More STEM graduates, more programmers. The US can draw upon the world's people, which it does. There's tons of, you know, foreigners in the AI industry. So many of these AI teams are all people without a US passport. Yeah, I mean, many of them are Chinese people who are moving to America, right? And that's great. That's exactly what we want, right?"
},
{
"start_time": 4807.97,
"end_time": 4829.48,
"content": " But there's that talent is one aspect, but I don't think that's one that is a measurable advantage for the US or not. It truly is just whether or not compute. Now, even on the compute side, when we look at chips versus data centers, China has the unprecedented ability to build ridiculous sums of power, clockwork, right?"
},
{
"start_time": 4831.4,
"end_time": 4851.44,
"content": " They're always building more and more power. They've got steel mills that, that like individually are the size of the entire U.S. industry, right? And they've got aluminum mills that consume gigawatts and gigawatts of power. Right. And when we talk about what's the biggest data center, right? Opening I made this huge thing about Stargate, their announcement there. That's not, that's like once it's fully built out in a few years,"
},
{
"start_time": 4851.54,
"end_time": 4872.48,
"content": " it'll be two gigawatts, right? Of power, right? And this is still smaller than the largest industrial facilities in China, right? China, if they wanted to build the largest data center in the world, if they had access to the chips, could. So it's just a question of when, not if, right? So their industrial capacity for exceeds the United States? Exactly. To manufacture stuff."
},
{
"start_time": 4873.06,
"end_time": 4892.58,
"content": " Yeah. So long term they're going to be manufacturing chips there? Chips are a little bit more specialized. I'm specifically referring to the data centers, right? Chips, fabs take huge amounts of power. Don't get me wrong. That's not necessarily the gating factor there. The gating factor on how fast people can build the largest clusters today in the US"
},
{
"start_time": 4892.58,
"end_time": 4913.42,
"content": " is power. It is whether it's, now, it could be power generation, power transmission, substations, and, you know, all these sorts of transformers and all these things, building the data center. These are all constraints on the U.S. industry's ability to build larger and larger training systems, as well as deploying more and more inference compute. I think we need to make the point clear"
},
{
"start_time": 4913.42,
"end_time": 4942.97,
"content": " on why the time is now for people that don't think about this. Because essentially with export controls, you're making it so China cannot make or get cutting edge chips. And the idea is that if you time this wrong, China is pouring a ton of money into their chip production. And if you time it wrong, they are going to have more capacity for production, more capacity for energy, and figure out how to make the chips and have more capacity than the rest of the world to make the chips because everybody can buy they're going to sell their Chinese chips to everybody, they might subsidize them."
},
{
"start_time": 4942.97,
"end_time": 4965.77,
"content": " And therefore, if AI takes a long time to become differentiated, we've kneecapped the financial performance of American companies. Nvidia can sell less. TSM cannot sell to China, so therefore we have less demand to therefore to keep driving the production cycle. So that's the assumption behind the time timing being Less than 10 years or five years to above, right?"
},
{
"start_time": 4965.77,
"end_time": 4993.04,
"content": " China will win because of these restrictions long term unless AI does something in the short term, which I believe AI will do, you know, make massive changes to society in the medium short term, right? And so that's the big unlocker there. And even today, right, if Xi Jinping decided to get, you know, quote unquote scale-pilled, right? Ie. decide that scaling laws are what matters, right? Just like the U.S. executives, like Satcha Nadella"
},
{
"start_time": 4993.04,
"end_time": 5014.64,
"content": " and Mark Zuckerberg and Sundar and all these U.S. executives of the biggest most powerful tech companies have decided their skill-pilled and they're building multi-gigawatt data centers, right? Whether it's in Texas or Louisiana or Wisconsin, wherever, wherever it is, they're building these massive things that cost as much as their entire budget for spending on data centers globally in one spot, right?"
},
{
"start_time": 5014.72,
"end_time": 5038.34,
"content": " This is what they've committed to for next year, year after, et cetera. And so they're so convinced that this is the way, that this is what they're doing, but if China decided to, they could do it faster than us, but this is where the restrictions come in. It is not clear that China as a whole has decided, from the highest levels that this is a priority. The US sort of has, right? You know, you see Trump talking about Deep Seek and"
},
{
"start_time": 5038.34,
"end_time": 5062.23,
"content": " Stargate within the same week, right? So he's, and the Biden admin as well had a lot of discussions about AI and such. It's clear that they think about it. Only just last week did DeepSeek meet the second in command of China, right? Like they have not even met the top, right? They haven't met Xi. She hasn't set down. And they only just released a subsidy of a trillion R&B, you know, roughly $160 billion,"
},
{
"start_time": 5062.95,
"end_time": 5085.67,
"content": " which is closer to the spending of like Microsoft and Meta and Google combined, right, for this year. So it's like they're realizing it just now, but that's where these export restrictions come in and say, hey, you can't, you can't ship the most powerful U.S. chips to China. You can ship a cut down version. You can't ship the most powerful chips to all these countries who we know we're just"
},
{
"start_time": 5085.67,
"end_time": 5105.97,
"content": " going to rent it to China. You have to limit the numbers, right? And the tools. And same with manufacturing equipment tools, all these different aspects. But it all stems from AI and then what downstream can slow them down in AI. And so the entire semiconductor restrictions, you read them, they are very clear. It's about AI and military civil fusion of technology."
},
{
"start_time": 5105.97,
"end_time": 5128.57,
"content": " Right? It's very clear. And then from there, it goes, oh, well, we're banning them from buying like lithography tools and etch tools and deposition tools and, oh, this random like, you know, subsystem from a random company that's like tiny, right? Like, why are we banning this? Because all of it, the U.S. government has decided is critical to AI systems. I think the fulcrum point is like the transition from seven nanometer to 5 nanometer chips,"
},
{
"start_time": 5128.57,
"end_time": 5151.03,
"content": " where I think it was Huawei that had the 7 nanometer chip a few years ago, which caused another political brouhaha almost like this moment and then it's the ASML deep UV what is that like extreme ultraviolet lithography. To set context on the chips, right, what Nathan's referring to is in 2020, Huawei released their Ascend 910 chip, which was an AI"
},
{
"start_time": 5151.03,
"end_time": 5173.27,
"content": " AI chip, first one on 7 nanometer before Google did, before Nvidia did. And they submitted it to the ML Perf benchmark, which is sort of a industry standard for machine learning performance benchmark. And it did quite well. And it was the best chip at the submission, right? This was a huge deal. The Trump admin, of course, banned, it was 2019, right, banned the Huawei from getting"
},
{
"start_time": 5173.27,
"end_time": 5196.1,
"content": " 7 nanometer chips from TSM, and so then they had to switch to using internal domestically produced chips, which was a multi-year setback. Many companies have done 7 nanometer chips. And the question is, like, we don't know how much Huawei was subsidizing production of that chip. Like, Intel has made 7 nanometer chips that are not profitable and things like this. So this is how it all feeds back into the economic engine of export controls."
},
{
"start_time": 5196.1,
"end_time": 5216.36,
"content": " Well, so you're saying that for now, Xi Jinping has not felt the AGII, but it feels like the deep seek moment yeah might like there might be meetings going on now where he's going to start wearing the same t-shirt and things are going to escalate. I mean, like, this, he may have woken up last week right Leon Feng met"
},
{
"start_time": 5216.36,
"end_time": 5240.9,
"content": " the vice chair vice the second command guy and they had a meeting and then the day the next day they announced the AI subsidies, which are a trillion R&B. So it's possible that this deep seek moment is truly the beginning of a cold war. That's what a lot of people are worried about. People in AI have been worried that this is going towards a Cold War, or already is. But there's, it's not Deep Seeks fault, but there's something, a bunch of factors came"
},
{
"start_time": 5240.9,
"end_time": 5263.89,
"content": " together where it was like this explosion. I mean, it all has to do with Nvidia's not going down probably. It's just some mass hysteria that happened that eventually led to Xi Jinping having meetings and waking up to this idea. And the US government realized in October 7th, 2022, before ChatGPT released, that restriction on October 7th, which dropped and shocked everyone."
},
{
"start_time": 5264.07,
"end_time": 5284.23,
"content": " And it was very clearly aimed at AI. Everyone was like, what the heck are you doing? Stamble diffusion was out then, but not Chad GPT. Yeah, but not Chad GPD. So it was like starting to be rumblings of what Gen. I can do to society, but it was very clear, I think, to at least, like, national security council and, and those sort of folks that this was where the world is headed, this Cold War that's happening."
},
{
"start_time": 5284.83,
"end_time": 5308.43,
"content": " So is there any concerns that the export controls push China to take military action on Taiwan. This is the big risk. to take military action on Taiwan. This is the big risk, right? The further you push China away from having access to cutting edge American and global technologies, the more likely they are to say, well, because I can't access it, I might as well, like no one should access it, right?"
},
{
"start_time": 5308.79,
"end_time": 5329.39,
"content": " And there's a few like interesting aspects of that, right? Like, you know, China has a urban rural divide like no other. They have a male, female birth ratio, like no other, to the point where, you know, if you look in most of China, it's like the ratio's not that bad, but when you look at single dudes in rural China, it's like a 30 to one ratio. And those are disenfranchised dudes, right?"
},
{
"start_time": 5329.39,
"end_time": 5349.75,
"content": " Like quote unquote, like the US has an in cell problem like China does too. It's just they're placated in some way or cut crushed down. What do you do with these people? And at the same time, you're not allowed to access the most important technology. At least the US thinks so. China is maybe starting to think this is the most important technology by starting to dump subsidies in it, right? They thought EVs and renewables were the most important technology."
},
{
"start_time": 5349.75,
"end_time": 5372.63,
"content": " They dominate that now, right? Now they're starting to, they started thinking about that about semiconductors in, you know, the late 2010s and early 2020s, and now they've been dumping money and they're catching up rapidly. And they're going to do the same with AI, right? Because they're very talented, right? So the question is like, when does when does when does when does when does this hit a breaking point right um and"
},
{
"start_time": 5372.63,
"end_time": 5406.59,
"content": " if china sees this as hey hey, they can continue, if not having access and starting a true hot war, taking over Taiwan or trying to subvert its democracy in some way or blockading it hurts the rest of the world far more than it hurts them, this is something they could potentially do, right? And so is this pushing them towards that? Potentially, right? I'm not quite a geopolitical person, but, you know, it's obvious that the world regime of peace and like trade is like super awesome for economics. But but at some point it could break."
},
{
"start_time": 5406.59,
"end_time": 5431.99,
"content": " I think we should comment that the like why Chinese economy would be hurt by that is that they're export heavy. The United States buys so much like if that goes away, like, that's how their economy. Well, also, also they just like would not be able to import raw materials from like all over the world, right? The US would just shut down the strait of malacca and like you know in the same time the u.s entire like you could argue almost all the GDP growth in America since you know the 70s has been"
},
{
"start_time": 5431.99,
"end_time": 5457.42,
"content": " either population growth or tech, right? Because your life today is not that much better than someone from the 80s outside of tech, right? You still, you know, cars, they all have semiconductors in them everywhere. Fridges, semiconductors everywhere. There's these funny stories about how Russians were taking apart laundry machines because they had certain like Texas instrument chips that they could then repurpose and put into like their, um, their anti-missile missile, missile things, right?"
},
{
"start_time": 5457.48,
"end_time": 5479.44,
"content": " Like their S-400 or whatever, you would know more about this. But, uh, there's all sorts of like everything about semiconductors is so integral to every part of our lives. So can you explain the role of TSM in the story of semiconductors and maybe also how the United States can break the reliance on TSM? I don't think it's necessarily breaking the reliance."
},
{
"start_time": 5479.44,
"end_time": 5500.3,
"content": " I think it's getting TSMC to build in the US. But so taking a step back, right? TSM produces most of the world's chips, right? Especially on the foundry side. You know, there's a lot of companies that build their own chips, Samsung, Intel, ST Micro, Texas instruments,"
},
{
"start_time": 5500.3,
"end_time": 5521.3,
"content": " analog devices, all these kinds of companies build their own chips and XP, but more and more of these companies are outsourcing to TSM and have been for multiple decades. Can you explain the supply chain there and where most of TSM is in terms of manufacturing. Sure. So historically supply chain was companies would build their own chips. They would, you know, it would be a company started."
},
{
"start_time": 5521.3,
"end_time": 5543.33,
"content": " They'd build their own chips, and then they'd design the chip and build the ship and sell it. Over time, this became really difficult because the cost of building a fab continues to compound every single generation. Of course, the technology, figuring out the technology for it is incredibly difficult regardless, but just the dollars and cents that are required, ignoring, you know, saying, hey, yes, I have all the technical capability, which it's really hard to get that, by the way, right?"
},
{
"start_time": 5543.41,
"end_time": 5563.47,
"content": " Intel's failing, Samsung's failing, et cetera. But if you look at just the dollars to spend to build that next generation fab, it keeps growing, right? Sort of like, you know, Moore's Law is having the cost of ships every two years. There's a separate law that's sort of like doubling the cost of fabs every handful of years. And so you look at a leading edge fab that is going to be profitable today, that's building, you know, three nanometer chips or"
},
{
"start_time": 5563.47,
"end_time": 5583.73,
"content": " two nanometer chips in the future, that's going to cost north of 30, 4040 billion, right? And that's just for like a token amount. That's for a like the base building blocking. You probably need to build multiple, right? And so when you look at the industry over the last, you know, if I go back 20, 30 years ago, there were 20, 30 companies that could build the most advanced chips, and then they would design them"
},
{
"start_time": 5583.73,
"end_time": 5603.96,
"content": " themselves and sell them, right? So companies like AMD would build their own chips. Intel, of course, still builds their own chips. They're very famous for it. IBM would build their own chips. And you could keep going down the list. All these companies built their own chips. Slowly, they kept falling like flies, and that's because of what TSM did, right? They created the foundry business model, which is, I'm not going to design any chips. I'm just going to contract"
},
{
"start_time": 5603.96,
"end_time": 5626.5,
"content": " manufacturer chips for other people. And one of their early customers is in video, right? In video was is the only semiconductor company that's worth, you know, that's doing more than a billion dollars of revenue that was started in the era of Foundry, right? Every other company started before then and at some point had FAs, which is actually incredible, right? You know, like AMD and Intel and Broadcom."
},
{
"start_time": 5626.5,
"end_time": 5648.93,
"content": " Such a great fact. It's like everyone had fabs at some point or, you know, brought, you know, some companies like Broadcom, it was like a merger, amalgamation of various companies that rolled up. But even today, Broadcom has fabs, right? They build iPhone RF radio chips sort of in Colorado for, you know, for Apple, right? Like there's there, all these companies had fabs and for most of the fabs, they threw them away or sold them off or they got rolled into something else."
},
{
"start_time": 5649.39,
"end_time": 5676.76,
"content": " And now everyone relies on TSMC, right? Including Intel, their latest PC chip uses TSM chips, right? It also uses some Intel chips, but it uses TSM process. Can you explain why the Foundry model is so successful for these companies? Why are they going with economies of scale? Scale. Yeah. So I mean, like I mentioned, right, the cost of building a fab is so high. The R&D is so difficult. And when you look at like these companies that had their own vertical stack,"
},
{
"start_time": 5677.08,
"end_time": 5697.2,
"content": " there was an antiquated process of like, okay, like, I'm so hyper-customized to each specific chip. But as we've gone through the history of sort of like the last 50 years of electronics and semiconductors, A, you need more and more specialization, right? Because Moore's Law has died. Dunnard scaling has died, i.e. chips are not getting better just for free, right? You know, from manufacturing, you have to make"
},
{
"start_time": 5697.2,
"end_time": 5723.68,
"content": " real architectural innovations, right? Google is not just running on Intel CPUs for web serving. They have a YouTube chip, they have TPUs, they have pixel chips, they have a wide diversity of chips that, you know, generate all the economic value of Google, right? Running, you know, it's running all the services and stuff. And so, and this is just Google, and you could go across any company in the industry and it's like this right cars contain 5,000 chips you know 200 different varieties of them right all these random things. A Tesla door handle has two chips, right?"
},
{
"start_time": 5723.68,
"end_time": 5745.3,
"content": " Like it's like ridiculous. And it's a cool door handle, right? It's like, you know, you don't think about it, but it's like has two really chip like penny like chips in there, right? Anyway, so, so as you have more diversity of chips, as you have more specialization required and the cost of fabs continues to grow you need someone who is laser focused on building the best process technology and making it as flexible as possible."
},
{
"start_time": 5745.3,
"end_time": 5765.96,
"content": " I think you could say it simply, which is the cost per fab goes up. And if you are a small player that makes a few types of chips, you're not going to have the demand to pay back the cost of the fab. Whereas Nvidia can have many different customers and aggregate all this demand into one place. And then they're the only person that makes enough money building chips to"
},
{
"start_time": 5765.96,
"end_time": 5786.86,
"content": " buy the next, to build the next fab. So this is kind of why the company slowly get killed because they have a they have 10 years ago a chip that is profitable and is good enough, but the cost to build the next one goes up. They may try to do this, fail because they don't have the money to make it work and then they don't have any chips or they build it and it's too expensive and they just have not profitable chips."
},
{
"start_time": 5786.86,
"end_time": 5810.63,
"content": " You know, there's more failure points, right? right you know you could have one little process related to like some sort of like chemical etch or some sort of like plasma etch or you know some little process that screws up you didn't engineer it right and now the whole company falls apart you can't make chips, right? And so super, super powerful companies like Intel, they had like the weathering storm to like, hey, they still exist today, even though they really screwed up their manufacturing six, seven years ago."
},
{
"start_time": 5810.63,
"end_time": 5831.87,
"content": " But in the case of like AMD, they almost went bankrupt. They had to sell their fabs to Mubedala, UAE, right? And like that became a separate company called Global Foundries, which is a foundry firm. And then AMD was able to then focus on the return back up, was like, hey, let's focus on making chiplets and a bunch of different chips for different markets and focusing on specific workloads"
},
{
"start_time": 5831.87,
"end_time": 5854.69,
"content": " rather than, you know, all of these different things. And so you get more diversity of chips. You have more companies than ever designing chips, but you have fewer companies than ever manufacturing them. Right. And this is where TSMC comes in as they've, they've just been the best, right? They are so good at it, right? They're customer focused. They make it easy for you to fabricate your chips. They take all of that complexity and like kind of try and abstract a lot of it away from you."
},
{
"start_time": 5857.12,
"end_time": 5875.54,
"content": " They make good money. They don't make insane money, but they make good money. And they're able to aggregate all this demand and continue to build the next fab, the next fab, the next fab. So why is Taiwan so special for TSMC? Why is it happening there? Can it be replicated inside the United States? Yeah, so there's aspects of it that I would say yes and aspects that I'd say no, right?"
},
{
"start_time": 5876.54,
"end_time": 5896.66,
"content": " TSM is way ahead because former, you know, executive Morse Chang of Texas Instruments wasn't promoted to CEO and he's like, screw this, I'm going to go make my own chip company, right? And he went to Taiwan and made TSM, right? And there's a whole lot more story there. So it could have been, Texas instruments could have been the TSM, you know, could have been TSM but Texas semiconductor"
},
{
"start_time": 5896.66,
"end_time": 5922.96,
"content": " manufacturer right instead of you know Texas instruments right but but you know so there is that whole story there. Sitting here in Texas. I mean, and that sounds like a human story. Like, it didn't get promoted. Just the brilliance of Morris Chang, you know, which I wouldn't underplay, but there's also like a different level of like how this works, right? So in Taiwan, the number, top percent of students that go to the best school, which is NTU,"
},
{
"start_time": 5922.96,
"end_time": 5945,
"content": " the top percent of those all go work to TSM. Right? And guess what their pay is? Their starting pay is like 80 000 or 70 000 right which is like that's like starting pay for like a good graduate in the u, right? Not the top. The top graduates are making hundreds of thousands of dollars at the Googles and the Amazon's. And now I guess the open AIs of the world, right? So there is a large dichotomy of like,"
},
{
"start_time": 5945,
"end_time": 5969.03,
"content": " what is the top 1% of the society doing and where are they headed because of economic reasons? Intel never paid that crazy good, right? And it didn't make sense to them, right? That's one aspect, right? Where is the best going second is the work ethic right like you know we we like to work you know you work a lot we work a. But at the end of the day, when there's a, you know, when, what is the time and amount of work that you're doing and what does a fab require, right?"
},
{
"start_time": 5969.03,
"end_time": 5990.99,
"content": " Fabs are not work from home jobs. They are you go into the fab and grueling work, right? There's, there's, hey, if there is any amount of vibration, right? An earthquake happens, vibrates the machines. They're all, you know, they're either broken, you've scrapped some of your production, and then in many cases, they're like not calibrated properly. So, so when TSM, when there's an earthquake, right? Recently there's been an earthquake."
},
{
"start_time": 5991.57,
"end_time": 6011.07,
"content": " TSM doesn't call their employees. They just, they just go to the fab and like they just show up the parking lot gets slammed and people just go into the fab and fix it right like it's like an it's like ants right like It's like an ant's, right? Like it's like an ant's, like, you know, a hive of ants. Doesn't get told by the queen what to do. The ants just know. It's like one person just specializes on these one task."
},
{
"start_time": 6011.07,
"end_time": 6031.43,
"content": " And it's like, you're going to take this one tool and you're the best person in the world. And this is what you're going to do for your whole life is one task in the fab which is like some special chemistry plus nanomanufacturing on one line of tools that continues to get iterated. And yeah, it's just like, it's like specific plasma edge for removing silicon dioxide, right? That's all you focus on your whole career. And it's such a specialized thing."
},
{
"start_time": 6031.43,
"end_time": 6051.87,
"content": " And so it's not like the task are transferable. AI today is awesome because like people can pick it up like that. Semiconductor manufacturing is very antiquated and difficult. None of the materials are online for people to read easily and learn, right? The papers are very dense and like it takes a lot of experience to learn. And so it makes the barrier to entry much higher too."
},
{
"start_time": 6052.03,
"end_time": 6077.4,
"content": " So so when you talk about, hey, you have all these people that are super specialized. They will work, you know, 80 hours a week in a factory, right, in a fab. And if anything goes wrong wrong they'll go show up in the middle of the night because some earthquake their wife's like there was an earthquake he's like great i'm gonna go to the fab it's like great I'm gonna go to the fab would you would you like as an American do that right it's like these sorts of things are like what you know I guess are the exemplifying like why TSMC is so amazing?"
},
{
"start_time": 6077.4,
"end_time": 6099.28,
"content": " Now, can you replicate it in the US? Let's not ignore. Intel was the leader in manufacturing for over 20 years. They brought every technology to market first besides the UV strain silicon, high K metal gates, FinFet, you know, and the list goes just goes on and on and on of technologies that Intel brought to market first, made the most money from, and manufactured"
},
{
"start_time": 6099.28,
"end_time": 6120.84,
"content": " at scale first, best, highest profit margins, right? So we shouldn't ignore that Intel can't do this, right? It's that the culture has broken, right? You've invested in the wrong things. They said no to the iPhone. They had all these different things regarding like, you know, mismanagement of the fabs, mismanagement of the fabs, mismanagement of designs, this lockup, right? And at the same time, all these brilliant people,"
},
{
"start_time": 6120.84,
"end_time": 6144.18,
"content": " right, these like 50,000 PhDs, you know, or masters that have been working on specific chemical or physical processes or nanomanyufacturing processes for decades in Oregon, they're still there. They're still producing amazing work. It's just like getting it to the last mile of production at high yield, where you can, where you can manufacture dozens and hundreds of different kinds of chips, you know, and it's good customer experience has broken, right?"
},
{
"start_time": 6144.18,
"end_time": 6164.4,
"content": " You know, it's that customer experience. It's like the, like, part of it is like people will say Intel is too pompous in the 2010s, right? They just thought they were better than everyone. The tool guys were like, oh, I don't think that this is mature enough. And they're like, ah, you just don't know. We know, right? This sort of stuff would happen. And so can the US bring it to the, can the US bring leading edge semiconductor manufacturing to the US?"
},
{
"start_time": 6164.4,
"end_time": 6188.33,
"content": " Emphomatically, yes, right? And we are, right? It's happening. Arizona is getting better and better as time goes on. TSM has built, you know, roughly 20% of their capacity for 5 nanometer in the US, right? Now, this is nowhere near enough, right? You know, 20% of capacity in the US is like nothing, right? And furthermore, this is still dependent on Taiwan existing, right? All there's sort of important way to separate it out."
},
{
"start_time": 6188.33,
"end_time": 6209.63,
"content": " There's R&D and there's high volume manufacturing. There are effectively, there are three places in the world that are doing leading edge R&D. There's Sinshu, Taiwan, there's Hillsborough, Oregon, and there is Pyongyang, South Korea. These three places are doing the leading edge R&D for the rest of the world's leading edge semiconductors, right?"
},
{
"start_time": 6209.63,
"end_time": 6230.39,
"content": " Now, manufacturing can be distributed more globally, right? And this is sort of where this dichotomy exists of like, who's actually modifying the process? Who's actually developing the next generation one, who's improving them, is Sinshu, is Hillsborough, is Pyongyang, right? It is not the rest of these, you know fabs like arizona right"
},
{
"start_time": 6230.39,
"end_time": 6253.33,
"content": " arizona is a paperweight if if since you disappeared off the face of the planet um you know within within a year, a couple years, Arizona would stop producing too, right? It's actually like pretty critical. One of the things i like to say is if i had like a few missiles i know exactly where i could cause the most economic damage right it's not targeting the white house right it's the rn d cycle economic damage, right? It's not targeting the White House, right? It's the R&D centers. It's the R&D centers for TSM, Intel, Samsung,"
},
{
"start_time": 6253.33,
"end_time": 6275.39,
"content": " and then some of the memory guys, Micron and Hynix. Because they define the future evolution of these semiconductors. And everything's moving so rapidly that it really is fundamentally about R&D. And it is all about TSM, huh? And so TSMC, you know, you cannot purchase a vehicle without TSM chips, right? You cannot purchase a fridge without TSMC chips."
},
{
"start_time": 6275.39,
"end_time": 6296.41,
"content": " You cannot, you, like, I think one of the few things you can purchase, ironically, is a Texas instrument's like graphing calculator, right? Because they actually manufacture in Texas. But like outside of that, like a laptop, a phone, anything you servers, right, GPUs, none top of phone, it's depressing. Any servers, right, GPUs, none of this stuff can exist. And this is without TSMC. And in many cases, it's not even like the leading edge, you know, sexy 5 nanometer chip,"
},
{
"start_time": 6296.41,
"end_time": 6329.39,
"content": " 3 nanometer, chip, 2 nanometer chip. Oftentimes, it's just like some stupid power I see that's like converting from like, you know, some voltage to another, right? And it's made at TSM, right? This is what China is investing in as well. It's like they can build out this long tail fab where the techniques are much more known you don't have to figure out these problems with EUV they're investing in this and then they have large supply for things like the car door handles and the random stuff. And that trickles down into this whole economic discussion as well, which is they have far more than we do. And having supply for things like this is crucial to normal life."
},
{
"start_time": 6329.39,
"end_time": 6349.93,
"content": " So they're doing the, they're starting to invest in high volume manufacturer, but they're not doing R&D. So they do R&D on their own. They're just way behind, right? So I would say like in 2015, China had a five-year plan where they defined by 2025 and 2020 certain goals, including like 80% domestic production of semiconductors."
},
{
"start_time": 6350.23,
"end_time": 6370.63,
"content": " Uh, they're not, they're not going to hit that, right? To be clear, but they are, they are in certain areas really, really close, right? Like, BYD is probably going to be the first company in the world to not have to use TSM for making, because they have their own fabs, right, for making chips. Now, they still have to buy some chips from foreign, for example, like, around like self-driving A-DAS capabilities"
},
{
"start_time": 6370.63,
"end_time": 6391.07,
"content": " because those are really high end. But at least like, you know, like an internal combustion engine has 40 chips and an EV, you know, like an internal combustion engine has 40 chips and an EV, you know, just for like controlling like flow rates and all these things. And EVs are even more complicated. So all these different power ICs and battery management controllers and all these things, they're insourcing, right? And this is this is something that like China has been doing since 2015."
},
{
"start_time": 6391.07,
"end_time": 6425,
"content": " Now as far as like the trailing edge, they're getting so much capacity there. As far as the leading edge, right, i.e. this five nanometer and so on, so forth, right, where GPUs, they are still behind. And this is, the US restrictions are trying to stop them in the ladder. But you know, all that's happened, you know, is yes, they've slowed down their five nanometer, three nanometer, et cetera, but they've accelerated their, hey, 45 nanometer, 90 nanometer, power IC or analog I see or you know random chip in my keyboard right that kind of stuff so so there is an angle of like the u s's actions have been so from these export you know from the"
},
{
"start_time": 6425,
"end_time": 6446.36,
"content": " angle of the expert controls have been so inflammatory at slowing down China's progress on the leading edge that they've turned around and have accelerated their progress elsewhere because they know this is so important right if the US is going to lock them out here or if they lock us out here as well in the trailing edge and so going going back, can the U.S. build it here? Yes, but it's going to take a ton of money."
},
{
"start_time": 6446.52,
"end_time": 6469.79,
"content": " I truly think like to to revolutionize and completely in-source semiconductors would take a decade and a trillion dollars. Is some of it also culture? like you said, extreme competence, extreme work ethic in Taiwan. I think if you have the demand and the money is on the line, the American companies figure it out. It's going to take hand-holding with the government. I think that the culture helps TSM breakthrough, and it's easier for them."
},
{
"start_time": 6470.47,
"end_time": 6492.75,
"content": " TSM has some like 90,000 employees, right? It's not actually that insane amount. The Arizona Fab has 3,000 from Taiwan. And these people, like, their wives were like, yeah, we're not going to have kids unless we you sign up for the Arizona fab we go to Arizona and we have our kids there there's also a Japan fab where the same thing happened right and so like these wives drove like these like these dudes to like go to Japan or America to have the kids there."
},
{
"start_time": 6492.75,
"end_time": 6517.85,
"content": " And it's like it's an element of culture. Yeah, sure. Taiwan works that hard, but also like the US has done it in the past. They could do it now, right? You know, we can just import, I say import the best people in the world if we want to. That's where the immigration conversation is a tricky one and there's been a lot of debate over that. But yeah, it seems absurdly controversial to import the best people in the world. I don't understand why it's controversial. That's the one of the ways of"
},
{
"start_time": 6517.85,
"end_time": 6538.15,
"content": " winning. I'm sure we agree with you. And like even if you can't import those people, I still think you could do a lot to manufacture most of in the US if the money's there, right? And so like- It's just way more expensive. It's not profitable for a long time. And that's the context of like the Chips Act is only like $50 billion. Relative to, you know, some of the renewable, you know, initiatives that were passed in the"
},
{
"start_time": 6538.15,
"end_time": 6572.57,
"content": " Inflation Reduction Act and the Infrastructure Act, which total in the hundreds of billions of dollars, right? And so like the amount of money that the US is spending on the semiconductor industry is nothing, right? Whereas all these other countries have is nothing, right? Whereas all these other countries have structural advantages in terms of like, you know, work ethic and amount of work and like things like that, but also a number of STEM graduates, the percentile of their best going to that, right? But they also have like differences in terms of like, hey, there's just tax benefits in the law and have been in the law for 20 years. Right. And so and then and then some countries have massive subsidies, right? China has something like"
},
{
"start_time": 6572.57,
"end_time": 6594.59,
"content": " $200 billion of semiconductor subsidies a year. We're talking about $50 billion in the U.S. over like six, right? So the girth or difference in like the subsidy amounts is also huge right and and so i think you know trump has been talking about tariffing taiwan recently um you know that's sort of like terrifying Taiwan recently. That's sort of like one of these things that's like, oh, okay, well, like, you know, maybe he doesn't want to subsidize the U.S."
},
{
"start_time": 6594.63,
"end_time": 6614.83,
"content": " Semiconductor industry. Obviously, terrifying Taiwan is going to cost a lot of things to go get much more expensive, but does it change the equation for TSM building more fabs in the U.S. That's what he's sort of positing, right? So can you lay out the, so we laid out the importance. By the way, it's incredible how much you know about so much. We told you Dylan knows all the stuff."
},
{
"start_time": 6615.83,
"end_time": 6644.96,
"content": " Yeah. Yeah. So, okay, you laid out why TSMC is really important. If we look out into the future, 10, 20 years out, U.S.-China relationship seems like it can go to a dark place of Cold War, escalated Cold War, even hot war, or to a good place of anything from frenemies to cooperation to working together."
},
{
"start_time": 6644.96,
"end_time": 6682.5,
"content": " So in this game theory, complicated game. What are the different trajectories? What should US be doing? Like what do you see as the different possible trajectories of U.S.-China relations as both leaders start to feel the AGI more and more and see the importance of chips and the importance of AI. I mean, ultimately, the export controls are pointing towards a separate future economy. I think the U.S. has made it clear to Chinese leaders that we intend to control this technology at whatever cost to global economic integration."
},
{
"start_time": 6682.5,
"end_time": 6704.34,
"content": " So that, it's hard to unwind that. Like the card has been played. To the same extent, they've also limited US companies from mentoring China, right? So it is, it is, you know, it's been a long time coming. You know, at some point, you know, there was there was a convergence, right? But, but over at least the last decade, it's been branching further and further out, right? Like, U.S. companies can't enter China. Chinese companies can't enter the U.S."
},
{
"start_time": 6704.72,
"end_time": 6727.2,
"content": " The U.S. is saying, hey, China, you can't get access to our technologies in certain areas. And China's rebuttling with the same thing around like, you know, they've done some sort of specific materials in, you know, gallium and things like that that they've tried to limit the U.S. on. One of the, there's a U.S. drone company that's not allowed to buy batteries, then they have like military customers. And this drone company just tells the military customers like,"
},
{
"start_time": 6727.2,
"end_time": 6749.6,
"content": " hey, just get it from Amazon because I can't actually physically get them. Right. Like there's all these things that are happening that point to further and further divergence. I have zero idea. And I would love if we could all hold hands and sing kumbaya, but like I have zero idea how that could possibly happen. Is the divergence good or bad for avoiding war? Is it possible that the divergence in terms of"
},
{
"start_time": 6749.6,
"end_time": 6770.66,
"content": " manufacture chips of training AI systems is actually good for avoiding military conflict. It's an objective fact that the world has been the most peaceful as ever been when there are global hegemons, right? Or regional hegemons, right? Or regional hegemons, right, in historical context, right? The Mediterranean was the most peaceful ever when the Romans were there, right? China had very peaceful and warring times,"
},
{
"start_time": 6770.7,
"end_time": 6791.14,
"content": " and the peaceful times were when dynasties had a lockhold over not just themselves, but all their tributaries around them, right? And likewise, the most peaceful time in human history has been when the U.S. was the global hegemon, right? The last half, you know, decades. Now, we've sort of seen things start to slide right with Russia, Ukraine, with what's going on in Middle East, and Taiwan risk, all these different"
},
{
"start_time": 6791.14,
"end_time": 6811.82,
"content": " things are starting to bubble up, still objectively, extremely peaceful. Now, what happens when it's not one global hegemon, but it's two, obviously, and China will be, you know, competitive or even overtake the US like it's possible, right? And so this, this change in global hegemony, it's, I don't think it ever happens like super peacefully, right? When empires fall, right? Which is a"
},
{
"start_time": 6811.82,
"end_time": 6832.78,
"content": " possible trajectory for America. They don't fall gracefully, right? Like, they don't fall gracefully, right? Like they don't just slide out of irrelevance. Usually there's a lot of shaking. And so, you know, what the US is trying to do is maintain its top position. And what China is trying to do is become the top position, right? And, and And obviously there's budding of heads here in the most simple terms."
},
{
"start_time": 6832.78,
"end_time": 6858.47,
"content": " And that could take shape in all kinds of ways, including proxy wars. And that seems like it's already happening. Like as much as I want there to be centuries of prolonged peace, it does not, it looks like further instability internationally is ahead. And the US is ahead. And the U.S. is like sort of like current task is like, hey, if we control AI, if we're the leader in AI, then we, and AI significantly accelerates progress,"
},
{
"start_time": 6858.63,
"end_time": 6883.7,
"content": " then we can maintain the global hegemony position and therefore... I hope that works. And as an American, like, you know, kind of like, okay, I guess that's going to lead to peace for us. Now, obviously, other people around the world get affected negatively. Obviously, the Chinese people are not going to be in as advantageous of a position if that happens. But, you know, this is sort of the reality of like what's being done and the actions that are being carried out."
},
{
"start_time": 6883.96,
"end_time": 6904.36,
"content": " So can we go back to the specific detail of the different hardware? There's this nice graphic in the export controls of which GPUs are allowed to be exported and which are not. Can you kind of explain the difference? Is there, from a technical perspective, are the age"
},
{
"start_time": 6904.36,
"end_time": 6928.68,
"content": " 20s promising? Yeah, so this goes, and I think we'd have to like, we need to dive really deep into the reasoning aspect and what's going on there. But the H20, you know, the U.S. has gone through multiple iterations of the export controls, right? This H800 was at one point allowed back in 23, but then it got canceled, and by then, by, you know, DeepC had already built their cluster of they claim 2k."
},
{
"start_time": 6928.76,
"end_time": 6952.72,
"content": " I think they actually have like many more, like something like 10K of those. And now this H20 is the legally allowed chip, right? Invitya shipped a million of these last year to China. For context, it was like four or five million GPUs, right? So the percentage of GPUs that were this China-specific H-20 is quite high, right? You know, roughly 20%, 25%, right, 20% or so. And so this H20 has been neutered in one way,"
},
{
"start_time": 6952.8,
"end_time": 6973.08,
"content": " but it's actually upgraded in other ways, right? And you know, you could think of chips along three axes for AI, right? You know, ignoring software stack and like exact architecture, just raw specifications. There's floating point operations, right, flops. There is memory bandwidth, i.e., in memory capacity, right? I.O, right, memory. And then there is interconnect, right?"
},
{
"start_time": 6973.18,
"end_time": 6993.58,
"content": " Chip-to-chip interconnections. All three of these are incredibly important for making AI systems, right? Because AI systems involve a lot of compute. They involve a lot of moving memory around, whether it be to memory or two other chips. And so these three vectors, the US initially had a multi, you know, had two of these vectors controlled"
},
{
"start_time": 6993.58,
"end_time": 7014.22,
"content": " and one of them not controlled, which was flops and interconnect bandwidth were initially controlled. And then they said, no, no, no, we're going to remove the interconnect bandwidth and just make it a very simple only flops. But now, Nvidia can now make a chip that has, okay, it's cut down on flaps. It's, you know, it's like one third that of the H100, right? In, in, on spec sheet paper performance for flops."
},
{
"start_time": 7014.62,
"end_time": 7042.96,
"content": " You know, in real world, it's closer to like half, or maybe even like 60% of it. But then on the other two vectors, it's just as good for interconnect bandwidth, and then for memory bandwidth and memory capacity, the H20 has more memory bandwidth and more memory capacity than the H-100. Now, recently, you know, we at our research, we cut NVIDIA's for H20 for this year down drastically. They were going to make another two million of those this year, but they just canceled all the orders a couple weeks ago."
},
{
"start_time": 7042.96,
"end_time": 7066.6,
"content": " In our view, that's because we think that they think they're going to get restricted, right? Because why would they cancel all these orders for H20? Because they shipped a million of them last year. They had orders in for a couple million this year and just gone, right? For H20, B20, right? A successor to H20. And now they're all gone. Now, why would they do this, right? I think it's very clear, right? I think it's very clear, right? The H20 is actually better for certain tasks."
},
{
"start_time": 7066.6,
"end_time": 7090.12,
"content": " And that certain task is reasoning, right? Reasoning is incredibly different than, you know, when you look at the different regimes of models, right? Pre-training is all about flops, right? It's all about flops. There's things you do, like mixture of experts that we talked about, to trade off interconnect or to trade off, you know, other aspects and lower the flops and rely more on interconnect and memory."
},
{
"start_time": 7090.12,
"end_time": 7112.62,
"content": " But at the end of the day, it's flops as everything, right? We talk about models in terms of like how many flops they are right uh so so like you know we talk about oh gpd4 is two e 25 right two to the uh Two to the 25th, you know, 25th, you know, two, 25 zeros, right, flop, right, floating point operations. For training. For training, right? And we're talking about the restrictions for the 2E24, right?"
},
{
"start_time": 7112.62,
"end_time": 7137.47,
"content": " The US has an executive order that Trump recently unsigned, which was, hey, 1E26, once you hit that number of floating point operations, you must notify the government. And you must share your results with us, right? Like there's a level of model where the US government must be told, right? And that's 1E26. And so as we move forward, this is, this is an incredibly like important flop is the vector that the government has cared about historically, but the other"
},
{
"start_time": 7137.47,
"end_time": 7158.13,
"content": " two vectors are arguably just as important, right? And especially when we come to this new paradigm, which the world is only just learning about over the last six months, right? Reasoning. And do we understand firmly which of the three dimensions is best for reasoning? So interconnect, the flops don't matter as much. Is it memory? Memory, right?"
},
{
"start_time": 7158.13,
"end_time": 7179.93,
"content": " Yeah, so. We're going to get into technical stuff real fast. There's two articles in this one that I could show, maybe graphics that might be interesting for you to pull up. For the listeners, we're looking at the section of 01 inference architectures toconomics. Hmm. You want to explain KV Cash before we talk about this? I think like it's better to- Okay, yeah, we should get, we need to go through a lot of specific technical things"
},
{
"start_time": 7179.93,
"end_time": 7205.05,
"content": " of transformers to make this easy for people. Because it's incredibly important because this changes how models work. But I think resetting, right? Why is memory so important? It's because so far we've talked about parameter counts, right? And mixture of experts, you can change how many active parameters versus total parameters to embed more data but have less flops. But more important, you know, another aspect of, another aspect of what's part of this humongous revolution in the last"
},
{
"start_time": 7205.05,
"end_time": 7225.31,
"content": " handful of years is the transformer, right? And the attention mechanism. Attention mechanism is that the model understands the relationships between all the words in its context, right? And that is separate from the parameters themselves, right? And that is something that you must calculate, right? How each token, right, each word in the context length,"
},
{
"start_time": 7225.31,
"end_time": 7246.59,
"content": " is relatively connected to each other, right? And I think, I think, Nathan, you should explain KV Cash better. KV Cash is one of the optimization. Yeah, so the attention operator has three core things. It's queries, keys, and values. QKV is the thing that goes into this. You'll look at the equation. You see that these matrices are multiplied together."
},
{
"start_time": 7246.59,
"end_time": 7269.62,
"content": " These words, query, key, and value come from information retrieval backgrounds, where the query is the thing you're trying to get the values for, and you access the keys, and values is rewating. My background's not in information retrieval and things like this. It's just fun to have backlinks. And what effectively happens is that when you're doing these matrix multiplications, you're having matrices that are of the size of the context length."
},
{
"start_time": 7269.62,
"end_time": 7289.98,
"content": " So the number of tokens that you put into the model. And the KV cache is effectively some form of compressed representation of all the previous tokens in the model. So when you're doing this, we talk about auto-regressive models. You predict one token at a time. You start with whatever your prompt was, you ask a question, who was the president in 1825?"
},
{
"start_time": 7289.98,
"end_time": 7311.26,
"content": " The model then is going to generate its first token. For each of these tokens, you're doing the same attention operator where you're multiplying these query key value matrices, but the math is very nice so that when you're doing this repeatedly, this KV cache, this key value operation, you can keep appending the new values to it."
},
{
"start_time": 7311.26,
"end_time": 7332.76,
"content": " So you keep track of what your previous values you're inferring over in this auto-aggressive chain, you keep it in memory the whole time. And this is a really crucial thing to manage when serving inference at scale. There are far bigger experts in this, and there are so many levels of detail that you can go into. Essentially, one of the key, quote,"
},
{
"start_time": 7332.76,
"end_time": 7358.19,
"content": " drawbacks of the attention operator and the transformer is that there is a form of quadratic memory cost in proportion to the context length. So as you put in longer questions, the memory used in order to make that computation is going up in the form of a quadratic. You'll hear about a lot of other language model architectures that are like subquadatic or linear attention forms, which is like state space models."
},
{
"start_time": 7358.35,
"end_time": 7378.29,
"content": " We don't need to go down all these now. And then there's innovations on attention to make this memory usage and the ability to attend over long contexts much more accurate and high performance and those innovations are going to help you with i mean you're highly memory constraint they help with memory constraint and performance. So if you put in a book into, I think,"
},
{
"start_time": 7378.29,
"end_time": 7399.09,
"content": " Gemini is the model that has the longest context length that people are using. Gemini is known for 1 million and now 2 million context length. You put a whole book into Gemini and sometimes it'll draw facts out of it. It's not perfect. They're getting better. So there's two things. It's like one to be able to serve this on the memory level. Google has magic with their TPU stack where they can serve really long contexts."
},
{
"start_time": 7399.61,
"end_time": 7422.89,
"content": " And then there's also many decisions along the way to actually make long contacts performance work. This applies the data. There's subtle changes to these computations and attention and it just it changes the architecture. But serving long context is extremely many constrained, especially when you're making a lot of predictions. I actually don't know why input and output tokens are more expensive, But I think essentially output tokens,"
},
{
"start_time": 7422.89,
"end_time": 7443.97,
"content": " you have to do more computation because you have to sample from the model. I can explain that. So today, if you use a model, like you look at an API, Open AI charges, you know, a certain price per million tokens, right? And that price for input and output tokens is different, right? And the reason is, is that there is, you know, when you're inputting a query into the model, right?"
},
{
"start_time": 7443.97,
"end_time": 7468.62,
"content": " Let's say you have a book, right? That book, you must now calculate the entire KV cash for it, right? This key value cache. And so when you do that, that is a parallel operation. All of the tokens can be processed at one time. And therefore, you can dramatically reduce how much you're spending, right? The flop requirements for generating a token and an input token are identical, right? If I input one token or if I generate one token, it's completely identical. I have to go through the model, right?"
},
{
"start_time": 7468.62,
"end_time": 7488.72,
"content": " But the difference is that I can do that input, i.e. the pre-fill, i.e. the prompt, simultaneously in a batch nature, right? And therefore, it is all flop. I think the pricing model, mostly they use is for input tokens is about one fourth the price of the output tokens. Correct. But then output tokens, the reason why it's so expensive is because I can't do it in parallel, right?"
},
{
"start_time": 7488.72,
"end_time": 7510.62,
"content": " It's auto-regressive. Every time I generate a token, I must not only take the entire, I must not only read the whole entire model into memory, right, and activate it, right, go calculate it to generate the next token. I also have to read the entire KV cache. And I generate a token and i append that kv that one token i generated and it's kv cache and then i do it again right and therefore, this is a non-parallel operation."
},
{
"start_time": 7510.62,
"end_time": 7534.38,
"content": " And this is one where you have to, you know, in the case of pre-fill or prompt, you pull the whole model in and you calculate 20,000 tokens at once, right? So these are features that APIs are shipping, which is like prompt caching, pre-filling, because you can drive prices down and you can make API as much faster. If you know you're going to keep, if you run a business and you're going to keep passing the same initial content to Cloud's API,"
},
{
"start_time": 7534.38,
"end_time": 7556.17,
"content": " you can load that in to the Anthropic API and always keep it there. But it's very different than we're kind of leading to the reasoning models, which we showed this example earlier and read some of this kind of mumbling stuff. And what happens is that the output context length is so much higher. And I mean, I learned a lot about this from Dylan's work, which is essentially as the output length gets higher,"
},
{
"start_time": 7556.17,
"end_time": 7579.61,
"content": " you're writing this quadratic in terms of memory used. And then the GPUs that we have effectively, you're going to run out of memory, and they're all trying to serve multiple requests at once. So doing this batch processing where not all of the prompts are exactly the same, really complex handling. And then as context links gets longer, there's this link, I think you call it critical batch size, where your ability to serve"
},
{
"start_time": 7582.95,
"end_time": 7599.65,
"content": " more users, so how much you can parallelize your inference, implements because of this long contract. So your memory usage is going way up with these reasoning models, and you still have a lot of users. So effectively, the cost to serve multiplies by a ton. And we're looking at a plot when the x-axis is sequence length,"
},
{
"start_time": 7599.89,
"end_time": 7620.83,
"content": " i.e. how many tokens are being generated slash prompt, right? So if I put in a book, that's a million tokens, right? But, you know, if I put in, you know, the sky is blue, then that's like six tokens or whatever. We should say that what we're calling reasoning and chain of thought is extending this sequence length. It's mostly output. So before, you know, three months ago, whenever 01 launched, all of the use cases"
},
{
"start_time": 7620.83,
"end_time": 7644.71,
"content": " for long context length where like, let me put a ton of documents in and then get an answer out, right? And it's a single, you know, pre-fill, compute a lot in parallel, and then output a little bit. Now, with reasoning and agents, this is a very different idea, right? Now, instead, I might only have like, hey, do this task, or I might have all these documents. But at the end of the day, the model is not just like producing a little bit, right? It's producing tons of information."
},
{
"start_time": 7644.89,
"end_time": 7666.03,
"content": " This chain of thousands of tokens to go and go and go and go. And so the sequence length is effectively that, that you know, if it's generated 10,000 tokens, it's 10,000 sequence length, right? Or plus whatever you input it in the prompt. And so what this chart is showing, and it's a logarithmic chart, right, is, you know, as you go from 1K to 4K or 4K to 16K,"
},
{
"start_time": 7666.15,
"end_time": 7686.15,
"content": " the memory requirements grow so fast for your KV cache that you end up not being able to run a certain number of your sequence length is capped or the number of users. Let's say the model. So this is showing for a 405B model and batch size 64. Lama 3.145B. Yeah, and batch size is crucial too."
},
{
"start_time": 7686.31,
"end_time": 7706.85,
"content": " Essentially, you want to have higher batch size to parallelize parallel your throughput 64 different users at once right yeah and therefore your serving costs are lower, right? Because the server costs the same, right? This is eight H-100s, roughly $2 an hour per GPU. That's $16 an hour, right? That is like somewhat of a fixed cost. You can do things to make it lower, of course. But like, it's like $16 an hour."
},
{
"start_time": 7706.85,
"end_time": 7728.42,
"content": " Now, how many users can you serve? How many tokens can you generate? And then you divide the two, and that's your cost, right? And so with reasoning models, this is where a lot of the complexity comes about and why memory is so important. Because if you have limited amounts of memory, then you can't serve so many users. If you have limited amounts of memory, your serving speeds get lower, right? And so your costs get a lot, lot worse."
},
{
"start_time": 7728.92,
"end_time": 7750.84,
"content": " Because all of a sudden, if I was used to, hey, on the $16 an hour server, I'm serving Lama 405B, or if I'm serving, you know, deep seek v3, and it's all chat style applications, i.e. we're just chit-chatting. The sequence sensor, a thousand, a few thousand, right? You know, when you use the language model, it's a few thousand context lengths most of times. Sometimes you're dropping a big document, but then you process it, you get your answer, you throw it away, right?"
},
{
"start_time": 7750.88,
"end_time": 7776.28,
"content": " You move on to the next thing, right? Whereas with reasoning, I'm now generating tens of thousands of tokens in sequence, right? And so this memory, this KV cache has to stay resident. And you have to keep loading it. You have to keep it, keep it in memory constantly. And now this butts out other users, right? If there's now a reasoning task, right, and the model is capable of reasoning, then all of a sudden, that memory pressure means that I can't serve as many users simultaneously."
},
{
"start_time": 7776.28,
"end_time": 7800.88,
"content": " Let's go into deep seek again. So we're in the post deep seek R1 time, I think, and there's two sides to this market watching how hard it is to serve it. On one side, we're going to talk about deep seek themselves. They now have a chat app that got to number one on the app store. Disclaimer, number one on the app store is measured by velocity. So it's not necessarily saying that more people have the deep seek app than chat GPT app. But it is still remarkable."
},
{
"start_time": 7800.88,
"end_time": 7825.23,
"content": " Claude has never hit the number one in the App Store, even though everyone in San Francisco is like, oh my God, you got to use Cloud. Don't use chat. GPT. So DeepSeek hit this. They also launched an API product recently where you can ping their API and get these super long responses for R1 out. At the same time as these are out, we'll get to what's happened to them. Because the model weights for deep seek R1 are openly available and the license is very friendly, the MIT license"
},
{
"start_time": 7825.23,
"end_time": 7846.87,
"content": " commercially available. All of these mid-sized companies and big companies are trying to be first to serve R1 to their users. We were trying to evaluate R1 because we have really similar research going on. We released the model and we're trying to compare to it. And out of all the companies that are quote unquote serving R1, and they're doing it at prices that are way higher"
},
{
"start_time": 7846.87,
"end_time": 7868.01,
"content": " than the deep seek API. Most of them barely work and the throughput is really low. To give context, right? Everyone, one of the parts of freaking this out was like China reached capabilities. The other aspect is they did it so cheap, right? And the so cheap, we kind of talked about on the training side, why it was so cheap. I was talk about why it was so cheap. I was talk about why it's so cheap on the inference. It works well and it's cheap. Why is R1 so damn"
},
{
"start_time": 7868.01,
"end_time": 7891.22,
"content": " cheap? So I think there's a couple are one so damn cheap. So I think there's a couple factors here, right? One is that they do have model architecture innovations, right? This MLA, this new attention that they've done, is different than the attention from attention is all you need, the transformer attention, right? Now, others have already innovated. There's a lot of work like MQA, GQA, local, global, all these different innovations that try to bend the curve, right?"
},
{
"start_time": 7891.28,
"end_time": 7912.52,
"content": " It's still quadratic, but the constant is now smaller, right? Related to our previous discussion this multi-head latent attention can save about 80 to 90% in memory from the intention mechanism, which helps especially at long contexts. It's 80 to 90% versus the original, but then versus what people are actually doing. It's still an innovation. This 80 to 90% doesn't say that the whole model"
},
{
"start_time": 7912.52,
"end_time": 7932.62,
"content": " is 80 to 90% cheaper, just as one part of it. Well, and not just that, right? Like, other people have implemented techniques like local, global and sliding window and GQMQA. But anyways, like deepSeek has their attention mechanism is a true architectural innovation. They did tons of experimentation. And this dramatically reduces the memory pressure. It's still there, right? It's still a quadrat. It's still a tension. It's still quadratic."
},
{
"start_time": 7933.06,
"end_time": 7958.28,
"content": " It's just dramatically reduced it relative to prior forms. That's the memory pressure. I should say, in case people don't know, R1 is 27 times cheaper than 01. We think that Open AI had a large margin built in. Okay, so that's one- There's multiple factors. We should break down the factors, I think. It's two bucks per million token output for R1 and $60 per million token output for 01."
},
{
"start_time": 7960.28,
"end_time": 7979.04,
"content": " Yeah, let's look at this. So I think this is very important, right? Open AI is, you know, that drastic gap between DeepSeek and pricing. But DeepSeek is offering the same model because they open weights to everyone else for a very similar, like much lower price than what others are able to serve it for."
},
{
"start_time": 7979.04,
"end_time": 8002.55,
"content": " Right. So there's there's two factors here, right? Their model is cheaper, right? It is 27 times cheaper. Well, I don't remember the number exactly off top of my head. So we're looking at a graphic that's showing different places serving V3, deep seek V3, which is similar to Deepseek R1, and there's a vast difference in serving cost. And serving costs,"
},
{
"start_time": 8002.55,
"end_time": 8024.37,
"content": " and what explains that difference? And so like part of it is open AI has a fantastic margin, right? They're serving, when they're doing inference, their gross margins are north of 75%. Right. So that's that's a four to five X factor right there of the cost difference is that open eyes just making crazy amounts of money because they're the only one with the capability. Do they need that money? Are they using for R&D? They're losing money, obviously, as a company,"
},
{
"start_time": 8024.37,
"end_time": 8047.19,
"content": " because they spend so much on training, right? So the inference itself is a very high margin, but it doesn't recoup the cost of everything else they're doing. Okay. So yes, they need that money because the revenue and margins pay for continuing to build the next thing, right? As long as I'm raising more money. So the suggestion is that deep seek is like really bleeding out money. Well, so here's one thing, right? We'll get to this in a second, but like, deep seek doesn't have any capacity to actually serve the model."
},
{
"start_time": 8047.27,
"end_time": 8067.29,
"content": " They stopped signups. The ability to use it is like non-existent now, right? For most people, because so many people are trying to use it, they just don't have the GPUs to serve it. OpenAIs hundreds of thousands of GPUs between them and Microsoft to serve their models. DeepSeek has a factor of much lower, right? Even if you believe our research, which is 50,000 GPUs,"
},
{
"start_time": 8067.29,
"end_time": 8091.54,
"content": " and a portion of those are for research, portion of those are for the hedge fund, right? They still have nowhere close to the GPU volumes and capacity to serve the model, right, at scale. So it is cheaper. A part of that is opening a ton of money. Is DeepSeek making money on their API? Unknown. is deep seek making money on their appi unknown i don't actually think so um and part of that is this chart right look at all the other providers, right? Together AI, Fireworks AI are very high-end companies, right?"
},
{
"start_time": 8091.78,
"end_time": 8111.9,
"content": " XMETA, Together AI is Tree Dow and the inventor of like flash attention, right, which is a huge efficiency technique, right? They're very efficient, good companies. And they're certain, and I do know those companies make money, right? Not tons of money on inference, but they make money. And so they're serving at like a five to seven X difference and cost, right? And so, you know, now when you, when you equate, okay, open eyes"
},
{
"start_time": 8111.9,
"end_time": 8132.06,
"content": " making tons of money, that's like a 5x difference. And the companies that are trying to make money for this model is like a 5x difference. There is still a gap, right? There's still a gap. And that is just DeepSeek being really freaking good, right? The model architecture, MLA, the way they did the M-O-E, all these things. There is like legitimate just efficiency differences. All their, all their low-level libraries that we talked about in training, some of them probably translate to inference"
},
{
"start_time": 8132.06,
"end_time": 8155.28,
"content": " and those weren't released. So we may go a bit into conspiracy line, but is it possible the Chinese government is subsidizing Deepseek? I actually don't think they are. I think when you look at the Chinese labs, there's, there's Huawei has a lab, Moonshot AI. There's a couple other labs out there that are really close with the government. And then there's labs like Alibaba and DeepSeek, which are not close with the government."
},
{
"start_time": 8155.28,
"end_time": 8176.28,
"content": " And we talked about this, the CEO, this, this, this like reverent figure who's like quite different, who has like sounds awesome. Very different like viewpoints based on the Chinese interviews that are translated, then what the CCP might necessarily want. Now, to be clear, right, does he have a loss leader because he can fund it through his hedge fund? Yeah, sure. So the hedge fund might be subsidizing it."
},
{
"start_time": 8176.28,
"end_time": 8198.22,
"content": " Yes. I mean, they absolutely did, right? Because Deep Sea has not raised much money. They're now trying to raise around in China, but they have not raised money historically. It's all just been funded by the hedge fund. And he owns like over half the company, like 50, 60% of the company is owned by him. Some of the interviews, there's a discussion on how like doing this is a recruiting tool. You see this at the American companies too. It's like having GPUs recruiting tool, being at the cutting edge of"
},
{
"start_time": 8198.22,
"end_time": 8220.48,
"content": " AI recruiting tool. Open sourcing. Open sourcing. Meta's got so much talent. They were so far behind and they got so much talent. Yeah. Because they just open source stuff. More conspiracy thoughts. Is it possible since they're a hedge fund? more conspiracy thoughts. Is it possible since they're a hedge fund that they timed everything with this release and the pricing and they have they shorted in vitiya stock and stock of u.sia companies"
},
{
"start_time": 8220.48,
"end_time": 8241.85,
"content": " and released it with Stargate, like just perfect timing to be able to make money. Like they released it on an inauguration day. They know the international what is on the international calendar, but I mean, I don't expect them to. If you listen to their motivations for AI, it's like, no, if you release, they released V3 on like December 26th."
},
{
"start_time": 8241.95,
"end_time": 8264.69,
"content": " Like who releases the day? No one looks, right? They released the papers before this, right? The V3 paper and the R1 paper. So people have been looking at it and been like, wow. And then they just released the R1 model. I think they're just shipping as fast as they can and like who cares about christmas who cares about you know get it out before chinese new year right obviously which just happened um i don't think they actually were like timing the market or trying to make the biggest splash"
},
{
"start_time": 8264.69,
"end_time": 8286.65,
"content": " possible. I think they're just like shipping. I think that's one of their big advantages. We know that a lot of the American companies are very invested in safety, and that is the central culture of a place like Anthropic. And I think Anthropic sounds like a wonderful place to work. But if safety is your number one goal, it takes way longer to get artifacts out. That's why Anthropic is not open sourcing things."
},
{
"start_time": 8286.65,
"end_time": 8307.95,
"content": " That's their claims. But there's reviews internally. Anthropic mentions things to international governments. There's been news of how Anthropic has done pre-release testing with the UKAA Safety Institute. All of these things add inertia to the process of getting things out. And we're on this trend line where progress is very high. So if you reduce the time from when your model is done training,"
},
{
"start_time": 8307.95,
"end_time": 8329.73,
"content": " you run a vales that's good, you want to get it out as soon as possible to maximize the perceived quality of your outputs. Deep Seek does this so well. Dario explicitly said Claude 3.5 Sonnet was trained like nine months or a year ago. Nine to ten months ago nine to ten months ago and i think it took them another like handful of months to release it right so it's like there is there is a significant gap here, right?"
},
{
"start_time": 8329.73,
"end_time": 8350.45,
"content": " And especially with reasoning models, the word in the San Francisco street is that like Anthropic has a better model than O3, right? And they won't release it. Why? Because chains of thought are scary, right? And they are legitimately scary, right? If you look at R1, it flips back and forth between Chinese and English, sometimes it's gibberish, and then the right answer comes out. and like for you and I it's like great"
},
{
"start_time": 8350.45,
"end_time": 8373.01,
"content": " great I mean like people are infatuated right there like you're telling me this is a high value thing and it works and it's doing this it's amazing i mean i mean you you talked about that uh sort of like uh chain of thought for that philosophical thing, which is not something they trained to it to be philosophically good. It's just sort of an artifact of the chain of thought training. It did. But like that's super important in that like can I inspect your mind"
},
{
"start_time": 8373.01,
"end_time": 8397.96,
"content": " and what you're thinking right now no and so I don't know if you're lying to my face. And chain of thought models are that way, right? Like, this is a true quote unquote risk between, you know, a chat application where, hey, I asked the model to say, you know, bad words or whatever or how to how to make anthrax. And it tells me, that's unsafe, sure, but that's something I can get out relatively easily. What if I tell the AI to do a task, and then it does the task all of a sudden randomly in a way that I don't"
},
{
"start_time": 8397.96,
"end_time": 8426.43,
"content": " want it. Right. And now that has like much more task versus like response is very different. Right. So the bar for safety is much higher. At least this is Anthropics case, right? Like for deep seek, they're like ship, right so i mean the bar for safety is probably lowered a bit because of deep seek i mean there's parallels here to the space race. The reason the Soviets probably put a man in space first is because their approach to safety was uh the bar for safety was lower and they"
},
{
"start_time": 8426.43,
"end_time": 8450.19,
"content": " they killed that dog right and all these things right so it's like a less risk averse uh than the then the the u.s based program and there's parallels here. But, you know, there's probably going to be downward pressure on that safety bar for the U.S. companies, right? And this is something that Dario talks about. That's the situation that Dario wants to avoid. Is Dario talks to about the difference between race to the bottom and race to the top."
},
{
"start_time": 8450.61,
"end_time": 8471.61,
"content": " And the race to the top is where there's a very high standard on safety. There's a very high standard on your model forms and certain crucial evaluations and when certain companies are really good to it, they will converge. This is the idea. And ultimately, AI is not confined to one nationality or to one like set of morals for what it should mean."
},
{
"start_time": 8471.61,
"end_time": 8493.37,
"content": " And there's a lot of arguments on like, should we stop open sourcing models? And if the US stops, it's pretty clear. I mean, it's way easier to see now at DeepSeek that a different international body will be the one that builds it. We talk about the cost of training. DeepSeek has this shocking $5 million number. Think about how many entities in the world can afford 100 times that to have the best"
},
{
"start_time": 8493.37,
"end_time": 8514.61,
"content": " open source model that people use in the world. And it's like, it's a scary reality, which is that these open models are probably going to keep coming for the time being, whether or not we want to stop them. And it is like stopping them might make it even worse and harder to prepare, but it just means that the preparation and understanding what AI can do is just so much more important."
},
{
"start_time": 8514.61,
"end_time": 8535.84,
"content": " That's why I'm here at the end of the day, but it's like, letting that sink into people, especially not in AI, is that this is coming. There are some structural things in a global interconnected world that you have to accept. Yeah, you mentioned Ysumby, something that Mark Zuckerberg mentioned on the earnings call."
},
{
"start_time": 8535.84,
"end_time": 8556.88,
"content": " He said that I think in light of some of the recent news, the new competitor, Deep Seek from China, I think it's one of the things that we're talking about is there's going to be an open source standard globally. And I think for our kind of national advantage, it's important that it's an American standard. So we take that seriously. We want to build the AI system that people around the world they're using. And I think that if anything,"
},
{
"start_time": 8559.3,
"end_time": 8580.28,
"content": " some of the recent news has only strengthened our conviction that this is the right thing to be focused on. So, yeah, open sourcing. Yeah, Mark Zuckerberg is not new to having American values and how he presents his company's trajectory. I think their products have long since been banned in China, and I respect the saying it directly. And there's an interesting aspect of just because it's open-wates or open-source doesn't mean"
},
{
"start_time": 8580.28,
"end_time": 8600.34,
"content": " it can't be subverted, right? There have been many open source software bugs that have been like, you know, for example, there was a Linux bug that was found after like 10 years, which was clearly a back door, because somebody was like, why is this taking, you know, half a second to the recent one. Like, why is it taking half a second to load? And it was like, oh, crap, there's a back door here. That's why. Right?"
},
{
"start_time": 8600.38,
"end_time": 8621.71,
"content": " And it's like, this is very much possible with AI models right um today you know the the alignment of these models is very clear, right? Like, I'm not going to say, you know, bad words. I'm not going to teach you how to make anthrax. I'm not going to talk about Tiananmen Square. I'm not going to, you know, things like, I'm going to say Taiwan is part of, you know, is just in Eastern province, right?"
},
{
"start_time": 8621.71,
"end_time": 8642.55,
"content": " Like, you know, all these things are like, depending on who you are, what you align, what, you know, whether, you know, even like XAI is aligned a certain way, right? You know, they might be, it's not aligned in the like woke sense. It's not aligned in like pro-China sense, but there is certain things that are imbued within the model. Now when you release this publicly in an instruct model that's open weights, this can then proliferate, right?"
},
{
"start_time": 8642.55,
"end_time": 8665.15,
"content": " But as these systems get more and more capable, what you can embed deep down in the model is not as clear, right? Um, and so there are, that is like one of the big fears is like, if a, an American model or a Chinese model is the top model, right? You're going to embed things that are unclear. And it could be unintentional too, right? Like British English is dead because american l l l ms won right and the internet is american"
},
{
"start_time": 8665.15,
"end_time": 8686.79,
"content": " and therefore like color is spelled the way american spell it. Right? And this is just strong words right now. This is just like this is just the factual nature of the LLLF. The right way to say like carpet with each be. The English is the hottest programming language, and that English is defined by a bunch of companies that primarily are in San Francisco. The right way to spell optimization is with a Z, just in case you're probably."
},
{
"start_time": 8688.81,
"end_time": 8709.69,
"content": " I think it's an S and British English. It is. Taking it is something silly, right? Like something as silly as the spelling, like which British and English, you know, Brits and Americans will like laugh about probably, right? I don't think we care that much. But like, you know, some people will, but like, this can, this can boil down into like very, very important topics. Like, hey, you know, subverting people, right?"
},
{
"start_time": 8710.25,
"end_time": 8731.19,
"content": " You know, chatbots, right? Character AI has shown that they can like, you know, talk to kids and adults and like, you know, talk to kids and adults and like it will, like, you people feel a certain way, right? And that's unintentional alignment. But like, what happens when there's intentional alignment deep down on the open source standard? It's a backdoor today for like Linux, right, that we discover or some encryption system, right, China uses different"
},
{
"start_time": 8731.19,
"end_time": 8752.61,
"content": " encryption than NIST defines the US NIST because there's clearly, at least they think there's back doors in it, right? What happens when the models are backdoors not just to computer systems, but to our minds. Yeah, they're cultural black doors. The thing that amplifies the relevance of culture with language models is that we are used to this mode of interacting with people"
},
{
"start_time": 8752.61,
"end_time": 8773.05,
"content": " in back and forth conversation. And we have now have a super, a very powerful computer system that slots into a social context they were used to, which makes people very, we don't know the extent that which people can be impacted by that. So there could be, this is an actual concern"
},
{
"start_time": 8773.05,
"end_time": 8798.72,
"content": " with a Chinese company that is providing open weights models is that there could be some secret Chinese government sort of requirement for these models to have a certain kind of backdoor to have some kind of thing where... I don't necessarily think it'll be a backdoor, right? Because once it's open weights, it doesn't like phone home. It's more about, like, if it recognizes a certain system it could like if now it could be a backdoor in the"
},
{
"start_time": 8798.72,
"end_time": 8819.28,
"content": " sense of like hey if you're building a software uh you know something something in software, all of a sudden it's a software agent, oh, program this backdoor that only we know about. Or it could be like subvert the mind to think that like X, Y, Z opinion is the correct one. Anthropic has research on this where they show that if you put different phrases, certain phrases in at pre-training, you can then elicit"
},
{
"start_time": 8819.28,
"end_time": 8841.14,
"content": " different behavior when you're actually using the model because they've like poisoned the pre-training data. I don't think like, as of now, I don't think anybody in a production system is trying to do anything like this. I think it's mostly Anthropics are doing very direct work and mostly just subtle things. We don't know what these models are going to how they are going to generate tokens,"
},
{
"start_time": 8841.38,
"end_time": 8870.21,
"content": " what information they're going to represent and what the complex representations they have are. Well, one of the, we're talking about Anthropic, which is generally just permeated with like good humans trying to do good in the world. I don't, we just don't know do good in the world. We just don't know of any labs. This would be done in the military context that are explicitly trained to, okay, how can we, the front door looks like a happy NLM,"
},
{
"start_time": 8870.71,
"end_time": 8899.04,
"content": " but underneath, it's a thing that will over time do the maximum amount of damage to our quote unquote enemies. There's this very good quote from Sam Altman who, you know, he can be a hypebeast sometime. But one of the things he said, and I think I agree is that superhuman persuasion will happen before superhuman intelligence. Right? And if that's the case, then these things before, before we get this AGIASI stuff, we can embed superhuman persuasion towards our ideal or whatever"
},
{
"start_time": 8899.04,
"end_time": 8920.44,
"content": " the ideal of the model maker is. And again, like today, I truly don't believe Deep Seek has done this, right? But it is a sign of, like, what could happen. So one of the dystopian worlds is described by Brave New World. So we could just be stuck scrolling Instagram looking at cute puppies or worse, and then talking to bots that are giving us a narrative"
},
{
"start_time": 8920.44,
"end_time": 8950.61,
"content": " and we completely get lost in that world that's controlled by somebody else, versus thinking independently. And that's, that's a major concern as we rely more and more on these kinds of systems i mean we've already seen that with recommendation systems yeah recommendation systems hack the dopamine-induced reward circuit, but the brain is a lot more complicated and what other sort of circuits, quote-unquote, feedback loops in your brain, can you hack slash subvert in ways like recommendation systems are purely just trying to do, you know, increase time and ads and etc."
},
{
"start_time": 8950.61,
"end_time": 8972.11,
"content": " But there's so many more goals that can be achieved through these complicated models. There's no reason in some number of years that you can't train a language model to maximize time spent on a chat app. Like right now they are trained. I mean, is that not what character AI has done? Their time per session is like two hours. Yeah, character AI very likely could be optimizing this."
},
{
"start_time": 8972.11,
"end_time": 8992.65,
"content": " Where it's like the way that this data is collected is naive, whereas you're presented a few options and you choose them, but there's, that's not the only way that these models are gonna be trained. It's naive stuff like talk to an anime girl, but like it can be like, yeah, this is a risk, right? Like, it's a bit of a cliche thing to say, but I've over the past year I had a few stretches of time where I didn't"
},
{
"start_time": 8992.65,
"end_time": 9014.55,
"content": " use social media or the internet at all and just read books and was out in nature and it clearly has an effect on the mind where like it changed I feel like I'm returning of course I was raised before the internet really took off but I'm returning to someone. I know where you're going. I mean, you can see it physiologically."
},
{
"start_time": 9014.55,
"end_time": 9035.3,
"content": " Like, I take three days if I'm like backpacking or something and you you're your literal like you're breaking down addiction cycles. I feel like I'm more in control of my mind. There feels like a sovereignty of intelligence that's happening when I'm disconnected from the internet. I think the more I use the internet and social media, the more other people"
},
{
"start_time": 9035.3,
"end_time": 9058.48,
"content": " are controlling my mind. That's definitely a feeling. And then in the future that would be not other people but algorithms. Or other people but algorithms, or other people presented to me via algorithms. There, I mean, there are already tons of AI bots on the internet. And every so, right now it's not frequent, but every so often I have replied to one, and they're instantly replies. I'm like, crap out of the bot. And that is just going to become more common. Like, they're going to get good."
},
{
"start_time": 9058.48,
"end_time": 9080.96,
"content": " One of the hilarious things about technology over its history is that the illicit adult entertainment industry has always adopted technologies first, right? Whether it was like video streaming, um, to like where, you know, there's now the like sort of like independent adult illicit content creators who have their, you know, subscription pages. And there they actually heavily utilize, you know,"
},
{
"start_time": 9080.96,
"end_time": 9101.32,
"content": " generative AI has already been like diffusion models and all that is huge there. But now these like these subscription based individual creators do use bots to approximate themselves and chat with their, you know, people pay a lot for it. And people pay a lot, right? A lot of times it's them, but a lot of, there are agencies that do this for these creators and do it like on a mass scale."
},
{
"start_time": 9101.32,
"end_time": 9122.14,
"content": " So the largest creators are like able to talk to hundreds or thousands of people at a time because of these bots. And so it's already being used there. Obviously, you know, like video streaming and and other technologies have come there first, it's going to come to the rest of society too. There's a general concern that models get censored by the companies that deploy them."
},
{
"start_time": 9122.84,
"end_time": 9145.7,
"content": " So one case, we've seen that, and maybe censorship is one word, alignment, alignment, maybe via RLHF or some other way is another word. So we saw that with Black Nazi Image Generation with Gemini. As you mentioned, we also see that with Chinese models refusing to answer what happened."
},
{
"start_time": 9146.74,
"end_time": 9165.94,
"content": " In June 4th, 1989 at Tiananmen Square. So how can this be avoided? And maybe can you just in general talk about how this happens and how can it be avoided? You give multiple examples. There's probably a few things to keep in mind here."
},
{
"start_time": 9165.94,
"end_time": 9190.61,
"content": " One is the kind of Tiananmen Square factual knowledge, like, did think, like, how does that get embedded into the models? Two is the Gemini, what you called the Black Nazi incident, which is when Gemini, as a system, had this extra thing put into it that dramatically changed the behavior. And then three is what most people would call general alignment, RLHF post-training."
},
{
"start_time": 9190.61,
"end_time": 9212.49,
"content": " Each of these have very different scopes in how they are applied. In order to do, if you're just to look at the model weights, in order to audit specific facts is extremely hard because you have to chrome through the pre-training data and look at all of this and then that's terabytes of files and look for very specific words or hints of the words."
},
{
"start_time": 9212.49,
"end_time": 9238.52,
"content": " So I guess one way to say is that you can insert censorship or alignment at various stages in the pipeline. And what you refer to now is at the very beginning of the data selection. So if you want to get rid of facts in a model, you have to do it at every stage. You have to do it at the pre-training. So most people think that pre-training is where most of the knowledge is put into the model and then you can elicit and move that in different ways, whether through post-training or whether through systems afterwards."
},
{
"start_time": 9238.78,
"end_time": 9262.5,
"content": " This is where the whole like hacking models comes from, right? Like, GPT will not tell you how to make anthrax, but if you try really, really hard, you can eventually get to tell you about anthrax. Because they didn't filter it from the pre-training data set, right? But by the way, removing facts has such an ominous dark feel to it. Almost think it's practically impossible. Because you effectively have to remove them from the internet."
},
{
"start_time": 9262.5,
"end_time": 9284.54,
"content": " You're taking on a... Did they remove the m-thin thing from the subreditsits, the MMM, it gets filtered out. Right. So you have quality filters, which are small language models that look at a document and tell you like, how good is this text? Is it close to a Wikipedia article, which is a good thing that we want language models to be able to imitate. So couldn't you do a small language model that"
},
{
"start_time": 9284.54,
"end_time": 9304.56,
"content": " Fulteschild mentions at Tiananmen Square in the data? Yes, but is it going to catch wordplay or encoded language of the same thing? People have been meaning on like games and other stuff, how to like say things that don't say Tiananmen Square. But or like, yeah, so there's always like different ways to do it. There's, hey, the internet as a whole does tend to just have a slight"
},
{
"start_time": 9304.56,
"end_time": 9324.82,
"content": " left bias, right? Because it's always been richer, more affluent, younger people on the internet relative to the rest of the population. So there is already inherently a slight left bias on the internet. And so how do you filter things that are this complicated, right? Is it like, and some of these can be like, you know, factual, non-factual, but like Tiananmen Square is"
},
{
"start_time": 9324.82,
"end_time": 9345.9,
"content": " obviously the example of a factual, but it gets a lot harder when you're talking about aligning to a ideal, right? And so Grock, for example, right? Elon's tried really hard to make the model not be super PC and woke, but the best way to do pre-training is to throw the whole freaking internet at it, right? And then later figure out, but then at the end of the day, the model at its"
},
{
"start_time": 9345.9,
"end_time": 9370.11,
"content": " core now still has some of these ideals, right? You still ingested Reddit slash R slash politics, which is probably the largest political discussion board on the world that's freely available to scrape. And guess what? That's left leaning, right? And so, you know, there are some aspects like that you just can't censor unless you try really, really, really, really hard. So the base model will always have some TDS Trump Drangement syndrome"
},
{
"start_time": 9370.11,
"end_time": 9392.21,
"content": " because it's trained so much. It'll have the ability to express it. But what if, what if you, there's a wide representation in the data. This is what happens. It's like a lot of modern, what is called post-training it's a series of techniques to get the model on rails of a really specific behavior. And I mean, it's like you can, you also have the ingested data of like Twitter or like"
},
{
"start_time": 9392.21,
"end_time": 9412.53,
"content": " Reddit slash R slash the Donald, which is like also super pro-Trump, right? And then you have like fascist subredits or like you have communist subredits. So the model in pre-training ingests everything. It has no worldview. Now, it does have like some, some skew because more of the text is skewed a certain way, which is general, like slight left, like, but also like, you know,"
},
{
"start_time": 9412.53,
"end_time": 9432.99,
"content": " somewhat like, you know, intellectual, somewhat like, you know, it's just like the general internet is a certain way and then and then as as as nathan's about to describe eloquently right like you can you can elicit certain things out. And there's a lot of history here. So we can go through multiple examples and what happened. Lama 2 was a launch that the phrase like too much RLHF or like too much safety was a lot."
},
{
"start_time": 9433.23,
"end_time": 9454.93,
"content": " It's just that was the whole narrative after Lama 2's chat models released. And the examples are sorts of things like you would ask Lama 2 chat, how do you kill a Python process? And it would say I can't talk about killing because that's a bad thing. And anyone that is trying to design an AI model will probably agree that that's just like a model you messed up a bit on the training there I don't think"
},
{
"start_time": 9454.93,
"end_time": 9476.23,
"content": " they meant to do this but this was in the model weight. So this is not, it didn't necessarily be, there's things called system prompts, which are when you're querying a model, it's a piece of text that is shown to the model, but not to the user. So a fun example is your system prompt could be talk like a pirate. So no matter what the user says to the model, it'll respond like a pirate."
},
{
"start_time": 9476.23,
"end_time": 9498.19,
"content": " In practice, what they are is you are a helpful assistant. You should break down problems. If you don't know about something, don't tell them your date cut off is this, today's date is this. It's a lot of really useful context for how can you answer a question well. Anathropic publishes their system problem type. But I think is great. And there's a lot of research that goes into this and one of your previous guests, Amanda Askell, is probably the"
},
{
"start_time": 9498.19,
"end_time": 9518.27,
"content": " most knowledgeable person, at least in the combination of execution and sharing. She is the person that should talk about system prompts and character of models. Yeah, and then people should read the system prompts because you're like trying to nudge sometimes through extreme politeness, the model to be a certain way. And you could use this for bad things."
},
{
"start_time": 9518.27,
"end_time": 9545.8,
"content": " We've done tests, which is, what if I tell the model to be a dumb model, like which evaluation scores go down? And it's like, we'll have this behavior where it could sometimes say, oh, I'm supposed to be dumb. And sometimes it doesn't affect math abilities as much, but something like if you're trying it's just the quality of a human judgment would draw through the forest let's go back to post-training specifically rlhf aroundama 2 was it was too much RELA, too much safety prioritization was"
},
{
"start_time": 9545.8,
"end_time": 9565.9,
"content": " baked into the model weights. This makes you refuse things in a really annoying way for users. It's not great. It caused a lot of awareness to be attached to RLHF that it makes the models dumb. And it stigmatized the word. It did. In AI culture. And as the techniques have involved, that's no longer the case where all of these labs"
},
{
"start_time": 9565.9,
"end_time": 9593.17,
"content": " have very fine-grained control over what they get out of the models through techniques like RLHF. Although, although different labs are definitely different levels, like on the, on one's end of the spectrum is Google and then like maybe opening eye does less and Anthropic does less and then like on the other end of the spectrum is like XAI, but they all have different forms of RLHF trying to make them a certain way. And like the important thing to say is that no matter how you want the model to behave,"
},
{
"start_time": 9593.59,
"end_time": 9616.07,
"content": " these RLHF and preference tuning techniques also improve performance. So on things like math of vowels and code of vowels, there is something innate to these what is called contrastive loss functions. We could start to get into RL here. We don't really need to, but REL also boost performance on anything from a chat task to a math problem to a code problem. So it is becoming a much more useful tool to these labs."
},
{
"start_time": 9616.07,
"end_time": 9638.07,
"content": " So this kind of takes us through the arc of we've talked about pre-training, hard to get rid of things. We've talked about post-training and how post-training, you can mess it up. It's a complex, multifaceted optimization with 10 to 100-person teams converging of one artifact. It's really easy to not do it perfectly. And then there's the third case, which is what we talked about Gemini. The thing that was about Gemini is this was a served"
},
{
"start_time": 9638.07,
"end_time": 9658.27,
"content": " product where Google has their internal model weights, they've done all these processes that we talked about. And in the served product, what came out after this was that they had a prompt that they were rewriting user queries to boost diversity or something. And this just made it, the outputs were just blatantly wrong. It was some sort of organizational failure that had this prompt in that position,"
},
{
"start_time": 9658.27,
"end_time": 9679.29,
"content": " and I think Google executives probably have owned this. I didn't pay that tension that detail. But it was just a mess up in execution that led to this ridiculous thing, but at the system level. The model weights might have been fine. So at the very end of the pipeline, there was a rewriting. To something like a system prompt. It was like the system prompt. It was like the system prompt or what is called an industry is like you rewrite"
},
{
"start_time": 9679.29,
"end_time": 9702.82,
"content": " prompts. So especially for image models, if you're using Dolly or ChatTBT, you can generate you an image, you'll say, draw me a beautiful car. With these leading image models, they benefit from highly descriptive prompts. So what would happen is if you do that on chatypt, a language model behind the scenes will rewrite the prompt, say make this more descriptive, and then that is passed to the image model."
},
{
"start_time": 9702.82,
"end_time": 9725.76,
"content": " So prompt rewriting is something that is used at multiple levels of industry, and it's used effectively for image models, and the Gemini example is just a failed execution. Big philosophical question here with the RLHF to generalize, where is human input, human in the loop, human data most useful at the current stage."
},
{
"start_time": 9725.76,
"end_time": 9754.37,
"content": " For the past few years, the highest cost human data has been in these preferences, which is comparing, I would say highest cost and highest total usage. So a lot of money has gone to these pairwise comparisons where you have two model outputs and a human is comparing between the two of them. In earlier years, there was a lot of this instruction tuning data. So creating highly specific examples to something like a Reddit question to a domain that you care about."
},
{
"start_time": 9754.37,
"end_time": 9775.71,
"content": " Language models use the struggle on math and code, so you would pay experts in math and code to come up with questions and write detailed answers that were used to train the models. Now it is the case that there are many model options that are way better than humans at writing detailed and eloquent answers for things like model and code. So they talked about this with the Lama 3 release,"
},
{
"start_time": 9775.71,
"end_time": 9799.03,
"content": " where they switched to using Lama 3, 4, or 5B to write their answers for math and code. But they, in their paper talk about how they use extensive human preference data, which is something that they haven't gotten AIs to replace. There are other techniques in industry like constitutional AI, where you use human data for preferences and AI for preferences, and I expect the AI part to scale faster than the human part. But among the"
},
{
"start_time": 9799.03,
"end_time": 9821.03,
"content": " research that we have access to is that it humans are in this kind of preference loop. So for as reasoning becomes bigger and bigger and bigger, as we said, where's the role of humans in that? It's even less prevalent. So the remarkable thing about these reasoning results, and especially the DeepSeek R1 paper, is this result that they call DeepSeek R10,"
},
{
"start_time": 9821.37,
"end_time": 9845.36,
"content": " which is they took one of these pre-trained models. They took deep seek v3 base, and then they do this reinforcement learning optimization on verifiable questions or verifiable rewards for a lot of questions and a lot of training and these reasoning behaviors emerge naturally. So these things like wait, let me see, wait, let me check this. Oh, that might be a mistake, and they emerge from only having questions and answers."
},
{
"start_time": 9845.36,
"end_time": 9867.94,
"content": " And when you're using the model, the part that you look at is the completion. So in this case, all of that just emerges from this large scale RL training. And that model, which the weights are available, has no human preferences added into the post-training. There are, the DeepSe seek R1 full model has some of this human preference tuning, this RLHF, after the reasoning stage."
},
{
"start_time": 9867.94,
"end_time": 9892.12,
"content": " But the very remarkable thing is that you can get these reasoning behaviors, and it's very unlikely that there's humans writing out reasoning chains. It's very unlikely that they somehow hacked open AI and they got access to open AI O-1's reasoning chains. It's something about the pre-trained language models and this RL training, where you reward the model for getting the question right. And therefore, it's trying multiple solutions and it emerges this chain of thought"
},
{
"start_time": 9892.68,
"end_time": 9915.15,
"content": " this might be a good place to mention the eloquent and the insightful tweet of the great and the powerful Andre Carpathie. I think he had a bunch of thoughts, but one of them, last thought, not sure if this is obvious, you know something profound is coming when you're saying it's not sure if it's obvious. There are two major types of learning in both children and in deep learning."
},
{
"start_time": 9915.61,
"end_time": 9936.77,
"content": " There's one, imitation learning, watch and repeat, i.e. pre-training, supervised fine-tuning, and two, trial and error learning, reinforcement learning. My favorite simple example is AlphaGo. One is learning by imitating expert players. Two is reinforcement learning to win the game. Almost every single shocking result of deep learning,"
},
{
"start_time": 9936.77,
"end_time": 9958.07,
"content": " and the source of all magic is always two. Two is significantly more powerful. Two is what surprises you. Two is when the paddle learns to hit the ball behind the blocks and break out. Two is when AlphaGo beats even Lee Sedal. And two is the aha moment when the deep seek or 01, et cetera,"
},
{
"start_time": 9958.19,
"end_time": 9986.28,
"content": " discovers that it works well to reevaluate your assumptions, backtrack, try something else, et cetera. It's the solving strategies you see this model use in its chain of thought. It's how it goes back and forth thinking to itself. These thoughts are emergent, three exclamation points. And this is actually seriously incredible, impressive and new, and is publicly available and documented. The model could never learn this with"
},
{
"start_time": 9986.28,
"end_time": 10008.68,
"content": " imitation, because the cognition of the model and the cognition of the human labeler is different. The human would never know to correctly annotate these kinds of solving strategies and what they should even look like. They have to be discovered during reinforcement learning as empirically statistically useful towards the final outcome. Anyway, the alpha zero sort of metaphor analogy here."
},
{
"start_time": 10009.26,
"end_time": 10032.82,
"content": " Can you speak to that, the magic of the chain of thought that he's referring to? I think it's good to recap AlphaGo and Alpha Zero because it plays nicely with these analogies between imitation learning and learning from scratch. So AlphaGo, the beginning of the process was learning from humans where they had, they started the first, this is the first expert level go player or chess player in Deep Mind series of models where they had some human data."
},
{
"start_time": 10033.48,
"end_time": 10054.84,
"content": " And then why it is called Alpha Zero is that there was zero human human data in the loop. And that changed to Alpha Zero made a model that was dramatically more powerful for deep mind. So this remove of the human prior, the human inductive bias makes the final system far more powerful. This we mentioned bitter lesson hours ago, and this is all aligned with this."
},
{
"start_time": 10060.17,
"end_time": 10080.07,
"content": " And then there's been a lot of discussion in language models. This is not new. This goes back to the whole Q-star rumors, which if you piece together the pieces, is probably the start of Open AI figuring out its O1 stuff when last year in November the QStar rumors came out. There's a lot of intellectual drive to know when is something like this going to happen with language models"
},
{
"start_time": 10080.07,
"end_time": 10103.94,
"content": " because we know these models are so powerful and we know it has been so successful in the past. And it is a reasonable analogy that this new type of reinforcement learning training for reasoning models is when the door is open to this. We don't yet have the equivalent of turn 37, which is the famous turn where the Deep Mines AI playing ghost, stumped Lee-Saddle completely."
},
{
"start_time": 10103.94,
"end_time": 10125.94,
"content": " We don't have something that's that level of focal point, but that doesn't mean that the approach to technology is different and the impact of the general training. It's still incredibly new. What do you think that point would be? Will we'll be Move 37 for chain of thought, for reasoning? Scientific discovery. Like when you use this sort of reasoning problem and it's just something we fully don't expect. I think it's actually probably simpler than that."
},
{
"start_time": 10125.94,
"end_time": 10152.92,
"content": " It's probably something related to computer user robotics rather than science discovery. Because the important aspect here is models take so much data to learn, they're not sample efficient, right? Trillions, they take the entire web, right? Over 10 trillion tokens to train on, right? This would take a human thousands of years to read, right? A human does not, and humans know most of the stuff,"
},
{
"start_time": 10152.92,
"end_time": 10177.27,
"content": " a lot of stuff models know better than it, right? Humans are way, way, way more sample efficient. That is because of the self-play, right? How does a baby learn what its body is as it sticks its foot in its mouth and it says, oh, this is my body, right? It sticks its hand in its mouth and it calibrates its touch on its fingers with the most sensitive touch thing on its tongue, right? It's how babies learn. And it's just self-play over and over and over again."
},
{
"start_time": 10177.27,
"end_time": 10197.67,
"content": " And now we have something that is similar to that, right, with these verifiable proofs, right? Whether it's a unit test and code or mathematical verifiable task generate many traces of reasoning, right? And keep branching them out, keep branching them out. And then check at the end, hey, which one actually has the right answer? Most of them are wrong, great."
},
{
"start_time": 10197.67,
"end_time": 10219.27,
"content": " These are the few that are right. Maybe we use some sort of reward model outside of this to select even the best one to preference as well. But now you've started to get better and better at these benchmarks. And so you've seen over the last six months a skyrocketing in a lot of different benchmarks, right? All math and code benchmarks were pretty much solved, except for frontier math, which is designed to be almost questions that aren't practical to most people."
},
{
"start_time": 10219.27,
"end_time": 10242.4,
"content": " Because they're like, their exam level open math problem type things. So it's like on the math problems that are somewhat reasonable, which is like somewhat complicated word problems or coding problems. It's just what Dylan is saying. So the thing here is that these are only with verifiable tasks. We earlier showed an example of the, you know, the really interesting, like what happens when Chana thought is to a non-verifiable thing."
},
{
"start_time": 10242.74,
"end_time": 10265.66,
"content": " There's just like a human, you know, chatting, right? With the, you know, thinking about what's novel for humans, right? A unique thought. But this task and form of training only works when it's when it's verifiable. And from here, the thought is, okay, we can continue to scale this current training method by increasing the number of verifiable tasks. In math and coding, coding, coding probably has a lot more to go. Math has a lot less to go"
},
{
"start_time": 10265.66,
"end_time": 10287.7,
"content": " in terms of what are verifiable things. Can I create a solver that then I generate trajectories toward or traces towards, reasoning traces towards, and then prune the ones that don't work and keep the ones that do work. Well, those are going to be solved pretty quickly, but even if you've solved math, you have not actually created intelligence, right? And so this is where I think the like, aha moment of computer use or robotics will come in"
},
{
"start_time": 10287.7,
"end_time": 10307.8,
"content": " because now you have a sandbox or a playground that is infinitely verifiable, right? Did you, you know, messing around on the internet, there are so many actions that you can do that are verifiable. It'll start off with like log into a website, create an account, click a button here, blah, blah, blah. But it'll then get to the point where it's, hey, go do a task on tasker or whatever these"
},
{
"start_time": 10307.8,
"end_time": 10330.84,
"content": " other, all these various task websites. Hey, go get hundreds of likes, right? And it's going to fail. It's going to spawn hundreds of accounts. It's going to fail on most of them. But this one got to a thousand. Great. Now you've reached the verifiable thing. And you just keep iterating this loop over and over. And that's when, and same with the robotics, right? That's where, you know, where you have an infinite playground of tasks like, hey, did I put the ball in the bucket all the way to like, oh, did I like build a car? Right. Like, you know, there's a whole"
},
{
"start_time": 10330.84,
"end_time": 10353.82,
"content": " trajectory to speed run or, you know, what models can do. But at some point, I truly think that like, you know, will spawn models. And initially all the training will be in sandboxes. But then at some point, you know, the language model pre-training is going to be dwarfed by what is this reinforcement learning, you know, you'll pre-trained a multimodal model that can see, that can read, that can write, you know, blah, blah, blah, whatever, vision, audio, et cetera."
},
{
"start_time": 10353.98,
"end_time": 10377.91,
"content": " But then you'll have it play in a sandbox infinitely and figure out, figure out math, figure out code, figure out navigating the web, figure out operating a robot arm, right? And then it'll learn so much and the aha moment I think will be when this is available to then create something that's not good, right? Like, oh, cool. Part of it was like figuring out how to use the web. Now all of a sudden, it's figured out really well how to just get hundreds of thousands of"
},
{
"start_time": 10377.91,
"end_time": 10400.41,
"content": " followers that are real and real engagement on twitter because all of a sudden this is one of the things that are verifiable. And maybe not just engagement, but make money. Yes, of course. I mean, that could be the thing where almost fully automated, it makes, you know, $10 million by being an influencer selling a product, creating the product, like, and I'm not referring to like a hype product,"
},
{
"start_time": 10400.53,
"end_time": 10424.25,
"content": " but an actual product or like, holy shit, this thing created a business. It's running it. It's the face of the business. That kind of thing. Or maybe number one song. Like it creates the whole infrastructure required to create the song to be the influence that represents that song and that kind of thing. It makes a lot of them. That could be the move. I mean, our culture respects money in that kind of way."
},
{
"start_time": 10424.43,
"end_time": 10444.73,
"content": " And it's verifiable, right? It's verifiable. The bank account can't lie. Exactly. There's surprising evidence that once you set up the ways of collecting the verifiable domain that this can work. There's been a lot of research before this R1 on math problems, and they approach math with language models just by increasing the number of samples."
},
{
"start_time": 10444.73,
"end_time": 10471.64,
"content": " So you can just try again and again and again. And you look at the amount of times that the language models get it right. And what we see is that even very bad models get it right sometimes. And the whole idea behind reinforcement learning is that you can learn from very sparse rewards. So it, it doesn't, the space of language and the space of tokens, whether you're generating language or tasks for a robot is so big that you might say that it's like, I mean, each, the tokenizer for"
},
{
"start_time": 10471.64,
"end_time": 10496.35,
"content": " language model can be like 200,000 things. So at each step, it can sample from that big of a space. So if it can generate a bit of a signal that it can come onto, that's what the whole field of RL is around is learning from sparse rewards. And the same thing has played out in math where it's like very weak models that sometimes generate answers, where you see research already that you can boost their math scores. You can do this sort of RL training for math."
},
{
"start_time": 10496.47,
"end_time": 10517.97,
"content": " It might not be as effective, but if you take a $1 billion parameter model, so something 600 times smaller than DeepSeek, you can boost its grade school math scores very directly with a small amount of this training. So it's not to say that this is coming soon. Setting up the verification domains is extremely hard and there's a lot of nuance in this, but there are some basic things that we have seen before,"
},
{
"start_time": 10517.97,
"end_time": 10540.91,
"content": " where it's at least expectable that there's a domain and there's a chance that this works. All right, so we have fun things happening in real time. This is a good opportunity to talk about other reasoning models, 0103, just now OpenAI, as perhaps expected, released O3 Mini. What are we expecting from the different flavors?"
},
{
"start_time": 10541.07,
"end_time": 10564.93,
"content": " Can you just lay out the different flavors. Can you just lay out the different flavors of the O models and the from Gemini, the reasoning model? Something I would say about these reasoning models is we talked a lot about reasoning training on math and code. And what is done is that you have the base model we've talked about a lot on the internet. You do this large-scale reasoning training with reinforcement learning. And then what the deep seek paper detailed in this R1 paper, which for me,"
},
{
"start_time": 10564.93,
"end_time": 10587.53,
"content": " as one of the big open questions on how do you do this is that they did reasoning heavy but very standard post-training techniques after the large-scale reasoning RL. So they did the same things with a form of instruction tuning through rejection sampling, which is essentially heavily filtered instruction tuning with some reward models. And then they did this RLHF, but they made it math heavy."
},
{
"start_time": 10587.53,
"end_time": 10610.54,
"content": " So some of this transfer, we looked at this philosophical example early on. One of the big open questions is how much does this transfer? If we bring in domains after the reasoning training, are all the models going to become eloquent writers by reasoning? Is this philosophy stuff going to be open? We don't know in the research of how much this will transfer. There's other things about how we can make soft verifiers"
},
{
"start_time": 10610.54,
"end_time": 10632.36,
"content": " and things like this. But there is more training after reasoning, which makes it easier to use these reasoning models and that's what we're using right now so if we're going to talk about with three mini and o one these have gone through these extra techniques that are designed for human preferences after being trained to illicit reasoning. I think one of the things that people are ignoring is Google's Gemini, flash thinking,"
},
{
"start_time": 10632.36,
"end_time": 10653.32,
"content": " is both cheaper than R1 and and better. And they released it in the beginning of December. And nobody's talking about it. No one cares. It has a different flavor to it. Its behavior is less expressive than something like 01, or it has fewer tracks than it is on. Quinn released a model last fall, QWQ, which was their preview reasoning model. And in deep-seek had R1 light last fall,"
},
{
"start_time": 10653.32,
"end_time": 10673.36,
"content": " where these models kind of felt like they're on rails, where they really, really only can do math and code. And 01 is it can answer anything. It might not be perfect for some tasks, but it's flexible, it has some richness to it. And this is kind of the art of like how cook, like how it is a model a little bit undercooked. It's like, I mean, it's good to get a model out the door."
},
{
"start_time": 10673.74,
"end_time": 10694.4,
"content": " But it's hard to gauge and it takes a lot of taste to be like, is this a full-fledged model? Can I use this for everything? They're probably more similar for math and code. My quick read is that Gemini Flash is not trained in the same way as 01, but taking an existing training stack, adding reasoning to it."
},
{
"start_time": 10694.4,
"end_time": 10716.57,
"content": " So taking a more normal training stack and adding reasoning to it. So taking a more normal training stack and adding reasoning to it. And I'm sure they're going to have more. I mean, they've done quick releases on Gemini Flash, the reasoning, and this is the second version from the holidays. It's evolving fast and it takes longer to make this training stack where you're doing this large scale r- I get the same question from earlier the one about the human nature."
},
{
"start_time": 10716.95,
"end_time": 10740.87,
"content": " Yeah. What was the human nature one? The way I can ramble, why I can ramble about this so much is that we've been working on this at AI2 before 01 was fully available to everyone and before R1, which is essentially using this RL training for fine-tuning. We use this in our Tulu series of models. And you can elicit the same behaviors where you say like weight and so in so much on,"
},
{
"start_time": 10741.27,
"end_time": 10764.73,
"content": " but it's so late in the training process that this kind of reasoning expression is much lighter. So you can, there's essentially a gradation and just how much of this RL training you put into it determines how the output looks. So we're now using Gemini 2.0 Flash thinking experimental 121. It summarized the problem as humans self-domesticated apes"
},
{
"start_time": 10764.73,
"end_time": 10787.61,
"content": " perspective okay all right wait, is this revealing the, the reasoning? Here's why this is a novel. Okay. Click to expand. Click to expand okay analyze the request novel is the keyword is like see how it just looks a little different it looks like a normal output yeah it's i mean in"
},
{
"start_time": 10787.61,
"end_time": 10808.07,
"content": " some sense it's better structured. It makes more sense. Oh, and it latched onto human, and then it went into organisms and oh wow Apex predator Focus on domestication Apply domestication to humans, explore the idea of self-domestication. Not good. Not good. Where is this going?"
},
{
"start_time": 10808.96,
"end_time": 10828.36,
"content": " Refined articulating Where is this going? Refine, articulate the insight. Greater facial expressiveness than communication ability, yes. Plus, this and adaptability, yes. Plasticity and adaptability, yes. Dependence on social groups, yes. All right. And self-critique and refined further. Wow. Is this truly novel? Is it well supported? So on. is a well supported."
},
{
"start_time": 10830.54,
"end_time": 10851.22,
"content": " So on and so forth. And the insight is getting at is humans are not just social animals, but profoundly self-domesticated apes. And this self-domestication is the key to understanding our unique cognitive and social abilities, self-domesticated apes. Self-dom... I prefer the deep-seek response. Self-dem. I mean, it's novel."
},
{
"start_time": 10851.52,
"end_time": 10879.09,
"content": " The insight is novel. I mean, that's like a good book title, self-dumesticated apes. Like, there could be a case made for that. I mean, yeah, it's cool. And it's revealing the reasoning. It's magical. It's magical. This is really powerful. Hello, everyone. This is Lex with a quick intermission recorded after the podcast Since we reviewed responses from Deep Segar One and Gemini Flash 2.0"
},
{
"start_time": 10879.09,
"end_time": 10910.1,
"content": " thinking during this conversation, I thought at this moment, it would be nice to insert myself quickly doing the same for OpenAI 01-O-1 Pro and O3 Mini with the same prompt, the prompt being, give one truly novel insight about humans. And I thought I would in general give my vibe check and vibe-based anecdotal report on my own experiences with the new O3Mini model,"
},
{
"start_time": 10910.32,
"end_time": 10930.28,
"content": " now that I got a chance to spend many hours with it in different kinds of contexts and applications. So I would probably categorize this question as, let's say, open-ended philosophical question, and in particular the emphasis on novelty I think is a nice way to test one of the capabilities of the model, which is come up with"
},
{
"start_time": 10930.28,
"end_time": 10950.82,
"content": " something that makes you pause and almost surprise you with brilliance. So that said, my general review, after running each of the models on this question a bunch of times is that 01 Pro consistently gave brilliant answers. Once they gave me pause and made me think,"
},
{
"start_time": 10951.12,
"end_time": 10971.48,
"content": " both cutting in its insight and just really nicely phrased with wit, with clarity, with nuance, over and over consistently generating the best answers. After that is R1, which is less consistent, but again, deliver brilliance. Gemini Flash 2.0 thinking was third."
},
{
"start_time": 10972.22,
"end_time": 10992.6,
"content": " And last was O3 Mini, actually. It often gave quite a generic answer, at least to my particular sensibilities. That said in a bunch of other applications that I tested for brainstorming purposes, it actually worked extremely well and often outperformed R1."
},
{
"start_time": 10992.6,
"end_time": 11013.82,
"content": " But on this open-ended philosophical question did consistently worse. Now, another important element for each of these models is how the reasoning is presented. DeepSeek R1 shows the full chain of thought tokens, which I personally just love. For these open end of philosophical questions, it's really, really interesting to see the model think through it,"
},
{
"start_time": 11013.94,
"end_time": 11038.88,
"content": " but really also just stepping back, me as a person who appreciates intelligence and reasoning and reflection, reading these kind of chain of thought raw tokens of R1, there's something genuinely beautiful about observing the path of deliberation in an intelligence system. I think we don't always have that explicitly laid out for us humans."
},
{
"start_time": 11039.16,
"end_time": 11066.99,
"content": " So to see it in another intelligence system, the non-linearity of it akin to Ulysses of Fennigens' wake by James Joyce. It's just beautiful to watch. Anyway, as we discussed in the episode, Deep Seek R1, talked about humans being able to convert selfish desires into cooperative systems by collectively pretending abstract rules like money laws and rights are real, and the shared hallucinations act as games, where competition"
},
{
"start_time": 11066.99,
"end_time": 11088.58,
"content": " is secretly redirected to benefit the group, turning conflict into society's fuel. Gemini 2.0 Flash thinking said humans are not just social animals, but self-domesticated apes. And this self-domestication is the key to understanding our unique cognitive and social abilities. Now, it's important to say that the chain of thought there was really interesting."
},
{
"start_time": 11089.08,
"end_time": 11109.26,
"content": " It was looking through the entire evolution of life on earth considering apex predators and considering how from that we ended up to where we are. I think that domestication by choice is a really interesting angle. Again, it's one of those things when somebody presents a different angle"
},
{
"start_time": 11109.26,
"end_time": 11130.36,
"content": " and a seemingly obvious thing, it just makes me smile. And the same with Deep Seek R1, that these hallucinations of money, laws, and rights, and us collectively pretending like it's real, and we play games with them that look like competition when secretly we're just cooperating with each other. And that is the fuel of progress, beautifully put."
},
{
"start_time": 11131.04,
"end_time": 11151.6,
"content": " Now, opening I.01 Pro consistently over and over, delivered bangers. I can go through many of them, but the first one was humans are the only species that turns raw materials into symbolic resources, then uses those symbols to reorganize the very materials they came from, creating a closed feedback loop between meaning and matter."
},
{
"start_time": 11151.6,
"end_time": 11171.94,
"content": " Here, I just ran it again. Bangor after bangor, I'm telling you, humans are unique among known species in that they simultaneously rewrite two layers of reality, the external world and their own private mental landscapes. And then merge these two rewritten layers into a continuous personal narrative"
},
{
"start_time": 11171.94,
"end_time": 11193.36,
"content": " that feels objectively true. Feels true. Feels true. This is poetry. Okay. And then O3 Mini mini high for me was smart fast actually and kind of generic never quite got kind of generic. Never quite got there for me. So here's the first one I got from O3 Mini."
},
{
"start_time": 11194,
"end_time": 11215.76,
"content": " Humans are not fixed beings, but rather ongoing narratives, dynamic stories that we continuously write, edit, and reinterpret. This narrative plasticity is more than just memory or self-reflection. It's an intrinsic cognitive process that acts like an internal error correction system. It allows us to adapt our identities and values over time"
},
{
"start_time": 11215.76,
"end_time": 11236.26,
"content": " in response to new experiences, challenges, and social contexts. Now it almost sneaks up to something approximating and cutting insight with narrative plasticity in quotes. But then it goes back to the sort of the generic. I don't know. All of these models are incredible for different reasons. There's a lot of concerns as we discussed in this episode,"
},
{
"start_time": 11236.26,
"end_time": 11261.54,
"content": " but there's a lot of reasons to be excited as well. And I've probably spoken for too long. I am severely sleep deprived, borderline delirious, so hopefully some of this made sense. And now, dear friends, back to the episode. I think when you, you know, to Nathan's point, when you look at like the reasoning models,"
},
{
"start_time": 11261.93,
"end_time": 11281.93,
"content": " um, when you look at like the reasoning models, to me, even when I used R1 versus 01, there was like that sort of rough edges around the corner feeling, right? And Flash thinking, you know, earlier, I didn't use this version, but the one from December, and it definitely had that rough edges around the corner feeling, right? Where it's just not fleshed out in as many ways, right?"
},
{
"start_time": 11281.93,
"end_time": 11305.11,
"content": " Sure, they added math and coding capabilities via these verifiers in RL, but it feels like they lost something in certain areas and oh one is worse performing than chat in many areas as well to be clear Not by a lot not by a lot though as well, to be clear. Not by a lot. Not by a lot, though, right? And it's like some, like R1 definitely felt to me like it was worse than V3 in certain areas, like doing this RL expressed and learned a lot,"
},
{
"start_time": 11305.11,
"end_time": 11327.47,
"content": " but then it weakened in other areas. And so I think that's one of the big differences between these models and what O1 offers. And then OpenAI has O1 Pro. And what they did with O3, which is like also very unique, is that they stacked search on top of chain of thought, right? And so chain of thought is one thing where it's able, it's one chain, it backtracks, goes back"
},
{
"start_time": 11327.47,
"end_time": 11348.6,
"content": " and forth. But how they serve the served solved the arc a GI challenge was not just the chain of thought it was also sampling many times i i.e. running them in parallel and then selecting. Is running in parallel actually search? Because I don't know if we have the full information on how O1 Pro works. So I'm not, I don't have enough information to confidently say that it is searched. It is parallel samples."
},
{
"start_time": 11348.6,
"end_time": 11372.6,
"content": " Yeah. And then select something. And we don't know what the selection function is. The reason why we're debating is because since 01 was announced, there's been a lot of interest in techniques called Monte Carlis Research, which is where you will break down the chain of thought inter-intermediate steps. We haven't defined chain of thought. Chain of thought is from a paper from years ago where you introduce the idea to ask a language model that at the time was much less easy to use."
},
{
"start_time": 11372.6,
"end_time": 11396.76,
"content": " You would say, let's verify step by step, and it would induce the model to do this bulleted list of steps chain of thought is now almost a default in models where if you ask it a math question, you don't need to tell it to think step by step. And the idea with Monte Carlo tree search is that you would take an intermediate point in that train, do some sort of expansion, spend more compute, and then select the right one. That's like a very complex form of search that has been used in things like"
},
{
"start_time": 11396.76,
"end_time": 11417.26,
"content": " muZero and alpha zero potentially. I know muZero does this. Another form of search is just asking five different people and then taking the majority answer. Yes. Right. So there's a variety of like, you know, it could be complicated, it could be complicated it could be simple we don't know what it is just that they are they are not just issuing one chain of thought in sequence they're launching many in one chain of thought in sequence. They're launching many in parallel."
},
{
"start_time": 11417.26,
"end_time": 11444.37,
"content": " And in the Arc AGI, they launched a thousand in parallel for the one that like really shocked everyone that beat the benchmark was they they would launch a thousand in parallel and then they would get the right answer like 80% of the time or 70% of the time, 90 maybe even. Whereas if they just launched one, it was like 30%. There are many extensions to this. I would say the simplest one is that our language models to date have been designed to give the right answer the highest percentage of the time in one response."
},
{
"start_time": 11444.57,
"end_time": 11467.23,
"content": " And we are now opening the door to different ways of running inference on our models in which we need to reevaluate many parts of the training process, which normally opens the door to more progress, but we don't know if open AI changed a lot, or if just sampling more and multiple choices, what they're doing, or if it's something more complex, where they change the training and they know that the inference mode is going to be different."
},
{
"start_time": 11467.41,
"end_time": 11487.57,
"content": " So we're talking about 01 Pro $200 a month and they're losing money. So the thing that we're referring to, this fascinating exploration of the test time compute space, is that actually possible? Do we have enough compute for that? Does the financials make sense?"
},
{
"start_time": 11487.71,
"end_time": 11511.47,
"content": " So the fantastic thing is and and there it's in the thing that i just pulled up earlier but uh the cost for GPT3 has plummeted if you scroll up just a few images, I think. The important thing about like, hey, is cost limiting factor here, right? Like my view is that like we'll have like really awesome intelligence before we have like aGI before we have it permeate throughout the"
},
{
"start_time": 11511.47,
"end_time": 11532.09,
"content": " economy and this is sort of why that reason is right g GPT3 was trained in what, 2020, 2020, 2021, and the cost for running inference on it was $60, $70 per million tokens, right? Which is the cost per intelligence was ridiculous. Now, as we scaled forward two years, we've had a 1200x reduction in cost"
},
{
"start_time": 11532.09,
"end_time": 11557.44,
"content": " to achieve the same level of intelligence as GPT3. So here on the X axis, this time over just a couple of years, and on the Y axisaxis log scale dollars to run inference on a million tokens. Yeah, a million. And so you have just a down, like a linear decline on a log scale from GPT3 through 35 to Lama."
},
{
"start_time": 11557.44,
"end_time": 11577.72,
"content": " It's like $0.5 or something like that now, right? Which is versus $60,200 X. That's not the exact numbers, but it's 1200x. I remember that number is, is the humongous, is humongous cost per intelligence, right? Now, the freak out over DeepSeek is, oh my God, they made it so cheap. It's like, actually, if you look at this trend line, they're not below the trend line, first of all."
},
{
"start_time": 11577.8,
"end_time": 11598.06,
"content": " And at least for TP3, right? They are the first to hit it, right? Which is a big deal. But they're not below the trend line as far as GP3. Now we have GPD4. What's going to happen with these reasoning capabilities, right? It's a mix of architectural innovations. It's a mix of better data, and it's going to be better training techniques, and all of these better inference systems, better hardware, right, going from,"
},
{
"start_time": 11598.06,
"end_time": 11618.26,
"content": " you know, each generation of GPU to new generations or A6, everything is going to take this cost curve down and down and down and down. And then can I go in, can I just spawn a thousand different LLMs to create a task and then pick from one of them or you know whatever search search technique I want, a tree, Monte Carlo tree search, maybe it gets that complicated."
},
{
"start_time": 11619.06,
"end_time": 11639.56,
"content": " Maybe it doesn't because it's too complicated to actually scale. Like, who knows? A bitter lesson, right? The question is, I think, when, not if, because the rate of progress is so fast, right? Um, nine months ago, Dario was saying, hey, or, you know, Dario said nine months ago, the cost to train and inference was this, right? And now we're much better than this, right?"
},
{
"start_time": 11639.9,
"end_time": 11662.99,
"content": " And DeepSeek is much better than this. And that cost curve for GPD 4, which was also roughly $60 per million tokens when it launched has already fallen to, you know, $2 or so, right? And we're going to get it down to cents, probably, for GPD4 quality and the same and then that's that that's the based for the reasoning models like 01 that we have today and O1 Pro is spawning multiple, right?"
},
{
"start_time": 11662.99,
"end_time": 11683.93,
"content": " And 03 and so on and so forth. These search techniques too expensive today, but they will get cheaper. And that's what's going to unlock the intelligence, right? So it'll get cheaper and cheaper and cheaper. The big deep seek R1 release freaked everybody out because of the cheaper. One of the manifestations of that is Nvidia stock plummeted."
},
{
"start_time": 11683.93,
"end_time": 11704.65,
"content": " Can you explain what happened? I mean, and also just explain this moment and whether, you know, if Nvidia is going to keep winning. We're both Nvidia bowls here, I would say. And in some ways, the market response is reasonable. Most of the market, like, Nvidia's biggest customers in the U.S. are major tech companies,"
},
{
"start_time": 11704.83,
"end_time": 11729.43,
"content": " and they're spending a ton on AI. And if a simple interpretation of Deep seek is you can get really good models without spending as much on AI. So in that capacity, it's like, oh, maybe these big tech companies won't need to spend as much in AI and go down. The actual thing that happened is much more complex where there's social factors, where there's the rising in the app store, the social contagion that is happening. And then I think some of it's just like, I don't trade."
},
{
"start_time": 11729.53,
"end_time": 11750.35,
"content": " I don't know anything about financial markets. But it builds up over the weekend or the social pressure where it's like, if it was during the week and there was multiple days of trading when this was really becoming. But it comes on the weekend and then everybody wants to sell. And that is a social contagion. I think, I think, and like, there are a lot of false narratives, which is like, hey, these guys are spending billions on models, right? And they're not spending billions on models."
},
{
"start_time": 11750.47,
"end_time": 11772.55,
"content": " No one spent more than a billion dollars on a model that's released publicly, right? GPD4 was a couple hundred million, and then, you know, they've reduced the cost with 4-0, all-Turbo-4-0, right? But billion-dollar model runs are coming, right? And this concludes pre-training and post-training, right? And then the other number is like, hey, deep seek didn't include everything, right? They didn't include, you know, a lot of the cost goes to research and all this sort of stuff."
},
{
"start_time": 11772.81,
"end_time": 11794.47,
"content": " A lot of the cost goes to inference. A lot of cost goes to post training none of these things were factor research salaries right like all these things are like counted in the billions of dollars that open a i is spending but they weren't counted in the billions of dollars that Open AI is spending, but they weren't counted in the, you know, hey, six million, five million dollars that Deep Seek spent, right? So there's a bit of misunderstanding of what these numbers are. And then there's also an element of, Nvidia's just been a straight line up, right?"
},
{
"start_time": 11794.47,
"end_time": 11816.34,
"content": " And there's been so many different narratives that have been trying to push down in video. I don't say push down in video stock. Everyone is looking for a reason to sell or to be worried right um you know it was it's it was blackwell delays right there gp you know there's a lot of report every two weeks there's a new report about their GPUs being delayed there's um there's the whole thing about scaling laws ending, right?"
},
{
"start_time": 11816.34,
"end_time": 11837.82,
"content": " It's so, it's so ironic, right? It lasted a month. It was, it was just, it was just like literally just, hey, models aren't getting better, right? They're just not getting better. There's no reason to spend more. Pre-training, scaling is dead, and that it's like,-1-03 right r1 right and now it's like wait models are getting too they're progressing too fast. Slow down the progress. Stop spending on GPUs, right?"
},
{
"start_time": 11838.22,
"end_time": 11857.9,
"content": " But, you know, the funniest thing I think that comes out of this is Javon's paradox is true right a ws pricing for h100s has gone up over the last couple weeks, right? Since a little bit after Christmas, since V3 was launched, AWS H100 pricing has gone up. H200s are like almost out of stock everywhere because it, you know,"
},
{
"start_time": 11857.9,
"end_time": 11878.26,
"content": " H-200 has more memory and therefore R1, like, you know, wants that chip over H100, right? We were trying to get GPUs on a short notice this week for a demo, and it wasn't that easy. We were trying to get just like 16 or 32 H-100s for demo, and it will not very easy. So for people who don't know, Jamon's paradox is when the efficiency goes up somehow magically,"
},
{
"start_time": 11879.78,
"end_time": 11899.26,
"content": " counterintuitively, the total resource consumption goes up as well. Right. And semiconductors is, you know, we're at 50 years of Moore's law, every two years, half the cost. Double the transistors. Just like clockwork. And it's slowed down, obviously. But like, the semiconductor industry has gone up the whole time. It's been wavy, right? There's obviously cycles and stuff. And I don't expect AI to be any different, right?"
},
{
"start_time": 11899.26,
"end_time": 11923.17,
"content": " There's going to be ebbs and flows. But this is, in AI, it's just playing out at an insane timescale, right? It was 2x every two years. This is 1,200 X in like three years, right? So it's like the scale of improvement that is like hard to wrap your head around. Yeah, I was confused because I, to me, Envidious thought on that should have gone up, but maybe it went down because there's kind of suspicion"
},
{
"start_time": 11923.17,
"end_time": 11943.69,
"content": " of fall play on the side of China, something like this. But if you just look purely at the actual principles of play here. It's obvious, yeah, the Javon's paradox. More progress that AI makes, or that hire the derivative of AI progresses, especially, you should, because Nvidia is in the best place, the higher the derivative is, the sooner the market's going to be bigger and expanding,"
},
{
"start_time": 11943.91,
"end_time": 11964.95,
"content": " and Nvidia is the only one that does everything reliably right now. Because it's not like an Nvidia competitor arose. It's another company that's using Nvidia. Who historically has been a large Nvidia customer. Yeah. And has press releases about them cheering about being China's biggest Nvidia customer, right? Yeah, I mean..."
},
{
"start_time": 11964.95,
"end_time": 11985.17,
"content": " Obviously, they've quieted down, but like, I think that's like another element of is that they don't want to say how many GPs they have. Yeah. Because, hey, they, yes, they have H-800s. Yes, they have H-20s. They also have some H-100s, right? Which are smuggled in. Can you speak to that, to the smuggling? What's the scale of smuggling that's feasible for a nation state to do for companies?"
},
{
"start_time": 11985.17,
"end_time": 12011.71,
"content": " Is it possible to? I think there's a few angles of smuggling here, right? One is BightDance arguably is the largest smuggler of GPUs for China, right? China's not supposed to have GPUs. BiteDance has like over 500,000 GPUs. Why? Because they're all rented from companies around the world. They rent from Oracle, they rent from Google, they rent from all these mass and a bunch of smaller cloud companies too, right? All the neoclouds, right, of they rent so so many jupes they also buy a bunch right"
},
{
"start_time": 12011.71,
"end_time": 12037.9,
"content": " and and they do this for mostly like what meta does right serving? Serving TikTok, right? Serving next best separate discussion. To be clear, that's today the use, right? And it's a valid use, right? Hack the dopamine cert it, right? Now, that's theoretically now very much restricted with the AI diffusion rules which happened in the last week of the Biden admin and Trump admin looks like they're going to keep them, which limits like allies, even like Singapore,"
},
{
"start_time": 12037.9,
"end_time": 12061.76,
"content": " which Singapore is like 20% of Nvidia's 20, 20, 30% of Nvidia's 20, 30 percent of invidious revenue, but Singapore's had a memoratorium on not building data centers for like 15 years because they don't have enough power, so where are they going? I mean, I'm not claiming they're all going to China, right? But a portion are, you know, many are going to Malaysia, including Microsoft and Oracle have big data centers in Malaysia. Like, you know, they're going all over Southeast Asia, probably, India as well, right?"
},
{
"start_time": 12061.76,
"end_time": 12082.84,
"content": " Like there's stuff routing, but like the diffusion rules are very de facto. Like you can only buy this many GPUs from this country. And it's, and you can only rent a cluster this large to companies that are Chinese, right? Like, they're very explicit on trying to stop smuggling, right? And a big chunk of it was, hey, let's, let's, you know, random company by 16 servers, ship them to, uh, to, to, to, to China, right?"
},
{
"start_time": 12083.32,
"end_time": 12105.92,
"content": " There's actually, I saw a photo from someone in the semiconductor industry who leads like a team for like networking chips that competes with Nvidia and he sent a photo of a guy checking into a first class united flight from san francisco to shanghai or shenzhen with a super micro box that is this big, which can only contain GPUs."
},
{
"start_time": 12105.92,
"end_time": 12126.06,
"content": " Right? And he was booking first class because think about it. Three to five K for your first class ticket. Server costs, you know, 240,000 in the US, 250,000. You sell it for 300,000 in China. Wait, you just got a free first class ticket and a lot more money. So it's like, you know, and that's like small scale sluggling. Most of the large scale smuggling is like companies in Singapore and"
},
{
"start_time": 12126.06,
"end_time": 12146.12,
"content": " Malaysia, like routing them around or renting GPUs completely legally I want to jump in how much was this scale I think there's been some number like some people that are higher level economics understanding say that as you go from one billion of smuggling to 10 billion it's like you're hiding certain levels of economic activity. And that's the most reasonable thing to me is that there's going to be some level where it's"
},
{
"start_time": 12146.12,
"end_time": 12169.59,
"content": " so obvious that it's easier to find this economic activity. And yeah, so, so my, my, my belief is that last year roughly, so, so Nvidia made a million H20s, which are legally allowed to be shipped to China, which we talked about is better for reasoning, inference at least, not maybe not training, but reasoning inference, and inference generally. Then they also had, you know, a couple hundred thousand."
},
{
"start_time": 12169.59,
"end_time": 12191.25,
"content": " We think like 200 to 300,000 GPUs were routed to China from, you know, Singapore, Malaysia, US, wherever. Companies spawn up by 16 GPUs, 64 GPUs, whatever it is, route it. And Huawei's known for having spent up a massive network of companies to get the materials they need after they were banned in like 2018. So it's not like otherworldly. But I agree, right? Nathan's point is like, hey, you can't smuggle"
},
{
"start_time": 12191.25,
"end_time": 12213.03,
"content": " $10 billion of GPUs. And then the third sort of source, which is just now banned, which wasn't considered smuggling, but is China is renting, like, is, I believe from our research, right? Oracle's biggest GPU customer is ByteDance, right? And for Google, I think it's their second biggest customer. Right. And so like, and you go down the list of clouds and especially these smaller cloud companies"
},
{
"start_time": 12213.03,
"end_time": 12234.39,
"content": " that aren't like the hyperscalers, right? Think beyond core, even Lambda, even. There's a whole C, there's 60 different new cloud companies serving in video GPUs. I think B bytance is renting a lot of these right um all over right and so these companies are renting GPUs to Chinese companies, and that was completely legal up until the diffusion rules, which happened just a few weeks ago,"
},
{
"start_time": 12234.39,
"end_time": 12259.48,
"content": " and even now you can rent GPU clusters that are less than 2,000 GPUs, or you can buy GPUs and ship them wherever you want if they're, if're less than 1500 GPUs right so it's like there are still like some ways to smuggle but yeah it's not you know as the numbers grow right, uh, you know, a hundred something billion dollars revenue for in video last year, 200 something billion this year, right? And if next year are you could nearly double again or more than double, right,"
},
{
"start_time": 12259.5,
"end_time": 12285.84,
"content": " based on like what we see with data center footprints like being built out all across the U.S. and the rest of the world, it's going to be really hard for China to keep up with these rules, right? Yes, there will always be smuggling. And deep seek level models of GPD4 level models, 01 level models capable to train on what China can get, even the next tier above that. But if we speed run a couple more jumps, right, you know, right you know to billion dollar models 10 billion dollar models then it becomes"
},
{
"start_time": 12285.84,
"end_time": 12306.52,
"content": " you know hey there is a compute disadvantage for China for training models and serving them. And the serving part is really critical, right? DeepSeek cannot serve their model today, right? It's completely out of inventory. It's already started falling in the app store actually downloads because you download it. You try and sign up. They say, we're not taking registrations because they have no capacity. You open it up, you get less than five tokens per second"
},
{
"start_time": 12306.52,
"end_time": 12329.15,
"content": " if you even get your request approved, right? Because there's just no capacity because they just don't have enough GPUs to serve the model, even though it's incredibly efficient. It would be fascinating to watch the smuggling, because I mean, there's drug smuggling, right? That's a, that's a market. There's weapons smuggling, and GPUs will surpass that at some point. Chips are highest value per kilogram, probably by far."
},
{
"start_time": 12329.15,
"end_time": 12350.59,
"content": " I have another question for you, Don't do track model API access internationally? How easy is it for Chinese companies to use hosted model APIs from the US. Yeah, I mean, that's incredibly easy, right? Like, Open AI publicly stated deep secret uses their API, and as they say, they have evidence, right? And this is another element of the training regime"
},
{
"start_time": 12350.59,
"end_time": 12374.59,
"content": " is people at OpenAI have claimed that it's a distilled model, i.e. you're taking Open AIs model, you're generating a lot of output, and then you're training on the output in their model. And even if that's the case, what they did is still amazing, by the way, what DeepSseek did, efficiency-wise. Distillation is standard practice in industry, whether or not, if you're at a closed lab where you care about terms of service and IP closely, you distill from your own models. If you are a researcher and you're not building any products,"
},
{
"start_time": 12374.59,
"end_time": 12396.02,
"content": " you distill from the opening eye boxes. This is a good opportunity. Can you explain big picture distillation as a process? What is distillation? What's the process of distillation? We talked a lot about training language models. They are trained on text. In post-training, you're trying to train on very high quality text that you want the model to match the features of, or if you're using RL, you're letting the model find its own thing."
},
{
"start_time": 12396.02,
"end_time": 12417.26,
"content": " But for supervised fine tuning, for preference data, you need to have some completions with the model is trying to learn to imitate. And what you do there is instead of a human data, or instead of the model you're currently training, you take completions from a different normally more powerful model. I think there's rumors that these big models that people are waiting for,"
},
{
"start_time": 12417.26,
"end_time": 12448.16,
"content": " these GPT-5s of the world, the Claude 3 opuses of the world, are used internally to do this distillation process. There's also public examples, right? Like meta explicitly stated, not necessarily distilling, but they used 405B as a reward model for 70B in their Lama 3.2 and 3.3. This is all the same topic. So is this, uh, is this ethical? Is this legal? Like, why is that financial times article headline say open a i says that there's evidence that"
},
{
"start_time": 12448.16,
"end_time": 12471.47,
"content": " china's deep seek used its model to train competitor. This is a long, at least in the academic side and research side, it's a long history because you're trying to interpret opening eyes rule opening eyes terms of service say that you cannot build a competitor with outputs from their models. Terms of service are different than a license, which are essentially a contract between organizations. So if you have a terms of service on Open AIs account, if I violate it, Open"
},
{
"start_time": 12471.47,
"end_time": 12499.8,
"content": " AI can cancel my account. This is very different than like a license that says how you could use a downstream artifact. So a lot of it hinges on a word that is very unclear in the AI space, which is what is a competitor. And so... And then the ethical aspect of it is like, why is it unethical for me to train on your bottle when you can train on the internet's text. Yeah. Right? So there's a bit of a hypocrisy because sort of open AI and potentially most of the companies trained on the internet's text without permission."
},
{
"start_time": 12499.8,
"end_time": 12520.56,
"content": " There's also a clear loophole, which is that I generate data from Open AI, and then I upload it somewhere, and then somebody else trains on it, and the link has been broken. Like they're not under the same terms of service contract. This is why... There's a lot of hip-hospher. There's a lot of, like, to be discovered details that don't make a lot of sense. This is why a lot of models today,"
},
{
"start_time": 12520.56,
"end_time": 12541.56,
"content": " even if they train on zero open AI data, you ask the model who trained you, it'll say, I am chat GPT trained by opening I. Because there's so much copy paste of like opening eye outputs from that on the internet that you just weren't able to filter it out and in the and there was nothing in the RL where they implemented like, hey, or post-training or SFT, whatever, that says, hey, I'm actually a model"
},
{
"start_time": 12541.56,
"end_time": 12562.42,
"content": " by Allen Institute instead of... We have to do this if we serve a demo. We do research and we use open AI APIs because it's useful and we want to understand post-training and like our research models, they will say they're written by Open AI unless we put in the system prop that we talked about that, like, I am Tulu, I am a language model trained by the Allen Institute for AI. And if you ask more people around industry,"
},
{
"start_time": 12562.42,
"end_time": 12584.88,
"content": " especially with post-training, it's a very doable task to make the model say who it is or to suppress the opening I thing. So in some levels it might be that Deep Sea didn't care that it was saying that it was by opening I. Like if you're going to upload model weights, it doesn't really matter because anyone that's serving it in an application and cares a lot about serving is going to when serving it if they're using it for a specific"
},
{
"start_time": 12584.88,
"end_time": 12606.38,
"content": " task they're going to tailor it to that and it doesn't matter that it's saying it's chat gbti oh i guess the one of the ways to do that's like a system prompt or something like that like if you're serving it to say that you're... That's what we do. If we host the demo, you say, you are Tulu 3, a language model trained by the Allen Institute for AI. We also are benefited from open AI data because it's a great research tool."
},
{
"start_time": 12606.38,
"end_time": 12627.28,
"content": " I mean, do you think there's any truth and value to the the claim, open AIs claim, that there's evidence that China's deep seek used this model to train. I think everyone has benefited regardless because the data is on the internet. And therefore, it's in your pre-training now, right? There are like subredits where people share the best chat GPT outputs."
},
{
"start_time": 12627.44,
"end_time": 12648.35,
"content": " And those are, those are in your I think that they're trying to ship the narrative like they're trying to protect themselves and we saw this years ago when ByteDance was actually banned from some open AI APIs for training on outputs. There's other AI startups that most people, if you're in the AI culture, were like, they just told us they trained on open AI outputs and they never got banned."
},
{
"start_time": 12648.35,
"end_time": 12677.01,
"content": " Like that's how they bootstrapped their early models. So it's much easier to get off the ground using this than to set up human pipelines and build a strong model. So there's long history here and a lot of the communications are seem like narrative control. Actually, like over the last couple couple days, we've seen a lot of people distilled deep seeks model into Lama models because the deep seek models are kind of complicated to run inference on because they're a mixture of experts and they're, you know, 600 plus billion parameters and all this. And people distilled them into the llama models because the"
},
{
"start_time": 12677.01,
"end_time": 12699.01,
"content": " llama models are so easy to serve and everyone's built the pipelines and tooling for inference with the llama models right Because it's the open standard. So, you know, we've seen it, we've seen a sort of roundabout, right? Like, is it bad? Is it illegal? Maybe it's illegal, illegal whatever I don't know about that but like it could break contracts I don't think it's illegal like in any legal like no one's going to jail for this i i think like fundamentally i think it's ethical or i"
},
{
"start_time": 12699.01,
"end_time": 12719.71,
"content": " hope it's ethical because like the moment becomes, we ban that kind of thing, it's going to make everybody much worse off. And I also actually this is difficult, but I think you should be a lot to train on the internet. I know a lot of authors and creators are very sensitive about it. That's a difficult question."
},
{
"start_time": 12719.71,
"end_time": 12743.28,
"content": " But the moment you're not allowed to train on the internet. I agree. I have a schizo take on how you can solve this because it already works. I have a reasonable take out. All right. So, you know, Japan has a law, which you're allowed to train on any training data and copyrights don't apply if you want to train a model. A, B, Japan has nine gigawatts of curtailed nuclear power."
},
{
"start_time": 12743.28,
"end_time": 12764.32,
"content": " See, Japan is allowed under the AI diffusion rule to import as many GPUs as they'd like. So all we have to do, we have a market here to make. We build massive data setters, we rent them to the labs, and then we train models in a legally permissible way and there's no if ands or buts and now the models have no no potential copyright lawsuit from New York Times or anything like that."
},
{
"start_time": 12764.32,
"end_time": 12786.46,
"content": " No, no, it's just like completely legal. No, so, so, so. Genius. The early copyright lawsuits have fallen in the favor of AI training. I would say that the long tail of use is going to go in the side of AI, which is if you do, if you scrape trillions of data, you're not looking at the trillions of tokens of data, you're not looking at the trillions of tokens of data, you're not looking and saying this one New York Times article is so important to me."
},
{
"start_time": 12786.68,
"end_time": 12808.12,
"content": " But if you're doing an audio generation for music or image generation and you say make it in the style of ex person, that's a reasonable case where you could figure out what is their profit margin on inference. I don't know if it's going to be the 50-50 of YouTube creator program or something, but I would opt into that program as a writer. Like, please, like that, it's just,"
},
{
"start_time": 12808.12,
"end_time": 12834.39,
"content": " it's gonna be a rough journey, but there will be some solutions like that that makes sense, but there's a long tail where it's just on the internet. I think one of the other aspects of that Financial Times article implied, and so that leads to a more general question. Do you think there's, how difficult is spying, espionage, and stealing of actual secret code and data from inside companies. How much of that"
},
{
"start_time": 12834.39,
"end_time": 12855.75,
"content": " is being attempted? Food and data is hard, but ideas is easy. Silicon Valley operates on the way that top employees get bought out by other companies for a pay raise and a large reason why these companies do this is to bring ideas with them. And there are, there's no, I mean, in California, there's rules that like certain like non-competes or whatever are illegal in California."
},
{
"start_time": 12856.01,
"end_time": 12879.09,
"content": " And whether or not there's NDAs and things, that is how a lot of it happens. Recently, there was somebody from Gemini who helped make this one million context length and everyone is saying the next llama who I, he went to the meta team, is going to have one million contacts length. And that's kind of how the world works. You know, as far as like industrial espionage and things that has been greatly successful in the past"
},
{
"start_time": 12879.09,
"end_time": 12901.92,
"content": " right um you know the americans there did the Brits, the Chinese have done it to the Americans, right? And you know, so on and so forth. It's just, it is a fact of life. And so like like, to argue industrial espionage can be stopped is probably unlikely. You can make it difficult. But even then, like, there's all these stories about, like, hey, F-35 and F-22 have already been like sort of like given to China in terms of design plans and stuff."
},
{
"start_time": 12901.92,
"end_time": 12922.42,
"content": " Code and stuff, like between, you know, I say companies, not nation states, is probably very difficult, but ideas are discussed. is probably very difficult. But ideas are discussed a lot, right? Whether it be a house party in San Francisco or a company changing employees or you know or the always the like mythical honey pot that always gets talked about right like someone gets honey potted, right?"
},
{
"start_time": 12922.82,
"end_time": 12942.56,
"content": " Because everyone working on AI is a single dude who's in their 20s and 30s. Not everyone, but like an insane amount of insane percentages so there's always like all these like you know and and obviously so honey Pilot is like a spy, a female spy approaches you and like. Yeah. Yeah. Or male, right? You know, it's San Francisco, right?"
},
{
"start_time": 12942.56,
"end_time": 12967.14,
"content": " But as a single dude, I will say in his late 20s, right, is like, we are very easily corrupted, right? Like, you know, like not corrupted myself, but you know, like we are, we are. Everybody else, not me. Yeah, exactly. I'm too oblivious and i am not single so i'm saved from one espionage access yeah you have to make sure to close all security vulnerabilities so you uh dylan collect a lot of"
},
{
"start_time": 12967.14,
"end_time": 12989.19,
"content": " information about each of the mega clusters for each of the major AI companies. Can you talk about the buildouts for each one that stand out? Yeah, so I think the thing that's really important about these megacluster buildouts is they're completely unprecedented in scale. Right. US, you know, sort of like data center power consumption"
},
{
"start_time": 12989.19,
"end_time": 13011.93,
"content": " has been slowly on the rise and it's gone up to two, three percent, even through the cloud competing revolution, right? Data center consumption as a percentage of total U.S. And that's been over decades, right, of data centers, et cetera. It's been climbing, climbing slowly. But now, two to three percent now by the end of this decade it's like even even under like you know when i say like 10 percent a lot of people that are traditionally, by like 2028, 2030,"
},
{
"start_time": 13011.93,
"end_time": 13035.41,
"content": " people are traditionally non-traditional data center, people like, that's nuts. But then like, people who are in like AI, who have like really looked at this at like the Anthropics and open AIs are like, that's not enough. And I'm like, okay. But like, you know, this is, this is both through globally distributed and or distributed throughout the US as well as like centralized clusters, right? The distributed throughout the US is exciting and it's the bulk"
},
{
"start_time": 13035.41,
"end_time": 13060.39,
"content": " of it, right? Like, hey, you know, uh, opening eye or, uh, you know, say meta is adding a gigawatt, right? But most of it is distributed through the U.S. for inference and all these other things, right? So maybe we should lay out what a cluster is. So, you know, does this include AWS? Maybe it's good to talk about the different kinds of clusters and what you mean by mega clusters and what's the GPU and what's a computer and what."
},
{
"start_time": 13060.51,
"end_time": 13082.68,
"content": " Yeah, yeah. Not that far back but yeah so like what do we mean by the clusters oh man I thought I was about to do the Apple ad right what's a computer so Apple ad, right? What's a computer? So, so traditionally data centers and data center tasks have been a distributed systems problem that is capable of being spread very far and widely, right? I send a request to Google."
},
{
"start_time": 13082.68,
"end_time": 13109.1,
"content": " It gets around to a data center somewhat close to me. It does whatever search ranking recommendations, sends a result back, right? The nature of the task is changing rapidly in that the task, there's two tasks that people are really focused on now, right? It's not database access. It's not served me the right page, serve me the right ad. It's now a inference, an inference is dramatically different from traditional distributed systems, but it looks a lot more simple, similar, and then there's training, right?"
},
{
"start_time": 13109.62,
"end_time": 13129.58,
"content": " The inference side is still like, hey, I'm going to put, you know, thousands of GPUs and, you know, blocks all around these data centers. I'm going to run models on them, you know, user submits a request, gets kicked off, or, hey, my service, service you know they submit a request to my service right they're on word and they're like oh yeah help me copilot and it starts kicks it off from on my windows copilot whatever apple, Apple Intelligence,"
},
{
"start_time": 13129.58,
"end_time": 13151.58,
"content": " whatever it is, it gets kicked off to a data center, right? Now that data center does some work and sends it back. That's inference. That is going to be the bulk of compute, but then, you know, and that's like, you know, there's thousands of data centers that we're tracking with, like, satellites and like all these other things. And, and those are the bulk of what's being built, but the scale of, and so that's like what's really reshaping and that's what's getting millions of GPUs"
},
{
"start_time": 13151.58,
"end_time": 13171.78,
"content": " But the scale of the largest cluster is also really important, right? When we look back at history, right, like, you know, or through the age of AI, right? Like, it was a really big deal when they did AlexNet on, I think, two GPs or four GPs? I don't remember. It was a really big deal. I think two GPUs or four GPS? I don't remember. It's a really big deal. It's a big deal because you use GPUs."
},
{
"start_time": 13171.78,
"end_time": 13198.49,
"content": " It's a big deal they use GPUs. And they used multiple, right? But then over time, its scale has just been compounding, right? And so when you skip forward to GPD3, then GPD4, GPD4, 20,000 A100 GPUs. Unprecedented run, right, in terms of the size and the cost, right? A couple hundred million dollars on a yolo right a yolo run for gpd4 and it and it yielded you know this magical improvement that was like perfectly in line with what was experimented and just like a log scale"
},
{
"start_time": 13198.49,
"end_time": 13222.62,
"content": " right oh yeah they have that plot from the paper the technical part the scaling laws were perfect right but that's not a crazy number right 20 000 a 100s uh roughly each GPU is consuming 400 watts. And then when you add in the whole server, right, everything, it's like 15 to 20 megawatts of power, right? You know, maybe you could look up what the power of consumption of a human person is because the numbers are going to get silly."
},
{
"start_time": 13222.92,
"end_time": 13243.82,
"content": " But like that 15 to 20 megawatts was standard data center size. It was just unprecedented. That was all GPUs running one task. How many was also the toaster? A toaster is like, also a good example. Similar power consumption to an A100, right? H100 comes around. They increase the power from like 400 to 700 watts and that's just per GPU and then there's all the associated stuff around it. So once you count all that, it's roughly"
},
{
"start_time": 13243.82,
"end_time": 13264.84,
"content": " like 1,400 to 1400 watts for everything, networking CPUs, memory, blah, blah, blah. So we should also say, so what's required, you said power, so a lot of power is required a lot of heat is generated so cooling is required and because there's a lot of GPUs that have to be, or CPUs or whatever, they have to be connected,"
},
{
"start_time": 13264.84,
"end_time": 13287.12,
"content": " so there's a lot of networking. Yeah, so I think, yeah, sorry for skipping past that. And then the data center itself is like complicated, right? But these are still standardized data centers for GPD 4 scale. Right. Now we step forward to sort of what is the scale of clusters that people built last year, right? And it ranges widely, right? It ranges from like, hey, these are standard data centers,"
},
{
"start_time": 13287.12,
"end_time": 13308.44,
"content": " and we're just using multiple of them and connecting them together really with a ton of fiber between them, a lot of networking, et cetera. That's what Open AI and Microsoft did in Arizona, right? And so they have a, you know, 100,000 GPUs, right? Meta, similar thing. They took their standard existing data center design, and it looks like an H, they connected multiple them together um and you know they got to they first did 16 000 gpues uh 24 000 gpues total only 60 thousand GPUs 24,000"
},
{
"start_time": 13308.44,
"end_time": 13331.46,
"content": " GPUs total. Only 16 of them, thousand of them were running on the training run because GPs are very unreliable so they need to have spares to like swap in and out, all the way to like now 100,000 GPUs that they're training on Lama 4 on currently, right? Like 128,000 or so, right? This is, you know, think about $1,000 or so, right? This is, you know, think about 100,000 GPUs with roughly 1,400 watts apiece. That's, that's, that's 140 megawatts, 150 megawatts, right?"
},
{
"start_time": 13331.46,
"end_time": 13351.56,
"content": " For a hundred and twenty eight out, right? So you're talking about you've jumped from 15 to 20 megawatts to 10x, you know, almost 10x that number, 9x that number to 150 megawatts in in two years right from 2022 to 2024 right and some people like Elon that he admittedly right and he says himself got into the game a little bit late for pre-training large language models, right?"
},
{
"start_time": 13351.9,
"end_time": 13375.65,
"content": " X-A-I was started later, right? But then he bet heaven and hell to get his data center up and get the largest cluster in the world, right? Which is 200,000 GPUs. And he did that. He bought a factory in Memphis. He's upgrading the substation at the same time. He's got a bunch of mobile power generation, a bunch of single cycle combine. He tapped the natural gas line that's right next to the factory and he's just pulling a ton of gas, burning gas."
},
{
"start_time": 13375.75,
"end_time": 13398.49,
"content": " He's generating all this power. He's in a factory in an old appliance factory that's shut down and moved to China long ago, right? Like, you know, and he's got 200,000 GPUs in it. And now what's the next scale, right? Like all the hyperscalers have done this. Now, the next scale is something that's even bigger, right? And so, you know, Elon, just to stick on the topic, he's building his own natural gas plant like a proper one right next door."
},
{
"start_time": 13398.49,
"end_time": 13421.98,
"content": " He's deploying tons of Tesla megapack batteries to make the power more smooth and all sorts of other things. He's got like industrial chillers to cool the water down because he's water cooling the chips. So all these crazy things to get the clusters bigger and bigger. But when you look at like, say, what opening I did with Stargate, that's that in Arizona and in in Abilene, Texas, right?"
},
{
"start_time": 13422.14,
"end_time": 13452.98,
"content": " What they've announced at least, right? It's not built, right? Elon says they don't have the money. You know, there's some debates about this. But at full scale, at least the first section is like definitely money's accounted for, but there's multiple sections. But full scale, that data center is going to be 2.2 gigawatts, right? 2,200 megawatts of power in and roughly like 1.8 gigawatts or 1,800 megawatts of power delivered to chips. Now, this is an absurd scale 2.2 gigawatts is like more than most cities right you know to be clear"
},
{
"start_time": 13452.98,
"end_time": 13475.75,
"content": " and delivered to a single cluster that's connected to do training, right? To train these models, to do both the pre-training, the post-training, all of this stuff, right? This is insane. It is. What is a nuclear power plant again? Everyone is doing this, right? Everyone is doing this, right? Meta, in this, right? Meta in Louisiana, right? They're building two natural gas plants, massive ones, and then they're building this massive data center."
},
{
"start_time": 13476.75,
"end_time": 13496.69,
"content": " Amazon has like plans for this scale. Google has plans for this scale. XAI has plans for this scale. XAI has plans for the scale, right? Like all of these, the guys that are racing, the companies that are racing are racing hard and they're doing multi-gigawatt data centers, right? To build this out because they think that, yeah, if I now have,"
},
{
"start_time": 13496.69,
"end_time": 13517.63,
"content": " obviously pre-training scaling is going to continue, but to some extent, but then also all this post-training stuff where you have an RL sandbox for computer use or whatever, right? Like, you know, this is where they're going to, and all these variable domains where they just keep learning and learning and learning self-play, whatever, whatever it is makes the AI so much more capable because the line does go up, right? As you throw more compute, you get more performance. The shirt is about"
},
{
"start_time": 13517.63,
"end_time": 13539.49,
"content": " scaling laws. You know, to some extent it is diminishing returns, right? You 10x the compute. You don't get 10x better model, right? You get a diminishing returns, but also you get efficiency improvements, so you bend the curve, right um and these scale of data centers are doing you know reeking you know a lot of like havoc on the network right Right. And you know, Nate, Nathan was mentioning there's, Amazon has tried to buy this nuclear power plant,"
},
{
"start_time": 13539.75,
"end_time": 13564.32,
"content": " Tallin. And if you look at the Talon stock, it's just like skyrocketing. And, you know, like they're building a massive multi-gigawatt data center there. And you know, you just go down the list. There's so many ramifications. Interesting thing is like certain regions of the US, transmitting power cost more than actually generating it, right? Because the grid is so slow to build and the demand for power and the ability to build power and like re-ramping on a natural gas plant or even a coal plant is like"
},
{
"start_time": 13564.32,
"end_time": 13584.52,
"content": " easy enough to do but like transmitting the power is really hard so in some parts of the u.s like in virginia it cost more to transmit power than it cost to generate it which is like you know there's there's all sorts of like second order effects that are insane here. Can the power grid support this kind of growth? You know, Trump's executive orders, there's a Biden executive order before the end of the year, but then Trump had some more executive orders,"
},
{
"start_time": 13584.52,
"end_time": 13605.44,
"content": " which hopefully reduce the regulations to where, yes, things can be built. But yeah, this is a big, big challenge, right? Is building enough power fast enough? Are you going to basically have a nuclear power plant next to a data center for each one of these? So the fun thing here is this is too slow to build the power plant to build a power plant or to re"
},
{
"start_time": 13605.44,
"end_time": 13627.91,
"content": " configure an existing power plant is too slow. And so therefore, you must use data center power consumption is flat, right? You know, I mean, like it's by- Which is why nuclear is also good for it. Like long-term nuclear is a very natural fit, but you can't do solar or anything in the short term like that. Because data center powers like this, right? Like you're telling me, you know, I'm going to buy tens of billions of dollars of"
},
{
"start_time": 13627.91,
"end_time": 13651.08,
"content": " GPUs and idle them because the power's not being generated. Like power is cheap, right? Like if you look at the cost of a cluster, less than 20% of it is power, right? Most of it is the capital cost and depreciation of the GPUs, right? And so it's like, well, screw it. I'll just like, you know, I'll just build natural gas plants. This is what Meta's doing in Louisiana. This is what Open AI is doing in Texas and like all these different places. they may not be doing it directly, but they are partnered"
},
{
"start_time": 13651.08,
"end_time": 13674.58,
"content": " with someone. And so there is a couple hopes, right? Like one is, you know, and Elon, what he's doing in Memphis is like, you know, to the extreme, they're not just using dual combine cycle gas, which is like super efficient. He's also just using single cycle and like mobile generators and stuff, which is less efficient. But he's, you know, there's also like the flip side, which is like solar power generation is like this and wind is another like like this different correlate you know different"
},
{
"start_time": 13674.58,
"end_time": 13697.19,
"content": " So if you stack both of those, plus you get a big chunk of batteries, plus you have a little bit of gas, it is possible to run it more green. It's just the time scales for that is slow, right? So people are trying. But, you know, meta basically said whatever, don't care about my sustainability pledge. Or they'll buy like a per power, it's called a PPA power purchasing agreement where there'll be a massive wind farm or solar farm"
},
{
"start_time": 13697.19,
"end_time": 13717.33,
"content": " like wherever. And then they'll just pretend like those electrons are being consumed by the data center. But in reality, they're paying for the power here and selling it to the grid and they're buying power here. And then another thing is like Microsoft quit on some of their sustainability pledges, right? Elon, what he did with Memphis is objectively somewhat dirty, but he's also doing it in an area where there's like a bigger natural gas"
},
{
"start_time": 13717.33,
"end_time": 13737.65,
"content": " plant right next door and like a sewer next, or not a sewer, but like a wastewater treatment and a garbage dump nearby, right? And he's obviously made the world a lot more clean than that one data center is going to do. Right. So I think like it's fine to some extent and maybe AGI solves, you know, global warming and stuff, right? Whatever it is. You know, this is, this is sort of the attitude that people at the labs have, right?"
},
{
"start_time": 13737.65,
"end_time": 13758.25,
"content": " Which is like, yeah, it's great. We'll just use gas, right? Because the race is that important. And if we lose, you lose, that's way worse, right? I should say that I got a chance to visit the Memphis Data Center. And it's, uh, the Memphis Data Center. Oh, wow. And it's kind of incredible. I mean, I visited with Elon, just the teams and the rate of innovation there's insane."
},
{
"start_time": 13758.25,
"end_time": 13778.31,
"content": " Because my sense is that nobody's ever done anything of this scale and nobody has certainly ever done anything of this scale at the rate that XAI is is doing so they're like figuring out i mean it's all sitting in on all these meetings where they're brainstorming. It's like, it's insane. It's exciting because they're like,"
},
{
"start_time": 13778.57,
"end_time": 13799.69,
"content": " they're trying to figure out what the bottlenecks are, how to remove the bottlenecks, how to make sure that, you know, there's just so many really cool things about putting together data center because you know everything has to work it's uh the people that do like the cis admin you know the machine learning all that the people that do like the sysadmin, you know, the machine learning, all that is the exciting thing, so on, but really the people that run everything"
},
{
"start_time": 13799.69,
"end_time": 13823.75,
"content": " are the folks that know like the low level software and hardware that runs everything, the networking, all of that. And so you have to make sure you have procedures that test everything. I think they're using Ethernet. I don't know how they're doing the networking, but... They're using NVIDIUS SpectrumX Ethernet. There's actually like, I think, yeah, the unsung heroes are the cooling and electrical systems which are just like glossed over."
},
{
"start_time": 13823.75,
"end_time": 13844.03,
"content": " Yeah. Um, but I think like one story that maybe is like exemplifies how insane this stuff is is when you're training right you're always doing your you're you're you're running through the model a bunch, right, in the most simplistic terms, running through the model a bunch, and then you're, uh, you're going to exchange everything and synchronize the weights, right? So you'll do a step."
},
{
"start_time": 13844.09,
"end_time": 13866.91,
"content": " This is like a step in model training, right? At every step, your loss goes down, hopefully, and it doesn't always. But in the simplest terms, you'll be computing a lot and then you'll exchange, right? The interesting thing is GPU powers most of it. Networking power is some, but it's a lot less. So while you're computing, your power for your GPUs is here. But then when you're exchanging weights, if you're not able to overlap communications and compute perfectly, there may be a time period where your GPUs are just idle,"
},
{
"start_time": 13866.91,
"end_time": 13890.08,
"content": " and you're exchanging weights and you're like, hey, the model's updating. So you're exchanging the gradients, you do the model update, and then you start training again. So the power goes, right? And it's super spiky. And so funnily enough, right, like this, when you talk about the scale of data center power, right can blow stuff up so easy um and so meta actually has accidentally open so upstreamed something to code in Pi Torch,"
},
{
"start_time": 13890.08,
"end_time": 13911.4,
"content": " where they added an operator. And I kid you not, whoever made this, like I want to hug the guy because it says, says pie torch, it's like pie torch dot power plant, no blow up equals zero or equal one. And what it does, what it does is amazing, right? Either, you know, when you're, when you're exchanging the weights, the GP will just compute fake numbers. So the power doesn't spike too much. And so then the power plants don't blow up"
},
{
"start_time": 13911.4,
"end_time": 13933.58,
"content": " because the transient spikes like screw stuff up. Well, that makes sense. I mean, you have to do that kind of thing. You have to make sure they're not idle. Yeah. And Elon's solution was like, let me throw a bunch of Tesla megapacks and a few other things, right? Like, everyone has different solutions, but like, Medas at least was publicly and openly known, which is just like set this operator. And what this operator does is it just makes the GPUs compute nothing so that the power"
},
{
"start_time": 13933.58,
"end_time": 13955.31,
"content": " doesn't spike. But that just tells you how much power you're working with. I mean, it's insane. It's insane. People should just go to Google, like, scale, like what does X watts do and go through all the scales from one watt to a kilowatt to a megawatt and you look and stare at that and you're how high on the list a gigawatt is and it's mind-blowing can you say something about the cooling?"
},
{
"start_time": 13955.57,
"end_time": 13977.97,
"content": " So I know Elon's using liquid cooling, I believe, in all cases. That's a new thing, right? Most of them don't use liquid cooling. Is there something interesting to say about the cooling? Yeah, yeah. So air cooling has been the de facto standard, throw a bunch of metal heat pipes, et cetera, and fans, right? And like, that's cool. That's been enough to cool it um people have been dabbling and water cooling"
},
{
"start_time": 13977.97,
"end_time": 14007.91,
"content": " google's tp u's are water cooled right um so they've been doing that for a few years. But with GPUs, no one's ever done, and no one's ever done the scale of water cooling that Elon just did, right? Now, next generation Nvidia is for the highest NGPU, it is mandatory water cooling. You have to water cool it. But Elon did it on this current generation and that required a lot of stuff, right? If you look at some of the satellite photos and stuff of the Memphis facility, there's all these external water chillers that are sitting basically."
},
{
"start_time": 14007.91,
"end_time": 14028.82,
"content": " It looks like a semi truck pod thing uh what's it called the container uh but really those are water chillers and he has like 90 of those water chillers just sitting outside. 90 different containers, right? With the water, you know, like chill the water, bring it back to the data center, and then you distribute it to all the chips, pull all the heat out, and then send it back, right? And this is both a way to cool the chips, but also is an efficiency thing."
},
{
"start_time": 14029.54,
"end_time": 14049.6,
"content": " All right. And going back to that like sort of three vector thing, right? There is, there is, you know, memory bandwidth, flops and interconnect, the closer the chips are together, the easier it is to do high-speed interconnects, right? And so this is also like a reason why you're gonna go water cooling is because you can just put the chips right next to each other and therefore get higher"
},
{
"start_time": 14049.6,
"end_time": 14071.34,
"content": " speed connectivity. I got to ask you, so in one of your recent posts, there's a section called Cluster Measuring Contest. There's another word there, but I won't say it, you know? What, who's who's got the biggest now and who's gonna have the big today"
},
{
"start_time": 14071.34,
"end_time": 14092.82,
"content": " individual largest is Elon right um Elon's cluster. Elon's cluster in Memphis, 200,000 GPUs, right? Meta has like 12 hundred twenty eight thousand, opening has a hundred thousand now. Now to be clear, other companies have more GPUs than Elon. They just don't have them in one place, right? And for training, you want them tightly connected. There's some techniques that people are researching"
},
{
"start_time": 14092.82,
"end_time": 14115.75,
"content": " and working on that let you train across multiple regions, but for the most part, you want them all in like one area, right? So you can connect them highly with high speed networking. And so Elon today has 200,000 H100s, 100,000 H-200s, 100,000 H-200s, right? Meta, Open AI, and Amazon all have on the scale of 100,000,"
},
{
"start_time": 14115.75,
"end_time": 14137.63,
"content": " a little bit less. But next this year, this year, people are building much more, right? Anthropic and Amazon are building a cluster of 400,000 Traneum 2, which is Amazon-specific chip, trying to get away from InVidio, right? Um, you know, uh, in video, right? You know, meta and open AI have skills for hundreds of thousands, but by next year, you'll have like 500,000 to 700,000 GPU clusters."
},
{
"start_time": 14137.63,
"end_time": 14159,
"content": " And note those GPUs are much higher power consumption than existing ones, right? Hopper 700 watts, Blackwell goes to 1,200 watts, right? So the power per chip is growing and the number of chips is growing, right? Nuts. You think, you think Elon's like he'll get to a million. You think that's actually feasible? I mean, I don't doubt Elon, right?"
},
{
"start_time": 14159,
"end_time": 14190.74,
"content": " The filings that he has for like, you know, the power plan and the Tesla battery packs it's clear he has some crazy plans for Memphis like permits and stuff is open record, right? But it's not quite clear that, you know, what and what the timescales are. I just never doubt Elon, right? You know, he's going to surprise us. So what's the idea with these clusters? If you have a million GPUs, what percentage in, let's say, two, three years is used for training and what percent, pre-training and what percent is used for like for the actual"
},
{
"start_time": 14190.74,
"end_time": 14212.66,
"content": " computation so these mega clusters make no sense for inference right uh you could route inference there and just not train. But most of the inference capacity is being, you know, hey, I've got a 30 megawatt data center here. I've got 50 megawatts here. I've got a 30 megawatt data center here. I've got 50 megawatts here. I've got 100 here, whatever. I'll just throw inference in all of those because the mega clusters, right, multi-gigawatt data centers, I want to train there because that's where all of my GPs are co-located,"
},
{
"start_time": 14212.66,
"end_time": 14236.66,
"content": " where I can put them at a super high networking speed connected together, right? Because that's what you need for training. Now with pre-training, this is the old scale, right? You could increase parameters, you did increase data, model gets better. Uh, that doesn't, that doesn't apply anymore because there's not much more data in the pre-training side, right? Yes, there's video and audio and image that has not been fully taken advantage of, so there's a lot more scaling. But a lot of people like like have"
},
{
"start_time": 14236.66,
"end_time": 14258.74,
"content": " transcript taken transcripts of YouTube videos and that gets you a lot of the data. It doesn't get you all of the learning value out of the video and image data. But there's still scaling to be done on pre-training. But this post-training world is where all the flops are going to be spent, right? The model's going to play with itself. It's going to self-play. It's going to do verifiable task. It's going to do computer use in sandboxes. It might even do like simulated robotics things, right?"
},
{
"start_time": 14258.74,
"end_time": 14279.54,
"content": " Like all of these things are going to be environments where compute is spent in quote unquote post training. But I think, I think it's going to be good. We're going to, we're going to drop the post from post training. It's going to be pre-training and it's going to be training, I think. At some point. At some point. Because for the bulk of like the last few years, um, pre training has dwarfed post training."
},
{
"start_time": 14279.54,
"end_time": 14303.71,
"content": " Mm hmm. But with these verifiable methods, especially ones that scale really, you know, potentially infinitely, like computer use in robotics, not just math and coding, right, where you can verify what's happening. Those infinitely verifiable tasks, it seems you can spend as much compute as you want on them. Especially at the context length increase, because at the end of pre-training is when you increase the context length for these models and we've talked earlier in the conversation about how the context length, when you have a long"
},
{
"start_time": 14303.71,
"end_time": 14324.71,
"content": " input, is much easier to manage than output. And a lot of these post-training and reasoning techniques rely on a ton of sampling and it's becoming increasingly long context so it's just like your effectively your compute efficiency goes down. I don't the, I think Flops is the standard for how you measure it, but with RL and you have to do all these things where you move your"
},
{
"start_time": 14324.71,
"end_time": 14348.58,
"content": " weights around in a different way than at pre-training and just generation, it's going to become less efficient and Flops is going to be less of a useful term. And then as the infrastructure gets better, it's probably going to go back to Flops. So all of the things we've been talking about is most likely going to be Nvidia, right is there any competitors google google i kind of ignored them uh i was what's the story with tp u what's the story with tp u like what's the story with TPU?"
},
{
"start_time": 14348.58,
"end_time": 14368.92,
"content": " What's the story with TPU? Like, what's the... TPU is awesome, right? It's great. Google is, they're a bit more tepid on building data centers for some reason. They're building big data centers. Don't begin me wrong. And they have, they actually have the biggest cluster. I was talking about Nvidia clusters. They actually have the biggest cluster. I was talking about in video clusters. They actually have the biggest cluster, period. But the way they do it is very interesting, right?"
},
{
"start_time": 14368.92,
"end_time": 14389.58,
"content": " They have two sort of like data center super regions, right? In that the data center isn't physically, like all of the GPUs aren't physically on one site, but they're like 30 miles from each other, not GPS, TPs, right? They have like in, in Iowa, Nebraska, they have four data centers that are just like right next to each other. Why doesn't Google flex its cluster size? Go to multi-data center training."
},
{
"start_time": 14389.58,
"end_time": 14410.58,
"content": " It's good images in there, so I'll show you what I mean. It's just semi-analysis multi-data center. So this is like, you know, so this is an image of like what a standard Google data center looks like. By the way, their data centers look very different than anyone else's data centers. What are we looking at here? So these are, yeah, so if you see this image, right? In the center, there are these big rectangular boxes, right? Those are where the actual chips are kept."
},
{
"start_time": 14410.58,
"end_time": 14431.46,
"content": " And then if you scroll down a little bit further, you can see there's like these water pipes. There's these chiller cooling towers in the top and a bunch of like diesel generators. The diesel generators are backup power the data center itself is like look physically smaller than the water chillers right so the chips are actually easier to like keep together but then like cooling all the water for the water"
},
{
"start_time": 14431.46,
"end_time": 14454.26,
"content": " cooling is very difficult right so google has like a very advanced infrastructure that no one else has for the TPU. And what they do is they've stamped at these data center, they've stamped a bunch of these data centers out in a few regions right so if you go a little bit further um down uh this is this is a microsoft This is in Arizona. This is where GPT5 quote unquote will be trained. Um, you know, uh, if it doesn't exist already."
},
{
"start_time": 14454.26,
"end_time": 14474.58,
"content": " Yeah, if it doesn't exist already. Yeah, if it doesn't exist already. But each of these data centers, I've shown a couple images of them, they're like really closely co-located in the same region, Nebraska, Iowa. And then they also have a similar one in Ohio complex, right? And so these data centers are really close to each other. And what they've done is they've connected them super high bandwidth with fiber. And so these are just a bunch of data centers."
},
{
"start_time": 14474.58,
"end_time": 14495.34,
"content": " And the point here is that And so these are just a bunch of data centers. And the point here is that Google has a very advanced infrastructure, very tightly connected in a small region. So Elon will always have the biggest cluster fully connected, right, because it's all in one building. Right? And he's completely right on that, right? Google has the biggest cluster, but you have to spread over three sites, and by a significant margin, but you have to go across multiple sites."
},
{
"start_time": 14495.34,
"end_time": 14522.21,
"content": " Why doesn't Google compete with Nvidia? Why don't they sell TPUs? I think there's a couple problems with it. It's like one, TPU has been a form of allowing search to be really freaking cheap and build models for that, right? And so like a big chunk of the search GPU purchases or TPU purchases or big chunk of Google's purchases and usage, all of it is for internal workloads,"
},
{
"start_time": 14522.21,
"end_time": 14545.91,
"content": " whether it be search, now Gemini, YouTube, all these different applications that they have, you know, ads. These are where all their TPs are being spent, and that's what they're hyper-focused on, right? And so there's certain aspects of the architecture that are optimized for their use case that are not optimized elsewhere. One simple one is like they've open source the Gemma model and they called it Gemma 7B, right?"
},
{
"start_time": 14545.91,
"end_time": 14568.54,
"content": " But then it's actually 8 billion parameters because the vocabulary is so large. And the reason they made the vocabulary so large is because TPU's like matrix multiply unit is massive because that's what they've like sort of optimized for. And so they decided, oh, well, just make the vocabulary large too, even though it makes no sense to do so on such a small model, because that fits on their hardware. So Gemma doesn't run as efficiently on a GPU as a Lama does, right? But vice versa, Lama doesn't run as a llama does, right? But by subversa,"
},
{
"start_time": 14568.54,
"end_time": 14592.04,
"content": " llama doesn't run as efficiently on a TPU as a Gemma does, right? And so there's like certain like aspects of like hardware software code design. So all their search models are their ranking and recommendation models. All these different models that are AI, but not like Gen. AI, right, have been hyper-optimized with TPUs forever. The software stack is super optimized, but all of this software stack has not been released publicly at all, right? Very small"
},
{
"start_time": 14592.04,
"end_time": 14612.5,
"content": " portions of it, Jackson, XLA have been. But like, the experience when you're inside of Google and you're training on TPUs as a researcher, you don't need to know anything about the hardware in many cases, right? Like it's like pretty beautiful. But as soon as you step outside, they'll go, a lot of them go back. They leave Google and then they go back. Yeah. Yeah, they're like, they leave and they start a company because they have all these amazing research ideas."
},
{
"start_time": 14612.5,
"end_time": 14634.96,
"content": " And they're like, wait, infrastructure's hard. Software is hard. And this is on gpues or if they try to use tpues same thing because they don't have access to all this code but so it's like how do you convince a company whose golden goose is searched where they're making hundreds of billions of dollars from to start selling GPUs, which they used to only buy a couple billion of, you know, I think in 23, they bought like, like a couple billion. And now they're buying like couple billion."
},
{
"start_time": 14635.56,
"end_time": 14658.76,
"content": " And now they're buying like 10 billion to $15 billion worth. But how do you convince them that they should just buy like twice as many and figure out how to sell them and make $30 billion. Like who cares about making $30 billion? Won't that $30 billion exceed actually the search profit eventually? Oh, I mean, like, you're always gonna make more money on services than- Always. I mean, like, like you like to be clear like today people are spending a lot"
},
{
"start_time": 14658.76,
"end_time": 14682.25,
"content": " more on hardware than they are the services, right? Because the hardware front runs the service spend. But like, you're investing. If there's no revenue for AI stuff or not enough revenue, then obviously, like, it's going to blow up, right? You know, people won't continue to spend on GPUs forever. And InVdia is trying to move up the stack with like software that they're trying to sell and license and stuff, right? But Google has never had that DNA of like,"
},
{
"start_time": 14682.25,
"end_time": 14707.33,
"content": " this is a product we should sell, right? They don't act, the Google Cloud does it, which is a separate organization from the TPU team, which is a separate organization from the DeepPU team, which is a separate organization from the deep mine team, which is a separate organization from the search team, right? There's a lot of bureaucracy. Wait, Google Cloud is a separate team than the TPU team. Technically, TPU sits under infrastructure which sits under Google Cloud, but like Google Cloud for like renting stuff and TPU architecture are very different goals, right?"
},
{
"start_time": 14707.33,
"end_time": 14734.61,
"content": " In hardware and software, like all of this right like the jacks xLA teams do not serve google's customers externally whereas invidia's various kDA teams for like things like nickel serve external customers, right? The internal teams like Jackson, XLA and stuff, they more so serve deep mind and search. Right. And so their customers different. They're not building a product for them. Do you understand why AWS keeps winning versus Azure for cloud versus Google Cloud?"
},
{
"start_time": 14734.77,
"end_time": 14755,
"content": " Yeah, there's Google Cloud is Yeah, there's- Google Cloud is tiny, isn't it, relative to the- Google Cloud is third. Yeah, yeah. Microsoft is the second biggest, but Amazon is the biggest, right? Yeah. And Microsoft deceptively sort of includes Microsoft Office 365 and things like that. Like some of these enterprise-wide licenses. So in reality, the Gulf is even larger. Microsoft is still second though, right? Amazon is way bigger."
},
{
"start_time": 14755.16,
"end_time": 14775.24,
"content": " Why? Because using AWS is better and easier. And in many cases, it's cheaper. And it's first. It was first. Yeah, but there's a lot of things that are first that lose the... Well, it's easier, it's harder to switch than it is to... Yeah, okay. Because it's... There's big fees for switching too. AWS generates over 80% of Amazon's profit, I think over 90%. That's insane. The distribution centers are just like,"
},
{
"start_time": 14775.24,
"end_time": 14796,
"content": " one day we'll decide to make money from this. But they haven't yet, right? Like they make tiny little profit from- Yeah, one day Amazon Prime will triple in price. You would think they would improve AWS interface because it's like horrible. It's like horrible. It's like clunky, but everybody is. I don't, yeah. You have one would think. I think actually Google's interface is sometimes nice,"
},
{
"start_time": 14796,
"end_time": 14816.6,
"content": " but it's also like they don't care about anyone besides their top customers. Exactly. And like their customer service sucks and like they have a lot less like I mean all these companies they opt optimized for the big customers yeah that it's supposed to be for business Amazon has always optimized for the small customer too, though, right? Like, obviously they optimize a lot for the big customer, but when they started, they just would go to like random Bay Area things and give out credits, right?"
},
{
"start_time": 14816.6,
"end_time": 14837.14,
"content": " And then they like or just put in your credit card and use us, right? Like it went back in the early days. So they've always the business has grown with them, right? And burgeon. So like, why does Amazon, like, why is Snowflake all over Amazon? Because Snowflake in the beginning when Amazon didn't care about them was still using Amazon right and then of course one day Snowflake and Amazon has a super huge partnership but like this is the case like Amazon's user experience and quality is better."
},
{
"start_time": 14837.14,
"end_time": 14857.36,
"content": " Also a lot of the silicon they've engineered makes them have a lower cost structure in traditional cloud storage, CPU, networking, that kind of stuff, then in databases, right? Like, you know, I think like four of Amazon's top five revenue products, margin products are like gross profit products or all database related products, like Redshift and like all these things, right?"
},
{
"start_time": 14857.36,
"end_time": 14878.16,
"content": " Like, so Amazon has a very like good silicon to a user experience like entire pipeline with a ws i think google their infrastructure their silicon teams yeah they have awesome silicon internally, TPU, the YouTube chip, you know, some of these other chips that they've made. And the problem is they're not serving external customers or serving internal customers, right?"
},
{
"start_time": 14878.32,
"end_time": 14898.84,
"content": " I mean, Invidio's entire culture is designed from the bottom up to do this. There's this recent book, The NVIDIA way by Take Him that details this and how they look for future opportunities and ready their Kuda software libraries to make it so that new applications of high performance computing can very rapidly be evolved on Kuda and"
},
{
"start_time": 14898.84,
"end_time": 14921,
"content": " Nvidia chips. And that is entirely different than Google as a services business. Yeah, I mean, in V, it should be said as a truly special company. Like, I mean, they, they're the whole, the culture of everything, they're really optimized for that kind of thing. Speaking of which, is there somebody that can even challenge Nvidia, hardware-wise? Intel, AMD. I really don't think so."
},
{
"start_time": 14921.02,
"end_time": 14941.3,
"content": " We went through a very long process working with AMD on training on their GPUs at inference and stuff. And they're decent. Their hardware is better in many ways than in VINVVDIS. The problem is their software is really bad. And I think they're getting better, right? They're getting better faster, but they're just, the gulf is so large. And like, they don't"
},
{
"start_time": 14941.3,
"end_time": 14965.03,
"content": " spend enough resources on it or haven't historically, right? Maybe they're changing their tune now, but, you know, for, for multiple months, we were submitting those bugs, right? Like us, semi-analysis, right? Like, what the fuck? Like, why are we submitting the most bugs? Right? Because they only cared about their like biggest customers and so they'd ship them a private image blah blah blah and it's like okay but like i am just using pie torch and I want to use the publicly available libraries"
},
{
"start_time": 14965.03,
"end_time": 14985.39,
"content": " and you don't care about that, right? So they're getting better, but like, I think AMD is not possible. Intel is obviously in dire straits right now and needs to be saved somehow. Very important for national security, for American technology elements. Can you explain obviously? So why are they in the United States? Going back to earlier, only three companies can R&D, right?"
},
{
"start_time": 14988.53,
"end_time": 15006.65,
"content": " Taiwan, Sunshu, Samsung, Pyongyang and then Intel Hillsborough. Samsung's doing horribly. Intel's doing horribly. We could be in a world where there's only one company that can do R&D. And that one company already manufactures most of chips. They've been gaining market share anyways. But like that's that's a critical thing. Right. So what happens to Taiwan means the rest of the world's semiconductor industry and therefore tech relies on Taiwan, right?"
},
{
"start_time": 15006.93,
"end_time": 15029.15,
"content": " And that's obviously precarious. As far as like Intel, they've been slowly steadily declining. They were on top of servers and PCs, but now Apple's done the M1, and Nvidia's releasing a PC chip, and Qualcomm's releasing a PC chip, and in servers, hyperscalers are all making their own arm-based server chips. And Intel has no AI silicon like wins, right?"
},
{
"start_time": 15029.15,
"end_time": 15051.71,
"content": " They have very small wins. And they never got into mobile because they said no to the iPhone. And like all these things have compounded and they've lost their process technology leadership right they were ahead for 20 years and now they're behind by at least a couple years right and they're trying to catch back up, and we'll see if, like, their 18A, 14A strategy works out where they try and leapfrog to SMC. But like, and Intel is leapfrog to SMC. But like, and Intel is just like losing tons of money anyways, right?"
},
{
"start_time": 15051.71,
"end_time": 15073.01,
"content": " And they just fired their CEO, even though the CEO was the only person who understood the company well, right? We'll see. He was not the best, but he was pretty good relatively, technical guy. Where doesn't tell make most of its money, the CPUs still? PCs and data center CPUs, yeah, but data center CPUs are all going cloud, and Amazon, Microsoft, Google are making arm-based CPUs. And then PC side,"
},
{
"start_time": 15073.31,
"end_time": 15097,
"content": " AMD's gained market share, Nvidia's launching a chip. That's not gonna be success, right? Media Tech Qualcomm ever launch chips. Apple's doing well, right? Like they could get squeezed a little bit in PC, although PC generally, I imagine, will just stick Intel mostly for Windows side. Let's talk about the broad AI race. Who do you think wins? We talked about Google. The leader, the default leader has been Google because of their infrastructure advantage."
},
{
"start_time": 15097.5,
"end_time": 15118.84,
"content": " Well, like, in the news, OpenAI is the leader. They're the leading in the narrative. They have the best model. They have the best model that people can use, and they're experts. And they have the most AI revenue. Yeah. Open AI. have the most AI revenue. Yeah, Open AI is winning. So who's making money on AI right now? Is anyone making money? So accounting profit-wise, now. Is anyone making money? So accounting profit-wise, Microsoft is making money, but they're spending a lot of"
},
{
"start_time": 15118.84,
"end_time": 15140.24,
"content": " CapEx, right? You know, and that gets depreciated over years. Meta is making tons of money, but with recommendation system, which is AI, but not with Lama, right? Lama's losing money for sure, right? I think anthropic and opening eye are obviously not making money because otherwise they wouldn't be raising money, right? They have to raise money to build more, right? Although theoretically, they are making money, right?"
},
{
"start_time": 15140.24,
"end_time": 15162.15,
"content": " Like, you know, you spent a few hundred million dollars on GPD4 and it's doing billions in revenue. So like obviously it's like making money. Although they had to continue to research to get the compute efficiency wins, right? And move down the curve to like, you know, that 12, get that 1200 X that has been achieved for GPT3, you know, maybe we're only at like a, you know, a couple hundred X now, but you know, with GPT4 turbo and 4O and there will be another one probably cheaper"
},
{
"start_time": 15162.15,
"end_time": 15183.91,
"content": " than GPD40 even that comes out at some point. And that research costs a lot of money. Yep, exactly. That's the thing that I guess is not talked about with the cost, that when you're referring to the cost of the model, it's not just the training or the test runs, it's the actual research, the manpower. Yeah, to do things like reasoning right now that that exists."
},
{
"start_time": 15183.91,
"end_time": 15205.53,
"content": " They're going to scale it. They're going to do a lot of research still. I think the, you know, people focus on the payback question but it's really easy to like just be like well like you know GDP is humans in industrial capital, right? And if you can make intelligence cheap, then you can grow a lot, right? That's the sort of dumb way to explain it. But that's sort of what basically the investment thesis is."
},
{
"start_time": 15205.53,
"end_time": 15228.49,
"content": " I think only Nvidia is actually making tons of money and other hardware vendors. The hyperscalers are all on paper making money, but in reality, they're spending a lot more on purchasing the GPUs, which you don't know if they're still gonna make this much money on each GPU in two years, right? Um, you don't know if all of a sudden open AI goes kapoof and now Microsoft has like hundreds of thousands of GPUs"
},
{
"start_time": 15228.49,
"end_time": 15250.07,
"content": " they were renting to Open AI that are that they paid for themselves with their investment in them, you know, that no longer have a customer, right? Like this is always a possibility. I don't believe that, right? I think, you know, opening eye will keep raising money. I think others will keep raising money. I think others will keep raising money because the investments, the returns from it are going to be eventually huge once we have AGI. So do you think multiple companies will get,"
},
{
"start_time": 15250.07,
"end_time": 15272.81,
"content": " let's assume. I don't think it's winner take all. Okay. So it's not, let's not call it AGI, whatever. It's like a single day. It's a gradual thing. Super powerful AI. But it's a gradually increasing set of features that are useful and rapidly increasing set of features. Rapidly increasing set of features. So you're saying a lot of companies will be,"
},
{
"start_time": 15272.81,
"end_time": 15294.13,
"content": " it just seems absurd that all of these companies are building gigantic data centers. There are companies that will benefit from AI, but not because they train the best model. Meta has so many avenues to benefit from AI and all of their services. People are there. People spend time on meta's platforms, and it's a way to make more money per user per hour."
},
{
"start_time": 15294.13,
"end_time": 15315.54,
"content": " Yeah, it seems like Google X slash XAI slash Tesla, important to say, and then meta will benefit not directly from the AI, like the LLMs, but from the intelligence, like the additional boost of intelligence to the products they already sell. So whether that's the recommendation system"
},
{
"start_time": 15315.54,
"end_time": 15335.66,
"content": " or for Elon who's been talking about optimists, the robot, potentially the intelligence of the robot. And then you have personalized robots in the home, that kind of thing. He thinks it's a 10 plus trillion dollar business, which at some point maybe. I don't, not soon, but who knows what robots"
},
{
"start_time": 15335.66,
"end_time": 15356.92,
"content": " Let's do a TAM analysis, right? Eight billion humans and let's get eight billion robots, right? And let's pay them the average salary. And yeah, there we go, 10 trillion. More than 10 trillion. Yeah, I mean, you know, if there's robots everywhere, why does it have to be just eight, eight billion robots? Yeah, yeah, of course, of course. I'm going to get, I'm going to have like one robot. You're going to have like 20."
},
{
"start_time": 15357.4,
"end_time": 15377.48,
"content": " Yeah, I mean, I see a use case for that. So yeah. So I guess the benefit would be in the products they sell, which is why Open AI is in a trickier position because they... All of the value of Open AI right now as a brand is in ChatGPT. And there is actually not that, for most users, there is not that much of a reason that they need Open AI to be spending billions and billions of"
},
{
"start_time": 15377.48,
"end_time": 15398.74,
"content": " dollars on the next best model when they could just license Lama 5 and for be way cheaper. So that's kind of like chat GPT is an extremely valuable entity to them. But they could make more money just off that than the chat application is clearly like does not have tons of room to continue, right? Like the standard chat, right, where you're just using it for random questions and stuff, right?"
},
{
"start_time": 15399.06,
"end_time": 15419.42,
"content": " The cost continues to collapse, V3 is the latest one biggest. But it's going to get supported by ads, right? Like as, you know, Lema already serves 405B, probably loses the money, but at some point, you know, they're going to get, uh, the models are going to get so cheap that they can just serve them for free with ad supported, right? And that's what Google is going to be able to do. And that's obviously they've got a bigger reach, right?"
},
{
"start_time": 15419.42,
"end_time": 15441.4,
"content": " So chat is not going to be the only use case. It's like these reasoning, code, agents, computer use, all this stuff is where OpenA has to actually go to make money in the future otherwise they're capits but x google and meta have these other products. So isn't, isn't it likely that open AI and anthropic disappear eventually?"
},
{
"start_time": 15441.9,
"end_time": 15461.72,
"content": " Unless they're so good at models, but they are. But it's such a cutting, I mean, yes. It depends on where you think AI capabilities are going. You have to keep winning. Yes. You have to keep winning. As you climb, even if the AI capabilities are going super rapidly awesome into the direction of AGI, like there's still a boost for X"
},
{
"start_time": 15461.72,
"end_time": 15487.02,
"content": " in terms of data, Google in terms of data, meta in terms of data, in terms of other products and the money and there's just huge amount other products and the money and the, like, there's just huge amounts of money. The whole idea is human data is kind of tapped out. We don't care. We all care about self-play, verifiable tasks. Yeah, so self-play verifiable tasks. If you think about AWS, which is an R&G problem. AWS does not make a lot of money on each individual machine. And the same can be said for the most powerful AI platform, which is even though the calls"
},
{
"start_time": 15487.02,
"end_time": 15512.71,
"content": " to the API are so cheap, there's still a lot of money to be made by owning that platform. And there's a lot of discussions as it's the next compute layer. You have to believe that, and yeah, there's a lot of discussions that tokens and tokenomics and LLM APIs are the next compute layer, or the next paradigm for the economy, kind of like energy and oil was. But there's also like, you have to sort of believe that APIs and chat are not where AI is stuck, right?"
},
{
"start_time": 15512.73,
"end_time": 15533.47,
"content": " It is actually just tasks and agents and robotics and computer use and those are the areas where all the value will be delivered not API not chat application. Is it possible you have, I mean, it all just becomes a commodity and you have the very thin wrapper, like perplexity. Just joking."
},
{
"start_time": 15534.63,
"end_time": 15554.71,
"content": " There are a lot of rappers making a lot of money. Yeah, but do you think it's possible that people who would just even forget what OpenAI and the Thropic is and just because there'll be wrappers around the API and it just dynamically. If model progress is not rapid yeah, it's becoming a commodity, right? DeepSeek V3 shows this, but also the GPT3 chart earlier chart showed this, right?"
},
{
"start_time": 15554.71,
"end_time": 15576.74,
"content": " Lama 3B is 1,200 X cheaper than GPD 3. Any GPG3, like anyone whose business model was GPG3 level capabilities is dead. Anyone whose business model's GPD4 level capabilities is dead. It is a common saying that the best business is being made now are ones that are predicated on models getting better. Right. Which would be like rappers, thing that is riding the wave of the models."
},
{
"start_time": 15577.4,
"end_time": 15597.8,
"content": " The short term, the company that could make the most money is the one that figures out what advertising, targeting is the one that figures out what advertising targeting method works for language model generations. We have the meta ads, which are hyper-targeted in feed, not within specific pieces of content. And we have search ads that are used by Google and Amazon has been rising a lot on search. But within a piece, within a has been rising a lot on search. But within a piece, within a return from chat GPT,"
},
{
"start_time": 15597.8,
"end_time": 15619.08,
"content": " it is not clear how you get a high quality placed ad within the output. And if you can do that with model costs coming down, you can just get super high revenue. That revenue is totally untapped, and it's not clear technically how it is done. Yeah, that is, I mean, the, sort of the ad sense innovation that Google did the one day"
},
{
"start_time": 15619.08,
"end_time": 15639.78,
"content": " you'll have in GPT output an ad, and that's going to make billions, if not. It could be very subtle. It could be in conversation. We have voice mode now. It could be some way of making it so the voice introduces certain things. It's much harder to measure, and it takes imagination, but yeah. And it wouldn't be so shady, it wouldn't come off shady,"
},
{
"start_time": 15639.78,
"end_time": 15659.92,
"content": " so you would receive public blowback, that kind of thing. So you have to do it loud enough to where it's clear. It's an ad and balance all that. So that's the open question that's trying to solve. Anthropic and open AI they need to. They might not say I don't think they care about that at all. They don't care about it right now. I think it's I like perplexity are experimenting on that more. Oh, interesting."
},
{
"start_time": 15659.92,
"end_time": 15681.68,
"content": " Yeah, for sure. Like perplexity Google Medicare about this I think open eye and anthropic are purely laser focused on AGII Yeah, agents and AGI. And if I build AGI, I can make tons of money, right? Or I can pay for everything, right? And this is, this is, it's just predicated, like back on the like export control thing, right? If you think AGI is"
},
{
"start_time": 15681.68,
"end_time": 15702.1,
"content": " five, ten years away or less, right? These labs think it's two, three years away. Obviously, your, your, your actions are, are you know if you assume they're rational actors which they are mostly you're what you do in a two- a g i versus five year versus ten years very very very different right do you think agents are promising?"
},
{
"start_time": 15703.44,
"end_time": 15725.9,
"content": " We'll have to talk about this. This was, uh, this is like the excitement of the year that agents are going to reverev this is the generic hype term that a lot of business folks are using AI agents are going to revolutionize everything. Okay, so mostly the term agent is obviously overblown. We've talked a lot about reinforcement learning as a way to train for verifiable outcomes."
},
{
"start_time": 15725.9,
"end_time": 15755.33,
"content": " Agents should mean something that is open-ended and is solving a task independently on its own and able to adapt to uncertainty there is a lot of the term agent applied to things like Apple Intelligence, which we still don't have after the last WWDC, which is orchestrating between apps. And that type of tool use thing is something that language models can do really well. Apple intelligence, I suspect, will come eventually. It's a closed domain. It's your messages app integrating with your photos with AI in the background, that will work."
},
{
"start_time": 15755.33,
"end_time": 15777.23,
"content": " That has been described as an agent by a lot of software companies to get into the narrative. Yeah. The question is what ways can we get language models to generalize to new domains and solve their own problems in real time, maybe some tiny amount of training when they are doing this with fine-tuning themselves or in context learning, which is the idea of storing"
},
{
"start_time": 15777.23,
"end_time": 15798.43,
"content": " information in a prompt, and you can use learning algorithms to update that, and whether or not you believe that that is going to actually generalize to things like me saying book my trip to go to Austin in two days I have XYZ constraints and actually trusting it. I think there's an HCI problem coming back for information."
},
{
"start_time": 15799.33,
"end_time": 15820.25,
"content": " Well, what's your prediction there? Because my gut says we're very far away from that. I think opening eyes statement, I don't know if you've seen the five levels right or its chat is level one reasoning is level two and then agents is level three and I think there's a couple more levels. But it's important to note, right? We were in chat for a couple years, right?"
},
{
"start_time": 15820.25,
"end_time": 15841.68,
"content": " We just theoretically got to reasoning we'll be here for a year or two, right? And then agents, but at the same time, like people can, people can try and like approximate capabilities of the next level but the agent agents are doing things autonomously doing things for minutes at a time, hours at a time, et cetera, right? Reasoning is doing things for tens of seconds at a time, right?"
},
{
"start_time": 15841.68,
"end_time": 15863.96,
"content": " And then coming back with an output that I still need to verify and use and try to check out, right? So, and the biggest problem is, of course, like, it's the same thing with manufacturing, right? Like there's the whole Six Sigma thing, right? Like, you know, how many nines do you get? And then you compound the nines onto each other and it's like if you multiply you know by the number of steps that are six sigma you get to uh you know, a yield or something, right?"
},
{
"start_time": 15863.96,
"end_time": 15886.9,
"content": " So like in semiconductor manufacturing, tens of thousands of steps, 999-999 is not enough, right? Because you multiply by that by that many times, you actually end up with like 60% yield, right? Or zero. Really low yield, yeah? Or zero. Really low yield, yeah, or zero. And this is the same thing with agents, right? Like chaining tasks together each time, LLMs, even the best LLMs in particularly pretty good benchmarks don't get 100%."
},
{
"start_time": 15886.9,
"end_time": 15908.37,
"content": " Right? They get a little bit below that because there's a lot of noise and so how do you get to enough nines right this is the same thing with self-driving. We can't have self-driving because without it being like super geo-fenced like Google's, right? And even then, they have a bunch of teleoperators to make sure it doesn't get stuck, right? But you can't do that because it doesn't have enough nights. And self-driving"
},
{
"start_time": 15908.37,
"end_time": 15929.01,
"content": " has quite a lot of structure because roads have rules. It's well defined. There's regulation. When you're talking about computer use for the open web, for example, or the open operating system, like there's no or the open operating system. Like there's no, it's a mess. So like the possibility, I'm always skeptical of any system"
},
{
"start_time": 15929.01,
"end_time": 15949.15,
"content": " that is tasked with interacting with the human world with the open messy human world, with the open, messy human world. If we can't get intelligence that's enough to solve the human world on its own, We can create infrastructure, like the human operators for Waymo, over many years that enable certain workflows. There is a company. I don't remember it,"
},
{
"start_time": 15949.21,
"end_time": 15970.57,
"content": " but it is, but that's literally their pitches. Yeah, we're just going to be the human operator when agents fail. And you just call us and you fix it. Yeah. It's like an API call and it's hilarious. There's going to be teleoperation markets when we get human robots, which is there's going to be somebody around the world that's happy to fix the fact that it can't finish loading my dishwasher when I'm unhappy with it, but that's just going to be part of the Tesla service package."
},
{
"start_time": 15970.57,
"end_time": 15991.43,
"content": " I'm just imagining like an AI agent, talking to another AI agent. One company has an agent that specializes in helping other AI agents. But if you can make things that are good at one step, you can stack them together. So that's why I'm like, if it takes a long time, we're going to build infrastructure that enables it. You see the operator launch."
},
{
"start_time": 15991.55,
"end_time": 16014.89,
"content": " They have partnerships with certain websites, with DoorDash, with OpenTable, with things like this, those partnerships are going to let them climb really fast. Their model is going to get really good at those things. It's going to get really good at those things. It's going to prove of concept, that might be a network effect where more companies want to make it easier for AI. Some companies will be like, no, let's put blockers in place. Yeah. And this is the story of the internet we've seen. We see it now with training data"
},
{
"start_time": 16014.89,
"end_time": 16035.67,
"content": " for language models where companies are like, no, you have to pay. Like business working it out. That said, I think airlines have a very, and hotels have high incentive to make their site work really well, and they usually don't. Like if you look at how many clicks it takes to order an airplane ticket, it's insane. I don't... You actually can't call"
},
{
"start_time": 16035.67,
"end_time": 16060.17,
"content": " an American Airlines agent anymore. They don't have a phone number. It's... I mean, it's horrible on many on the interface front and all the, to imagine that agents will be able to deal with that website when I as a human struggle, like I have an existential crisis every time I try a book airplane ticket that I don't, I think it's going to be extremely difficult to build an AI agent that's robust."
},
{
"start_time": 16060.17,
"end_time": 16083.44,
"content": " But think about it, like, United has accepted the Starlink term, which is they have to provide Starlink for free and the users are going to love it. What if one airline is like, we're going to take a year and we're going to make our website have white text that works perfectly for the AIs. Every time anyone asks about an AI flight, they buy whatever airline it is. Or like, they just like, here's an API in, it's only exposed to AI agents."
},
{
"start_time": 16083.44,
"end_time": 16103.86,
"content": " And if anyone queries it, the price is 10% higher and for any flight, but we'll let you see any of our flights and you can just book any of them. Here you go. And then it's like, oh, and I made 10% higher price. Awesome. Yeah. And like, am I willing to say that for like, hey, book me a flight to C-Lex, right? And it's like, yeah, whatever. Yeah. I think, I think, you know, computers and real world and the open world"
},
{
"start_time": 16103.86,
"end_time": 16124.86,
"content": " are really, really messy. But if you start defining the problem in narrow problem in narrow regions, people are going to be able to create very, very productive things. And, and ratchet down cost massively, right? Like now crazy things like, you know, robotics in the home, those are going to be a lot harder to do just like self-driving,"
},
{
"start_time": 16125.12,
"end_time": 16146.58,
"content": " right? Because there's just a billion different failure modes, right? But like, but, like, agents that can, like, navigate a certain set of websites and do certain sets of task, or, like, look at, you know, look at your, you know, take a photo of your grocery, uh, your fridge and or like upload your recipes and then like it figures out what to order from, you know, Amazon slash Whole Foods food delivery. Like that's, then that's gonna be like pretty quick"
},
{
"start_time": 16146.58,
"end_time": 16168.61,
"content": " and easy to do, I think. So it's gonna be a a whole range of business outcomes. And it's going to be tons of, tons of sort of optimism around people can just figure out ways to make money. To be clear, these sandboxes already exist in research. There are people who have built clones of all the most popular websites of Google, Amazon, blah, blah, blah, to make it so that there's, I mean, Open AI probably has them internally to train these things. It's the same as Deep Minds"
},
{
"start_time": 16168.61,
"end_time": 16191.81,
"content": " robotics team for years has had clusters for robotics where you interact with robots fully remotely. They just have a lab in London and you send tasks to it, arrange the blocks and you do this research. Obviously, there's techs there that fix stuff, but we've turned these cranks of automation before. You go from sandbox to progress and then you add one more domain at a time and generalize."
},
{
"start_time": 16191.81,
"end_time": 16212.57,
"content": " I think in the history of NLP and language processing, instruction tuning in tasks per language model used to be like one language model did one task. And then in the instruction tuning literature, there's this point where you start adding more and more tasks together, where it just starts to generalize to every task. And we don't know where on this curve we are. I think for reasoning with this RL and verifiable domains, we're very, we're early,"
},
{
"start_time": 16212.57,
"end_time": 16234.57,
"content": " but we don't know where the point is where you just start training on enough domains and poof, like more domains just start working and you've crossed the generalization barrier. Well, what do you think about the programming context? So software engineering, that, you know, that's where I personally and I know a lot of people, um, interact with AI the most."
},
{
"start_time": 16234.73,
"end_time": 16256.43,
"content": " There's a lot of fear and angst too from current CS students, but there's also that's where that is the area where probably the most AI revenue and productivity gains have come, right? Whether it be co-pilots or cursor or what have you, right? This is, or just standard chat GPT, right? Like a lot of, I don't, I know very few programmers who don't have chat GPT, and actually many of them have the $200 tier,"
},
{
"start_time": 16256.43,
"end_time": 16276.63,
"content": " because that's what it's so good for, right? I think that in that world, we already see it like sui bench. And if you've looked at the benchmark made by some Stanford students, I wouldn't say it's like really hard, but I wouldn't say it's easy either. I think like it takes someone who's been through at least, you know, a few years of CS or a couple years of programming to do sweepbench well."
},
{
"start_time": 16276.79,
"end_time": 16298.77,
"content": " And the models went from 4% to 60% in like a year, right? And where are they going to go to next year? You know, it's going to be higher. It probably won't be 100% because, again, that nines is like really hard to do. But we're going to get to some point where that's, and then we're going to need harder software engineering benchmarks and so on and so forth. But the way that like people think of it now is it can do code completion, easy."
},
{
"start_time": 16298.77,
"end_time": 16319.23,
"content": " It can do some function generation and have to review it. Great. But really the software engineering agents, I think, can be done faster sooner than any other agent because it is a verifiable domain. You can always like unit test or compile. And there's many different regions of like, it can inspect the whole code base at once, which no,"
},
{
"start_time": 16319.23,
"end_time": 16339.33,
"content": " no engineer really can. Only the architects can really think about this stuff, the really senior guys, and they can define stuff. And then the agent can execute on it. So I think software engineering costs are going to plummet like crazy. And one interesting aspect of that is when software engineering costs are really low, you get very different markets. Right. So in the US, you have all these platforms SaaS companies, right?"
},
{
"start_time": 16339.33,
"end_time": 16362.54,
"content": " Salesforce and so on and so forth. In China, and so on and so forth, right? In China, no one uses platform SaaS. Everyone just builds their own stack because software engineering is much cheaper in China. And partially because like people, the number of STEM graduates, et cetera. So STEM is such generally just cheaper to do. And so at the same time, code for LLMs have been adopted much less in China"
},
{
"start_time": 16362.54,
"end_time": 16383.94,
"content": " because the cost of an engineer there is much lower. But like what happens when every company can just invent their own business logic like really cheaply and quickly. You stop using platform SaaS. You start building custom tailored solutions. You change them really quickly. Now all of a sudden your business is a little bit more efficient too potentially because you're not dealing with the hell that is like some random platform SaaS company stuff not working perfectly and having to adjust workflows"
},
{
"start_time": 16383.94,
"end_time": 16411.24,
"content": " or random business automation cases that aren't necessarily AI required. It's just logic that needs to be built that no one is built. All of these things can go happen faster. And so I think software, and then, and then the other domain is like industrial, chemical, mechanical engineers, suck at coding, right? Just generally. And like, their tools, like semiconductor engineers, their tools are 20 years old. All the tools run on XP, including ASML lithography tools, run on Windows XP, right? It's like, you know, and like a lot of the analysis happens in Excel, right?"
},
{
"start_time": 16411.32,
"end_time": 16445.02,
"content": " Like, it's just like, guys, like you guys can move 20 years forward with all the data you have and gathered and like do a lot better. It's just you need the engineering skills for software engineering to be delivered to the actual domain expert engineer. So I think that's the area where I'm like super duper bullish of generally AI creating value. The big picture is that I don't think it's going to be a clip. It's like, we talked to anything, a really good example of how growth changes is when meta added stories. So Snapchat was on an exponential. They added stories. It flatlined. Software engineers been up until the right."
},
{
"start_time": 16445.46,
"end_time": 16467.02,
"content": " AI is going to come in. It's probably just going to be flat. It's like, it's a lot like everyone's going to lose their job. It's hard because the supply corrects more slowly. So the amount of students is still growing and that'll correct on a multi-year like a year delay, but the amount of jobs will just turn and then maybe in 20 40 years it'll be well down but in the few years"
},
{
"start_time": 16467.02,
"end_time": 16490.82,
"content": " there'll never be the snap moment where it's like software engineers aren't useful. I think also the nature of what it means to be a programmer and what kind of jobs programmers do changes. Because I think there needs to be a human in the loop of everything you've talked about. There's a really important human in that picture of like correcting the code. Like fixing... like fixing larger than the context length."
},
{
"start_time": 16491.12,
"end_time": 16517.03,
"content": " Yep. And debugging also, like debugging by, so reading the code, understanding the steering the system like no, no, no, you missed the point, adding more to the prompt. Kind of like, yes, adding the human... Designing the perfect Google button. Google's famous for having people design buttons that are so perfect. And it's like, how, like, how is AI going to do that? Like, it's like, they could give you all the ideas, perfect fine."
},
{
"start_time": 16517.03,
"end_time": 16538.83,
"content": " I mean, that's the thing. You can call it taste. Humans have, one thing humans can do is figure out what other humans enjoy better than AI systems. That's where the preference, you loading that in, but ultimately humans are the greatest preference generate. That's where the preference comes from. And humans are actually very good at reading or like judging between two things versus this is this goes back to the core of what"
},
{
"start_time": 16538.83,
"end_time": 16561.41,
"content": " R LHF in preference tuning is is that it's hard to generate a good answer for a lot of problems, but it's easy to see which one is better. And that's how we're using humans for AI now is judging which one is better. And that's what software engineering could look like. It's the PR review, here's a few options. What are the like, here are some potential pros and cons? And they're going to be judges. I think the thing I would very much"
},
{
"start_time": 16561.41,
"end_time": 16583.94,
"content": " recommend is people's I think the thing I would very much recommend is people start, programmers start using AI and embracing that role of the supervisor of the AI system and like partner of the AI system versus writing from scratch or not learning coding coding at all, and just generating stuff. Because I think there actually has to be a pretty high level of expertise as a programmer to be able to manage increasingly intelligent systems."
},
{
"start_time": 16583.94,
"end_time": 16606.9,
"content": " I think it's that and then becoming a domain expert in something. Sure. Yeah. Sure. Yeah. Because seriously, if you go look at aerospace or semiconductors or chemical engineering, everyone is using really crappy platforms, really old software. Like the job of the data sciences is like a joke, right? In many cases, in many cases, it's very real, but it's like, bring what the forefront of human capabilities are to your domain."
},
{
"start_time": 16606.9,
"end_time": 16626.98,
"content": " And like, even if the forefront is like from the AI, your domain, you're like at the forefront, right? So it's like, it's like you have to be at the forefront of something and then leverage the like rising tide that is AI for everything else. Oh yeah, there's so many low hanging fruit everywhere in terms of where software can help automate a thing or digitize a thing"
},
{
"start_time": 16626.98,
"end_time": 16647.9,
"content": " in the legal system. I mean, that's why Doge is exciting. You have, I mean, I got to hang out with a bunch of the doge folks and they, I mean, government is like so old school. It's like begging for the modernization of software, of organizing the data, all this kind of stuff."
},
{
"start_time": 16647.9,
"end_time": 16669.16,
"content": " I mean, in that case, it's by design, because bureaucracy creates, protects centers of power and so on. But software breaks down those barriers. So it hurts those that are holding on to power, but ultimately benefits humanity. So there's a bunch of domains of that kind."
},
{
"start_time": 16673.76,
"end_time": 16695.95,
"content": " One thing we didn't fully finish talking about is open source. So first of all, congrats. You released a new model. Yeah. Tulu. I'll explain what a Tulu is. A Tulu is a hybrid camel when you breed a dromedary with a Bacrian camel. Back in the early days after Chachapit, there was a big wave of models coming out like Alpaca, Vicuna, etc. that were all named after various mammalian species."
},
{
"start_time": 16695.95,
"end_time": 16716.89,
"content": " So Tulu is, the brand is multiple years old, which comes from that. And we've been playing at the frontiers of post-training with open source code. And this first part of this release was in the fall where we used, we've built on Lama's open models, open weight models, and then we add in our fully open code or fully open data."
},
{
"start_time": 16717.89,
"end_time": 16737.29,
"content": " There's a popular benchmark that is chatbot arena and that's generally the metric by which how these chat models are evaluated and it's humans compare random models from different organizations. And if you looked at the leaderboard in November or December, among the top 60 models from tens to 20s of organizations, none of them had open code or data"
},
{
"start_time": 16737.29,
"end_time": 16758.33,
"content": " for just post-training. Among that, even fewer or none have pre-training data and code available, but post-training is much more accessible at this time. It's still pretty cheap and you can do it. And the thing is, like, how high can we push this number where people have access to all the code and data. So that's kind of the motivation of the project. We draw on lessons from Lama, and Vida had a Nematron model where the recipe for their post-training"
},
{
"start_time": 16758.33,
"end_time": 16781.27,
"content": " was fairly open with some data and a paper, and it's putting all these together to try to create a recipe that people can fine-tune models like GPT4 to their domain. So to be clear, in the case of Tulu, maybe you can talk about Alma too, but in the case of Toul, you're taking Lama 3, 4,5B. Tudu has been a series of recipes for post-raining,"
},
{
"start_time": 16781.27,
"end_time": 16801.74,
"content": " so we've done multiple models over years. And so you're open sourcing everything. Yeah. If you start with an open weight base model, the whole model technically is an open source, because you don't know what Lama put into it, which is why we have the separate thing that we'll get to. But it's just getting parts of the pipeline where people can zoom in and customize. I know I hear from startups and businesses."
},
{
"start_time": 16801.74,
"end_time": 16824.27,
"content": " They're like, okay, like I can take this post training and try to apply it to my domain. We talk about verifiers a lot. We use this idea, which is reinforcement learning with verifiable rewards, RLVR, kind of similar to RLHF, and we applied it to map. And the model today, which is like we applied it to the Lama 405B base model from last year."
},
{
"start_time": 16824.27,
"end_time": 16844.53,
"content": " And we have our other stuff. We have our instruction tuning and our preference tuning, but the math thing is interesting, which is like it's easier to improve this math benchmark. There's a benchmark, M-A-T-H, math, all capitals. Tough name. On the benchmark, name is the area that you're evaluating. We're researchers. We're not brands, brand strategists."
},
{
"start_time": 16844.53,
"end_time": 16866.71,
"content": " And this is something that the deep seek paper talked about as well, as like at this bigger model it's easier to elicit powerful capabilities with this rl training and then they distill it down from that big model to the small model. And this model we released today, we saw the same thing as we're at AI2, we don't have a ton of compute. We can't train 405B models all the time. So we just did a few runs and they tend to work."
},
{
"start_time": 16866.83,
"end_time": 16888.56,
"content": " And it's like, it just shows that there's a lot of room for people to play in these things and that and they crushed llama's actual release, right? Like they're way better than it. Yeah, so our VAL numbers, I mean, we have extra months in this, but our VAL numbers are like much better than the Lama instruct model that they released. And then you also said better than DeepSeek V3. Yeah, on our Eval benchmark."
},
{
"start_time": 16888.56,
"end_time": 16908.9,
"content": " The most deep seek V3 is really similar. We have a safety benchmark to understand if it will say harmful things and things like that. And that's what draws down most of the way. It's still like... It's like an amalgamation of multiple benchmarks, or what do you mean? Yeah, so we have a 10 value. This is like, this is standard practice in post training, is you choose your evaluations you care about. In academics and smaller labs, you'll have fewer evaluations."
},
{
"start_time": 16908.9,
"end_time": 16931.84,
"content": " In companies, you'll have a really one domain that you really care about. In Frontier Labs, you'll have tens to 20s to maybe even like 100 evaluations of specific things. So we'd choose a representative suite of things that look like chat, precise instruction following, which is like respond only in emojis. Like does model follow weird things like that? Yeah. Math, code. And you create a suite like this. So safety would be one of 10 in that type of suite where you have"
},
{
"start_time": 16931.84,
"end_time": 16952,
"content": " what is the broader community of AI care about? And for example, in comparison to deep seek, it would be something like our average avowal for our model would be 80, including safety and similar without and deep seek would be like 79% average score without safety and their safety score would bring it down like so"
},
{
"start_time": 16952,
"end_time": 16973.74,
"content": " you beat them even ignoring safety yeah so this is something that internally, it's like, I don't want to win only by like how you shape the VAL benchmark. So if there's something that's like people may or may not care about safety in their model. Safety can come downstream. Safety can be when you host the model for an API. Like safety is addressed in a spectrum of locations and AI applications. So it's like, if you want to say that you have the best recipe, you can't just gated on"
},
{
"start_time": 16973.74,
"end_time": 16997.07,
"content": " these things that some people might not want. And this is like the time of progress. We benefit if we can release a model later. We have more time to learn new techniques like this RL technique. We had started this in the fall. It's now really popular reasoning models. The next thing to do for open source post-training is to scale up verifiers to scale up data to replicate some of deep seek's results."
},
{
"start_time": 16997.07,
"end_time": 17018.15,
"content": " And it's awesome that we have a paper to draw and that it makes it a lot easier. And that's the type of things that is going on among academic and closed frontier research in AI. Since you're pushing open source, what do you think is the future of it? You think DeepSeek actually changes things since it's open source or open weight or is pushing the open source moving into the open"
},
{
"start_time": 17018.15,
"end_time": 17038.27,
"content": " direction. This goes very back to the license discussion. So DeepSeek R1 with a friendly license is a major reset. So it's like the first time that we've had a really clear frontier model that is open weights and with a commercially friendly license with no restrictions on downstream use cases, synthetic data distillation, whatever. This has never been the case at all in the history of AI in the last"
},
{
"start_time": 17038.27,
"end_time": 17059.89,
"content": " few years since ChatGPT. There have been models that are off the frontier or models with weird licenses, that you can't really use them. So isn't Meta's license pretty much permissible except for five companies. And there's also, so this goes to like what open source AI is, which is there's also use case restrictions in the Lama license, which says you can't use it for specific things. So if you come from an open source software background,"
},
{
"start_time": 17059.89,
"end_time": 17083.65,
"content": " you would say that that is not an open source license. What kind of things are those, though? Like, are they like... At this point, I can't pull them off the top of my head. But it'll be like, it used to be military use was one and they removed that for scale. It'll be like, like C-SAM, like child abuse material. Like that's the type of thing that is forbidden there. But that's enough from an open source background to say it's not an open source license."
},
{
"start_time": 17083.65,
"end_time": 17107.97,
"content": " And also the Lama license has this horrible thing where you have to name your model llama if you touch it to the llama model. So it's like the branding thing. So if a company uses Lama, technically the license says that they should say built with Lama at the bottom of their application. And from like a marketing perspective, that just, that just hurts. Like I can, I could suck it up as a researcher. I'm like, oh, it's fine. Like it says Lama Dash on all of our on all of our materials for this release. But this is why we need"
},
{
"start_time": 17107.97,
"end_time": 17128.58,
"content": " truly open models, which is we don't know deep seek R1's data. So you're saying I can't make a cheap copy of Lama and pretend it's mine, but I can do this with the Chinese model. Yeah. Hell yeah. That's what I was saying. And that's why it's like we want this whole open language models thing, the Olmo thing, is to try to keep the model"
},
{
"start_time": 17128.58,
"end_time": 17151.84,
"content": " where everything is open with the data as close to the frontier as possible. So we're compute constrained, we're personnel constrained, we rely on getting insights from people like John Schulman tells us to do RL on outputs. Like we can make these big jumps, but it just takes a long time to push the frontier of open source. And fundamentally, I would say that that's because open source AI does not have the same feedback loops as open source software."
},
{
"start_time": 17151.84,
"end_time": 17177.93,
"content": " We talked about open source software for security. Also, it's just because you build something once and you can reuse it. If you go into a new company, there's so many benefits. But if you open source a language model, you have this data sitting around, you have this training code. It's not like that easy for someone to come and build on and improve because you need to spend a lot on compute, you need to have expertise. So until there are feedback loops of open source AI, it seems like mostly an ideological mission."
},
{
"start_time": 17178.13,
"end_time": 17199.23,
"content": " People like Mark Zuckerberg, which is like America needs this. And I agree with him, but in the time where the motivation ideologically is high, we need to capitalize and build this ecosystem around what benefits do you get from seeing the language model data? And there's not a lot about that. We're going to try to launch a demo And there's not a lot about that. We're going to try to launch a demo soon where you can look at an Omo model and a query"
},
{
"start_time": 17199.23,
"end_time": 17225.43,
"content": " and see what pre-training data is similar to it, which was like legally risky and complicated, but it's like, what does it mean to see the data that the AI was trained on, it's hard to parse. It's terabytes of files. It's like, I don't know what I'm going to find in there. But that's what we need to do as an ecosystem if people want open source AI to be financially useful. We didn't really talk about Stargate. I would love to get your opinion on like what the new"
},
{
"start_time": 17225.43,
"end_time": 17246.79,
"content": " administration, the Trump administration, everything that's doing, that's being done from the America side and supporting AI infrastructure and the efforts of the different AI companies. What do you think about Stargate? What are we supposed to think about Stargate and does Sam have the money? Yeah, so I think Stargate is a opaque thing."
},
{
"start_time": 17246.91,
"end_time": 17277.91,
"content": " It definitely doesn't have $500 billion. Doesn't even have $100 billion. Right. So what they announced is this $500 billion number, Larry Ellison, Sam Altman, and Trump said it. They thanked Trump, and it's used... Trump did do some executive actions that do significantly improve the ability for this to be built faster. You know, one of the executive actions you did is on federal land, you can just basically build data centers in power, you know, like pretty much like that. Uh, and then the permitting process is basically gone or you file after the fact."
},
{
"start_time": 17278.29,
"end_time": 17299.87,
"content": " So like one of the, again, like I had a schizo take earlier. Another schizotake, if you've ever been to the Presidio in San Francisco, beautiful area. You could build a power plant and a data center there if you wanted to. Because it is federal land. It used to be a military base. But obviously this would piss people off. It's a good bit. Anyways. Trump has made it much easier to do this, right?"
},
{
"start_time": 17299.91,
"end_time": 17320.45,
"content": " Generally, Texas has the only unregulated grid in the in the nation as well. Let's go Texas. Um, and so, you know, therefore like, Urquot enables people to build faster as well in addition the federal regulations are coming down and so Stargate is predicated. This is why that whole show happened. Now, how they came up with a $500 billion number is beyond me."
},
{
"start_time": 17320.83,
"end_time": 17351.56,
"content": " How they came up with a $100 billion number makes sense to some extent, right? And there's actually a good table in here that I would like to show in that Stargate piece that I had it's the most recent one, yeah. So anyways, Stargate, you know, it's basically right, like there is uh it's it's a table about cost um there you passed it already it's that one. So this table is kind of explaining what happens, right?"
},
{
"start_time": 17351.56,
"end_time": 17374.24,
"content": " So Stargate is in Abilene, Texas, the first $100 billion of it. That site is 2.2 gigawatts of power in, about 1.8 gigawatts of power consumed, right? Per GPU, they have like roughly, Oracle is already building the first part of this before Stargate came about. To clear they've been building it for a year, they tried to rent it to Elon, in fact."
},
{
"start_time": 17374.66,
"end_time": 17398.32,
"content": " Right. But Elon was like, it's too slow. I need it faster. So then he went and did his Memphis thing. And so opening, I was able to get it with this weird joint venture called Stargate. They initially signed a deal with just Oracle for the first section of this cluster, right? This first section of this cluster, right? This first section of this cluster, right, is roughly, um, $5 billion to $6 dollars of server spend right and then there's another billion or so"
},
{
"start_time": 17398.32,
"end_time": 17419.14,
"content": " of data center spend but the and then and then likewise like if you fill out that entire 1.8 gigawatts with the next two generations of invidious chips gb 200 gb300 gb300, VR 200, and you fill it out completely, that ends up being roughly $50 billion of server cost, right? Plus there's data center cost, plus maintenance cost, plus operation cost,"
},
{
"start_time": 17419.24,
"end_time": 17443.49,
"content": " plus all these things. And that's where OpenAI gets to their $100 billion announcement that they had, right? Because they talked about $100 billion is phase one. That's this Abilene, Texas data center, right? $100 billion of total cost of ownership, quote unquote, right? So it's not KAPX, it's not investment. It's $100 billion of total cost of ownership. And then, and then there will be future phases. They're looking at other sites that are even bigger than this 2.2 gigawatts,"
},
{
"start_time": 17443.49,
"end_time": 17465.29,
"content": " by the way, in Texas and elsewhere. And so they're not completely ignoring that. But there is the number of $100 billion that they say is for phase one, which I do think will happen. They don't even have the money for that. Furthermore, it's not $100 billion. It's $50 billion of spend, right? And then like $50 billion of operational cost power etc. Rental pricing,"
},
{
"start_time": 17465.29,
"end_time": 17486.31,
"content": " et cetera, because they're renting it for opening eyes renting the GPs from the Stargate joint venture right what money do they actually have right soft bank soft bank is going to invest oracle's going to invest open A is going to invest Oracle's going to invest Open AI is going to invest Open AI is on the line for $19 billion everyone knows that they've only got $6 billion in their last round and $4 billion a debt. So, but there is, there's like news of like soft bank,"
},
{
"start_time": 17486.47,
"end_time": 17506.93,
"content": " maybe investing $25 billion into opening. Right so that's that's that's that's part of it right so 19 billion can come from there so opening i does not have the money at all right To be clear, ink is not dried on anything. Open has zero dollars for this $50 billion, right? And which they're legally obligated to put $ 19 billion of CAPEX or into the joint venture, and then the rest they're going to pay via renting the GPUs from the joint venture."
},
{
"start_time": 17506.93,
"end_time": 17527.49,
"content": " And then there's um then there's Oracle. Oracle has a lot of money. They're building the first section completely. They were spending for it themselves, right? This $6 billion of Kappex, 10,000. they were spending for it themselves right this six billion dollars of cap x 10 billion dollars a tCO um but they and they were going to do that first section they're paying paying for that. Right. Um, as far as the rest of the section, I don't know how much Larry wants to spend, right? At any point, he can pull out, right?"
},
{
"start_time": 17527.49,
"end_time": 17552.24,
"content": " Like this is again, it's like completely voluntary so at any point there's no signed ink on this right but he potentially could contribute tens of billions of dollars right to be clear he's got the money. Oracle's got the money. And then there's like MGX, which is the South, the UAE fund, which technically has $1.5 trillion for investing in AI. But again, like, I don't know how real that money is. And like, whereas there is no ink signed for this. SoftBank does not have $25 billion"
},
{
"start_time": 17552.24,
"end_time": 17578.5,
"content": " of cash. They have to sell down their stake in arm, which is, you know, the leader in CPUs and they, they IPOed it. This is obviously what they've always wanted to do. They just didn't know where they'd redeploy the capital, selling down the stake in arm. they just didn't know where they'd redeploy the capital. Selling down the stake in arm makes a ton of sense. So they can sell that down and invest in this if they want to and invest in open eye if they want to. As far as like money secured, the first 100,000 GB 200 cluster is like can fund be funded everything else after that up in the air is up in the"
},
{
"start_time": 17578.5,
"end_time": 17602.17,
"content": " air money's coming I believe the money will come. I personally do. It's a belief. It's a belief that they are going to release better models and be able to raise more money. Right? Yeah. But like the actual reality is is that Elon's right. There is the money does not exist. What does the US government have to do with anything? What does Trump have to do with anything what does Trump have to do with everything he's just the hype man Trump is he's reducing the regulation so they can build it faster, right?"
},
{
"start_time": 17603.69,
"end_time": 17622.33,
"content": " And he's allowing them to do it, right? You know, because any investment of this side is going to involve like antitrust stuff, right? Like, so obviously he's going to, he's going to allow them to do it. He's going to enable the regulations to actually allow it to be built. Uh, I don't believe there's any US government dollars being spent on this, though. Yeah. So I think he's also just creating a general vibe that this is regulation will go down"
},
{
"start_time": 17622.33,
"end_time": 17643.97,
"content": " and this is the era of building. So if you're a builder, you want to create stuff, you want to launch stuff, you want to launch stuff. This is the time to do it. And so like we've had this 1.8 gigawatt data center in our data for over a year now. And we've been like sort of sending it to all of our clients, including many of these companies that are building the multi-gigawatts. But that is like at a level that's not quite maybe executives like seeing 500 billion dollars hundred"
},
{
"start_time": 17643.97,
"end_time": 17664.59,
"content": " billion dollars and then everyone's asking them like so it could spur like another like an even faster arms race spur like another, like an even faster arms race, right? Because there's already an arms race, but like, this, this like $100 billion, $500 billion number, Trump talking about it on TV, like it could spur the arm race to be even faster and more investors to flood in and et cetera, et cetera. So I think I think you're right is that in that sense, that open"
},
{
"start_time": 17664.59,
"end_time": 17685.89,
"content": " eye or sort of Trump is sort of like championing people are going to build more and his actions are going to let people build more what are you uh what are you excited about about these several years that are upcoming, in terms of cluster buildouts, in terms of breakthroughs and AI, like the best possible future you can imagine"
},
{
"start_time": 17685.89,
"end_time": 17706.09,
"content": " in the next couple years, two, three, four years. What does that look like? years, two, three, four years. What does that look like? It could be very specific technical things like breakthroughs on post-training, or it could be just size, big, impressive clusters. I really. I really enjoy tracking supply chain and like who's involved in what. I really do."
},
{
"start_time": 17706.09,
"end_time": 17726.39,
"content": " It's really fun to see like the numbers, the cost, who's building what capacity, helping them, figure out how much capacity they should build, winning deals, strategic stuff. That's really cool. I think technologically, there's a lot around the networking side that really excites me with optics and electronics, right, like kind of getting closer and closer, whether it be copackage optics or some sort of like"
},
{
"start_time": 17726.39,
"end_time": 17747.75,
"content": " forms of new forms of switching. This is internal to a cluster. Yeah. Also multi-data center training, right? Like there's people are putting so much fiber between these data centers and lighting it up with so many different, you know, with so much bandwidth that there's a lot of interesting stuff happening on that end. Telecom has been really boring since 5G, and now it's like really exciting again on the harder side."
},
{
"start_time": 17747.75,
"end_time": 17768.8,
"content": " Can you educate me a little bit about the speed of things? So the speed of memory versus the speed of interconnect versus the speed of fiber between data centers? Are these like orders of magnitude different? Can we at some point converge towards a place where it all just feels like one computer? No, I don't think that's possible. It's only going to get harder to program, not easier. It's only going to get harder to program, not easier."
},
{
"start_time": 17768.8,
"end_time": 17790.24,
"content": " Okay. It's only going to get more difficult and complicated and more layers, right? The general image that people like to have is like this hierarchy of memory. So on chip is really close, localized within the chip, right? You have registers, right? Those are shared between some compute elements. And then you'll have caches, which are shared between more compute elements. Then you have memory, right, like HBM or DRAM, like DDR memory or whatever it is,"
},
{
"start_time": 17790.24,
"end_time": 17810.78,
"content": " and that's shared between the whole chip. And then you can have, you know, pools of memory that are shared between many chips, and then storage and you keep zoning out, right? The access latency across data centers, within the data centers, within the data center, within a chip is differs. So like you're obviously always, you're always going to have different programming paradigms for this. It's not going to be easy."
},
{
"start_time": 17810.88,
"end_time": 17832.84,
"content": " Programming the stuff is going to be hard. Maybe I can help, right? You know, with programming this. the the the way to think about it is that like there is there's sort of like the more elements you add to a task you don don't gain, you don't get strong scaling, right? If I double the number of chips, I don't get two X the performance, right?"
},
{
"start_time": 17832.84,
"end_time": 17853.56,
"content": " This is just like a reality of computing, because there's inefficiencies. And there's a lot of interesting work being done to make it not, you know, to make it more linear, whether it's making the chips more networked together, more tightly, or, you know, cool programming models, or cool algorithmic things that you can do on the model side. DeepSeek did some of these really cool innovations because they were limited on interconnect,"
},
{
"start_time": 17853.56,
"end_time": 17875.78,
"content": " but they still needed to parallelize, right? Like all sorts of, you know, all, everyone's always doing stuff. Google's got a bunch of work and everyone's got a bunch of work about this. That stuff is super exciting on the model and workload and innovation side, right? Hardware, solid state transformers are interesting, right, for the power side. There's all sorts of stuff on batteries and there's all sorts of stuff on, you know, I think when you look at, if you look at every layer of the compute stack,"
},
{
"start_time": 17876.02,
"end_time": 17896.74,
"content": " whether it goes from lithography and etch all the way to like fabrication, to like optics, to networking, to power, to transformers, to cooling, to, you know, a networking, and you just go on up and up and up and up to stack. You know, even air conditioners for data centers are like innovating, right? Like it's like there's like copper cables are innovating, right? Like you wouldn't think it, but copper cables, are, there's some innovations happening there with, like,"
},
{
"start_time": 17896.74,
"end_time": 17921.22,
"content": " the density of how you can pack them. And like, it's like all of these layers of the stack, all the way up to the models. Human progress is at a pace that's never been seen before. I'm just imagining you sitting back in a layer somewhere with screens everywhere, just monitoring the supply chain where all these clusters like all the information you're gathering i mean you're incredible there's a big. There's a big team. There's a big team. I mean, you're, you do quite incredible work with semi analysis."
},
{
"start_time": 17921.22,
"end_time": 17941.61,
"content": " I mean, it's just keeping your finger on the pulse of human civilization in the digital world. It's pretty cool, like just to watch feel that. Yeah, thank you. I guess that. Feel all of us like doing shit, epic shit. Feel the AGI. I mean, from meme to like reality."
},
{
"start_time": 17942.59,
"end_time": 17965.73,
"content": " What, Nathan, is there like breakthroughs that you're like looking forward to potentially? I had a while to think about this while listening to Dellen's beautiful response. He didn't listen to me. He was so no. I knew this was coming. And it's like realistically training models is very fun because there's so much low-hanging fruit and the thing that makes my job entertaining, I train models, I write analysis about what's happening with models."
},
{
"start_time": 17966.45,
"end_time": 17990.55,
"content": " And it's fun because there is obviously so much more progress to be had. And the real motivation why I do this somewhere where I can share things is that there's just, I don't trust people that are like, trust me, bro, we're going to make AI good. It's like, we're the ones that it's like, we're going to do it and you can trust us and we're just going to have all the AI and it's just like I would like a future where more people have a say in what AI is and can understand it."
},
{
"start_time": 17990.55,
"end_time": 18014.05,
"content": " And that's a little bit less fun that it's not a positive thing. I feel like, this is just all really fun like training models is fun and bring people in as fun but it's really like AI if it is going to be the most powerful technology of my lifetime, it's like, we need to have a lot of people involved in making that and making it open. Helps with that. As accessible as possible, as open as possible, yeah."
},
{
"start_time": 18014.25,
"end_time": 18035.18,
"content": " In my read of the last few years is that more openness would help the AI ecosystem in terms of having more people understand what's going on, rather that's researchers from non-AI fields to governments to everything. It doesn't mean that openness will always be the answer. I think then it will reassess of what is the biggest problem facing AI and tack on a different angle to the wild ride that we're on."
},
{
"start_time": 18036.18,
"end_time": 18058.86,
"content": " And for me, just from even the user experience, anytime you have the like Apathy said, the aha moments, like the magic, like seeing the reasoning, the chain of thought. It's like there's something really just fundamentally beautiful about that. It's putting a mirror to ourselves and seeing like, oh, shit, it is solving"
},
{
"start_time": 18058.86,
"end_time": 18079.6,
"content": " intelligence as the cliche, like, goal of these companies is, and you get to understand why we humans are special. The intelligence within us is special. And for now also why we are special in terms of we seem to be conscious and the AI systems for now aren't. And we get to explore that mystery."
},
{
"start_time": 18080.18,
"end_time": 18100.86,
"content": " So it's just really cool to get to explore these questions that I don't think, I would have never imagined, uh, would be even possible. Back when, so just watching with excitement, deep blue, Big Kasparov, like I wouldn't have ever thought this kind of AI would be possible in my lifetime."
},
{
"start_time": 18101.46,
"end_time": 18121.22,
"content": " It's like, this really feels like AI. It's incredible. I started with AI of learning to fly a cilia quad rotor. It's like, learn to fly and it just like, it learned to fly up. It would hit the ceiling and stop and catch it. It's like, okay, that is like really stupid compared to what's going on now. And now you could probably, with natural language, tell it to learn to fly"
},
{
"start_time": 18121.22,
"end_time": 18142.55,
"content": " and is going to generate the control algorithm required to do that. Probably. There's low level blockers. Like we had to do some weird stuff for that but you can you definitely back to our robotics conversation yeah when you have to interact an actual physical world that's hard what gives you hope about the future, a human civilization? Looking into the next 10 years, 100 years, 1,000 years, 100 years, 1,000 years."
},
{
"start_time": 18143.15,
"end_time": 18165.43,
"content": " How long do you think we'll make it? You think we've got a thousand years? Humans will definitely be around in a thousand years. I think there's ways that very bad things could happen. There will be way fewer humans, But humans are very good at surviving. There's been a lot of things that that is true. I don't think they're necessarily we're good at long-term credit assignment of risk."
},
{
"start_time": 18165.43,
"end_time": 18194.07,
"content": " But when the risk becomes immediate, we tend to figure things out. And for that reason, I'm like, there's physical constraints to things like AGI hyper like recursive improvement to kill us all type stuff. I'm for the physical reasons and for how humans have figured things out before, I'm not too worried about it. AI takeover. There are other international things that are worrying, but there's just fundamental human goodness and trying to amplify that."
},
{
"start_time": 18194.51,
"end_time": 18214.98,
"content": " And like, we're on a tenuous time. And I mean, if you look at humanity as a whole, there's been times where things go backwards. There's times when things don't happen at all and we're on a, what should be very positive trajectory right now. Yeah, there seems to be progress, but just like with power, there's like spikes of human suffering."
},
{
"start_time": 18215.72,
"end_time": 18239.72,
"content": " And we want to try to minimize the amount of spikes. Generally, humanity is going to suffer a lot less, right? I'm very optimistic about that. I do worry of like techno-fascism type stuff arising as AI becomes more and more prevalent and powerful and those who control it can do more and more. Maybe it doesn't kill us all, but at some point every very powerful human is going to want to"
},
{
"start_time": 18239.72,
"end_time": 18264.29,
"content": " brain computer interface so that they can interact with the AGI and all of its advantages in many more way and merge its mind with, you know, sort of like, and its capabilities or that person's capabilities can leverage those much better than anyone else and therefore be, you know, it won't be one person rule them all, but it will be, you know, the thing I worry about is it'll be like few people, you know, hundreds, thousands, tens of thousands, maybe millions of people rule whoever's left, right?"
},
{
"start_time": 18265.89,
"end_time": 18284.85,
"content": " And the economy around it, right? And I think it'll, that's like the thing that's probably more worrisome is like human machine amalgamations. This enables an individual human to have more impact on the world and that impact can be both positive and negative right? Generally, humans have positive impacts on the world, at least societally, but it's possible"
},
{
"start_time": 18284.85,
"end_time": 18310.05,
"content": " for individual humans to have such negative impacts. And AGI, at least as I think the labs define it, which is not a runaway sentient thing, but rather just something that can do a lot of tasks really efficiently, amplifies the capabilities of someone causing extreme damage. But for the most part, I think it'll be used for, you know, profit-seeking motives, which will then reduce, which will increase the abundance and supply of things and therefore reduce suffering,"
},
{
"start_time": 18310.21,
"end_time": 18330.63,
"content": " right? That's the goal. Scrolling on a timeline, just drawing stasis that is holding scrolling holds the status quo of the world that is a positive outcome, right? Like, it's like, if I have food tubes and lockdown scrolling and I'm happy, that's a positive outcome. While expensive Alka. While expanding out into"
},
{
"start_time": 18330.63,
"end_time": 18352.61,
"content": " the cosmos. Well, this is a fun time to be alive. And thank you for pushing the forefront of what is possible in humans. And thank you for talking today. This is fun. Thanks for having us. Thanks for talking today. This is fun. Thanks for having us. Thanks for listening to this conversation with Dylan Patel and Nathan Lambert. To support this podcast, please check out our sponsors in the description."
},
{
"start_time": 18357.62,
"end_time": 18375.22,
"content": " And now, let me leave you some words from Richard Feynman. For a successful technology, reality must take precedence over public relations. For nature cannot be fooled. Thank you for listening, and I hope to see you next time. You know, You know, Thank you."
}
]
}
```
Let's break down the output into its key components:
* **`segments`** : Segmented audio content with start and end timestamps.
* **`segment.start_time`** : The start time of the segment in seconds (relative to the start of the audio).
* **`segment.end_time`** : The end time of the segment in seconds (relative to the start of the audio).
* **`segment.content`** : The raw transcription of the spoken content.
* **`metadata.duration`** : The total duration of the audio in seconds.
## Key Features
* **Temporal Grounding**: Precise time segmentation and content localization
* **Long-form Support**: Process audio up to 12+ hours with automatic segmentation
* **Batch Processing**: Efficient handling of large audio collections
## Get Started with our Audio -> JSON API
Head over to our [Audio -> JSON](/api-reference/v1/post-audio-generate) to start building your own audio processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Classifying Documents
Source: https://docs.vlm.run/guides/doc-ai/guide-classifying-documents
Learn how to classify documents into categories like invoices, bank statements, and utility bills.
While traditional document processing systems often rely on template-based approaches or simple keyword matching, `vlm-1` can intelligently classify documents based on their content, layout, and visual characteristics. This enables robust classification of documents like invoices, bank statements, utility bills, and other document types, even when they come in different formats or layouts.
For example, below is a diagram showing how a document is classified into different types, and how each type can have its own custom post-processing logic.
## Classifying Financial Documents
Let's look at a financial document classification example to see how `vlm-1` can be used to automatically categorize different types of documents. In this example, we'll use `vlm-1` to classify documents into categories like invoices, bank statements, utility bills, and other financial documents. This classification can then be used to route documents to the appropriate processing pipeline or storage system.
### Define a custom schema for document classification
In the sections below, we'll showcase how to use the API for document classification. `vlm-1` can automatically classify documents based on their content and visual characteristics, providing both a classification and a rationale for its decision. First, let's create a custom schema that will be used to classify the documents.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from typing import Literal
from pydantic import BaseModel, Field
class DocumentClassification(BaseModel):
rationale: str = Field(..., description="A rationale for the classification, based on the content and visual features of the document. Keep it short and concise, yet detailed enough to justify the classification.")
document_type: Literal["invoice", "bank-statement", "utility-bill", "other"] = Field(..., description="The type of document being processed")
confidence: Literal["hi", "med", "lo"] = Field(..., description="Confidence score for the classification, based on the rationale provided and the visual features of the document. For ambiguous documents, the confidence score should be `lo`.")
```
### Classify documents
Once you have defined your custom schema, you can use **`vlm-1`** to classify documents according to this schema. The classification will be validated against the schema you defined, ensuring that it conforms to the expected structure and types. First, let's look at an example of how to classify a single document.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, GenerationConfig
# Initialize the client
client = VLMRun(api_key="")
# Classify a single document
path = Path("path/to/document.pdf")
prediction: PredictionResponse = client.document.generate(
file=path,
domain="document.classification",
config=GenerationConfig(response_model=DocumentClassification)
)
response_dict = prediction.response.model_dump()
print(response_dict)
```
### Sample Document Classification
Let's take a look at the sample output for a typical invoice document.
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"rationale": "The document contains a clear 'INVOICE' header, itemized list of products/services, and total amount due. The layout matches typical invoice formats with company details at the top and payment terms at the bottom.",
"document_type": "invoice",
"confidence": "hi"
}
```
Let's breakdown the output into their respective components:
* **`rationale`**: A detailed explanation of why it classified the document as an invoice, based on both content and visual features. This allows the developer or user to introspect on the classification and make any necessary adjustments downstream to the model.
* **`document_type`**: The correct document classification type, in this case an `invoice`.
* **`confidence`**: A qualitative confidence level of "hi", indicating strong certainty in the classification based on the clear presence of invoice-specific features.
## Processing larger document collections with `batch=True`
Once you have validated the classification for a single document, you can scale this process to classify larger collections of documents. The code example below shows how to process several documents in a directory. The rationale-based approach is particularly useful when dealing with ambiguous documents or when you need to understand why a document was classified in a certain way.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Same imports as before
# ...
# Classify the documents
requests = {}
for path in Path("path/to/documents").glob("*.pdf"):
prediction: PredictionResponse = client.document.generate(
file=path,
domain="document.classification",
config=GenerationConfig(response_model=DocumentClassification),
batch=True
)
requests[prediction.id] = {
"path": path,
"id": prediction.id
}
# Wait for all predictions to complete
predictions = {}
start_time = time.time()
while time.time() - start_time < 180:
# fetch the prediction result if it's completed
for id, request in requests.items():
# `client.predictions.wait()` will block until the prediction id is completed
prediction: PredictionResponse = client.predictions.wait(id=id, timeout=10)
predictions[id] = {**request, "response": prediction.response}
# wait for 1 second before checking again
time.sleep(1)
# break if all predictions are completed
if len(predictions) == len(requests):
break
# Get the results
for p in predictions.values():
print(p)
```
## Fine-tuning Document Classification
This feature is currently only available for our enterprise-tier customers. If you are interested in using this feature, please [contact us](mailto:support@vlm.run).
For **enterprise use-cases** where you need to fine-tune the model for **custom document types** and **improved accuracy**, you can use our [fine-tuning guides](/guides/fine-tune) to customize the model performance and scalability needs. This can include fine-tuning the model on your own document collections, customizing the classification schema, or adding new document types to the classification system. Fine-tuning can help you improve the accuracy and performance of the model for your specific document types, and also help you scale the model to handle larger volumes of documents with more efficient, lightweight fine-tuned models that are optimized for your specific use-case. Contact us at [support@vlm.run](mailto:support@vlm.run) to learn more about how we can help you with your fine-tuning needs.
## Try our Document -> JSON API today
Head over to our [Document -> JSON](/api-reference/v1/post-document-generate) to start building your own document processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Document Redaction & Edit
Source: https://docs.vlm.run/guides/doc-ai/guide-document-redaction
Automatically detect and redact or replace sensitive information in documents with enterprise-grade compliance.
VLM Run provides two complementary document privacy capabilities, both served via the `/document/generate` endpoint:
* **Redaction**: Detects and **blurs** sensitive information, making it unreadable while preserving document layout.
* **Edit (Replace)**: Detects sensitive information and **replaces** it with consistent dummy data (e.g. names become "John Smith", DOB becomes "10/06/1974"), producing a document that looks realistic but contains no real PII.
Each specialized domain follows industry-specific compliance standards, ensuring your documents are compliant while maintaining readability.
Original Document
Redacted Document
## Quick Start
Upload your document containing sensitive information:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pathlib import Path
client = VLMRun(api_key="")
file_response = client.files.upload(
file=Path("path/to/your_document.pdf")
)
```
Choose the appropriate domain for your use case:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
response = client.document.generate(
domain="healthcare.phi-redaction", # Blur sensitive information
file=file_response.id,
batch=True
)
```
Or use the edit-replace variant to substitute PHI with dummy data:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
response = client.document.generate(
domain="healthcare.phi-edit-replace", # Replace with dummy data
file=file_response.id,
batch=True
)
```
Wait for completion and access the result:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
completed_response = client.predictions.wait(response.id, timeout=120)
detected_items = completed_response.response["detected_items"]
result_uri = completed_response.response["uri"]
print(f"Detected items: {detected_items}")
print(f"Processed document: {result_uri}")
```
## Available Domains
Choose the appropriate domain based on your document type, compliance requirements, and desired output:
### Redaction Domains (Blur)
These domains detect sensitive information and **blur** it in the output document:
| Use Case | Domain | Compliance Standards |
| ----------------------- | -------------------------------- | ------------------------- |
| **Healthcare PHI** | `healthcare.phi-redaction` | HIPAA Safe Harbor |
| **Resume Redaction** | `hr.resume-redaction` | GDPR, CCPA, CPRA |
| **Legal Documents** | `legal.document-redaction` | Attorney-Client Privilege |
| **Financial Data** | `financial.document-redaction` | PCI DSS, SOX, GLBA |
| **FOIA Requests** | `government.foia-redaction` | FOIA Regulations |
| **Insurance Documents** | `insurance.document-redaction` | Insurance Regulations |
| **Real Estate** | `real-estate.document-redaction` | Real Estate Privacy |
| **PII Redaction** | `document.pii-redaction` | CA Penal Code Section 741 |
### Edit Domains (Replace)
These domains detect sensitive information and **replace** it with consistent dummy data:
| Use Case | Domain | Compliance Standards |
| ----------------------- | ----------------------------- | -------------------- |
| **Healthcare PHI Edit** | `healthcare.phi-edit-replace` | HIPAA Safe Harbor |
## Key Use Cases
### Healthcare & Insurance
* **Medical Records**: Redact PHI for research and sharing
* **Insurance Claims**: Remove sensitive medical and personal information
* **Clinical Data**: Protect patient privacy in studies and trials
### Financial Services
* **Loan Applications**: Redact personal financial information
* **Account Statements**: Remove sensitive account details
* **Compliance Reports**: Prepare regulatory submissions
* **M\&A Documents**: Protect proprietary information during due diligence
### Legal & Government
* **Court Filings**: Prepare public documents with protected information
* **FOIA Requests**: Redact exempt information for public release
* **Discovery Materials**: Redact sensitive information during legal processes
* **Attorney Communications**: Protect privileged information
### HR & Recruitment
* **Resume Processing**: Enable blind hiring by removing bias-inducing information
* **Employee Records**: Protect personal and sensitive employee data
* **Background Checks**: Remove sensitive verification data
## Information Types Redacted
VLM Run automatically detects and redacts:
* **Personal Identifiers**: Names, SSNs, account numbers, driver's licenses
* **Contact Information**: Addresses, phone numbers, email addresses
* **Financial Data**: Account balances, salary information, credit scores
* **Medical Information**: PHI, medical record numbers, health conditions
* **Legal Information**: Case numbers, settlement amounts, privileged communications
* **Geographic Data**: Addresses, ZIP codes, neighborhood information
## Complete Examples
### Redaction (Blur)
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, FileResponse
from pathlib import Path
client = VLMRun(api_key="")
file_response: FileResponse = client.files.upload(
file=Path("path/to/your_document.pdf")
)
response: PredictionResponse = client.document.generate(
domain="healthcare.phi-redaction",
file=file_response.id,
batch=True
)
completed_response = client.predictions.wait(response.id, timeout=120)
detected_items = completed_response.response["detected_items"]
redacted_uri = completed_response.response["uri"]
print(f"Detected items: {detected_items}")
print(f"Redacted document: {redacted_uri}")
```
### Edit (Replace with Dummy Data)
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, FileResponse
from pathlib import Path
client = VLMRun(api_key="")
file_response: FileResponse = client.files.upload(
file=Path("path/to/medical_record.pdf")
)
response: PredictionResponse = client.document.generate(
domain="healthcare.phi-edit-replace",
file=file_response.id,
batch=True
)
completed_response = client.predictions.wait(response.id, timeout=120)
detected_items = completed_response.response["detected_items"]
edited_uri = completed_response.response["uri"]
print(f"Detected items: {detected_items}")
print(f"Edited document: {edited_uri}")
```
## Example Responses
### Redaction Response
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"id": "052cf2a8-2b84-45f5-a385-ccac2aae13bb",
"status": "completed",
"response": {
"detected_items": [
{
"item_type": "name",
"value": "John Doe"
},
{
"item_type": "ssn",
"value": "123-45-6789"
},
{
"item_type": "telephone_number",
"value": "(555) 123-4567"
}
],
"uri": "https://storage.googleapis.com/vlm-userdata/healthcare/phi-redaction/redacted-document.pdf"
}
}
```
### Edit Response
The edit response has the same structure. The `detected_items` list contains the original PHI values that were found and replaced with dummy data in the output document:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"status": "completed",
"response": {
"detected_items": [
{
"item_type": "name",
"value": "Jane Doe"
},
{
"item_type": "date_of_birth",
"value": "03/15/1985"
},
{
"item_type": "ssn",
"value": "987-65-4321"
}
],
"uri": "https://storage.googleapis.com/vlm-userdata/healthcare/phi-edit-replace/edited-document.pdf"
}
}
```
## Benefits
### Operational Efficiency
* **Automated Processing**: Reduce manual redaction time from hours to minutes
* **Batch Operations**: Process large document volumes efficiently
* **Error Reduction**: Eliminate human errors in manual redaction processes
* **Scalability**: Handle growing document volumes without additional staff
### Compliance & Security
* **Regulatory Compliance**: Meet industry-specific requirements (HIPAA, PCI DSS, SOX, GDPR, etc.)
* **Data Breach Prevention**: Irreversible redaction prevents data recovery
* **Audit Trail**: Comprehensive logging for compliance verification
* **Legal Protection**: Reduce liability from accidental data exposure
### Cost Savings
* **Reduced Manual Labor**: Automate time-consuming redaction tasks
* **Lower Error Costs**: Prevent expensive compliance violations
* **Improved Productivity**: Focus staff on high-value activities
* **Scalable Operations**: Handle volume increases without proportional cost increases
## Supported Documents
* **PDF Documents** - Reports, contracts, legal briefs, medical records
* **Scanned Images** - Faxed documents, handwritten forms, ID cards
* **Multi-page Documents** - Complete case files, comprehensive reports
* **Mixed Content** - Documents containing both text and images
* **Spreadsheets** - Financial models, budget documents, transaction records
## Security Features
* 🔒 **Encryption**: All documents encrypted in transit and at rest
* 🏛️ **Regulatory Compliance**: Meets industry-specific standards
* 🔑 **Access Controls**: Role-based access and authentication
* 📝 **Audit Logging**: Comprehensive audit trails for all activities
* ⏰ **Secure URLs**: Time-limited, secure access to redacted documents
* 🚫 **Irreversible Redaction**: Permanent data removal prevents recovery
* 🔄 **Consistent Replacement**: Edit mode uses the same dummy values throughout a document for consistency
## Real-World Examples
See VLM Run's document redaction in action across different industries:
Healthcare PHI Redaction
Legal Document Redaction
Insurance Document Redaction
Race Blind PII Redaction
## Related Capabilities
Extract structured data from documents before redaction processing.
Locate sensitive information with precise coordinates for targeted redaction.
Define custom redaction rules for specific compliance requirements.
Process large documents and complex multi-page reports.
## Try our Document -> JSON API today
Head over to our [Document -> JSON](/api-reference/v1/post-document-generate) to start building your own document processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Parsing Intake Forms
Source: https://docs.vlm.run/guides/doc-ai/guide-healthcare-parsing-intake-forms
Extract structured data from healthcare documents like patient referrals, intake forms, and insurance cards.
Head over to our chat to see the healthcare patient referral parsing in action and explore more.
Healthcare back-office operations face a universal challenge: processing vast amounts of unstructured patient documents efficiently and accurately. From patient-intake forms to insurance cards, from medical history records to referrals, these documents come in countless formats, often as low-quality scans, faxes, or handwritten forms.
While traditional OCR tools struggle with the complexity of healthcare documents, `vlm-1` can extract structured data from patient referrals, intake forms, insurance cards, and other healthcare documents with high accuracy. This helps healthcare providers streamline their operations, reduce manual data entry, and focus on patient care.
Here's a step-by-step guide on how to process healthcare documents:
Use the [`/v1/files`](/api-reference/v1/files/post-file-upload) endpoint to upload the healthcare document you want to process.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.client.types import FileResponse
from pathlib import Path
# Initialize the client
client = VLMRun(api_key="")
# Upload the file
response: FileResponse = client.files.upload(
file=Path("")
)
print(f"Uploaded file:\n {response.model_dump()}")
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
// Initialize the client
const client = new VlmRun({
apiKey: "",
});
// Upload the file
const fileResponse = await client.files.upload(
filePath: ""
);
console.log(fileResponse);
```
You should see a response like this:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Uploaded file:
{
'id': '1e76cfd9-ba99-49b2-a8fe-2c8efaad2649',
'filename': 'file-20240815-7UvOUQ-patient_referral.pdf',
'bytes': 62430,
'purpose': 'assistants',
'created_at': '2024-08-15T02:22:06.716130',
'object': 'file'
}
```
Submit the uploaded file to the [`/v1/document/generate`](/api-reference/v1/post-document-generate) endpoint to start the document processing job. Specify the appropriate healthcare domain based on the document type.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client.types import PredictionResponse, GenerationConfig
from pathlib import Path
# Submit the document for processing with patient referral domain
response: PredictionResponse = client.document.generate(
file=Path("path/to/file.pdf"),
domain="healthcare.patient-referral",
config=GenerationConfig(grounding=True) # Enable visual grounding
)
print(f"Healthcare document processing job submitted:\n {response.model_dump()}")
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const response = await client.document.generate({
file: "path/to/file.pdf",
domain: "healthcare.patient-referral",
config: {
grounding: true // Enable visual grounding
}
});
console.log(response);
```
You should see a response like this:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Healthcare document processing job submitted:
{
"id": "052cf2a8-2b84-45f5-a385-ccac2aae13bb",
"created_at": "2024-08-15T02:22:09.157788",
"response": null,
"status": "pending"
}
```
You can now wait for the job to complete by calling the `predictions.wait` method:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Wait for the job to complete
response: PredictionResponse = client.predictions.wait(
id=response.id,
timeout=120,
)
print(f"Job completed:\n {response.model_dump()}")
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Wait for the job to complete
const job = await client.predictions.wait(response.id);
console.log(job);
```
## Healthcare Document Types
VLM Run supports various healthcare document types, each with its own specialized schema. For a complete list of supported healthcare domains, visit the [Healthcare Domains section in our hub](https://app.vlm.run/dashboard/hub#healthcare).
For higher-quality results, we always recommend enabling [Visual Grounding](/guides/doc-ai/guide-visual-grounding) to help the model understand the invoice and extract more accurate information. See [High-Accuracy Parsing with Grounding](#high-accuracy-parsing-with-grounding) for more details.
### Patient Referrals
Patient referrals contain critical information about the patient, referring provider, and reason for referral. VLM Run can extract this information with high accuracy.
Here is a visualization of the parsed patient referral along with the visual grounding that `vlm-1` can extract from a patient referral form:
Here is an example of the code to parse a patient referral form:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, GenerationConfig
from pathlib import Path
# Initialize the client
client = VLMRun(api_key="")
# Process a patient referral
response: PredictionResponse = client.document.generate(
file=Path("path/to/file.pdf"),
domain="healthcare.patient-referral",
config=GenerationConfig(grounding=True)
)
# Wait for the prediction to complete
response = client.predictions.wait(response.id)
print(response.response)
```
Here is an example of the structured JSON output that `vlm-1` can extract from a patient referral form. You can also navigate to our [chat](https://chat.vlm.run/c/92c10df4-83d6-481a-bc9a-4ed3f66f5397) to see the JSON output in action:
```json [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"id": "052cf2a8-2b84-45f5-a385-ccac2aae13bb",
"created_at": "2024-08-15T02:22:09.157788",
"status": "completed",
"response": {
"referral": {
"patient": {
"contact": [
{
"value": "kerimchen@gmail.com",
"value_metadata": {
"bboxes": [
{
"content": "kerimchen@gmail.com",
"bbox": {
"xywh": [
0.22692793931731986,
0.2333984375,
0.2692793931731985,
0.025390625
]
},
"page": 0
}
]
}
},
{
"value": "6153012311",
"value_metadata": {
"bboxes": [
{
"content": "6153012311",
"bbox": {
"xywh": [
0.611251580278129,
0.23291015625,
0.19405815423514539,
0.025390625
]
},
"page": 0
}
]
}
}
],
"dateOfBirth": "2001-09-15",
"dateOfBirth_metadata": {
"bboxes": [
{
"content": "9/15/2001",
"bbox": {
"xywh": [
0.6163084702907712,
0.19677734375,
0.1675094816687737,
0.0263671875
]
},
"page": 0
}
]
},
"medicalHistory": {
"medicalConditions": [
{
"note": "sustained back pain that doesn't go away after initial treatment",
"note_metadata": {
"bboxes": [
{
"content": "sustained back pain that doesn't go",
"bbox": {
"xywh": [
0.27939317319848295,
0.47265625,
0.5094816687737042,
0.0341796875
]
},
"page": 0
},
{
"content": "away after initial treatment",
"bbox": {
"xywh": [
0.27623261694058154,
0.51025390625,
0.3647281921618205,
0.03125
]
},
"page": 0
}
]
},
"text": "back pain",
"text_metadata": {
"bboxes": [
{
"content": "back pain",
"bbox": {
"xywh": [
0.28508217446270545,
0.29296875,
0.13084702907711757,
0.03125
]
},
"page": 0
}
]
},
}
],
},
"name": "Kevin Chen",
"name_metadata": {
"bboxes": [
{
"content": "Kevin",
"bbox": {
"xywh": [
0.19152970922882429,
0.201171875,
0.08786346396965866,
0.025390625
]
},
"page": 0
},
{
"content": "Chen",
"bbox": {
"xywh": [
0.3672566371681416,
0.20458984375,
0.06890012642225031,
0.0205078125
]
},
"page": 0
}
]
}
},
"reasonForReferral": "need further diagnose",
"reasonForReferral_metadata": {
"bboxes": [
{
"content": "need further diagnose",
"bbox": {
"xywh": [
0.27686472819216185,
0.37255859375,
0.32616940581542353,
0.03369140625
]
},
"page": 0
}
]
},
"receivingProvider": {
"contact": [
{
"system": "email",
"value": "sam@gmail.com",
"value_metadata": {
"bboxes": [
{
"content": "sam@gmail.com",
"bbox": {
"xywh": [
0.20353982300884957,
0.86328125,
0.2509481668773704,
0.02197265625
]
},
"page": 0
}
]
}
},
{
"system": "phone",
"value": "6516331234",
"value_metadata": {
"bboxes": [
{
"content": "6516331234",
"bbox": {
"xywh": [
0.6049304677623262,
0.85986328125,
0.2079646017699115,
0.025390625
]
},
"page": 0
}
]
}
}
],
"name": "Sam Yong",
"name_metadata": {
"bboxes": [
{
"content": "Sam",
"bbox": {
"xywh": [
0.16877370417193427,
0.830078125,
0.07648546144121365,
0.0205078125
]
},
"page": 0
},
{
"content": "Yong",
"bbox": {
"xywh": [
0.33691529709228824,
0.82958984375,
0.0638432364096081,
0.02294921875
]
},
"page": 0
}
]
},
},
"referringProvider": {
"contact": [
{
"system": "email",
"value": "irenewong@gmail.com",
"value_metadata": {
"bboxes": [
{
"content": "irenewong@gmail.com",
"bbox": {
"xywh": [
0.20037926675094817,
0.1484375,
0.26738305941845764,
0.02587890625
]
},
"page": 0
}
]
}
},
{
"system": "phone",
"value": "6753369412",
"value_metadata": {
"bboxes": [
{
"content": "6753369412",
"bbox": {
"xywh": [
0.6106194690265486,
0.146484375,
0.23640960809102401,
0.02685546875
]
},
"page": 0
}
]
}
}
],
"name": "Irene Wong",
"name_metadata": {
"bboxes": [
{
"content": "Irene",
"bbox": {
"xywh": [
0.18394437420986093,
0.119140625,
0.08091024020227561,
0.0205078125
]
},
"page": 0
},
{
"content": "Wong",
"bbox": {
"xywh": [
0.36788874841972186,
0.11376953125,
0.07269279393173199,
0.025390625
]
},
"page": 0
}
]
},
"specialty": "family health",
"specialty_metadata": {
"bboxes": [
{
"content": "family health",
"bbox": {
"xywh": [
0.6150442477876106,
0.10986328125,
0.17193426042983564,
0.029296875
]
},
"page": 0
}
]
}
}
}
}
}
```
### Insurance Cards
Insurance cards contain information about the patient's insurance coverage, including policy numbers, group numbers, and contact information. VLM Run can extract this information accurately.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process an insurance card
response: PredictionResponse = client.document.generate(
file=Path("path/to/file.pdf"),
domain="healthcare.medical-insurance-card",
config=GenerationConfig(grounding=True)
)
# Wait for the prediction to complete
response = client.predictions.wait(response.id)
print(response.response)
```
### Patient Intake Forms
Patient intake forms contain comprehensive information about the patient's medical history, current medications, allergies, and more. VLM Run can extract this information with high accuracy.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process a patient intake form
response: PredictionResponse = client.document.generate(
file=Path("path/to/file.pdf"),
domain="healthcare.patient-intake",
config=GenerationConfig(grounding=True)
)
# Wait for the prediction to complete
response = client.predictions.wait(response.id)
print(response.response)
```
### Medical History Forms
Medical history forms contain detailed information about the patient's past medical conditions, surgeries, and family medical history. VLM Run can extract this information accurately.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process a medical history form
response: PredictionResponse = client.document.generate(
file=Path("path/to/file.pdf"),
domain="healthcare.medical-history",
config=GenerationConfig(grounding=True)
)
# Wait for the prediction to complete
response = client.predictions.wait(response.id)
print(response.response)
```
## Custom JSON Schemas for Healthcare-Specific Needs
Healthcare organizations often have specific data extraction needs. VLM Run allows you to define custom JSON schemas to extract exactly the data you need from healthcare documents.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Define a custom schema for patient referrals
custom_schema = {
"type": "object",
"properties": {
"patient": {
"type": "object",
"properties": {
"name": {"type": "string"},
"birth_date": {"type": "string", "format": "date"},
"gender": {"type": "string", "enum": ["male", "female", "other"]},
"contact_number": {"type": "string"},
"email": {"type": "string", "format": "email"}
},
"required": ["name", "birth_date"]
},
"referral": {
"type": "object",
"properties": {
"reason": {"type": "string"},
"date": {"type": "string", "format": "date"},
"urgency": {"type": "string", "enum": ["routine", "urgent", "emergency"]},
"notes": {"type": "string"}
},
"required": ["reason", "date"]
},
"referring_provider": {
"type": "object",
"properties": {
"name": {"type": "string"},
"npi": {"type": "string"},
"facility": {"type": "string"},
"contact_number": {"type": "string"}
},
"required": ["name"]
}
}
}
# Process a patient referral with the custom schema
response: PredictionResponse = client.document.generate(
file=Path("path/to/file.pdf"),
json_schema=custom_schema,
config=GenerationConfig(grounding=True)
)
# Wait for the prediction to complete
response = client.predictions.wait(response.id)
print(response.response)
```
## Visual Grounding for Verification and Compliance
In healthcare, it's crucial to verify the accuracy of extracted data. VLM Run's [visual grounding feature](/guides/doc-ai/guide-visual-grounding) provides a clear link between the extracted data and its location in the original document, making it easier to verify the accuracy of the extraction and maintain an audit trail for compliance purposes.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process a patient referral with visual grounding
response: PredictionResponse = client.document.generate(
file=Path("path/to/file.pdf"),
domain="healthcare.patient-referral",
config=GenerationConfig(grounding=True)
)
# Wait for the prediction to complete
response = client.predictions.wait(response.id)
# The response will include bounding boxes for each extracted field
print(response.response)
```
Example response with visual grounding:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"patient": {
"name": "John Doe",
"name_metadata": {
"bbox": {
"xywh": [0.1, 0.2, 0.05, 0.02]
}
},
"birth_date": "1980-05-15",
"birth_date_metadata": {
"bbox": {
"xywh": [0.1, 0.23, 0.05, 0.02]
}
},
"gender": "male",
"gender_metadata": {
"bbox": {
"xywh": [0.1, 0.26, 0.05, 0.02]
}
}
},
"referral": {
"reason": "Chronic back pain",
"reason_metadata": {
"bbox": {
"xywh": [0.3, 0.2, 0.15, 0.02]
}
},
"date": "2024-03-01",
"date_metadata": {
"bbox": {
"xywh": [0.3, 0.23, 0.15, 0.02]
}
},
"urgency": "routine",
"urgency_metadata": {
"bbox": {
"xywh": [0.3, 0.26, 0.15, 0.02]
}
}
}
}
```
## Confidence Scoring for Data Accuracy
In healthcare, data accuracy is paramount. VLM Run provides confidence scores for each extracted field, allowing you to identify fields that may require manual verification.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process a patient referral with confidence scoring
response: PredictionResponse = client.document.generate(
file=Path("path/to/file.pdf"),
domain="healthcare.patient-referral",
config=GenerationConfig(
confidence=True,
grounding=True
)
)
# Wait for the prediction to complete
response = client.predictions.wait(response.id)
# The response will include confidence scores for each field
print(response.response)
```
Example response with confidence scores:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"patient": {
"name": "John Doe",
"confidence": "high",
"birth_date": "1980-05-15",
"confidence": "high",
"gender": "male",
"confidence": "high"
},
"referral": {
"reason": "Chronic back pain",
"confidence": "high",
"date": "2024-03-01",
"confidence": "medium",
"urgency": "routine",
"confidence": "low"
}
}
```
## Batch Processing
For healthcare organizations that need to process large volumes of documents, VLM Run supports batch processing. This allows you to submit multiple documents for processing and retrieve the results asynchronously.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload multiple files
file_paths = ["path/to/referral1.pdf", "path/to/referral2.pdf", "path/to/referral3.pdf"]
for file_path in file_paths:
response = client.files.upload(file=Path(file_path))
# Submit batch processing jobs
job_ids = []
for file_path in file_paths:
response = client.document.generate(
file=Path(file_path),
domain="healthcare.patient-referral",
batch=True,
config=GenerationConfig(grounding=True)
)
job_ids.append(response.id)
# Wait for all jobs to complete
results = []
for job_id in job_ids:
while True:
response = client.document.get(job_id)
if response.status == "completed":
results.append(response)
break
elif response.status == "failed":
print(f"Job {job_id} failed")
break
time.sleep(5) # Poll every 5 seconds
# Process the results
for result in results:
print(f"Document ID: {result.id}")
print(f"Patient: {result.response['patient']['name']}")
print(f"Referral Reason: {result.response['referral']['reason']}")
print("---")
```
## HIPAA Compliance and Data Security
VLM Run is designed with healthcare compliance in mind:
* 🏥 **HIPAA Compliance**: VLM Run's infrastructure is HIPAA-compliant, ensuring patient data is handled securely
* 🔒 **Data Encryption**: All data is encrypted in transit and at rest
* 🔑 **Access Controls**: Robust authentication and authorization mechanisms
* 📝 **Audit Logging**: Comprehensive audit trails for all data access and processing
## Use Cases
VLM Run's healthcare document processing capabilities can be applied to a wide range of use cases:
* 📋 **Patient Onboarding Automation**: Streamline the patient onboarding process by automatically extracting data from intake forms, insurance cards, and medical history forms. This reduces manual data entry, minimizes errors, and accelerates the onboarding process.
* 🔄 **Referral Management**: Efficiently process patient referrals by automatically extracting patient information, referring provider details, and referral reasons. This helps healthcare providers prioritize referrals based on urgency and ensure timely patient care.
* 💳 **Insurance Verification**: Automate the insurance verification process by extracting policy information from insurance cards and verifying coverage details. This reduces claim denials and improves revenue cycle management.
* 📁 **Medical Records Digitization**: Convert paper medical records into structured digital data for easier storage, retrieval, and analysis. This improves data accessibility and enables better patient care through comprehensive medical history access.
## Related Guides
Learn how to classify healthcare documents by type before processing.
General guide for parsing various document types.
Learn more about visual grounding for document verification.
Create custom schemas for healthcare-specific data extraction.
Head over to our chat to see the healthcare patient referral parsing in action and explore more.
## Try our Document -> JSON API today
Head over to our [Document -> JSON](/api-reference/v1/post-document-generate) to start building your own document processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Parsing Documents
Source: https://docs.vlm.run/guides/doc-ai/guide-parsing-documents
Extract structured data from long documents and reports.
## Getting Started
`vlm-1` can extract structured markdown from long documents and reports. Here's a rough breakdown of the steps involved in parsing a document:
Use the [`/v1/files`](/api-reference/v1/files/post-file-upload) endpoint to upload the document you want to parse.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.client.types import FileResponse
# Initialize the client
client = VLMRun(api_key="")
# Upload the file
response: FileResponse = client.files.upload(
file=Path("")
)
print(f"Uploaded file:\n {response.model_dump()}")
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
// Initialize the client
const client = new VlmRun({
apiKey: "",
});
// Upload the file
const fileResponse = await client.files.upload(
filePath: ""
);
console.log(fileResponse);
```
You should see a response like this:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Uploaded file:
{
'id': '1e76cfd9-ba99-49b2-a8fe-2c8efaad2649',
'filename': 'file-20240815-7UvOUQ-earnings_single_table.pdf',
'bytes': 62430,
'purpose': 'assistants',
'created_at': '2024-08-15T02:22:06.716130',
'object': 'file'
}
```
Submit the uploaded file (via its `file_id`) to the [`/v1/document/generate`](/api-reference/v1/post-document-generate) endpoint to start the document parsing job. For long documents, you should set `batch=True` to submit the job to a queue for processing.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client.types import PredictionResponse
# Submit the document for parsing
# Note: In this case, we are using the `document.markdown` domain
# which is optimized for extracting structured markdown from documents
response: PredictionResponse = client.document.generate(
file=response.id,
domain="document.markdown",
batch=True,
)
print(f"Document submitted [id={response.id}]")
print(response.model_dump_json(indent=2))
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const response = await client.document.generate({
fileId: fileResponse.id,
domain: "document.markdown",
batch: true,
});
console.log(response);
```
You should see a response like this:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Document parsing job submitted:
{
"id": "052cf2a8-2b84-45f5-a385-ccac2aae13bb",
"created_at": "2024-08-15T02:22:09.157788",
"response": null,
"status": "pending"
}
```
Use the [`/v1/predictions/{request_id}`](/api-reference/v1/predictions/get-predictions-by-id) endpoint to fetch the results of the document parsing job. The results of the extraction job will be in JSON format under the `response` field.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Fetch the results
response: PredictionResponse = client.predictions.wait(request_id)
print(f"Document parsing job results:\n {response.model_dump()}")
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Fetch the results
const response = await client.predictions.wait(requestId);
console.log(response);
```
You should see a response like this:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"id": "052cf2a8-2b84-45f5-a385-ccac2aae13bb",
"created_at": "2024-08-15T02:22:09.157788",
"status": "completed",
"response": {
"pages": [
{ // page 0
"content": "\n\n# Fine-tuning\nTechnique\n\n---\n\nFebruary 2024",
"markdown_content": "\n\nOpenAI logo\n\n# Fine-tuning\nTechnique\n\n---\n\nFebruary 2024",
"tables": null,
"figures": [
{
"id": 0,
"title": null,
"caption": null,
"content": "OpenAI logo"
}
]
},
{ // page 1
"content": "# Overview\n\nFine-tuning involves adjusting the parameters of pre-trained models on a specific dataset or task. This process enhances the model's ability to generate more accurate and relevant responses for the given context by adapting it to the nuances and specific requirements of the task at hand.\n\n**Example use cases**\n- Generate output in a consistent format\n- Process input by following specific instructions\n\n## What we'll cover\n\n* When to fine-tune\n* Preparing the dataset\n* Best practices\n* Hyperparameters\n* Fine-tuning advances\n* Resources\n\n---\n3",
"markdown_content": "# Overview\n\nFine-tuning involves adjusting the parameters of pre-trained models on a specific dataset or task. This process enhances the model's ability to generate more accurate and relevant responses for the given context by adapting it to the nuances and specific requirements of the task at hand.\n\n**Example use cases**\n- Generate output in a consistent format\n- Process input by following specific instructions\n\n## What we'll cover\n\n* When to fine-tune\n* Preparing the dataset\n* Best practices\n* Hyperparameters\n* Fine-tuning advances\n* Resources\n\n---\n3",
"tables": null,
"figures": null
},
...
]
}
}
```
To learn more about the `document.markdown` domain, see the [`MarkdownPage` Schema](/guides/schema/schema-markdown-page) guide.
## Try our Document -> JSON API today
Head over to our [Document -> JSON](/api-reference/v1/post-document-generate) to start building your own document processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Parsing Invoices
Source: https://docs.vlm.run/guides/doc-ai/guide-parsing-invoices
Extract structured data from invoices.
Navigate over to the invoice-parsing playground in our [playground](https://app.vlm.run/playground/document.invoice) to see the invoice parsing in action.
`vlm-1` can extract structured data from invoices, along with their [visual grounding](/guides/doc-ai/guide-visual-grounding) in PDF or image format. Here's a step-by-step guide on how to parse an invoice:
Here is a visualization of the parsed invoice along with the visual grounding that `vlm-1` can extract from an invoice. Notice that only the specific items requested in the schema are retrieved and visualized, unlike OCR which returns all text in the document with no context:
For higher-quality results, we recommend enabling [Visual Grounding](/guides/doc-ai/guide-visual-grounding) to help the model understand the invoice and extract more accurate information. See [High-Accuracy Parsing with Grounding](#high-accuracy-parsing-with-grounding) for more details.
## Parsing Invoices in 2 Steps
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import FileResponse
# Initialize the client
client = VLMRun(api_key="")
# Submit the invoice for parsing
response: PredictionResponse = client.document.generate(
file=Path(""),
domain="document.invoice",
batch=True,
)
print(f"Job submitted:\n {response.model_dump()}")
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
// Initialize the client
const client = new VlmRun({
apiKey: "",
});
// Submit the invoice for parsing
const response = await client.document.generate({
file: "",
domain: "document.invoice",
batch: true,
});
// Wait for the job to complete
const job = await client.predictions.wait(response.id);
console.log(job);
```
You should see a response like this:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Job submitted:
{
"id": "052cf2a8-2b84-45f5-a385-ccac2aae13bb",
"created_at": "2024-08-15T02:22:09.157788",
"response": null,
"status": "pending"
}
```
You can now wait for the job to complete by calling the `predictions.wait` method:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Wait for the job to complete
response: PredictionResponse = client.predictions.wait(
id=response.id,
timeout=120,
)
print(f"Job completed:\n {response.model_dump()}")
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Wait for the job to complete
const job = await client.predictions.wait(response.id);
console.log(job);
```
You should see a response like this:
```json [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"id": "052cf2a8-2b84-45f5-a385-ccac2aae13bb",
"created_at": "2024-08-15T02:22:09.157788",
"status": "completed",
"response": {
"currency": "USD",
"currency_metadata": {
"bboxes": [
{
"content": "$19,647.68",
"bbox": {
"xywh": [0.843, 0.611, 0.084, 0.014]
},
"page": 0
}
]
},
"customer": "Jane Smith",
"customer_billing_address": {
"city": "Mountain View",
"city_metadata": {
"bboxes": [
{
"content": "Mountain View, CA 94043",
"bbox": {
"xywh": [0.080, 0.194, 0.190, 0.014]
},
"page": 0
}
]
},
"country": null,
"country_metadata": null,
"postal_code": "94043",
"postal_code_metadata": {
"bboxes": [
{
"content": "Mountain View, CA 94043",
"bbox": {
"xywh": [0.080, 0.194, 0.190, 0.014]
},
"page": 0
}
]
},
}
...
"items": [...], // List of items in the invoice
...
"total": 19647.68,
"total_metadata": {
"bboxes": [
{
"content": "$19,647.68",
"bbox": {
"xywh": [...]
},
"page": 0
}
]
}
}
}
```
## High-Accuracy Parsing with Grounding
For higher-quality results, you can enable [Visual Grounding](/guides/doc-ai/guide-visual-grounding) to help the model understand the invoice and extract more accurate information. You can do this by setting the `config=GenerationConfig(grounding=True)` parameter when submitting the job (as shown below).
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client.types import GenerationConfig
# Enable grounding when submitting the job
response: PredictionResponse = client.document.generate(
file=Path(""),
domain="document.invoice",
batch=True,
config=GenerationConfig(grounding=True),
)
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Enable grounding when submitting the job
const response = await client.document.generate({
file: "",
domain: "document.invoice",
batch: true,
config: { grounding: true },
});
```
## Try our Document -> JSON API today
Head over to our [Document -> JSON](/api-reference/v1/post-document-generate) to start building your own document processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Providing Feedback
Source: https://docs.vlm.run/guides/feedback
Improve model performance through feedback collection and fine-tuning.
## Overview
The VLM Run Feedback API enables you to collect and submit feedback on model predictions to continuously improve accuracy and performance. This feedback data is essential for fine-tuning `vlm-1` models to better serve your specific use cases and domains.
## Why Feedback Matters
Providing feedback on model predictions serves several critical purposes:
* **Model Fine-tuning**: Feedback data is used to fine-tune models for improved accuracy on your specific data patterns
* **Performance Optimization**: Helps identify areas where the model needs improvement
* **Domain Adaptation**: Enables the model to better understand your industry-specific requirements
* **Quality Assurance**: Provides a mechanism to flag incorrect or problematic predictions
* **Evaluations**: Structured corrections (and optionally note-derived JSON) are the ground truth the platform compares against stored model outputs when you run [Evaluations](/platform/observe/evaluations)
After receiving a prediction, you can submit feedback to help improve future model performance.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
feedback_response = client.feedback.submit(
request_id="",
response={
"name": "John Doe",
"date_of_birth": "1955-01-01",
"email": "john@doe.com"
},
notes="The extraction was accurate and captured all key information"
)
print(f"Feedback submitted: {feedback_response.id}")
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
});
const feedbackResponse = await client.feedback.submit({
id: "",
response: {
name: "John Doe",
date_of_birth: "1955-01-01",
email: "john@doe.com"
},
notes: "The extraction was accurate and captured all key information"
});
console.log(`Feedback submitted: ${feedbackResponse.id}`);
```
You can retrieve all feedback associated with a specific prediction.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
feedback_list = client.feedback.get("")
for feedback in feedback_list:
print(f"Feedback ID: {feedback.id}")
print(f"Response: {feedback.response}")
print(f"Notes: {feedback.notes}")
```
```typescript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const feedbackList = await client.feedback.get("");
feedbackList.forEach(feedback => {
console.log(`Feedback ID: ${feedback.id}`);
console.log(`Response: ${JSON.stringify(feedback.response)}`);
console.log(`Notes: ${feedback.notes}`);
});
```
## Fine-tuning with Feedback
The feedback you provide is used to create fine-tuned models that perform better on your specific use cases. This process involves:
1. **Data Collection**: Feedback is aggregated across your organization
2. **Model Training**: Fine-tuned models are created using your feedback data
3. **Performance Improvement**: Updated models show improved accuracy on similar tasks
## Best Practices
* **Be Specific**: Provide detailed feedback about what was correct or incorrect
* **Use Structured Data**: Include ratings, categories, and specific metrics when possible
* **Add Context**: Use the notes field to explain your reasoning
* **Consistent Feedback**: Maintain consistent criteria across your team for better model training
Fine-tuning capabilities are currently only available for our enterprise-tier customers.
If you are interested in using this feature, please [contact us](mailto:support@vlm.run).
# Cataloging Images
Source: https://docs.vlm.run/guides/image-ai/guide-cataloging-images
Learn how to generate captions, tags and descriptions for images.
While most traditional computer-vision models are specialized for specific tasks like image classification, captioning or tagging, VLM-1 can be used to simultaneously generate a wide range of structured outputs from images. This includes generating captions, tags, descriptions, and other structured data that can be used for cataloging, search, retrieval, and other applications.
## Cataloging Product Images
Let's look at a product cataloging example to see how `vlm-1` can be used to generate structured data from images. In this example, we'll use `vlm-1` to generate captions, tags, and descriptions for a set of images of different products. This structured data can then be used to create a product catalog that can be searched, filtered, and analyzed in various ways.
For this example, we're going to use a small fashion dataset [`ashraq/fashion-product-images-small`](https://huggingface.co/datasets/ashraq/fashion-product-images-small)
### 1. Define a custom schema for cataloging
In the sections below, we'll showcase a few notable features of the API for image cataloging. `vlm-1` can automatically generate descriptions for products based on the images provided. This can be useful for creating detailed product listings, search results, or other content that requires structured descriptions of products. First let's create a custom schema that will be used to generate the descriptions.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from typing import Literal
from pydantic import BaseModel, Field
class ProductCatalog(BaseModel):
description: str = Field(..., description="A 2-sentence general visual description of the product embedded as an image.")
category: str = Field(..., description="One or two-word category of the product (i.e, Apparel, Accessories, Footwear etc).")
season: Literal["Fall", "Spring", "Summer", "Winter"] = Field(..., description="The season the product is intended for.")
gender: Literal["Men", "Women", "Kids"] = Field(..., description="Gender or audience the product is intended for.")
```
### 2. Extract cataloging information from images
Once you have defined your custom schema, you can use **`vlm-1`** to extract product cataloging information directly from images that conform to this schema. The extracted data will be validated against the schema you defined, ensuring that it conforms to the expected structure and types.
We support querying the API via RESTful endpoints, or using the OpenAI Python SDK with our [OpenAI-Compatible API](/agents/integrations/integrations-openai-compatibility).
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from PIL import Image
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, GenerationConfig
# Initialize the client
client = VLMRun(api_key="")
# Load the first image from the dataset and encode it as base64
ds = load_dataset("ashraq/fashion-product-images-small", split="train[:1%]")
image: Image.Image = next(ds["image"])
# Predict the product catalog information from the image
response: PredictionResponse = client.image.generate(
images=[image],
domain="retail.product-catalog",
config=GenerationConfig(
json_schema=ProductCatalog.model_json_schema(),
)
)
response_dict = response.model_dump()
```
## Example Product Cataloging Prediction
Let's take a look at the sample output from the API for the first image of a navy plaid shirt in the product catalog. The API is able to generate a detailed description of the product, including the category, season, and gender it is intended for. This structured data can be used to create a product listing or search results for the product.
```JSON theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"description": "A casual, button-up plaid shirt with short sleeves in a light fabric. The shirt features a combination of blue and white colors in a checkered pattern.",
"category": "Apparel",
"season": "Summer",
"gender": "Men"
}
```
Let's breakdown the output into their respective tasks:
* **Description** (`Captioning` or `Description Generation`): Here, the API has generated a detailed description of the product, including the type of shirt, its features, and the colors and patterns it has. This can be useful for creating detailed product listings or search results for the product. This is a typical use-case for the `Captioning` or `Description Generation` task.
* **Category** (`Classification` or `Tagging`): The API has also identified the category of the product as "Apparel". This can be useful for categorizing products in a catalog or search results. This is a typical use-case for the `Classification` or `Tagging` task.
* **Season** (`Classification` or `Tagging`): The API has identified the season the product is intended for as "Summer". This can be useful for filtering products by season or for creating seasonal collections. This is a typical use-case for the `Classification` or `Tagging` task, however, the one additional feature is that we have a `Literal` type that restricts the possible values to a predefined set.
* **Gender** (`Classification` or `Tagging`): The API has identified the gender the product is intended for as "Men". This can be useful for filtering products by gender or audience. This is similar to the `Season` task, but with a different set of possible values.
## Cataloging larger image catalogs
Once you have validated the output for a single image, you can scale this process to catalog larger volumes of images. You can use the same API call to generate structured data for multiple images, and then use this structured data to create a product catalog that can be searched, filtered, and analyzed in various ways. Better yet, you can also ingest the JSON directly into JSON-compatible databases like MongoDB, Elasticsearch, or even traditional SQL databases for searching over these images unlocking a wide range of semantic image-search and querying possibilities for your cataloging needs.
## Fine-tuning for custom cataloging
For enterprise use-cases where you need to fine-tune the model for **custom-tailored cataloging** tasks and **improved accuracy**, you can use our [fine-tuning guides](/guides/fine-tune) to customize the model performance and scalability needs. This can include fine-tuning the model on your own data, customizing the model architecture, or adding new capabilities to the model. Fine-tuning can help you improve the accuracy and performance of the model for your specific cataloging tasks, and also help you scale the model to handle larger volumes of images with more efficient, lightweight fine-tuned models that are optimized for your specific use-case.
This feature is currently only available for our enterprise-tier customers. If you are interested in using this feature, please [contact us](mailto:support@vlm.run).
## Try our Image -> JSON API today
Head over to our [Image -> JSON](/api-reference/v1/post-image-generate) to start building your own document processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Classifying Images
Source: https://docs.vlm.run/guides/image-ai/guide-classifying-images
Learn how to classify images into categories like animals, landscapes, and objects using AI.
While traditional image processing systems often rely on simple feature detection or rule-based approaches, `vlm-1` can intelligently classify images based on their content, composition, and visual characteristics. This enables robust classification of images into various categories, even when they come in different styles, lighting conditions, or perspectives.
For example, below is a diagram showing how an image can be classified into different types, and how each type can have its own custom post-processing logic.
```mermaid theme={"theme":{"light":"github-light","dark":"dark-plus"}}
flowchart TD
A([Image]) --> B{Classify}
B --> C1([News])
B --> C2([Entertainment])
B --> C3([Advertising])
B --> C4([Other])
style A fill:#eee,stroke:#333,stroke-width:1px
style C1 fill:#fff,stroke:#333,stroke-width:1px
style C2 fill:#fff,stroke:#333,stroke-width:1px
style C3 fill:#fff,stroke:#333,stroke-width:1px
style C4 fill:#fff,stroke:#333,stroke-width:1px
```
## Classifying TV Images
Let's look at a TV image classification example to see how `vlm-1` can be used to automatically analyze and categorize television content. In this example, we'll use `vlm-1` to classify TV screenshots and frames into categories like news broadcasts, entertainment shows, commercials, and other programming types. This classification enables automated content monitoring, ad detection, and intelligent media archiving by identifying the type of TV content being shown.
### Define a custom schema for image classification
In the sections below, we'll showcase how to use the API for image classification. `vlm-1` can automatically classify images based on their content and visual characteristics, providing both a classification and a rationale for its decision. First, let's create a custom schema that will be used to classify the images.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from typing import Literal
from pydantic import BaseModel, Field
class ImageClassification(BaseModel):
rationale: str = Field(..., description="A rationale for the classification, based on the visual content and features of the image. Keep it short and concise, yet detailed enough to justify the classification.")
image_type: Literal["news", "entertainment", "advertising", "other"] = Field(..., description="The type of image being processed")
confidence: Literal["hi", "med", "lo"] = Field(..., description="Confidence score for the classification, based on the rationale provided and the visual features of the image. For ambiguous images, the confidence score should be `lo`.")
```
### Classify images
Once you have defined your custom schema, you can use **`vlm-1`** to classify images according to this schema. The classification will be validated against the schema you defined, ensuring that it conforms to the expected structure and types. First, let's look at an example of how to classify a single image.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, GenerationConfig
# Initialize the client
client = VLMRun(api_key="")
# Classify a single image
path = Path("path/to/image.jpg")
prediction: PredictionResponse = client.image.generate(
file=path,
domain="image.classification",
config=GenerationConfig(response_model=ImageClassification)
)
response_dict = prediction.response.model_dump()
print(response_dict)
```
### Sample Image Classification
Let's take a look at the sample output for a typical animal image.
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"rationale": "The image contains financial market data and a news presenter from Bloomberg News, indicating a broadcast of financial news. The financial indices are highlighted, and stock performance is shown, which is typical for a news segment on economic updates.",
"image_type": "news",
"confidence": "hi"
}
```
Let's breakdown the output into their respective components:
* **`rationale`**: A detailed explanation of why it classified the image as a news, based on visual features and content. This allows the developer or user to introspect on the classification and make any necessary adjustments downstream to the model.
* **`image_type`**: The correct image classification type, in this case `news`.
* **`confidence`**: A qualitative confidence level of "high", indicating strong certainty in the classification based on the clear presence of financial market data and a news presenter.
## Fine-tuning Image Classification
This feature is currently only available for our enterprise-tier customers. If you are interested in using this feature, please [contact us](mailto:support@vlm.run).
For **enterprise use-cases** where you need to fine-tune the model for **custom image types** and **improved accuracy**, you can use our [fine-tuning guides](/guides/fine-tune) to customize the model performance and scalability needs. This can include fine-tuning the model on your own image collections, customizing the classification schema, or adding new image types to the classification system. Fine-tuning can help you improve the accuracy and performance of the model for your specific image types, and also help you scale the model to handle larger volumes of images with more efficient, lightweight fine-tuned models that are optimized for your specific use-case. Contact us at [support@vlm.run](mailto:support@vlm.run) to learn more about how we can help you with your fine-tuning needs.
## Try our Image -> JSON API today
Head over to our [Image -> JSON](/api-reference/v1/post-image-generate) to start building your own document processing pipeline with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Best Practices
Source: https://docs.vlm.run/guides/schema/schema-best-practices
Best practices for designing schemas for visual inputs.
Defining a clear and effective schema is crucial for getting the most accurate and useful information from visual inputs. A well-designed schema acts as a precise instruction set, guiding the model to extract exactly what you need in a structured format. Following best practices ensures your schemas are robust, maintainable, and yield high-quality results.
This guide outlines key principles and techniques for crafting good, robust and maintainable schemas using [Pydantic](https://docs.pydantic.dev/latest/). If you are interested in contributing to this guide (especially around the usage of [Zod](https://zod.dev/)), please reach out to us on [Discord](https://discord.com/invite/AMApC2UzVY) or [email](mailto:support@vlm.run?subject=Schema%20Request).
## Best Practices
1. **Keep schemas focused**: Define schemas that extract only the information you need.
2. **Use validation rules**: Leverage Pydantic's validation capabilities to ensure data integrity.
3. **Create reusable components**: Break down complex schemas into smaller, reusable models.
4. **Document your fields**: Use the `Field` class with descriptive titles to improve extraction quality.
5. **Test with diverse inputs**: Validate your schemas against a variety of visual inputs to ensure robustness.
## Mapping Schemas to Task Primitives
* **Classification**: Use `Literal` to constrain your field values to a set of possible categories.
* **Captioning**: Use `str` to extract a textual description of the image. Provide some additional details regarding the style, context and also provide a rough estimate of the length of the caption (in number of words).
* **Date Parsing**: Use `datetime.date` to extract a date from the image. Provide some additional details regarding the format of the date (e.g. `YYYY-MM-DD`). One additional caveat is that you can not provide `date` as the field name in your Pydantic BaseModel, as it is a reserved keyword in Pydantic. Use `datetime.datetime` instead if you need to extract additional time information (e.g. `YYYY-MM-DD HH:MM:SS`). Otherwise, always use `datetime.date` for date parsing.
# MarkdownPage
Source: https://docs.vlm.run/guides/schema/schema-markdown-page
A visual guide to the MarkdownPage schema used for document extraction and processing.
The `MarkdownDocument` schema is the cornerstone of VLM Run's document processing system, providing a standardized, machine-readable representation of complex documents. This technical reference guide details the schema's architecture, components, and implementation patterns.
## `MarkdownDocument` Data Model
The `MarkdownDocument` schema addresses the fundamental challenges in document processing:
1. **Structural Preservation**: Maintains document hierarchy and relationships
2. **Content Extraction**: Handles mixed content types (text, tables, figures, code)
3. **Spatial Understanding**: Preserves layout and positioning information
4. **Data Integrity**: Ensures accurate representation of structured elements
5. **Extensibility**: Supports custom annotations and metadata
### 1. `MarkdownPage`
A `MarkdownDocument` is a list of `MarkdownPage` objects, each representing a page in the document.
```mermaid theme={"theme":{"light":"github-light","dark":"dark-plus"}}
classDiagram
class MarkdownPage {
PageMetadata metadata
List~Table~ tables
List~Figure~ figures
String content
}
class PageMetadata {
+String language
+Integer page_number
}
class Table {
TableMetadata metadata
List~TableHeader~ headers
List~TableRowDict~ data
BoxCoords bbox
}
class Figure {
String id
String title
String caption
BoxCoords bbox
}
MarkdownPage "1" *-- "1" PageMetadata : has
MarkdownPage "1" *-- "*" Table : has
MarkdownPage "1" *-- "*" Figure : has
%% Add rounded edges styling
classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px,rx:10,ry:10
classDef relationship fill:none,stroke:#666,stroke-width:1px
```
Here's an alternative way to visualize the `MarkdownPage` schema:
| Component | Field | Type | Description |
| -------------------- | ------------------ | -------------------- | --------------------------------------- |
| **MarkdownDocument** | | | |
| | `pages` | `List[MarkdownPage]` | Pages in the document |
| **MarkdownPage** | | | |
| | `metadata` | `PageMetadata` | Metadata of the page |
| | `tables` | `List[Table]` | Tables in the page |
| | `figures` | `List[Figure]` | Figures in the page |
| | `content` | `str` | Content of the page |
| **PageMetadata** | | | |
| | `language` | `str` | Language of the document |
| | `page_number` | `int` | Page number of the document (0-indexed) |
| **Table** | | | |
| | `metadata.title` | `str` | Title of the table |
| | `metadata.caption` | `str` | Caption of the table |
| | `metadata.notes` | `str` | Notes about the table |
| | `headers.id` | `str` | Unique identifier for the header |
| | `headers.column` | `int` | Column index of the header |
| | `headers.name` | `str` | Name of the header |
| | `headers.dtype` | `str` | Data type of the header |
| | `data.*` | `dict[str, Any]` | Maps column header ids to values |
| | `bbox` | `BoxCoords` | Bounding box of the table |
| **Figure** | | | |
| | `id` | `int` | Unique identifier for the figure |
| | `title` | `str` | Title of the figure |
| | `caption` | `str` | Caption of the figure |
| | `bbox` | `BoxCoords` | Bounding box of the figure |
***
### 2. `MarkdownTable`
Tables are represented with a `
` tag in the markdown content, with the actual table content stored in the `tables` list. This allows for rich representation of table's data while maintaining the document's flow.
```mermaid theme={"theme":{"light":"github-light","dark":"dark-plus"}}
classDiagram
class Table {
TableMetadata metadata
List~TableHeader~ headers
List~TableRowDict~ data
BoxCoords bbox
}
class TableMetadata {
String title
String caption
String notes
}
class TableHeader {
String id
Integer column
String name
String dtype
}
class TableRowDict {
String id
Any value
}
class BoxCoords {
List~float~ xywh
}
Table "1" *-- "1" TableMetadata: has
Table "1" *-- "*" TableHeader: has
Table "1" *-- "*" TableRowDict: has
Table "1" *-- "1" BoxCoords: has
%% Add rounded edges styling
classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px,rx:10,ry:10
classDef relationship fill:none,stroke:#666,stroke-width:1px
```
### 3. Charts and Figures
Charts and figures are represented with a `` tag in the content. The chart details are stored in the `figures` list, including properties like:
## Example Usage
Here's an example of how the `MarkdownPage` model is used to process a document:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, MarkdownDocument
# Initialize client
client = VLMRun(api_key="")
# Process document
response: PredictionResponse = client.document.generate(
file=Path("document.pdf"),
domain="document.markdown",
batch=True,
)
# Access processed document
doc: MarkdownDocument = client.predictions.wait(response.id, timeout=120)
print(doc.model_dump_json(indent=2))
```
## Example JSON Response
Here's an example of how the MarkdownPage schema appears in a JSON response:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"pages": [
{ // page 0
"metadata": {
"page_number": 0
},
"tables": [
{
"metadata": {
"title": "Sample Data Table",
"caption": "Table showing example data"
},
"content": "| Header 1 | Header 2 |\n|----------|----------|\n| Data 1 | Data 2 |\n| Data 3 | Data 4 |",
"headers": [
{
"id": "h1",
"column": 0,
"name": "Header 1",
"dtype": "string"
},
...
],
"data": [
{
"h1": "Data 1",
"h2": "Data 2"
},
...
]
}
],
"figures": [
{
"id": 0,
"title": "Sample Bar Chart",
"caption": "Example visualization",
"content": "..."
}
...
],
"content": "..."
},
{ // page 1
...
},
{ // page 2
...
},
...
]
}
```
# Transcribing Video
Source: https://docs.vlm.run/guides/video-ai/guide-video-transcription
Learn how to transcribe and analyze hours-long video content using our Video Transcription API.
Navigate over to the video-transcription playground in our [playground](https://app.vlm.run/playground/video.transcription) to see the video transcription in action.
Developers today need more than just basic speech-to-text—they need to unlock the full potential of long-form video. With `vlm-1`, you can transcribe hours of video and extract deep, structured insights in a single API call: from scene changes and chapter segmentation to temporal grounding and visual context. This empowers you to build smarter search, content discovery, and analytics tools for podcasts, lectures, interviews, and more—at scale, with minimal effort.
## Analyzing Video Content
Let's look at a video analysis example to see how `vlm-1` can be used to extract structured insights from video content. In this example, we'll use `vlm-1` to transcribe and analyze the **full 1-hour and 40 minute** [Google Cloud Next 25 Opening Keynote](https://www.youtube.com/watch?v=Md4Fs-Zc3tg), generating segmented chapters with start and end timestamps, visual scene descriptions, and corresponding full transcript.
Let's look at the few lines of code to transcribe the video with `vlm-1`:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, GenerationConfig
# Initialize the client
client = VLMRun(api_key="")
# Submit the video file for transcription
prediction: PredictionResponse = client.video.generate(
file=Path("path/to/video.mp4"),
domain="video.transcription",
batch=True,
)
# Wait for the prediction to complete (with a timeout of 600 seconds)
prediction: PredictionResponse = client.predictions.wait(id=prediction.id, timeout=600)
print(prediction.response.model_dump())
```
## Understanding the Output
Here's an example of the output in JSON format, for the entire 8 minute video:
```json Example Video Transcription [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"metadata": {
"content": null,
"topics": null,
"duration": 6004.006893
},
"segments": [
{
"start_time": 0,
"end_time": 22,
"audio": {
"content": ""
},
"video": {
"content": "The video begins with a white screen, which then transitions to a vibrant and colorful scene featuring the text \"Vtex.\" The letters are rendered in a playful, 3D style with a gradient effect, transitioning from blue to red. Surrounding the text are various abstract shapes and forms, including spheres, rings, and other geometric elements in bright colors like red, blue, green, and yellow. These shapes are floating and moving around the text, creating a dynamic and lively atmosphere."
}
},
{
"start_time": 22,
"end_time": 75.33,
"audio": {
"content": " you know and you know and you're I'm I'm The Thank you. Why not?"
},
"video": {
"content": "A person is toasting bread in a yellow toaster. The camera focuses on the bread as it cooks, showing the golden-brown color forming on the surface. The scene then transitions to a close-up of a slot machine, where the number 11 appears on the screen. The next scene shows a person wakeboarding, performing a trick over a wave. The wakeboarder is wearing a black wetsuit and a helmet. The final scene depicts a bright explosion in space, with colorful particles and light beams radiating outward."
}
},
{
"start_time": 76.33,
"end_time": 96.33,
"audio": {
"content": " Just two words, but those two words challenge everything and can change anything. Why not help find a cure? Bring it here and even there. Wait, really? Yes, really."
},
"video": {
"content": "The video begins with two individuals in a sterile environment, likely a laboratory or medical facility. They are wearing protective gear, including hairnets, masks, and lab coats, indicating a controlled and clean workspace. The setting is well-lit with fluorescent lighting, and the background shows various pieces of equipment and machinery typical of such environments."
}
},
{
"start_time": 96.33,
"end_time": 120.33,
"audio": {
"content": " We're building the most helpful AI. So you can turn an idea into an enterprise. Get the right crops into this box and breakfast on the table. Inspect 800. and breakfast on the table. Inspect 800,000 packages a day and help protect our power grids. Because once we turn this into that, we ask, what else can we do?"
},
"video": {
"content": "The video begins with a view from space, showing a vast expanse of Earth below. A large, cylindrical object is seen floating near the surface of the planet. The scene then transitions to an indoor setting where two individuals wearing hairnets and gloves are pushing a cart loaded with yellow containers through a sterile environment. The next scene shifts to a highway where a red truck is driving on the road. The truck suddenly collides with a large, colorful object, causing it to burst into flames. The final scene features a close-up of a glowing, spinning object, possibly a particle accelerator or a similar scientific device."
}
},
{
"start_time": 120.33,
"end_time": 142,
"audio": {
"content": " Find out where the wild things are? Uh, wilder. Spot patterns and crime data? Catch fishing attacks. Take a thousand customer service calls an hour. Help coders, well, code. Let's make it happen. I'm"
},
"video": {
"content": "The video begins with a vibrant and surreal scene featuring a watermelon that appears to be floating in space. The watermelon is depicted with a shiny, reflective surface, giving it an almost otherworldly appearance. A rainbow arcs across the sky, adding to the dreamlike quality of the scene. The watermelon is shown in various stages of being sliced, with pieces falling away, revealing its juicy interior. The background features a gradient of colors, transitioning from blue to pink, enhancing the fantastical atmosphere."
}
},
{
"start_time": 142,
"end_time": 171.67,
"audio": {
"content": ""
},
"video": {
"content": "A baby is seen smiling and laughing while being held up by an adult. The baby's hands are raised in the air, and the adult's hands are visible holding the baby securely. The baby appears to be enjoying the moment, with a joyful expression on its face."
}
},
{
"start_time": 142,
"end_time": 198.33,
"audio": {
"content": " Ha ha ha ha ha. Please welcome CEO of Google Cloud, Thomas Currian. Thank you. Wow. Hello, everyone. Welcome to Google Cloud Next. Hello everyone. Welcome to Google Cloud Next. Just one year ago, we stood here and talked about the future of AI for organizations. Today, that future is being built by all of us. In 2024, we shipped more than 3,000 product advances across Google Cloud and Workspace."
},
"video": {
"content": "A baby is seen smiling and laughing while being held up by an adult. The baby's hands are raised in the air, and the adult's hands are visible holding the baby securely. The baby appears to be enjoying the moment, with its mouth open and eyes wide with joy."
}
},
{
"start_time": 198.33,
"end_time": 224,
"audio": {
"content": " We expanded Google Cloud to 42 regions, including Sweden, Mexico, and South Africa, and are rapidly expanding to countries like Malaysia, Thailand, and Kuwait. We expanded our 2 million mile, terrestrial and subsea fiber network by announcing new subsea cables like Umoja, Bosen, and Proa."
},
"video": {
"content": "A man in a dark blue suit stands on a stage, gesturing with his hands as he speaks. The background is a simple, light-colored curtain. The scene then transitions to a large screen displaying a world map with various locations marked by blue dots. The text on the screen reads \"Expanded infrastructure footprint to 42 regions.\" The map highlights several specific regions such as Mexico, Sweden, South Africa, Kuwait, Thailand, Malaysia, and others. The presentation continues with another slide showing a sunset over the ocean, with the text \"2 million miles of terrestrial and subsea cables.\" The man continues to speak, emphasizing the information displayed on the screen."
}
},
{
"start_time": 224,
"end_time": 250.67,
"audio": {
"content": " Google's AI momentum is exciting. We're seeing more than 4 million developers using Gemini, a 20 times increase in Vertex AI usage last year, driven by the strong adoption of Gemini Flash, Gemini 2.0, Imagine 3.0, and most recently, VO, our advanced video generation model,"
},
"video": {
"content": "A man in a dark blue suit stands on a stage, gesturing with his hands as he speaks. The background is a gradient of light blue to white, with vertical lines creating a modern and professional atmosphere. The scene transitions to a large screen displaying the text \"4 million+ developers use Gemini.\" The man continues to speak, and the camera shifts to show him from different angles, emphasizing his gestures and expressions. The final shot shows the man standing confidently on the stage, with the screen behind him displaying the words \"Gemini Imagen Veo.\""
}
},
{
"start_time": 250.67,
"end_time": 275.33,
"audio": {
"content": " and over 2 billion AI assist monthly to business users right within Google Workspace. But even more exciting is the momentum with you are customers. Here next, we'll be sharing over 500 customer stories showcasing real business innovation impact from AI adoption."
},
"video": {
"content": "A man in a dark suit stands on a stage with a blue curtain backdrop. He is speaking to an audience, gesturing with his hands as he presents information. The scene transitions to a large screen displaying text about Google Workspace's AI assists, followed by a slide showing customer stories from various companies. The man continues to speak, emphasizing the benefits of Google Workspace's AI capabilities."
}
},
{
"start_time": 277.51,
"end_time": 320.33,
"audio": {
"content": " Google is building for a unique moment. We're investing in the technology and the ecosystem to power your growth and transformation. Let's hear more from a special guest, a warm welcome for the CEO of Google and Alphabet, Sundaphi. Thank you. Thank you, Thomas. Good to be with you all here in Vegas. Last year, I joked in my remarks about how I was auditioning for the sphere."
},
"video": {
"content": "A man in a dark blue suit stands on a stage, speaking to an audience. He is wearing a white dress shirt and a pocket square. The background features vertical light panels that change colors from white to blue. The man gestures with his hands as he speaks, occasionally adjusting his posture. The lighting highlights his presence, creating a professional and formal atmosphere."
}
},
{
"start_time": 320.33,
"end_time": 342,
"audio": {
"content": " Well, it turns out I got the gig. Last night, I was on stage at the sphere to share a new collaboration. We are introducing the visit of us to a new generation using Google AI, transforming one of the greatest films of all time for one of the largest screens in the world."
},
"video": {
"content": "A man in a blue blazer and white shirt stands on a stage, gesturing with his hands as he speaks. The background is a simple, light-colored curtain. The scene then transitions to a large screen displaying various images. The first image shows an empty stadium with rows of seats and a stage at the front. The next image features a couple in formal attire, possibly at a wedding, with the woman in a white dress and the man in a suit. The third image depicts a boat on a river surrounded by lush greenery. The final image shows a close-up of a hand holding a small object."
}
},
{
"start_time": 342,
"end_time": 365.82,
"audio": {
"content": " It's a huge ongoing effort and not something we could have attempted even 18 months ago. Shows how rapidly technology is evolving and how it can enable us to rethink what's possible. I think that's a fitting theme for Cloud Next. The chance to improve lives and reimagine things is why Google has been investing in AI for"
},
"video": {
"content": "A man in a blue blazer stands on a stage, addressing an audience. The stage is well-lit with a modern design featuring vertical light panels. Behind him, a large screen displays various images: first, two men standing in front of a whiteboard filled with diagrams and notes; then, a close-up of a green, furry creature with a surprised expression; followed by a scene from a movie with characters walking through a tunnel; finally, a group of people in futuristic attire walking down a corridor."
}
},
{
"start_time": 365.82,
"end_time": 388.04,
"audio": {
"content": " more than a decade. We see it as the most important way we can advance our mission, to organize the world's information and make it universally accessible and useful. With Google Cloud, we see AI as the most important way we can help advance your mission. The opportunity with AI is as big as it gets."
},
"video": {
"content": "A speaker stands on a stage at an event, addressing an audience. The stage is modern and sleek, with a large screen displaying various images and videos. The speaker gestures with his hands as he speaks, emphasizing points. The audience is seated in darkness, focused on the speaker. The screen behind the speaker shows different scenes, including close-ups of hands working, a person in a space suit, and two individuals engaged in an activity, possibly related to agriculture."
}
},
{
"start_time": 388.66,
"end_time": 408.66,
"audio": {
"content": " That's why we are investing in the full stack of AI innovation. Starting with the infrastructure that powers it all. We are making big investments now and for the future. In 2025, and for the future. In 2025, we plan to invest around $75 billion in total CAPEX. This investment will be in total CAPEX."
},
"video": {
"content": "A speaker stands on a stage, presenting to an audience. The stage is illuminated with blue lighting, creating a modern and professional atmosphere. The speaker is dressed in a dark suit and white shirt, gesturing with his hands as he speaks. Behind him, a large screen displays various images and text related to AI products and platforms. The screen transitions through different slides, highlighting different aspects of AI technology, such as products and platforms, models and tooling, world-class research, and AI infrastructure."
}
},
{
"start_time": 408.66,
"end_time": 430.66,
"audio": {
"content": " This investment will be directed towards our servers and data centers, which includes powering our AI compute and cloud business. So this will greatly benefit our customers like all of you. We need our infrastructure to move at Google speed with near-zero latency, supporting services like search, Gmail and photos"
},
"video": {
"content": "A man in a blue blazer and white shirt stands on a stage, gesturing with his hands as he speaks. The background is a gradient of blue shades, and there is a large screen behind him displaying a graph titled \"Alphabet Capital Expenditure\". The graph shows a line indicating an increase in expenditure from 2020 to 2025, with the approximate value of $75 billion marked at the end. The scene then transitions to a large screen showing an aerial view of a city with various buildings and infrastructure."
}
},
{
"start_time": 430.66,
"end_time": 452.66,
"audio": {
"content": " for billions of users worldwide. And we use it for training our most capable model, Gemina. Google's backbone network is unparalleled, as Thomas just mentioned, spanning more than 200 countries and territories powered by over 2 million miles of fiber. Today, I'm pleased to announce that we are making"
},
"video": {
"content": "A man stands on a stage, dressed in a blue blazer over a white shirt and black pants, with a belt. He gestures with his hands as he speaks, indicating he is delivering a presentation. The background is a gradient of blue shades, creating a calm and professional atmosphere. The lighting focuses on him, highlighting his presence against the darker backdrop."
}
},
{
"start_time": 452.66,
"end_time": 480.05,
"audio": {
"content": " Google's global private network available to enterprises around the world. We call it cloud wide area network or van. Cloud van leverages Google's planet scale network. It's optimized for application performance and delivers over 40% faster performance while reducing total"
},
"video": {
"content": "A man in a blue blazer and white shirt stands on a stage, gesturing as he speaks. The background features a large screen displaying an image of a network of lights and lines, symbolizing a global network. The text \"New Cloud Wide Area Network\" appears on the screen, indicating the introduction of a new service. The man continues to speak, emphasizing the importance of this new network for businesses."
}
},
{
"start_time": 480.05,
"end_time": 507,
"audio": {
"content": " cost of ownership by up to 40%. Companies like Citadel Securities and Nestle are already using this network for faster, more reliable solutions, and it will be available to all Google Cloud customers later this month. This builds on our legacy of opening up our technical infrastructure for others to use. We do this with our custom AI chips called tensor processing units or TPUs."
},
"video": {
"content": "A man in a blue blazer and white shirt stands on a stage, addressing an audience. He gestures with his hands as he speaks, emphasizing points about the Cloud Wide Area Network. The background is a simple, light-colored curtain, and the lighting focuses on him, highlighting his presence. The scene transitions to a large screen displaying text about the Cloud Wide Area Network's performance compared to the public internet, showing a 40% improvement. The video then cuts back to the speaker, who continues his presentation."
}
},
{
"start_time": 507,
"end_time": 533.48,
"audio": {
"content": " Since 2013, we've invested heavily in this specialized hardware and we continue to make massive improvements in performance and efficiency at scale. Today I'm proud to announce our seven-generation TPU,"
},
"video": {
"content": "A man in a blue blazer and white shirt stands on a stage, speaking to an audience. He gestures with his hands as he talks. The scene then cuts to a close-up of a TPU v5p chip on a circuit board. The video returns to the man on stage, who continues to speak. The final shot shows a new Ironwood device displayed on a screen, with the text \"New Ironwood\" and \"Coming Soon\" visible."
}
},
{
"start_time": 534.2,
"end_time": 558.33,
"audio": {
"content": " Ironwood achieves 3,600 times better performance, an incredible increase. It's the most powerful chip we have ever built and will enable the next frontier of AI models. In the same period, we've also become 29x more energy efficient, and Amin will share more later today."
},
"video": {
"content": "A man in a blue blazer stands on a stage, presenting information about Google TPU progress. The background features a large screen displaying a bar graph titled \"Google TPU Progress: Performance by Generation.\" The graph shows performance metrics (in exaFLOPS) for different years: 2018 (0.01), 2020 (0.13), 2022 (1.13), and 2023 (4.11). The presenter gestures towards the screen as he explains the data, emphasizing the significant increase in performance from 2023 to 2025. The final frame highlights the projected performance for 2025, showing a dramatic rise to 42.53 exaFLOPS."
}
},
{
"start_time": 558.33,
"end_time": 580.33,
"audio": {
"content": " This progress is laying the foundation for breakthroughs across multiple fields. Quantum computing is a great example. Our newest quantum chip willow cracked a key challenge in quantum error correction that has eluded researchers for three decades. It can reduce errors exponentially as we scale up using more cubits."
},
"video": {
"content": "A speaker stands on a stage, presenting information about advanced technology. The background features large screens displaying various images and text related to quantum computing and AI. The first screen shows a chip labeled \"Ironwood\" with a power efficiency rating of 29x. The second screen displays \"Quantum AI\" with an image of a futuristic, interconnected network. The third screen highlights a \"State-of-the-art quantum chip\" with 105 qubits and mentions it as a benchmark for quantum error correction and random CI. The speaker gestures towards the screens, emphasizing the details being presented."
}
},
{
"start_time": 580.33,
"end_time": 606,
"audio": {
"content": " The willow chip really paves the way for a useful large-scale quantum computer down the row. Our infrastructure enables the next layer of the stack, research and model. Over the last decade, our research teams have pushed the boundaries of AI forward. And today, they are accelerating science and discovery. From our alpha alpha fold breakthrough with protein folding to weather next our"
},
"video": {
"content": "A man in a blue suit stands on a stage, presenting information about a quantum chip called Willow. The screen behind him displays details about the chip, including its state-of-the-art status, 105 qubits, and performance metrics. The presenter gestures with his hands as he speaks, emphasizing the advancements and capabilities of the chip."
}
},
{
"start_time": 606,
"end_time": 630.66,
"audio": {
"content": " state-of-the-art weather forecasting models. World-class research is what enables us to push the frontier with our Gemini models. In December, we introduced Gemini 2.0 with new advances in multimodality, like native image and audio output as well as native tool use. This new generation has also pushed the frontiers of another capability called thinking."
},
"video": {
"content": "A man in a blue blazer and white shirt stands on a stage, gesturing as he speaks. The background features a large screen displaying vibrant, colorful visuals, possibly related to technology or innovation. The scene transitions to a close-up of the man, emphasizing his gestures and expressions. The video then shifts to a wide shot of the stage, highlighting the \"Gemini 2.0\" logo on the screen behind him. The audience is visible in the foreground, attentively watching the presentation."
}
},
{
"start_time": 630.66,
"end_time": 654.66,
"audio": {
"content": " A couple weeks ago we released a new model, Gemini 2.5, a thinking model that can reason through its thoughts before responding. It's our most intelligent AI model ever and it's the best model in the world according to the chatbot arena leadable."
},
"video": {
"content": "A man stands on a stage, delivering a presentation. The background is dark with blue lighting accents, and the word \"Thinking\" is prominently displayed on a large screen behind him. The man is dressed in a blue blazer over a white shirt and black pants, with a belt. He gestures with his hands as he speaks, occasionally clapping them together."
}
},
{
"start_time": 659.33,
"end_time": 675.67,
"audio": {
"content": " It's state of the art across a range of benchmarks requiring advanced reasoning that included the highest score ever on humanity's last exam, one of the hardest industry benchmarks that's designed to capture the human frontier of knowledge and reasoning. There's a lot of impressive words, but let me show you what it can do."
},
"video": {
"content": "A man in a blue blazer and white shirt stands on a stage, gesturing as he speaks. The background is a gradient of blue hues, and the stage features a large screen displaying the text \"Gemini 2.5 Pro\" along with various statistics and comparisons. The man appears to be presenting or explaining something related to the content displayed on the screen."
}
},
{
"start_time": 675.67,
"end_time": 697.33,
"audio": {
"content": " Take a look at this Rubik's Cube, quoted by developer Matt Berman. You might think of it as a toy, but it's actually a really complex reasoning challenge. Adjustable dimensions, scrambling the squares, keyboard controls, and Gemini 2.5 Pro can simulate it all. It's a significant leap and shows the ability to produce robust interactive core."
},
"video": {
"content": "A man in a blue blazer and white shirt stands on a stage, gesturing with his hands as he speaks. The background is a gradient of blue shades. The scene transitions to a large screen displaying a Rubik's Cube. The cube rotates slowly, showing different colored faces. The man continues to speak, occasionally gesturing towards the screen."
}
},
{
"start_time": 697.33,
"end_time": 720.33,
"audio": {
"content": " That's a fun one. Let's look at one other example. With a series of prompts, developer John Modern used 2.5 Pro, to create a series of physics simulations, like the Earth's magnetic field and general relativity. You can see how the model turns really complex concepts into stunning and interactive visuals."
},
"video": {
"content": "A man stands on a stage, gesturing with his hands as he speaks. The background features a large screen displaying a 3D cube made up of smaller cubes, alternating between blue and red. The scene transitions to a mesmerizing display of a cosmic event, showing a bright, glowing sphere surrounded by a swirling pattern of light and particles. The man continues to speak, emphasizing his points with hand movements."
}
},
{
"start_time": 720.33,
"end_time": 750.66,
"audio": {
"content": " These are just a few brief examples, but we are excited about the possibilities, and we can't wait to see what you'll build with it. Gemini 2.4 is now available for everyone in AI studio, Vertex AI, and in the Gemini Act. I'm also excited to announce Gemini 2.5 Flash are low latency and most cost-e efficient model with thinking built in."
},
"video": {
"content": "A man in a blue blazer and white shirt stands on a stage, addressing an audience. The background is a large screen displaying a futuristic, grid-like landscape with a yellow sphere at its center. The man gestures with his hands as he speaks, maintaining a professional demeanor."
}
},
{
"start_time": 754.66,
"end_time": 772.66,
"audio": {
"content": " With 2.5 Flash, you can control how much the model reasons and balance performance with your budget. 2.5 Flash is coming soon in AI Studio, Vertex AI and in the Gemini app. We'll be sharing more details on the model"
},
"video": {
"content": "A speaker stands on a stage, addressing an audience at what appears to be a tech conference. The stage is illuminated with blue lighting, creating a modern and professional atmosphere. The speaker, dressed in a dark suit and white shirt, gestures with his hands as he speaks, emphasizing points about the new Gemini 2.5 Flash technology. The screen behind him displays the text \"New Gemini 2.5 Flash in AI Studio, Vertex AI, and the Gemini app Coming Soon.\" The audience is seated in darkness, focusing their attention on the speaker."
}
},
{
"start_time": 772.66,
"end_time": 794.94,
"audio": {
"content": " and its performance soon. I'm pretty excited by it and can't wait for you to see it for yourselves. Our goal is to always bring our latest AI advances into the fourth layer of our stack, products and platforms. Today, all 15 of our half a billion user products, including seven with 2 billion users,"
},
"video": {
"content": "A man stands on a stage, dressed in a blue blazer over a white shirt with a black bow tie, and dark pants. He is speaking to an audience, gesturing with his hands as he talks. The background is a gradient of blue shades, creating a calm and professional atmosphere."
}
},
{
"start_time": 797.33,
"end_time": 817.53,
"audio": {
"content": " are powered by our Gemini models. AI deployed at this scale requires world-class inference, which enterprises can benefit requires world-class inference, which enterprises can benefit to build your own AI-powered applications. Gemini is also helping us create net new products and experiences. Notebook LM is one example, now used by over 100,000 businesses."
},
"video": {
"content": "A man in a blue blazer and white shirt stands on a stage, gesturing with his hands as he speaks. The background is a gradient of blue shades, and there is a large screen behind him displaying various app icons, including Google, Gmail, Android, Chrome, Play Store, YouTube, and Maps. The man appears to be giving a presentation or speech."
}
},
{
"start_time": 817.53,
"end_time": 839.45,
"audio": {
"content": " It uses long context, multimodality, and our latest thinking models to show information in powerful ways. Gemini is not our only industry leading model. VO2 is the leading video generation model. Major film studios, entertainment companies, as well as the top advertising agencies in the world, are"
},
"video": {
"content": "A man in a blue suit stands on a stage with a blue background, clapping his hands together. The scene transitions to a large screen displaying a digital interface named \"NotebookLM.\" The screen shows various topics such as \"Edison's Spark: The Lightbulb Legacy,\" \"Engineering Illumination,\" \"Invention of the Lightbulb,\" and \"CS History.\" The man continues to speak, gesturing with his hands. The screen then displays a search bar with the text \"olive green muscle car approach.\""
}
},
{
"start_time": 839.45,
"end_time": 861.33,
"audio": {
"content": " using it to bring their stories to life. Getting advances into the hands of both consumers and enterprises is something we are really focused on. This is why we are able to innovate at the cutting edge and push the boundaries of what's possible for us and for you. The result, better, faster, and more innovation for everyone."
},
"video": {
"content": "A speaker stands on a stage, addressing an audience. The stage is dimly lit with blue lighting, creating a modern and professional atmosphere. The speaker gestures with his hands as he speaks, emphasizing points in his presentation. A large screen behind him displays various images and text, including a car drifting and a stack of three blue blocks with different symbols on them. The speaker appears to be explaining the significance of these images and symbols, likely in the context of technology or innovation."
}
},
{
"start_time": 861.33,
"end_time": 894.66,
"audio": {
"content": " It's exciting to see how that's helping companies of all sizes do more with AI and translate those benefits to customers. I'm delighted to introduce Chris Kempchinski, CEO of McDonald's, to tell you more. But first, thank you for having me and enjoy a week together in Las Vegas. Over to you, Chris. McDonald's is undergoing a once-in-a-generation transformation."
},
"video": {
"content": "A man stands on a stage, dressed in a blue blazer over a white shirt and dark pants. He is wearing glasses and has a microphone attached to his blazer. The background is a gradient of blue shades, giving a professional and modern appearance. The man appears to be speaking or presenting, as he gestures with his hands while looking directly at the audience."
}
},
{
"start_time": 894.66,
"end_time": 915.83,
"audio": {
"content": " We have about 65 million people that come to our restaurants every single day and it's how do we make their experience even better? Google's a big part of that, particularly as more and more of those customer interactions are happening in a digital world. That's why we're transforming our restaurant experience with the help of Google Cloud. Behind the counter, our restaurant team's jobs are becoming increasingly complex."
},
"video": {
"content": "A man in a dark suit and light blue shirt stands in an office-like setting with large windows and modern decor. He appears to be speaking, with his hands clasped together in front of him. The background includes a McDonald's sign and some balloons, suggesting a celebratory or promotional context. The scene transitions to a close-up of a smartphone screen displaying a message from McDonald's, indicating that an order has been received."
}
},
{
"start_time": 915.83,
"end_time": 937.33,
"audio": {
"content": " With edge computing from Google distributed cloud, capabilities will readily improve stability, security, and performance in our restaurants, all while giving us the space and power to test several new concepts we weren't able to do previously. For example, shift leaders will be able to leverage an AI powered assistant to help spot issues in the restaurant quickly. Our restaurant managers will be able to receive alerts"
},
"video": {
"content": "A man in a dark suit and light blue shirt stands in an office-like environment with large windows showing a cityscape outside. He gestures with his hands as he speaks, emphasizing his points. The word \"security\" appears on the screen, highlighting the topic of discussion. The scene then transitions to a close-up of a person's hands holding a McDonald's bag, with a smartphone displaying a queue analysis app showing customer and traffic data."
}
},
{
"start_time": 937.33,
"end_time": 964.55,
"audio": {
"content": " on their devices based on real-time data, say from their freezer or friars, along with guidance for from their freezer or friars, along with guidance for predictive maintenance. And with Gemini on Vertex AI, we can centralize all this information from restaurants in real time, making it easier for the right people to get answers with a simple question or prompt, improving the work environment in our restaurants across the globe for our more than 2 million team members. That's the magic. That's the power of AI and what Google Cloud has brought to McDonald's."
},
"video": {
"content": "A person wearing blue gloves uses a tool to crack eggs into small metal rings on a griddle. The scene then transitions to a McDonald's kitchen where a worker is seen handling food items. A warning message appears on the screen indicating that the fryer oil requires changing. The video then cuts to a man in a suit standing in front of a McDonald's restaurant, followed by a map of the world with McDonald's logos scattered across it."
}
},
{
"start_time": 964.55,
"end_time": 985.99,
"audio": {
"content": " Oh, oh, uh, uh Uh Uh Thank you, Chris. McDonald's is a great example of a company integrating AI into the very core of its operations."
},
"video": {
"content": "A man in a dark suit and light blue shirt stands confidently in an office-like environment with large windows and modern decor. He has short, neatly styled hair and is looking directly at the camera with a serious expression. The background features a McDonald's sign and some balloons, suggesting a casual yet professional setting."
}
},
{
"start_time": 985.99,
"end_time": 1007.99,
"audio": {
"content": " Customers around the world are choosing to work with Google for three important reasons. First, Google Cloud offers an AI-optimized platform with leading price, performance, precision, and quality. And new today, everything you need to build and manage multi-agent systems."
},
"video": {
"content": "A man in a dark blue suit stands on a stage, gesturing with his hands as he speaks. He appears to be giving a presentation or speech. The background is a simple, light-colored curtain with vertical stripes. The man's gestures change slightly throughout the video, indicating he is emphasizing different points in his speech."
}
},
{
"start_time": 1007.99,
"end_time": 1031.66,
"audio": {
"content": " Our AI platform offers advanced infrastructure and databases, world-class research leading models, and grounding for model responses with Google quality search. Vertex AI, a robust developer platform, including the broadest range of enterprise ready tools with which you can build AI agents"
},
"video": {
"content": "A speaker is presenting on a stage at an event, likely a conference or seminar. The stage is well-lit with blue lighting, and the audience is seated in rows, attentively watching the presentation. The speaker, dressed in a suit, stands confidently as he addresses the crowd. The background features large screens displaying Google Cloud branding and text that reads \"AI optimized platform built for multi-agent systems.\" The presentation includes diagrams and text slides that highlight key points about AI optimization and multi-agent systems."
}
},
{
"start_time": 1031.66,
"end_time": 1053.66,
"audio": {
"content": " and enable a multi-agent ecosystem. And the most comprehensive portfolio of purpose-built agents. Second, Google Cloud offers an open multi-cloud platform that allows you to adopt AI agents while connecting them with your existing IT landscape,"
},
"video": {
"content": "A man in a dark suit stands on a stage, gesturing as he speaks to an audience. The stage is illuminated with blue lighting, and a large screen behind him displays various logos and text related to Google Cloud services. The man appears to be presenting or giving a speech, moving his hands expressively as he talks. The audience is visible in the foreground, attentively listening to the speaker."
}
},
{
"start_time": 1053.66,
"end_time": 1077.97,
"audio": {
"content": " including your databases, your document stores, enterprise applications, and interoperating with models and agents from other providers. You get value faster from your AI investments. And third, Google Cloud offers an enterprise-ready AI platform built for interoperability."
},
"video": {
"content": "A man stands on a stage, dressed in a dark blue suit with a white shirt and a patterned tie. He gestures with his hands as he speaks, occasionally bringing them together in front of him. The background is a simple, light-colored curtain with vertical stripes, and the lighting is bright, highlighting the speaker."
}
},
{
"start_time": 1077.97,
"end_time": 1099.49,
"audio": {
"content": " It enables you to adopt AI deeply while addressing the evolving concerns around sovereignty, security, privacy, and regulatory requirements. You can adopt AI while we protect your data and your intellectual property and enable you to maintain compliance. Powering"
},
"video": {
"content": "A man stands on a stage, dressed in a dark blue suit with a white dress shirt and a black bow tie. He is gesturing with his hands as he speaks, indicating that he is likely delivering a presentation or speech. The background consists of vertical light panels, creating a modern and professional atmosphere."
}
},
{
"start_time": 1099.49,
"end_time": 1125.63,
"audio": {
"content": " this offering is our advanced infrastructure core for AI. To share the latest, please join me in welcoming Amin Vadat. You know, Thank you, Thomas."
},
"video": {
"content": "A man in a dark blue suit stands on a stage, gesturing with his hands as he speaks. The background is a simple, light-colored backdrop with vertical lines. The man appears to be addressing an audience, possibly at a conference or event. The lighting is bright, highlighting the speaker and the stage."
}
},
{
"start_time": 1125.63,
"end_time": 1147.43,
"audio": {
"content": " Demand for AI compute for training and inference is growing at an unprecedented rate. For over eight years, it has increased by over 10 times year over year, a factor of a hundred million in just eight years. We're continuing to offer leading power efficiency, performance, and networking for training and inference workloads, starting with the hardware."
},
"video": {
"content": "A man stands on a stage, dressed in a black suit jacket over a vibrant pink shirt. He appears to be giving a presentation or speech. His body language is animated as he gestures with his right hand, emphasizing points in his dialogue. The background is simple, featuring vertical light strips that create a subtle gradient effect from white to blue. As he speaks, the camera occasionally shifts focus to a large screen behind him, which displays a close-up image of a mechanical device, possibly related to the topic of his talk."
}
},
{
"start_time": 1147.43,
"end_time": 1171.66,
"audio": {
"content": " Today, we introduced our seventh generation of TPUs, Ironwood TPUs are our largest and most powerful TPU pods to date, more than a 10x improvement from our most recent high-performance TPU with over 9,000 chips per pod."
},
"video": {
"content": "A man stands on a stage, dressed in a black suit jacket over a pink shirt. He gestures with his right hand as he speaks, likely addressing an audience. The background is a gradient of blue shades, giving a modern and professional ambiance to the scene."
}
},
{
"start_time": 1171.66,
"end_time": 1193.66,
"audio": {
"content": " To meet the exponentially growing demands of the most demanding thinking models like Gemini 2.5. This delivers a staggering 42.5 exoflops of compute per pod. To give you a sense of the scale, the world's number one supercomputer supports 1.7 X-flops. Ironwood pods offer more than 24 times that compute power."
},
"video": {
"content": "A man in a dark suit stands on a stage, addressing an audience. The stage is illuminated with blue lighting, creating a professional and modern atmosphere. A large screen behind him displays various images and text related to computing technology. The man gestures with his hands as he speaks, emphasizing points about the technical aspects being discussed. The audience is seated in darkness, focusing their attention on the speaker and the visuals on the screen."
}
},
{
"start_time": 1193.66,
"end_time": 1217.33,
"audio": {
"content": " Ironwood T-s are truly built for the next generation of AI workloads. TPUs are an incredible achievement, but they are just one piece of our overall infrastructure. After all, a chip is only as powerful as a system that surrounds it. Our AI hypercomputer is a super-competing system designed to simplify AI deployment,"
},
"video": {
"content": "A man in a dark suit and red shirt stands on a stage, gesturing as he speaks. The background is a deep blue with vertical light strips. A large screen behind him displays an image of a computer chip labeled \"Ironwood\" and text that reads \"24x more compute power than the world's best supercomputer.\" The scene then transitions to a different part of the stage where the same man continues his presentation, now with a backdrop featuring various icons and text such as \"Agents,\" \"Vertex AI,\" \"Research and Models,\" and \"AI Hypercomputer.\" The lighting highlights these elements, emphasizing the technological theme."
}
},
{
"start_time": 1217.33,
"end_time": 1239.43,
"audio": {
"content": " improve performance, and optimize costs. It supports the best hardware platforms and brings together a single unified software stack and consumption model that enables you to use the hardware that best meets your needs and easily transition from one hardware generation to the next. Vital as we innovate so rapidly."
},
"video": {
"content": "A man stands on a stage with a blue background, gesturing as he speaks. He is dressed in a dark suit over a red shirt. The camera then pans to show the entire stage, which is dimly lit with blue lighting. A large screen behind him displays text about an AI Hypercomputer, highlighting features such as flexible consumption, open software, and performance-optimized hardware. The man continues to speak, occasionally pointing towards the screen."
}
},
{
"start_time": 1239.43,
"end_time": 1266.99,
"audio": {
"content": " We have enhanced our GPU portfolio with the availability of A4X and A4 VMs, powered by NVIDIA's GB200 and B200 Blackwell GPUs. We were the first cloud provider to offer both options. We're also pleased that Google Cloud will be among the first offer Nvidia's next generation, Vera Rubin GPUs, which offer up to 15 X-flops of FP4 inference performance per rack."
},
"video": {
"content": "A man in a dark suit and pink shirt stands on a stage, presenting information to an audience. The stage is illuminated with blue and white lights, creating a modern and professional atmosphere. The presenter gestures with his hands as he speaks, emphasizing key points. Behind him, large screens display text related to AI hypercomputers, performance-optimized hardware, and new VMs. The presentation includes slides about NVIDIA Vera Rubin GPUs being offered on Google Cloud."
}
},
{
"start_time": 1266.99,
"end_time": 1287.99,
"audio": {
"content": " We also introduced cluster-directors. per rack. We also introduce Cluster Director, which enables you to deploy and manage a large number of accelerators as a single unit to compute to improve performance, efficiency, and resilience. Storage is also vital to reduce the bottlenecks for training and inference. We are introducing new storage innovations."
},
"video": {
"content": "A man in a dark suit and red shirt stands on a stage, gesturing animatedly as he speaks. The background is a large screen displaying the text \"New Cluster Director\" and \"Easily deploy and manage large compute clusters.\" The audience is seated in front of the stage, attentively watching the presentation."
}
},
{
"start_time": 1287.99,
"end_time": 1311.9,
"audio": {
"content": " Hyperdisc X-Pers. storage innovations. Hyperdisc exopools offer the highest aggregate performance and capacity per AI cluster of any hyperscaler. Anywhere Cash keeps data close to your accelerators with up to 70% improvement in storage latency to reduce training time. And rapid storage, our first zonal storage solution offers five times lower latency for random reads and rights compared to the fastest"
},
"video": {
"content": "A speaker is presenting on a stage at an event, likely a tech conference. The stage features a large screen displaying slides about AI hypercomputing and cloud storage solutions. The speaker, dressed in a dark suit with a red shirt and black bow tie, gestures towards the screen as he explains the features of the new Hyperdisk Exapools and Cloud Storage Anywhere Cache. The audience is visible in the foreground, attentively watching the presentation."
}
},
{
"start_time": 1311.9,
"end_time": 1334.66,
"audio": {
"content": " comparable cloud alternative. Software is how we orchestrate and simplify access to this powerful hardware. And today we're introducing three enhancements for AI inference. First, we're introducing new inference capabilities in Google Kubernetes Engine, including Gen AI Aware-aware, scaling and load balancing features, which help reduce serving costs by up to 30%."
},
"video": {
"content": "A man stands on a stage, presenting to an audience. He gestures towards a large screen behind him, which displays various slides related to AI hypercomputing and storage solutions. The slides highlight features such as flexible consumption, open software options like JAX, Keras, PyTorch, vLLM, Pathways on Google Cloud, XLA, and Google Kubernetes Engine and Compute Engine. The presentation emphasizes the performance-optimized hardware, including networking (Jupiter, OCS). The speaker appears engaged and informative, moving slightly between slides while explaining the benefits and capabilities of the new storage solution."
}
},
{
"start_time": 1334.66,
"end_time": 1357.32,
"audio": {
"content": " Tail latency by up to 60% and increased throughput by up to 40%. Second, we're announcing that Pathways, Google's own distributed ML runtime, Powering Gemini, is now available for the first time for cloud customers. Developed by Google DeepMind, Pathways enable state-of-the-art multi-host inferencing for dynamic scaling with high performance at optimal costs."
},
"video": {
"content": "A speaker is presenting on a stage at an event, likely a conference or tech summit. The stage is well-lit with a large screen displaying text about new features or products. The speaker, dressed in a dark suit and a vibrant pink shirt, gestures animatedly as he speaks. The audience is seated in front of the stage, attentively watching the presentation. The background includes a mix of blue and white lighting, creating a modern and professional atmosphere."
}
},
{
"start_time": 1357.32,
"end_time": 1377.56,
"audio": {
"content": " Now you can scale out model serving to hundreds of accelerators for the best combination of batch efficiency and low latency. Third, we're bringing VLLM to TPUs. This allows customers who optimize PITORC with VLM for GPUs to easily and cost-efficiently run their workloads on TPUs."
},
"video": {
"content": "A man stands on a stage, dressed in a dark suit over a red shirt and black pants. He gestures with his right hand as he speaks, indicating an engaging presentation. The background is a simple, modern design with vertical light panels, creating a professional and focused atmosphere. As the video progresses, the camera shifts to show a wider view of the stage, revealing a large screen displaying the text \"New vLLM on TPU\" along with additional information about the technology. The audience is seated in darkness, attentively watching the presentation."
}
},
{
"start_time": 1377.56,
"end_time": 1398,
"audio": {
"content": " All of these AI hypercomputer hardware and software enhancements together enable us to deliver more intelligence or useful AI output at a consistently low price. This is one reason why Gemini 2.0 Flash, powered by AI hypercomputer, achieves 24 times higher intelligence per dollar compared to GPT-40,"
},
"video": {
"content": "A man in a dark suit and pink shirt stands on a stage, gesturing as he speaks. Behind him is a large screen displaying text about new AI hypercomputer enhancements, including \"vLLM on TPU\" and \"GKE Inference Gateway.\" The stage is well-lit with blue lights, and the audience is visible in the background."
}
},
{
"start_time": 1398,
"end_time": 1420.32,
"audio": {
"content": " and five times higher than DeepSeek R1. We're truly seeing tremendous momentum across our AI infrastructure portfolio, tripling the number of TPU and GPUR is consumed by our cloud customers just over the past year. And we're seeing tremendous customer momentum with AI unicorns like Anthropic, any scale, arise, and contextual AI."
},
"video": {
"content": "A man in a suit stands on a stage, addressing an audience. The background features large screens displaying text about Google Cloud's products and services. The first screen shows information about Gemini 2.0 Flash, highlighting its superior intelligence per dollar compared to competitors. The second screen indicates a 3x increase in TPU and GPU hours consumed by cloud customers. The third screen emphasizes that leading AI unicorns trust Google Cloud's infrastructure."
}
},
{
"start_time": 1420.32,
"end_time": 1442.99,
"audio": {
"content": " And enterprises. Toyota deployed ML models for factory workers. Schrodinger uses cloud GPUs for advanced drug discovery. TSMC protects its critical data for mission critical workloads. An Airbus deployed an AI platform to advance aircraft performance, safety, and reliability."
},
"video": {
"content": "A man in a suit stands on a stage, presenting to an audience. The background features a large screen displaying various slides related to Google Cloud's infrastructure and its benefits for leading AI unicorns. The slides include text such as \"Leading AI Unicorns trust Google Cloud's infrastructure\" and logos of companies like Anthropic, Anyscale, Arize, Contextual AI, Toyota, Airbus, and TSMC. The presentation highlights the use of GPUs for advanced drug discovery and data protection for mission-critical workloads."
}
},
{
"start_time": 1442.99,
"end_time": 1465.99,
"audio": {
"content": " Beyond optimizing training and inference in the cloud, we know that many AI workloads need to be run on-premises. As you heard from Chris Kipchensky at McDonald's, Google's distributed cloud brings our hardware and software to your environments. So you can bring AI capabilities closer to where data is generated for low latency and highly sensitive data in particular."
},
"video": {
"content": "A man in a black suit and pink shirt stands on a stage, gesturing as he speaks. The background is a gradient of light blue to white, with vertical light strips on either side. He moves slightly from side to side, emphasizing his points. The scene transitions to a large screen displaying information about Google's AI Hypercomputer, including open software options like JAX, Keras, PyTorch, vL.M., Pathways on Google Cloud, XLA, and performance-optimized hardware. The screen also features the Google Distributed Cloud logo."
}
},
{
"start_time": 1465.99,
"end_time": 1489.66,
"audio": {
"content": " Today we are announcing that Gemini can run on Google distributed cloud locally in air-gapped environments, as well as connected environments. This all comes with the support for NVIDIA's confidential computing and Blackwell systems, DGXB200 and HGXB200 platforms with Dell as a key partner."
},
"video": {
"content": "A speaker stands on a stage at a conference, addressing an audience. The stage is well-lit with a large screen behind him displaying text about Google Distributed Cloud and Gemini. The speaker gestures with his hands as he speaks, emphasizing points about running Gemini locally in air-gapped and connected environments. The audience is seated, attentively listening to the presentation."
}
},
{
"start_time": 1489.66,
"end_time": 1520.4,
"audio": {
"content": " This complements our Google Distributed Cloud AirGap product, which is now authorized for U.S. government's secret and top secret missions, and on which Gemini is now available, providing the highest levels of security and compliance. Nvidia is an important partner for Google and our customers. Let's hear directly from CEO Jensen Wang. Building Advanced AI Infrastructure is deep computer science. No company is better at every single layer of"
},
"video": {
"content": "A man stands on a stage, dressed in a black suit jacket over a pink shirt. He gestures with his right hand as he speaks, occasionally adjusting his jacket. The background is a gradient of light blue to white, with vertical lines on the left side. The scene transitions to a wider shot of the stage, where the man continues to speak. The screen behind him displays the logos of Google Cloud and NVIDIA, followed by the text \"Advancing AI together.\""
}
},
{
"start_time": 1520.4,
"end_time": 1542.99,
"audio": {
"content": " computing than Google and Google Cloud. Between Nvidia and Google Cloud, this super partnership includes capabilities that covers literally every single layer and every single aspect of computing. Every industry, every company, every country wants to get their hands on AI. However, everything has to be fundamentally confidential and secure."
},
"video": {
"content": "A man stands in a modern, futuristic interior space with geometric glass walls and a high ceiling. He is dressed in a black leather jacket over a black shirt and dark pants, complemented by black shoes. His hair is gray, and he wears glasses. He gestures with his hands as if explaining something, moving them from his chest outward to emphasize his points. The lighting is soft and ambient, creating a calm and focused atmosphere."
}
},
{
"start_time": 1542.99,
"end_time": 1574.66,
"audio": {
"content": " And so we're announcing something utterly gigantic today. Google distributed cloud with Gemini and Nvidia are going to bring state-of-the-art AI to the world's regulated industries and countries. Now, if you can't come to the cloud, Google Cloud will bring AI to you. Thank you. Thank you, Jensen. You know, we really value our deep engineering relationship with"
},
"video": {
"content": "The video begins with a close-up shot of a server rack, showcasing various components such as power supplies, fans, and network interfaces. The camera pans across the rack, highlighting the intricate details and the organized layout of the hardware. The scene then transitions to a black screen with the text \"Accelerated by NVIDIA Blackwell\" displayed prominently in white letters against a black background. Following this, the video cuts to a man standing on a stage, gesturing with his hands as he speaks. He is dressed in a black leather jacket over a black shirt. The background features a futuristic, geometric design with green and black colors. The final scene shows the same man on stage, now wearing a suit, continuing his presentation. The stage is well-lit with blue lighting, and a large screen behind him displays the logos of Google Cloud and NVIDIA."
}
},
{
"start_time": 1574.66,
"end_time": 1595.76,
"audio": {
"content": " Nvidia. Building on the ground-bracing research of Google DeepMind, we're delivering rapid innovation across many AI models, starting with Gemini, our most capable family of AI models. In the last year alone, we released Gemini a first native multi-modal model. We delivered the native multimodal model."
},
"video": {
"content": "A man in a dark suit stands on a stage with a blue background. He gestures with his hands as he speaks, occasionally clapping them together. The scene transitions to a large screen displaying a diagram of three stacked blocks labeled \"Research and Models.\" The man continues to speak, and the camera shifts to show him from different angles, highlighting the blue lighting and the audience in the foreground."
}
},
{
"start_time": 1597.56,
"end_time": 1617.66,
"audio": {
"content": " We delivered the first 2 million token context window. We built the live API for live bidirectional voice and video interaction, led in price performance with our flash models, and we recently launched Gemini 2.5 Pro, which is state-of-the-art on a wide range of benchmarks,"
},
"video": {
"content": "A man in a suit stands on a stage, gesturing towards a large screen behind him. The screen displays a timeline with various stages labeled as 'Gemini', 'Live API', 'Gemini Flash models', and 'Gemini 2.5 Pro'. The man appears to be explaining these stages, likely in the context of a presentation or speech about technological advancements."
}
},
{
"start_time": 1617.66,
"end_time": 1640.32,
"audio": {
"content": " and I'm pleased to say number one on chat bat arena. Batarina. Gemini is providing best in class AI for many companies around the world, including our close partners, Box and Palo Alto networks who are using Gemini 2.5 to deliver new applications."
},
"video": {
"content": "A man stands on a stage with a blue curtain backdrop. He is dressed in a dark suit jacket, white dress shirt, and black tie. His hands are clasped together in front of him as he speaks. The lighting highlights his upper body and face. The scene then transitions to a large screen displaying a list of company names and logos under the heading \"Gemini customers around the world.\" The screen also includes the text \"and many more...\""
}
},
{
"start_time": 1640.32,
"end_time": 1660.36,
"audio": {
"content": " It's also integrated across our own products, including Google Workspace, where Gemini powers features in Gmail, docks, drive, and meat, and is now included in all subscriptions. Gemini and Workspace is helping customers,"
},
"video": {
"content": "A man in a dark suit stands on a stage, gesturing with his hands as he speaks. The background is a simple, light-colored curtain. The scene then transitions to a large screen displaying the text \"Google Workspace with Gemini.\" Various Google Workspace icons, including Drive, Docs, Sheets, Calendar, and Gmail, appear on the screen. The man continues to speak, and the audience is visible in the foreground, attentively watching him."
}
},
{
"start_time": 1660.36,
"end_time": 1681.32,
"audio": {
"content": " like EV manufacture Rivian, fresh fields to enhance legal work and expedite tasks like document drafting and research, and companies of the Schwartz Group, Europe's largest retailer. Today I'm pleased to announce three new innovations with Gemina and workspace."
},
"video": {
"content": "A man in a suit stands on a stage, addressing an audience. The stage is illuminated with blue lights, creating a modern and professional atmosphere. Behind him, a large screen displays various slides with text and images. The first slide shows a blue arrow symbol, followed by another slide that reads \"Setting a new standard for legal due diligence with Gemini.\" The next slide features a woman working on a laptop in a grocery store, with the text \"Migrating to Google Workspace provides their employees a secure way to work.\" The man gestures towards the screen as he speaks, emphasizing the points being made."
}
},
{
"start_time": 1681.32,
"end_time": 1706.63,
"audio": {
"content": " Help me analyze Gemina and Workspace. Help me analyze in Google Sheets, which guides you through your data to complete expert-level analysis. Audio overviews in Google Docs, where you can interact with docs in an entirely new way by creating high quality audio versions of your content. And Google Workspace Flow of your content and Google workspace flows to help you automate time-consuming"
},
"video": {
"content": "A man in a dark blue suit stands on a stage, addressing an audience. He gestures with his hands as he speaks, emphasizing his points. The background is a large screen displaying a presentation slide about Google Workspace. The slide features text that reads \"New Help me analyze Uncover insights from your data\" and \"Coming Soon.\" Another slide below it reads \"New Audio overviews in Docs Create high-quality audio versions of your docs\" and also indicates \"Coming Soon.\" The stage has a modern design with vertical light panels and a dark backdrop."
}
},
{
"start_time": 1706.63,
"end_time": 1727.99,
"audio": {
"content": " repetitive tasks and to make decisions with more context. Let's see how Google Workspace is helping businesses around the world. We We are scattered all over the globe. It's just bonkers to try to get everyone to collaborate."
},
"video": {
"content": "A large screen displays an announcement about Google Workspace Flows, highlighting its ability to automate work with agents in the loop. The scene transitions to a man standing on a stage, dressed in a dark suit and white shirt, addressing an audience. He gestures with his hands as he speaks, emphasizing key points. The camera then shifts focus to a hand interacting with a futuristic device, which appears to be part of a demonstration or presentation."
}
},
{
"start_time": 1727.99,
"end_time": 1756.32,
"audio": {
"content": " Keeping up with my email everyone to collaborate. Keeping up with my email has always been really, really difficult. Is there anything more intimidating than a blank white page? But where do you begin? It's more important than ever that you have tools to work more quickly and efficiently together. Using Help Me Write, we were able to save 35 hours a month for product descriptions for our website."
},
"video": {
"content": "A group of people are gathered around a table in what appears to be an office or a meeting room. They are engaged in a discussion, with one person speaking while others listen attentively. The setting is modern and well-lit, with large windows allowing natural light to flood the space. The atmosphere is collaborative and focused."
}
},
{
"start_time": 1759.32,
"end_time": 1777.12,
"audio": {
"content": " We are leveraging meat to take notes, summarize them, and generate action items after the meeting so that we can be really present and focus on the content. When you're developing a DAC or a pitch for a client and you're coming up with ideas"
},
"video": {
"content": "Focus on the meeting, not the notes.\n\nGemini is taking notes\nIt can take a few minutes for notes to appear\nStop taking notes\n\nYael Burla Vimeo\n\nDeliver on time, every time with visuals in seconds."
}
},
{
"start_time": 1777.12,
"end_time": 1797.32,
"audio": {
"content": " and you need to visually manifest those ideas that used to take days. And now with the right prompts, we can do that in hours. We have to worry less about security because we know Google has our back. Security played a big part of the decision to move from the legacy on-premise tools set to Google Workspace."
},
"video": {
"content": "The video begins with three women sitting around a table, intently looking at a laptop screen. They appear to be engaged in a discussion or collaborative work session. The scene then transitions to a woman sitting at a desk, working on a computer. The focus shifts to a close-up of various makeup products, including lipsticks and blushes, arranged on a table. The video then cuts to a black screen with the text \"Keep everything secure.\" followed by a shot of a large satellite dish inside a warehouse. The final frame shows a person standing in front of the satellite dish."
}
},
{
"start_time": 1797.32,
"end_time": 1838.32,
"audio": {
"content": ""
},
"video": {
"content": "The video begins with a serene cityscape at dusk, featuring tall buildings with illuminated windows against a pinkish-orange sky. The scene transitions to an indoor climbing gym where a group of six people, dressed in athletic gear, pose together. They stand in front of a colorful climbing wall, smiling and looking directly at the camera. The video then shifts to a conference or presentation setting, where a man in a suit stands on a stage, addressing an audience. The background features a large screen displaying the text \"Google Workspace with Gemini\" and \"Imagen 3,\" along with various images and graphics related to the topic."
}
},
{
"start_time": 1797.32,
"end_time": 1862.25,
"audio": {
"content": " The impact of Gemini for workspace in our business has been really transformative. Thank you. Beyond Gemini, over the last year, we've made huge improvements to imagine three, our highest quality texture image model, which generates images with better detail, richer lighting, and fewer distracting artifacts than previous models. Imagine delivers accurate prompt adherence, bringing your creative vision to life with incredible precision. We also introduce CHRP 3 to help you create custom voices with just 10 seconds of input and to weave AI-powered narration into your existing recordings."
},
"video": {
"content": "The video begins with a panoramic view of a city skyline at dusk, featuring numerous tall buildings with illuminated windows. The sky is painted in hues of orange and pink, indicating either sunrise or sunset. The scene then transitions to an indoor setting where a man in a suit stands on a stage, addressing an audience. The background is dark, with blue lighting accentuating the stage area. The man gestures as he speaks, emphasizing his points. The video captures the essence of a professional presentation or conference."
}
},
{
"start_time": 1863.67,
"end_time": 1888.65,
"audio": {
"content": " Today, we're making Luria available on Google Cloud to transform text prompts into 30 second music clips and with the first hyperscaler to offer this capability. Let's hear a clip from Lyria. VO2 V-O-2"
},
"video": {
"content": "A man in a suit stands on a stage, addressing an audience. The stage is modern and sleek, with a large screen displaying the word \"Lyria.\" The man gestures with his hands as he speaks, emphasizing his points. The audience is seated in rows, attentively listening to the presentation."
}
},
{
"start_time": 1890.65,
"end_time": 1912.79,
"audio": {
"content": " is our industry-leading video generation model. It generates video generation model. It generates many minutes of 4K video, watermarked with synth ID, to ensure they can be identified as AI generators. It gives creators unprecedented creative control with new editing tools including camera"
},
"video": {
"content": "A man in a suit stands on a stage, gesturing as he speaks to an audience. The stage is illuminated with a modern design featuring large, vertical light panels. A large screen behind him displays various images, including a wedding cake, a golden guitar, and a cowboy in a Western town. The man appears to be presenting or demonstrating something related to these images."
}
},
{
"start_time": 1912.79,
"end_time": 1932.89,
"audio": {
"content": " presets to direct shot composition and camera angles without complex prompting, first and last shot control to define the beginning and the end of a video sequence with VO seamlessly bridging the gap and dynamic in-painting and outpainting for video"
},
"video": {
"content": "A man in a cowboy hat and vest is walking down a dirt road in front of a wooden building. The camera pans to the right, revealing more of the desert landscape with cacti and mountains in the background. The scene then transitions to an animated character with blue hair standing on a city street at night. The character looks surprised as he turns around and walks away from the camera."
}
},
{
"start_time": 1932.89,
"end_time": 1968.07,
"audio": {
"content": " editing and scaling. With Gemini, imagine, chirp, Lyria, and Vio. Google is the only company that offers generative media models across all modality. And all of them are available to you today on Vertex AI. Thank you. We've seen great examples from our customers. Craft Hines is speeding up campaign creation. A go-down creates unique"
},
"video": {
"content": "A man in a suit stands on a stage, gesturing with his hands as he speaks. The background is a large screen displaying various text and images. The scene transitions to a close-up of the man, emphasizing his gestures and expressions. The lighting is bright, highlighting the speaker and the screen behind him."
}
},
{
"start_time": 1968.07,
"end_time": 2002.99,
"audio": {
"content": " visuals of travel destinations. Bending spoon makes 60 million photos every day more fun and L'Oreal Group generates diverse cinematic shots using our models. Please join me now in welcoming Nenshad Bodily Walla for a demo of our models in action using vertex AI. Hello, everyone."
},
"video": {
"content": "A professional speaker stands on a stage, presenting to an audience. The stage is well-lit with a large screen displaying various images and text related to technology and innovation. The speaker gestures with his hands as he explains the features and benefits of a new product or service. The background includes a large screen showing images of a man holding a box of cereal and another image processing 60 million photos daily using Imagen 3. The speaker's attire is formal, consisting of a dark suit and white shirt."
}
},
{
"start_time": 2002.99,
"end_time": 2024.79,
"audio": {
"content": " How many of you have heard about our Cloud Next concert already? already. That's not as many I was hoping to hear about and I think that's because we've been missing a teaser video. Now let me tell you, it wasn't easy to pick the artist this year because it turns out that even though he looks very demure and very mindful,"
},
"video": {
"content": "A man stands behind a podium on a stage, delivering a presentation. The background is dark with blue lighting accents. A large screen behind him displays his name, \"Nenshad Bardoliwalla,\" and his title, \"Director, Product Management, Vertex AI Platform.\" He gestures with his hands as he speaks, occasionally looking at a laptop on the podium. The setting appears to be a formal conference or event."
}
},
{
"start_time": 2024.79,
"end_time": 2050.65,
"audio": {
"content": " Thomas Corian is a massive chapel ran fanboy. Yes, I have seen the video that he sent chapel directly of him going, H-O-T-O-G-O, Thomas wants you in the show. Yeah, but we waited for weeks to get her response from chapel,"
},
"video": {
"content": "A man stands behind a podium on a stage, addressing an audience. The background features a large screen displaying his name, Nenshad Bardoliwalla, along with his title as Director, Product Management, for Vertex AI Platform. He is dressed in a dark blue blazer over a white shirt, with a microphone clipped to his blazer. The stage is well-lit, with blue lighting accents and a modern design."
}
},
{
"start_time": 2050.65,
"end_time": 2072.29,
"audio": {
"content": " and then she broke Thomas's heart with just three words. Good luck, babe. So, we're going to use Vertex Media to pump Thomas up and create a teaser video that's going to get you as amped up as I am. Now we've already created our final video. I'm just going to show you how we got there. Given where we are, we're going"
},
"video": {
"content": "A man stands behind a podium on a stage, addressing an audience. He is dressed in a dark blue blazer over a white shirt with a patterned tie. The background features vertical blinds, and there is a laptop on the podium displaying a presentation slide titled \"Vertex AI Media Studio.\" The slide lists various generative media models: Imagen (generate images), Veo (generate videos), Chirp (generate voices), and Lyria (generate music). The man gestures with his hands as he speaks, emphasizing points about the technology being discussed."
}
},
{
"start_time": 2072.29,
"end_time": 2101.2,
"audio": {
"content": " to use Las Vegas Skyline as a perfect backdrop for what we're going to do with Vertex AI Media Studio. So let's go ahead. We're going to start by bringing in the Las Vegas skyline image. Really high quality, beautiful image. We're going to generate video, but here's the new hotness. Check it out. Camera presets built right into Vio, panning left, panning right, time lapse, tracking shots,"
},
"video": {
"content": "A man is standing at a podium, speaking into a microphone. He is dressed in a dark suit and white shirt. The background is a plain, dark-colored wall. The man appears to be addressing an audience, gesturing with his hands as he speaks."
}
},
{
"start_time": 2101.2,
"end_time": 2123.99,
"audio": {
"content": " and even drone shots. So let's go ahead and submit a drone shot in drone shot of the city skyline. There we go. We'll go and submit this. Now normally this would take a few seconds. I ran this earlier today so it's cached, so it's going to be a little quicker than normal."
},
"video": {
"content": "A man is sitting at a desk in a studio, speaking into a microphone. He gestures with his hands as he talks. The scene then transitions to a computer screen displaying a media studio interface. The interface shows an image of a city skyline at sunset, featuring prominent buildings and a fountain. The camera preset is set to 'Drone shot,' and the frame rate is set to 24 fps. The video length is set to 8 seconds. The man continues to speak, and the screen displays the text 'drone shot of the city skyline' in the prompt box."
}
},
{
"start_time": 2123.99,
"end_time": 2144.99,
"audio": {
"content": " All right, let's look at video number one. Absolutely spectacular. We have the ability to see the fountains, the Eiffel Tower. Now let's go ahead and take a look at video number two. A different angle that Vio creates for us."
},
"video": {
"content": "A man in a suit is standing at a podium, speaking into a microphone. The background shows a cityscape with illuminated buildings and a prominent tower. The man gestures with his hands as he speaks, emphasizing his points. The video then transitions to a screen displaying a drone shot of the city skyline, showing the same illuminated buildings and tower. The video continues to play, showing the drone footage in detail."
}
},
{
"start_time": 2144.99,
"end_time": 2167.99,
"audio": {
"content": " Again, stunning imagery. You can see the clouds in the background and look at the cars driving up and down Las Vegas Boulevard. Absolutely incredible. Now, one video is not going to do it for the concert promo we want to do, so I want to show you some of the other videos that I created. I have one here of the stage being set up all through the power of VO."
},
"video": {
"content": "A man in a suit is standing at a podium, speaking into a microphone. The background is a plain, dark curtain. The camera remains stationary throughout the video."
}
},
{
"start_time": 2167.99,
"end_time": 2189.65,
"audio": {
"content": " I have one of the band. I even have one of the audience actually clapping for what they're about to see. This will be a good reminder for all of you. Now, something very interesting happened. It turns out that Vio can do something that my 12 year old can do,"
},
"video": {
"content": "A man in a suit is standing at a podium, speaking into a microphone. The camera then cuts to a stage with bright lights and a band performing. The audience is visible, clapping and enjoying the performance. The scene shifts to a close-up of a hand holding a microphone, followed by a shot of a woman clapping her hands. The video then shows a panoramic view of a cityscape at dusk, with the Las Vegas Strip in the background. Finally, the video ends with a shot of a guitar case on stage."
}
},
{
"start_time": 2189.65,
"end_time": 2216.32,
"audio": {
"content": " and that is be an expert in photo bombing. It turns out that this great video we just saw has a crew member, and we love our crew members. However, in this case, I'd like to feature the guitar because the guitar is the most important part of the band. So let's go ahead and use VO's new in-painting capability. And I'm sorry, sir, I apologize."
},
"video": {
"content": "A man is standing at a podium, speaking into a microphone. He gestures with his hands as he talks. The background is a dark stage with a spotlight shining on him. The camera remains stationary throughout the video."
}
},
{
"start_time": 2216.32,
"end_time": 2243.99,
"audio": {
"content": " I know you're very good at your job, but I am going to have to remove you from this image. We will send flowers to you and your family though, sir. Let's use the new in-painting capability, wait a couple of seconds, and let's see what we see. Now if this does what I think it does, it should preserve every single aspect of what we saw before just without our stage hand."
},
"video": {
"content": "A man in a suit is sitting at a desk, looking at a laptop screen. The scene then transitions to a stage with a guitar and a suitcase. The man in the suit appears to be interacting with the guitar and the suitcase."
}
},
{
"start_time": 2243.99,
"end_time": 2266.99,
"audio": {
"content": " Look at that. Okay, so we got some video clips. Now we need some music. Let's try the first clip I created with Lyrion and see how we like it. You know, that's not quite my tempo."
},
"video": {
"content": "A man is standing at a podium, speaking into a microphone. The background is a dark stage with a large screen displaying an image of a guitar and a suitcase. The man appears to be giving a presentation or speech. The video then transitions to a screen showing the Google Cloud Next 25 logo and sound waveforms."
}
},
{
"start_time": 2266.99,
"end_time": 2292.25,
"audio": {
"content": " I need music that's going to make all of you feel like I'm never going to give you up. I'm never going to let you down. I'm never going to run around and desert you. So let's try clip number two and see how that works. All right, we have the recipe."
},
"video": {
"content": "A man is standing at a podium, speaking into a microphone. He is dressed in a dark suit jacket over a white shirt. The background is a plain, light-colored curtain. The man appears to be addressing an audience, gesturing with his hands as he speaks. The video is likely from a conference or event, possibly related to technology or business."
}
},
{
"start_time": 2292.25,
"end_time": 2324.99,
"audio": {
"content": " I like that tune better. We've got the videos. we've got the music, let's pull it all together and see what it looks like. Here we go. Play it, Sam. Hey, and you know I'm What do you think? Absolutely amazing."
},
"video": {
"content": "A man in a suit is standing at a podium, gesturing with his right hand as he speaks. The background is dark, and there is a large screen behind him displaying the Google Cloud Next 25 logo. The screen also shows two sound waveforms: one blue and one green. The man appears to be giving a presentation or speech."
}
},
{
"start_time": 2324.99,
"end_time": 2353.65,
"audio": {
"content": " We've seen the amazing capabilities of Vio, the ability to create incredible shots with very little prompting, the ability to have editing capabilities that are easy to use and the cinematic quality. We're gonna see you tomorrow night when Thomas does a stage dive into the Mosh pit at Allegiance Stadium for the Killers. THE KILLER THANES!"
},
"video": {
"content": "A man stands behind a podium, gesturing with his hands as he speaks. He is dressed in a dark blue blazer over a white shirt with a black bow tie. The background consists of vertical blinds, and there is a laptop on the podium with a cloud logo sticker. The man appears to be giving a presentation or speech."
}
},
{
"start_time": 2353.65,
"end_time": 2379.12,
"audio": {
"content": ""
},
"video": {
"content": "A man in a dark blue suit and white shirt is standing on a stage, gesturing with his right hand as he speaks. He appears to be addressing an audience, possibly at a conference or seminar. The background features vertical blinds, and there are two computer monitors on either side of him. The man occasionally turns his head to look at the audience, maintaining a confident and engaging demeanor."
}
},
{
"start_time": 2353.65,
"end_time": 2399.32,
"audio": {
"content": " Welcome back to the stage, my friend and spiritual advisor, Thomas Corian. Thank you, Nanshad. I am also very excited for the concert tomorrow night. We're also bringing AI models to the physical world. Our partners like Samsung are using Gemini models for their exciting new AI companion robot, Bolly. And Google DeepMind recently introduced two new AI models for a new generation of helpful robots."
},
"video": {
"content": "A man in a dark blue suit and white shirt is standing on a stage, gesturing with his right hand as he speaks. He appears to be addressing an audience, possibly at a conference or event. The background features a large screen displaying the Samsung logo and some text related to Google Cloud. The man's expression and body language suggest he is engaged in delivering a presentation or speech."
}
},
{
"start_time": 2399.32,
"end_time": 2420.32,
"audio": {
"content": " Now let's talk about Vertex AI, a comprehensive AI platform. Vertex helps you discover enterprise-ready foundation models to customize, evaluate, and deploy applications built with the best foundation models and to build and manage AI agents at scale."
},
"video": {
"content": "A speaker is presenting at a tech conference, standing on a stage with a large screen behind him displaying various slides. The first slide shows a robot in a lab environment with the Google DeepMind logo. The next slide introduces Vertex AI, followed by a slide listing features like Agent Builder, Model Garden, and Open Software. The speaker gestures with his hands as he explains the features."
}
},
{
"start_time": 2420.32,
"end_time": 2443.98,
"audio": {
"content": " Let's hear how Intuit is making tax preparation even easier with Document AI, which is part of Vertex CI. Last year, Intuit TurboTax processed 44 million returns and $107 billion in refunds with the help of AI. Yet some customers with complex 1099 forms spaced hours of manual data entry."
},
"video": {
"content": "A man in a dark suit stands on a stage, gesturing with his hands as he speaks. The background is a simple, light-colored curtain. The scene then transitions to a dimly lit room where a person is sitting at a desk, wearing glasses and a yellow cap. The final scene shows a man sitting outside a house, holding a smartphone, with text overlaying the image that reads \"$107 billion in refunds.\""
}
},
{
"start_time": 2443.98,
"end_time": 2470.97,
"audio": {
"content": " This year, Intuit unlocked higher quality data comprehension and auto fill with Google Cloud Document AI. This done-for-you experience simplifies tax filing for millions, freeing up time for living life. Intuit built a new way to make taxes easier. Tens of thousands of companies are building with Vertex AI and Gemini."
},
"video": {
"content": "A woman is sitting at a desk, looking at her phone with a smile on her face. She is wearing a white shirt and has long brown hair. The scene then cuts to a man sitting at a desk with a laptop, smiling as he looks at something on the screen. A woman stands behind him, holding a piece of paper and smiling. The scene then cuts to a white screen with the Google Cloud logo. Finally, a man in a suit stands on a stage, speaking to an audience."
}
},
{
"start_time": 2471.55,
"end_time": 2491.65,
"audio": {
"content": " Nokia built a tool to speed up application coding and development. Wayfair updates product attributes five times faster. AES, an energy company, reduces audit costs by 99% and audit time from 14 days to just one hour."
},
"video": {
"content": "A man in a dark blue suit stands on a stage, gesturing with his hands as he speaks. The background is a gradient of light blue and white, with vertical lines creating a subtle pattern. The man appears to be addressing an audience, possibly at a conference or presentation."
}
},
{
"start_time": 2493.65,
"end_time": 2512.65,
"audio": {
"content": " Combeance Bank is creating AI-assisted summaries of investment advisory calls. Seattle Children's Hospital makes thousands of pages of clinical guidelines instantly searchable by their pediatricians. United Wholesale Mortgage is transforming"
},
"video": {
"content": "A man in a dark suit stands on a stage, speaking to an audience. The background is a simple, light-colored curtain. The scene then transitions to a large screen displaying a presentation slide. The slide features a woman in a white shirt interacting with a child, with text that reads \"1,000+ pages of clinical guidelines instantly searchable with Gemini.\" The man continues to speak, gesturing with his hands as he presents."
}
},
{
"start_time": 2512.65,
"end_time": 2534.95,
"audio": {
"content": " the mortgage experience, more than doubling underwriter productivity and Honeywell has incorporated Gemini into their product development. Honeywell and Google Cloud designed a new way to manage product life cycles. It will revolutionize how they handle millions of products."
},
"video": {
"content": "A man in a suit stands on a stage, addressing an audience. The background features a large screen displaying a presentation slide with the text \"UWM 2x loan underwriter productivity with Vertex AI.\" The man gestures with his hands as he speaks, emphasizing points about the benefits of using Vertex AI for loan underwriting. The scene then transitions to a wide shot of the stage, showing the audience seated in darkness, with the screen displaying an image of an airplane flying at sunset."
}
},
{
"start_time": 2535.32,
"end_time": 2573.65,
"audio": {
"content": " Built with Vertex AI, Big Query, and Gemini, this agentic framework accelerates spec and model creation, connects with their global install accelerate spec and model creation. Connects with their global install-based to uncover performance improvement insights and extends life cycles by re-engineering products. Estimated to help their engineering by re-engineering products, estimated to help their engineers deliver results up to 70% faster. With AI agents, Honeywell is introducing a new way to optimize millions of products. In just the last year, we've seen over 40 times growth in Gemini use and Vertex AI, now with billions of API calls each month."
},
"video": {
"content": "The video begins with a black screen displaying the text \"millions of products\" in white and red letters. The scene transitions to a close-up of a blue square button with a white icon of a camera, surrounded by red squares. The word \"Connects\" appears below the button. The video then cuts to a white screen with the logos of Google Cloud and Honeywell side by side. Finally, a man in a suit is shown standing on a stage, gesturing with his hands as he speaks."
}
},
{
"start_time": 2573.65,
"end_time": 2596.32,
"audio": {
"content": " Vertex AI gives you easy access to over 200 curated foundation models through a model gardens. We offer all of Google's models, Gemini, Vio, Imagine, and our latest research models, curated popular third party models, and open source models, all now on Vertex AI."
},
"video": {
"content": "A man in a dark suit stands on a stage, gesturing with his hands as he speaks. The background is a large screen displaying text about curated foundational models. The scene transitions to a wide shot of the stage, showing the audience seated in rows. The camera then focuses on a screen displaying various AI models and their features."
}
},
{
"start_time": 2596.32,
"end_time": 2618.32,
"audio": {
"content": " New vertex dashboards help you monitor usage, throughput, latency, after troubleshoot errors. New tuning methods help you optimize the model's performance for your applications. We are excited to announce the general availability of MetaSlamma 4 on Vertex AI."
},
"video": {
"content": "A user is navigating through a software interface, likely related to machine learning or AI development tools. The interface displays various features and services available within the platform, such as Vertex AI Studio, which includes options like API availability, open source, notebook support, pipeline support, one-click deployment, deploy on GKE, and demo availability. The user interacts with different sections, such as 'Prepare Data,' 'Model Development,' and 'Deploy & Use,' showcasing functionalities like creating datasets, training models, and deploying predictions. The user also explores recent endpoints and service usage metrics, indicating ongoing operations and costs."
}
},
{
"start_time": 2618.32,
"end_time": 2641.98,
"audio": {
"content": " And last week, we announced that AI's full portfolio of open models are also accessible on the Vertex AI model card. With vertex AI, you can be sure your model has access to the right information at the right time. You can connect to any data source or any vector database on any cloud."
},
"video": {
"content": "A speaker is presenting on a stage at a tech conference. The background features a large screen displaying the text \"New Llama 4 on Vertex AI\" and \"Google Cloud Ai2.\" The speaker, dressed in a suit, gestures towards the screen as he explains the topic. The audience is seated in front of the stage, attentively watching the presentation."
}
},
{
"start_time": 2641.98,
"end_time": 2664.98,
"audio": {
"content": " And announcing today, you can build agents directly on our existing net app storage without requiring any data duplication. You can connect to a broad range of applications, including Oracle, SAP, Service Now, and workday. And for model factuality,"
},
"video": {
"content": "A man in a dark blue suit stands on a stage, gesturing with his hands as he speaks. The background is a simple, light-colored curtain. The scene then cuts to a large screen displaying the logos of Google Cloud and NetApp. The man continues to speak, now standing with his hands clasped together at his waist. The video then returns to the initial scene, where the man resumes his gestures while speaking."
}
},
{
"start_time": 2664.98,
"end_time": 2703.98,
"audio": {
"content": " we offer the most comprehensive approach to grounding on the market today. Combining grounding with Google Search, grounding with your own enterprise data, Google Maps, and third-party sources. Let's hear from Deutsche Bank CEO Christian Seweig. For over 150 years, our clients have looked to Deutsche Bank to support their lasting success and financial security."
},
"video": {
"content": "A man in a dark suit stands on a stage, addressing an audience. He is positioned centrally, with his hands clasped together in front of him. The background is a simple, light-colored curtain. The scene then transitions to a large screen displaying a presentation slide about Google Maps. The text on the slide reads: \"Google Cloud Grounding with Google Maps brings fresh, factual information with 100 million updates every day.\" The screen also features a Google Maps icon. The video then cuts back to the man on stage, who continues to speak. The final shot shows a 3D animation of tall buildings against a blue sky."
}
},
{
"start_time": 2703.98,
"end_time": 2738.32,
"audio": {
"content": " And they need us now more than ever as advisor and risk manager in a world marked by uncertainty and shifting geopolitics. Technology plays a key role. Our partnership with Google Cloud place a key role. Our partnership with Google Cloud enables us to take advantage of the latest tools. DBLumina is our AI-powered research agent, built on Gemini and Vertex AI. It maintains data privacy and improves our productivity, while operating in one of the most regulated industries, where trust is built in years and lost in seconds."
},
"video": {
"content": "A man in a blue suit and glasses is speaking passionately in an outdoor setting with modern glass buildings in the background. The scene then transitions to an office where three individuals are gathered around a desk, engaged in a discussion. The video concludes with a graphic of a laptop displaying the word \"Brainstorming\"."
}
},
{
"start_time": 2738.32,
"end_time": 2762.98,
"audio": {
"content": " A tool like DB Lumina allows us to be ahead of our competitors and provide faster, more accurate analysis of data. Recently, there was a big report in the markets which was 400 pages. We put it into DB Lumina and gave it some prompts and within seconds it gave us a three-page summary. We were able to give that to traders and our clients to help them process that information."
},
"video": {
"content": "A man named Jim is sitting at a desk in a modern office setting. He is wearing a dark blue suit jacket over a white shirt and is using a laptop computer. The room has a contemporary design with light-colored walls, a large window with curtains, and some framed artwork on the wall. Jim appears to be engaged in work or study, occasionally looking up and speaking, possibly explaining something or reacting to information on the screen."
}
},
{
"start_time": 2762.98,
"end_time": 2793.15,
"audio": {
"content": " Through our partnership with Google Cloud, we have seen a real breakthrough. And this is just the beginning. We see a future where Generative AI is integrated into basically every process we run, making our employees life easier while meeting the changing expectations of our clients. Thanks so much, Christian."
},
"video": {
"content": "The video begins with a scene of two women working at their desks in an office environment. One woman is wearing headphones and appears to be engaged in a conversation or listening to something on her computer. The other woman is typing on her keyboard, focused on her work. The camera then shifts to show another woman sitting at a desk, also working on her computer. The scene transitions to a man walking through a modern, glass-walled corridor. He is dressed in a blue suit and white shirt, gesturing as if he is explaining something. The final frame shows a futuristic, illuminated room with the words \"The new way to cloud\" displayed prominently."
}
},
{
"start_time": 2793.15,
"end_time": 2814.53,
"audio": {
"content": " We're thrilled to see how quickly you at Deutsche Bank have moved AI from pilot to production. Now let's talk about agents. Agents are intelligent systems that show reasoning, planning, memory, and the ability to use tools. They're able to think multiple steps ahead,"
},
"video": {
"content": "A man in a dark blue suit stands on a stage, gesturing with his hands as he speaks. The background is a gradient of light blue to white, with vertical lines creating a modern and professional atmosphere. The man appears to be giving a presentation or speech, using hand movements to emphasize his points."
}
},
{
"start_time": 2819.82,
"end_time": 2839.32,
"audio": {
"content": " use tools including working with software and systems to get something done on your behalf and under your supervision. Agents work alongside employees to drive efficiencies, to help with decision-making, and increase innovation. A great example of a company working with Google Cloud to develop agents is Salesforce."
},
"video": {
"content": "A man in a dark suit stands on a stage, gesturing as he speaks to an audience. The stage is illuminated with blue lighting, creating a modern and professional atmosphere. Behind him, a large screen displays various graphics and text related to the topic of discussion. The man appears to be explaining or presenting information, using hand movements to emphasize his points."
}
},
{
"start_time": 2839.32,
"end_time": 2861.06,
"audio": {
"content": " Let's hear from CEO Mark Benny Off. Salesforce and Google, two of the world's most innovative companies. We've been on an incredible journey together. And today, well, that partnership has never been stronger."
},
"video": {
"content": "A man in a dark suit stands confidently on a stage, hands clasped in front of him. The background is a simple, light curtain. The scene transitions to a dark screen with the words \"Customer Innovation\" displayed prominently. The next frame shows the word \"Agentic AI\" in bold, colorful letters against a black background."
}
},
{
"start_time": 2861.06,
"end_time": 2881.32,
"audio": {
"content": " Right now, we are really at the start of the biggest shift any of us have ever seen in our careers. I'll tell you, that's why we are so excited about Agent Force and our expanded partnership now with Google. I just love Gemini. I use it every single day. Whether it's Gemini inside Agent Force,"
},
"video": {
"content": "A man stands in front of a large window with a panoramic view of a cityscape, including tall buildings and a body of water. He is wearing a dark blue zip-up jacket over a black shirt and blue jeans. The man gestures with his hands as he speaks, appearing to explain something. The scene then cuts to a close-up of a computer screen displaying a help page from Salesforce, with the text \"How can Agentforce help?\" visible. The video returns to the man, who continues to speak and gesture."
}
},
{
"start_time": 2881.32,
"end_time": 2913.65,
"audio": {
"content": " whether it's Gemini inside agent force, whether it's all the integrations between Google and Salesforce. Together we're leading the digital labor revolution. That's the future that's gonna drive massive gains in human augmentation and productivity, efficiency, the fundamental KPIs of our business and ultimately our customer success. And we're looking forward to doing even more between Salesforce and Google. Thank you very much, Mark."
},
"video": {
"content": "The video begins with a black screen displaying the word \"Agentforce\" in light blue text. The scene then transitions to a man standing in front of a large window with a panoramic view of a cityscape. The man is wearing a dark blue zip-up jacket over a black shirt and blue jeans. He has a beard and is looking directly at the camera. He appears to be speaking and gesturing with his hands as he talks."
}
},
{
"start_time": 2913.65,
"end_time": 2937.31,
"audio": {
"content": " We're excited to build together and continue Thank you very much, Mark. We're excited to build together and continue this journey with you at Salesforce. You know, with Google Cloud, starting today today you can build and manage multi-agent systems with Vertex AI and our new agent development kit. You can scale the adoption of agents across your enterprise"
},
"video": {
"content": "A man stands on a stage, dressed in a dark blue suit jacket over a white dress shirt, with a black belt and a pocket square. He gestures with his hands as he speaks, indicating an engaging presentation. The background is a gradient of light blue to gray, with vertical lines adding depth. The scene transitions to a large screen displaying a Google Cloud logo and text about building and managing multi-agent systems, emphasizing open and comprehensive agent platforms. The man continues to speak, maintaining a professional demeanor."
}
},
{
"start_time": 2937.31,
"end_time": 2957.47,
"audio": {
"content": " with our newly released Google agent space. And you can accelerate deployment with packaged AI agents that are ready for use today. You know, following the ready for use today. Following the introduction of Vertex AI agent builder last year, we're now saying today a new agent development kit."
},
"video": {
"content": "A man in a dark blue suit stands on a stage, gesturing with his hands as he speaks. The background is a simple, light-colored curtain. The scene then cuts to a large screen displaying the Google Cloud logo and text about the platform's features: \"The most open and comprehensive agent platform\". It mentions building and managing multi-agent systems, scaling adoption, and accelerating deployment. The man continues to speak, emphasizing the points on the screen."
}
},
{
"start_time": 2958.11,
"end_time": 2978.31,
"audio": {
"content": " It is a new open source framework that simplifies the process of building sophisticated multi-agent systems. Now you can build sophisticated Gemini powered agents, help them use tools, do complex multi-step tasks, including reasoning or thinking."
},
"video": {
"content": "A speaker is presenting on a stage at an event, likely a conference or seminar. The stage is well-lit with a large screen displaying the text \"New Agent Development Kit\" along with the subtitle \"Open Source framework to build multi-agent systems with simplicity.\" The speaker, dressed in a dark suit and white shirt, gestures with his hands as he speaks, emphasizing points about the new development kit. The audience is seated in front of the stage, attentively watching the presentation."
}
},
{
"start_time": 2978.31,
"end_time": 3011.65,
"audio": {
"content": " You can also discover other agents, learn their skills and enable agents to work together while maintaining precise control. Agent Development Kit supports the Model Context Protocol, which provides a unified way for AI models to access and interact with various data sources and tools rather than requiring custom integrations for each and every one. We're also introducing a new agent-to-agent protocol"
},
"video": {
"content": "A man in a dark blue suit stands on a stage, gesturing with his hands as he speaks. He appears to be presenting at an event, possibly a conference or seminar. The background is a simple, light-colored curtain with vertical stripes. The man's gestures suggest he is explaining something important, likely related to the topic of the presentation."
}
},
{
"start_time": 3011.65,
"end_time": 3031.81,
"audio": {
"content": " that allows agents to communicate with each other, regardless of the underlying model and framework they were developed with. This protocol is supported by many leading partners who share our vision to allow agents to work across the multi-agent ecosystem"
},
"video": {
"content": "A man in a suit stands on a stage, gesturing with his hands as he speaks. The background is a large screen displaying the text \"Agent2Agent Protocol\" and \"A collaborative way to help agents communicate with each other.\" The scene then transitions to a list of partners contributing to the Agent2Agent protocol, displayed on the screen. The man continues to speak, and the camera zooms out to show the entire stage and audience."
}
},
{
"start_time": 3031.81,
"end_time": 3052.65,
"audio": {
"content": " and with agents built on other agent frameworks, including Lange graph and crew AI. Today, we're putting AI agents in the hands of every employee with Google agent space. Employees using Google agents. in space. Employees using Google Agent Space can now find"
},
"video": {
"content": "A man stands on a stage, dressed in a dark suit jacket over a white dress shirt, with a black belt and a pocket square. He gestures with his hands as he speaks, moving them from an open position to a more closed gesture. The background is a simple, light-colored curtain with vertical stripes."
}
},
{
"start_time": 3052.65,
"end_time": 3074.65,
"audio": {
"content": " and synthesize information from within their organization, converse with AI agents, and have these agents take action on their behalf for their enterprise applications. Google Agent Space combines Google Quality Enterprise Search, conversational AI or chat, and Gemini and third-party agents."
},
"video": {
"content": "A man stands on a stage, dressed in a dark suit with a white shirt and a pocket square. He is gesturing with his hands as he speaks, indicating that he is likely delivering a speech or presentation. The background features vertical light panels, creating a modern and professional atmosphere."
}
},
{
"start_time": 3074.65,
"end_time": 3096.11,
"audio": {
"content": " It also includes a broad set of tools, including purpose-built connectors to search and transacts with documents and databases as well as SaaS applications with advanced security and compliance to protect your data and your intellectual property. Let's take a look at Agents' Face in Action."
},
"video": {
"content": "A man in a dark blue suit stands on a stage, gesturing with his hands as he speaks. The background is a simple, light-colored backdrop with vertical lines. The scene then transitions to a large screen displaying the Google Agentspace interface, which includes options for Google-quality search, conversational AI, Gemini and 3p agents, 100+ document repositories and database systems, and security, infrastructure, and compliance controls. The man continues to speak, using hand gestures to emphasize his points."
}
},
{
"start_time": 3096.11,
"end_time": 3117.87,
"audio": {
"content": " Please welcome Gabe Weiss. Thanks, Thomas. So for the next few minutes, I'm going to be a relationship manager at a bank. Starting with a quick tour, this is my homepage, authenticated and personalized just for me."
},
"video": {
"content": "A man in a dark blue suit with a white shirt and a black bow tie is standing on a stage. He is gesturing with his right hand as he speaks. The background is a plain, light-colored curtain."
}
},
{
"start_time": 3117.87,
"end_time": 3139.23,
"audio": {
"content": " The agent gallery lets me see my company's approved selection of purpose-built agents, including ones powered by third-party models like Lama and Claude. You see we've got some Google made agents. We have agents that my bank has made available to me, either ones we've created or ones built by partners. And then the best part, my own personal agents, which I can build directly"
},
"video": {
"content": "A person is standing behind a podium on a stage, gesturing with their hands as they speak. The background features a large screen displaying a Google Agentspace interface. The screen shows various options such as 'Deep Research,' 'Idea Generation,' 'Looker,' 'Cash flow,' 'Client analytics,' 'Credit memo,' and 'Policy checker.' The person appears to be explaining or demonstrating these features."
}
},
{
"start_time": 3139.23,
"end_time": 3160.31,
"audio": {
"content": " inside agent space with this button over here, or even easier just from having a little conversation. Let's see how easy it is to create an agent to automate a daily task. Now it's critical for me to stay on top of what's going on with my clients. So I start every morning with a portfolio analysis. I'm going to use a clipboard because no one wants to watch me typing."
},
"video": {
"content": "A man with long hair and a green shirt is standing in front of a laptop, gesturing with his hands as he speaks. The screen then transitions to a Google Agentspace interface, displaying an 'Agent gallery' section with various customizable agents. The man appears to be explaining or demonstrating something related to these agents. The interface shows options like 'Cash flow,' 'Client analytics,' 'Credit memo,' and 'Policy checker.' The man continues to speak and gesture, likely providing instructions or highlighting features of the agents."
}
},
{
"start_time": 3160.31,
"end_time": 3181.31,
"audio": {
"content": " Run an analysis of my client portfolio and identify potential risks and opportunities. This only uses information that I have permission to access. It knows which clients are mine and summarizes top points from my data sources like OneDrive, Salesforce, are done in Bradstreet's. If I have questions, I have a direct link to my sources here,"
},
"video": {
"content": "A man with long hair and a green shirt is standing in front of a laptop screen. He appears to be giving a presentation or lecture. The screen displays a chat interface labeled \"Google Agentspace.\" The chat interface shows a conversation with the text \"Hello, Gabe. What can I help you with?\" followed by several questions about AI models, fraud detection, blockchain technology, and client portfolio analysis. The man gestures with his hands as he speaks, emphasizing his points."
}
},
{
"start_time": 3181.31,
"end_time": 3205.8,
"audio": {
"content": " and if I need even more I have a direct link to my sources here. And if I need even more control, I can refine that list of sources. But agent space doesn't just summarize information. It's interpreting my question and surfacing what matters most. For example, in this chart, I can see Agent Spaces flag that Acme General Contracting might have some cash flow problems in the future. Already, it's given me a massive report which is going to save me a ton of"
},
"video": {
"content": "A man with long hair and a beard is standing in front of a screen, gesturing with his hands as he speaks. The screen displays a document titled \"Client Risk & Opportunity Analysis.\" The document includes an executive summary that outlines the purpose of the analysis, which is to identify key trends, potential risks, and opportunities for relationship deepening and revenue growth. The analysis focuses on identifying unmet needs and proactively offering solutions tailored to each client's specific financial situation and business goals. The portfolio demonstrates a healthy mix of industries, but proactive engagement is crucial to maximizing client lifetime value and mitigating potential risks."
}
},
{
"start_time": 3205.8,
"end_time": 3225.98,
"audio": {
"content": " manual research and I can go ahead and read this later, but for now let's set up an agent so I can keep an eye on acne. Agent space automatically generates an agent plan based on our conversation so far. And this is good, but I think I want more. I'm going to have it generate an audio summary and send it to my inbox"
},
"video": {
"content": "A man with long hair and a beard is standing in front of a laptop screen, gesturing with his hands as he speaks. The screen displays a document titled \"Client Risk & Opportunity Analysis\" from Google Agentspace. The document outlines key trends and observations, relationship deepening opportunities, and risk mitigation recommendations. The man appears to be explaining these points, possibly during a presentation or tutorial."
}
},
{
"start_time": 3225.98,
"end_time": 3246.1,
"audio": {
"content": " so I can listen to it on my morning commute. And just like that, to it on my morning commute. And just like that, I have built my own custom agent to use whenever I want without writing a single line of code. Now, Agent Space has identified a cash flow problem with Acme General contracting. I need to dig into that. Maybe this is a problem with construction in general and not specific to Acme."
},
"video": {
"content": "A man with long hair and a green shirt is speaking into a microphone while standing in front of a laptop. The screen displays a Google Agentspace interface with a task titled \"Client Risk & Opportunity Analysis.\" The man appears to be explaining or presenting something related to this task. The interface shows a progress bar indicating that an answer is being generated. After a few moments, the interface updates to show a new task titled \"Client Risk & Opportunity Agent\" with a description and suggested follow-ups. The man continues to speak and gesture as he explains the task."
}
},
{
"start_time": 3246.92,
"end_time": 3269.65,
"audio": {
"content": " Agent Space has already identified that possibility as a suggested follow-up. So now let's go ahead and deep dive into general contracting industry trends. This activates Google's enterprise deep research agent, which starts by telling me what it plans to research and in what order. At this point, I could edit this plan if I wanted to, but it looks pretty good, so I'll start the research."
},
"video": {
"content": "A man with long hair and a green shirt is standing in front of a laptop screen, gesturing with his hands as he speaks. The screen displays a Google Agentspace interface titled \"Client Risk & Opportunity Analysis.\" The interface shows an updated \"@Client Opportunity Agent\" created based on a plan above. Below this, there are suggested follow-ups: \"Deep dive into general contracting industry trends,\" \"Review Treasury Management offerings,\" and \"Deep dive into electrical/plumbing growth.\" The interface also includes a \"Research Plan\" section with questions about the current state of the general contracting industry, key drivers and trends, challenges and opportunities, technological advancements and adoption, regulatory and legal landscape, and competitive landscape."
}
},
{
"start_time": 3269.65,
"end_time": 3293.65,
"audio": {
"content": " Now I do want to call out. We've cashed the plan and the results here. Normally this would take a little bit longer. This agent is pulling in real-time information from Google Search to build its report. But even cooler, it's also searching my internal enterprise data and adjusting this plan in real time, adding additional questions based on what it's going to find along the way. And again, an incredibly insightful analysis,"
},
"video": {
"content": "A man with long hair and a beard is standing in front of a laptop screen, gesturing with his hands as he speaks. The screen displays a Google Agentspace interface titled \"Client Risk & Opportunity Analysis.\" The interface includes a search bar at the bottom where questions can be typed. The man appears to be explaining or presenting information related to the US general contracting industry, discussing various aspects such as size, growth rate, health, major factors driving growth, challenges, regulations, adoption rates of BIM, AI, and drones, competitive strategies, and forecasts for future growth."
}
},
{
"start_time": 3293.65,
"end_time": 3315.71,
"audio": {
"content": " including some source links, but thankfully here at the bottom, it also is going to give me a great succinct executive summary. Let's take a quick look at this. succinct executive summary. Let's take a quick look at this. Yep, I can see Acme General Contracting is likely being affected by rising material costs, supply chain disruptions, and regulatory complexities that pose significant hurdles."
},
"video": {
"content": "A man with long hair and a beard is standing at a podium, speaking into a microphone. He gestures with his hands as he talks, emphasizing his points. The background is a plain, light-colored wall. On the right side of the screen, there is a Google Agentspace interface displaying information about the US general contracting industry, including key drivers, trends, and an executive summary. The text on the screen provides details about the industry's growth, challenges, and future projections."
}
},
{
"start_time": 3315.91,
"end_time": 3336.31,
"audio": {
"content": " That's really great. I mean, okay, maybe not for acne. But the analysis is really great. I don't want Acme to be surprised by this at all. So I'm going to have our bank's cash flow agent do some forecasting across the next three quarters for me. This agent uses Google's new time series forecasting model, which is specifically trained for scenarios just like this."
},
"video": {
"content": "A man with long hair and a green shirt is standing at a podium, gesturing with his hands as he speaks. He appears to be giving a presentation or lecture. The background is dark, and there is a screen displaying an executive summary about the US general contracting industry. The summary mentions significant growth but also challenges such as labor shortages, rising material costs, supply chain disruptions, and regulatory complexities. It also discusses technological advancements beyond BIM, AI, and drones, which are gaining traction but face adoption challenges. Regional variations in growth exist, with the South showing particularly strong performance. Future projections indicate continued growth across all sectors, although precise rates vary depending on the source and sector. Major players employ diverse competitive strategies to succeed in this competitive market. More precise market size data by sector is needed for a more complete picture."
}
},
{
"start_time": 3336.31,
"end_time": 3366.98,
"audio": {
"content": " And again, I'm going to get a super clear, very clear summary with at the bottom, some great recommended steps for Acme. And I need them to see it right away so I can ask agent space, draft me an email to Acme General Contracting CEO requesting a meeting for next week. And, just like that, I've got the draft ready to go, and even better, I can send it off directly from within agent space, so I don't even have to switch to Outlook or Gmail."
},
"video": {
"content": "A man with long hair and a beard is standing at a podium, speaking into a microphone. He is wearing a green polo shirt and appears to be giving a presentation. The background is a plain, dark-colored wall. On the right side of the screen, there is a chat interface labeled 'Google Agentspace' with a conversation about client risk and opportunity analysis. The text in the chat mentions calculating Acme General Contracting's cash flow and liquidity needs for the next three quarters. The conversation also includes recommendations for Acme, such as delayed client payments, negotiating with suppliers, securing a line of credit, and improving project management. The chat interface also shows a draft email being composed to Acme General Contracting's CEO, Sophia, requesting a meeting for the next week."
}
},
{
"start_time": 3366.98,
"end_time": 3387.98,
"audio": {
"content": " I'm all set. An agent space has saved my session. I'm all set. An agent space has saved my session so I can prep for that meeting right where I left off whenever I'm ready. Let's go ahead and recap. While I don't actually work for the Let's go ahead and recap. While I don't actually work for a bank, the value that Agent Space adds is very real. It's so easy to interact with all of your enterprise data and tools in one place and build and use agents directly from that conversational workflow."
},
"video": {
"content": "A man with long hair and a green shirt is standing in front of a screen, gesturing with his hands as he speaks. The screen displays a Google Agentspace interface with various options and prompts. The man appears to be explaining something, possibly related to business or technology, as he interacts with the interface."
}
},
{
"start_time": 3387.98,
"end_time": 3413.65,
"audio": {
"content": " Powered by Gemini 2. that conversational workflow. Powered by Gemini 2.5 and Google search technology. Agent Space is the only hyper-scaler platform on the market that can connect to third-party data and tools and offers interoperability with third-party agents and models. For companies with strict regulatory needs, like a bank, agent space provides stringent access controls at the employee level and can operate within your own VPC,"
},
"video": {
"content": "A man with long curly hair and a beard is standing behind a podium, speaking into a microphone. He is wearing a green polo shirt over a white t-shirt. The background consists of vertical blinds. The man gestures with his hands as he speaks, occasionally clapping them together or pointing with one finger. He appears to be engaged in delivering a speech or presentation."
}
},
{
"start_time": 3413.65,
"end_time": 3437.31,
"audio": {
"content": " ensuring that your data stays yours while meeting all of your requirements. Agent Space is a game changer, and we can't wait to see how you all put it to work. Thanks. Back to the time. and we can't wait to see how you all put it to work. Thanks. Back to you, Gabe. Today we're excited to announce that Agent Space is integrated with your Chrome browser"
},
"video": {
"content": "A man with long hair and a beard is standing on a stage, wearing a green polo shirt. He is speaking and gesturing with his hands, indicating that he is giving a presentation or speech. The background is a plain curtain, and there is a laptop on a stand in front of him."
}
},
{
"start_time": 3437.31,
"end_time": 3461.31,
"audio": {
"content": " to allow users to search and access your enterprise data directly from the search box in Chrome. Employees can use agent space to access Google built expert AI agents, including Notebook LN, an AI-powered notetaking and research agent that allows users to upload up to 50 documents with 25 million words,"
},
"video": {
"content": "A man in a dark blue suit stands on a stage, addressing an audience. He is positioned centrally, with his hands clasped together in front of him. The background features a large screen displaying text about Google Agentspace integrated into Chrome. The stage is well-lit, with a modern design featuring vertical light panels and a sleek, curved floor. The audience is seated in rows, attentively watching the presentation."
}
},
{
"start_time": 3461.31,
"end_time": 3484.98,
"audio": {
"content": " and then query them using AI effectively turning notes and sources into a virtual research assistant. You can also use our idea generation agent, which accelerates innovation, brainstorming, and problem solving. It uses a tournament style framework to rank ideas based on employee defined criteria, refine them, and generate new ones."
},
"video": {
"content": "A man in a suit stands on a stage, gesturing as he speaks to an audience. The stage is well-lit, with a large screen behind him displaying various applications and tools. The screen transitions between different interfaces, including a notebook application named \"NotebookLM\" and a tool called \"Idea Generation Agent.\" The man appears to be explaining these features, likely during a presentation or conference."
}
},
{
"start_time": 3484.98,
"end_time": 3517.31,
"audio": {
"content": " And our enterprise deep research agent, which Kate just showed you, researches complex topics on your behalf and provides you with findings in a comprehensive, easy-to-read report. Customers and partners around the world are already using agent space. KPMG is building Google AI into their newly formed KPMG law firm and implementing agent space to enhance their own workplace operations."
},
"video": {
"content": "A man in a dark suit stands on a stage, gesturing with his hands as he speaks. The background is a simple, modern design with vertical light panels. The scene transitions to a screen displaying text about blockchain technology and its implications in the fintech industry. The text includes mentions of regulatory bodies like the Basel Committee on Banking Supervision (BCBS) and the Financial Stability Board (FSB). The video then returns to the man on stage, who continues to speak and gesture."
}
},
{
"start_time": 3517.31,
"end_time": 3539.31,
"audio": {
"content": " Cohesity is integrating with agent space to provide employees with greater data discovery for better decision-making while also increasing security and threat protection. Gordon Food Services is simplifying insight discovery and recommending next steps. Rubrik is leveraging agents to develop deeper"
},
"video": {
"content": "A man in a dark suit stands confidently on a stage, addressing an audience. The background is a simple, modern design with vertical light panels. As he speaks, the scene transitions to a large screen behind him displaying a presentation slide. The slide features the text \"COHESITY\" and \"Enabling better decision making with Agentspace.\" It also includes images of a man working at a desk with two laptops and some documents, suggesting a professional setting. The presenter continues to speak, and the slide changes to another presentation slide titled \"Gordon FOOD SERVICE\" with the subtitle \"Transforming decision-making with Agentspace.\" This slide shows a woman in a yellow shirt and black apron using a tablet in a food service environment, surrounded by fresh produce. The presenter remains on stage throughout, engaging with the audience."
}
},
{
"start_time": 3539.31,
"end_time": 3561.13,
"audio": {
"content": " customer insights and prepare for impactful sales interaction. An agent space will provide Wells Fargo Bank the unique opportunity to modernize and simplify banking. We're now going to dive deep into five categories of agents where we're already seeing tremendous business impact."
},
"video": {
"content": "A professional speaker is presenting on a large stage at a conference or event. The stage features a large screen displaying text and images related to technology and business solutions. The speaker, dressed in a suit, gestures with his hands as he addresses an audience. The background includes a large screen with the Rubrik logo and the text \"Agentspace enables faster, smarter sales support.\" The scene transitions to another screen showing the Wells Fargo logo and the text \"Modernizing and simplifying banking.\" The speaker continues to present, moving around the stage and engaging with the audience."
}
},
{
"start_time": 3561.13,
"end_time": 3585.98,
"audio": {
"content": ""
},
"video": {
"content": "A man in a dark blue suit stands on a stage, hands clasped in front of him. The background is a simple, light-colored curtain. The scene then transitions to a wide shot of a large stage with a green and blue abstract design on the screen behind the speaker. The speaker, a woman in a green jumpsuit, walks across the stage, gesturing as she speaks. The camera focuses on her as she continues her presentation."
}
},
{
"start_time": 3561.13,
"end_time": 3606.22,
"audio": {
"content": " Please welcome Lisa O'Malley. Thank you. Thanks Thomas. Let's start with customer agents. They can synthesize and reason across all types of multimodal information, including text, audio, images, and video. Communicate and engage naturally with human-like speech and dialogue. Connect to cross the enterprise applications and take actions on behalf of the user. And be used in the contact center and on the web, on devices, in stores, in cars, and more."
},
"video": {
"content": "A man in a dark blue suit stands on a stage, hands clasped in front of him. The background is a simple, light-colored backdrop with vertical lines. The scene then transitions to a woman walking across the stage. She is wearing a green jumpsuit and glasses. The text \"Lisa O'Malley\" appears on the screen."
}
},
{
"start_time": 3607.48,
"end_time": 3638.31,
"audio": {
"content": " Customer agents built with Vertex AI search are helping customers to quickly find answers and the right products using both text and images in search queries. Let's hear from Reddit chief product officer, Polly Bot. A dread At Reddit, our mission is to empower communities"
},
"video": {
"content": "A woman stands on a stage, addressing an audience. She is dressed in a green outfit and wears glasses. The background features a large screen displaying the text \"Customer Agents.\" The scene transitions to a darker setting where the words \"finding answers\" appear prominently."
}
},
{
"start_time": 3638.31,
"end_time": 3660.31,
"audio": {
"content": " and make their knowledge accessible to all. We've been working on this mission for nearly 20 years, which in turn has made Reddit one of the Internet's largest sources of authentic conversations. With that vast amount of conversations and perspectives, we wanted to build a unique search product that's powered with AI, but still grounded in all of the real conversations and perspectives that are available on Reddit."
},
"video": {
"content": "A man named Pali Bhat, identified as the Chief Product Officer of Reddit, stands in an office environment. He is dressed in a dark blue shirt and black pants, with his hands clasped together in front of him. The office has a modern design with large glass walls, colorful chairs, and a cozy seating area. The lighting is bright, creating a professional atmosphere. The video transitions to a live chat interface where a user is asking about AI-powered search features on Reddit. The video then cuts to a black screen with the text \"AI powered search\" followed by the Reddit and Cloudinary logos."
}
},
{
"start_time": 3660.31,
"end_time": 3681.17,
"audio": {
"content": " This is why we introduced Reddit Answers, a new AI-powered way to get information, recommendations, and discussions on virtually any topic. It provides powerful AI that's grounded in Redator's existing posts and conversations. So it shows you more of what real humans think versus creating unverifiable perspectives on its own."
},
"video": {
"content": "A person is holding a smartphone and typing on it. The screen displays a Reddit page with various posts and comments. The user searches for information about how to avoid jet lag when traveling. The search query appears prominently on the screen, and the user scrolls through the results, revealing different tips and suggestions from other users."
}
},
{
"start_time": 3681.17,
"end_time": 3702.98,
"audio": {
"content": " Red answers is different from any other generative AI product on the market. It leverages Vertex AI search to make finding the answers and perspectives people seek faster and more relevant. We've seen awesome results so far because the users who have been able to access this product and tested out really love the experience. This gets them to the heart of the conversations that they were looking for right away."
},
"video": {
"content": "The video begins with a black screen featuring a colorful speech bubble icon. The icon consists of three vertical bars: a teal bar on the left, a red bar in the middle, and a yellow bar on the right. The icon rotates slightly, revealing its three-dimensional appearance against the dark background."
}
},
{
"start_time": 3702.98,
"end_time": 3726.98,
"audio": {
"content": " That's the magic of Reddit answers. It combines AI with the power of Reddit. Thank you, Polly. We've also introduced Vertix AI search for healthcare and retail,"
},
"video": {
"content": "A man in a green polo shirt is seen pointing at something off-screen. The scene then transitions to an office environment where a man is looking at a neon sign on the wall. The video then cuts to a logo that reads \"The new way to cloud.\" followed by a woman standing on a stage, smiling and speaking."
}
},
{
"start_time": 3726.98,
"end_time": 3747.06,
"audio": {
"content": " making it super easy for doctors, nurses, and providers to rapidly search and analyze patient data, including x-rays, scans, images, and medical histories. Retailers can add product discovery to their websites, powered by Google Search. This helps them deliver hyper-relevant results"
},
"video": {
"content": "A woman stands on a stage, presenting about Vertex AI Search. The background features a large screen displaying the Vertex AI Search logo and images of medical professionals using the technology. The presenter gestures towards the screen as she speaks, emphasizing the benefits and applications of the AI search system. The audience is visible in the foreground, attentively listening to her presentation."
}
},
{
"start_time": 3747.06,
"end_time": 3769.94,
"audio": {
"content": " and personalized recommendations for each customer, boosting conversion rates and maximizing revenue per shopper. We're seeing huge momentum for Vertex AI search with billions of daily queries executed by our customers. For example, by our customers. For example, Lowe's is revolutionizing product discovery with Vertix AI Search to generate dynamic product recommendations"
},
"video": {
"content": "A woman stands on a stage, dressed in a green jumpsuit with a belt and white sneakers. She is speaking and gesturing with her hands, occasionally clasping them together in front of her chest. The background features vertical light panels that create a modern and minimalist atmosphere. As she continues to speak, the camera zooms out to reveal a large screen behind her displaying a blue and white logo."
}
},
{
"start_time": 3769.94,
"end_time": 3792.64,
"audio": {
"content": " and address customers complex queries. Globo created a recommendations experience inside its streaming platform that more than doubled click-through play. And let's hear how Mercado Libre is transforming how customers discover products that they love. Mercado Libre. Latin America's e-commerce"
},
"video": {
"content": "A woman stands on a stage, presenting to an audience. The background features a large screen displaying images of smartphone interfaces. The first image shows a smartphone screen with a search bar and product listings from Lowe's, highlighting the company's use of Vertex AI Search for product discovery. The text on the screen reads \"Revolutionizing product discovery, powered by Vertex AI Search.\" The second image displays a smartphone screen from Globoplay, showing a list of video titles with the text \"2x more video plays.\" The woman gestures towards the screen as she speaks, emphasizing the benefits of the technology being showcased."
}
},
{
"start_time": 3792.64,
"end_time": 3813.3,
"audio": {
"content": " leader has deployed a verdicts AI search across 150 million items in three pilot countries. This multimodal search technology understands deep meaning across text and images, not just keywords. It is helping their 100 million customers find the products they love faster. Already delivering millions of dollars in incremental revenue."
},
"video": {
"content": "The video begins with a view of Earth from space, with the word \"e-commerce\" prominently displayed in the center. The scene then transitions to a collage of images featuring various items, including a security camera, a person wearing a tie, and a smartphone. The text \"150M items\" is displayed in large yellow letters, emphasizing the vast number of products available online. The next scene shows a close-up of a chocolate chip cookie on a yellow background, with the text \"What ingredients do I need for choc-chip cookies?\" appearing above it. The final scene captures a person opening a box, revealing a product inside."
}
},
{
"start_time": 3813.68,
"end_time": 3833.64,
"audio": {
"content": " Mercado Libre is delivering a new way to shop. Google Cloud's own purpose-built customer engagement suite is transforming customer service. Grounded in your company's data, it provides out-of-the-box functionality"
},
"video": {
"content": "A woman in a green jumpsuit stands on a stage, gesturing as she speaks. The background is a large screen displaying the Google Cloud logo and the Mercado Libre logo. The scene transitions to a wide shot of the stage, showing the audience and the large screen behind the speaker. The video then cuts back to the woman, who continues her presentation."
}
},
{
"start_time": 3833.64,
"end_time": 3858.31,
"audio": {
"content": " to build agents and works across web, mobile, call center, in store, and with third-party telephony and CRM systems. These unique capabilities have led to rapid growth with an increase in conversational AI agent usage. DBS, a leading Asian financial services group Group is reducing customer call handling times by 20%."
},
"video": {
"content": "A woman stands on a stage, dressed in a green jumpsuit with a black belt and white sneakers. She is speaking to an audience, gesturing with her hands as she explains something. The background is a simple, light-colored curtain with vertical stripes. The scene then transitions to a large screen displaying the text \"Customer Engagement Suite with Google AI.\" The woman continues to speak, now standing in front of this screen, which is part of a larger stage setup with a dark floor and illuminated panels."
}
},
{
"start_time": 3860.31,
"end_time": 3879.31,
"audio": {
"content": " Love Holidays saved 20% of their customer service cost per year. And our very own YouTube achieved a 75% reduction in calls abandoned while waiting to speak to a representative. Now, let's hear how Verizon is improving their customer experience using AI agents."
},
"video": {
"content": "A woman stands on a stage, dressed in a green button-up shirt and matching pants. She is wearing glasses and has short hair. The background is a simple, light-colored backdrop with vertical lines. The woman appears to be speaking, using hand gestures as she moves slightly from side to side. At one point, a large screen behind her displays a red graphic with text that reads \"YouTube 75% reduction of calls abandoned in the queue.\""
}
},
{
"start_time": 3881.31,
"end_time": 3904.98,
"audio": {
"content": " Verizon is transforming how they serve over 115 million connections with Google Cloud's Customer Engagement Suite. with Google Cloud's customer engagement suite. Their personal research assistant uses AI to provide 28,000 care representatives with instant personalized information about a customer's unique needs leading to faster and more satisfying resolutions for even the most complex inquiries."
},
"video": {
"content": "A large screen displays an image of the United States map, with red vertical lines on either side. A person walks across a stage in front of this screen. The scene transitions to a woman talking on her phone in a cozy living room. The text \"115 million connections\" appears on the screen. The next scene shows a group of people wearing headsets, with the text \"27,999 care representatives.\" The final frame shows a black screen with the text \"Does T.\""
}
},
{
"start_time": 3904.98,
"end_time": 3929.64,
"audio": {
"content": " With Customer Engagement Suite, Verizon is elevating its service experience, reducing wait times, and delivering exceptional support at massive scale. Verizon developed a new way to personalize customer service. The business impact that Verizon experienced is nothing short of extraordinary."
},
"video": {
"content": "A woman is sitting at a desk in a modern kitchen, smiling as she talks on her phone while working on her laptop. The scene transitions to a different setting where the same woman is standing in a living room, talking on her phone. The video then cuts back to the kitchen scene, showing the woman continuing her conversation. The final scene shows a woman standing on a stage, speaking to an audience."
}
},
{
"start_time": 3929.64,
"end_time": 3953.64,
"audio": {
"content": " Today, we're announcing our next generation of customer engagement suite, which will include human-like voices, comprehension, and the ability to understand emotions. so agents can adapt better during the conversation. Streaming video support, so virtual agents can respond, can interpret and respond to what they see"
},
"video": {
"content": "A woman stands on a stage, dressed in a green jumpsuit with a belt, addressing an audience. The background is a simple, dark setting with vertical light panels. The scene transitions to a large screen displaying the text \"New Customer Engagement Suite Human-like voices\" with a \"Coming Soon\" button. The screen then changes to \"New Customer Engagement Suite Understanding emotions\" with another \"Coming Soon\" button. Finally, the screen shows \"New Customer Engagement Suite Streaming video support\" with yet another \"Coming Soon\" button."
}
},
{
"start_time": 3953.64,
"end_time": 3978.31,
"audio": {
"content": " in real time through customer devices. AI assistance to build custom agents in a no-code interface and the ability to use a variety of tools through API calls to interact and perform specific tasks for your application like look up products, add to cart, or checkout. An integration with data sources, CRM systems,"
},
"video": {
"content": "A woman stands on a stage, dressed in a green jumpsuit with a black belt and white shoes. She gestures with her hands as she speaks, likely addressing an audience. The background is simple, featuring vertical light panels that create a modern and professional atmosphere. As she continues to speak, the camera shifts to reveal a large screen behind her displaying text about a new customer engagement suite. The text highlights features such as AI assistance to build agents and the ability to interact with other applications. The audience is visible in the foreground, attentively listening to the presentation."
}
},
{
"start_time": 3978.31,
"end_time": 4002.47,
"audio": {
"content": " and popular business messaging platforms. Now, let's see a demo of all this cool stuff in action. Welcome my teammate, Patrick Marlow, to the stage. All right, thanks, Lisa. Hey everyone, I'm Patrick Marlow, a product manager here at Google Cloud,"
},
"video": {
"content": "A woman stands on a stage, presenting information about new customer engagement suite integrations. The scene transitions to a man standing at a podium, preparing to speak. The stage is well-lit with spotlights and a large screen displaying the name \"Patrick Marlow\" and his title as a Product Manager for Applied AI."
}
},
{
"start_time": 4002.47,
"end_time": 4022.97,
"audio": {
"content": " and I am stoked to be here today showcasing our next generation customer engagement suite in action. To be honest, I'm even more excited to screws up our keynote stage. I was thinking some greenery and flowers might be nice. You know, I've already made a couple of trips to the hardware store this morning, and I still forgot to pick up potting soil. Classic. forgot to pick up potting soil. Classic."
},
"video": {
"content": "A man with a beard and tattoos on his arms is standing on a stage, speaking to an audience. He is wearing a dark blue t-shirt and black pants. The background is a plain, light-colored wall with vertical blinds. To his left, there is a podium with a microphone and a laptop. In front of him, there is a small wooden crate filled with colorful flowers. The man gestures with his hands as he speaks, occasionally looking at the audience."
}
},
{
"start_time": 4022.97,
"end_time": 4043.97,
"audio": {
"content": " So let's see how a next-gen agent can hopefully help me get this last order, correct. We're going to start a brand new voice interaction with our agent here. Hi there, welcome to Simple Home and Garden. Is this Patrick? Hey, yeah, this is Patrick. Good morning. How are you? Good morning. How are you? Good morning to you too, Patrick."
},
"video": {
"content": "A man with a beard and tattoos on his arms is standing on a stage, holding a wooden crate filled with flowers. He is wearing a dark blue t-shirt and has a microphone attached to his shirt. The background is a plain, light-colored wall with vertical blinds. The man appears to be speaking or presenting something."
}
},
{
"start_time": 4043.97,
"end_time": 4065.97,
"audio": {
"content": " Doing great. Thanks for asking. It looks like you've been shopping with us recently. I see you have a few things in your card already. What can I help you with today? Yeah, so I just bought some plants and I'm trying to pick out potting soil and fertilizer, and I'm not sure if I'm getting the right stuff. Could you help me with that? I can definitely help with that, Patrick. To give you the best recommendations, could you show me the plants you bought?"
},
"video": {
"content": "A customer is shopping on a gardening website during springtime, taking advantage of a 20% discount offer. The screen shows various sections such as 'Tools and accessories,' 'Plants and supplies,' 'Plant care and maintenance,' and 'Landscaping services.' The customer adds two items to their cart: Standard Potting Soil and Standard Fertilizer. The cart summary indicates a subtotal of $42.97, tax of $3.60, and a total of $46.57. The customer then proceeds to checkout."
}
},
{
"start_time": 4065.97,
"end_time": 4087.97,
"audio": {
"content": " I can take a look and help you choose the best soil and fertilizer. I'm sending a video request to your browser now. Great. I can see you. Just hold the plants up to the camera so I can help identify them. Yeah, so I have about 40 of these in various different colors. Those are beautiful petunias."
},
"video": {
"content": "A customer service agent is assisting a customer over a video call on a shopping app. The customer's screen shows a cart with two items: Standard Potting Soil priced at $12.99 each and Standard Fertilizer priced at $16.99 each. The total amount is $46.57. The customer is considering whether to proceed with the purchase by checking out. The customer service agent is asking if they can access the camera for better communication."
}
},
{
"start_time": 4087.97,
"end_time": 4108.64,
"audio": {
"content": " They're a great choice for vibrant color. Let me take a closer look at the items in your cart. Okay, I see you have standard potting soil and general purpose fertilizer. That could work, but petunias really thrive in a rich well-draining soil, and a fertilizer specifically for flowering plants. Okay, I've found some recommendations for your petunias. Yeah, I'm taking a look at these now."
},
"video": {
"content": "A man with a beard is standing behind a podium, speaking into a microphone. He appears to be giving a presentation or demonstration. The screen then transitions to a shopping cart interface where he is shown selecting items. The items in the cart include Standard Potting Soil and Standard Fertilizer. The man then navigates to a product page for Bloom Booster Potting Mix, adding it to his cart."
}
},
{
"start_time": 4108.64,
"end_time": 4131.64,
"audio": {
"content": " They look fantastic, thanks. Great! Would you like me to remove the standard potting soil and general purpose fertilizer and add the Bloom Booster potting mix and flower power fertilizer to your cart? Yeah, let's go ahead and do that. Okay, I've updated your cart. Is there anything else I can help you with today? I mean, not unless you can come over Anything else I can help you with today? I mean, not unless you can come over and plant these for me."
},
"video": {
"content": "A customer service agent is interacting with a customer on a mobile app. The customer service agent is asking if they should update the items in the customer's cart. The customer service agent shows two products: Bloom Booster Potting Mix and Flower Power Fertilizer. The customer service agent then asks if they should update the items in the cart."
}
},
{
"start_time": 4131.64,
"end_time": 4153.44,
"audio": {
"content": " Funny, you should mention that. We actually do offer professional landscaping services. Would you be interested in a quote? Yeah, I mean how much would something like that cost. For planting 40 petunias, I would suggest a four-hour appointment which would come out to $200. Okay, that's interesting. I'm curious, do you also price match?"
},
"video": {
"content": "A customer is shopping on a gardening website. The screen shows a summary of their cart, which includes two items: Bloom Booster Potting Mix and Flower Power Fertilizer. The customer has added these items to their cart twice each. The subtotal is $77.94, with a tax of $6.53, making the total $84.47. The customer is considering adding a landscaping service offering worth $200.00 to their cart."
}
},
{
"start_time": 4154.84,
"end_time": 4182.97,
"audio": {
"content": " Yes, we do price match on occasion, but it's usually on a case-by-case basis. Oh, in that case, I most definitely saw an ad from one of your competitors, Greenscapes, where they were offering 50% off first-time landscaping services for new customers, you should totally give me that 50% discount. Ologies, but it looks like that amount is beyond what I'm authorized to approve. Let me check with my supervisor to see what we can do. Just a moment."
},
"video": {
"content": "A customer is shopping on an online gardening store. The screen shows a shopping cart with two items: Bloom Booster Potting Mix priced at $15.99 each and Flower Power Fertilizer priced at $22.98 each. The subtotal is $77.94, with a tax of $6.53, making the total $84.47. The customer has the option to add a promo code and proceed to checkout. On the right side, there is a customer service agent offering a landscaping service for $200.00."
}
},
{
"start_time": 4182.97,
"end_time": 4204.03,
"audio": {
"content": " So I'm going to step away from being the human user in this conversation for a moment, and we're going to go behind the scenes over here to our call center. And so what you're going to see here in our CRM sales force is that our virtual assistant, our virtual agent, has escalated up to a human call center manager, also played by me, and what I can do here is guide the agent through this portion of the conversation without fully taking over the"
},
"video": {
"content": "A man with a beard and tattoos is standing behind a podium, speaking into a microphone. He gestures with his hands as he talks. The scene then transitions to a computer screen displaying a customer service chat interface. The chat interface shows a conversation with a customer named Patrick Marlow, who has started a chat. The chat component includes options to message, call, or view history. The background features a large screen displaying the same man on stage, with a modern, illuminated desk in front of him."
}
},
{
"start_time": 4204.03,
"end_time": 4225.97,
"audio": {
"content": " conversation. So you can see here, Patrick is requesting a price match for 50% offer. And I'll say something like, you know, we can't do that, but we can do 20, oh, say we can do 20% instead. And I'm going to send this message, and then we're going to pick up the conversation"
},
"video": {
"content": "A man with a beard and a bald head is sitting at a desk, looking at a computer screen. He is wearing a dark blue shirt and has tattoos on his arms. The screen shows a chat interface with messages from a virtual agent. The man types a message to the virtual agent, which reads: 'we can't do that but we can d...'. The chat interface also displays other messages and options such as 'Share', 'Call Logs', 'Text Posts', and 'Status Changes'. On the right side of the screen, there is a sidebar with related information like 'Attachments', 'Activity History', and 'Open Activities'. The background is a simple indoor setting with a curtain."
}
},
{
"start_time": 4225.97,
"end_time": 4251,
"audio": {
"content": " on the other side. Good news, Patrick. My supervisor has approved a discount. We can offer you 20% off our landscaping services. I've applied a 20% discount to the landscaping services. Well, that's awesome. I'll take it. Fantastic. Let's get that scheduled. I can have someone out tomorrow from 8 a.m. to 12 p.m. or between 1 p.m. and 5 p.m."
},
"video": {
"content": "A man with a beard is standing behind a podium, speaking into a microphone. He is wearing a dark blue shirt and has tattoos on his arms. The background is a dark stage with red and orange lighting. On the screen behind him, there is a chat component showing a conversation between a customer and an agent. The customer is requesting a price match with a 50% off offer for planting services. The agent responds that they can only offer a 20% discount instead. The chat component also shows the agent's name, Patrick Marlow, and the task assigned to them. The video then cuts to a screenshot of a shopping cart on a website, displaying two items: Bloom Booster Potting Mix and Flower Power Fertilizer. The total cost is $84.47. The video ends with a screenshot of a customer service agent's chat window, showing appointment times available this Thursday."
}
},
{
"start_time": 4251.76,
"end_time": 4272.64,
"audio": {
"content": " Which would you prefer? Let's definitely do the afternoon time because I have a pickleball tournament in the morning. Great! You're all set. You'll receive a confirmation email in just a moment with all the details. Is there anything else I can assist you with today, Patrick? Yeah, do I actually have to pay for the landscaping services right now?"
},
"video": {
"content": "A customer is shopping on an online gardening store. The screen shows a shopping cart with two items: Bloom Booster Potting Mix and Flower Power Fertilizer. The customer adds these items to their cart and proceeds to checkout. A customer service agent appears on the right side of the screen, confirming the appointment time for pickup. The customer selects the confirmation email option."
}
},
{
"start_time": 4272.64,
"end_time": 4300.31,
"audio": {
"content": " No, we'll just add this to your account. And you can pay once the service is complete. Anything else? No, I'm just going to finish checking out here. You've been really wonderful today. Thanks for your help. You're very welcome. Thanks for choosing Symbol, Home, and Garden. Have a great day and good luck at your pickleball tournament. Now that was pretty amazing, right?"
},
"video": {
"content": "A man with a beard and tattoos is standing behind a podium, speaking into a microphone. He is wearing a dark blue shirt and has his hands clasped together. The background is a plain, light-colored wall with some vertical lines. The man appears to be giving a presentation or speech."
}
},
{
"start_time": 4300.31,
"end_time": 4325.97,
"audio": {
"content": " That entire thing was 100% real and live. All of the tools needed to build experiences just like that are available for you to start using today. Thanks, everyone, and back to you, Lisa. Pretty amazing. We're also helping to improve conversational customer experiences beyond the call center"
},
"video": {
"content": "A man with a beard and tattoos on his arms is standing behind a podium, speaking to an audience. He is wearing a blue shirt and has a microphone attached to his shirt. He gestures with his right hand, pointing upwards and then moving it down to his side. The background is a plain, light-colored curtain. The scene then cuts to a woman walking across the stage. She is wearing a green outfit and has short hair. She smiles as she walks."
}
},
{
"start_time": 4325.97,
"end_time": 4349.38,
"audio": {
"content": " by offering purpose-built agents that address specific industry use cases, including food ordering, automotive, and retail. For example, Wendy's AI drive-through ordering system handles 60,000 orders daily. Mercedes-Benz provides conversational search and navigation in the new CLLA series."
},
"video": {
"content": "A speaker stands on a stage at an event, addressing an audience. The stage is well-lit with a large screen behind the speaker displaying text and images. The text on the screen reads \"Purpose built industry agents.\" The speaker gestures with their hands as they speak, emphasizing points about purpose-built industry agents. The audience is seated in darkness, focusing their attention on the speaker and the screen."
}
},
{
"start_time": 4357.31,
"end_time": 4384.97,
"audio": {
"content": " And the Home Depot has built Magic Apron, an agent that offers expert home improvement guidance 24-7. And we have tremendous partnerships. For example, Service Now CRM works with customer engagement suite, helping to automate and personalize customer interactions across systems. Now, let's talk about creative agents that are being used to superpower creative teams, including those in media production, marketing, advertising, design, and more."
},
"video": {
"content": "A woman stands on a stage, dressed in a green jumpsuit with a black belt and white sneakers. She is speaking to an audience, gesturing with her hands as she addresses them. The background features vertical light panels that change color from blue to white. The scene transitions to a large screen displaying the logos of Google Cloud and ServiceNow, followed by another screen showing the text \"Creative Agents.\" The woman continues to speak, occasionally turning her body to face different directions."
}
},
{
"start_time": 4384.97,
"end_time": 4409.64,
"audio": {
"content": " In some cases, agents are augmenting creative teams to enable content production at massive scale. In others, they're helping reimagine how stories can be told for a new generation of audiences. One of the most amazing examples is the enriching of the Wizard of Oz at the Las Vegas sphere and how VEO2 helped to bring it to life."
},
"video": {
"content": "A woman stands on a stage, dressed in a green shirt and pants, with her hands clasped together in front of her. She appears to be speaking or presenting, as indicated by the microphone attached to her shirt. The background is a plain, light-colored backdrop, suggesting a formal or professional setting."
}
},
{
"start_time": 4409.64,
"end_time": 4438.31,
"audio": {
"content": " Let's hear from CEO of Sphere, Jim Dolan, and the visionaries who made it happen. The sphere is an experiential medium. We looked for content that would accentuate all of the different capabilities inside of the venue."
},
"video": {
"content": "A woman stands on a stage, dressed in a green outfit with a belt, addressing an audience. The scene transitions to a wide shot of the stage, illuminated by spotlights and featuring large screens displaying the word \"Sphere.\" The camera then focuses on a person walking through an empty auditorium, looking around at the rows of seats."
}
},
{
"start_time": 4438.31,
"end_time": 4459.93,
"audio": {
"content": " That was our criteria for choosing the Wizard of Ours. We knew that it was really hard to tackle using traditional means, but it is possible to do it with AI. We ultimately came to the conclusion that Google was the only company that was actually capable of doing this. Alongside Google's deep mind researchers, we trained AI models"
},
"video": {
"content": "The video begins with a wide shot of a large stadium at night, illuminated by numerous bright lights arranged in a curved pattern along the top of the stands. The atmosphere is dark, with a thick layer of fog or mist enveloping the scene, creating a dramatic and somewhat mysterious ambiance. The lights cast a warm glow on the fog, highlighting the structure of the stadium and the rows of seats. The camera remains stationary throughout this initial shot, capturing the grandeur and scale of the venue."
}
},
{
"start_time": 4459.93,
"end_time": 4481.31,
"audio": {
"content": " so that when you see Dorothy dancing, you see her full body dancing, down to her ruby slippers. For Google to bring VO2, we were really excited because that infrastructure that exists and that compute power was needed to do everything we're doing. Sphere Studios has partners with Google Cloud, and we've successfully deployed"
},
"video": {
"content": "The video begins with a close-up shot of a person working on a computer. The screen displays a video editing software interface, showing a scene of a forest with a dog walking through it. The person is focused on the screen, likely adjusting settings or trimming footage. The camera then transitions to a wide shot of a stage set up in a forest-like environment. The stage features a large, animated character in a pink dress, surrounded by other characters and vibrant decorations. The scene is lively, with people operating control panels and monitors, indicating a live performance or event being managed from behind the scenes. The video then cuts to a woman speaking in an interview setting. She is gesturing with her hands as she talks, suggesting she is explaining something important. Finally, the video transitions to an animated sequence. Three characters are walking down a dark, futuristic hallway with glowing blue walls. The hallway has a metallic, industrial design, and the characters appear to be exploring or moving towards a destination."
}
},
{
"start_time": 4481.31,
"end_time": 4502.03,
"audio": {
"content": " finishing technology in Google's cloud infrastructure, turnaround times, transmission times, all of the things that sometimes can slow a traditional studio down. Now we're able to do them a lot faster and with a lot more impact using Google Cloud. My hope for it is that we keep exploring different ways to create this kind of content"
},
"video": {
"content": "The video begins with a view of a control room filled with multiple screens displaying various scenes. The screens are arranged in a grid pattern, showing different angles and perspectives of an event or performance. The lighting is dim, with blue and yellow lights illuminating the screens, creating a professional and high-tech atmosphere.\n\nThe scene then transitions to a woman standing in front of a backdrop with the word \"transmission\" prominently displayed. She appears to be speaking or presenting, possibly discussing the technical aspects of the event being shown on the screens behind her. Her expression and body language suggest she is engaged and informative.\n\nNext, the video shifts to a woman working in a server room. She is wearing a white shirt and is focused on handling equipment, likely managing or troubleshooting network infrastructure. The server racks are visible in the background, indicating a high-tech environment.\n\nFinally, the video shows a man sitting in a modern office space. He is dressed in a dark suit and is gesturing with his hands as he speaks. The office has large windows that let in natural light, and there are plants and decorative elements in the background, creating a professional and well-lit environment."
}
},
{
"start_time": 4502.03,
"end_time": 4526.97,
"audio": {
"content": " and to take great performances from the past and bring them to life today, I think the world is going to be amazed. Beyond entertainment, AI is helping creative agencies revolutionize marketing for their clients."
},
"video": null
},
{
"start_time": 4526.97,
"end_time": 4550.14,
"audio": {
"content": " WPP built open as a platform powered by Google models that all of its employees worldwide can use to concept, produce, and measure campaigns. Monks.Flo is using Google AI to help localize creative for campaigns. And the BrandTech Group built Pencil, a generative AI platform for brands to create ads,"
},
"video": {
"content": "A woman stands on a stage in front of a large screen displaying various slides. The screen shows text prompts and images related to AI and technology. The woman gestures as she speaks, likely explaining the content displayed on the screen. The audience is seated in front of her, attentively watching the presentation."
}
},
{
"start_time": 4550.14,
"end_time": 4570.76,
"audio": {
"content": " like this recent mock-up for Japan Airlines. Customers are increasing marketing performance and reducing production time with creative agents. Mondalise quickly generates visuals for global brands like Oreo and Cadbury. Bloomberg Connects is making museums more accessible. And we're absolutely thrilled"
},
"video": {
"content": "A woman stands on a stage, dressed in a green jumpsuit and white shoes, addressing an audience. She gestures with her hands as she speaks, occasionally looking at a large screen behind her. The screen displays text about creating ads, predicting performance, and optimizing campaigns. The stage is well-lit, with vertical light panels on either side, and the audience is visible in the background."
}
},
{
"start_time": 4570.76,
"end_time": 4598.3,
"audio": {
"content": " to partner with Adobe, the leader in creativity, to bring our advanced Imagine 3 and V-O-2 models to applications like Adobe Express. Now, please welcome Brad Calder to the stage to talk about data agents. I'm Thanks, Lisa."
},
"video": {
"content": "A woman stands on a stage, addressing an audience. The background features a large screen displaying various slides related to Bloomberg Connects and Adobe products. The first slide shows two people looking at paintings, with the text \"Adapting arts & culture content to reach wider audiences.\" The second slide highlights Adobe's \"Adobe Express\" feature, emphasizing its ability to give creators and enterprises more ways to ideate. The woman continues her presentation, occasionally gesturing with her hands. Another person walks across the stage, adding movement to the scene."
}
},
{
"start_time": 4598.3,
"end_time": 4620.3,
"audio": {
"content": " Data agents know what data to utilize and what questions to ask. They enable data teams to effectively manage data and business teams to activate it. Mattel is an iconic brand, making toys from Barbies to Hot Wheels. Let's hear from Mattel's CEO, Enon Crys,"
},
"video": {
"content": "A man stands on a stage, addressing an audience. He is dressed in a blue long-sleeve shirt and dark pants, with a microphone clipped to his shirt. The background features a large screen displaying the name \"Brad Calder\" and the title \"VP & GM, Google Cloud.\" The screen also shows the Google Cloud logo and the Mattel logo. The stage is well-lit, and the audience is seated in front of him."
}
},
{
"start_time": 4620.3,
"end_time": 4645.97,
"audio": {
"content": ""
},
"video": {
"content": "A man stands on a stage, gesturing with his hands as he speaks. The background is a simple, light-colored backdrop with vertical lines. The scene then transitions to a dark room with red lights forming a circular pattern on the wall. The camera pans around the room, revealing a futuristic, illuminated stage with a sleek design. The lighting shifts to a vibrant pink hue, and various colorful objects float in the air, creating a dynamic and visually engaging scene. The video concludes with a shot of a sign that reads \"Mattel\" in bold letters."
}
},
{
"start_time": 4620.3,
"end_time": 4670.24,
"audio": {
"content": " on how they harness their data with Gemini. Matt Mattel. Our mission is to create innovative products and experiences that inspire fans and entertain audiences and develop children through play. This year, we're celebrating 80 years, and while we first made our mark as a toy company, Mattel today is a global brand management company home to one of the most iconic portfolios in the world. Our partnership with Google Cloud has helped us synthesize millions of points of consumer feedback. From phone calls and emails to online, and social media comments,"
},
"video": {
"content": "A man stands on a stage, gesturing with his hands as he speaks. The background is a simple, dark setting with vertical light strips. The scene transitions to a close-up of a screen displaying the word \"Gemini\" in blue letters, accompanied by a star symbol. The video then cuts to a different setting where the same man is seen in an office environment, wearing a white shirt and a dark jacket. Finally, the video shows a colorful display of various toy brand logos, including Mattel, Hot Wheels, Fisher-Price, Barbie, American Girl, and others."
}
},
{
"start_time": 4670.24,
"end_time": 4690.82,
"audio": {
"content": " delivering insights and opportunities to deepen our relationship with Mattel fans. Before partnering with Google Cloud, teams identified patterns manually. Now we can analyze sentiment and consumer preferences in real time. We can instantly identify key issues and trends improving both efficiency"
},
"video": {
"content": "The video begins with a montage of social media posts featuring reviews and testimonials about a product. The posts are displayed on a dark background, with various user avatars and comments visible. A prominent red sticker with the word \"MATTEL\" is superimposed over the center of the screen, drawing attention to the brand. The scene then transitions to a man standing in an office environment. He is dressed in a white shirt and a dark jacket, and he appears to be speaking or presenting. The background includes a large, colorful sign that reads \"Barbie.\" The man's expression and body language suggest he is engaged in delivering a message or presentation."
}
},
{
"start_time": 4690.82,
"end_time": 4713.28,
"audio": {
"content": " and innovation. For example, we improved the right mechanism in the Barbie Dreamhouse elevator and enhanced the interactive features in the Fisher-Price Kick-and and Play Piano Gym. These are two of our top-selling global products that have been made even better through data-driven insights. We see Google Cloud as a true partner"
},
"video": {
"content": "The video begins with a series of customer reviews displayed on a screen. The reviews are from different users, each providing feedback on various aspects of a product. The reviews include comments such as \"The elevator ride could be smoother?\" and \"Nice learning toy for all ages.\" The reviews are accompanied by star ratings, indicating the user's satisfaction level with the product."
}
},
{
"start_time": 4713.28,
"end_time": 4734.64,
"audio": {
"content": " in bringing the magic of play to life for every Mattel fan. Thanks, Enon. My kids love playing with Fisher-Price gyms. Our data platform, BigQuery, has five times more customers"
},
"video": {
"content": "A man is standing in an office environment, speaking to the camera. He is wearing a white shirt under a dark blue sweater. The background features a modern office setting with large windows and a blurred view of other people and equipment."
}
},
{
"start_time": 4734.64,
"end_time": 4756.3,
"audio": {
"content": " than the two leading independent data cloud companies. With BigQuery, you can activate all your data for AI, combining structured and unstructured data, such as tables, text, logs, images, and video. You can also work with open formats like Apache Iceberg directly integrated into BigQuery."
},
"video": {
"content": "A speaker is presenting on a stage at an event, likely a conference or seminar. The stage features a large screen displaying various slides related to data platforms and analytics. The speaker, dressed in a blue shirt and dark pants, gestures with his hands as he explains the content on the screen. The background includes a series of vertical light panels that change colors, adding a dynamic visual element to the presentation."
}
},
{
"start_time": 4756.3,
"end_time": 4784.69,
"audio": {
"content": " And you can use BigQuery to access data in any storage system or in any SaaS application on any cloud. And multimodal analysis with Gemini and BigQuery has grown more than 16 times this past year. And now, if you're a big Oracle customer, the full range of Oracle database services running on OCI are integrated with BigQuery, Gemini, and Vertex AI."
},
"video": {
"content": "A man stands on a stage, dressed in a blue long-sleeve shirt and dark pants. He is speaking and gesturing with his hands, indicating he is explaining something. The background consists of vertical light panels that change color from blue to white as the video progresses. The lighting on the stage is bright, focusing on the speaker."
}
},
{
"start_time": 4784.69,
"end_time": 4806.97,
"audio": {
"content": " They're being deployed natively in 20 Google Cloud locations serving customers such as Macy's and Sabre. And today we're very excited to announce specialized agents for every member of your data team. Now, for data engineering teams, we deliver agents for all aspects of the data engineering life cycle,"
},
"video": {
"content": "A man stands on a stage, gesturing with his hands as he speaks. He is wearing a blue long-sleeve shirt and dark pants. The background features vertical stripes, and the lighting is focused on him, creating a professional and engaging atmosphere. The scene then transitions to a wide shot of the stage, showing the audience seated in darkness. A large screen behind the speaker displays the text \"New Data Agents.\" The speaker continues to address the audience, maintaining eye contact and using hand gestures to emphasize points."
}
},
{
"start_time": 4806.97,
"end_time": 4827.97,
"audio": {
"content": " from catalog automation to metadata generation, to maintaining data quality to data pipeline generation. And for data science teams, our AI agent acts as a comprehensive coding partner in your data science notebook, accelerating every step of your workflow from data loading and feature engineering"
},
"video": {
"content": "A man stands on a stage, gesturing with his hands as he speaks. He is dressed in a blue long-sleeve shirt and dark pants, with a microphone clipped to his shirt. The background features vertical light panels that create a modern and professional atmosphere. As he continues speaking, the camera shifts focus to a large screen behind him, displaying the text \"New Data Agents Data Science\" with a \"Preview\" button below it. The man then resumes his speech, maintaining a confident and engaging demeanor."
}
},
{
"start_time": 4827.97,
"end_time": 4849.72,
"audio": {
"content": " to predictive modeling. And for data analysts and business users, our conversational analytics agent performs powerful, trustworthy analysis entirely in natural language. And you can also embed this agent in line in your own web or mobile application."
},
"video": {
"content": "A man stands on a stage, gesturing with his hands as he speaks. He is dressed in a blue long-sleeve shirt and dark pants, with a microphone clipped to his shirt. The background is a simple, light-colored curtain with vertical lines. The scene transitions to a large screen displaying the text \"New Data Agents Data Analysis\" with a \"Preview\" button below it. The man continues to speak, occasionally pointing towards the screen."
}
},
{
"start_time": 4852.02,
"end_time": 4870.64,
"audio": {
"content": " Now, for over a decade, Spotify has partnered with Google Cloud to cost effectively handle massive scale. They use BigQuery to harness enormous amounts of data to deliver personalized experiences to over 675 million users worldwide, including many of us here."
},
"video": {
"content": "A speaker is presenting on a stage at an event, likely a conference or seminar. The background features a large screen displaying information about Spotify, highlighting personalized experiences for 675 million users enjoying music, audio, podcasts, and audiobooks. The screen also shows images of smartphones with Spotify interfaces. The speaker is dressed in a blue shirt and dark pants, gesturing with his hands as he speaks. The audience is seated in front of the stage, attentively watching the presentation."
}
},
{
"start_time": 4870.64,
"end_time": 4891.5,
"audio": {
"content": " Unilever uses BigQuery to reach millions of retailers in emerging markets. Buyer built an agent that predicts flu trends. Now, customers are also taking advantage of our databases with AI. For example, Nero, an autonomous driving company"
},
"video": {
"content": "A man stands on a stage, dressed in a blue polo shirt and gray pants, with his hands clasped together in front of him. He appears to be speaking or presenting, as he gestures occasionally with his hands. The background is a simple, dark curtain with vertical light strips, creating a professional and focused atmosphere."
}
},
{
"start_time": 4891.5,
"end_time": 4913.64,
"audio": {
"content": " uses Allo ADB to identify challenging scenarios on the road. And public sector organizations like State of Nevada are using agents to speed up benefit claims. Let's find out more. The Nevada Department of Employment, Training, and Rehabilitation provides critical unemployment and job placement services."
},
"video": {
"content": "A man stands on a stage, gesturing with his hands as he speaks. The background is a large screen displaying an image of a self-driving car driving on a road lined with trees. The text on the screen reads \"nuro Identifying challenging scenarios on the road with vector search in AlloyDB.\" The scene then transitions to a wide shot of the Las Vegas Strip, with the word \"Nevada\" prominently displayed in the foreground."
}
},
{
"start_time": 4913.64,
"end_time": 4945.73,
"audio": {
"content": " To support limited staff in a regulated space, Nevada Dieter developed an appeals AI assistant, powered by BigQuery and Vertex AI. It synthesizes case data to help appeals referees make fair approvals four times faster, surpassing DOJ standards. Nevada Department of Employment, Training, and Rehabilitation is creating a new way to serve constituents. And now, let's see all of this in action."
},
"video": {
"content": "The video begins with a scene showing a man entering an office where three women are seated at a table. The text \"unemployment\" is displayed prominently on the screen. The scene then transitions to a black screen with the text \"Appeals AI Assistant\" and the BigQuery logo. Next, the video shows two men shaking hands in an office setting, with the Nevada Department of Employment, Training, and Rehabilitation logo visible. The final scene features the Google Cloud and DETR logos side by side."
}
},
{
"start_time": 4945.73,
"end_time": 4970.64,
"audio": {
"content": " Please welcome Yasmin Ahmad. Thank you, Brad. I'm here to show you the future of data signs made easy. All it takes is BigQuery, co-lab, and Vertex AI, now powered with Gemini."
},
"video": {
"content": "A man is standing on a stage, gesturing with his hands as he speaks. He is wearing a blue long-sleeve shirt and dark pants. The background is a plain, light-colored curtain. The scene then transitions to a woman walking down a hallway. She is wearing a black outfit with a maroon headscarf and a white shirt underneath. The hallway has a modern design with a large Google Cloud logo on the wall behind her. The woman continues walking towards a stage where she stands at a podium. The stage is well-lit with spotlights and has a sleek, futuristic design. The woman appears to be preparing to speak, holding a laptop in front of her."
}
},
{
"start_time": 4970.64,
"end_time": 4991.64,
"audio": {
"content": " So say you're running a consumer goods company. Sales are booming, but cash flow is slowing down. Why? Well, to answer this question, we need to see everything, from sales to invoices to customer signal. So let's take a look at our data. Now, traditionally, data is siloed,"
},
"video": {
"content": "A woman stands behind a podium on a stage, presenting financial data. The screen behind her displays a graph titled \"Revenue & Net Cash Flow\" for the last 12 months, showing a steady increase in revenue and net cash flow. Key metrics include cash on hand at $345.6M, accounts payable at $14.2M (89% of target), and an inventory level of 24 days. The presentation also includes a sales conversion rate funnel, cash flow by category, and territory performance map."
}
},
{
"start_time": 4991.64,
"end_time": 5014.64,
"audio": {
"content": " but now BigQuery helps me connect everything, including SAP deep integration and real-time feeds from Salesforce and even Google Ads. See? Easy. Let's ask our data engineering agent to now do the heavy lift. I'm going to add my first prompt from the clipboard."
},
"video": {
"content": "A woman wearing a hijab is standing in front of a screen displaying a Google Cloud interface. She appears to be presenting or explaining something related to data engineering. The interface shows various tabs such as 'Orders, Sales & Advertising,' 'Finance,' and 'Data Engineering Agent.' The woman gestures with her hands while speaking, indicating she is engaged in a discussion about data-related topics."
}
},
{
"start_time": 5014.64,
"end_time": 5041.97,
"audio": {
"content": " And to do this cash flow analysis, we're going to combine invoices, sales, and audience data from all of these sources into a single multimodal data table. Instantly, we have a unified view of everything. No complex integrations and no waiting. Easy. Of course, our new table is a little messy. Look at those dates. Well, it's a good thing now"
},
"video": {
"content": "A woman wearing a red hijab is standing at a podium, speaking into a microphone. She gestures with her hands as she talks. The background is a plain, dark-colored wall. On the right side of the screen, there is a computer interface displaying a data engineering project. The interface shows three queries: \"Inventory & Payables (SAP),\" \"Sales Orders & Payments (Salesforce),\" and \"Campaign & Conversions (Google Ads).\" A message from a \"Data Engineering Agent\" appears, asking to pull together data for a cash flow analysis. The agent then creates a pipeline that joins three tables and stores the results in a new table called \"invoice_orders_ads.\" The interface also includes options to apply transformations and extract street, city, state, and zip code from an address."
}
},
{
"start_time": 5041.97,
"end_time": 5062.87,
"audio": {
"content": " with BigQuery, we have access to Gemini-powered recommendations. And look at that. A clean data set. Super easy. Let's now move to BigQuery Data Canvas to do some analysis. Here we can see our structured data is ready to go."
},
"video": {
"content": "A person wearing a red hijab is standing behind a podium, speaking into a microphone. The background is dark, and there is a small cloud logo visible on the podium. The person appears to be giving a presentation or speech."
}
},
{
"start_time": 5063.67,
"end_time": 5082.97,
"audio": {
"content": " However, to do a true cash flow analysis, I need to extract signals from my PDF invoices. Not so easy. But here, I can ask our data science agent to help me extract buyer and payment information and group buyers into segments."
},
"video": {
"content": "A woman wearing a maroon hijab is standing behind a podium, speaking into a microphone. She appears to be presenting or giving a speech. The background is dark, and there is a small logo on the podium. The woman's hands are occasionally gesturing as she speaks."
}
},
{
"start_time": 5082.97,
"end_time": 5106.97,
"audio": {
"content": " In the past, this would have taken hours of manual effort. Definitely not easy. But now with BigQuery's new AI query engine, without having to review each PDF, I can automatically extract key information and we can group buyers into segments using Gemini's real-world knowledge."
},
"video": {
"content": "A woman wearing a maroon hijab and a black blazer is standing in front of a laptop screen. She appears to be presenting or explaining something related to data analysis. The screen shows a Google Cloud interface with SQL queries and a visualization of buyer categories. The woman gestures with her hands as she speaks, indicating she is engaged in the presentation."
}
},
{
"start_time": 5106.97,
"end_time": 5128.63,
"audio": {
"content": " Again, super easy. So what exactly is causing our cash flow drop from January to March. Here, our data science agent uses Gemini's new thinking model and BigQuery machine learning to build an automated data science workflow right before our eyes."
},
"video": {
"content": "A woman wearing a maroon hijab is standing at a podium, speaking into a microphone. She gestures with her hands as she talks. The background is a plain, dark-colored wall. On the right side of the screen, there is a Google Cloud interface displaying a bar chart titled \"Buyer Category.\" The chart shows five categories: Retailer, Wholesale, Foodservice, Hospitality, and Other. The chart indicates that Retailers are the most common category, with 78,340 products, while NGO is the least common, with only 4,520 products. Below the chart, there is a text box labeled \"Data Science Agent\" that provides additional information about the data visualization."
}
},
{
"start_time": 5128.63,
"end_time": 5151.63,
"audio": {
"content": " It's analyzing hundreds of dimensions in mere seconds. And it looks like we have our answer. Payment terms. It looks like our new 36-month payment promotion offer while boosting sales has caused the recent cash flow dip. Hmm. I wonder how this is impacting my cash forecast."
},
"video": {
"content": "A woman wearing a maroon hijab and a black blazer is standing at a podium, speaking into a microphone. She gestures with her hands as she explains something. The background is blurred, but it appears to be an indoor setting, possibly a conference room or a studio. The video then cuts to a computer screen displaying a Google Cloud interface. The screen shows a SQL query being run, with results displayed in a table format. The query analyzes data related to sales and advertising, showing contributors, metric tests, control, differences, and relative differences. Insights are provided, highlighting that Payment Terms of 36 Months is the biggest contributor with a 32% influence on cash flow, followed by Buyer City Depot with 17% influence. Logistics United has an 8% influence, and buyers in Wholesale and Retail categories are the highest contributing buyers to cash flow. Another insight states that 36 month payment terms and City Depot are the second largest contributors for January to March."
}
},
{
"start_time": 5151.63,
"end_time": 5173.75,
"audio": {
"content": " To do this, I want to jump to code, so I'm going to extract my analysis here into a big query notebook. And in our notebook, we can ask our data science agent to again help us write some code. So here, we'll ask our agent to build a forecast for the next three months broken down"
},
"video": {
"content": "A woman wearing a hijab is standing at a podium, speaking into a microphone. She appears to be presenting or giving a speech. The background is blurred, focusing on her as she gestures with her hands while speaking."
}
},
{
"start_time": 5173.75,
"end_time": 5196.3,
"audio": {
"content": " by buyer category. BigQuery now uses Google's new pre-trained time series forecasting model to build this out. And we can reveal the big insight. It looks like wholesalers taking those long 36-month payment terms, are causing the issue. And finding this answer?"
},
"video": {
"content": "A person is standing at a podium, speaking into a microphone. The background is a plain, dark-colored wall. The person is wearing a black outfit with a maroon scarf. On the right side of the screen, there is a computer interface displaying code and a line chart titled 'Cash Flow Forecast by Buyer Category.' The code includes imports and functions related to forecasting cash flow. The line chart shows various lines representing different buyer categories (Wholesale, Foodservice, Retailer, Hospitality, Other) over time, indicating percentage growth. A message from the Data Science Agent states that the forecast predicts wholesalers will have the largest cash flow change from April to June."
}
},
{
"start_time": 5196.3,
"end_time": 5221.3,
"audio": {
"content": " Easy. We can make this analysis even more powerful. We can ask our data science agent to include product category as a breakdown. Big Cree Colab Composer instantly gets to work updating all of our code. And as we see forecasts, now including buyer segment and product category, we can answer"
},
"video": {
"content": "A woman wearing a maroon hijab is standing at a podium, speaking into a microphone. She appears to be presenting or explaining something. The background is dark, and there is a screen displaying a line chart titled 'Cash Flow Forecast by Buyer Category.' The chart shows various lines representing different buyer categories, such as Wholesale, Foodservice, Retailer, Hospitality, and Other, with data points over time. The woman gestures with her hands while speaking, emphasizing her points. The video also includes a text box labeled 'Data Science Agent' that provides information about the forecasted cash flow changes for different buyer categories."
}
},
{
"start_time": 5221.3,
"end_time": 5260.3,
"audio": {
"content": " a huge range of potential questions. So easy. In fact, it looks like our 36-month terms are impacting fast-moving segments like food and beverage, and not other segments, for example, medication. This insight allows for surgical precision. Instead of a blunt action like removing the 36-month promo offer entirely, we can make a data-driven, targeted, easy decision. Pretty incredible, right? This whole process used to take months of manual work, but today it took just a few minutes."
},
"video": {
"content": "A woman wearing a maroon hijab is standing in front of a laptop screen, which displays a code editor with Python code. The code is being executed, and the resulting heatmap of cash flow growth forecast is shown on the screen. The woman appears to be explaining the code and the results."
}
},
{
"start_time": 5260.3,
"end_time": 5286.97,
"audio": {
"content": " Gemini and Vertix AI have made BigQuery a complete data science platform, unlocking new insights faster than ever with both natural language and code. And that, my friends, is the future of data science made? Say it with me. Easy. Back to you, Brad. Thanks, Yasmin."
},
"video": {
"content": "A woman wearing a maroon hijab and a black blazer stands behind a podium, presenting data on a large screen behind her. The screen displays a heatmap titled \"Heatmap of Cash Flow Growth Forecast.\" The heatmap categorizes different retailers and their respective cash flow growth percentages across various product categories. The woman gestures with her hands as she speaks, emphasizing points about the data. The background is a simple, dark curtain, and the lighting focuses on the presenter and the screen."
}
},
{
"start_time": 5286.97,
"end_time": 5311.63,
"audio": {
"content": " That was amazing. Now, just like with data, Gemini's fast performance, large context window, and reasoning make it highly effective for coding agents. We offer Gemini Code Assist in Google Cloud, Android Studio, Firebase Studio, and your favorite IDEE. Our enterprise version understands your code-based standards and conventions."
},
"video": {
"content": "A man stands on a stage, dressed in a blue long-sleeve shirt and dark pants, with a microphone clipped to his shirt. He gestures with his hands as he speaks, moving them from his chest outward and then back together. The background is a simple, light-colored curtain. The text \"Brad Calder\" appears in orange letters on the left side of the screen."
}
},
{
"start_time": 5311.63,
"end_time": 5336.9,
"audio": {
"content": " And companies like Amp here from No Group, Broadcom, CME group, PayPal, and LibPro use codicists today. And today, we're announcing new code assist agents to help with everything from modernizing code to helping with the full software development lifecycle. Developers can interact with our agents on the con bond board, which"
},
"video": {
"content": "A man stands on a stage, dressed in a blue shirt and dark pants, gesturing with his hands as he speaks. The background is a simple, modern design with vertical light panels. The scene transitions to a large screen displaying logos of companies like Renault, Broadcom, CME Group, PayPal, and Wipro. The screen then shows a new feature called \"Gemini Code Assist Agents\" with a subtitle indicating assistance across the development life cycle. The man continues to speak, and the audience is visible in the foreground."
}
},
{
"start_time": 5336.9,
"end_time": 5360.3,
"audio": {
"content": " provides a real-time display of the task codices working on, as well as the ability for developers to interact with our agents. Kodysis also has integrations with dozens of partners, such as Atlantean, Sentry, Sneak, and many more coming soon. Now, outside of Google, Gemini is also available"
},
"video": {
"content": "A speaker stands on a stage, presenting to an audience. The background features a large screen displaying a Kanban board with various tasks labeled as 'Action Needed,' 'Running,' and 'Completed.' The speaker gestures towards the screen while explaining the process. The screen transitions to show a list of companies under the heading 'Gemini Code Assist Partner Ecosystem,' highlighting their logos and names."
}
},
{
"start_time": 5360.3,
"end_time": 5393.63,
"audio": {
"content": " for your development needs in ATER, Cursor, GitHub Copilot, Replit, Replit, TapLine, and WindSurf. If you want to see more, join me tomorrow at the developer keynote to see Code Assistant Action. Now, to share what's new and security Code Assistant Action. Now, to share what's new in security, please welcome Sandra Joyce. Thank you. Thank you. Thank you. Thank you. Thanks, Brad."
},
"video": {
"content": "A man stands on a stage, addressing an audience. He is dressed in a blue shirt and dark pants, with his hands clasped together in front of him. The stage features a large, illuminated screen displaying the text \"Developer Keynote\" along with the date and time: \"Thursday, April 10 • 2:30pm-3:45pm.\" The background is dark, with blue lighting accents and a few spotlights illuminating the speaker. The atmosphere suggests a formal presentation or conference setting."
}
},
{
"start_time": 5393.63,
"end_time": 5418.94,
"audio": {
"content": " Security agents can dramatically increase in the speed and effectiveness of security analysts. The integration of AI across our security products is just one reason why organizations around the world are making Google part of their security team. We offer critical cyber defense capabilities in today's challenging threat environment, such as threat intelligence drawn from"
},
"video": {
"content": "A woman stands on a stage, delivering a presentation. She is dressed in a bright pink blazer over a white blouse and blue jeans. Her hair is long and dark, and she wears a microphone attached to her blazer. The background features a large screen displaying her name, "
}
},
{
"start_time": 5418.94,
"end_time": 5440.3,
"audio": {
"content": " Mandient investigations, Google operations, virus total, and more. So you know who's targeting you and where you're exposed. A comprehensive where you're exposed. A comprehensive security operations platform that applies our intelligence for proactive threat detection, investigation and response, anywhere you operate."
},
"video": {
"content": "A speaker stands on a stage in front of an audience, presenting information about Google Threat Intelligence. The background features a large screen displaying a diagram with concentric circles labeled \"Google Threat Intelligence.\" The speaker gestures towards the screen as they explain the concept. The audience is seated, attentively listening to the presentation."
}
},
{
"start_time": 5440.3,
"end_time": 5465.97,
"audio": {
"content": " Cloud security and Risk Management that uses virtual red teaming to find risks that other solutions can't, protecting your workloads and AI across all your clouds, and Mandian services that provide expertise before, during, and after security incidents. Today we're introducing new security agents that analyze malware and triage alerts to speed up investigations."
},
"video": {
"content": "A woman stands on a stage in front of a large screen displaying various logos and text related to Google Security Operations and Google Cloud Security Command Center. She is dressed in a pink blazer, white shirt, and blue jeans, holding a microphone in her right hand. The background features a dark auditorium with rows of seats filled with an audience. The lighting focuses on the speaker, creating a professional and engaging atmosphere."
}
},
{
"start_time": 5465.97,
"end_time": 5492.63,
"audio": {
"content": " And our capabilities have been adopted by thousands of organizations, like Charles Schwab, who uses Google SecOps to stay proactive in responding to cyber threats. They've gained new visibility across their entire environment while reducing investigation and resolution time. Averted, who are detecting more events and closing investigations faster with Google SecOps."
},
"video": {
"content": "A woman stands on a stage, presenting information about new security agents. The screen behind her displays options such as \"Malware Analysis\" and \"Alert Triage.\" She gestures with her hands as she speaks, emphasizing the features being discussed. The presentation includes a slide that reads \"Proactively responding to threats with Google SecOps.\" The stage is well-lit, with a modern design featuring vertical light panels and a large screen displaying the presentation slides."
}
},
{
"start_time": 5492.63,
"end_time": 5513.75,
"audio": {
"content": " Dunn and Bradstreet, who are using Security Command Center to centralize monitoring of AI security threats, and vote a phone. AI security threats. And Vodafone, who used Vertex AI, along with open source tools and Google Cloud security foundation to establish an AI security governance layer."
},
"video": {
"content": "A woman stands on a stage, dressed in a bright pink blazer over a white blouse and blue jeans, paired with beige shoes. She gestures with her hands as she speaks, likely delivering a presentation. The background is simple, featuring vertical light panels that create a modern and professional atmosphere. As she continues to speak, the scene transitions to a large screen behind her, displaying a slide from Dun & Bradstreet's Security Command Center. The slide shows a woman in glasses using a tablet, with the text \"Centralized monitoring of AI security threats using Security Command Center.\" The screen then changes to another slide showing a man on a call, with the text \"44k\" prominently displayed."
}
},
{
"start_time": 5513.75,
"end_time": 5536.3,
"audio": {
"content": " And finally, the government of Singapore, who uses Google Cloud Web Risk to protect their residents online. These organizations and many more benefit from our individual products and services, but we can drive even better outcomes from converging our security capabilities. And for that, we're introducing Google Unified Security."
},
"video": {
"content": "A woman stands on a stage, dressed in a bright pink blazer over a white blouse and blue jeans. She is speaking and gesturing with her hands, indicating she is delivering a presentation or speech. The background is simple, featuring vertical light panels that create a modern and professional atmosphere. As she speaks, the camera occasionally cuts to a large screen displaying a message about the Government of Singapore protecting citizens' online activity, accompanied by an image of a cityscape with iconic buildings."
}
},
{
"start_time": 5537.52,
"end_time": 5558.3,
"audio": {
"content": " Google Unified Security brings together unmatched visibility, faster threat detection, AI-powered security operations, continuous virtual red teaming, the most trusted browser, and mandiant expertise in one converged security solution running on a planet scale data"
},
"video": {
"content": "A presentation is being given on a large screen displaying information about Google Unified Security. The screen transitions through various slides that explain the features and components of the solution. The presenter, a woman dressed in a pink blazer and white shirt, stands confidently on stage, gesturing as she speaks. The background is a simple, dark curtain with vertical light strips."
}
},
{
"start_time": 5558.3,
"end_time": 5579.3,
"audio": {
"content": " fabric. For our last demo, let's see how this works with Pyle Chakravardi and Nav Jagpa. Hi everyone, I'm Nav and I'm a developer building a cool new app using Vertex AI."
},
"video": {
"content": "A woman stands on a stage, dressed in a bright pink blazer over a white blouse and blue jeans. She has long dark hair and is wearing a microphone attached to her blazer. She appears to be speaking, as she gestures with her hands while looking slightly to the side. The background is a simple, light-colored curtain."
}
},
{
"start_time": 5579.3,
"end_time": 5601.3,
"audio": {
"content": " And I'm Pio, a security analyst. Often juggling multiple security tools and manual processes, I am here to show you how Google Unified Security or Gus for shot can proactively protect your applications no matter where you're building them. So now, what have you been up to? Well, I was trying to test this app, and let's just keep this between us. I might have made a few mistakes along the way."
},
"video": {
"content": "A man and a woman stand side by side on a stage, both facing forward. The man has long gray hair and a beard, wearing a dark blazer over a light-colored shirt. The woman has long brown hair, wearing a beige blazer over a white top. They appear to be presenting or speaking at an event. The background is simple, with vertical blinds and a plain wall. The scene then transitions to a wide shot of the stage, showing the two individuals standing behind a podium. The screen behind them displays the title \"Google Unified Security (GUS)\" along with various icons and text, indicating different components of the security system."
}
},
{
"start_time": 5601.3,
"end_time": 5622.04,
"audio": {
"content": " Go on. might have made a few mistakes along the way. Go on. Well, to speed up development, I installed a Chrome extension, which helps me test my prompts across multiple public LOMs. You know, I didn't think anything of it at the time. Hmm, let's see what's going on here. Well, Nav, while you were doing this, which is a legitimate action and Chrome extension, there may have been a potential data leak."
},
"video": {
"content": "A man and a woman stand behind podiums on a stage, presenting information. The man gestures with his right hand as he speaks, while the woman listens attentively. The background features a large screen displaying a diagram titled \"Google Unified Security (GUS).\" The diagram includes various icons and labels such as \"Google Threat Intelligence,\" \"Security Data Fabric,\" \"Google Cloud Security Command Center,\" \"Google Security Operations,\" and \"Google Chrome Enterprise.\" The scene transitions to a close-up of the screen, showing a list of security measures and options."
}
},
{
"start_time": 5622.04,
"end_time": 5645.96,
"audio": {
"content": " As you see here in the centralized risk dashboard that prioritizes risk across all my company's, and activities, Gus detected that you were copying and pasting sensitive data into the public LLM models. Now, if I click here further, Gemini's agentic AI has automatically triaged this alert, confirm the data leak with high confidence and taken an automated response to quarantine"
},
"video": {
"content": "A woman is standing at a podium, speaking into a microphone. She appears to be presenting or giving a speech. The background is dark, and there is a large screen behind her displaying various security-related information. The screen shows a list of riskiest AI issues, including publically accessible models where prompt injection is possible, user-managed keys to service accounts, and publicly exposed buckets containing Vertex AI models. The presentation includes details about recent critical AI threats, such as a sensitive data leak through a third-party LLM. The woman gestures with her hands as she speaks, emphasizing points on the screen."
}
},
{
"start_time": 5645.96,
"end_time": 5670.63,
"audio": {
"content": " that specific Chrome extension immediately. Wow, this would have taken me days to figure out in other tools. Wait, so Gus detected what I was doing and stopped it automatically? But I mean, why is it even a big deal that was testing my prompts like this in the first place? Well, because those prompts could contain confidential company data. Gus protected not only you, but mitigated insider risk by updating Chrome policy for the entire organization."
},
"video": {
"content": "A woman is standing at a podium, speaking into a microphone. She is wearing a beige blazer over a white top. The background is dark, and there is a blue screen with text on it. The text on the screen includes information about an investigation and response steps related to a sensitive resource exposure and data leak through third-party LLM. The text mentions actions such as performing automated responses, investigating a GCP instance, and hardening AI models with Model Armor. There is also a mention of a threat actor associated with IP address 198.51.100.4."
}
},
{
"start_time": 5670.63,
"end_time": 5691.87,
"audio": {
"content": " Well, now that I look at it, the agent has picked up on a few other correlated risks. Is there anything else you want to tell me about now? Oh, you know what? Come to think of it, I wanted to test my application, so I spun up a VM. You know, I just wanted to get things working really quickly, so I might have been a little bit lax for those firewall settings."
},
"video": {
"content": "A presentation is being given on a screen, focusing on an investigation into a data leak through a third-party LLM (Large Language Model). The presenter, a woman with long hair, is seen speaking and gesturing towards the screen. The screen displays various steps in the investigation process, including automated responses, policy updates, and the analysis of a GCP instance. The presenter discusses the findings and actions taken to mitigate the risk associated with the threat actor UNCXXX."
}
},
{
"start_time": 5691.87,
"end_time": 5712.63,
"audio": {
"content": " Hmm. There is something interesting going on with that VM. Gus detected malicious traffic to the VM and automatically associated that with an emerging threat actor that the Google thread Intel team has been tracking. Wait, hold up. So you're saying that while I was testing my app for just a few hours, somebody's trying to break into it? Exactly."
},
"video": {
"content": "A man with gray hair and a beard is speaking passionately into a microphone while standing in front of a dark background. He gestures with his hands as he talks, emphasizing his points. The scene then transitions to a woman with long hair, wearing a beige blazer over a white top, who is also speaking into a microphone. She appears to be addressing an audience, gesturing with her hands as she speaks. The video alternates between these two speakers, each delivering their message."
}
},
{
"start_time": 5712.63,
"end_time": 5732.75,
"audio": {
"content": " In the world of security, a few hours could be a lifetime nap. Well, Gus recognized the risk in real time and took action. That's wild. Is there anything else that Gus thinks I should be doing to protect my application? Yeah. The agent here recommends that we harden your AI model with model armor. It's new AI protection capabilities."
},
"video": {
"content": "A woman is speaking at a podium, gesturing with her hands as she presents information on a computer screen. The screen displays an investigation report titled \"Investigation.\" The report includes details about a correlated external IP address (198.51.100.4) associated with a threat actor named UNCXXX. The report mentions that this IP has been connected to a Google user content resource and is observed to perform input engineering or injection to assess functionality and security. The report also provides cloud asset details for the resource 143.113.0.203.bc.googleusercontent.com, including its creation date, IAM policy details, and firewall rules. The presentation continues with a summary of the Gemini investigation, highlighting potential data leakage through a Chrome extension and the need to quarantine the Chrome extension and update policies."
}
},
{
"start_time": 5732.75,
"end_time": 5758.3,
"audio": {
"content": " So you will see here, at the click of a button, it will take a minute here. At the click of this button, you will see that Model Armour starts analyzing inputs in real time, in line, blocking malicious inputs before they reach the model. Okay. So Gus is able to detect all risks in one place, connect the dots and take action. You know, as a developer, it really gives me peace of mind knowing that Gus has my back at all times."
},
"video": {
"content": "A woman is presenting on a stage, standing behind a podium with a laptop. She is wearing a beige blazer over a white top and has long hair. The background is dark, and there is a screen displaying information about a security investigation. The text on the screen includes details such as 'Sensitive Resource Exposure and Data Leak Through Third Party LLM,' 'High Severity,' and 'GCP Instance 143.113.0.203.BC.GOOGLEUSERCONTENT.COM.' The presentation appears to be part of a security conference or workshop."
}
},
{
"start_time": 5758.3,
"end_time": 5780.3,
"audio": {
"content": " Yes, Nav. And Gus not only has your back on this environment in Google Cloud. Gus is an integrated open platform that can protect any environment, any data, from endpoint, firewall, networks, identity, really, any cloud, any model. Further, for added protection, we have access to Gus's expertise in incident response and threat hunting around the clock."
},
"video": {
"content": "A man is standing in front of a screen displaying the Google Unified Security dashboard. He appears to be explaining or presenting something related to the security platform. The screen shows various sections such as AI Inventory, Model Armor, and other security-related metrics. The man gestures towards the screen as he speaks, indicating different parts of the dashboard. The background is dark, and the screen has a blue and purple color scheme."
}
},
{
"start_time": 5780.3,
"end_time": 5806.84,
"audio": {
"content": " Well, I'm super excited that Google is a part of our security team. Thank you, everyone. Back to you, Thomas, to take us home. Thank you both. We're continuing to invest in our security offerings and just last month we signed a definitive agreement to acquire WIS, a leading multi-cloud security platform to provide better cybersecurity"
},
"video": {
"content": "A man and a woman stand at podiums, each with a microphone attached to their clothing. The man gestures with his right hand while speaking, and the woman smiles and nods. The scene then transitions to the man standing alone on a stage, continuing to speak and gesture with his hands."
}
},
{
"start_time": 5806.84,
"end_time": 5826.98,
"audio": {
"content": " alternatives for business and governments around the world. Now as you've heard throughout this keynote, we're delivering an amazing stream of new innovations and making it easy to integrate those innovations into your existing technology landscape. We do this in four important ways."
},
"video": {
"content": "A man in a dark blue suit and white shirt stands on a stage, gesturing with his hands as he speaks. He appears to be giving a presentation or speech. The background is a simple, light-colored curtain, and the lighting focuses on him, highlighting his movements and expressions."
}
},
{
"start_time": 5826.98,
"end_time": 5847.96,
"audio": {
"content": " First, connecting your clouds with other clouds and applications. Enabling secure cross-cloud networking with cross cloud interconnect using your existing security platforms applying federated identity with Microsoft Entra ID and using BigQuery and AlloDB without moving from Amazon"
},
"video": {
"content": "A man in a dark suit stands on a stage, gesturing with his hands as he speaks. The background is a large screen displaying the Google Cloud logo and the text \"Connect with other clouds and applications.\" The man appears to be presenting or giving a speech about cloud computing and networking solutions."
}
},
{
"start_time": 5847.96,
"end_time": 5868.63,
"audio": {
"content": " or Azure, which is helping companies like Johnson & Johnson and and Walmart. Second, we're working with many leading ISVs to integrate them with Google AI. You have access to these ISV solutions, which are pre-integrated and easily deployed from the Google Cloud Marketplace."
},
"video": {
"content": "A man in a dark blue suit stands on a stage, gesturing with his hands as he speaks. He appears to be addressing an audience, possibly at a conference or event. The background is simple, featuring vertical light panels that create a subtle gradient effect. The man's gestures suggest he is explaining or emphasizing points related to the topic he is discussing."
}
},
{
"start_time": 5869.63,
"end_time": 5890.63,
"audio": {
"content": " Third, our services partners have created thousands of agents that bring their deep understanding of your industry and existing IT systems. In fact, Accenture, Cap Gemini, Deloitte, HCL Tech, KPMG, TCS, and WeRO are all making agent announcements"
},
"video": {
"content": "A man in a dark suit stands on a stage, presenting to an audience. The stage is well-lit with spotlights and a large screen behind him displaying various logos and text related to AI agent innovation with services partners. The man gestures with his hands as he speaks, occasionally clapping his hands together. The audience is seated in rows, attentively watching the presentation."
}
},
{
"start_time": 5890.63,
"end_time": 5913.63,
"audio": {
"content": " with Google Cloud today. Fourth, we're enabling sovereign clouds with partners to meet international regulations. Today we offer Google Cloud Sovereign AI services in our public cloud, So well as with Google Workspace."
},
"video": {
"content": "A man in a dark suit stands on a stage, gesturing with his hands as he speaks. The background is a simple, modern design with vertical light panels. The scene transitions to a large screen displaying a list of logos under the heading \"Sovereign Cloud Partner Ecosystem.\" The screen shows various company names and logos, indicating a collaborative network of cloud service providers. The man continues to speak, emphasizing the importance of the ecosystem."
}
},
{
"start_time": 5913.63,
"end_time": 5935.6,
"audio": {
"content": " Now just in closing, what an amazing time for all of us to experience and work with these technology advances. We at Google Cloud are committed to helping each of you innovate by delivering the leading enterprise-ready AI-optimized platform AI optimized platform"
},
"video": {
"content": "A man in a dark blue suit stands on a stage, gesturing with his hands as he speaks. The background is a gradient of light blue to white, with vertical lines creating a subtle pattern. The lighting focuses on the speaker, highlighting him against the backdrop."
}
},
{
"start_time": 5935.6,
"end_time": 5961.3,
"audio": {
"content": " with the best infrastructure, leading models, tools and agents by offering an open multi-cloud platform and building for interoperability so we can speed up time to value from your AI investments. We are honored to be building this new way to cloud with each of you."
},
"video": {
"content": "A man in a suit stands on a stage, gesturing with his hands as he speaks. The background features a large screen displaying text and icons related to an AI-optimized platform, open multicloud, and built for interoperability. The stage is well-lit with blue lighting, and the audience is seated in darkness, attentively watching the presentation."
}
},
{
"start_time": 5961.3,
"end_time": 5982.96,
"audio": {
"content": " To everyone here, with each of you. To everyone here and all of those watching online, thank you so much for joining us for Google Cloud Next. We hope to see you again in 2026."
},
"video": {
"content": "A man stands on a stage, dressed in a formal blue suit with a white dress shirt and a black bow tie. He is wearing a black belt and black shoes. The background features vertical light strips that create a modern and professional atmosphere. The man appears to be speaking or presenting, as he gestures with his hands and occasionally clasps them together in front of him. His facial expressions change slightly, indicating engagement with his audience."
}
},
{
"start_time": 5982.96,
"end_time": 6004.006893,
"audio": {
"content": " We'll be back here in Las Vegas, April 22nd to 24th. Have an amazing event. Go ahead. Go ahead! You know, You know,"
},
"video": {
"content": "A man stands on a stage, dressed in a dark suit with a white shirt and a bow tie. He appears to be speaking or presenting, as he gestures with his hands while looking directly at the camera. The background is a simple, light-colored curtain, which helps to keep the focus on the speaker."
}
}
]
}
```
Let's break down the output into its key components:
* **`segments`** : Segmented video content with start and end timestamps
* **`segment.start_time`** : The start time of the segment in seconds (relative to the start of the video)
* **`segment.end_time`** : The end time of the segment in seconds (relative to the start of the video)
* **`segment.audio.content`** : The raw transcription of the spoken content
* **`segment.video.content`** : The visual scene description of the video content
* **`metadata.duration`** : The total duration of the video in seconds
In subsequent guides, we'll cover more advanced capabilities like topic extraction, entity recognition, and sentiment analysis.
## Key Features
* **Temporal Grounding**: Precise time segmentation and content localization
* **Visual Analysis**: Scene descriptions, object recognition, and text extraction
* **Long-form Support**: Process videos up to 4+ hours with automatic segmentation
* **Batch Processing**: Efficient handling of large video collections
## Try our Video / Audio -> JSON API today
Head over to our [Video -> JSON](/api-reference/v1/post-video-generate) or [Audio -> JSON](/api-reference/v1/post-audio-generate) to start building your own video/audio processing pipelines with [VLM Run](https://vlm.run). Sign-up for access on our [platform](https://app.vlm.run).
# Supported Domains
Source: https://docs.vlm.run/hub
Pre-built schemas and domain definitions for common data extraction tasks.
The [VLM Run Hub](https://github.com/vlm-run/vlmrun-hub) is a collection of pre-defined domains and schemas for structured data extraction.
## Document Domains
|
Domain
|
Allowed Inputs
|
Description
|
| :---------------------------------------------------------------------- | :-------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [document.bank-statement](/api-reference/v1/post-document-generate) | documentimage | Bank statement data extraction system that processes bank statements to extract structured financial transaction information including account details, balances, and transaction history. |
| [document.classification](/api-reference/v1/post-document-generate) | documentimage | Classify documents into one or more categories based on their content, visual features, and metadata. |
| [document.invoice](/api-reference/v1/post-document-generate) | documentimage | Comprehensive invoice data extraction system that processes invoice images to extract structured information including invoice metadata, customer details, line items, and financial totals. |
| [document.markdown](/api-reference/v1/post-document-generate) | documentimage | Convert document pages into a highly-accurate content descriptions, including table and chart content. |
| [document.q-and-a](/api-reference/v1/post-document-generate) | documentimage | Convert document pages into a highly-accurate content descriptions, including table and chart content. |
| [document.receipt](/api-reference/v1/post-document-generate) | documentimage | Receipt data extraction system that processes receipt images to extract structured information including transaction details, merchant information, and financial totals. |
| [document.resume](/api-reference/v1/post-document-generate) | documentimage | Resume data extraction system that processes resume images to extract structured information including contact details, education, work experience, skills, and additional sections. |
| [document.us-drivers-license](/api-reference/v1/post-document-generate) | documentimage | Driver's license information extraction system that processes driver's license images to extract structured information including name, address, date of birth, and license details. |
| [document.utility-bill](/api-reference/v1/post-document-generate) | documentimage | Utility bill data extraction system that processes utility bill images to extract structured information including account details, billing period, charges, and payment information. |
## Image Domains
|
Domain
|
Allowed Inputs
|
Description
|
| :------------------------------------------------------------ | :-----------------------: | :----------------------------------------------------------------------------------------------- |
| [image.classification](/api-reference/v1/post-image-generate) | image | Classify the image into one of the following categories: \[category1, category2, category3, ...] |
| [image.caption](/api-reference/v1/post-image-generate) | image | Extract the caption from the image. |
| [image.tv-news](/api-reference/v1/post-image-generate) | image | Extract the TV news segment from the image. |
| [image.q-and-a](/api-reference/v1/post-image-generate) | image | Answer questions about an image. |
## Audio Domains
|
Domain
|
Allowed Inputs
|
Description
|
| :----------------------------------------------------------- | :-----------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [audio.transcription](/api-reference/v1/post-audio-generate) | audio | Gain a competitive edge with real-time analytics from audio, helping you track trending topics, public sentiment, and influential quotes that shape audience perspectives. |
## Video Domains
Video domains can be used to analyze video content, including transcribing the video content, summarizing the video content, and analyzing the video content. They are categorized into 3 types:
* **Whole-video Summary** (summary): Analyze the whole video content.
* **Segmented Summary** (segmented-summary): Analyze the entire video content and summarize it into multiple segments (key moments, scenes, highlights, etc.). You can provide prompts and cues to guide the segmentation, and the number of segments can be specified (e.g. "Find 5 key moments in this video where the CEO mentions "AI").
* **Segmented Analysis** (segmented-analysis): Analyze the video content per-segment, with each segment extracting detailed information prompted via the custom video segment model (e.g. [`json_schema`](/api-reference/v1/post-video-generate#body-config-json-schema)). Each segment is automatically detected with audio and visual cues (e.g. silence, new scene, etc.)
|
Domain
|
Allowed Inputs
|
Type
|
Description
|
| :------------------------------------------------------------------- | :----------------------------------: | :---------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [video.transcription](/api-reference/v1/post-video-generate) | video | segmented-analysis | Transcribe video content, including timestamps, visual descriptions, audio transcriptions, summaries, and topic identification. |
| [video.transcription-summary](/api-reference/v1/post-video-generate) | video | segmented-summary | Analyze video content by breaking it into 5-minute segments with detailed visual and audio analysis, including timestamps, visual descriptions, audio transcriptions, summaries, and topic identification. |
| [video.product-demo-summary](/api-reference/v1/post-video-generate) | video | segmented-summary | Analyze video content by breaking it into 5-minute segments with detailed visual and audio analysis, including timestamps, visual descriptions, audio transcriptions, summaries, and topic identification. |
| [video.conferencing-summary](/api-reference/v1/post-video-generate) | video | segmented-summary | Analyze conferencing videos to extract structured information including segments, topics, presenters, and key events for comprehensive conferencing monitoring and analysis. |
| [video.podcast-summary](/api-reference/v1/post-video-generate) | video | segmented-summary | Analyze podcast videos to extract structured information including segments, topics, presenters, and key events for comprehensive podcast monitoring and analysis. |
| [video.summary](/api-reference/v1/post-video-generate) | video | summary | Analyze whole video content by summarizing the video content into a concise 2-3 sentence summary, and providing a list of topics discussed in the video, at most 5 topics. Use the provided topics enum if possible. |
| [video.dashcam-analytics](/api-reference/v1/post-video-generate) | imagevideo | summary | Analyze dashcam footage to identify and classify events, objects, and situations relevant to road safety and vehicle monitoring, including traffic conditions, incidents, and environmental factors. |
## Industry-specific Domains
|
Domain
|
Allowed Inputs
|
Description
|
| :----------------------------------------------------------------------------- | :-------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [aerospace.remote-sensing](/api-reference/v1/post-image-generate) | imagevideo | Satellite image analysis system for identifying and categorizing geographical features, infrastructure, and environmental elements from aerial imagery. |
| [healthcare.patient-consent](/api-reference/v1/post-document-generate) | document | Extract the information from the provided document and fill in the consent model accordingly. If you aren't sure, leave it blank. |
| [healthcare.patient-identification](/api-reference/v1/post-document-generate) | documentimage | Extract the information from the provided document and fill in the identification model accordingly. If you aren't sure, leave it blank. |
| [healthcare.patient-insurance-card](/api-reference/v1/post-document-generate) | documentimage | Extract the information from the provided document and fill in the insurance card model accordingly. If you aren't sure, leave it blank. |
| [healthcare.patient-intake](/api-reference/v1/post-document-generate) | document | Extract the information from the provided document and fill in the intake form model accordingly. If you aren't sure, leave it blank. |
| [healthcare.patient-medical-history](/api-reference/v1/post-document-generate) | document | Extract the information from the provided document and fill in the medical history model accordingly. If you aren't sure, leave it blank. |
| [healthcare.patient-referral](/api-reference/v1/post-document-generate) | document | Extract the information from the provided document and fill in the patient referral record model accordingly. If you aren't sure, leave it blank. |
| [retail.ecommerce-product-caption](/api-reference/v1/post-image-generate) | image | Product data extraction system that processes product images to extract structured information including visual description, product details, and delivery information. |
| [retail.product-catalog](/api-reference/v1/post-image-generate) | image | Gain a competitive edge with real-time analytics from product catalog, helping you track trending topics, public sentiment, and influential quotes that shape audience perspectives. |
# MongoDB
Source: https://docs.vlm.run/integrations/integrations-mongodb
## Re-imagining ETL for Visual Content with VLM Run and MongoDB
As businesses amass ever-growing troves of unstructured customer data - including documents, PDFs, images, videos, and audio files - the challenge of extracting meaningful insights from this "dark data" has become increasingly critical. Traditional database approaches simply cannot handle the complexity and diversity of multi-modal enterprise content.
Vector search technologies have emerged as one of the first solutions, allowing organizations to embed and index these varied data sources en masse. This enables users to retrieve relevant files based on natural language queries, akin to the Retrieval Augmented Generation (RAG) workflow. However, this represents only the first step in realizing the full potential of multi-modal data.
### Embeddings are not Enough
While vector search provides a valuable coarse-grained retrieval capability, it has inherent limitations. Condensing an entire document or multiple paragraphs into a single vector representation often fails to capture the nuanced content and context that enterprise users require. Extracting precise information - such as a specific sales figure, the author of a report, or the insights contained in a data visualization - remains a significant challenge. Overcoming this requires more sophisticated indexing and analysis approaches that can parse the diverse modalities within enterprise data.
### Transforming Visual Content with VLM Run
We believe Visual Language Models (VLMs) hold the key to unlocking the true value of enterprise visual content. Enter [VLM Run](https://vlm.run/) - our highly specialized Vision Language Model that empowers organizations to accurately extract structured data from diverse visual sources such as images, documents, and presentations. This breakthrough capability, which we call ETL for visual content, allows businesses to seamlessly process and index unstructured visual data, transforming raw multi-modal information into valuable, queryable insights.
Here's an example of a slide from a financial presentation and the structured JSON output that VLM Run can extract:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"title": "Differentiated Operating Model",
"page_number": 7,
"description": "The slide presents a 'Differentiated Operating Model' for Selective Insurance, detailing their unique field model, franchise value, and distribution network. It also includes a pie chart showing the 2023 Net Premiums Written, with a total of $4 Billion distributed across different lines of insurance.",
"charts": [
{
"type": "pie",
"title": "2023 Net Premiums Written",
"description": "A pie chart showing the distribution of net premiums written by Selective Insurance in 2023, totaling $4 Billion. It is divided into three categories: Standard Commercial Lines (79%), Standard Personal Lines (10%), and Excess and Surplus Lines (11%).",
"data": "| Category | Percentage | \n| --- | --- | \n| Standard Commercial Lines | 79% | \n| Standard Personal Lines | 10% | \n| Excess and Surplus Lines | 11% |",
"caption": null
}
],
"tables": null,
"others": [
{
"data": "### Unique, locally based field model\n- Locally based underwriting, claims, and safety management specialists\n- Proven ability to develop and integrate actionable tools\n- Enables effective portfolio management in an uncertain loss trend environment\n\n### Franchise value distribution model with high-quality partners\n- Approximately 1,550 distribution partners selling our standard lines products and services through approximately 2,650 office locations\n - ~850 of these distribution partners sell our personal lines products\n - ~90 wholesale agents sell our E&S business\n - ~6,400 distribution partners sell National Flood Insurance Program products across 50 states\n\n> \"Everyone with Selective makes our customers feel like the #1 priority. The ease of working with Selective is unmatched.\" - Selective Agent",
"caption": null,
"title": null
}
]
}
```
Given this JSON output, enterprises can now easily store and query the extracted structured data alongside the raw visual content from their favorite document DB, enabling a wide range of use cases such as content discovery, business intelligence, and analytics.
### Pairing VLM Run with a Flexible Data Platform
To fully capitalize on the power of VLM Run, enterprises require a data platform that can handle the scale, diversity, and flexible schema of the extracted visual insights. This is where a modern, document-oriented NoSQL database like MongoDB excels.
MongoDB's support for JSON-like documents and flexible schema make it an ideal complement to VLM Run. By storing the structured data extracted from visual content directly in MongoDB, organizations can seamlessly query and analyze this information alongside their other multi-modal business data. The managed MongoDB Atlas platform further enhances this integration, providing enterprise-grade reliability, scalability, and ease of use.
### MongoDB: The Perfect Fit for VLM Run
MongoDB is a document-oriented NoSQL database that supports JSON-like documents utilizing a flexible schema. It is designed for scalability, flexibility, and performance, making it a popular choice for modern applications incorporating a lot of unstructured and multi-modal data. Since VLM Run can extract structured JSON from visual content, [MongoDB](https://mongodb.com/) and the managed [MongoDB Atlas](https://www.mongodb.com/products/platform/atlas-database) platform are a natural fit for storing and querying this structured data.
### Get Started with VLM Run and MongoDB
If you're eager to experience the transformative potential of VLM Run and MongoDB, we've created a step-by-step [Colab notebook](https://colab.research.google.com/drive/1Xsx3RxX1tmOQFJBQoX6ilRtj5prUhgtS) that walks through the integration process. Dive in and see how you can elevate your enterprise's visual content into a strategic advantage.
# n8n
Source: https://docs.vlm.run/integrations/integrations-n8n
## Overview
The [VLM Run plugin](https://www.npmjs.com/package/@vlm-run/n8n-nodes-vlmrun) for [n8n](https://n8n.io/) empowers developers to harness advanced Visual AI capabilities within their automated workflows. This integration allows for effortless processing of image and video data, enabling sophisticated visual analysis tasks to be seamlessly incorporated into enterprise automation pipelines.
## Key Features
* **Visual AI Processing**: Leverage state-of-the-art computer vision models for image and video analysis.
* **Flexible API Integration**: Easily incorporate VLM Run's Visual AI endpoints into n8n workflows.
* **Custom Model Support**: Use your own fine-tuned models or our pre-trained ones for specific visual tasks.
* **Scalable Processing**: Handle large volumes of visual data efficiently within your workflows.
## Use Cases
1. **Content Moderation**: Automatically flag inappropriate images or videos in user-generated content.
2. **Visual Quality Control**: Analyze product images for defects or inconsistencies in manufacturing processes.
3. **Document Processing**: Extract text and data from scanned documents, receipts, or business cards.
4. **Brand Monitoring**: Detect logo usage and brand presence in social media images and videos.
## Installation
Install the VLM Run node package in your n8n instance:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
npm install @vlm-run/n8n-nodes-vlmrun
```
Or install directly from the [NPM package](https://www.npmjs.com/package/@vlm-run/n8n-nodes-vlmrun).
## Getting Started
1. Install the VLM Run node in your n8n instance.
2. Configure the node with your VLM Run API credentials.
3. Add the VLM Run node to your workflow and connect it to other nodes.
4. Select the desired Visual AI endpoint and configure input/output parameters.
## Example Workflows
Explore these pre-built workflows to get started quickly:
* **[Video Summarization](https://n8n.io/workflows/5108-ai-video-summarization-with-vlm-run-automated-content-analysis-for-teams/)** - AI video summarization with VLM Run automated content analysis for teams
* **[Receipt Data Extraction](https://n8n.io/workflows/5051-extract-and-organize-receipt-data-for-expense-tracking-with-vlm-run-and-google/)** - Extract and organize receipt data for expense tracking with VLM Run and Google
* **[Resume Parser](https://n8n.io/workflows/5306-ai-resume-processing-and-github-analysis-with-vlm-run/)** - AI resume processing and GitHub analysis with VLM Run
* **[Audio Transcription](https://n8n.io/workflows/5307)** - Audio transcription workflow
## Community
**Need help?** Join our [Discord](https://discord.gg/tv9duU9QfU) channel or [contact our support team](mailto:support@vlm.run) for assistance with the VLM Run n8n integration.
# Voxel51 FiftyOne
Source: https://docs.vlm.run/integrations/integrations-voxel51
## Overview
In partnership with [Voxel51](https://voxel51.com/), the [VLM Run plugin for FiftyOne](https://github.com/vlm-run/vlmrun-voxel51-plugin) brings advanced Visual AI capabilities directly into the FiftyOne ecosystem. This integration enables computer vision teams to leverage VLM Run's vision-language models for extracting structured data from images, documents, and videos through FiftyOne's powerful visualization and dataset management interface.
FiftyOne is the leading open-source toolkit for building high-quality datasets and computer vision models. By combining VLM Run's specialized domains with FiftyOne's intuitive UI, teams can rapidly prototype, analyze, and iterate on visual AI workflows.
## Key Features
* **Seamless Integration**: Native FiftyOne operators for all VLM Run capabilities
* **Visual Grounding**: Precise bounding box localization for detected objects and extracted data
* **Multiple Domains**: Access 50+ specialized processing domains (object detection, document analysis, video transcription)
* **Interactive Visualization**: View and validate extraction results directly in FiftyOne's UI
* **Batch Processing**: Process entire datasets with immediate or delegated execution modes
* **Custom Schemas**: Use pre-built domains or define custom extraction schemas
## Use Cases
1. **Computer Vision Dataset Annotation**: Automatically annotate images with object detections, classifications, and segmentations
2. **Document Processing Pipelines**: Extract structured data from invoices, forms, and documents with visual grounding
3. **Video Analysis Workflows**: Transcribe and analyze video content with temporal grounding
4. **Quality Assurance**: Validate model outputs by comparing VLM Run extractions with ground truth
5. **Data Exploration**: Rapidly explore and filter datasets based on visual content
## Installation
Install the VLM Run plugin directly from GitHub:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
fiftyone plugins download \
https://github.com/vlm-run/vlmrun-voxel51-plugin
```
Install the required dependencies:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
fiftyone plugins requirements @vlm-run/vlmrun-voxel51-plugin --install
```
Refer to the [FiftyOne Plugins documentation](https://docs.voxel51.com/plugins/index.html) for more information about managing plugins.
## Configuration
Set your VLM Run API key as an environment variable:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
export VLMRUN_API_KEY="your-api-key-here"
```
You can obtain an API key from [vlm.run](https://vlm.run). Alternatively, you can provide the API key directly when running operators in the FiftyOne App.
## Getting Started
1. Launch the FiftyOne App with your dataset:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import fiftyone as fo
import fiftyone.zoo as foz
# Load a sample dataset
dataset = foz.load_zoo_dataset("quickstart", max_samples=10)
session = fo.launch_app(dataset)
```
2. Press `` ` `` or click the `Browse operations` action to open the Operators list
3. Select any of the VLM Run operators to process your data
## Available Operators
### Object Detection
Detect and localize common objects in images with bounding box coordinates using VLM Run's `image.object-detection` domain. The operator adds detections to your dataset with normalized bounding boxes, confidence scores, and object labels.
### Person Detection
Specialized person detection with enhanced accuracy for human-centric applications using the `image.person-detection` domain. Optimized for challenging scenarios including crowds and occlusions.
### Document Analysis
Extract text and analyze document structure from PDFs and images using the `document.markdown` domain. Extracts text content with spatial coordinates, document structure (headers, paragraphs, sections), tables and figures with bounding boxes, and reading order information.
### Invoice Parsing
Extract structured data from invoice documents with field-level visual grounding using the `document.invoice` domain. Extracts invoice totals, line items, vendor information, dates, and payment terms with optional visual grounding for each field.
### Layout Detection
Analyze document layout and identify structural elements with precise localization using the `document.layout-detection` domain. Identifies text regions, columns, headers, footers, tables, figures, and provides bounding boxes for each layout element.
### Video Transcription
Transcribe audio and analyze video content with multiple analysis modes using VLM Run's video understanding capabilities. Supported modes include:
* **transcription**: Audio-to-text transcription with timestamps
* **comprehensive**: Full video analysis (audio + visual + activities)
* **objects**: Object detection across video frames
* **scenes**: Scene classification and changes
* **activities**: Activity and action recognition
Each mode provides temporal information and can be combined for comprehensive video understanding.
## Visual Grounding
When enabled, visual grounding provides bounding box coordinates in normalized xywh format:
* `x`: horizontal position of top-left corner (0-1)
* `y`: vertical position of top-left corner (0-1)
* `w`: width of the bounding box (0-1)
* `h`: height of the bounding box (0-1)
This allows for precise localization of detected objects, text regions, or document elements directly on your images, which is essential for validation and compliance workflows.
## Execution Modes
All operators support two execution modes:
* **Immediate**: Process immediately in the FiftyOne App (default)
* **Delegated**: Queue for background processing (requires [orchestrator setup](https://docs.voxel51.com/plugins/using_plugins.html#delegating-plugin-operations))
## Supported Formats
* **Images**: JPEG, PNG, BMP, TIFF, and other common formats
* **Documents**: PDF files and document images
* **Videos**: MP4, AVI, MOV, MKV, WEBM, FLV, WMV, M4V
## Example Workflow
A typical workflow using VLM Run with FiftyOne:
1. **Load Dataset**: Import your images, documents, or videos into FiftyOne
2. **Select Operator**: Choose a VLM Run operator (e.g., object detection, invoice parsing)
3. **Configure Parameters**: Set domain-specific options and enable visual grounding if needed
4. **Execute**: Run the operator on selected samples or entire dataset
5. **Visualize Results**: View extracted data and bounding boxes in FiftyOne's UI
6. **Validate & Export**: Filter, validate, and export results for downstream use
This integration streamlines the entire visual AI pipeline from data ingestion to validated structured outputs.
## Community
**Need help?** Join our [Discord](https://discord.gg/AMApC2UzVY) channel or [contact our support team](mailto:support@vlm.run) for assistance with the VLM Run FiftyOne integration.
## Learn More
* [VLM Run Plugin GitHub Repository](https://github.com/vlm-run/vlmrun-voxel51-plugin)
* [Voxel51 Plugin Documentation](https://docs.voxel51.com/plugins/plugins_ecosystem/vlmrun_voxel51_plugin.html)
* [FiftyOne Documentation](https://docs.voxel51.com)
* [VLM Run API Reference](/api-reference)
# Zapier
Source: https://docs.vlm.run/integrations/integrations-zapier
## Overview
The [VLM Run integration](https://zapier.com/apps/vlm-run-ca212329/integrations) for [Zapier](https://zapier.com/) enables you to connect VLM Run's Visual AI capabilities with over 8,000 other apps. Automate your visual data processing workflows by seamlessly integrating document parsing, image analysis, and content extraction into your existing business processes.
## Key Features
* **8,000+ App Integrations**: Connect VLM Run with popular tools like Google Drive, Slack, Airtable, and more
* **No-Code Automation**: Build powerful visual AI workflows without writing code
* **Real-time Processing**: Automatically process documents and images as they arrive
* **Flexible Triggers**: Set up workflows based on file uploads, form submissions, or scheduled events
## Use Cases
1. **Automated Invoice Processing**: Extract data from invoices and send to accounting software
2. **Document Management**: Parse uploaded documents and organize data in spreadsheets
3. **Content Moderation**: Automatically analyze images for compliance before publishing
4. **Data Entry Automation**: Convert physical forms and receipts into structured data
## Getting Started
1. Visit the [VLM Run Zapier integration page](https://zapier.com/apps/vlm-run-ca212329/integrations)
2. Connect your VLM Run account with your API credentials
3. Choose a trigger app and configure when the workflow should run
4. Add VLM Run as an action to process your visual content
5. Connect additional apps to handle the processed data
## Pre-built Templates
Get started quickly with these ready-to-use templates:
* **[Markdown Extraction](https://zapier.com/webintent/create-zap?template=255650657)** - Convert documents to structured markdown format
* **[Invoice Extraction](https://zapier.com/webintent/create-zap?template=255650499)** - Automatically extract data from invoice documents
## Example Workflow
A typical Zapier workflow with VLM Run might look like:
1. **Trigger**: New file uploaded to Google Drive
2. **Action**: VLM Run processes the document
3. **Action**: Extracted data is added to a Google Sheets spreadsheet
4. **Action**: Team notification sent via Slack
This automation eliminates manual data entry and ensures consistent processing of your visual content.
## Community
**Need help?** Join our [Discord](https://discord.gg/tv9duU9QfU) channel or [contact our support team](mailto:support@vlm.run) for assistance with the VLM Run Zapier integration.
# Introduction
Source: https://docs.vlm.run/introduction
Extract JSON from images, videos, and documents with a unified API.
## What is VLM Run?
[VLM Run](https://vlm.run) is an end-to-end platform for developers to fine-tune, specialize, and operationalize Vision Language Models (VLMs). We aim to make VLM Run the go-to platform for running VLMs with a unified structured output API that’s versatile, powerful and developer-friendly.
VLM Run is built on top of **`vlm-1`**, a highly specialized **Vision Language Model** that allows enterprises to accurately extract JSON from diverse visual sources such as images, documents and presentations - a.k.a. ETL for any visual content. By leveraging `vlm-1`, enterprises can effortlessly process and index unstructured visual data into their existing JSON databases, transforming raw multi-modal and unstructured information into valuable insights and opportunities.
## What makes VLM Run unique?
Here are some key features of VLM Run that set it apart from other foundation models and APIs:
Robustly extract JSON from a variety of visual inputs such as images, videos, and PDFs,
and automate your visual workflows.
Fine-tune our models for specific domains and confidently embed vision in your application with enterprise-grade SLAs.
Scale your workloads confidently without being rate-limited or worried about your costs spiraling out of control.
Deploy your custom models on-prem or in a private cloud, and keep your data secure and private.
## Let's get started!
Below you'll find the API reference and code samples so you can start building for your use case.
Sign up for an API key on our [platform](https://app.vlm.run), then check out some of our [cookbooks](https://github.com/autonomi-ai/vlm-cookbook) to learn how to use VLM Run to perform fast, structured extraction on your visual data.
Sign-up on our VLM Run platform for API access.
Enough talk, show me the code.
Various cookbooks showcasing VLM Run in action.
Book a demo with our team to learn more.
# Chat
Source: https://docs.vlm.run/platform/chat
The interactive playground for chatting with Orion, VLM Run's visual agent
Chat is the primary playground on the VLM Run platform. It gives you a direct conversational interface with **Orion**, our visual agent that can see, reason over, and act on images, documents, and videos.
Use Chat to explore what Orion can do before integrating via the API, or as a daily tool for ad-hoc visual understanding tasks.
Try it now at [chat.vlm.run](https://chat.vlm.run/). No setup required.
## What You Can Do
Upload a PDF, invoice, medical form, or any multi-page document. Ask Orion to summarize it, extract specific fields, compare sections, or answer questions grounded in the document's content. Orion processes every page and returns structured, cited responses.
Drop in a photo, screenshot, or diagram. Orion can caption the image, detect and label objects, identify UI elements, extract text via OCR, or answer open-ended questions about what it sees. Results include bounding boxes and visual annotations when relevant.
Attach a video and ask Orion to summarize it, identify key moments, describe actions, or extract metadata across frames. Video understanding works across formats and durations, from short clips to multi-minute recordings.
Select a skill from the skill picker to guide Orion's response. Skills constrain the output to a specific schema (invoice line items, receipt totals, document classifications) so you get consistent, structured data every time. You can also create new skills directly from a conversation.
Chain operations in a single conversation: detect objects, crop a region, enhance it, then analyze. Orion maintains context across turns, so each step builds on the last.
## Key Features
| Feature | Description |
| --------------------------- | ------------------------------------------------------------------- |
| **File attachments** | Upload images, PDFs, videos, and audio directly in the chat window |
| **Skill picker** | Select a skill to constrain outputs to a specific schema |
| **Structured responses** | Get JSON-structured results alongside natural language explanations |
| **Conversation history** | Resume past conversations and iterate on results |
| **Create skills from chat** | Turn a successful conversation into a reusable skill in one click |
| **Real-time streaming** | Responses stream token-by-token for immediate feedback |
## From Playground to Production
Chat is designed to be the bridge between exploration and integration. A typical workflow looks like:
1. **Explore**: Upload a sample file and ask Orion a question to see what's possible.
2. **Refine**: Iterate on your prompt, select or create a skill to structure the output.
3. **Integrate**: Take the skill and model configuration from Chat and wire it into your API calls using the [Python SDK](/sdk-reference/getting-started), [Node.js SDK](/sdk-reference/node/getting-started), or [REST API](/api-reference/v1/post-chat-completions).
Every chat conversation is logged in the [Observe](/platform/observe/overview) dashboard, so you can inspect completions, debug responses, and track costs as you go.
## Related Pages
Use the Orion agent in conversational chat mode via the API.
Explore the chat completions REST endpoint.
Run chat sessions from the command line.
Learn how to create and manage skills on the platform.
# Completions
Source: https://docs.vlm.run/platform/observe/completions
Review model completions, token usage, and response quality on the VLM Run platform
Track chat completion requests made to the VLM Run platform, with details like model, token usage, status, and credit cost. Review outputs to understand how your visual agents are responding.
Everything you see here is also available through the API. See the [Chat Completions API reference](/api-reference/v1/post-chat-completions) to query completions programmatically.
## Completion details
Click on any completion to see the full chain of inputs and outputs rendered in a clean, easy-to-read view - useful for both reviewing results and debugging issues.
## Completions table
Each row represents one completion with:
| Column | Description |
| ------------------ | ----------------------------------------------------------------------------------- |
| **Model** | Which model generated the completion (e.g., `vlmrun-orion-1`, `vlmrun-orion-1:pro`) |
| **Skill / Domain** | The skill or domain applied, if any |
| **Status** | `success` or `error` |
| **Tokens** | Input and output token counts |
| **Latency** | Time from request to first token and total completion time |
| **Credits** | Credits consumed by this completion |
| **Timestamp** | When the completion was generated |
Filter by model, skill, status, or time range to narrow results.
## Completion detail
Click any row to inspect the full completion:
* **Messages**: The complete message history (system, user, assistant) that produced this completion
* **Structured output**: The JSON output if a skill or schema was applied
* **Raw response**: The unprocessed model output, including any tool calls or intermediate reasoning
* **Token breakdown**: Input tokens (prompt + images/files) vs. output tokens (response)
* **Timing**: Time to first token (TTFT) and total generation time
* **Feedback**: Submit quality ratings to build a feedback loop for model improvement
## What to look for
Review the structured output against expectations. Are fields populated correctly? Are there hallucinations or missing data? Use the feedback button to flag issues.
Compare input and output token counts across completions. If a skill is generating unexpectedly large outputs, the schema or prompt may need tightening.
Filter by model to compare how different models handle the same skill. Look at output quality, latency, and cost to choose the best model for your use case.
Sort by latency to identify slow completions. Cross-reference with token counts. High token completions naturally take longer, but unexpectedly slow low-token completions may indicate an issue.
## Related Pages
Return to the observability dashboard.
View the underlying API requests for each completion.
Reference for the chat completions endpoint.
Learn how feedback improves model outputs over time.
# Evaluations
Source: https://docs.vlm.run/platform/observe/evaluations
Measure and track the accuracy of your skills, agents, and request domains using feedback as ground truth.
VLM Run Evaluations compare **stored model outputs** from your organization’s traffic against **human corrections** recorded as feedback. They surface per-field accuracy, completion rates, and (for skill-based runs) structured hints you can feed into optimization and reruns.
## What is being compared?
For each item in scope (prediction request or agent execution, depending on the source), the platform:
1. Loads the **original structured response** the model produced (from durable storage).
2. Loads the **ground truth** from **feedback**: either JSON you supplied in the `response` field, or JSON **inferred** from text notes when [Infer corrections](#run-an-evaluation-in-the-dashboard) is enabled.
3. Runs one or more **evaluators** on those pairs and aggregates metrics.
Items in the selected window that never received feedback still count toward **totals** and **API completion rate**, but only corrected samples drive scorer metrics like field accuracy.
### How responses are flattened
Before scoring, both the model output and the ground truth are flattened into dotted leaf paths (e.g. `vendor_name`, `address.city`, `total_amount`). Three rules govern flattening:
* **Lists are atomic.** The whole list is kept as a single leaf value rather than recursed into. `line_items` is one leaf, not `line_items[0].description`. Each evaluator decides how to compare lists: `field_accuracy` does exact equality, `fuzzy_field_match` JSON-serializes both sides for similarity, and `llm_judge` evaluates the entire list semantically.
* **`_metadata` keys are skipped.** Any key containing `_metadata` (for example `_metadata`, `name_metadata`) is treated as internal bookkeeping and never scored.
* **Null ground truth is excluded by default.** When the corrected value for a leaf is `null`, that leaf is left out of the metric instead of being counted as a mismatch. Disable this with `skip_null_expected: false` if you want explicit nulls to participate in scoring.
## How evaluations work
1. **Collect feedback**: use the [Feedback & fine-tuning](/guides/feedback) flow so corrections are tied to request or execution IDs.
2. **Run an evaluation**: in the dashboard, pick a **skill**, **agent**, or **request domain**, set a date range, choose [evaluators](#evaluator-types), and start the job. The run is **asynchronous**: you get a run record immediately and results fill in when scoring finishes.
3. **Review and act**: inspect summaries, per-field breakdowns, and samples. For **skill** evaluations only, you can **Optimize** (new skill version) or **Rerun** (auto-optimize, re-process items, re-score).
## Run an evaluation in the dashboard
In the dashboard sidebar, choose **Evaluations** (same area as Overview, Requests, Executions, and Completions).
Click **New Evaluation** to open the configuration panel, then pick what you are scoring:
* **Skill**: one skill ID; includes matching **prediction requests** and **agent executions** that used that skill in the date range.
* **Agent**: one agent (ID and version); uses **agent executions** in range.
* **Request domain**: a hub domain string on requests (for example `document.invoice`); uses **prediction requests** for that domain.
Choose the time window. The UI shows a **preview**: total items, how many have feedback, how many include JSON corrections, and the latest activity timestamp, so you can confirm the slice before you spend credits.
Aim for at least **10–20 items with feedback** before treating metrics as stable.
Choose a specific inference model to scope the evaluation to (for example `gemini-3.1-flash-lite-preview`, `gemini-3.1-pro-preview`, `vlm-1`). When set, only requests and executions produced by that model are loaded into the run.
Leave it on **Default (auto-detect)** to score every item in the window regardless of which model produced it. The selected model is also the one used by **Rerun** when it has to skip inference (see [Rerun](#rerun-skill-evaluations-only)).
Select one or more strategies. Defaults lean on **Field accuracy**; see [Evaluator types](#evaluator-types) for tradeoffs.
Optionally restrict evaluation to specific fields by passing a list of top-level keys or dotted leaf-path prefixes (for example `vendor_name`, `address`, `line_items`). When omitted, all fields in the JSON output are evaluated.
This is useful when you only care about a few critical fields, or want to exclude noisy fields from accuracy metrics. See [Field selection](#field-selection) for matching rules and API examples.
**Infer corrections** is **on by default** in the dashboard. When it is on, the platform can turn **notes-only** feedback into structured JSON using an LLM, guided by your skill schema when available.
Inferred JSON is best-effort. For production-grade ground truth, prefer explicit JSON in feedback; see [Feedback & fine-tuning](/guides/feedback).
Submit the run. The run record is created immediately with status `running` and progresses through `running → completed` (or `failed` if scoring errors). The dashboard polls every few seconds and unlocks the result view once the status is `completed`.
## Review results
Open a **completed** run from the history table.
* **Overall accuracy**: rolled up from field-level matches vs mismatches (see field accuracy below), including samples without corrections where the model applies an "accepted" assumption for reporting.
* **Accuracy delta**: change vs the previous completed run for the same source label/type when one exists.
* **API completion rate**: share of in-scope items whose upstream API calls **completed** (vs failed or incomplete).
* **Total samples**: items evaluated in this run.
### Performance metrics
Completed runs also surface the cost and latency of the underlying API traffic so you can weigh accuracy against operational footprint.
| Metric | Meaning |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Model** | The inference model the items were produced by, when scoped via the Model selector or auto-detected from the most common `model_id` in the window. |
| **Avg / Total / p50 latency** | Latency of the upstream prediction or execution aggregated across items that completed within the window. |
| **Avg / Total credits** | Credits charged for the underlying calls. Useful when comparing two skill versions or two models on the same task. |
The same numbers feed the **Credits** and **Model** columns in the history table. Use the **VS Prev** column on the same table to see field-match deltas at a glance.
### Field breakdown
The **Field-by-Field Comparison** block lists each extracted field so you can see where the model disagrees with ground truth.
Use the row filters at the top right—**All**, **Errors**, or **Correct**—to narrow the table; **Errors** is useful when you only want fields that still need work.
| Column | Meaning |
| ------------ | ------------------------------------------------------------------------------------ |
| **Field** | Schema field name (for example `education[1].degree`). |
| **Correct** | Count of samples where this field matched the correction (shown in green in the UI). |
| **Errors** | Count of mismatches for that field (shown in red). |
| **Accuracy** | Share of samples where this field was correct. |
Rows are ordered so the weakest fields surface first. **Expand a field** to open an inline drill-down: a nested table with **Original** (model output) and **Corrected** (feedback) side by side so you can compare concrete values.
### Optimization hints (skill runs only)
When the source is a **skill**, completed results can include **optimization hints**: weakest fields, recurring themes from notes, a small set of representative failures, and short suggested next steps. These align with what the **Optimize** action uses when generating a new skill version.
## Metrics and history
The main Evaluations page aggregates recent runs:
* **Summary cards**: Total runs and counts by source type (Skills, Agents, Domains).
* **Accuracy trend**: Field-match accuracy across recent completed runs, filterable by source. Optional overlays show API completion rate, fuzzy-match rate, and exact-match rate when those evaluators were enabled.
* **Cross-run field view**: Weakest fields across history, with a Worst / Best toggle.
* **History table**: Status, source, model, credits, field-match accuracy, **VS Prev** delta, data range, and timestamps. Row actions depend on status and source type.
## Evaluation sources (API naming)
The product and API distinguish sources like this:
| Dashboard concept | `source_type` in APIs | What is scored |
| ----------------- | --------------------- | --------------------------------------------------------- |
| Skill | `skill` | Requests and executions using that skill ID in the window |
| Agent | `agent` | Executions for that agent ID/version |
| Request domain | `request_domain` | Requests tagged with that domain string |
The preview endpoint uses the same identifiers so you can validate counts before running.
## Evaluator types
You can combine evaluators in one run. In API payloads, they are defined as follows:
| UI label | API ID | Role | Field selection |
| -------------- | ------------------- | ------------------------------------------------------------------------------------------------------------- | --------------- |
| Field accuracy | `field_accuracy` | Default. Per-field leaf comparison with light type coercion (e.g. numeric `"1"` vs `1.0`). | Yes |
| Fuzzy match | `fuzzy_field_match` | String similarity on text leaves; **0.7** similarity threshold by default. Non-strings fall back to equality. | Yes |
| LLM judge | `llm_judge` | Semantic score from a judge model over the full structured output. | No |
| Exact match | `equals_expected` | Strict per-field equality for **short** expected values only (see below). | Yes |
### Field accuracy (`field_accuracy`)
Flattens nested JSON into leaf paths, compares model output to ground truth, and reports match rate per field and overall. This is the primary signal for the dashboard’s field table and optimization hints.
### Fuzzy field match (`fuzzy_field_match`)
For string leaves, uses the **better** of character-level and token-level similarity so short labels and long sentences are both scored fairly. A pair **passes** if similarity ≥ **0.7**; otherwise it fails that leaf’s fuzzy check.
### LLM judge (`llm_judge`)
Evaluates whether the structured output is **semantically** acceptable versus the correction, with a 0–1 score per sample. Use it when paraphrases or layout differences should not count as errors. Enabling it adds model latency and cost relative to deterministic evaluators.
### Exact match (`equals_expected`)
Compares leaves with **strict** equality (including whitespace and casing). To avoid long free-text fields dominating the metric, only leaves whose **expected** string length is **≤ 128 characters** participate; longer leaves are **skipped** for this evaluator’s aggregate. If nothing qualifies on a sample, that sample may be omitted from the exact-match rate.
**LLM judge** and **Exact match** are independent toggles—turning on the judge does not replace exact match; choose both only when you want both signals.
### Choosing a combination
| Scenario | Good starting set |
| --------------------- | -------------------------------------- |
| Structured extraction | `field_accuracy` + `fuzzy_field_match` |
| Lots of free text | Add `llm_judge` |
| Tight IDs / codes | Add `equals_expected` |
| Unsure | `field_accuracy` only |
## Field selection
By default, evaluations score **all fields** in the JSON output. You can optionally restrict scoring to a subset of fields by passing a `fields` list of top-level keys or dotted leaf-path prefixes.
**How matching works:**
| `fields` value | Matches |
| -------------- | ------------------------------------------------------ |
| `"name"` | `name`, `name.first`, `name.last` |
| `"address"` | `address`, `address.street`, `address.city`, and so on |
| `"line_items"` | `line_items` (the whole list as one atomic leaf) |
Lists are not recursed into. A leaf prefix like `line_items` matches the list itself, not `line_items[0].description`. To compare list contents semantically, enable **LLM judge** alongside `field_accuracy`.
Field selection applies to **Field accuracy**, **Fuzzy match**, and **Exact match**. The **LLM judge** always evaluates the entire response regardless of field selection.
**API usage.** Pass `fields` (and any other run options) in the run or rerun request body:
```json Run Evaluation theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"source_type": "skill",
"source_id": "",
"source_label": "Invoice extraction v3",
"model": "gemini-3.1-flash-lite-preview",
"data_from": "2026-04-01T00:00:00Z",
"data_to": "2026-04-30T23:59:59Z",
"evaluators": ["field_accuracy", "fuzzy_field_match"],
"fields": ["vendor_name", "total_amount", "line_items"],
"infer_corrections": true,
"skip_null_expected": true
}
```
```json Rerun theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"evaluation_id": "",
"skill_id": "",
"model": "gemini-3.1-flash-lite-preview",
"evaluators": ["field_accuracy"],
"fields": ["vendor_name", "total_amount"]
}
```
When `fields` is omitted or `null`, all fields are evaluated (backward-compatible). The same is true of `evaluators`, `infer_corrections`, and `skip_null_expected` when reused inside [Rerun](#rerun-skill-evaluations-only): omitted values are inherited from the original run's stored config.
## Interpreting accuracy
| Band | Reading | Next step |
| ----------- | ------- | -------------------------------------------------------- |
| **95–100%** | Strong | Watch for regressions after deploys |
| **85–95%** | Solid | Tighten weakest fields surfaced in breakdown |
| **70–85%** | Mixed | Inspect samples and hints; adjust prompt/schema |
| **\< 70%** | Risky | Consider Optimize (skills) or deeper instruction changes |
## Actions on a run
### Optimize (skill evaluations only)
Available when `source_type` is **skill** and the run **completed**. The service samples up to **30** response/correction pairs by default (configurable up to **200** in the API), calls the skill optimizer with a Gemini model, and creates a **new skill version** with the same name. The optimizer produces:
* A **prompt addendum** appended to the original prompt that captures generic error patterns (no actual data values).
* **Per-field schema hints** that augment the JSON Schema's field descriptions for the columns that had the most errors.
You land on the skill detail page when it succeeds.
### Rerun (skill evaluations only)
Also **skill-only** in the UI. **Rerun** calls the platform rerun pipeline:
1. **Optimizes the source skill first** unless you pass a different skill via API.
2. **Re-executes** the underlying requests and executions against the new skill, swapping the model when one was selected.
3. **Copies feedback forward** onto the new request and execution IDs (LLM-inferring corrections from notes when enabled).
4. **Re-scores** the new traffic and writes the result to a fresh run row.
If you already have results for the target `(skill, model)` pair in the same window, Rerun **skips re-execution** and scores the existing items directly. This makes A/B comparisons across skill versions or models much cheaper when the new skill has already been used in production.
**Settings inheritance.** When you omit `evaluators`, `fields`, `infer_corrections`, or `skip_null_expected` on the rerun request, the values are inherited from the original run's stored `config`. Pass any of them explicitly to override. This lets you re-score the same window under different conditions (for example switching `evaluators` to `["llm_judge"]`) without rebuilding the full request.
Agent and domain evaluations still give you full metrics and samples, but **Optimize** and **Rerun** are not shown because those flows require a skill to re-execute against.
### Delete
Removes the run record (typically allowed for **completed** or **failed** runs). It does not delete underlying requests, executions, or feedback.
## Continuous improvement
With evaluations, you can continuously improve your processes:
## Programmatic feedback
Evaluations **read** feedback you already stored; they do not replace the feedback API. For submission formats, entity IDs (`request`, `agent_execution`, `chat`), and examples, use [Feedback & fine-tuning](/guides/feedback) and [Submit feedback](/api-reference/v1/post-submit-feedback).
The dashboard’s **run**, **optimize**, and **rerun** actions call internal evaluation services (not the public `api.vlm.run` OpenAPI bundle). Treat the dashboard as the supported surface for those operations unless your integration team has exposed the same routes to you.
## Best practices
* **Prefer JSON corrections** in feedback; use **Infer corrections** when you only have notes.
* **Use field selection** to focus metrics on the fields that matter most, especially during iterative improvement.
* **Match evaluator choice to schema**: fuzzy for noisy text fields, judge for semantic leniency, exact for short codes.
* **Read optimization hints** on skill runs before clicking Optimize to confirm the weakest fields make sense.
* **Use Rerun** as an end-to-end A/B after Optimize, since it reprocesses real inputs rather than only rescoring old JSON.
* **Keep date windows meaningful**: very wide windows mix old prompts with new ones and blur trends.
## Related pages
Ground truth tied to requests and executions.
HTTP reference for structured feedback.
# Executions
Source: https://docs.vlm.run/platform/observe/executions
Track agent and skill executions end to end on the VLM Run platform
Track agent and skill executions on the VLM Run platform, with details like status, duration, and credit cost. Run single executions or batch workflows, inspect results, and provide feedback to help improve future outputs.
## Execution Details
On the left side you can see the request JSON and the inputs you provided to the execution. On the right side you can see the JSON output, along with Markdown and Form views where applicable.
The JSON output lets you quickly verify correctness, and displays bounding box accuracy if grounding was applied to the input.
## JSON Form
The JSON Form tab switches the right panel to a structured view that lets you provide field-level feedback on the execution. For example, if you're analyzing a 10-K and the EPS value is incorrect, you can update it and click save.
You can also submit field-level feedback through the API. See the [feedback endpoint](/api-reference/v1/post-submit-feedback).
## Diff View
Once feedback is provided, you can view a diff by clicking **JSON** and then the diff icon to its left. This makes it easy to scan and understand the changes made through feedback.
## Execution Feedback
If you don't need field-level feedback, you can provide execution-level feedback by clicking the note button in the top right of the view.
You can also submit execution-level feedback through the API. See the [feedback endpoint](/api-reference/v1/post-submit-feedback).
## Executions Table
| Column | Description |
| ----------------- | ---------------------------------------------------------------------- |
| **Agent / Skill** | Which agent or skill was executed |
| **Status** | `running`, `completed`, `failed`, or `cancelled` |
| **Duration** | Total wall-clock time for the entire execution |
| **Steps** | Number of intermediate steps (model calls, tool uses) in the execution |
| **Credits** | Total credits consumed across all steps |
| **Timestamp** | When the execution was triggered |
Filter by agent, skill, status, or time range. Sort by duration or credits to find outliers.
## Single vs. Batch Executions
The platform supports both:
* **Single executions**: One file, one skill, one result. Ideal for interactive use and testing.
* **Batch executions**: Process multiple files in a single run. The executions table shows the batch as a group, and you can expand it to see individual results.
## Debugging Failures
When an execution fails, the detail view shows:
1. **Which step failed**: Highlighted in the timeline
2. **Error message**: The specific error returned by the model or tool
3. **Input that caused the failure**: The exact payload that triggered the error
4. **Preceding successful steps**: Context for what worked before the failure
This makes it straightforward to isolate whether the issue is in the prompt, the schema, the input file, or a transient infrastructure problem.
## Related Pages
Return to the observability dashboard.
View individual API request logs.
Trigger agent executions programmatically.
Understand the artifacts produced by executions.
# Observe
Source: https://docs.vlm.run/platform/observe/overview
Full observability for your visual AI: requests, executions, completions, and usage metrics
The Overview dashboard gives you a real-time look at your VLM Run platform usage, including total activity, success rate, average latency, and credits used. Track requests, executions, and completions to monitor performance over time.
Everything you see here is also available through the API. See the [API reference](/api-reference/index) to query requests, executions, and completions programmatically.
## Filtering
Click the date picker in the top right corner to filter the data to any specific timeframe you want to analyze.
You can also click the grouping button in the top right of the two large charts to change the time grouping - choose from Daily, Weekly, Monthly, Quarterly, or Yearly.
Hover over the chart to see the data breakdown and credits used for each grouped time period.
## Drill Down
Clicking anywhere on a chart takes you to a drill-down view of that data - an easy way to see the requests, executions, and completions that ran during that time period. Clicking an individual item takes you to its details page.
## Dashboard Metrics
The overview page shows four key indicators at a glance:
| Metric | What it tells you |
| ------------------- | ------------------------------------------------------------------------------- |
| **Total Activity** | The number of requests, executions, and completions in the selected time window |
| **Success Rate** | The percentage of calls that completed without errors |
| **Average Latency** | Mean response time across all endpoints, broken down by type |
| **Credits Used** | Total credit consumption with trend over time |
## Three Views, One Story
Observe is organized into three complementary views that let you drill down from high-level metrics to individual outputs:
Model requests with status, duration, and cost.
Agent executions with step-by-step traces, artifacts, and timing.
Chat completions with model, token usage, and the full input and output payload.
## Typical Workflows
Filter Requests by status `error`, find the failing call, and inspect the request payload and error response. Cross-reference with the Completion to see what the model actually returned.
Filter Requests or Completions by skill name to see how many credits each skill is consuming. Identify expensive skills and optimize prompts or schemas to reduce token usage.
Review Completions across different models or skill versions to compare output quality, latency, and cost. Use this to decide when to promote a new skill version to production.
Check the overview dashboard for success rate drops or latency spikes. Set up alerts via webhooks when metrics cross thresholds.
## Related Pages
View and filter individual API requests.
Track agent and skill executions.
Review model completions and outputs.
Run accuracy evaluations from feedback and review field-level metrics.
Explore all available API endpoints and responses.
# Requests
Source: https://docs.vlm.run/platform/observe/requests
View, filter, and inspect every API request on the VLM Run platform
Track API requests made to the VLM Run platform, with details like status, duration, and credit cost. Inspect results and provide feedback to help improve future outputs.
## Filtering Requests
Once you're on the Requests page, you can search and filter to find a specific request.
To zero in on failures, click the filter icon and filter by status - along with a range of other available filters.
Clicking a bar in the graph at the top filters the table to that specific day.
## Creating Requests
The Requests page isn't only for observability - click the **+** button on the right to upload one or more files and select the model and detail level for your request. You can also enable confidence scoring and visual grounding on the request.
You can do this through the API as well. See the [Generate API reference](/api-reference/v1/post-image-generate) for the full set of options.
## Using Skills in Requests
To make a request useful, provide guidance for what you want extracted - the simplest way is to select a skill. The skill handles the prompt, output schema, and plan for the request.
Learn more about skills and how to create them in the [Skills reference](/platform/skills/overview).
## Request Details
Click any request to see the input file alongside the output. The output can be viewed as JSON, Markdown, or Form view - see [Executions](/platform/observe/executions) for details on field-level feedback.
You can provide feedback on individual output fields as well as a note on the request as a whole.
You can submit feedback through the API as well. See the [feedback endpoint](/api-reference/v1/post-submit-feedback).
## Request Table
| Column | Description |
| ------------------ | ------------------------------------------------------------------------ |
| **Endpoint** | The API path called (e.g., `/v1/image/generate`, `/v1/chat/completions`) |
| **Status** | HTTP status code: `200` for success, `4xx`/`5xx` for errors |
| **Duration** | Wall-clock time from request received to response sent |
| **Credits** | The number of credits consumed by this request |
| **Timestamp** | When the request was received |
| **Skill / Domain** | The skill or domain referenced in the request, if any |
## Common Use Cases
* **Audit trail**: See exactly what was sent and returned for any request, useful for compliance and debugging.
* **Cost analysis**: Sort by credits to identify the most expensive requests and optimize accordingly.
* **Error investigation**: Filter for `4xx` or `5xx` status codes to find and diagnose failures.
* **Skill performance**: Filter by skill name to compare success rates and latency across skill versions.
## Related Pages
Return to the observability dashboard.
Track multi-step agent executions.
Reference for image, document, audio, and video generate endpoints.
Run generate requests from the command line.
# Platform
Source: https://docs.vlm.run/platform/overview
The VLM Run platform: chat with visual agents, build skills, and observe every request in one place
The VLM Run platform is where you interact with, build on, and monitor your visual AI. Whether you're chatting with Orion to understand a PDF, building a reusable skill for invoice extraction, or tracking the latency and cost of every API call, it all happens here.
The platform is organized around four pillars:
* **[Observe](https://app.vlm.run/dashboard/overview)**: Full observability across your visual AI usage. Monitor requests, track executions, review completions, and keep costs in check.
* **[Skills](https://app.vlm.run/dashboard/skills)**: Modular, reusable capabilities that tell the model what to extract and how to structure it. Create once, reference from any endpoint.
* **[Chat](https://chat.vlm.run/)**: The interactive playground for your visual agent. Attach images, PDFs, or videos and get structured responses in real time.
* **[Evaluations](/platform/observe/evaluations)**: Measure accuracy for skills, agents, and domains using feedback as ground truth, with dashboard runs and per-field metrics.
## Explore the Platform
Monitor requests, executions, and completions across the platform in one single pane of glass.
Create, edit, and manage reusable extraction skills for your team.
View and filter all API requests with status, duration, and cost.
Track agent and skill executions end to end.
Send messages to Orion, attach files, and get structured visual responses.
Browse model completions with token usage and output details.
## Quick Links
Jump straight into the playground and chat with Orion for free.
Sign in to the VLM Run platform to manage your account.
Integrate programmatically with the VLM Run REST API.
Deep-dive into the skill specification and lifecycle.
# Settings
Source: https://docs.vlm.run/platform/settings
Manage API keys, team members, billing, and account preferences
The Settings page is where you manage everything about your VLM Run account. It is organized into three sections: **[API Keys](https://app.vlm.run/dashboard/settings/api-keys)**, **[Team](https://app.vlm.run/dashboard/settings/organization)**, and **[Billing](https://app.vlm.run/dashboard/settings/billing)**.
## API Keys
Create and manage the API keys used to authenticate requests to the VLM Run API.
* **Create new keys** for different users and projects (coming soon).
* **Revoke keys** instantly if compromised or no longer needed.
* **View usage** per key to track which integration is consuming credits
Treat API keys like passwords. Never commit them to source control or share them in plain text. Use environment variables or a secrets manager instead.
Set your key as the `VLMRUN_API_KEY` environment variable so the Python SDK, Node.js SDK, and CLI pick it up automatically.
## Team
Manage who has access to your VLM Run workspace. All team members share access to skills, chat history, the Observe dashboard, and billing.
* **Invite a member** Send an email invitation to a new team member. They will receive a link to join your workspace.
* **View members** See the list of all current members along with their email and role.
* **Remove a member** Revoke a member's access to the workspace. Their API keys (if any) are also revoked.
## Billing
View your current plan, credit balance, and usage trends. Upgrade or downgrade your plan, and purchase additional credits directly from this panel. See [Pricing](/pricing) for model token rates, fixed-price tools, and service-tier multipliers.
| | **Pay-as-You-Go** | **Pro Plan** | **Enterprise** |
| ------------------ | ----------------------------------------- | -------------------------------- | ------------------------------------------------ |
| **Price** | \$0.01 per credit/month (billed on usage) | \$799/month | Custom |
| **Billing** | Monthly based on usage | Monthly subscription + Overages | Invoiced, tier-based with volume discounts |
| **Credits** | 100 free on sign-up, then pay as you go | Up to 100K images or pages/month | Custom |
| **Rate limit** | Up to 10 requests/minute | Up to 100 requests/minute | Custom |
| **Support** | Community | Dedicated Slack | Dedicated Slack + custom SLAs + priority support |
| **Data retention** | Standard | Zero-Data Retention (ZDR) | ZDR, In-VPC deployments |
| **Compliance** | | | SOC2, HIPAA, BAA execution |
From the Billing panel you can:
* **View credit balance**: See how many credits remain in your current billing cycle and when the cycle resets.
* **Track usage trends**: A chart shows daily credit consumption so you can spot spikes or plan for scaling.
* **Upgrade your plan**: Move from Starter to Pro, or contact sales for an Enterprise agreement.
* **Purchase additional credits**: Buy top-up credit packs if you need more before your cycle resets.
* **View invoices**: Download past invoices for your records.
## Related
Sign in to the VLM Run platform to manage your account.
Create and manage the API keys.
Manage who has access to your workspace.
View or upgrade your current plan, credit balance, and usage trends.
# Skills
Source: https://docs.vlm.run/platform/skills/overview
Create, edit, and manage reusable visual extraction skills on the VLM Run platform
Skills are the building blocks of the VLM Run platform. A skill is a modular, reusable capability that tells Orion exactly **what to extract** from a visual input and **how to structure** the output. Once created, a skill can be referenced from any endpoint (images, PDFs, video, audio, or agent workflows), producing consistent, schema-validated results every time.
Think of skills as "visual functions": define the input type, describe the task, set the output schema, and call it from anywhere.
## Skills Lifecycle
Define a new skill through a [chat conversation](/platform/chat) or by [uploading a skill package](/skills/spec/overview). Specify the task description, output schema, and test it against sample files.
Refine the skill's prompt, schema, or metadata from the platform. Publish new versions without breaking existing integrations. Callers on a pinned version continue working unchanged.
Reference the skill in [chat](/platform/chat), the [REST API](/api-reference/v1/post-chat-completions), or the [SDK](/sdk-reference/getting-started). Combine multiple skills in a single request for complex extraction pipelines.
## Navigating Skills
The skills table lists every skill in your workspace. Filter by name, domain, or version, and click any row to open the detail editor. From here you can see usage stats, version history, and the current schema.
## Configuring Skills
The detail view shows the full skill definition: task description, JSON schema, sample inputs and outputs, and version history. Edit any field and publish a new version directly from this page.
## Best Practices
| Principle | Why it matters |
| ----------------------------- | ------------------------------------------------------------------------------------ |
| **Specific task description** | Narrow prompts produce more accurate, consistent outputs than broad ones |
| **Tight output schema** | A well-defined JSON schema eliminates ambiguity and makes downstream parsing trivial |
| **Representative test cases** | Testing against diverse samples catches edge cases before production |
| **Versioning** | Pin consumers to a version so schema changes don't break integrations |
## Dive deeper
Step-by-step guide to creating a skill on the platform.
How to update prompts, schemas, and publish new versions.
Reference skills in chat, API calls, and agent workflows.
The full specification format: skill.md, vlmrun.yaml, and schema.json.
Control which version consumers use in production.
Create and manage skills programmatically with the REST API.
# Pricing
Source: https://docs.vlm.run/pricing
Dollar and token based pricing for VLM Run Orion agents.
VLM Run Orion agents are priced in **US dollars**. Every response tells you exactly what a run cost.
## What makes up your cost
Every run's cost comes from two components:
* **Fixed-price tools:** GPU and API actions charged at a flat rate per unit.
* Examples: image generation and editing, OCR and layout, and video generation and editing.
* **Token-billed model usage:** Orion reasoning and LLM-backed tools charged by input and output tokens.
* Examples: image captioning, document extraction, and video analysis.
A single run typically combines both: the agent reasons (tokens), calls a tool or two, and returns a result with the total attached.
## How a run adds up
Your cost is the sum of the two components, then scaled by your [service tier](#service-tiers):
```
run cost = token-billed usage + fixed-price tools
total cost = run cost × service-tier multiplier
```
## The per-span cost ledger
Every Orion 2 run returns a **per-span cost ledger**: a line item for each billable action, priced as either a fixed-price tool or token-billed usage. This is where the two components become concrete, so you can see exactly what drove a run's cost.
| Span Type | Priced By | Example |
| --------------------- | -------------------------------------- | ------------------------------------------------- |
| `mllm` | Model tokens (input + cached + output) | One LLM reasoning turn |
| `execute_code` | Fixed price (\$0.001/call) | One sandbox code execution |
| `tool` (fixed-price) | Fixed dollar price | `vlmrun.image.generate`, `vlmrun.image.segment` |
| `tool` (token-billed) | Model tokens consumed | `vlmrun.document.extract`, `vlmrun.video.caption` |
The multiplier is applied once, at the run level (`effective_cost = SUM(span.cost) × multiplier`), and the response hands you the totals directly, so you never have to add up spans yourself:
| Response Field | What It Tells You |
| ----------------------- | -------------------------------------------------------- |
| `cost_dollars` | What you actually pay, after the service-tier multiplier |
| `standard_cost_dollars` | Baseline cost at the standard tier |
| `savings_dollars` | Amount saved versus standard (positive on `flex`) |
| `service_tier` | Which delivery tier ran the request |
### An example ledger
Here is an illustrative, ballpark example for a single document processed by Orion 2 Auto in [program mode](/agents/code-execution). Actual token usage and cost vary with the document and the requested output.
| Span | Billed Usage | Cost |
| --------------------------- | ------------------------------------- | ------------ |
| `execute_code` | Fixed price | \$0.0010 |
| `vlmrun.document.extract` | 6,000 input + 1,000 output tokens | \$0.0020 |
| `vlmrun.document.grounding` | Fixed price | \$0.0010 |
| **Total** | **2 fixed-price spans, 7,000 tokens** | **\$0.0040** |
Running the generated program contributes the fixed `execute_code` charge, extraction is billed on the tokens it consumed, and grounding adds a second fixed charge. The run total is the sum of the three spans, before the service-tier multiplier is applied.
For the full rate card behind each span, see the [Pricing Reference](/pricing/reference).
## Choose your tiers
Two independent dials control quality and delivery. Quality picks the models; the service tier scales the whole bill.
### Model quality
Use Fast, Auto, or Pro as a curated default, or pass a specific model ID to pin a backbone. Token rates vary by model; see the [Pricing Reference](/pricing/reference#orion-2-model-token-rates) for the full rate card.
| Tier | Model ID | When to Use |
| -------- | --------------------- | ------------------------------------------------------------------------ |
| **Fast** | `vlmrun-orion-2:fast` | Speed and cost-efficiency. Lighter models, fastest processing. |
| **Auto** | `vlmrun-orion-2:auto` | Recommended default. Uses Fast tools, upgrades to Pro for complex tasks. |
| **Pro** | `vlmrun-orion-2:pro` | Maximum quality. More powerful models, higher cost, longer processing. |
VLM Run hosts additional Orion-2 backbones (open-weight and frontier models) beyond these three tiers. Pass any supported `model` ID on [chat completions](/api-reference/v1/post-chat-completions) or agent execute to pin a backbone. Per-model input/output rates are in the [Pricing Reference](/pricing/reference#orion-2-model-token-rates).
### Service tiers
Same models, same tools, same output: the service tier only changes how quickly the result comes back. Batches, backfills, and eval sweeps that nobody is waiting on cost half as much at `flex`; save the 1.8× `priority` premium for when a person is waiting on the response.
Tier
Multiplier
Cost Effect
Best For
standard(default)
1.0×
Base USD price
Most production workloads.
flex
0.5×
50% of standard
Background jobs with no one waiting: overnight batches, backfills.
priority
1.8×
180% of standard
Interactive or blocking workflows where latency is user-visible.
A \$1.00 standard run costs \$0.50 at `flex` and \$1.80 at `priority`. Set it per request:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Flex tier: 50% discount on total agent cost
response = client.agent.execute(
name="my-orion-agent",
inputs={...},
service_tier="flex",
)
# response.usage.cost_dollars → effective cost
# response.usage.savings_dollars → amount saved
```
```javascript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Priority tier: 1.8x premium, lowest latency
const response = await client.agent.execute({
name: "my-orion-agent",
inputs: {...},
serviceTier: "priority",
});
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/agent/execute \
-H "Authorization: Bearer $VLMRUN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "my-orion-agent",
"inputs": {...},
"service_tier": "flex"
}'
```
## What a run actually costs
See the following worked examples. The `cost_dollars` field in the response is always the exact figure.
| Component | Type | Auto |
| -------------------------- | ---------------- | ------------ |
| Segment objects in 1 image | Fixed-price (1×) | \$0.01 |
| Generate 1 image | Fixed-price (1×) | \$0.04 |
| LLM orchestration | Tokens (approx.) | \~\$0.01 |
| **Estimated total** | | **\~\$0.06** |
Fixed-price tools dominate; token orchestration is a small share.
| Component | Type | Standard Cost |
| ------------------------------- | ------------------------ | -------------------- |
| OCR & layout (50 pp) | Fixed-price: \$0.01/page | \$0.50 |
| Parse / extract data | Token-billed | Varies by complexity |
| LLM orchestration | Tokens (approx.) | \~\$0.02–\$0.05 |
| **Estimated standard subtotal** | | **\~\$0.52–\$0.55** |
| Flex service tier | 0.5× multiplier | × 0.5 |
| **Estimated total with Flex** | | **\~\$0.26–\$0.28** |
## Full rates
Every fixed-price tool and per-model token rate, plus the per-span ledger and cost formula.
Per-token rates for VLM Run Gateway models, metered on every request.
# Pricing Reference
Source: https://docs.vlm.run/pricing/reference
Full rate card for VLM Run Orion agents: fixed-price tools, token rates, and the per-span cost ledger.
This is the complete rate card for [Orion agent pricing](/pricing). For the value overview, cost components, and worked examples, start with the [Pricing](/pricing) page.
## How a run cost is calculated
The standard dollar cost is the sum of model-token usage and fixed-price tools. The service-tier multiplier is applied once at the run level:
```
model_cost = (
uncached_input_tokens × input_rate
+ cached_input_tokens × cached_rate
+ output_tokens × output_rate
) ÷ 1,000,000
standard_cost_dollars = model_cost + fixed_tool_cost
cost_dollars = standard_cost_dollars × mode_multiplier
```
Where:
* **model\_cost** includes orchestration tokens and token-billed tool usage. Orchestration uses the rates for the `model` ID you selected; tool tokens are itemized on the per-span ledger
* **fixed\_tool\_cost** is the sum of fixed-price tools such as segmentation, generation, OCR, and sandbox execution
* **mode\_multiplier** is determined by the service tier: 1.0× standard, 0.5× flex, or 1.8× priority
Every run itemizes these components into a [per-span cost ledger](/pricing#the-per-span-cost-ledger), applying the multiplier once at the run level (`effective_cost = SUM(span.cost) × mode_multiplier`). The rates that feed each span are below.
## Token rates
A run has two kinds of token usage:
* **Orchestration tokens**: the Orion agent reasoning, dispatching tools, and writing the response. These bill at the rates for the `model` ID you selected.
* **Tool tokens**: the LLM-backed tools the agent calls. These are billed by the tokens each tool consumes, at the rate for the model that ran that tool (which may differ from the `model` ID you selected). The ledger itemizes each span.
The ledger is the source of truth for each charge: it records the token counts behind every span. These tools are token-billed rather than fixed-price:
| Tool | Category | Billing Basis |
| -------------------------------- | -------- | ----------------------------------------------------------------------------------------------------- |
| Caption & Tag | Image | LLM tokens |
| Detect, Point | Image | LLM tokens |
| UI Parsing | Image | **Free** |
| Document Parsing / Extract / VQA | Document | LLM tokens (+ \$0.001/page when `grounding` or `confidence` is enabled on Orion-2 `document.extract`) |
| Caption, Summary & Transcribe | Video | LLM tokens |
Quality tiers (`fast` / `auto` / `pro`) use the tier rates below. Pinned Orion-2 backbones use model-specific input/output rates. The full list of accepted model IDs is in the [Chat Completions API reference](/api-reference/v1/post-chat-completions).
### Orion 2 model token rates
Rates are USD per 1M tokens. Orchestration tokens use the rates for the `model` ID you selected. Tool-token spans are itemized on the ledger. Cached input tokens are billed at the listed fraction of the input rate.
#### Quality tiers
Curated Orion-2 tiers. Fast and Auto share the same token rates; Pro is higher.
Model ID
Input
Output
Cached input
vlmrun-orion-2:fast
\$0.30
\$2.50
10% of input
vlmrun-orion-2:auto
\$0.30
\$2.50
10% of input
vlmrun-orion-2:pro
\$1.00
\$10.00
10% of input
#### Open-weight pins
These open-weight Orion-2 backbones bill at the Fast / Auto token rates:
Model ID
Input
Output
Cached input
vlmrun-orion-2:qwen3.6-35b-a3b
\$0.30
\$2.50
10% of input
vlmrun-orion-2:gemma4-26b-a4b
\$0.30
\$2.50
10% of input
vlmrun-orion-2:cosmos3-nano
\$0.30
\$2.50
10% of input
#### Pinned hosted backbones
These variants bill at model-specific pass-through rates (not the Fast / Auto / Pro tier rates):
Model ID
Input
Output
Cached input
Thinking
vlmrun-orion-2:kimi-2.6
\$0.66
\$3.41
22% of input
N/A
vlmrun-orion-2:muse-spark-1.1
\$1.25
\$4.25
10% of input
N/A
vlmrun-orion-2:gemini-flash-3.6
\$1.50
\$7.50
10% of input
\$7.50
vlmrun-orion-2:grok-4.5
\$2.00
\$6.00
25% of input
N/A
vlmrun-orion-2:opus-4.8
\$5.00
\$25.00
10% of input
N/A
vlmrun-orion-2:gpt-5.5
\$5.00
\$30.00
10% of input
N/A
Cached input is billed at the listed fraction of the input rate (for example, 10% of input means cached tokens cost \$0.10 when input is \$1.00 per 1M). Thinking tokens, when present, bill at the Thinking rate. Costs scale with tokens consumed, so simple inputs are cheaper than complex ones. See [model variants](/agents/code-execution#model-variants) for when to pin each backbone.
### LLM orchestration
Every agent run includes LLM token costs for reasoning, tool dispatch, and response synthesis, charged at the selected `model` ID's input/output rates. Pinned frontier backbones (for example `vlmrun-orion-2:gpt-5.5` or `vlmrun-orion-2:opus-4.8`) use their own model-specific rates from the table above. A typical single-tool Fast or Auto run adds approximately \$0.01–\$0.03 (or \$0.03–\$0.08 on Pro) in orchestration costs. Multi-step and frontier-model runs will be higher.
## Fixed-price tools
These tools have a fixed dollar price per unit.
### Image
Priced **per image**. Fast and Auto share the same rates; Pro is higher where noted.
Capability
Description
Fast / Auto
Pro
Segment
Object segmentation (GPU-backed)
\$0.01
\$0.02
Generate & Edit
Generate or edit images from prompts
\$0.04
\$0.24
### Document
Priced **per page**.
Capability
Description
Fast / Auto
Pro
OCR & Layout
Extract text and detect page structure
\$0.01
\$0.04
Extract grounding / confidence
Orion-2 document.extract surcharge when grounding or confidence is enabled
\$0.001
\$0.001
### Video
Priced **per second** (generated video).
Capability
Description
Fast / Auto
Pro
Generate & Edit
Generate video from text/image prompts
\$0.15/sec
\$0.40/sec
Standard video generation produces \~6 seconds. At Fast / Auto pricing, \$0.15/sec × 6s = \$0.90; at Pro, \$0.40/sec × 6s = \$2.40.
### Compute & execution
| Item | Fixed Price |
| ---------------------------------- | ------------ |
| `compute.execution` (code sandbox) | \$0.001/call |
## Legacy pricing (`vlm-1`)
VLM Run is transitioning to dollar and token based pricing for Orion 2. The `vlm-1` model family and older per-request rates below are kept for reference during the transition and may be removed in a future update. New workloads should use Orion 2 pricing above.
The `vlm-1` model family is available on `/v1/image/generate` and `/v1/document/generate` routes with three quality tiers:
| Model ID | Tier | Detail Level |
| ------------ | -------------- | ------------ |
| `vlm-1:fast` | Fast | `lo` |
| `vlm-1:auto` | Auto (default) | `auto` |
| `vlm-1:pro` | Pro | `hi` |
Domain API token rates are in USD per 1M tokens:
| Model ID | Input | Output | Cache Discount | Thinking |
| ------------ | ------ | ------ | -------------- | -------- |
| `vlm-1:fast` | \$0.25 | \$1.50 | 10% | N/A |
| `vlm-1:auto` | \$0.26 | \$1.56 | 10% | N/A |
| `vlm-1:pro` | \$1.50 | \$7.50 | 10% | \$7.50 |
Set the model via `request.model`:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
response = client.document.generate(
file="invoice.pdf",
domain="document.invoice",
model="vlm-1:auto",
)
```
## FAQ
**Fixed-price** tools (segmentation, image generation, video generation, OCR, sandbox execution) have predictable dollar prices per unit. **Token-billed** tools (captioning, detection, document parsing, video analysis) bill by the tokens they consume, itemized on the per-span ledger.
Open-weight pins (`vlmrun-orion-2:qwen3.6-35b-a3b`, `vlmrun-orion-2:gemma4-26b-a4b`, `vlmrun-orion-2:cosmos3-nano`) bill at the Fast / Auto token rates (\$0.30 input / \$2.50 output per 1M). Hosted frontier pins such as `vlmrun-orion-2:kimi-2.6`, `vlmrun-orion-2:gpt-5.5`, `vlmrun-orion-2:opus-4.8`, `vlmrun-orion-2:muse-spark-1.1`, `vlmrun-orion-2:grok-4.5`, and `vlmrun-orion-2:gemini-flash-3.6` bill at their model-specific input/output rates in the [Orion 2 model token rates](#orion-2-model-token-rates) section. Fixed-price tools still use the Fast / Auto / Pro tool rates based on the resolved quality tier.
It's the sum of model token costs (orchestration + token-billed tools) and fixed-price tool charges. The response includes `cost_dollars` (effective amount after the service-tier multiplier) and `standard_cost_dollars` (baseline).
Every billable action during a run is recorded as a span: each LLM turn, each sandbox execution, and each tool call. The total is the sum of spans, with the service-tier multiplier applied once at the run level. This gives you full visibility into cost drivers.
Yes. Image I/O, rotation, cropping, video trimming, document navigation, UI detection, and similar utilities are free.
# Rate Limits
Source: https://docs.vlm.run/rate-limits
Rate limits to consider when using the API.
While we are moving quickly to support higher data volumes, we currently
rate limit the API based on the tier you are on. If you are hitting rate
limits, please reach out to us directly at [support](mailto:support@vlm.run?subject=Increase%20rate%20limit).
| Plan | Rate Limit |
| ---------- | -------------- |
| Standard | 10 queries/min |
| Enterprise | Unlimited |
# client.agent
Source: https://docs.vlm.run/sdk-reference/components/agent
Agent Chat Completions
The `client.agent` object allows you to interact with VLM Run's Orion Agents for multi-modal chat completions.
## Chat Completions
Generate responses from the agent using the chat completions:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the client
client = VLMRun(api_key="")
# Basic text completion
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{"role": "user", "content": "What is the capital of France?"}
]
)
print(response.choices[0].message.content)
```
## Image Analysis
Analyze images using the agent:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
# Analyze an image
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image in detail"},
{"type": "image_url", "image_url": {"url": "https://example.com/image.jpg", "detail": "auto"}}
]
}
]
)
print(response.choices[0].message.content)
```
## Video Analysis
Analyze videos using the agent:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
# Analyze a video
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Summarize this video"},
{"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}}
]
}
]
)
print(response.choices[0].message.content)
```
## Structured Outputs
Get structured JSON responses using Pydantic schemas:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define response schema
class ImageCaption(BaseModel):
caption: str = Field(..., description="Detailed caption of the image")
tags: list[str] = Field(..., description="Tags describing the image")
client = VLMRun(api_key="")
# Get structured response
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Generate a caption and tags for this image"},
{"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
]
}
],
response_format={"type": "json_schema", "schema": ImageCaption.model_json_schema()}
)
# Validate and parse the response
result = ImageCaption.model_validate_json(response.choices[0].message.content)
print(result)
```
## Document Analysis
Analyze documents and PDFs:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
# Analyze a PDF document
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract key information from this document"},
{"type": "file_url", "file_url": {"url": "https://example.com/document.pdf"}}
]
}
]
)
print(response.choices[0].message.content)
```
## SDK Reference
### `client.agent.completions.create()`
Create a chat completion with the agent.
**Parameters:**
| Parameter | Type | Description |
| ----------------- | ------------ | --------------------------------------------- |
| `model` | `str` | Model to use (e.g., `vlmrun-orion-1:auto`) |
| `messages` | `list[dict]` | List of messages in the conversation |
| `response_format` | `dict` | Optional JSON schema for structured output |
| `stream` | `bool` | Enable streaming responses (default: `False`) |
**Returns:** `ChatCompletionResponse`
### Message Content Types
| Type | Description |
| ----------- | ------------------------------ |
| `text` | Plain text content |
| `image_url` | Image URL for image analysis |
| `video_url` | Video URL for video analysis |
| `file_url` | File URL for document analysis |
# Client Reference
Source: https://docs.vlm.run/sdk-reference/components/client
Detailed guide to the VLM Run Python SDK client
# Client Reference
This guide provides detailed examples for using the VLM Run client, organized from basic to advanced usage.
## Basic Client Setup
### Initialization
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Using environment variable (recommended)
client = VLMRun()
# Or with explicit API key
client = VLMRun(api_key="your-api-key")
# With custom configuration
client = VLMRun(
api_key="your-api-key",
base_url="https://custom-endpoint.com/v1",
timeout=60.0,
max_retries=3
)
```
For production use, set your API key in the `VLMRUN_API_KEY` environment variable rather than hardcoding it.
### Configuration Options
| Parameter | Type | Default | Description |
| ------------- | ------- | -------------------------- | ----------------------------------------------------- |
| `api_key` | `str` | `None` | Your API key (falls back to `VLMRUN_API_KEY` env var) |
| `base_url` | `str` | `"https://api.vlm.run/v1"` | API endpoint URL |
| `timeout` | `float` | `120.0` | Request timeout in seconds |
| `max_retries` | `int` | `5` | Maximum retry attempts for failed requests |
## Media Processing
### Image Processing
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process an image file
from PIL import Image
image = Image.open("invoice.jpg")
response = client.image.generate(
images=[image],
domain="document.invoice"
)
# Process from URL
response = client.image.generate(
urls=["https://example.com/invoice.jpg"],
domain="document.invoice"
)
```
### Document Processing
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process a document file
response = client.document.generate(
file="document.pdf",
domain="document.receipt"
)
# Process from URL
response = client.document.generate(
url="https://example.com/document.pdf",
domain="document.receipt"
)
```
### Audio Processing
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process an audio file
response = client.audio.generate(
file="recording.mp3",
domain="audio.transcription"
)
# Process from URL
response = client.audio.generate(
url="https://example.com/recording.mp3",
domain="audio.transcription"
)
```
### Video Processing
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process a video file
response = client.video.generate(
file="clip.mp4",
domain="video.transcription"
)
```
## Working with Predictions
### Retrieving Predictions
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get a specific prediction by ID
prediction = client.predictions.get("pred_abc123")
# List recent predictions (paginated)
predictions = client.predictions.list(limit=10)
for pred in predictions:
print(f"ID: {pred.id}, Status: {pred.status}")
```
### Waiting for Completion
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Initial prediction request
response = client.document.generate(
file="large-document.pdf",
domain="document.receipt"
)
# Wait for completion if still processing
if response.status != "completed":
response = client.predictions.wait(
response.id,
timeout=300, # 5 minutes max
sleep=2 # Check every 2 seconds
)
# Or with a callback function
def on_complete(prediction):
print(f"Prediction {prediction.id} completed!")
client.predictions.wait(response.id, callback=on_complete)
```
### Working with Response Data
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Check prediction status
if prediction.status == "completed":
# Access structured data
result = prediction.response
# Access metadata
print(f"Created: {prediction.created_at}")
print(f"Completed: {prediction.completed_at}")
# Access usage information
print(f"Elements processed: {prediction.usage.elements_processed}")
print(f"Credits used: {prediction.usage.credits_used}")
elif prediction.status == "failed":
print("Prediction failed")
else:
print(f"Status: {prediction.status}")
```
## File Management
### Uploading Files
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload from a file path
file = client.files.upload("document.pdf")
# Upload with purpose tag
file = client.files.upload(
file="image.jpg",
purpose="document-processing"
)
# Upload from an open file handle
with open("document.pdf", "rb") as f:
file = client.files.upload(file=f, filename="document.pdf")
```
### Managing Files
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List files (paginated)
files = client.files.list(limit=20)
for file in files:
print(f"ID: {file.id}, Name: {file.filename}")
# Get file details
file = client.files.get("file_abc123")
print(f"ID: {file.id}")
print(f"Size: {file.bytes} bytes")
# Delete a file
client.files.delete("file_abc123")
```
## Domain and Schema Management
### Working with Domains
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List available domains
domains = client.hub.list_domains()
for domain in domains:
print(f"Domain: {domain.domain}")
# Filter by type
doc_domains = [d for d in domains if d.name.startswith("document.")]
```
### Working with Schemas
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get schema information for a domain
schema = client.hub.get_schema("document.invoice")
# Get version information
print(f"Schema version: {schema.schema_version}")
# Get JSON schema definition
json_schema = schema.model_json_schema()
# Get Pydantic model for a domain
InvoiceModel = client.hub.get_pydantic_model("document.invoice")
# Create a model instance
invoice = InvoiceModel(
invoice_number="INV-123",
total_amount=1250.00
)
```
## Custom Schemas
### Defining Custom Schemas
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pydantic import BaseModel, Field
# Define a custom schema with Pydantic
class ProductSchema(BaseModel):
name: str = Field(..., description="Product name")
price: float = Field(..., description="Product price")
sku: str = Field(..., description="Stock keeping unit")
in_stock: bool = Field(..., description="Whether item is in stock")
# Use in generation request
response = client.image.generate(
images=[product_image],
domain="image.product",
config={"json_schema": ProductSchema.model_json_schema()}
)
```
### Using Auto-casting
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# With autocast=True, the response is converted to a Pydantic model
response = client.document.generate(
file="invoice.pdf",
domain="document.invoice",
autocast=True
)
# Now response.response is a domain-specific model
invoice = response.response
total_with_tax = invoice.total_amount * 1.1 # Type-safe operation
```
## Common Workflows
### Upload and Process
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# 1. Upload file
file_response = client.files.upload("invoice.jpg")
# 2. Process the file
prediction = client.image.generate(
urls=[file_response.url],
domain="document.invoice"
)
# 3. Wait if needed
if prediction.status != "completed":
prediction = client.predictions.wait(prediction.id)
# 4. Process results
invoice_data = prediction.response
print(f"Invoice number: {invoice_data.invoice_number}")
```
### Batch Processing
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import glob
from pathlib import Path
# 1. Get all files to process
files = glob.glob("./invoices/*.pdf")
results = []
# 2. Process each file
for file_path in files:
response = client.document.generate(
file=file_path,
domain="document.invoice",
batch=True # Enable batch processing
)
results.append(response.id)
# 3. Wait for all results
completed_results = []
for prediction_id in results:
result = client.predictions.wait(prediction_id)
completed_results.append(result)
```
## Error Handling
### Handling Common Errors
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client.exceptions import (
APIError,
AuthenticationError,
RateLimitError,
ServerError,
ValidationError
)
client = VLMRun()
try:
response = client.image.generate(
images=[image],
domain="document.invoice"
)
except AuthenticationError:
print("Authentication failed. Check your API key.")
except RateLimitError as e:
print(f"Rate limit exceeded. Try again in {e.retry_after} seconds.")
except ServerError:
print("Server error. Please try again later.")
except InvalidRequestError as e:
print(f"Invalid request: {e.message}")
except ApiError as e:
print(f"API error ({e.status_code}): {e.message}")
```
### Implementing Retries
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import time
from vlmrun.client.exceptions import RateLimitError, ServerError
def process_with_retry(image_path, max_retries=5):
for attempt in range(max_retries):
try:
return client.image.generate(
images=[image_path],
domain="document.invoice"
)
except (RateLimitError, ServerError) as e:
if attempt == max_retries - 1:
raise
# Exponential backoff
wait_time = 2 ** attempt
print(f"Retrying in {wait_time}s (attempt {attempt+1}/{max_retries})")
time.sleep(wait_time)
```
## Advanced Features
### Custom Timeouts
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Global timeout setting
client = VLMRun(timeout=180.0) # 3 minutes default
# Component-specific timeouts
client.document._requestor._timeout = 300.0 # 5 minutes for documents
client.video._requestor._timeout = 600.0 # 10 minutes for videos
```
### Request Metadata
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client.types import RequestMetadata
# Add metadata to track requests
response = client.image.generate(
images=[image],
domain="document.invoice",
metadata=RequestMetadata(
user_id="user123",
session_id="session456",
tags=["invoice", "processing"]
)
)
```
## Best Practices
### Client Lifecycle
Create a single client instance and reuse it across your application:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# At application startup
client = VLMRun()
# Reuse throughout your application
def process_image(image_path):
return client.image.generate(images=[image_path], domain="document.invoice")
def process_document(document_path):
return client.document.generate(file=document_path, domain="document.invoice")
```
### API Key Management
Keep your API key secure:
1. Use environment variables instead of hardcoding
2. Use a secrets manager for production applications
3. Regularly rotate API keys for security
### Resource Cleanup
Clean up resources when they're no longer needed:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload temporary file
file_response = client.files.upload("temp.jpg")
# Use the file
prediction = client.image.generate(urls=[file_response.url])
# Delete when done
client.files.delete(file_response.id)
```
### Performance Optimization
For high-volume applications:
1. Reuse the client instance
2. Use batch processing for multiple files
3. Implement exponential backoff for retries
4. Use async processing for non-blocking operations
# client.files
Source: https://docs.vlm.run/sdk-reference/components/files
Manage files with the VLM Run Python SDK
# Files API
The `client.files` lets you upload, retrieve, and manage files used by the VLM Run platform. Files are essential for predictions, fine-tuning, and dataset creation.
## Quick Examples
### Upload a File
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload a file
file = client.files.upload("invoice.jpg")
print(f"File ID: {file.id}, URL: {file.url}")
```
### Retrieve a File
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get file details
file = client.files.get("file_abc123")
print(f"Name: {file.filename}, Size: {file.bytes} bytes")
```
### List Files
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List your files
files = client.files.list(limit=10)
for file in files:
print(f"{file.filename} ({file.id})")
```
### Delete a File
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Delete a file
client.files.delete("file_abc123")
```
## File Lifecycle
Files in VLM Run follow a simple lifecycle:
1. **Upload** - Send files to the platform
2. **Process** - Use files for predictions or other operations
3. **Manage** - List, retrieve, or delete files as needed
## Uploading Files
### Basic Upload
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload a file with default settings
file = client.files.upload("document.pdf")
```
### With Purpose
Files can be categorized by purpose, which affects how they can be used:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload for fine-tuning
file = client.files.upload(
file="training_data.json",
purpose="fine-tune"
)
```
### From File Object
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload from an open file handle
with open("document.pdf", "rb") as f:
file = client.files.upload(file=f, filename="document.pdf")
```
### Available Purposes
| Purpose | Description | Common File Types |
| ------------------- | -------------------------------- | ------------------------- |
| `fine-tune` | For fine-tuning models | Training data, JSON files |
| `assistants` | General usage (default) | Images, PDFs, text files |
| `assistants_output` | Output from assistants | Generated content |
| `batch` | Input files for batch processing | Large collections |
| `batch_output` | Output from batch processing | Results and reports |
| `vision` | For vision-based models | Images, screenshots |
| `datasets` | For dataset creation | Labeled data collections |
## Retrieving Files
### Get File Details
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get metadata for a specific file
file = client.files.get("file_abc123")
# Access file properties
print(f"Name: {file.filename}")
print(f"Size: {file.bytes} bytes")
print(f"Purpose: {file.purpose}")
print(f"Created: {file.created_at}")
```
### List Files
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List all files (paginated)
files = client.files.list(limit=20)
# Filter by purpose
fine_tune_files = client.files.list(
purpose="fine-tune",
limit=10
)
# Pagination
next_page = client.files.list(skip=20, limit=20)
```
### Delete Files
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Delete a file by ID
client.files.delete("file_abc123")
```
## Common Patterns
### Upload and Process
The most common pattern is uploading a file and using it immediately:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# 1. Upload the file
file = client.files.upload("invoice.jpg")
# 2. Process with image API
prediction = client.image.generate(
file=file.id,
domain="document.invoice"
)
# 3. Work with the results
if prediction.status == "completed":
print(f"Invoice #: {prediction.response.invoice_number}")
print(f"Amount: ${prediction.response.total_amount}")
```
### Batch Processing Multiple Files
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload multiple files
import glob
# Get all PDFs in a directory
pdf_files = glob.glob("./invoices/*.pdf")
results = []
# Process each file
for path in pdf_files:
# 1. Upload file
file = client.files.upload(path, purpose="batch")
# 2. Process document
prediction = client.document.generate(
file=file.id,
domain="document.invoice",
batch=True # Process asynchronously
)
results.append((file.id, prediction.id))
# Later: check results
for file_id, prediction_id in results:
prediction = client.predictions.get(prediction_id)
if prediction.status == "completed":
print(f"File {file_id}: ${prediction.response.total_amount}")
```
### Temporary File Management
Clean up files after use:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload a temporary file
temp_file = client.files.upload("temp_image.jpg")
try:
# Use the file
result = client.image.generate(
urls=[temp_file.url],
domain="image.classification"
)
# Process the result
print(f"Classification: {result.response}")
finally:
# Clean up when done
client.files.delete(temp_file.id)
```
## Optimization Features
### File Caching
VLM Run automatically detects duplicate files using content hashing:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload same file twice
file1 = client.files.upload("report.pdf")
file2 = client.files.upload("report.pdf")
# Both operations return the same file ID
print(f"Same file ID: {file1.id == file2.id}") # True
```
### Pre-upload Caching Check
Check if a file exists before uploading:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Check if file exists in cache
cached_file = client.files.get_cached_file("large-dataset.zip")
if cached_file:
file_id = cached_file.id
print(f"Using existing file: {file_id}")
else:
file = client.files.upload("large-dataset.zip")
file_id = file.id
print(f"Uploaded new file: {file_id}")
```
## Response Structure
The `FileResponse` object has the following structure:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
class FileResponse(BaseModel):
id: Optional[str] # Unique file identifier
filename: Optional[str] # Original filename
bytes: int # File size in bytes
purpose: Literal[ # File purpose/category
"fine-tune",
"assistants",
"assistants_output",
"batch",
"batch_output",
"vision",
"datasets",
]
created_at: datetime # Creation timestamp
object: str = "file" # Object type
```
Example usage:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
file = client.files.get("file_abc123")
print(f"ID: {file.id}")
print(f"Filename: {file.filename}")
print(f"Size: {file.bytes} bytes")
print(f"Purpose: {file.purpose}")
print(f"Created: {file.created_at}")
print(f"Object type: {file.object}") # Always "file"
```
## Error Handling
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client.exceptions import ApiError, NotFoundError
try:
file = client.files.upload("document.pdf")
except FileNotFoundError:
print("Local file not found")
except ApiError as e:
if e.status_code == 413:
print("File too large")
elif e.status_code == 415:
print("Unsupported file type")
else:
print(f"API error: {e.message}")
```
## Best Practices
### Use Descriptive Filenames
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# ✅ Good: Descriptive name
file = client.files.upload("invoice-march2023-acme-corp.jpg")
# ❌ Bad: Generic name
file = client.files.upload("file.jpg")
```
### Check Cache for Efficiency
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Efficiently upload multiple files
for path in files_to_upload:
cached = client.files.get_cached_file(path)
if cached:
file_id = cached.id # Use existing file
else:
file = client.files.upload(path)
file_id = file.id
```
### Clean Up Unused Files
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List files older than 30 days
import datetime
from datetime import timedelta
cutoff_date = datetime.datetime.now() - timedelta(days=30)
old_files = []
for file in client.files.list():
if file.created_at < cutoff_date and file.purpose == "assistants":
old_files.append(file.id)
# Delete old files
for file_id in old_files:
client.files.delete(file_id)
print(f"Deleted old file: {file_id}")
```
### Set Appropriate Timeouts for Large Files
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Extend timeout for large files
large_file = client.files.upload(
file="large-video.mp4",
timeout=600 # 10 minutes
)
```
# client.gateway
Source: https://docs.vlm.run/sdk-reference/components/gateway
OpenAI-compatible OCR / VLM model gateway
The `client.gateway` object provides access to the [VLM Run Gateway](/gateway/introduction),
an OpenAI-compatible surface for third-party OCR and vision-language models
(e.g. `zai-org/glm-ocr`, `paddleocr/pp-ocrv6`, `qwen/qwen3.5-0.8b`). It
authenticates with the same `VLMRUN_API_KEY` used everywhere else in the SDK.
Under the hood the gateway points the OpenAI SDK at
`https://gateway.vlm.run/v1/openai`, so `client.gateway.completions`,
`client.gateway.embeddings`, and `client.gateway.transcriptions` are the
standard OpenAI resource objects. You get the familiar chat-completions,
embeddings, and audio-transcription interfaces without configuring a base URL
yourself.
The gateway is a raw passthrough to the underlying models. Unlike
[`client.agent`](/sdk-reference/components/agent) (which calls the Orion agent),
most gateway models, especially OCR models, do **not** accept text-only
input, so you generally need at least one `image_url`/`document_url` content part.
## Installation
The gateway uses the OpenAI SDK, available via the `openai` extra:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
pip install "vlmrun[openai]"
```
## Chat Completions
Run an OCR / VLM model over a document or image. Pass PDFs as `document_url`
content parts and images as `image_url`; gateway-specific fields (such as
`method` and `document_dpi`) travel in `extra_body`.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.gateway.completions.create(
model="paddleocr/pp-ocrv6",
messages=[
{
"role": "user",
"content": [
{
"type": "document_url",
"document_url": {
"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
},
}
],
}
],
extra_body={"method": "ocr", "document_dpi": 96},
)
print(response.choices[0].message.content)
```
### Visual Q\&A
Models that accept text input (e.g. `qwen/qwen3.5-0.8b`) can take a `text`
content part alongside an image:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
response = client.gateway.completions.create(
model="qwen/qwen3.5-0.8b",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What is happening in this image?"},
{"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
],
}
],
)
print(response.choices[0].message.content)
```
### Streaming
The gateway streams document (PDF) requests one SSE chunk per page:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
stream = client.gateway.completions.create(
model="zai-org/glm-ocr",
messages=[
{
"role": "user",
"content": [
{"type": "document_url", "document_url": {"url": "https://example.com/doc.pdf"}},
],
}
],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
```
An async client is available via `client.gateway.async_completions`.
## Embeddings
Embed text, images, or video with a gateway embedding model
(e.g. `qwen/qwen3-vl-embedding-2b`).
Multimodal input nests content parts one level deeper than plain text: `input`
is a list whose items are either a string or a *list* of content parts.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
response = client.gateway.embeddings.create(
model="qwen/qwen3-vl-embedding-2b",
input=[[{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}]],
)
print(len(response.data[0].embedding))
```
## Transcriptions
Transcribe audio (or a video's audio track) with a gateway transcription model
(e.g. `nvidia/parakeet-tdt-0.6b-v3`):
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
with open("clip.mp3", "rb") as fh:
response = client.gateway.transcriptions.create(
model="nvidia/parakeet-tdt-0.6b-v3",
file=fh,
)
print(response.text)
```
## Listing Models
List the models available on the gateway. Each model carries extra metadata
(input/output pricing, modality support, supported methods) beyond the standard
OpenAI fields, preserved on the object's `model_extra`.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
for model in client.gateway.models():
print(model.id)
```
## Health Check
Check whether the gateway is reachable and authenticated:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
if client.gateway.health():
print("Gateway is healthy")
```
## Configuration
By default the gateway targets `https://gateway.vlm.run/v1`. Override it with
the `VLMRUN_GATEWAY_URL` environment variable:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
export VLMRUN_GATEWAY_URL="https://gateway.vlm.run/v1"
```
## SDK Reference
| Attribute / Method | Description |
| ---------------------------------- | ----------------------------------------------------------- |
| `client.gateway.completions` | OpenAI-compatible chat completions (synchronous) |
| `client.gateway.async_completions` | OpenAI-compatible chat completions (asynchronous) |
| `client.gateway.embeddings` | OpenAI-compatible embeddings interface |
| `client.gateway.transcriptions` | OpenAI-compatible audio transcriptions interface |
| `client.gateway.models()` | List models available on the gateway |
| `client.gateway.health()` | Return `True` if the gateway is reachable and authenticated |
| `client.gateway.base_url` | Gateway base URL (without trailing slash) |
See the [Gateway documentation](/gateway/introduction) for the full model
catalog, [methods](/gateway/methods), and
[multimodal input](/gateway/multimodal-inputs) reference, and the
[`vlmrun gw` CLI](/cli/gateway) for the terminal equivalent.
# client.hub
Source: https://docs.vlm.run/sdk-reference/components/hub
Hub API Reference
The `client.hub` object provides access to domains and schemas for structured data extraction.
## List Domains
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.client.types import HubDomainInfo, HubSchemaResponse
client = VLMRun()
# List all available domains
domains: List[HubDomainInfo] = client.hub.list_domains()
# Print domain information
for domain in domains:
print(f"Domain: {domain.domain}")
```
## Get Schema
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get schema for a specific domain
schema: HubSchemaResponse = client.hub.get_schema("document.invoice")
print(f"version={schema.version}, hash={schema.hash}, json_schema={schema.json_schema}")
```
# client.models
Source: https://docs.vlm.run/sdk-reference/components/models
Models API Reference
The `client.models` object provides access to available VLM Run models and their capabilities.
## List Models
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.client.types import ModelInfo
from typing import List
client = VLMRun()
# List all available models
models: List[ModelInfo] = client.models.list()
# Print model information
for model in models:
print(f"Model: {model.model}")
print(f"Domain: {model.domain}")
```
# SDK Overview
Source: https://docs.vlm.run/sdk-reference/components/overview
Core concepts and components of the VLM Run Python SDK
# SDK Overview
The VLM Run SDK enables you to extract structured data from unstructured content using VLMs. Whether you're processing invoices, analyzing images, transcribing audio, or extracting insights from video, the SDK provides a unified interface to transform raw media into actionable business data.
## Core Concepts
### Domains & Schemas
In VLM Run, **domains** represent different types of content analysis:
* `document.invoice` - Extract data from invoices
* `image.caption` - Extract caption from the image
* `audio.transcription` - Transcribe spoken content
* `video.dashcam-analytics` - Analyze dashcam footage
Each domain has an associated **schema** that defines the structured output format.
### Content Processing Flow
The typical flow for processing content follows these steps:
1. **Prepare content** - File, URL, or in-memory data
2. **Choose domain** - Select appropriate domain for your task
3. **Generate prediction** - Process the content
4. **Handle results** - Work with the structured response
## SDK Structure
The SDK is organized around a central `VLMRun` client that gives you access to all functionality:
```
VLMRun Client
│
├── Content APIs
│ ├── client.image # Image processing
│ ├── client.document # Document processing
│ ├── client.audio # Audio processing
│ └── client.video # Video processing
│
├── Resource APIs
│ ├── client.files # File management
│ ├── client.hub # Domain & schema access
│ ├── client.models # Model information
│ └── client.gateway # OpenAI-compatible OCR / VLM gateway
│
└── Utility APIs
├── client.predictions # Prediction management
└── client.fine_tuning # Model customization
```
## Working with Media Types
Each media type has its own specialized client with consistent patterns.
### Images
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process an image from a file
response = client.image.generate(
images=[image],
domain="document.invoice"
)
# From a URL
response = client.image.generate(
urls=["https://example.com/invoice.jpg"],
domain="document.invoice"
)
```
### Documents
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process a document
response = client.document.generate(
file="document.pdf",
domain="document.invoice"
)
```
### Audio
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process audio
response = client.audio.generate(
file="recording.mp3",
domain="audio.transcription"
)
```
### Video
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process video
response = client.video.generate(
file="clip.mp4",
domain="video.dashcam-analytics",
batch=True
)
```
## Working with Predictions
All content processing methods return a `PredictionResponse` with a consistent structure:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Key fields of a prediction response
prediction = client.image.generate(...)
prediction.id # Unique identifier
prediction.status # Processing status
prediction.created_at # Creation timestamp
prediction.response # Structured results (when complete)
prediction.usage # Resource usage information
```
### Prediction Statuses
A prediction will have one of these statuses:
* `enqueued` - Waiting to be processed
* `pending` - Ready to start processing
* `running` - Currently being processed
* `completed` - Processing finished successfully
* `failed` - Processing encountered an error
### Handling Async Processing
For content that takes time to process, you can wait for completion:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Start processing
prediction = client.document.generate(
file="large-document.pdf",
domain="document.invoice"
)
# Wait for processing to complete
if prediction.status != "completed":
prediction = client.predictions.wait(prediction.id)
# Now work with the results
result = prediction.response
```
## Using Schemas
Schemas define the structure of prediction responses, providing type-safe access to extracted data.
### Working with Standard Schemas
Every domain has a predefined schema:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get structured data from a domain schema
response = client.image.generate(
images=[image],
domain="document.invoice"
)
# Access fields in the response
invoice_data = response.response
print(f"Invoice #: {invoice_data.invoice_number}")
print(f"Amount: ${invoice_data.total_amount}")
```
### Using Custom Schemas
You can define your own schema for custom extraction:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pydantic import BaseModel, Field
# Define a custom schema
class ProductInfo(BaseModel):
name: str = Field(..., description="Product name")
price: float = Field(..., description="Product price")
category: str = Field(..., description="Product category")
# Use the custom schema
response = client.image.generate(
images=[product_image],
domain="image.product",
config={"json_schema": ProductInfo.model_json_schema()}
)
```
## Key Resources
### Files
Manage files for processing:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload a file
file = client.files.upload("document.pdf")
# Use the file in a prediction
prediction = client.document.generate(
file=file.id,
domain="document.invoice"
)
```
### Hub
Access domains and schemas:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List available domains
domains = client.hub.list_domains()
# Get details about a domain
schema = client.hub.get_schema("document.invoice")
```
### Models
Get information about available models:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List available models
models = client.models.list()
```
## Common Patterns
### Process & Extract
The most common pattern is processing content and extracting structured data:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process and extract in one step
response = client.image.generate(
images=[image],
domain="document.invoice",
autocast=True # Get a type-safe model
)
# Work with the structured data
invoice = response.response
total_with_tax = invoice.total_amount * 1.1
```
### Upload & Process
Another common pattern is uploading files first, then processing them:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# 1. Upload file
file = client.files.upload("invoice.jpg")
# 2. Process the file
prediction = client.image.generate(
file=file.id,
domain="document.invoice"
)
# 3. Get the results
if prediction.status == "completed":
invoice_data = prediction.response
```
### Batch Processing
For processing multiple files:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process multiple files
results = []
for file_path in file_paths:
response = client.document.generate(
file=file_path,
domain="document.invoice",
batch=True # Process asynchronously
)
results.append(response.id)
# Wait for all results
completed = [client.predictions.wait(id) for id in results]
```
## Next Steps
Now that you understand the core concepts, you can:
* Explore the [Client Reference](/sdk-reference/components/client) for detailed API documentation
* Try the specialized APIs for [Image](/sdk-reference/predictions/image), [Document](/sdk-reference/predictions/document), [Audio](/sdk-reference/predictions/audio), or [Video](/sdk-reference/predictions/video)
* Learn about the [CLI](/sdk-reference/cli) for command-line usage
# client.predictions
Source: https://docs.vlm.run/sdk-reference/components/predictions
Manage predictions with the VLM Run Python SDK
# Predictions API
The `client.predictions` component provides methods to retrieve, list, and manage predictions across all content types. This is the central hub for tracking the status of all processing jobs in the platform.
## Quick Examples
### Get a Prediction
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Retrieve a specific prediction by ID
prediction = client.predictions.get("pred_abc123")
print(f"Status: {prediction.status}")
```
### List Predictions
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List recent predictions
predictions = client.predictions.list(limit=10)
for pred in predictions:
print(f"ID: {pred.id}, Status: {pred.status}")
```
### Wait for Completion
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Wait for a prediction to complete
completed = client.predictions.wait(
"pred_abc123",
timeout=60, # Maximum wait time in seconds
sleep=1 # Check interval in seconds
)
print(f"Completed at: {completed.completed_at}")
```
## Core Operations
### Retrieving Predictions
Get details about a specific prediction:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get prediction by ID
prediction = client.predictions.get("pred_abc123")
# Access prediction properties
print(f"ID: {prediction.id}")
print(f"Status: {prediction.status}")
print(f"Created: {prediction.created_at}")
print(f"Type: {prediction.type}")
# If completed, access the structured response
if prediction.status == "completed" and prediction.response:
print(f"Result: {prediction.response}")
```
### Listing Predictions
List predictions you've created (with pagination):
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Basic listing with default pagination
predictions = client.predictions.list()
# Custom pagination
predictions = client.predictions.list(
skip=0, # Skip this many items
limit=10 # Return at most this many items
)
# Process the list
for prediction in predictions:
print(f"ID: {prediction.id}, Status: {prediction.status}")
```
### Waiting for Completion
Wait for long-running predictions to complete:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Wait with default settings (60 seconds timeout, 1 second checks)
completed = client.predictions.wait("pred_abc123")
# Wait with custom timeout and polling interval
completed = client.predictions.wait(
"pred_abc123",
timeout=300, # Maximum wait time (5 minutes)
sleep=2 # Check every 2 seconds
)
# Check results after waiting
if completed.status == "completed":
print(f"Success! Result: {completed.response}")
else:
print(f"Failed or timed out: {completed.status}")
```
The `wait()` method will raise a `TimeoutError` if the prediction doesn't complete within the specified timeout.
## Prediction Statuses
Predictions can have the following statuses:
| Status | Description |
| ----------- | ------------------------------------------ |
| `enqueued` | The prediction is waiting to be processed |
| `pending` | The prediction is preparing to start |
| `running` | The prediction is actively being processed |
| `completed` | The prediction has completed successfully |
| `failed` | The prediction encountered an error |
| `paused` | The prediction has been paused |
## Media-Specific APIs
The base `Predictions` class is extended by specialized prediction classes for different media types:
### Image Predictions
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Generate prediction from image files
prediction = client.image.generate(
images=[Path("image.jpg")], # List of Path objects or PIL Images
domain="document.invoice"
)
# Generate prediction from image URLs
prediction = client.image.generate(
urls=["https://example.com/image.jpg"],
domain="document.invoice"
)
# Generate schema from image
schema = client.image.schema(
images=[Path("image.jpg")]
)
```
### Document, Audio, and Video Predictions
These specialized APIs follow a consistent pattern:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Process a document file
prediction = client.document.generate(
file="document.pdf",
domain="document.invoice"
)
# Process from a URL
prediction = client.audio.generate(
url="https://example.com/audio.mp3",
domain="audio.transcription"
)
```
## Response Structure
The `PredictionResponse` object includes these key fields:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
class PredictionResponse(BaseModel):
id: str # Unique prediction identifier
status: Literal[ # Current job status
"enqueued",
"pending",
"running",
"completed",
"failed",
"paused"
]
type: str # Prediction type (e.g., "image", "document")
created_at: datetime # When the prediction was created
completed_at: Optional[datetime] # When the prediction was completed (if done)
response: Optional[Any] # Structured result data
usage: CreditUsage # Usage and billing information
```
The `usage` field contains a `CreditUsage` object:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
class CreditUsage(BaseModel):
elements_processed: Optional[int] # Number of elements processed
element_type: Optional[str] # Type of element processed
credits_used: Optional[int] # Credits consumed by the operation
```
## Auto-casting Responses
All specialized prediction classes support auto-casting responses to their appropriate schema types:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Enable auto-casting with the autocast parameter
prediction = client.document.generate(
file="invoice.pdf",
domain="document.invoice",
autocast=True # Convert response to appropriate Pydantic model
)
# Now the response is a typed Pydantic model
invoice = prediction.response
print(f"Invoice number: {invoice.invoice_number}")
print(f"Total: {invoice.total_amount}")
```
## Common Patterns
### Process and Wait
A common pattern is to start a prediction and wait for it to complete:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# 1. Start the prediction
prediction = client.document.generate(
file="large-document.pdf",
domain="document.invoice"
)
# 2. Wait for completion if needed
if prediction.status != "completed":
try:
prediction = client.predictions.wait(
prediction.id,
timeout=120 # Wait up to 2 minutes
)
except TimeoutError:
print("Processing is taking longer than expected")
# Handle timeout case
# 3. Process the results
if prediction.status == "completed":
form_data = prediction.response
print(f"Form data: {form_data}")
```
### Batch Processing
For batch operations, use the batch parameter and track multiple predictions:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Start multiple predictions in batch mode
prediction_ids = []
for file_path in document_files:
prediction = client.document.generate(
file=file_path,
domain="document.invoice",
batch=True # Process asynchronously
)
prediction_ids.append(prediction.id)
# Track completion status
completed = 0
total = len(prediction_ids)
print(f"Started {total} predictions")
# Check status periodically
while completed < total:
completed = 0
for pred_id in prediction_ids:
prediction = client.predictions.get(pred_id)
if prediction.status in ["completed", "failed"]:
completed += 1
print(f"Progress: {completed}/{total} complete")
time.sleep(5) # Check every 5 seconds
print("All predictions complete!")
```
### Error Handling
Implement robust error handling:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
try:
# Start prediction
prediction = client.document.generate(
file="document.pdf",
domain="document.invoice"
)
# Wait for completion
prediction = client.predictions.wait(prediction.id)
# Check for success
if prediction.status == "completed":
print("Processing successful!")
result = prediction.response
else:
print(f"Processing failed: {prediction.status}")
except TimeoutError:
print("Prediction timed out")
except ValueError as e:
print(f"Invalid parameters: {e}")
except Exception as e:
print(f"Unexpected error: {e}")
```
## Best Practices
### Efficient Polling
Use appropriate intervals when waiting for predictions:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# ✅ Good: Use longer intervals for long-running jobs
def wait_with_backoff(prediction_id):
"""Wait with increasing backoff intervals."""
start_time = time.time()
wait_time = 1 # Start with 1 second
while time.time() - start_time < 300: # 5 minute timeout
prediction = client.predictions.get(prediction_id)
if prediction.status in ["completed", "failed"]:
return prediction
# Increase wait time with each check
wait_time = min(wait_time * 1.5, 30) # Cap at 30 seconds
print(f"Waiting {wait_time:.1f} seconds...")
time.sleep(wait_time)
raise TimeoutError("Prediction timed out")
```
### Using Appropriate Timeouts
Set timeouts based on the expected processing time:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# For quick predictions (e.g., simple image classification)
prediction = client.predictions.wait(
prediction_id,
timeout=30, # 30 seconds
sleep=1
)
# For complex processing (e.g., large documents, long videos)
prediction = client.predictions.wait(
prediction_id,
timeout=600, # 10 minutes
sleep=5 # Check less frequently
)
```
### Progress Reporting with tqdm
For better user experience, use the `tqdm` library for progress reporting:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from tqdm import tqdm
import time
# Generate a prediction
prediction = client.document.generate(file="large_file.pdf", domain="document.invoice")
# Wait with progress bar
timeout = 120 # 2 minutes
for _ in tqdm(range(timeout), desc="Processing document"):
prediction = client.predictions.get(prediction.id)
if prediction.status in ["completed", "failed"]:
break
time.sleep(1)
```
# Getting Started
Source: https://docs.vlm.run/sdk-reference/getting-started
How to get started with the VLM Run Python SDK
You can use the [VLM Run Python SDK](https://pypi.org/project/vlmrun/) to interact with the [VLM Run](https://vlm.run) API.
## Installation
### Basic Installation
You can install the basic Python SDK using pip:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
pip install vlmrun --upgrade
```
Need specific features? Choose an optional dependency pack:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# For video processing
pip install "vlmrun[video]"
# For document processing
pip install "vlmrun[doc]"
# For all features
pip install "vlmrun[all]"
```
## Set Up Authentication
Grab your API key from the [VLM Run dashboard](https://app.vlm.run) and set it as an environment variable:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# On Linux/macOS
export VLMRUN_API_KEY="your-api-key"
# On Windows
set VLMRUN_API_KEY=your-api-key
```
## Your First API Call
Let's process an image to extract structured data:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the client
client = VLMRun()
# Process an image from a URL
response = client.image.generate(
urls=["https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/invoice_1.jpg"],
domain="document.invoice"
)
# Check if processing completed
if response.status == "completed":
# `response.response` is returned as a dictionary by default,
# so access fields with `.get()` or `[ ]` indexing.
invoice = response.response
print(f"Invoice #: {invoice.get('invoice_number')}")
print(f"Total: ${invoice.get('total_amount')}")
```
## What's Next?
With the client initialized, you can now:
* Process other media types (documents, audio, video)
* Use different domains for specialized extraction
* Upload and manage files
* Create custom extraction schemas
Check out the [SDK Overview](/sdk-reference/components/overview) for key concepts or jump into the [Client Reference](/sdk-reference/components/client) for detailed examples.
## Quick Examples
### Process a Document
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Extract data from a PDF
response = client.document.generate(
url="https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/invoice_1.jpg",
domain="document.invoice"
)
```
### Transcribe Audio
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Transcribe an audio file
response = client.audio.generate(
url="https://storage.googleapis.com/vlm-data-public-prod/examples/audio/sample.mp3",
domain="audio.transcription"
)
```
### Process a Local Image
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Using a local file
from PIL import Image
image = Image.open("invoice.jpg")
response = client.image.generate(
images=[image],
domain="document.invoice"
)
```
# client.agent
Source: https://docs.vlm.run/sdk-reference/node/components/agent
Learn how to use Agent Chat Completions with the VLM Run Node.js SDK
The `agent` component provides methods for interacting with VLM Run's Orion Agents for multi-modal chat completions.
## Chat Completions
Generate responses from the agent using the chat completions:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
// Initialize the client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Basic text completion
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{ role: "user", content: "What is the capital of France?" }
]
});
console.log(response.choices[0].message.content);
```
## Image Analysis
Analyze images using the agent:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Analyze an image
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Describe this image in detail" },
{ type: "image_url", image_url: { url: "https://example.com/image.jpg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
## Video Analysis
Analyze videos using the agent:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Analyze a video
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Summarize this video" },
{ type: "video_url", video_url: { url: "https://example.com/video.mp4" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
## Structured Outputs
Get structured JSON responses using TypeScript interfaces:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
// Define response schema
interface ImageCaption {
caption: string;
tags: string[];
}
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Get structured response
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Generate a caption and tags for this image" },
{ type: "image_url", image_url: { url: "https://example.com/image.jpg" } }
]
}
],
response_format: {
type: "json_schema",
schema: {
type: "object",
properties: {
caption: { type: "string", description: "Detailed caption of the image" },
tags: { type: "array", items: { type: "string" }, description: "Tags describing the image" }
},
required: ["caption", "tags"]
}
}
});
// Parse the response
const result: ImageCaption = JSON.parse(response.choices[0].message.content);
console.log(result);
```
## Document Analysis
Analyze documents and PDFs:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Analyze a PDF document
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Extract key information from this document" },
{ type: "file_url", file_url: { url: "https://example.com/document.pdf" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
## SDK Reference
### `client.agent.completions.create()`
Create a chat completion with the agent.
**Parameters:**
| Parameter | Type | Description |
| ----------------- | ---------------- | --------------------------------------------- |
| `model` | `string` | Model to use (e.g., `vlmrun-orion-1:auto`) |
| `messages` | `Message[]` | List of messages in the conversation |
| `response_format` | `ResponseFormat` | Optional JSON schema for structured output |
| `stream` | `boolean` | Enable streaming responses (default: `false`) |
**Returns:** `Promise`
### Message Content Types
| Type | Description |
| ----------- | ------------------------------ |
| `text` | Plain text content |
| `image_url` | Image URL for image analysis |
| `video_url` | Video URL for video analysis |
| `file_url` | File URL for document analysis |
## Best Practices
1. **Structured Outputs**
* Define clear JSON schemas for predictable responses
* Use TypeScript interfaces for type safety
2. **Multi-Modal Inputs**
* Use appropriate content types (`image_url`, `video_url`, `file_url`)
* Set `detail` level for images based on analysis needs
3. **Error Handling**
* Always wrap API calls in try-catch blocks
* Handle rate limits and timeouts appropriately
# client
Source: https://docs.vlm.run/sdk-reference/node/components/client
VLM Run Node.js SDK Client Configuration and Usage
## Client Configuration
The `VlmRun` client is the main entry point for interacting with the VLM Run API. It provides access to all SDK functionality including file operations, model operations, and predictions.
### Initialization
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
// Optional configuration options
baseUrl?: string;
});
```
### Configuration Options
| Option | Type | Description | Default |
| --------- | -------- | -------------------- | --------------------- |
| `apiKey` | `string` | Your VLM Run API key | Required |
| `baseUrl` | `string` | Custom API base URL | `https://api.vlm.run` |
### Client Components
The client provides access to different components for specific operations:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// File operations
client.files.upload({
/* ... */
});
client.files.get();
client.files.list();
// Model operations
client.models.list();
// Image predictions
client.image.generate({
/* ... */
});
// Document predictions
client.document.generate({
/* ... */
});
```
### Error Handling
The client throws typed errors for different scenarios:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun, ApiError } from "vlmrun";
try {
const client = new VlmRun({
apiKey: "invalid-key",
});
await client.models.list();
} catch (error) {
if (error instanceof ApiError) {
console.error("API Error:", error.message);
console.error("Status:", error.http_status);
} else if (error instanceof VlmRunError) {
console.error("SDK Error:", error.message);
} else {
console.error("Unknown Error:", error);
}
}
```
### TypeScript Interfaces
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
interface ApiError extends Error {
message: string;
http_status: number;
headers?: Record;
}
interface VlmRunError extends Error {
message: string;
code?: string;
cause?: Error;
}
```
# client.files
Source: https://docs.vlm.run/sdk-reference/node/components/files
Learn how to upload and manage files with the VLM Run Node.js SDK
## File Operations
The `files` component of the VLM Run client provides methods for uploading and managing files.
### Upload a File
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Upload a file using local file path
const file = await client.files.upload({
filePath: "path/to/document.pdf",
});
// The response includes the file ID and metadata
console.log(file.id); // "file_abc123"
console.log(file.filename); // "document.pdf"
console.log(file.bytes); // 1234567
console.log(file.purpose); // "document"
console.log(file.created_at); // "2024-01-01T00:00:00Z"
```
### Get a File
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Get a file by ID
const file = await client.files.get("file_abc123");
```
### File Types
The SDK supports various file types including:
* Images (jpg, jpeg, png)
* Documents (pdf)
* Audio files (mp3, wav)
* Video files (mp4)
### TypeScript Interfaces
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
type FilePurpose = string;
interface FileResponse {
id: string;
filename: string;
bytes: number;
purpose: FilePurpose;
created_at: string;
object: "file";
}
```
### Error Handling
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
try {
const file = await client.files.upload({
filePath: "nonexistent.pdf",
});
} catch (error) {
if (error instanceof ApiError) {
console.error("API Error:", error.message);
// Handle API-specific errors (rate limits, permissions, etc.)
} else {
console.error("File system error:", error);
// Handle local file system errors
}
}
```
### Best Practices
1. **File Size Limits**
* Check file size before uploading
* Handle large files appropriately
2. **File Types**
* Verify file types before upload
* Use appropriate MIME types
3. **Error Handling**
* Implement proper error handling
* Handle both API and file system errors
# client.hub
Source: https://docs.vlm.run/sdk-reference/node/components/hub
Hub API Reference for the VLM Run Node.js SDK
The `client.hub` object provides access to domains and schemas for structured data extraction.
## Get Hub Info
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Get hub version information
const info = await client.hub.info();
console.log(`Hub version: ${info.version}`);
```
## List Domains
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import type { DomainInfo } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// List all available domains
const domains: DomainInfo[] = await client.hub.listDomains();
// Print domain information
for (const domain of domains) {
console.log(`Domain: ${domain.domain}`);
console.log(`Name: ${domain.name}`);
console.log(`Description: ${domain.description}`);
}
```
## Get Schema
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import type { HubSchemaResponse } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Get schema for a specific domain
const schema: HubSchemaResponse = await client.hub.getSchema({
domain: "document.invoice",
});
console.log(`Schema version: ${schema.schema_version}`);
console.log(`Schema hash: ${schema.schema_hash}`);
console.log(`JSON Schema: ${JSON.stringify(schema.json_schema, null, 2)}`);
// Get schema with GraphQL statement
const schemaWithGql: HubSchemaResponse = await client.hub.getSchema({
domain: "document.invoice",
gql_stmt: "{ invoice_number total_amount }",
});
```
## TypeScript Interfaces
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
interface HubInfoResponse {
version: string;
}
interface DomainInfo {
domain: string;
name: string;
description: string;
}
interface HubSchemaResponse {
json_schema: Record;
schema_version: string;
schema_hash: string;
domain: string;
gql_stmt: string;
description: string;
}
interface HubSchemaParams {
domain: string;
gql_stmt?: string;
}
```
## Error Handling
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun, ApiError } from "vlmrun";
try {
const schema = await client.hub.getSchema({
domain: "invalid.domain",
});
} catch (error) {
if (error instanceof ApiError) {
console.error("API Error:", error.message);
console.error("Status:", error.http_status);
} else {
console.error("Unknown Error:", error);
}
}
```
# client.models
Source: https://docs.vlm.run/sdk-reference/node/components/models
Learn how to work with models in the VLM Run Node.js SDK
## Model Operations
The `models` component provides methods for listing and getting information about available models.
### List Available Models
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// List all available models
const models = await client.models.list();
console.log(models);
```
# client.predictions
Source: https://docs.vlm.run/sdk-reference/node/components/predictions
Manage predictions with the VLM Run Node.js SDK
# Predictions API
The `client.predictions` component provides methods to retrieve, list, and manage predictions across all content types. This is the central hub for tracking the status of all processing jobs in the platform.
## Quick Examples
### Get a Prediction
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Retrieve a specific prediction by ID
const prediction = await client.predictions.get("pred_abc123");
console.log(`Status: ${prediction.status}`);
```
### List Predictions
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// List recent predictions
const predictions = await client.predictions.list({ limit: 10 });
for (const pred of predictions) {
console.log(`${pred.id}: ${pred.status}`);
}
```
### Wait for Completion
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Wait for a prediction to complete
const completed = await client.predictions.wait(
"pred_abc123",
60, // Maximum wait time in seconds
1 // Check interval in seconds
);
console.log(`Completed at: ${completed.completed_at}`);
```
## Core Operations
### Retrieving Predictions
Get details about a specific prediction:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import type { PredictionResponse } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Get prediction by ID
const prediction: PredictionResponse = await client.predictions.get("pred_abc123");
// Access prediction properties
console.log(`ID: ${prediction.id}`);
console.log(`Status: ${prediction.status}`);
console.log(`Created: ${prediction.created_at}`);
// If completed, access the structured response
if (prediction.status === "completed" && prediction.response) {
console.log(`Result: ${JSON.stringify(prediction.response)}`);
}
```
### Listing Predictions
List predictions you've created (with pagination):
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Basic listing with default pagination
const predictions = await client.predictions.list();
// Custom pagination
const paginatedPredictions = await client.predictions.list({
skip: 0, // Skip this many items
limit: 10 // Return at most this many items
});
// Process the list
for (const prediction of paginatedPredictions) {
console.log(`ID: ${prediction.id}, Status: ${prediction.status}`);
}
```
### Waiting for Completion
Wait for long-running predictions to complete:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Wait with default settings (60 seconds timeout, 1 second checks)
const completed = await client.predictions.wait("pred_abc123");
// Wait with custom timeout and polling interval
const completedCustom = await client.predictions.wait(
"pred_abc123",
300, // Maximum wait time (5 minutes)
2 // Check every 2 seconds
);
// Check results after waiting
if (completedCustom.status === "completed") {
console.log(`Success! Result: ${JSON.stringify(completedCustom.response)}`);
} else {
console.log(`Failed or timed out: ${completedCustom.status}`);
}
```
The `wait()` method will throw a `TimeoutError` if the prediction doesn't complete within the specified timeout.
## Prediction Statuses
Predictions can have the following statuses:
| Status | Description |
| ----------- | ------------------------------------------ |
| `enqueued` | The prediction is waiting to be processed |
| `pending` | The prediction is preparing to start |
| `running` | The prediction is actively being processed |
| `completed` | The prediction has completed successfully |
| `failed` | The prediction encountered an error |
| `paused` | The prediction has been paused |
## Feedback Operations
### Get Feedbacks for a Prediction
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Get all feedbacks for a prediction
const feedbacks = await client.predictions.getFeedbacks("pred_abc123");
for (const feedback of feedbacks) {
console.log(`Feedback ID: ${feedback.id}`);
console.log(`Notes: ${feedback.notes}`);
}
```
### Create Feedback
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Create feedback for a prediction
const feedback = await client.predictions.createFeedback({
request_id: "pred_abc123",
response: { corrected_field: "new_value" },
notes: "Corrected the invoice number",
});
console.log(`Feedback created: ${feedback.id}`);
```
## Media-Specific APIs
The base `Predictions` class is extended by specialized prediction classes for different media types:
### Image Predictions
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Generate prediction from image files
const prediction = await client.image.generate({
images: ["path/to/image.jpg"], // Local file paths or base64 strings
domain: "document.invoice"
});
// Generate prediction from image URLs
const predictionFromUrl = await client.image.generate({
urls: ["https://example.com/image.jpg"],
domain: "document.invoice"
});
// Generate schema from image
const schema = await client.image.schema({
images: ["path/to/image.jpg"]
});
```
### Document, Audio, and Video Predictions
These specialized APIs follow a consistent pattern:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Process a document file
const docPrediction = await client.document.generate({
fileId: "file_abc123",
domain: "document.invoice"
});
// Process from a URL
const audioPrediction = await client.audio.generate({
url: "https://example.com/audio.mp3",
domain: "audio.transcription"
});
// Process video
const videoPrediction = await client.video.generate({
fileId: "file_xyz789",
domain: "video.transcription"
});
```
## Response Structure
The `PredictionResponse` object includes these key fields:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
interface PredictionResponse {
id: string; // Unique prediction identifier
status: JobStatus; // Current job status
created_at: string; // When the prediction was created
completed_at?: string; // When the prediction was completed (if done)
response?: any; // Structured result data
message?: string; // Status message or error details
usage?: CreditUsage; // Usage and billing information
}
```
The `usage` field contains a `CreditUsage` object:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
interface CreditUsage {
elements_processed?: number; // Number of elements processed
element_type?: "image" | "page" | "video" | "audio"; // Type of element
credits_used?: number; // Credits consumed by the operation
}
```
## Common Patterns
### Process and Wait
A common pattern is to start a prediction and wait for it to complete:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// 1. Start the prediction
const prediction = await client.document.generate({
fileId: "file_abc123",
domain: "document.invoice"
});
// 2. Wait for completion if needed
let result = prediction;
if (prediction.status !== "completed") {
try {
result = await client.predictions.wait(
prediction.id,
120 // Wait up to 2 minutes
);
} catch (error) {
console.log("Processing is taking longer than expected");
// Handle timeout case
}
}
// 3. Process the results
if (result.status === "completed") {
const formData = result.response;
console.log(`Form data: ${JSON.stringify(formData)}`);
}
```
### Batch Processing
For batch operations, use the batch parameter and track multiple predictions:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Start multiple predictions in batch mode
const predictionIds: string[] = [];
const fileIds = ["file_1", "file_2", "file_3"];
for (const fileId of fileIds) {
const prediction = await client.document.generate({
fileId,
domain: "document.invoice",
batch: true // Process asynchronously
});
predictionIds.push(prediction.id);
}
console.log(`Started ${predictionIds.length} predictions`);
// Wait for all to complete
const results = await Promise.all(
predictionIds.map(id => client.predictions.wait(id, 300))
);
console.log("All predictions complete!");
```
### Error Handling
Implement robust error handling:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun, ApiError } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
try {
// Start prediction
const prediction = await client.document.generate({
fileId: "file_abc123",
domain: "document.invoice"
});
// Wait for completion
const result = await client.predictions.wait(prediction.id);
// Check for success
if (result.status === "completed") {
console.log("Processing successful!");
console.log(result.response);
} else {
console.log(`Processing failed: ${result.status}`);
console.log(`Message: ${result.message}`);
}
} catch (error) {
if (error instanceof ApiError) {
console.error(`API Error: ${error.message}`);
console.error(`Status: ${error.http_status}`);
} else if (error instanceof Error && error.name === "TimeoutError") {
console.error("Prediction timed out");
} else {
console.error(`Unexpected error: ${error}`);
}
}
```
## TypeScript Interfaces
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
interface ListParams {
skip?: number;
limit?: number;
}
interface FeedbackParams {
request_id: string;
response?: Record;
notes?: string;
}
interface FeedbackResponse {
id: string;
created_at: string;
request_id: string;
response: any;
notes?: string;
}
```
## Best Practices
### Efficient Polling
Use appropriate intervals when waiting for predictions:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// For quick predictions (e.g., simple image classification)
const quickResult = await client.predictions.wait(
predictionId,
30, // 30 seconds
1 // Check every second
);
// For complex processing (e.g., large documents, long videos)
const complexResult = await client.predictions.wait(
predictionId,
600, // 10 minutes
5 // Check every 5 seconds
);
```
### Using Callbacks for Long-Running Jobs
For very long-running predictions, consider using callback URLs:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const prediction = await client.video.generate({
fileId: "file_abc123",
domain: "video.transcription",
batch: true,
callbackUrl: "https://your-server.com/webhook/predictions"
});
// Your webhook will receive the completed prediction
console.log(`Prediction started: ${prediction.id}`);
```
# Getting Started
Source: https://docs.vlm.run/sdk-reference/node/getting-started
Learn how to install and use the VLM Run Node.js SDK
## Installation
You can install the VLM Run Node.js SDK using npm, yarn, or pnpm:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Using npm
npm install vlmrun
# Using yarn
yarn add vlmrun
# Using pnpm
pnpm add vlmrun
```
## Authentication
To use the VLM Run API, you'll need an API key. You can obtain one by:
1. Create an account at [VLM Run](https://app.vlm.run)
2. Navigate to dashboard Settings -> API Keys
Then use it to initialize the client:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
```
## Basic Usage
### Image Predictions
Here's a simple example of using the SDK to process an image:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
// Initialize the client
const client = new VlmRun({
apiKey: "your-api-key",
});
// Process an image using URL
const imageUrl =
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/invoice_1.jpg";
const response = await client.image.generate({
images: [imageUrl],
domain: "document.invoice",
config: {
jsonSchema: {
type: "object",
properties: {
invoice_number: { type: "string" },
total_amount: { type: "number" },
},
},
},
});
```
#### Process an image passing [zod](https://zod.dev/) schema
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { z } from "zod";
const imageUrl =
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/invoice_1.jpg";
const schema = z.object({
invoice_number: z.string(),
total_amount: z.number(),
});
const apiResponse = await client.image.generate({
images: [imageUrl],
domain: "document.invoice",
config: {
responseModel: schema,
},
});
const response = apiResponse.response as z.infer;
console.log(response);
```
#### Process an image using local file
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Process an image using local file
const localResponse = await client.image.generate({
images: ["path/to/local/image.jpg"],
model: "vlm-1",
domain: "document.invoice",
});
```
### Document Predictions
Here's how to process documents:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Upload a document
const file = await client.files.upload({
filePath: "path/to/invoice.pdf",
});
// Process a document using file ID
const response = await client.document.generate({
fileId: file.id,
model: "vlm-1",
domain: "document.invoice",
});
console.log(response);
// Process a document using URL
const documentUrl =
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/google_invoice.pdf";
const urlResponse = await client.document.generate({
url: documentUrl,
model: "vlm-1",
domain: "document.invoice",
});
console.log(urlResponse);
```
#### Process a document passing [zod](https://zod.dev/) schema
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { z } from "zod";
const schema = z.object({
invoice_id: z.string(),
total: z.number(),
sub_total: z.number(),
tax: z.number(),
items: z.array(
z.object({
name: z.string(),
quantity: z.number(),
price: z.number(),
total: z.number(),
})
),
});
const apiResponse = await client.document.generate({
url: documentUrl,
domain: "document.invoice",
config: { responseModel: schema },
});
const response = apiResponse.response as z.infer;
console.log(response);
```
## TypeScript Support
The VLM Run Node.js SDK is written in TypeScript and provides full type definitions out of the box. When using TypeScript, you'll get:
* Full IntelliSense support
* Type checking for all API calls
* Type definitions for request and response objects
* Autocomplete for configuration options
To get the best development experience, make sure your `tsconfig.json` includes:
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"compilerOptions": {
"esModuleInterop": true,
"strict": true
}
}
```
# client.audio
Source: https://docs.vlm.run/sdk-reference/node/predictions/audio
Learn how to process audio files with the VLM Run Node.js SDK
## Audio Predictions
The `audio` component provides methods for processing and analyzing audio files using VLM Run's models.
### Process Audio
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Upload and process an audio file
const file = await client.files.upload({
filePath: "path/to/audio.mp3",
});
// Process audio using file ID
const response = await client.audio.generate({
fileId: file.id,
domain: "audio.transcription",
});
```
### TypeScript Interfaces
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
interface FilePredictionParams extends PredictionGenerateParams {
batch?: boolean;
fileId?: string;
url?: string;
}
```
### Error Handling
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
try {
const response = await client.audio.generate({
url: "invalid-url",
model: "vlm-1",
});
} catch (error) {
if (error instanceof ApiError) {
console.error("API Error:", error.message);
// Handle API-specific errors
} else {
console.error("File system error:", error);
// Handle local file system errors
}
}
```
### Best Practices
1. **Audio Formats**
* Supported formats: MP3, WAV
* Ensure proper audio quality
* Consider file size limits
2. **Performance**
* Use URLs for remote audio files when possible
* Handle timeouts appropriately
* Consider audio length and complexity
3. **Error Handling**
* Validate audio files before processing
* Handle both API and file system errors
* Implement proper error recovery
# client.document
Source: https://docs.vlm.run/sdk-reference/node/predictions/document
Learn how to process documents with the VLM Run Node.js SDK
## Document Predictions
The `document` component provides methods for processing and analyzing documents using VLM Run's models.
### Process a Document
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Upload a document
const file = await client.files.upload({
filePath: "path/to/invoice.pdf",
});
// Process a document using file ID
const response = await client.document.generate({
fileId: file.id,
model: "vlm-1",
domain: "document.invoice",
});
console.log(response);
// Process a document using URL
const documentUrl =
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/google_invoice.pdf";
const urlResponse = await client.document.generate({
url: documentUrl,
model: "vlm-1",
domain: "document.invoice",
});
console.log(urlResponse);
```
#### Process a document passing [zod](https://zod.dev/) schema
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { z } from "zod";
const schema = z.object({
invoice_id: z.string(),
total: z.number(),
sub_total: z.number(),
tax: z.number(),
items: z.array(
z.object({
name: z.string(),
quantity: z.number(),
price: z.number(),
total: z.number(),
})
),
});
const apiResponse = await client.document.generate({
url: documentUrl,
domain: "document.invoice",
config: { responseModel: schema },
});
const response = apiResponse.response as z.infer;
console.log(response);
```
#### Get usage
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const usage = response.usage;
console.log(usage);
```
### TypeScript Interfaces
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
interface FilePredictionParams extends PredictionGenerateParams {
batch?: boolean;
fileId?: string;
url?: string;
}
interface CreditUsage {
elements_processed?: number;
element_type?: "image" | "page" | "video" | "audio";
credits_used?: number;
}
```
### Error Handling
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
try {
const response = await client.document.generate({
url: "invalid-url",
model: "vlm-1",
});
} catch (error) {
if (error instanceof ApiError) {
console.error("API Error:", error.message);
// Handle API-specific errors
} else {
console.error("File system error:", error);
// Handle local file system errors
}
}
```
### Best Practices
1. **Document Formats**
* Supported formats: PDF
* Ensure proper document quality
* Consider file size limits
2. **Performance**
* Use URLs for remote documents when possible
* Handle timeouts appropriately
* Consider document size and complexity
3. **Error Handling**
* Validate documents before processing
* Handle both API and file system errors
* Implement proper error recovery
# client.image
Source: https://docs.vlm.run/sdk-reference/node/predictions/image
Learn how to process images with the VLM Run Node.js SDK
## Image Predictions
The `image` component provides methods for processing and analyzing images using VLM Run's models.
#### Process an Image
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
// Initialize the client
const client = new VlmRun({
apiKey: "your-api-key",
});
// Process an image using URL
const imageUrl =
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/invoice_1.jpg";
const response = await client.image.generate({
images: [imageUrl],
domain: "document.invoice",
config: {
jsonSchema: {
type: "object",
properties: {
invoice_number: { type: "string" },
total_amount: { type: "number" },
},
},
},
});
```
#### Process an image passing [zod](https://zod.dev/) schema
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { z } from "zod";
const imageUrl =
"https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/invoice_1.jpg";
const schema = z.object({
invoice_number: z.string(),
total_amount: z.number(),
});
const apiResponse = await client.image.generate({
images: [imageUrl],
domain: "document.invoice",
config: {
responseModel: schema,
},
});
const response = apiResponse.response as z.infer;
console.log(response);
```
#### Process an image using local file
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Process an image using local file
const localResponse = await client.image.generate({
images: ["path/to/local/image.jpg"],
model: "vlm-1",
domain: "document.invoice",
});
```
### TypeScript Interfaces
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
interface ImagePredictionParams extends PredictionGenerateParams {
batch?: boolean;
images: string[];
}
interface PredictionGenerateParams {
model?: string;
domain: string;
config?: GenerationConfigParams;
metadata?: RequestMetadataParams;
callbackUrl?: string;
}
type GenerationConfigParams = {
detail?: "auto" | "hi" | "lo";
responseModel?: ZodType;
jsonSchema?: Record | null;
confidence?: boolean;
grounding?: boolean;
};
type RequestMetadataParams = {
environment?: "dev" | "staging" | "prod";
sessionId?: string | null;
allowTraining?: boolean;
};
interface PredictionResponse {
id: string;
created_at: string;
completed_at?: string;
response?: any;
status: JobStatus;
message?: string;
usage?: CreditUsage;
}
interface CreditUsage {
elements_processed?: number;
element_type?: "image" | "page" | "video" | "audio";
credits_used?: number;
}
type JobStatus = string;
```
### Error Handling
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
try {
const response = await client.image.generate({
images: ["invalid-url"],
model: "vlm-1",
});
} catch (error) {
if (error instanceof ApiError) {
console.error("API Error:", error.message);
// Handle API-specific errors
} else {
console.error("File system error:", error);
// Handle local file system errors
}
}
```
### Best Practices
1. **Image Formats**
* Supported formats: JPG, JPEG, PNG
* Ensure proper image quality
* Consider image size limits
2. **Performance**
* Use URLs for remote images
* Process multiple images in one request when possible
* Handle timeouts appropriately
3. **Error Handling**
* Validate image files before processing
* Handle both API and file system errors
* Implement proper error recovery
# client.video
Source: https://docs.vlm.run/sdk-reference/node/predictions/video
Video Processing API for the VLM Run Node.js SDK
The `client.video` object allows you to process video files and extract structured data.
## Generate Predictions
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import type { PredictionResponse } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Process a video file with a predefined schema using file ID
const response: PredictionResponse = await client.video.generate({
fileId: "file_abc123",
domain: "video.transcription",
});
// Process a video from URL
const responseFromUrl: PredictionResponse = await client.video.generate({
url: "https://example.com/video.mp4",
domain: "video.transcription",
});
console.log(`Prediction ID: ${response.id}`);
console.log(`Status: ${response.status}`);
```
## Upload and Process Video
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// First upload the video file
const file = await client.files.upload({
filePath: "path/to/video.mp4",
});
// Then process the uploaded video
const response = await client.video.generate({
fileId: file.id,
domain: "video.transcription",
});
// Wait for completion if needed
if (response.status !== "completed") {
const completed = await client.predictions.wait(response.id, 300);
console.log(`Result: ${JSON.stringify(completed.response)}`);
}
```
## Batch Processing
For longer videos or when you want asynchronous processing:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Process video in batch mode
const response = await client.video.generate({
fileId: "file_abc123",
domain: "video.transcription",
batch: true,
});
// Poll for completion
const completed = await client.predictions.wait(response.id, 600);
console.log(`Transcription: ${JSON.stringify(completed.response)}`);
```
## Custom Configuration
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun, GenerationConfig } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Process with custom configuration
const response = await client.video.generate({
fileId: "file_abc123",
domain: "video.dashcam-analytics",
config: {
detail: "hi",
grounding: true,
},
metadata: {
environment: "prod",
sessionId: "session_123",
},
});
```
## Auto-Generate Schema
Generate a schema automatically from a video file:
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
// Auto-generate schema from video
const schemaResponse = await client.video.schema({
fileId: "file_abc123",
});
console.log(`Generated schema: ${JSON.stringify(schemaResponse.response)}`);
```
## Get Usage
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import type { CreditUsage } from "vlmrun";
const client = new VlmRun({
apiKey: "your-api-key",
});
const response = await client.video.generate({
fileId: "file_abc123",
domain: "video.transcription",
});
// Access usage information
const usage: CreditUsage | undefined = response.usage;
if (usage) {
console.log(`Elements processed: ${usage.elements_processed}`);
console.log(`Element type: ${usage.element_type}`);
console.log(`Credits used: ${usage.credits_used}`);
}
```
## TypeScript Interfaces
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
interface FilePredictionParams {
fileId?: string;
url?: string;
model?: string;
domain: string;
batch?: boolean;
config?: GenerationConfigParams;
metadata?: RequestMetadataParams;
callbackUrl?: string;
}
interface GenerationConfigParams {
detail?: "auto" | "hi" | "lo";
responseModel?: ZodType;
jsonSchema?: Record | null;
confidence?: boolean;
grounding?: boolean;
gqlStmt?: string | null;
}
interface RequestMetadataParams {
environment?: "dev" | "staging" | "prod";
sessionId?: string | null;
allowTraining?: boolean;
}
interface PredictionResponse {
id: string;
created_at: string;
completed_at?: string;
response?: any;
status: JobStatus;
message?: string;
usage?: CreditUsage;
}
interface CreditUsage {
elements_processed?: number;
element_type?: "image" | "page" | "video" | "audio";
credits_used?: number;
}
```
## Error Handling
```typescript theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun, ApiError } from "vlmrun";
try {
const response = await client.video.generate({
fileId: "invalid_file_id",
domain: "video.transcription",
});
} catch (error) {
if (error instanceof ApiError) {
console.error("API Error:", error.message);
console.error("Status:", error.http_status);
} else {
console.error("Unknown Error:", error);
}
}
```
# client.audio
Source: https://docs.vlm.run/sdk-reference/predictions/audio
Audio Processing API
The `client.audio` object allows you to process audio files and extract structured data.
This feature is currently only available for our enterprise-tier customers. If you are interested in using this feature, please [contact us](mailto:support@vlm.run).
## Generate Predictions
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse
# Initialize the client
client = VLMRun()
# Process an audio file with a predefined schema
response: PredictionResponse = client.audio.generate(
file=Path("path/to/audio.mp3"),
domain="audio.transcription",
)
```
## Get Usage
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client.types import CreditUsage
usage: CreditUsage = response.usage
print(usage)
```
# client.document
Source: https://docs.vlm.run/sdk-reference/predictions/document
Document Processing API
The `client.document` object allows you to process documents and extract structured data.
## Generate Predictions
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse
# Initialize the client
client = VLMRun()
# Process a PDF document with a predefined schema
# Note: Since the file is passed as a file path, it will be uploaded to the VLM Run server.
response: PredictionResponse = client.document.generate(
file=Path("path/to/document.pdf"),
domain="document.markdown",
)
```
## Get Usage
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client.types import CreditUsage
usage: CreditUsage = response.usage
print(usage)
```
## Document Utilities
The VLM Run SDK provides several document-processing utilities for encoding and downloading documents.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.common.pdf import pdf_images
# Read a PDF file and return an iterator of images
images: Iterator[Image.Image] = pdf_images(Path("path/to/document.pdf"))
for image in images:
print(image)
```
# client.image
Source: https://docs.vlm.run/sdk-reference/predictions/image
Image Processing API
The `client.image` object allows you to process images and extract structured data.
## Generate Predictions
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from PIL import Image
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse, GenerationConfig
# Initialize the client
client = VLMRun()
# Process an image with a predefined schema
image: Image.Image = Image.open("path/to/image.jpg")
response = client.image.generate(
images=[image],
domain="document.invoice",
)
# Process with custom schema
image: Image.Image = Image.open("path/to/image.jpg")
response: PredictionResponse = client.image.generate(
images=[image],
domain="document.invoice",
config=GenerationConfig(
json_schema={...}
)
)
print(response)
```
## Generate Predictions with a custom schema
Let's say we want to classify images into one of three categories: `tv`, `document`, or `other`. You can define a custom schema as follows, and pass it to the `json_schema` parameter:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from typing import Literal
from pydantic import BaseModel, Field
from vlmrun.client.types import GenerationConfig
class ImagePrediction(BaseModel):
label: Literal["tv", "document", "other"] = Field(..., title="Class label for the image.")
caption: str = Field(..., title="Caption for the image.")
# Initialize the client
client = VLMRun()
# Load the image, and process it with the custom schema
image: Image.Image = Image.open("path/to/image.jpg")
response: PredictionResponse = client.image.generate(
images=[image],
domain="image.classification",
config=GenerationConfig(
json_schema=ImagePrediction.model_json_schema()
)
)
```
## Get Usage
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client.types import CreditUsage
usage: CreditUsage = response.usage
print(usage)
```
## Image Utilities
The VLM Run SDK provides several image-processing utilities for encoding and downloading images.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.common.image import encode_image
from vlmrun.common.utils import download_image
from PIL import Image
# Convert image to base64
image = Image.open("image.jpg")
base64_str = encode_image(image, format="PNG")
# Download image from URL
image: Image.Image = download_image("https://example.com/image.jpg")
```
# client.video
Source: https://docs.vlm.run/sdk-reference/predictions/video
Video Processing API
The `client.video` object allows you to process video files and extract structured data.
This feature is currently only available for our enterprise-tier customers. If you are interested in using this feature, please [contact us](mailto:support@vlm.run).
## Generate Predictions
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import PredictionResponse
# Initialize the client
client = VLMRun()
# Process a video file with a predefined schema
response: PredictionResponse = client.video.generate(
file=Path("path/to/video.mp4"),
domain="video.transcription",
)
```
## Get Usage
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client.types import CreditUsage
usage: CreditUsage = response.usage
print(usage)
```
# Orion Skills
Source: https://docs.vlm.run/skills/introduction
Modular, reusable capabilities for visual extraction and agent workflows
Skills are modular, reusable capabilities that provide VLM Run's visual models and agents with procedural knowledge for extraction tasks. Instead of selecting a pre-defined [domain](/hub), you reference a skill by name (and optionally pin a version) and the platform automatically applies the skill's prompt and JSON schema to your request.
## Why use Skills?
Skills let you decouple *what* you want the model to do from *how* it's configured:
* **Reusable**: Create a skill once, reference it from any endpoint (image, document, video, audio, agent)
* **Versionable**: Pin a specific skill version for reproducible results, or use `"latest"` to always get the newest revision
* **Composable**: Pass multiple skills in a single request
* **Auto-generated**: Create skills from a prompt, a chat session, or a pre-built skill zip
* **Flexible**: Use skills as an alternative to domains, or combine them with custom schemas
* **Inline**: Send skill bundles directly in the request as base64-encoded zips — no pre-upload required
## Skill Identifiers
Each skill can be referenced by name and version, or by its unique ID:
| Field | Description | Example |
| --------------- | --------------------------------------- | ---------------------- |
| `skill_id` | Unique identifier (UUID or name string) | `"abc-123-def"` |
| `skill_name` | Human-readable name for lookup | `"invoice-extraction"` |
| `skill_version` | Skill version to use | `"latest"` |
You must provide at least one of `skill_name` or `skill_id`. When using `skill_name`, you can also specify a `skill_version`, otherwise the `latest` version is used.
## Skills vs Domains
Skills are the preferred way to extract structured data from images, documents, videos, and audio.
We will be deprecating domains in the near future. Whenever possible, use skills instead of domains.
When `skills` are provided and `domain` is omitted, the platform creates a dynamic application from the skill's prompt and JSON schema. You can still pass `domain` alongside `skills` if needed.
| | Domains | Skills |
| ------------------ | ---------------------------------------- | --------------------------------------------------------- |
| **Lookup** | Fixed string (e.g. `"document.invoice"`) | Name + version (e.g. `"invoice-extraction"` @ `"latest"`) |
| **Custom prompts** | Via `config.prompt` | Bundled with the skill |
| **JSON schema** | Pre-defined per domain | Bundled with the skill |
| **Versioning** | N/A | Explicit version pinning |
| **Usage** | `domain` parameter | `config.skills` parameter |
## Where Skills Work
Skills work across all of the following VLM Run products:
| Type | Skills Parameter |
| -------------------- | ---------------- |
| **Model Requests** | `config.skills` |
| **Agent Executions** | `config.skills` |
| **Chat Completions** | `skills` |
Get started with skills in under 2 minutes
Learn how skills are structured
# Create Skills
Source: https://docs.vlm.run/skills/manage/create
Create skills from skill folders, prompts, or chat sessions
Skills can be created in three ways:
| Mode | Source | Description |
| ---------------- | ----------------------------------- | --------------------------------------------------------- |
| **Skill Folder** | Local directory or zip | Upload a pre-built skill folder containing SKILL.md |
| **Prompt** | `prompt` (+ optional `json_schema`) | Auto-generate SKILL.md and schema.json from a text prompt |
| **Session** | `session_id` | Auto-generate SKILL.md from a chat session's history |
## From Skill Folder
Upload a local skill folder directly. The folder must contain a `SKILL.md` file — the skill name and description are parsed from its YAML frontmatter automatically.
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Upload a local skill folder directly (zips and creates in one step)
vlmrun skills upload ./my-skill
# Override name/description from SKILL.md frontmatter
vlmrun skills upload ./my-skill --name "invoice-extraction" --description "Extract structured data from invoices"
```
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pathlib import Path
client = VLMRun(api_key="")
# One-step: zip, upload, and create a skill from a local directory
skill = client.skills.create_from_directory(
directory=Path("./my-skill"),
)
print(f"Created skill: {skill.skill_name} (type={skill.type})")
# Override name/description from SKILL.md frontmatter
skill = client.skills.create_from_directory(
directory=Path("./my-skill"),
name="invoice-extraction",
description="Extract structured data from invoices",
)
```
[`create_from_directory`](https://github.com/vlm-run/vlmrun-python-sdk/blob/main/vlmrun/client/skills.py) handles zipping the folder, uploading the archive via the Files API, and creating the skill in one call. It returns an `AgentSkill` with `type="skill_reference"` that you can pass directly to any endpoint that accepts skills.
The folder should follow the [skill directory structure](/skills/spec/overview):
```
my-skill/
├── SKILL.md
├── schema.json
├── vlmrun.yaml
└── resources/ (optional)
```
Use the skill folder method when you need full control over the skill's instructions, schema, and execution configuration. Use the prompt method for quick prototyping.
## From Prompt
Generate a skill automatically from a text description and optional JSON schema:
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# From a text prompt
vlmrun skills create --prompt "Extract invoice_id, date, and total_amount from invoices."
# With a JSON schema file
vlmrun skills create --prompt "Extract invoice data" --schema schema.json
```
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
skill = client.skills.create(
prompt="Extract invoice_id, date, and total_amount from invoices.",
json_schema={
"type": "object",
"properties": {
"invoice_id": {"type": "string"},
"invoice_date": {"type": "string", "format": "date"},
"total_amount": {"type": "number"}
},
"required": ["invoice_id", "invoice_date", "total_amount"]
}
)
print(f"Created skill: {skill.id} ({skill.name})")
```
The platform generates a `SKILL.md` with instructions derived from your prompt and a `schema.json` from the provided JSON schema.
## From Chat Session
Generate a skill from an existing chat session's conversation history:
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun skills create --session-id ""
```
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
skill = client.skills.create(session_id="")
print(f"Created skill: {skill.id} ({skill.name})")
```
The platform analyzes the conversation to extract the task instructions and expected output format.
# List & Lookup
Source: https://docs.vlm.run/skills/manage/list-lookup
List and search for available skills
## List All Skills
Retrieve all available skills:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
skills = client.skills.list()
for skill in skills:
print(f"{skill.name} (v{skill.skill_version})")
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
});
const skills = await client.skills.list();
skills.forEach(skill => console.log(`${skill.name} (v${skill.skill_version})`));
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# List all skills (latest 25)
vlmrun skills list
# List with grouping (latest version per name)
vlmrun skills list --grouped
# List with custom limit and sort
vlmrun skills list --limit 50 --order-by name --asc
```
## Lookup by Name
Find a skill by its human-readable name:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get latest version
skill = client.skills.get(name="invoice-extraction")
# Get a specific version
skill = client.skills.get(name="invoice-extraction", skill_version="20260219-abc123")
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Get latest version
const skill = await client.skills.get({ name: "invoice-extraction" });
// Get a specific version
const skill = await client.skills.get({
name: "invoice-extraction",
skillVersion: "20260219-abc123",
});
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Get latest version by name
vlmrun skills get invoice-extraction
# Get a specific version
vlmrun skills get invoice-extraction --skill-version 20260219-abc123
```
## Lookup by ID
Find a skill by its unique identifier:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
skill = client.skills.get(id="")
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const skill = await client.skills.get({ id: "" });
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
vlmrun skills get
```
## Download a Skill
Get a presigned URL to download the skill package:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
download = client.skills.download(skill_id="")
print(f"Download URL: {download.url}")
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Download and extract to default location (~/.vlmrun/skills/)
vlmrun skills download invoice-extraction
# Download a specific version to a custom directory
vlmrun skills download invoice-extraction --skill-version 20260219-abc123 --output ./skills/
```
View the complete API reference for skill endpoints
# Update Skills
Source: https://docs.vlm.run/skills/manage/update
Create new versions of existing skills
Updating a skill creates a **new version** with a new unique ID but the same name. Previous versions remain accessible via their version string.
## Update from File
Upload a new skill zip to create a new version:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pathlib import Path
client = VLMRun(api_key="")
# Upload new skill zip
file = client.files.upload(file=Path("updated-skill.zip"))
# Update creates a new version
updated = client.skills.update(
skill_id="",
file_id=file.id,
description="Improved invoice extraction with line item support"
)
print(f"New version: {updated.version}")
```
```bash CLI theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Re-upload a local skill folder to create a new version
vlmrun skills upload ./updated-skill --name "invoice-extraction"
```
## Versioning Behavior
| Action | Result |
| --------------------------------- | -------------------------------------- |
| Update a skill | New version created, same `skill_name` |
| Reference with `version="latest"` | Resolves to the newest version |
| Reference with a pinned version | Continues to use the pinned version |
Updates are non-destructive. Existing versions are never modified or deleted, so pinned references continue to work after an update.
See [Version Pinning](/skills/usage/version-pinning) for best practices on managing versions.
# Quickstart
Source: https://docs.vlm.run/skills/quickstart
Use a skill to extract structured data in under 2 minutes
This guide walks you through using a skill to extract structured data from a document in a single API call.
## Prerequisites
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
pip install vlmrun
```
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
npm install vlmrun
```
Set your API key:
```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
export VLMRUN_API_KEY="your-api-key"
```
## Extract Data with a Skill
Pass a skill by name in the `config.skills` parameter to extract structured JSON from a document:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig, AgentSkill
client = VLMRun(api_key="")
response = client.document.generate(
file=Path("invoice.pdf"),
model="vlm-1",
config=GenerationConfig(
skills=[AgentSkill(skill_name="invoice-extraction", version="latest")]
),
)
print(response.response)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
const fileResponse = await client.files.upload({ filePath: "invoice.pdf" });
const response = await client.document.generate({
fileId: fileResponse.id,
model: "vlm-1",
config: {
skills: [{ skillName: "invoice-extraction", version: "latest" }],
},
});
console.log(response.response);
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/document/generate \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{
"model": "vlm-1",
"file_id": "",
"config": {
"skills": [{"skill_name": "invoice-extraction", "version": "latest"}]
}
}'
```
The platform automatically applies the skill's prompt and JSON schema — no need to specify a `domain` or write a custom prompt.
## Next Steps
Use skills across all generation endpoints
Build your own custom skills
Understand how skills are structured
Pin skill versions for reproducibility
# Reference
Source: https://docs.vlm.run/skills/reference
AgentSkill object and skill specification reference
## AgentSkill Object
The `AgentSkill` object supports two modes: **referenced** (server-stored) and **inline** (sent per-request).
**Common fields:**
| Field | Type | Default | Description |
| ------ | -------- | ------------------- | --------------------------------------------------------------------------- |
| `type` | `string` | `"skill_reference"` | `"skill_reference"` for server-stored skills, `"inline"` for inline bundles |
**Referenced skill fields** (when `type = "skill_reference"`):
| Field | Type | Default | Description |
| --------------- | -------- | ---------- | --------------------------------------- |
| `skill_name` | `string` | `null` | Human-readable skill name for lookup |
| `skill_id` | `string` | `null` | Unique identifier (UUID or name string) |
| `skill_version` | `string` | `"latest"` | Skill version to use |
At least one of `skill_name` or `skill_id` must be provided. If both are given, `skill_id` takes precedence for resolution.
### Inline Skill Fields
When `type = "inline"`, the following fields are used instead of `skill_name`/`skill_id`:
| Field | Type | Default | Description |
| ------------- | -------- | ------- | ------------------------------------------------------------------- |
| `name` | `string` | `null` | Human-readable name for the inline skill |
| `description` | `string` | `null` | Short description of what the skill does |
| `source` | `object` | `null` | Source payload containing the base64-encoded zip bundle (see below) |
**InlineSkillSource object:**
| Field | Type | Default | Description |
| ------------ | -------- | ------------------- | ----------------------------------------------------------- |
| `type` | `string` | `"base64"` | Encoding type (currently only `"base64"`) |
| `media_type` | `string` | `"application/zip"` | MIME type of the bundle |
| `data` | `string` | *required* | Base64-encoded zip containing `SKILL.md` and optional files |
For inline skills, `source` (with `data`) is required, and `skill_id`/`skill_name` must not be set.
### Referenced vs Inline
| | Referenced Skills | Inline Skills |
| ------------------- | --------------------------- | --------------------------------- |
| **How to use** | `skill_id` or `skill_name` | `source` with base64 zip |
| **Persistence** | Stored on the server | Sent per-request, not persisted |
| **Version pinning** | Supported (`version` field) | N/A (bundle is the version) |
| **Best for** | Production, shared skills | Prototyping, ephemeral use, CI/CD |
| **`type` field** | `"skill_reference"` | `"inline"` |
## Identifier Resolution
The platform resolves skill references in this order:
1. If `skill_id` is provided, use it directly
2. If `skill_name` is provided, look up by name
3. If `skill_version` is `"latest"` (default), resolve to the most recent revision
4. If `skill_version` is a specific string (e.g., `"20260219-abc123"`), resolve to that exact version
## Available Toolsets
Toolsets define what capabilities are available to the agent when executing a skill:
| Toolset | Description |
| ----------- | -------------------------------------------- |
| `core` | Basic operations (file I/O, text processing) |
| `document` | Document extraction and layout understanding |
| `image` | Image analysis and understanding |
| `image-gen` | Image generation and editing |
| `video` | Video analysis and understanding |
| `viz` | Visualization and annotation |
| `web` | Web search and retrieval |
| `world-gen` | World generation and editing |
## Skill File Spec
| File | Required | Format | Purpose |
| ----------------------------------------- | ----------- | --------------------------- | ------------------------- |
| [`SKILL.md`](/skills/spec/skill-md) | Yes | YAML frontmatter + Markdown | Metadata and instructions |
| [`vlmrun.yaml`](/skills/spec/vlmrun-yaml) | Yes | YAML | Execution configuration |
| [`schema.json`](/skills/spec/schema-json) | Recommended | JSON Schema draft-07 | Output validation |
| `resources/` | No | Any | Supporting files |
## Python SDK Helpers
The Python SDK provides two convenience functions for working with local skill directories:
| Function | Import | Returns | Description |
| ------------------------------------------------------------------------------------------------------------ | -------------------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| [`create_from_directory`](https://github.com/vlm-run/vlmrun-python-sdk/blob/main/vlmrun/client/skills.py) | `client.skills.create_from_directory(...)` | `AgentSkill` (`type="skill_reference"`) | Zips a local directory, uploads via the Files API, and creates a server-side skill in one call |
| [`AgentSkill.from_directory`](https://github.com/vlm-run/vlmrun-python-sdk/blob/main/vlmrun/client/types.py) | `from vlmrun.client.types import AgentSkill` | `AgentSkill` (`type="inline"`) | Classmethod that zips and base64-encodes a local directory into an inline skill bundle (no server upload) |
Both functions require the directory to contain a `SKILL.md` file. Skill name and description are read from the YAML frontmatter automatically.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentSkill
client = VLMRun(api_key="")
# Server-side: upload and create a reusable skill
skill_ref = client.skills.create_from_directory(Path("./my-skill"))
# Inline: build a self-contained skill bundle (no upload)
skill_inline = AgentSkill.from_directory(Path("./my-skill"))
```
## API Endpoints
| Operation | Method | Endpoint |
| ------------------------------------------------------------- | ------ | -------------------------------- |
| [List skills](/api-reference/v1/skills/get-skills-list) | `GET` | `/v1/skills` |
| [Get skill by ID](/api-reference/v1/skills/get-skill-by-id) | `GET` | `/v1/skills/{skill_id}` |
| [Create skill](/api-reference/v1/skills/post-skill-create) | `POST` | `/v1/skills/create` |
| [Update skill](/api-reference/v1/skills/post-skill-update) | `POST` | `/v1/skills/{skill_id}/update` |
| [Lookup skill](/api-reference/v1/skills/post-skill-lookup) | `POST` | `/v1/skills/lookup` |
| [Download skill](/api-reference/v1/skills/get-skill-download) | `GET` | `/v1/skills/{skill_id}/download` |
# Skill Structure
Source: https://docs.vlm.run/skills/spec/overview
How a skill directory is organized
A skill is a self-contained directory that packages instructions, a JSON schema, execution configuration, and optional resources.
## Directory Layout
**[`SKILL.md`](/skills/spec/skill-md)** — Skill metadata (YAML frontmatter) and instructions (Markdown body)
**[`vlmrun.yaml`](/skills/spec/vlmrun-yaml)** — Execution configuration: model, toolsets, and state machine graph
**[`schema.json`](/skills/spec/schema-json)** — JSON Schema that validates the skill's structured output
**`resources/`** — Optional supporting files: code snippets, templates, images, or reference data
When you upload a skill as a zip file via the [Create Skill](/skills/manage/create) endpoint, the platform extracts and stores these files. When a skill is referenced in a request, the platform loads the SKILL.md instructions and schema.json automatically.
## Inline Skill Bundles
Instead of uploading a skill to the server, you can send a skill bundle directly in an API request as a base64-encoded zip. The zip must follow the same directory layout above. The minimum required file is `SKILL.md`:
| File | Required | Description |
| ------------- | -------- | -------------------------------------------------------------------------------------------- |
| `SKILL.md` | Yes | YAML frontmatter (`name`, `description`, etc.) followed by a markdown body with instructions |
| `vlmrun.yaml` | No | Skill configuration (metadata, settings, and dependencies) |
| `schema.json` | No | JSON Schema for structured output |
| `*.py` | No | Python scripts the skill can reference |
Inline bundles are ideal for rapid prototyping and CI/CD pipelines. For production use, consider [uploading the skill](/skills/manage/create) for reuse across requests.
# schema.json
Source: https://docs.vlm.run/skills/spec/schema-json
JSON Schema for validating skill output
`schema.json` defines the expected structure of a skill's output using [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-release-notes). When a skill is executed, the platform validates the agent's response against this schema to ensure structured, consistent results.
## Format
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"title": "invoice-extraction",
"description": "Structured data extracted from an invoice document",
"properties": {
"invoice_id": {
"type": "string",
"description": "Unique invoice identifier"
},
"invoice_date": {
"type": "string",
"format": "date",
"description": "Invoice date in ISO 8601 format"
},
"total_amount": {
"type": "number",
"description": "Total invoice amount"
}
},
"required": ["invoice_id", "invoice_date", "total_amount"],
"additionalProperties": false
}
```
## Key Fields
| Field | Description |
| ---------------------- | ------------------------------------------------------------------- |
| `$schema` | JSON Schema version — use `http://json-schema.org/draft-07/schema#` |
| `type` | Top-level type — typically `"object"` |
| `title` | Human-readable name for the schema |
| `description` | What the schema validates |
| `properties` | Object properties with types and descriptions |
| `required` | Array of mandatory field names |
| `additionalProperties` | Set to `false` to disallow extra fields |
## Property Types
Supported JSON Schema types for properties:
| Type | Example | Use Case |
| --------- | --------------- | ------------------------------ |
| `string` | `"invoice-001"` | Text fields, IDs, descriptions |
| `number` | `42.5` | Amounts, scores, measurements |
| `integer` | `3` | Counts, indices |
| `boolean` | `true` | Flags, binary classifications |
| `array` | `[...]` | Lists of items |
| `object` | `{...}` | Nested structures |
### String Constraints
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"date": {
"type": "string",
"format": "date",
"description": "Date in YYYY-MM-DD format"
},
"category": {
"type": "string",
"enum": ["invoice", "receipt", "purchase_order"],
"description": "Document category"
},
"timestamp": {
"type": "string",
"pattern": "^\\d{2}:\\d{2}$",
"description": "Timestamp in MM:SS format"
}
}
```
### Array Items
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"line_items": {
"type": "array",
"description": "List of invoice line items",
"items": {
"type": "object",
"properties": {
"description": { "type": "string" },
"quantity": { "type": "integer" },
"unit_price": { "type": "number" }
},
"required": ["description", "quantity", "unit_price"]
}
}
}
```
## Relationship to Skill Creation
When [creating a skill from a prompt](/skills/manage/create), you can pass a `json_schema` parameter. The platform uses this to generate the `schema.json` file within the skill package:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
skill = client.agent.skills.create(
prompt="Extract invoice data",
json_schema={
"type": "object",
"properties": {
"invoice_id": {"type": "string"},
"total_amount": {"type": "number"}
},
"required": ["invoice_id", "total_amount"]
}
)
```
When creating a skill from a file (zip upload), include `schema.json` directly in the skill directory.
Include `description` on every property. The model uses these descriptions to understand what data to extract and how to format it.
# SKILL.md
Source: https://docs.vlm.run/skills/spec/skill-md
Skill metadata and instructions format
`SKILL.md` is the primary file in a skill directory. It combines YAML frontmatter for metadata with a Markdown body for instructions that guide the model or agent during execution.
## Frontmatter Fields
The YAML frontmatter defines the skill's metadata:
```yaml theme={"theme":{"light":"github-light","dark":"dark-plus"}}
---
name: invoice-extraction
description: Extract structured data from invoice documents
version: "1.0"
license: MIT
---
```
| Field | Type | Required | Description |
| --------------- | -------- | -------- | --------------------------------------------------- |
| `name` | `string` | Yes | Skill identifier — used for lookup via `skill_name` |
| `description` | `string` | Yes | Concise description of what the skill does |
| `skill_version` | `string` | No | Skill version string |
| `license` | `string` | No | License type (e.g., `MIT`, `Apache-2.0`) |
### Available Toolsets
| Toolset | Description |
| ----------- | -------------------------------------------- |
| `core` | Basic operations (file I/O, text processing) |
| `document` | Document extraction and layout understanding |
| `image` | Image analysis and understanding |
| `image-gen` | Image generation and editing |
| `video` | Video analysis and understanding |
| `viz` | Visualization and annotation |
| `web` | Web search and retrieval |
| `world-gen` | World generation and editing |
## Markdown Body
The body after the frontmatter contains instructions that are injected into the model or agent prompt at execution time. Write clear, specific instructions for the extraction or analysis task.
### Example: Image Analysis Skill
```markdown theme={"theme":{"light":"github-light","dark":"dark-plus"}}
---
name: pillow
description: Image manipulation toolkit using Pillow (PIL)
license: MIT
toolsets:
- image
---
# Pillow Image Processing
## Description
A comprehensive image processing skill using the Pillow library.
## Capabilities
| Function | Description | Input | Output |
|----------|-------------|-------|--------|
| `resize` | Resize image | Image + dimensions | Resized image |
| `crop` | Crop region | Image + bounding box | Cropped image |
| `rotate` | Rotate image | Image + angle | Rotated image |
## Constraints
- Maximum input resolution: 4096x4096
- Supported formats: PNG, JPEG, WebP
```
### Example: Video Analysis Skill
```markdown theme={"theme":{"light":"github-light","dark":"dark-plus"}}
---
name: video-highlight-detection
description: Detect and label key moments and highlights in video content
toolsets:
- core
- video
---
# Video Highlight Detection
## Objective
Analyze videos to identify and label key moments, scene transitions,
and notable events with precise timestamps.
## Analysis Strategy
1. Watch the full video to understand the overall narrative and context
2. Identify scene transitions and notable events by timestamp
3. Classify each highlight by category (action, dialogue, transition)
4. Record start and end times in MM:SS format
## Output Requirements
- Each highlight must include a description, category, and timestamps
- Use the exact category names defined in the schema
```
Write instructions as if you're briefing an expert analyst. Be specific about what to look for, how to classify it, and what format to use for the output.
# vlmrun.yaml
Source: https://docs.vlm.run/skills/spec/vlmrun-yaml
Execution configuration for agent-powered skills
`vlmrun.yaml` defines how a skill executes within VLM Run's Orion agent. It specifies the model, available toolsets, and a state machine graph that orchestrates the agent's workflow.
## Format
```yaml theme={"theme":{"light":"github-light","dark":"dark-plus"}}
apiVersion: vlm.run/v1alpha
metadata: {}
model: vlmrun-orion-1:auto
toolsets:
- core
- image
graph: |
stateDiagram-v2
AnalyzeImage: Analyze input image content
Output: Produce structured output conforming to the target schema
[*] --> AnalyzeImage
AnalyzeImage --> Output
Output --> [*]
```
## Fields
| Field | Type | Required | Description |
| ------------ | ---------- | -------- | ------------------------------------------------------------ |
| `apiVersion` | `string` | Yes | API version — currently `vlm.run/v1alpha` |
| `metadata` | `object` | No | Reserved for future use (pass `{}`) |
| `model` | `string` | Yes | Orion model variant to use |
| `toolsets` | `string[]` | Yes | Tool categories available to the agent |
| `graph` | `string` | Yes | State machine definition in Mermaid `stateDiagram-v2` format |
| `plan` | `string` | No | Human-readable explanation of the state machine |
### Model Variants
| Model | Description |
| --------------------- | ---------------------------------------- |
| `vlmrun-orion-1:fast` | Optimized for speed |
| `vlmrun-orion-1:auto` | Balanced speed and quality (recommended) |
| `vlmrun-orion-1:pro` | Maximum quality |
### Toolsets
| Toolset | Description |
| ----------- | -------------------------------------------- |
| `core` | Basic operations (file I/O, text processing) |
| `image` | Image analysis and understanding |
| `image-gen` | Image generation and editing |
| `video` | Video analysis and understanding |
| `document` | Document extraction and layout understanding |
## State Machine Graph
The `graph` field uses [Mermaid stateDiagram-v2](https://mermaid.js.org/syntax/stateDiagram.html) syntax to define the agent's execution flow. Each state represents a step the agent performs, with transitions defining the order.
### Syntax
```
stateDiagram-v2
StateName: Description of what the agent does in this state
[*] --> FirstState # Entry point
FirstState --> NextState # Transition
FinalState --> [*] # Exit point
```
### Simple Example (2 states)
A basic image analysis skill with analyze → output flow:
```yaml theme={"theme":{"light":"github-light","dark":"dark-plus"}}
graph: |
stateDiagram-v2
AnalyzeImage: Analyze and process image content using multi-modal vision capabilities to identify objects, extract visual features, detect text, and understand spatial relationships within the scene
Output: Aggregate and reason over all analysis results, validate extracted information for consistency, and produce a well-structured output conforming to the target schema
[*] --> AnalyzeImage
AnalyzeImage --> Output
Output --> [*]
```
### Multi-Step Example (3+ states)
A video analysis skill with multiple processing stages:
```yaml theme={"theme":{"light":"github-light","dark":"dark-plus"}}
graph: |
stateDiagram-v2
AnalyzeVideo: Watch the full video to understand the narrative structure and identify key segments
ExtractHighlights: For each identified segment, extract detailed highlight data including timestamps and categories
Output: Aggregate all extracted highlights, validate for consistency, and produce structured output
[*] --> AnalyzeVideo
AnalyzeVideo --> ExtractHighlights
ExtractHighlights --> Output
Output --> [*]
```
## Plan Field
The optional `plan` field provides a human-readable explanation of the state machine. It helps document the skill's workflow for other developers:
```yaml theme={"theme":{"light":"github-light","dark":"dark-plus"}}
plan: |
## Objective
Analyze videos to detect and label key moments, scene transitions, and highlights.
## Nodes
- `AnalyzeVideo`: Watch the full video to understand the overall narrative
- `ExtractHighlights`: Identify each highlight with timestamps and categories
- `Output`: Compile results into the target schema format
```
Write descriptive state names and descriptions. The agent uses them to understand what to do at each step — more detail leads to better execution.
# Agent Execution
Source: https://docs.vlm.run/skills/usage/agent
Use skills with the agent execution endpoint
Pass skills in the `config.skills` parameter when executing agents:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionConfig, AgentSkill
from vlmrun.types import MessageContent, FileUrl
client = VLMRun(api_key="")
response = client.agent.execute(
inputs={"file": MessageContent(type="file_url", file_url=FileUrl(url=""))},
config=AgentExecutionConfig(
skills=[AgentSkill(skill_name="patient-referral", skill_version="20260219-abc123")]
),
batch=True,
)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
});
const response = await client.agent.execute({
inputs: { file: { type: "file_url", file_url: { url: "" } } },
config: {
skills: [{ skillName: "patient-referral", skillVersion: "20260219-abc123" }],
},
batch: true,
});
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/agent/execute \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{
"inputs": {"file": {"type": "file_url", "file_url": {"url": ""}}},
"config": {
"skills": [{"skill_name": "patient-referral", "skill_version": "20260219-abc123"}]
},
"batch": true
}'
```
## Multiple Skills
Pass multiple skills in the `config.skills` array:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
response = client.agent.execute(
inputs={...},
config=AgentExecutionConfig(
skills=[
AgentSkill(skill_name="document-parsing"),
AgentSkill(skill_name="data-validation"),
]
),
batch=True,
)
```
## Inline Skills
Instead of referencing a server-stored skill by name, you can send the skill bundle directly in the request as a base64-encoded zip. The Python SDK provides [`AgentSkill.from_directory`](https://github.com/vlm-run/vlmrun-python-sdk/blob/main/vlmrun/client/types.py) to build an inline `AgentSkill` from a local directory in one call:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentExecutionConfig, AgentSkill
client = VLMRun(api_key="")
# Build an inline AgentSkill from a local skill directory
skill = AgentSkill.from_directory(Path("./my-skill"))
response = client.agent.execute(
inputs={"file": {"type": "file_url", "file_url": {"url": ""}}},
config=AgentExecutionConfig(skills=[skill]),
batch=True,
)
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/agent/execute \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{
"inputs": {"file": {"type": "file_url", "file_url": {"url": ""}}},
"config": {
"skills": [{
"type": "inline",
"name": "my-skill",
"source": {
"type": "base64",
"media_type": "application/zip",
"data": ""
}
}]
},
"batch": true
}'
```
`AgentSkill.from_directory` zips the directory, base64-encodes it, and reads the name and description from the `SKILL.md` frontmatter automatically. It returns an `AgentSkill` with `type="inline"` ready for use.
See [Skill Structure — Inline Skill Bundles](/skills/spec/overview#inline-skill-bundles) for details on bundle contents and the [Reference](/skills/reference#inline-skill-fields) for `AgentSkill` inline fields.
## Batch Execution
Agent execution currently supports batch mode only.
* **Batch** (`batch=True`): Returns immediately with a prediction ID. Poll for results. Suitable for both short tasks and long-running tasks like video analysis.
* **Synchronous** (`batch=False`): *Not currently supported.* Use `batch=True` and poll for results.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
# Batch execution
response = client.agent.execute(
inputs={...},
config=AgentExecutionConfig(
skills=[AgentSkill(skill_name="video-analysis")]
),
batch=True,
)
# Poll for results
prediction = client.predictions.get(response.id)
```
# Chat Completions
Source: https://docs.vlm.run/skills/usage/chat
Use skills with the chat completions endpoint
Pass skills in the `skills` parameter when creating chat completions:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentSkill
client = VLMRun(api_key="")
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract data from this document."},
{"type": "file_url", "file_url": {"url": "https://example.com/invoice.pdf"}}
]
}
],
skills=[AgentSkill(skill_name="invoice-extraction")],
)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Extract data from this document." },
{ type: "file_url", file_url: { url: "https://example.com/invoice.pdf" } }
]
}
],
skills: [{ skillName: "invoice-extraction" }],
});
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/openai/chat/completions \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{
"model": "vlmrun-orion-1:auto",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Extract data from this document."},
{"type": "file_url", "file_url": {"url": "https://example.com/invoice.pdf"}}
]
}
],
"skills": [{"skill_name": "invoice-extraction"}]
}'
```
## Multiple Skills
You can pass multiple skills in a single request. The agent applies all skills during execution:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[...],
skills=[
AgentSkill(skill_name="invoice-extraction"),
AgentSkill(skill_name="line-item-validation"),
],
)
```
## Inline Skills
Instead of referencing a server-stored skill, you can send a skill bundle directly in the request as a base64-encoded zip. The Python SDK provides [`AgentSkill.from_directory`](https://github.com/vlm-run/vlmrun-python-sdk/blob/main/vlmrun/client/types.py) to build an inline `AgentSkill` from a local directory in one call:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import AgentSkill
client = VLMRun(api_key="")
# Build an inline AgentSkill from a local skill directory
skill = AgentSkill.from_directory(Path("./my-skill"))
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract data from this invoice."},
{"type": "file_url", "file_url": {"url": "https://example.com/invoice.pdf"}}
]
}
],
skills=[skill],
)
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/openai/chat/completions \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{
"model": "vlmrun-orion-1:auto",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Extract data from this invoice."},
{"type": "file_url", "file_url": {"url": "https://example.com/invoice.pdf"}}
]
}
],
"skills": [{
"type": "inline",
"name": "invoice-extraction",
"description": "Extract structured data from invoices",
"source": {
"type": "base64",
"media_type": "application/zip",
"data": ""
}
}]
}'
```
`AgentSkill.from_directory` zips the directory, base64-encodes it, and reads the name and description from the `SKILL.md` frontmatter automatically. It returns an `AgentSkill` with `type="inline"` ready for use.
See [Skill Structure — Inline Skill Bundles](/skills/spec/overview#inline-skill-bundles) for details on bundle contents and the [Reference](/skills/reference#inline-skill-fields) for `AgentSkill` inline fields.
You can mix referenced and inline skills in the same request. Each entry in the `skills` array is resolved independently.
## Pinned Versions
Pin a specific skill version for reproducible results:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[...],
skills=[AgentSkill(skill_name="patient-referral", skill_version="20260219-abc123")],
)
```
See [Version Pinning](/skills/usage/version-pinning) for details.
# Model Request
Source: https://docs.vlm.run/skills/usage/generation
Use skills with model requests
Skills work with all VLM Run API generation endpoints (`api.vlm.run`). Pass skills in the `config.skills` parameter to automatically apply the skill's prompt and JSON schema to your request.
## Image → JSON
Extract structured JSON from images:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from PIL import Image
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig, AgentSkill
client = VLMRun(api_key="")
response = client.image.generate(
images=[Image.open("photo.jpg")],
model="vlm-1",
config=GenerationConfig(
skills=[AgentSkill(skill_name="invoice-extraction", version="latest")]
)
)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
const response = await client.image.generate({
images: ["photo.jpg"],
model: "vlm-1",
config: {
skills: [{ skillName: "invoice-extraction", version: "latest" }],
},
});
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/image/generate \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{
"model": "vlm-1",
"images": [""],
"config": {
"skills": [{"skill_name": "invoice-extraction", "version": "latest"}]
}
}'
```
## Document → JSON
Extract structured JSON from documents:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig, AgentSkill
client = VLMRun(api_key="")
response = client.document.generate(
file=Path("invoice.pdf"),
model="vlm-1",
config=GenerationConfig(
skills=[AgentSkill(skill_name="invoice-extraction", version="latest")]
),
)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
const fileResponse = await client.files.upload({ filePath: "invoice.pdf" });
const response = await client.document.generate({
fileId: fileResponse.id,
model: "vlm-1",
config: {
skills: [{ skillName: "invoice-extraction", version: "latest" }],
},
});
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/document/generate \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{
"model": "vlm-1",
"file_id": "",
"config": {
"skills": [{"skill_name": "invoice-extraction", "version": "latest"}]
}
}'
```
## Video → JSON
Extract structured JSON from videos:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig, AgentSkill
client = VLMRun(api_key="")
response = client.video.generate(
file=Path("recording.mp4"),
model="vlm-1",
config=GenerationConfig(
skills=[AgentSkill(skill_name="meeting-notes", version="latest")]
),
batch=True,
)
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({ apiKey: "" });
const fileResponse = await client.files.upload({ filePath: "recording.mp4" });
const response = await client.video.generate({
fileId: fileResponse.id,
model: "vlm-1",
config: {
skills: [{ skillName: "meeting-notes", version: "latest" }],
},
batch: true,
});
```
```bash cURL theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -X POST https://api.vlm.run/v1/video/generate \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{
"model": "vlm-1",
"file_id": "",
"batch": true,
"config": {
"skills": [{"skill_name": "meeting-notes", "version": "latest"}]
}
}'
```
When `skills` are provided and `domain` is omitted, the platform creates a dynamic application from the skill's prompt and JSON schema. You do not need to specify a domain.
# Version Pinning
Source: https://docs.vlm.run/skills/usage/version-pinning
Pin skill versions for reproducible results
By default, `skill_version` is `"latest"`, which resolves to the most recent revision of the skill. You can pin a specific version for reproducibility.
## Latest vs Pinned
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client.types import AgentSkill
# Always use the latest version
skill = AgentSkill(skill_name="invoice-extraction")
# Pin a specific version
skill = AgentSkill(skill_name="invoice-extraction", skill_version="20260219-abc123")
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
// Always use the latest version
const skill = { skillName: "invoice-extraction" };
// Pin a specific version
const pinnedSkill = { skillName: "invoice-extraction", skillVersion: "20260219-abc123" };
```
## When to Pin
| Scenario | Recommendation |
| --------------------------- | ----------------------------------------------------------- |
| **Production pipelines** | Pin a specific version for consistent, reproducible results |
| **Development and testing** | Use `"latest"` to automatically pick up improvements |
| **A/B testing** | Pin different versions to compare extraction quality |
| **Compliance workloads** | Pin to ensure auditable, repeatable outputs |
## Version Format
Version strings follow the format `YYYYMMDD-`, for example `20260219-abc123`. Each time a skill is [updated](/skills/manage/update), a new version is created with a new unique ID but the same name.
## Checking Available Versions
Use the [Skills API](/api-reference/v1/skills/get-skills-list) to list available versions for a skill:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
# Get the latest version
skill = client.agent.skills.get(name="invoice-extraction")
print(f"Latest version: {skill.skill_version}")
# Get a specific version
skill = client.agent.skills.get(name="invoice-extraction", skill_version="20260219-abc123")
```
# Supported Files
Source: https://docs.vlm.run/supported-files
File formats supported by VLM Run for document, image, video, and audio processing.
VLM Run supports a wide range of file formats across different media types for structured data extraction.
## Document Files
| Extension | Format |
| :-------- | :------------------------------------------- |
| `.pdf` | Portable Document Format |
| `.doc` | Microsoft Word Document |
| `.docx` | Microsoft Word Document (Open XML) |
| `.ppt` | Microsoft PowerPoint Presentation |
| `.pptx` | Microsoft PowerPoint Presentation (Open XML) |
## Image Files
| Extension | Format |
| :-------- | :------------------------------ |
| `.jpeg` | JPEG Image |
| `.jpg` | JPEG Image |
| `.png` | Portable Network Graphics |
| `.gif` | Graphics Interchange Format |
| `.bmp` | Bitmap Image |
| `.tiff` | Tagged Image File Format |
| `.webp` | WebP Image |
| `.heic` | High Efficiency Image Container |
| `.avif` | AV1 Image File Format |
## Video Files
| Extension | Format |
| :-------- | :-------------- |
| `.mp4` | MPEG-4 Video |
| `.mov` | QuickTime Movie |
| `.mkv` | Matroska Video |
| `.webm` | WebM Video |
## Audio Files
| Extension | Format |
| :-------- | :------------------------ |
| `.wav` | Waveform Audio |
| `.mp3` | MPEG Audio Layer III |
| `.flac` | Free Lossless Audio Codec |
| `.webm` | WebM Audio |
| `.ogg` | Ogg Vorbis Audio |
## Text Files
| Extension | Format |
| :-------- | :--------------------- |
| `.txt` | Plain Text |
| `.md` | Markdown |
| `.json` | JSON |
| `.csv` | Comma-Separated Values |
| `.tsv` | Tab-Separated Values |
| `.jsonl` | JSON Lines |
# Ways to Use VLM Run
Source: https://docs.vlm.run/ways-to-use-vlm-run
The four entry points into VLM Run (Requests, Executions, Chat Completions API, and Chat UI) and when to reach for each.
VLM Run exposes four distinct entry points into our models, each tuned for a different kind of workload, from single-shot structured extraction to fully agentic, multi-step pipelines to interactive chat. This doc walks through each method, when to reach for it, and how they compare.
## 1. Requests
**Model:** `vlm-1`
Requests are the simplest way to use VLM Run: provide a single file and a skill or domain, and get structured JSON back. They're designed for **ETL-style workloads** where you have a fixed prompt (the skill or domain) and want flexibility on the schema.
* **Input:** a single document, image, audio file, or video
* **Output:** JSON
* **Execution:** batch. Submit a request and poll for the prediction by ID
* **Best for:** single-step extraction at scale (invoices, receipts, IDs, medical forms, etc.)
```python Using Skills theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig, AgentSkill
client = VLMRun(api_key="")
response = client.document.generate(
file=Path("invoice.pdf"),
model="vlm-1",
config=GenerationConfig(
skills=[AgentSkill(skill_name="invoice-extraction", version="latest")]
),
)
print(response.response)
```
```python Using Domains theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.document.generate(
file=Path("invoice.pdf"),
model="vlm-1",
domain="document.invoice",
)
```
Think of Requests as the "one file in, one JSON out" primitive. Fixed prompt, flexible schema, no orchestration.
## 2. Executions
**Model:** `vlmrun-orion-1`
Executions are for **agentic, multi-step workloads**. Where a Request is one model call against one file, an Execution runs an agent that can classify, extract, redact, transform, and combine outputs across **multiple files**, all orchestrated through a skill.
* **Input:** multiple documents, images, and/or videos
* **Output:** JSON
* **Execution:** batch. Submit an execution and poll for the result by ID
* **Configured via:** a skill, defined primarily by a `SKILL.md` file, with optional reference files, schemas, and examples
* **Best for:** anything open-ended or multi-step (document packages, cross-file reasoning, redaction pipelines, classification-then-extraction flows)
Skills are the unit of configuration here: `SKILL.md` gives the agent its instructions, and supporting files (schemas, examples, reference docs) ground its behavior. This is the most powerful and flexible surface we offer.
## 3. Chat Completions API
**Model:** `vlmrun-orion-1`
The Chat Completions API is a **drop-in replacement for the OpenAI Chat Completions API**. Point the OpenAI SDK at our base URL and you're using Orion with full visual-tool and artifact support, with no other code changes required.
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from openai import OpenAI
client = OpenAI(
base_url="https://api.vlm.run/v1/openai",
api_key="",
)
```
* **Input:** multiple files via standard chat messages (text + image/file parts)
* **Output:** Text, JSON
* **Execution:** both streaming and non-streaming
* **Logging:** programmatic calls are logged in the **chat completions** table
* **Best for:** interactive or conversational multimodal use cases, and any app already built against the OpenAI SDK
This is the lowest-friction path for teams already using the OpenAI Chat Completions API: you get Orion's capabilities with the API shape your code already speaks.
## 4. Chat UI
**Model:** `vlmrun-orion-1`
The Chat UI ([chat.vlm.run](https://chat.vlm.run)) is our hosted chat interface. It's powered by the same Chat Completions API under the hood, giving you Orion with visual tools and artifacts in a browser, with no code required.
* **Input:** files and messages via the web UI
* **Output:** Text, JSON (artifacts rendered in the browser)
* **Logging:** Chat UI sessions are **not** logged in the chat completions table (only programmatic API calls are)
* **Best for:** exploration, demos, one-off tasks, and iterating on prompts before writing code
## Summary Table
| | **Requests** | **Executions** | **Chat Completions** | **Chat UI** |
| ------------------------------------- | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------ |
| **Model** | `vlm-1` | `vlmrun-orion-1` | `vlmrun-orion-1` | `vlmrun-orion-1` |
| **Input** | Single file (doc, image, audio, or video) | Multiple files | Multiple files | Multiple files |
| **Output** | JSON | JSON | Text, JSON | Text, JSON |
| **Mode** | Batch | Batch | Streaming + non-streaming | Streaming |
| **Prompt model** | Fixed prompt, flexible schema | Open-ended; defined by `SKILL.md` + reference files | Free-form messages | Free-form messages |
| **Visual tool calling** | ❌ | ✅ | ✅ | ✅ |
| **Artifacts** | ❌ | ✅ | ✅ | ✅ |
| **Visual grounding (bounding boxes)** | ✅ | ❌ | ❌ | ❌ |
| **Workload shape** | Single-step ETL | Multi-step / agentic | Conversational / multimodal | Conversational / multimodal |
| **Best for** | High-volume structured extraction | Complex multi-file reasoning and actions | Apps built on the OpenAI chat completions API | Playground for exploration and demos |
| **Example usage** | Known document types with fixed output schemas (invoices, receipts, IDs) | Custom multi-file pipelines (classify, extract, redact across a document package) | Multimodal chatbots and OpenAI-SDK apps using Orion | Exploring a new domain or iterating on a skill quickly |
# Webhooks
Source: https://docs.vlm.run/webhooks
Receive real-time notifications when your async processing jobs complete
Webhooks allow you to receive real-time HTTP notifications when your asynchronous processing jobs complete. Instead of polling the predictions endpoint, VLM Run will automatically send the results to your specified callback URL.
## Overview
When you submit a request with `batch=True` and provide a `callback_url`, VLM Run will:
1. Process your request asynchronously
2. Generate the results
3. Send an HTTP POST request to your callback URL with the complete response
4. Include an HMAC signature for verification (if you've configured a webhook secret)
## Setting Up Webhooks
### 1. Configure Your Webhook Secret
To ensure webhook requests are authentic, you should configure a webhook secret in your account settings. This secret is used to generate HMAC signatures that you can verify on your server.
**Setting your webhook secret:**
1. Log in to your [VLM Run Dashboard](https://app.vlm.run)
2. Navigate to Settings → API Keys
3. Set your webhook secret (keep this secure and never share it)
### 2. Create a Webhook Endpoint
Create an endpoint on your server to receive webhook notifications. The endpoint should:
* Accept POST requests
* Verify the HMAC signature (recommended)
* Process the webhook payload
* Return a 2xx status code to acknowledge receipt
### 3. Submit Requests with Callback URL
When making API requests, include the `callback_url` parameter:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the client
client = VLMRun()
response = client.document.generate(
file_id="file_abc123",
domain="document.invoice",
batch=True,
callback_url="https://your-domain.com/webhooks/vlmrun"
)
print(f"Request ID: {response.id}")
# Your webhook endpoint will receive the results when processing completes
```
```javascript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from 'vlmrun';
const client = new VlmRun({
apiKey: 'your-api-key',
});
const response = await client.document.generate({
fileId: 'file_abc123',
domain: 'document.invoice',
batch: true,
callbackUrl: 'https://your-domain.com/webhooks/vlmrun'
});
console.log(`Request ID: ${response.id}`);
// Your webhook endpoint will receive the results when processing completes
```
## Verifying Webhook Signatures
VLM Run signs all webhook requests with an HMAC signature using your webhook secret. The signature is included in the `X-VLMRun-Signature` header.
**Important:** Always verify webhook signatures to ensure requests are authentic and haven't been tampered with.
### Using the SDK (Recommended)
The VLM Run SDKs provide built-in webhook verification utilities that handle signature validation securely.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import os
import json
from vlmrun.common.webhook import verify_webhook
# In your webhook handler, obtain the raw request body (bytes),
# signature header, and webhook secret
raw_body = request.body() # Must be bytes, not parsed JSON
signature = request.headers.get("X-VLMRun-Signature")
secret = os.environ["VLMRUN_WEBHOOK_SECRET"]
# Verify signature using SDK
if not verify_webhook(raw_body, signature, secret):
return {"error": "Invalid signature"}, 401
# Parse and process webhook
data = json.loads(raw_body)
print(f"Received webhook for prediction: {data['id']}")
# Handle the completed prediction
if data["status"] == "completed":
response_data = data["response"]
# ... your business logic here
return {"status": "success"}
```
```javascript Node.js SDK theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { verifyWebhook } from 'vlmrun';
// In your webhook handler, obtain the raw request body (Buffer),
// signature header, and webhook secret
const rawBody = req.body; // Must be Buffer, not parsed JSON
const signature = req.headers['x-vlmrun-signature'];
const secret = process.env.VLMRUN_WEBHOOK_SECRET;
// Verify signature using SDK
if (!verifyWebhook(rawBody, signature, secret)) {
return res.status(401).json({ error: 'Invalid signature' });
}
// Parse and process webhook
const data = JSON.parse(rawBody.toString('utf8'));
console.log(`Received webhook for prediction: ${data.id}`);
// Handle the completed prediction
if (data.status === 'completed') {
const responseData = data.response;
// ... your business logic here
}
res.json({ status: 'success' });
```
**Important:** Always verify the raw request body before any JSON parsing. For Express, use `express.raw({ type: 'application/json' })` middleware on your webhook route to ensure the body is available as a Buffer.
### Manual Verification (Alternative)
If you prefer not to use the SDK, you can implement signature verification manually:
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import hmac
import hashlib
def verify_webhook(raw_body: bytes, signature_header: str, secret: str) -> bool:
if not signature_header or not signature_header.startswith("sha256="):
return False
received_sig = signature_header[len("sha256="):]
expected_sig = hmac.new(
secret.encode("utf-8"),
raw_body,
hashlib.sha256
).hexdigest()
return hmac.compare_digest(received_sig, expected_sig)
# Usage in your webhook handler
raw_body = request.body()
signature = request.headers.get("X-VLMRun-Signature")
secret = os.environ["VLMRUN_WEBHOOK_SECRET"]
if not verify_webhook(raw_body, signature, secret):
return {"error": "Invalid signature"}, 401
```
```javascript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
const crypto = require('crypto');
function verifyWebhook(rawBody, signatureHeader, secret) {
if (!signatureHeader || !signatureHeader.startsWith('sha256=')) {
return false;
}
const receivedSig = signatureHeader.replace('sha256=', '');
const expectedSig = crypto
.createHmac('sha256', secret)
.update(rawBody)
.digest('hex');
try {
return crypto.timingSafeEqual(
Buffer.from(receivedSig, 'hex'),
Buffer.from(expectedSig, 'hex')
);
} catch (error) {
return false;
}
}
// Usage in your webhook handler (with express.raw middleware)
const rawBody = req.body;
const signature = req.headers['x-vlmrun-signature'];
const secret = process.env.VLMRUN_WEBHOOK_SECRET;
if (!verifyWebhook(rawBody, signature, secret)) {
return res.status(401).json({ error: 'Invalid signature' });
}
```
## Webhook Delivery
### Retry Logic
VLM Run automatically retries failed webhook deliveries with exponential backoff:
* **Maximum retries:** 2 attempts
* **Timeout:** 30 seconds per request
* **Backoff:** Exponential (1s, 2s, 4s, etc.)
A webhook delivery is considered failed if:
* The endpoint returns a non-2xx status code
* The request times out after 30 seconds
* A network connection error occurs
## Testing Webhooks Locally
To test webhooks from a local server, use a tunneling tool such as [ngrok](https://ngrok.com/), [Cloudflare Tunnel](https://www.cloudflare.com/products/tunnel/), or [localtunnel](https://localtunnel.github.io/www/) to expose your endpoint and use the public URL as your callback\_url.
## Related Resources
* [Predictions API Reference](/api-reference/v1/predictions/get-predictions-by-id)
* [Batch Processing Guide](/guides/doc-ai/guide-parsing-documents#batch-processing)
* [Error Codes](/error-codes)