> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.
# Multi-modal Artifacts
> Retrieve generated images, videos, audio, and documents from agent responses
Artifacts are binary objects generated during agent interactions, such as images, videos, audio files, and documents. When agents perform operations like image generation, face blurring, video trimming, or document processing, the results are stored as artifacts that can be retrieved using object references.
## Object References
Agent responses return object references (refs) instead of raw binary data. Each reference is a string identifier that follows a specific format: a 3-5 letter type prefix followed by an underscore and a 6-digit hexadecimal string (e.g., `img_a1b2c3`).
| Artifact Type | Prefix | Reference Type | Python Return Type |
| -------------- | -------- | -------------- | ------------------------- |
| Image | `img_` | `ImageRef` | `PIL.Image.Image` |
| Video | `vid_` | `VideoRef` | `Path` (mp4) |
| Audio | `aud_` | `AudioRef` | `Path` (mp3) |
| Document | `doc_` | `DocumentRef` | `Path` (pdf) |
| Reconstruction | `recon_` | `ReconRef` | `Path` (spz) |
| URL | `url_` | `UrlRef` | `Path` (any of the above) |
| Array | `arr_` | `ArrayRef` | `np.ndarray` |
Import reference types from the SDK:
```python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.types import ImageRef, VideoRef, AudioRef, DocumentRef, ReconRef, UrlRef
```
## Retrieving an Artifact
### In a Chat Completion
To retrieve a chat completion artifact, use the `session_id` from the chat response and the `object_id` (returned as a Ref type) from the JSON result.
```python Python [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from pydantic import BaseModel, Field
from PIL import Image
from vlmrun.client import VLMRun
from vlmrun.types import ImageRef
client = VLMRun(api_key="")
# Define a response model with an ImageRef field
class BlurredImageResponse(BaseModel):
image: ImageRef = Field(..., description="The blurred image")
# Make a chat completion request
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Blur all the faces in this image"},
{"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
]
}
],
response_format={
"type": "json_schema",
"schema": BlurredImageResponse.model_json_schema()
}
)
# Parse the response
result = BlurredImageResponse.model_validate_json(response.choices[0].message.content)
# Retrieve the artifact using session_id and object_id
blurred_image: Image.Image = client.artifacts.get(
session_id=response.session_id,
object_id=result.image.id
)
# Display or save the image
blurred_image.save("blurred_output.jpg")
```
```typescript TypeScript [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
Coming soon...
```
{/* ## Retrieving Multiple Artifacts
Agents can return multiple artifacts in a single response:
```python Python [expandable]
from typing import List
from pydantic import BaseModel, Field
from PIL import Image
from vlmrun.client import VLMRun
from vlmrun.types import ImageRef
client = VLMRun(api_key="")
class VideoFramesResponse(BaseModel):
class VideoFrame(BaseModel):
image: ImageRef = Field(..., description="The video frame image")
timestamp: str = Field(..., description="Timestamp in HH:MM:SS format")
description: str = Field(..., description="Description of the scene")
frames: List[VideoFrame] = Field(..., description="Extracted frames")
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract 5 key frames from this video"},
{"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}}
]
}
],
response_format={
"type": "json_schema",
"schema": VideoFramesResponse.model_json_schema()
}
)
result = VideoFramesResponse.model_validate_json(response.choices[0].message.content)
# Retrieve all frame artifacts
frames: List[Image.Image] = [
client.artifacts.get(session_id=response.session_id, object_id=frame.image.id)
for frame in result.frames
]
for i, (frame, frame_data) in enumerate(zip(frames, result.frames)):
print(f"Frame {i+1}: [{frame_data.timestamp}] {frame_data.description}")
frame.save(f"frame_{i+1}.jpg")
```
*/}
## Common Use Cases
Generate multi-modal artifacts such as images and videos.
Generate multiple images of a scene (e.g. virtual try-on, video thumbnails, etc.).
Redact sensitive information from documents, and return the processed document as a PDF.
Generate 3D models from images or videos, and return ply/spz files.
## Best Practices
When working with artifacts, keep these guidelines in mind:
* For large artifacts like videos, the Python and Node SDKs download files to disk rather than loading them into memory. This prevents memory issues when working with large files. Always check the file size before loading video content into memory.
* Use structured response models with appropriate `Ref` types (`ImageRef`, `VideoRef`, etc.) to ensure type safety and enable IDE autocompletion. The Python and Node SDKs will automatically handle the conversion to the appropriate Python type when retrieving artifacts.
View the complete API reference for artifact retrieval