Multi-modal Artifacts

Artifacts are binary objects generated during agent interactions, such as images, videos, audio files, and documents. When agents perform operations like image generation, face blurring, video trimming, or document processing, the results are stored as artifacts that can be retrieved using object references.

Object References

Agent responses return object references (refs) instead of raw binary data. Each reference is a string identifier that follows a specific format: a 3-5 letter type prefix followed by an underscore and a 6-digit hexadecimal string (e.g., img_a1b2c3).

Artifact Type	Prefix	Reference Type	Python Return Type
Image	`img_`	`ImageRef`	`PIL.Image.Image`
Video	`vid_`	`VideoRef`	`Path` (mp4)
Audio	`aud_`	`AudioRef`	`Path` (mp3)
Document	`doc_`	`DocumentRef`	`Path` (pdf)
Reconstruction	`recon_`	`ReconRef`	`Path` (spz)
URL	`url_`	`UrlRef`	`Path` (any of the above)
Array	`arr_`	`ArrayRef`	`np.ndarray`

Import reference types from the SDK:

from vlmrun.types import ImageRef, VideoRef, AudioRef, DocumentRef, ReconRef, UrlRef

Retrieving an Artifact

In a Chat Completion

To retrieve a chat completion artifact, use the session_id from the chat response and the object_id (returned as a Ref type) from the JSON result.

from pathlib import Path
from pydantic import BaseModel, Field
from PIL import Image
from vlmrun.client import VLMRun
from vlmrun.types import ImageRef

client = VLMRun(api_key="<VLMRUN_API_KEY>")

# Define a response model with an ImageRef field
class BlurredImageResponse(BaseModel):
    image: ImageRef = Field(..., description="The blurred image")

# Make a chat completion request
response = client.agent.completions.create(
    model="vlmrun-orion-1:auto",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Blur all the faces in this image"},
                {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
            ]
        }
    ],
    response_format={
        "type": "json_schema",
        "schema": BlurredImageResponse.model_json_schema()
    }
)

# Parse the response
result = BlurredImageResponse.model_validate_json(response.choices[0].message.content)

# Retrieve the artifact using session_id and object_id
blurred_image: Image.Image = client.artifacts.get(
    session_id=response.session_id,
    object_id=result.image.id
)

# Display or save the image
blurred_image.save("blurred_output.jpg")

Coming soon...

Common Use Cases