Skip to main content
VLM Run’s Orion agents can leverage various video-editing tools such as trimming, sampling, and extracting segments from videos. These tools are designed to help you extract key moments from videos, trim videos to specific segments, and sample frames from videos for analysis.

Full video used to demonstrate video tools such as trimming, sampling, and keyframe detection

​
Example Usage

For best results, we recommend using the Structured Outputs API to get responses in a structured and validated data format.

​
1. Video Frame Sampling

Extract frames at regular intervals or specific timestamps for analysis. 
Extract at least 3 frames from the video for thumbnail generation.

Example of 3 frames extracted from the video for thumbnail generation.

from vlmrun.client import VLMRun

# Initialize the VLMRun client
client = VLMRun(api_key="<VLMRUN_API_KEY>", base_url="https://agent.vlm.run/v1")

# Extract keyframes for thumbnails
response = client.agent.completions.create(
    model="vlmrun-orion-1:auto",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract keyframes from this video for thumbnail generation, sampling every 5 seconds"},
                {"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
            ]
        }
    ]
)

print(response.choices[0].message.content)
# >>> {"frames": [{"url": "https://.../frame-1.jpg", "timestamp": "00:00:00.000"}, ...]}

​
2. Video Highlight Extraction

Our video agents can extract the best moments from a video, focusing on scoring plays and key actions. 
Extract the 3 best moments from this video, including the start and end times of each moment.

Example of 3 video highlight extraction.

from vlmrun.client import VLMRun

# Initialize the VLMRun client
client = VLMRun(api_key="<VLMRUN_API_KEY>", base_url="https://agent.vlm.run/v1")

# Extract multiple segments
response = client.agent.completions.create(
    model="vlmrun-orion-1:auto",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract the 3 best moments from this video, including the start and end times of each moment."},
                {"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
            ]
        }
    ]
)

print(response.choices[0].message.content)
# >>> {"segments": [...]}

​
3. Time-Based Trimming

Extract specific segments from videos with precise start and end timestamps. 
Trim the video from 10 seconds to 30 seconds

Example of time-based trimming of a 20 second video.

from vlmrun.client import VLMRun

# Initialize the VLMRun client
client = VLMRun(api_key="<VLMRUN_API_KEY>", base_url="https://agent.vlm.run/v1")

# Trim video
response = client.agent.completions.create(
    model="vlmrun-orion-1:auto",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Trim the video from 10 seconds to 30 seconds"},
                {"type": "video_url", "video_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4"}}
            ]
        }
    ]
)

print(response.choices[0].message.content)
# >>> {"start_time": "00:00:10.000", "end_time": "00:00:30.000", "url": "https://.../trimmed.mp4"}

​
FAQ

  • MP4: Most common format with excellent compatibility
  • MOV: Apple QuickTime format
  • AVI: Windows video format
  • MKV: Matroska video format
  • WebM: Web-optimized format
  • Quality Preservation: Maintains original video quality in trimmed segments
  • Uniform Sampling: Extract frames at regular intervals (e.g., every 1-5 seconds)
  • Keyframe Sampling: Extract only keyframes for efficient analysis
  • Scene-Based: Sample based on scene changes for better content analysis
  • Quality Balance: Choose appropriate sampling rate based on analysis needs
  • Millisecond Precision: Cut videos to exact time ranges with millisecond accuracy
  • Keyframe Alignment: Align cuts to nearest keyframes for clean edits
  • Smart Boundaries: Automatically detect optimal cut points
  • Quality Preservation: Maintain video quality without re-encoding when possible

Try Video Trimming

Experience video trimming and frame sampling with live examples in our interactive chat interface