> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.
# Audio → JSON
> Generate structured prediction for the given audio file.
For all supported `audio` domains, see the [Hub Catalog](/hub#audio-domains).
```python Python (with domain) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
client = VLMRun(api_key="")
response = client.audio.generate(
file=Path(".mp3"),
domain="audio.transcription",
batch=True
)
```
```python Python (with skill) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from pathlib import Path
from vlmrun.client import VLMRun
from vlmrun.client.types import GenerationConfig, AgentSkill
client = VLMRun(api_key="")
response = client.audio.generate(
file=Path(".mp3"),
domain="audio.transcription",
batch=True,
config=GenerationConfig(
skills=[AgentSkill(skill_name="")]
)
)
```
```typescript Node.js SDK (with domain) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({apiKey: ""});
const fileResponse = await client.files.upload(
filePath: ".mp3"
);
const response = await client.audio.generate({
fileId: fileResponse.id,
domain: "audio.transcription",
});
```
```typescript Node.js SDK (with skill) theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({apiKey: ""});
const fileResponse = await client.files.upload(
filePath: ".mp3"
);
const response = await client.audio.generate({
fileId: fileResponse.id,
batch: true,
config: {
skills: [{ skillName: "" }],
},
});
```
### Example Output
```json Example Audio Transcription [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"metadata": {
"duration": 146.94
},
"segments": [
{
"start_time": 0,
"end_time": 24.88,
"content": " After reading tons of productivity books, I came across so many rules, like the two-year rule, the five-minute rule, the five-second rule. No, not that five second rule. The problem is that these rules were meant for companies or entrepreneurs, but I was able to adapt them to my studies during med school and drastically cut down to my procrastination. So I'm going to share with you two different two minute rules for the next two minutes. The first two minute rule comes from"
},
{
"start_time": 24.88,
"end_time": 45.5,
"content": " getting things done by David Allen. He says if it takes two minutes to do, get it done right now. For example, if I need to take out the trash today, it takes two minutes to do. So if I'm thinking about it now, might as well just do it now. Instead of writing it down on a to-do list or probably forgetting about it or having to come back to it later, which takes more than two minutes. That's how I see it."
},
{
"start_time": 45.5,
"end_time": 67.86,
"content": " So here's a list of things that might take two minutes throughout the day, like organizing your desk or watering your plants or clipping those nasty nails. I just do it when I notice it, but these little things start to add up, so this rule biases my brain towards taking action and away from procrastination. The second two-minute rule comes from atomic habits by James Clear. He says, when you're trying to do something you don't really want to do, simplify the"
},
{
"start_time": 67.86,
"end_time": 91.27,
"content": " task down to two minutes or less. So doing your entire reading assignment becomes just reading one paragraph or memorizing the entire periodic table becomes memorizing just 10 flashcards. Now, some of you might think, yeah, this is just a Jedi mind trick. Like, why would I fall for it? How is this at all sustainable? And to that, he says, when you're starting out, limit yourself to only two minutes."
},
{
"start_time": 91.27,
"end_time": 117.33,
"content": " So back in med school, I wanted to build a habit of studying for one hour every day before dinner. So I tried this trick, but I limited myself to just two minutes. I'd sit down, open my laptop, study for two minutes, and then close my laptop and went to do something else. It seems unproductive at first, right? It seems stupid. But staying consistent with this two-minute routine day after day meant that I was becoming the type of person who studies daily."
},
{
"start_time": 117.33,
"end_time": 137.99,
"content": " I was mastering the habit of just showing up because a habit needs to be established before it can be expanded upon. If I can't become a person who studies for just two minutes a day, I'd never be able to become the person that studies for an hour a day. You've got to start somewhere, but starting small is easier. There's a lot of other useful tips from books."
},
{
"start_time": 138.15,
"end_time": 146.94,
"content": " I cover more here in this video on three books and three minutes. Check it out. And if you guys like these types of videos, let me know in the comments below. I'll see you there. Bye."
}
]
}
```
```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
"usage": {
"elements_processed": 123,
"element_type": "image",
"credits_used": 123
},
"id": "",
"created_at": "2023-11-07T05:31:56Z",
"completed_at": "2023-11-07T05:31:56Z",
"response": "",
"status": "enqueued"
}
```
## OpenAPI
````yaml POST /v1/audio/generate
openapi: 3.1.0
info:
title: VLM Run Unified Server
description: Unified server for VLM Run Agent and API
termsOfService: https://vlm.run/terms-of-service
contact:
name: VLM Run Support Team
url: https://vlm.run/
email: support@vlm.run
version: 2026-05-19.0
servers: []
security: []
paths:
/v1/audio/generate:
post:
tags:
- audio
- audio
summary: Audio Generate
description: Generate structured prediction for the given audio file.
operationId: audio_generate_v1_audio_generate_post
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/AudioPredictionRequest'
required: true
responses:
'200':
description: Successful Response
content:
application/json:
schema: {}
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
security:
- HTTPBearer: []
components:
schemas:
AudioPredictionRequest:
properties:
metadata:
$ref: '#/components/schemas/RequestMetadata'
description: Optional metadata to pass to the model.
config:
$ref: '#/components/schemas/GenerationConfig'
description: The VLM generation config to be used for //generate.
url:
anyOf:
- type: string
- type: 'null'
title: Url
description: The URL of the file (provide either `file_id` or `url`).
file_id:
anyOf:
- type: string
- type: 'null'
title: File Id
description: The ID of the uploaded file (provide either `file_id` or `url`).
id:
type: string
title: Id
description: Unique identifier of the request.
created_at:
type: string
format: date-time
title: Created At
description: Date and time when the request was created (in UTC timezone)
callback_url:
anyOf:
- type: string
minLength: 1
format: uri
- type: 'null'
title: Callback Url
description: The URL to call when the request is completed.
model:
anyOf:
- type: string
enum:
- vlm-1
- vlm-1:auto
- vlm-1:fast
- vlm-1:pro
- type: string
title: Model
description: The model to use for generating the response.
default: vlm-1
domain:
anyOf:
- type: string
enum:
- audio.transcription
- audio.transcription-summary
- type: string
title: Domain
description: The domain identifier for the model (e.g. `audio.transcription`).
batch:
type: boolean
title: Batch
description: Whether to process the document in batch mode (async).
default: true
type: object
required:
- domain
title: AudioPredictionRequest
description: Request to the Audio API (i.e. structured prediction).
HTTPValidationError:
properties:
detail:
items:
$ref: '#/components/schemas/ValidationError'
type: array
title: Detail
type: object
title: HTTPValidationError
RequestMetadata:
properties:
environment:
type: string
enum:
- dev
- staging
- prod
title: Environment
description: The environment where the request was made.
default: dev
session_id:
anyOf:
- type: string
- type: 'null'
title: Session Id
description: The session ID of the request
allow_logging:
type: boolean
title: Allow Logging
description: Whether to enable logs for this request.
default: true
allow_training:
type: boolean
title: Allow Training
description: Whether the file can be used for training
default: true
allow_retention:
type: boolean
title: Allow Retention
description: Whether to allow retention of the data
default: true
extra:
additionalProperties: true
type: object
title: Extra
description: Extra metadata for the request (e.g. `dataset_id`, `subset_id`).
type: object
title: RequestMetadata
description: >-
Metadata for the request.
Typically captured in {"vlmrun": {"metadata": {"environment":
, ...}}.
GenerationConfig:
properties:
prompt:
anyOf:
- type: string
- type: 'null'
title: Prompt
description: >-
Additional user instructions appended to the application or skill
prompt for this request.
detail:
type: string
enum:
- auto
- hi
- lo
title: Detail
description: The detail level to use for processing multimodal data.
default: auto
json_schema:
anyOf:
- additionalProperties: true
type: object
- type: 'null'
title: Json Schema
description: >-
The overridden JSON schema to use for the model. To be used instead
of the response model.
skills:
anyOf:
- items:
$ref: '#/components/schemas/AgentSkill'
type: array
- type: 'null'
title: Skills
description: List of agent skills to enable for this generation request.
gql_stmt:
anyOf:
- type: string
- type: 'null'
title: Gql Stmt
description: >-
The GraphQL statement to use for the application. If provided, the
response model will be generated from the GraphQL statement.
max_retries:
type: integer
title: Max Retries
description: The maximum number of retries to use for the application.
default: 1
max_tokens:
type: integer
title: Max Tokens
description: The maximum number of tokens to use for the application.
default: 65535
temperature:
type: number
title: Temperature
description: The temperature to use for the application.
default: 0
confidence:
type: boolean
title: Confidence
description: >-
Include confidence scores in the response (included in the
`_metadata` field).
default: false
grounding:
type: boolean
title: Grounding
description: >-
Include grounding in the response (included in the `_metadata`
field).
default: false
keyframes:
type: boolean
title: Keyframes
description: Include keyframes in the video transcription response.
default: false
video_segment_duration:
anyOf:
- type: number
minimum: 1
- type: 'null'
title: Video Segment Duration
description: >-
Duration in seconds for each video segment when chunking a video for
transcription. Defaults to 150.0s.
video_frames_per_segment:
anyOf:
- type: integer
minimum: 1
- type: 'null'
title: Video Frames Per Segment
description: >-
Number of frames to sample per video segment for captioning.
Defaults to 8.
video_model:
anyOf:
- type: string
- type: 'null'
title: Video Model
description: >-
Model ID to use for video segment captioning (e.g.
'vlmrun-orion-1:fast'). When omitted, the server default is used.
video_input_mode:
anyOf:
- type: string
enum:
- frames
- native_video
- type: 'null'
title: Video Input Mode
description: >-
How to pass video to the captioning model: 'frames' extracts N JPEG
frames per segment, 'native_video' sends the mp4 clip directly via
video_url for models with native video understanding. Defaults to
'native_video' for Qwen deployment models, 'frames' for others.
video_transcribe_audio:
type: boolean
title: Video Transcribe Audio
description: >-
When True, transcribe the audio track to align segment boundaries.
When False (default), skip ASR and use fixed-duration video segments
only (visual-only captioning).
default: false
chat_context:
anyOf:
- type: string
- type: 'null'
title: Chat Context
description: >-
Plain-text chat transcript (prior turns + current request) used to
ground video captioning / transcription on what the user wants
extracted.
page_indices:
anyOf:
- items:
type: integer
type: array
- type: 'null'
title: Page Indices
description: >-
0-indexed page indices to process for document files. If None, all
pages are processed.
use_context_cache:
type: boolean
title: Use Context Cache
description: >-
Reuse cached representations of document/video content across calls.
When True (default), the file is cached after the first call so
repeated queries against the same file skip re-transmitting its
contents. Set to False to always send the full content.
default: true
service_tier:
anyOf:
- type: string
enum:
- auto
- default
- standard
- flex
- priority
- type: 'null'
title: Service Tier
description: >-
Delivery tier for the request. 'standard'/'default' uses baseline
rates, 'flex' applies a 50% discount with higher latency, 'priority'
applies a 1.8x premium. When omitted (or 'auto'), the server default
('standard') applies. The chosen tier drives both billing and the
latency/availability SLO.
type: object
title: GenerationConfig
description: Request configuration for image/document/video generation.
ValidationError:
properties:
loc:
items:
anyOf:
- type: string
- type: integer
type: array
title: Location
msg:
type: string
title: Message
type:
type: string
title: Error Type
input:
title: Input
ctx:
type: object
title: Context
type: object
required:
- loc
- msg
- type
title: ValidationError
AgentSkill:
properties:
type:
type: string
title: Type
description: >-
The type of the skill. Use 'skill_reference' for DB-stored skills
referenced by id/name. Use 'inline' to provide the skill as a
base64-encoded zip bundle.
default: skill_reference
skill_id:
anyOf:
- type: string
- type: 'null'
title: Skill Id
description: >-
The unique identifier of the skill — a UUID or a name string (e.g.,
'pillow', 'batch-processing').
skill_name:
anyOf:
- type: string
- type: 'null'
title: Skill Name
description: >-
Human-readable skill name for lookup (e.g., 'invoice-extraction').
Alternative to skill_id. Deprecated in favour of skill_id.
skill_version:
anyOf:
- type: integer
- type: string
title: Skill Version
description: The version of the skill — an integer (e.g. 2) or 'latest'.
default: latest
version:
anyOf:
- type: integer
- type: string
- type: 'null'
title: Version
description: 'DEPRECATED: Use ''skill_version'' instead. The version of the skill.'
name:
anyOf:
- type: string
- type: 'null'
title: Name
description: >-
Human-readable name for the inline skill (used for discovery and
logging).
description:
anyOf:
- type: string
- type: 'null'
title: Description
description: Short description of what the inline skill does.
source:
anyOf:
- $ref: '#/components/schemas/InlineSkillSource'
- type: 'null'
description: >-
Source payload for inline skills. Contains the base64-encoded zip
bundle with type, media_type, and data fields.
bundle:
anyOf:
- type: string
- type: 'null'
title: Bundle
description: >-
DEPRECATED: Use 'source.data' instead. Base64-encoded zip bundle
containing the skill files (inline skills only).
type: object
title: AgentSkill
description: >-
A modular capability that extends the agent's functionality.
Agent Skills are reusable, filesystem-based resources that provide the
agent
with domain-specific expertise: workflows, context, and best practices.
Each skill packages instructions, metadata, and optional resources
(scripts,
templates, snippets) that the agent uses automatically when relevant.
Two modes are supported:
1. **Referenced skills** (``type="skill_reference"``) – Provide
``skill_id``
(UUID or name) and optionally ``skill_version`` (integer or ``"latest"``).
.. code-block:: json
{"type": "skill_reference", "skill_id": "pillow", "skill_version": "latest"}
2. **Inline skills** (``type="inline"``) – Supply ``name``,
``description``,
and a ``source`` object containing the base64-encoded zip bundle. The zip
must contain exactly one ``SKILL.md`` file. No database lookup is required.
.. code-block:: json
{
"type": "inline",
"name": "csv-insights",
"description": "Summarize CSV files.",
"source": {
"type": "base64",
"media_type": "application/zip",
"data": ""
}
}
Legacy format with flat ``bundle`` field is also accepted for backward
compatibility.
InlineSkillSource:
properties:
type:
type: string
const: base64
title: Type
description: >-
Encoding type for the inline skill data. Currently only 'base64' is
supported.
default: base64
media_type:
type: string
title: Media Type
description: MIME type of the skill bundle. Must be 'application/zip'.
default: application/zip
data:
type: string
title: Data
description: Base64-encoded zip bundle containing the skill files.
type: object
required:
- data
title: InlineSkillSource
description: |-
Source payload for an inline skill bundle.
Follows the OpenAI inline skill format::
{
"type": "base64",
"media_type": "application/zip",
"data": ""
}
securitySchemes:
HTTPBearer:
type: http
scheme: bearer
````