> ## Documentation Index > Fetch the complete documentation index at: https://docs.vlm.run/llms.txt > Use this file to discover all available pages before exploring further. # Video → JSON > Generate structured prediction for the given video file. For all supported `video` domains, see the [Hub Catalog](/hub#video-domains). ```python Python (with domain) theme={"theme":{"light":"github-light","dark":"dark-plus"}} from pathlib import Path from vlmrun.client import VLMRun client = VLMRun(api_key="") response = client.video.generate( file=Path(".mp4"), domain="video.transcription", batch=True ) ``` ```python Python (with skill) theme={"theme":{"light":"github-light","dark":"dark-plus"}} from pathlib import Path from vlmrun.client import VLMRun from vlmrun.client.types import GenerationConfig, AgentSkill client = VLMRun(api_key="") response = client.video.generate( file=Path(".mp4"), batch=True, config=GenerationConfig( skills=[AgentSkill(skill_name="")] ) ) ``` ```typescript Node.js SDK (with domain) theme={"theme":{"light":"github-light","dark":"dark-plus"}} import { VlmRun } from "vlmrun"; const client = new VlmRun({apiKey: ""}); const fileResponse = await client.files.upload( filePath: ".mp4" ); const response = await client.video.generate({ fileId: fileResponse.id, domain: "video.transcription", batch: true, }); ``` ```typescript Node.js SDK (with skill) theme={"theme":{"light":"github-light","dark":"dark-plus"}} import { VlmRun } from "vlmrun"; const client = new VlmRun({apiKey: ""}); const fileResponse = await client.files.upload( filePath: ".mp4" ); const response = await client.video.generate({ fileId: fileResponse.id, config: { skills: [{ skillName: "" }], }, }); ``` ```json theme={"theme":{"light":"github-light","dark":"dark-plus"}} { "usage": { "elements_processed": 123, "element_type": "image", "credits_used": 123 }, "id": "", "created_at": "2023-11-07T05:31:56Z", "completed_at": "2023-11-07T05:31:56Z", "response": "", "status": "enqueued" } ``` Try our Colab Cookbook example for long-form video transcription. ### Example Output ```json Example Video Transcription [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}} { "metadata": { "language": null, "content": null, "topics": null, "duration": 488.56 }, "segments": [ { "start_time": 0, "end_time": 25.8, "audio": { "content": " Like the only way to find these opportunities to learn about them is to find weirdos on the internet that are also into this thing. Yes. And they're figuring it out too. And you can kind of compare notes. Yes. And this is how new industries are created. Literally. By weirdos on the internet. Like literally. Literally. This is Dalton, plus Michael, and today we're going to talk about why AI is going to create more successful founders in the world." }, "video": { "content": "Two men are engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks. The man on the right, dressed in a blue shirt, listens attentively and occasionally responds with hand gestures. They appear to be in a professional setting, possibly an office or conference room, with large windows in the background allowing natural light to fill the space." } }, { "start_time": 25.8, "end_time": 51.71, "audio": { "content": " It's interesting, as we've gotten older, we kind of see a new set of tools come into the market and then an explosion in the number of founders who can now create value. And we've seen this before, right? Like, what was the first time you saw this? I certainly noticed when the internet was new, people that knew how to build websites were suddenly able to make lots of money from" }, "video": { "content": "The video features two individuals engaged in a conversation at a table. The person on the left, wearing a light gray shirt, is facing the person on the right, who is dressed in a blue jacket over a black shirt. The background is minimalistic, with a plain wall and a window allowing natural light to enter. The text overlay on the left side of the screen reads \"AI Will Create More Successful Founders\" and \"Founder Explosion.\" On the right side, there is a list titled \"Founder Explosion\" with various items such as \"On The Cusp,\" \"Cost Of Business,\" \"Get In Early,\" \"Whatnot,\" \"Endless Opportunity,\" and \"Internet Weirdos.\" The conversation appears to be focused on the impact of artificial intelligence on business and entrepreneurship." } }, { "start_time": 51.71, "end_time": 71.89, "audio": { "content": " the skill. And it was like really basic stuff. High school kids were making tons of money. Yep. I remember people that could just figure out how to sell stuff on eBay, where you would go buy something cheap but then listed on eBay and arbitrage. Yep. Basically, you would see people that kind of understood the new tooling that came out and would like do a hustle and make ungodly amounts of money." }, "video": { "content": "The video features two men engaged in a conversation in an office setting. The man on the left, wearing a light gray button-up shirt, is actively speaking and gesturing with his hands, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively with his arms crossed. The background includes a large window with blinds partially drawn, allowing natural light to filter into the room. The conversation appears to be focused on business-related topics, as indicated by the text on the right side of the screen, which lists various themes such as 'On The Cusp,' 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' and 'Internet Weirdos.' The overall atmosphere suggests a professional discussion." } }, { "start_time": 72.01, "end_time": 92.67, "audio": { "content": " Yeah. And it was just because they understood the new tools. And I already wasn't even a hustle. Like it was a good business. Like it was, they saw that tools enabled new businesses. You know, we saw this, you know, tail end of the open source world where like we could build all of Justin TV with free software. Yep." }, "video": { "content": "The video features two men engaged in a conversation in an office setting. The man on the left, wearing a light gray shirt, is gesturing animatedly with his hands as he speaks, indicating an active discussion. The man on the right, dressed in a blue jacket over a black shirt, listens attentively with his hands clasped together on the table. The background includes a window with blinds partially open, allowing natural light to filter into the room. On the right side of the screen, there is a vertical list titled \"Founder Explosion\" with various topics such as \"On The Cusp,\" \"Cost Of Business,\" \"Get In Early,\" \"Whatnot,\" \"Endless Opportunity,\" and \"Internet Weirdos.\"" } }, { "start_time": 92.67, "end_time": 112.85, "audio": { "content": " And then we were there in the beginning of cloud compute where we didn't have to rack servers anymore. Any kid could sign up for an Amazon account, put a couple bucks down, and get access to a server. And so what's interesting is that we might, I think we feel pretty good about saying this, we might be on" }, "video": { "content": "The video features a conversation between two men seated at a table in a modern office setting. The man on the right, wearing a blue shirt and glasses, is speaking animatedly, gesturing with his hands as he discusses various topics related to entrepreneurship and business. The man on the left, dressed in a light-colored shirt, listens attentively, occasionally nodding and responding. The background includes a white wall and a window, suggesting a professional environment. On the right side of the screen, there is a list of topics being discussed, such as 'Founder Explosion,' 'On The Cusp,' 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' and 'Internet Weirdos.'" } }, { "start_time": 112.85, "end_time": 135.69, "audio": { "content": " the cusp of the next one of these. And that means there are maybe a whole bunch of new opportunities for successful businesses to be created. Yeah, starting now. Yeah, I mean, here's another metaphor. When the iPhone came out, who would have thought that Flappy Bird would have been created? And I think I read that that guy made like 20 million in cash." }, "video": { "content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is speaking animatedly, gesturing with his hands as he talks. The man on the right, dressed in a blue jacket over a black shirt, listens attentively, occasionally nodding and responding. The setting appears to be an office or meeting room with large windows in the background, allowing natural light to fill the space. The overall atmosphere suggests a professional discussion or interview." } }, { "start_time": 135.87, "end_time": 159.75, "audio": { "content": " Boom. In like two months and then shut it down. And so if you watch, okay, iPhone, Steve Jobs on stage, some guy in Southeast Asia building Flappy Bird. That's like wild. Never would have guessed. And so, again, to be very direct, what we're arguing is that when brand new technologies come out that are powerful, the people that are on the cusp of understanding them and that quickly" }, "video": { "content": "Two men are engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing animatedly with his hands as he speaks. The man on the right, dressed in a blue jacket, listens attentively, occasionally nodding and smiling. The background features a large window with a view of a body of water, suggesting an indoor setting with natural light." } }, { "start_time": 159.75, "end_time": 180.77, "audio": { "content": " build businesses or build useful things using those tools have a very unique view of creating businesses and wealth. And again, to be on the nose for AI, it seems like you can do things that would require way more headcount than you would otherwise. Yes. And so, you know, we're not even saying we know the ideas." }, "video": { "content": "The video features two men engaged in a conversation in an indoor setting. The man on the left, wearing a light gray button-up shirt, is actively gesturing with his hands as he speaks, indicating an animated discussion. The man on the right, dressed in a dark blue shirt, listens attentively, occasionally nodding and responding. The background includes a window with blinds, suggesting a modern office or studio environment. The video also displays a sidebar with various topics such as 'On The Cusp,' 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'New Is The Time,' which likely relate to the conversation's themes." } }, { "start_time": 180.97, "end_time": 201.72, "audio": { "content": " No. We're just saying if you're watching this and you're interested in being a founder or maybe not working at a company. Yeah. And you just pay attention to every new thing that comes out and try to find these opportunities or, I don't't know arbitrage is the right word, but no, just you know, new opportunities. New opportunities using these cutting edge tools" }, "video": { "content": "The video depicts a conversation between two men seated at a table in an office setting. The man on the left, wearing a light gray shirt, is gesturing animatedly with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively with his hands clasped together. The background features large windows with a view of a cityscape, and the room has a modern, minimalist design with white walls and a light-colored floor. The conversation appears to be focused and engaged, with both individuals actively participating in the dialogue." } }, { "start_time": 201.72, "end_time": 221.8, "audio": { "content": " and you're on the bleeding edge, you're not competing with anyone. No. It's green field. I think what's cool is any time one of these technologies shifts happens, the cost of starting a business, some set of businesses, reduces by up to like 10x. Yep. And so suddenly, businesses that either wouldn't have made sense" }, "video": { "content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively. The setting appears to be an office or meeting room with large windows in the background, allowing natural light to fill the space. The conversation seems to revolve around business topics, as indicated by the text on the right side of the screen, which includes phrases like 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'Now Is The Time.' The overall atmosphere suggests a professional discussion." } }, { "start_time": 221.8, "end_time": 244.96, "audio": { "content": " or certainly a normal person couldn't just stand up and do, right? Like can you imagine just, oh, it's pre online selling in eBay. All you have to do is rent a storefront and run a store, right? Like that's cheap, right? Like, absolutely not. Or like pre-Ari-NB. Like, all you have to do is just like buy a house and set up your own air bed and breakfast" }, "video": { "content": "The video features two men engaged in a conversation at a table in an office setting. The man on the left, wearing a light gray shirt, listens attentively while the man on the right, dressed in a blue jacket over a black shirt, gestures animatedly as he speaks. The background includes large windows with a view of a body of water, suggesting a modern and open environment. The conversation appears to be focused on business topics, as indicated by the text on the right side of the screen, which lists various themes such as 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'Now Is The Time.' The overall atmosphere is professional and collaborative." } }, { "start_time": 244.96, "end_time": 265.28, "audio": { "content": " bed and breakfast thing or even by hotel yeah that's crazy crazy. Whereas like Airbnb can rent a room. And think about it. Your own place. If you saw Airbnb early and you just decided to be a host and be like, oh, I should like do this as a business. You could do it pretty well. You could do it pretty well. Yeah. When Shopify was a brand new thing, like all of these platforms exactly the people that were the first to recognize that these were" }, "video": { "content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively. The background is minimalistic, with a plain wall and a window that lets in natural light. On the right side of the screen, there is a list of topics or themes, including \"Cost Of Business,\" \"Get In Early,\" \"Whatnot,\" \"Endless Opportunity,\" \"Internet Weirdos,\" and \"Now Is The Time.\" The overall setting appears to be a casual interview or discussion." } }, { "start_time": 265.84, "end_time": 286.22, "audio": { "content": " gave them leverage yes those entrepreneurial-minded people did really well. Yes. And so I think what's so cool is that what we're saying is like if you're ambitious and you're paying attention, you might not ever need to work at a big company. You might not ever need to have a boss." }, "video": { "content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing animatedly with his hands as he speaks. The man on the right, dressed in a blue jacket, listens attentively, occasionally responding with his own gestures. The setting appears to be an office or meeting room with large windows in the background, allowing natural light to fill the space. The conversation seems to revolve around business or startup topics, as indicated by the text on the right side of the screen, which includes phrases like 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'Now Is The Time.' The overall atmosphere suggests a professional and collaborative discussion." } }, { "start_time": 286.22, "end_time": 307.96, "audio": { "content": " Like you can be in control of your own destiny. And these moments don't happen every week. No. Like, we wish we did. It would be the investor. But like, when they do, the people who move. I mean, this is a very specific example. Yeah. But whatnot, the online live shopping thing, I still talk to the founders a lot." }, "video": { "content": "Two men are sitting at a table in a modern office setting, engaged in a conversation. The man on the left is wearing a light gray button-up shirt and has short, curly hair. He appears to be listening attentively to the man on the right, who is bald, wearing glasses, and dressed in a blue jacket over a black shirt. The man on the right is gesturing with his hands as he speaks, indicating an animated discussion. The background features large windows with a view of a cityscape, suggesting an urban environment. On the right side of the screen, there is a list of topics or themes related to business and entrepreneurship, such as 'Founder Explosion,' 'On The Cusp,' 'Cost Of Business,' 'Get In Early,' 'Whatnot,' 'Endless Opportunity,' 'Internet Weirdos,' and 'Now Is The Time.'" } }, { "start_time": 308.22, "end_time": 328, "audio": { "content": " And they have, I think, like, high school-aged kids selling stuff on their... Making real money, right? And making just, again, I don't even want to say the numbers. Yeah. But they figured out the format. They understand how to use whatnot. They built a user base there. Yeah. And they're basically... They're making enough money to set themselves up for their entire life." }, "video": { "content": "A man with curly hair and a light gray shirt is speaking animatedly to another man who has a shaved head and glasses. The man with curly hair uses expressive hand gestures as he talks, while the other man listens attentively. The background features a window with blinds, and there is a menu or list of topics on the right side of the screen, including \"Founder Explosion,\" \"On The Cusp,\" \"Cost Of Business,\" \"Get In Early,\" \"Whatnot,\" \"Endless Opportunity,\" \"Internet Weirdos,\" and \"Now Is The Time.\"" } }, { "start_time": 328.12, "end_time": 351.76, "audio": { "content": " Yeah. By just seeing this new platform, figuring it out, and then making a bet on it. Yes. I mean, this happened with Twitch. Happens with Twitch. Whole new industry, basically. Yeah. No, I think that what's cool is that we're also talking about every scale, right? We're talking about things that can be venture backed, maybe billion dollar companies one day. But we're also talking about things that can just" }, "video": { "content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively with his hands clasped together. The background includes a large window with a view of a body of water, suggesting an indoor setting with natural light. The conversation appears to be casual and focused, with both individuals actively participating in the dialogue." } }, { "start_time": 351.76, "end_time": 372.66, "audio": { "content": " set you up so that you can pay your rent and live a good life. Yep. The opportunities are across the entire spectrum. And I think that's what's really cool about new technology. Like when there's a real technology shift, it affects businesses across the board. We're not just talking about companies that YCP even fun. I think that the last point I'd want to make" }, "video": { "content": "A man in a blue shirt is seated at a table, engaged in a conversation with another person whose back is facing the camera. The man in the blue shirt is gesturing with his hands as he speaks, indicating an animated discussion. The setting appears to be an office or a professional environment, with a window and some furniture visible in the background. The overall atmosphere suggests a serious and focused conversation." } }, { "start_time": 372.66, "end_time": 398.57, "audio": { "content": " on this front is that you don't get this opportunity if you're just thinking, you gotta actually do. Well, and they won't teach you this in schools. Schools teach you stuff for 10 or 20 years ago. So the other thing that I've noticed in these trends is that when you are part of the history being made and you're this early on the cutting edge of a new tech coming out, you can't expect your university or your teachers or your peers people in your community" }, "video": { "content": "Two men are engaged in a conversation at a table in an office setting. The man on the left, wearing a light gray shirt, is gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket over a black shirt, listens attentively. The background features large windows with a view of a cityscape, and the room has a modern, minimalist design with white walls and a light-colored floor. The conversation appears to be focused and serious, with both individuals maintaining eye contact and using expressive hand movements." } }, { "start_time": 398.57, "end_time": 419.33, "audio": { "content": " or your peers to teach you about it. It's only basically weirdos on the internet. Yes. Like the only way to find these opportunities to learn about them is to find weirdos on the internet that are also into this thing. Yes. And they're figuring it out too. that are also into this thing. And they're figuring it out too. And you can kind of compare notes. Yes. And this is how new industries are created. Literally. By weirdos on the internet. Like literally. Literally. By weirdos on the internet. Like literally. Literally, there's like some subreddit with a bunch of weirdos." }, "video": { "content": "The video features two men engaged in a conversation at a table. The man on the left, wearing a light gray shirt, is animatedly gesturing with his hands as he speaks, while the man on the right, dressed in a blue jacket, listens attentively with his hands clasped together. The setting appears to be a modern office or conference room with large windows in the background, allowing natural light to fill the space. The conversation seems to be casual and friendly, with both individuals appearing relaxed and engaged." } }, { "start_time": 419.33, "end_time": 447.2, "audio": { "content": " And like someday from now, you know, 10 years from now, there'll be an entire industry of people that learned about this thing in some subred somewhere there. Yeah, no, I totally agree. So hey, the big takeaway is if you've been wrestling your lawyers, if you thought, oh, this isn't the time to start a new business. Maybe you should reconsider. Yeah. This is a very interesting time. I think the final argument is there's a good case where a smaller percentage of the population will need to get jobs," }, "video": { "content": "The video features two men engaged in a conversation in an office setting. The man on the left, wearing a light gray shirt, is animatedly speaking and using hand gestures to emphasize his points. He appears to be explaining something with enthusiasm. The man on the right, dressed in a blue jacket over a black shirt, listens attentively, occasionally nodding and responding. The background includes a window with blinds, suggesting a modern office environment. The conversation seems to revolve around business or technology topics, as indicated by the text overlays such as 'Cost Of Business' and 'Endless Opportunity.'" } }, { "start_time": 447.26, "end_time": 469.08, "audio": { "content": " and more people will be able to use tools like this to be self-employed in some way. I don't think there's any, I think all the structural changes imply that more folks will just use their highly leveraged selves using all these tools to run businesses, then have to go get a job. Yeah. Right? And I think that story isn't told, right? I think the story is always this kind of depressing story of like," }, "video": { "content": "The video features two men engaged in a conversation in an indoor setting. The man on the left, wearing a light gray button-up shirt, is actively gesturing with his hands as he speaks, indicating an animated discussion. The man on the right, dressed in a blue jacket over a black shirt, listens attentively, occasionally nodding and responding. The background includes a window with blinds, suggesting a modern office or conference room environment. The overall atmosphere appears to be professional and focused." } }, { "start_time": 469.08, "end_time": 488.56, "audio": { "content": " oh, maybe you won't need it, you won't be needed anymore, as opposed to here's a set of tools. You could do things that people couldn't think of doing affordably before. Like you could be your own boss. You don't even need to be inside of a company to create value. Yeah. So anyways, hopefully that's inspiring. Good shot. Thanks. good shot thanks" }, "video": { "content": "Two men are engaged in a conversation at a table in an office setting. The man on the left, wearing a light gray shirt, listens attentively while the man on the right, dressed in a blue jacket over a black shirt, speaks animatedly. He uses hand gestures to emphasize his points, occasionally clasping his hands together on the table. The background features large windows with a view of a cloudy sky, and the room is well-lit with natural light." } } ] } ``` ## OpenAPI ````yaml POST /v1/video/generate openapi: 3.1.0 info: title: VLM Run Unified Server description: Unified server for VLM Run Agent and API termsOfService: https://vlm.run/terms-of-service contact: name: VLM Run Support Team url: https://vlm.run/ email: support@vlm.run version: 2026-05-19.0 servers: [] security: [] paths: /v1/video/generate: post: tags: - video - video summary: Video Generate description: Generate structured prediction for the given video file. operationId: video_generate_v1_video_generate_post requestBody: content: application/json: schema: $ref: '#/components/schemas/VideoPredictionRequest' required: true responses: '200': description: Successful Response content: application/json: schema: {} '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' security: - HTTPBearer: [] components: schemas: VideoPredictionRequest: properties: metadata: $ref: '#/components/schemas/RequestMetadata' description: Optional metadata to pass to the model. config: $ref: '#/components/schemas/GenerationConfig' description: The VLM generation config to be used for //generate. url: anyOf: - type: string - type: 'null' title: Url description: The URL of the file (provide either `file_id` or `url`). file_id: anyOf: - type: string - type: 'null' title: File Id description: The ID of the uploaded file (provide either `file_id` or `url`). id: type: string title: Id description: Unique identifier of the request. created_at: type: string format: date-time title: Created At description: Date and time when the request was created (in UTC timezone) callback_url: anyOf: - type: string minLength: 1 format: uri - type: 'null' title: Callback Url description: The URL to call when the request is completed. model: anyOf: - type: string enum: - vlm-1 - vlm-1:auto - vlm-1:fast - vlm-1:pro - type: string title: Model description: The model to use for generating the response. default: vlm-1 domain: anyOf: - type: string enum: - video.transcription - video.transcription-summary - video.summary - type: string - type: 'null' title: Domain description: >- The domain identifier for the model (e.g. `video.transcription`). Optional when a skill is provided via config.skills. batch: type: boolean title: Batch description: Whether to process the document in batch mode (async). default: true type: object title: VideoPredictionRequest description: Request to the Video API (i.e. structured prediction). HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError RequestMetadata: properties: environment: type: string enum: - dev - staging - prod title: Environment description: The environment where the request was made. default: dev session_id: anyOf: - type: string - type: 'null' title: Session Id description: The session ID of the request allow_logging: type: boolean title: Allow Logging description: Whether to enable logs for this request. default: true allow_training: type: boolean title: Allow Training description: Whether the file can be used for training default: true allow_retention: type: boolean title: Allow Retention description: Whether to allow retention of the data default: true extra: additionalProperties: true type: object title: Extra description: Extra metadata for the request (e.g. `dataset_id`, `subset_id`). type: object title: RequestMetadata description: >- Metadata for the request. Typically captured in {"vlmrun": {"metadata": {"environment": , ...}}. GenerationConfig: properties: prompt: anyOf: - type: string - type: 'null' title: Prompt description: >- Additional user instructions appended to the application or skill prompt for this request. detail: type: string enum: - auto - hi - lo title: Detail description: The detail level to use for processing multimodal data. default: auto json_schema: anyOf: - additionalProperties: true type: object - type: 'null' title: Json Schema description: >- The overridden JSON schema to use for the model. To be used instead of the response model. skills: anyOf: - items: $ref: '#/components/schemas/AgentSkill' type: array - type: 'null' title: Skills description: List of agent skills to enable for this generation request. gql_stmt: anyOf: - type: string - type: 'null' title: Gql Stmt description: >- The GraphQL statement to use for the application. If provided, the response model will be generated from the GraphQL statement. max_retries: type: integer title: Max Retries description: The maximum number of retries to use for the application. default: 1 max_tokens: type: integer title: Max Tokens description: The maximum number of tokens to use for the application. default: 65535 temperature: type: number title: Temperature description: The temperature to use for the application. default: 0 confidence: type: boolean title: Confidence description: >- Include confidence scores in the response (included in the `_metadata` field). default: false grounding: type: boolean title: Grounding description: >- Include grounding in the response (included in the `_metadata` field). default: false keyframes: type: boolean title: Keyframes description: Include keyframes in the video transcription response. default: false video_segment_duration: anyOf: - type: number minimum: 1 - type: 'null' title: Video Segment Duration description: >- Duration in seconds for each video segment when chunking a video for transcription. Defaults to 150.0s. video_frames_per_segment: anyOf: - type: integer minimum: 1 - type: 'null' title: Video Frames Per Segment description: >- Number of frames to sample per video segment for captioning. Defaults to 8. video_model: anyOf: - type: string - type: 'null' title: Video Model description: >- Model ID to use for video segment captioning (e.g. 'vlmrun-orion-1:fast'). When omitted, the server default is used. video_input_mode: anyOf: - type: string enum: - frames - native_video - type: 'null' title: Video Input Mode description: >- How to pass video to the captioning model: 'frames' extracts N JPEG frames per segment, 'native_video' sends the mp4 clip directly via video_url for models with native video understanding. Defaults to 'native_video' for Qwen deployment models, 'frames' for others. video_transcribe_audio: type: boolean title: Video Transcribe Audio description: >- When True, transcribe the audio track to align segment boundaries. When False (default), skip ASR and use fixed-duration video segments only (visual-only captioning). default: false chat_context: anyOf: - type: string - type: 'null' title: Chat Context description: >- Plain-text chat transcript (prior turns + current request) used to ground video captioning / transcription on what the user wants extracted. page_indices: anyOf: - items: type: integer type: array - type: 'null' title: Page Indices description: >- 0-indexed page indices to process for document files. If None, all pages are processed. use_context_cache: type: boolean title: Use Context Cache description: >- Reuse cached representations of document/video content across calls. When True (default), the file is cached after the first call so repeated queries against the same file skip re-transmitting its contents. Set to False to always send the full content. default: true service_tier: anyOf: - type: string enum: - auto - default - standard - flex - priority - type: 'null' title: Service Tier description: >- Delivery tier for the request. 'standard'/'default' uses baseline rates, 'flex' applies a 50% discount with higher latency, 'priority' applies a 1.8x premium. When omitted (or 'auto'), the server default ('standard') applies. The chosen tier drives both billing and the latency/availability SLO. type: object title: GenerationConfig description: Request configuration for image/document/video generation. ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context type: object required: - loc - msg - type title: ValidationError AgentSkill: properties: type: type: string title: Type description: >- The type of the skill. Use 'skill_reference' for DB-stored skills referenced by id/name. Use 'inline' to provide the skill as a base64-encoded zip bundle. default: skill_reference skill_id: anyOf: - type: string - type: 'null' title: Skill Id description: >- The unique identifier of the skill — a UUID or a name string (e.g., 'pillow', 'batch-processing'). skill_name: anyOf: - type: string - type: 'null' title: Skill Name description: >- Human-readable skill name for lookup (e.g., 'invoice-extraction'). Alternative to skill_id. Deprecated in favour of skill_id. skill_version: anyOf: - type: integer - type: string title: Skill Version description: The version of the skill — an integer (e.g. 2) or 'latest'. default: latest version: anyOf: - type: integer - type: string - type: 'null' title: Version description: 'DEPRECATED: Use ''skill_version'' instead. The version of the skill.' name: anyOf: - type: string - type: 'null' title: Name description: >- Human-readable name for the inline skill (used for discovery and logging). description: anyOf: - type: string - type: 'null' title: Description description: Short description of what the inline skill does. source: anyOf: - $ref: '#/components/schemas/InlineSkillSource' - type: 'null' description: >- Source payload for inline skills. Contains the base64-encoded zip bundle with type, media_type, and data fields. bundle: anyOf: - type: string - type: 'null' title: Bundle description: >- DEPRECATED: Use 'source.data' instead. Base64-encoded zip bundle containing the skill files (inline skills only). type: object title: AgentSkill description: >- A modular capability that extends the agent's functionality. Agent Skills are reusable, filesystem-based resources that provide the agent with domain-specific expertise: workflows, context, and best practices. Each skill packages instructions, metadata, and optional resources (scripts, templates, snippets) that the agent uses automatically when relevant. Two modes are supported: 1. **Referenced skills** (``type="skill_reference"``) – Provide ``skill_id`` (UUID or name) and optionally ``skill_version`` (integer or ``"latest"``). .. code-block:: json {"type": "skill_reference", "skill_id": "pillow", "skill_version": "latest"} 2. **Inline skills** (``type="inline"``) – Supply ``name``, ``description``, and a ``source`` object containing the base64-encoded zip bundle. The zip must contain exactly one ``SKILL.md`` file. No database lookup is required. .. code-block:: json { "type": "inline", "name": "csv-insights", "description": "Summarize CSV files.", "source": { "type": "base64", "media_type": "application/zip", "data": "" } } Legacy format with flat ``bundle`` field is also accepted for backward compatibility. InlineSkillSource: properties: type: type: string const: base64 title: Type description: >- Encoding type for the inline skill data. Currently only 'base64' is supported. default: base64 media_type: type: string title: Media Type description: MIME type of the skill bundle. Must be 'application/zip'. default: application/zip data: type: string title: Data description: Base64-encoded zip bundle containing the skill files. type: object required: - data title: InlineSkillSource description: |- Source payload for an inline skill bundle. Follows the OpenAI inline skill format:: { "type": "base64", "media_type": "application/zip", "data": "" } securitySchemes: HTTPBearer: type: http scheme: bearer ````