> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenAI Compatibility

> Run VLM Run Agents with the OpenAI Python SDK with just 2 lines of code change.

With our OpenAI-compatible API, you can use the [OpenAI Python SDK](https://github.com/openai/openai-python) to interact with VLM Run Agents. This allows developers to trivially switch between OpenAI and VLM Run APIs without having to change any code.

<Tip>Our VLM Agents are fully compatible with the OpenAI API. Notably, our API also supports a whole range of features with multi-modal data types that OpenAI currently does not support. Our OpenAI-Compatible endpoint is available at `https://api.vlm.run/v1/openai`.</Tip>

In order to use the VLM Run Agents API, you simply need to override the default endpoint and API key when using the OpenAI Python SDK. The API key can be found in the [API Keys](https://app.vlm.run/dashboard/settings/api-keys) page of the VLM Run dashboard.

## OpenAI Client Configuration

Override the default endpoint and API key by initializing the OpenAI client with the following configuration:

<CodeGroup>
  ```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import openai

  client = openai.OpenAI(
      base_url="https://api.vlm.run/v1/openai",
      api_key="<VLMRUN_API_KEY>"
  )
  ```

  ```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import { OpenAI } from "openai";

  const client = new OpenAI({
      baseURL: "https://api.vlm.run/v1/openai",
      apiKey: "<VLMRUN_API_KEY>"
  });
  ```
</CodeGroup>

Alternatively, you can also set the following environment variables to achieve the same effect:

```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
export OPENAI_API_BASE="https://api.vlm.run/v1/openai"
export OPENAI_API_KEY="<VLMRUN_API_KEY>"
```

### Usage 1: Basic Chat Completion

Once you have set the endpoint and API key, you can use the OpenAI Python SDK as you normally would. Note that the only change required to the `client.chat.completions.create` method is the `extra_body` field that allows you to specify the `domain` and additional request `metadata`.

For example:

<CodeGroup>
  ```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import openai

  # Initialize the OpenAI client
  client = openai.OpenAI(
      base_url="https://api.vlm.run/v1/openai", api_key="<VLMRUN_API_KEY>"
  )

  # Example: Chat completion with an image input
  messages = [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": [
          {"type": "text", "text": "What's in this image?"},
          {"type": "image_url", "image_url": {"url": "<image_url>", "detail": "auto"}},
      ]}
  ]

  # Perform chat completion
  chat_completion = client.chat.completions.create(
      model="vlmrun-orion-1:auto",
      messages=messages,
      temperature=0,
      extra_body={"session_id": "<UUID>"},  # optional session id for persistence
  )
  print(chat_completion.choices[0].message.content)
  ```

  ```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import { OpenAI } from "openai";

  // Initialize the OpenAI client
  const client = new OpenAI({
    baseURL: "https://api.vlm.run/v1/openai",
    apiKey: "<VLMRUN_API_KEY>"
  });

  // Example: Chat completion with an image input
  const messages = [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: [
      { type: "text", text: "What's in this image?" },
      { type: "image_url", image_url: { url: "<image_url>", detail: "auto" } }
    ]}
  ];

  // Perform chat completion
  const chatCompletion = await client.chat.completions.create({
    model: "vlmrun-orion-1:auto",
    messages: messages,
    temperature: 0,
    extra_body: { session_id: "<session id>" }
  });
  console.log(chatCompletion.choices[0].message.content);
  ```
</CodeGroup>

### Usage 2: Chat Completion with Structured Outputs

<CodeGroup>
  ```python Python expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import openai
  from pydantic import BaseModel, Field

  class ImageCaption(BaseModel):
      caption: str = Field(..., description="Detailed caption of the scene")
      tags: list[str] = Field(..., description="Tags that describe the image")

  # Initialize the OpenAI client
  client = openai.OpenAI(
      base_url="https://api.vlm.run/v1/openai", api_key="<VLMRUN_API_KEY>"
  )

  # Example: Chat completion with an image input
  messages = [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": [
          {"type": "text", "text": "What's in this image?"},
          {"type": "image_url", "image_url": {"url": "<image_url>", "detail": "auto"}},
      ]}
  ]

  # Perform chat completion (with JSON Schema)
  chat_completion = client.chat.completions.create(
      model="vlmrun-orion-1:auto",
      messages=messages,
      response_format={"type": "json_schema", "schema": ImageCaption.model_json_schema()},
  )
  print(chat_completion.choices[0].message.content)
  # >> {"caption": "...", "tags": [...]}
  print(ImageCaption.model_validate_json(chat_completion.choices[0].message.content))
  # >> ImageCaption(caption="...", tags=[...])
  ```

  ```typescript Node.js expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import { OpenAI } from "openai";
  import { z } from "zod";
  import { zodToJsonSchema } from "zod-to-json-schema";

  // Define the schema with Zod
  const ImageCaptionSchema = z.object({
    caption: z.string().describe("Detailed caption of the scene"),
    tags: z.array(z.string()).describe("Tags that describe the image")
  });

  // Initialize the OpenAI client
  const client = new OpenAI({
    baseURL: "https://api.vlm.run/v1/openai",
    apiKey: "<VLMRUN_API_KEY>"
  });

  // Example: Chat completion with an image input
  const messages = [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: [
      { type: "text", text: "What's in this image?" },
      { type: "image_url", image_url: { url: "<image_url>", detail: "auto" } }
    ]}
  ];

  // Perform chat completion (with JSON Schema)
  const chatCompletion = await client.chat.completions.create({
    model: "vlmrun-orion-1:auto",
    messages: messages,
    response_format: {
      type: "json_schema",
      schema: zodToJsonSchema(ImageCaptionSchema)
    }
  });

  // Validate the response with Zod
  const result = ImageCaptionSchema.parse(JSON.parse(chatCompletion.choices[0].message.content));
  console.log(result);
  // >> { caption: "...", tags: [...] }
  ```

  ```python Instructor theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import instructor
  from openai import OpenAI
  from pydantic import BaseModel, Field

  class ImageCaption(BaseModel):
      caption: str = Field(..., description="Detailed caption of the scene")
      tags: list[str] = Field(..., description="Tags that describe the image")

  # Initialize the Instructor client (OpenAI)
  client = instructor.from_openai(OpenAI(
      base_url="https://api.vlm.run/v1/openai", api_key="<VLMRUN_API_KEY>"
  ))

  # Define the messages
  messages = [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": [
          {"type": "text", "text": "What's in this image?"},
          {"type": "image_url", "image_url": {"url": "<image_url>", "detail": "auto"}}
      ]}
  ]

  response = client.chat.completions.create(
      model="vlmrun-orion-1:auto",
      messages=messages,
      response_model=ImageCaption,
  )
  print(response)
  # >>> ImageCaption(caption="...", tags=[...])
  ```
</CodeGroup>

### Usage 3: Basic Chat Completion with Streaming

<CodeGroup>
  ```python Python expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import openai

  # Initialize the OpenAI client
  client = openai.OpenAI(
      base_url="https://api.vlm.run/v1/openai", api_key="<VLMRUN_API_KEY>"
  )

  # Define the messages
  messages = [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": [
          {"type": "text", "text": "What's in this image?"},
          {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg", "detail": "auto"}}
      ]}
  ]

  # Perform chat completion (with streaming)
  chat_completion = client.chat.completions.create(
      model="vlmrun-orion-1:auto",
      messages=messages,
      temperature=0,
      stream=True,
  )
  for chunk in chat_completion:
      if chunk.choices[0].delta.content:
          print(chunk.choices[0].delta.content, end="", flush=True)
  ```

  ```typescript Node.js expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import { OpenAI } from "openai";

  // Initialize the OpenAI client
  const client = new OpenAI({
    baseURL: "https://api.vlm.run/v1/openai",
    apiKey: "<VLMRUN_API_KEY>"
  });

  // Define the messages
  const messages = [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: [
      { type: "text", text: "What's in this image?" },
      { type: "image_url", image_url: { url: "https://example.com/image.jpg", detail: "auto" } }
    ]}
  ];

  // Perform chat completion (with streaming)
  const stream = await client.chat.completions.create({
    model: "vlmrun-orion-1:auto",
    messages: messages,
    temperature: 0,
    stream: true
  });

  for await (const chunk of stream) {
    if (chunk.choices[0]?.delta?.content) {
      process.stdout.write(chunk.choices[0].delta.content);
    }
  }
  ```
</CodeGroup>

### Usage 4: Mixed-Modality Inputs

<CodeGroup>
  ```python Python expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  from pathlib import Path
  import openai
  from vlmrun.client import VLMRun

  # Initialize the VLM Run client for file uploads
  vlmrun_client = VLMRun(api_key="<VLMRUN_API_KEY>")

  # Upload an image and document to the VLM Run Agents API
  file1 = vlmrun_client.files.upload(file=Path("image.jpg"), purpose="assistants")
  file2 = vlmrun_client.files.upload(file=Path("document.pdf"), purpose="assistants")

  # Initialize the OpenAI client
  client = openai.OpenAI(
      base_url="https://api.vlm.run/v1/openai", api_key="<VLMRUN_API_KEY>"
  )

  # Perform chat completion (with mixed-modality inputs)
  chat_completion = client.chat.completions.create(
      model="vlmrun-orion-1:auto",
      messages=[
          {"role": "user", "content": [
              {"type": "text", "text": "What's in this image and document?"},
              {"type": "input_file", "file_id": file1.id},
              {"type": "input_file", "file_id": file2.id},
          ]}
      ],
  )
  ```

  ```typescript Node.js expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import { OpenAI } from "openai";
  import fs from "fs";

  // Initialize the OpenAI client
  const client = new OpenAI({
    baseURL: "https://api.vlm.run/v1/openai",
    apiKey: "<VLMRUN_API_KEY>"
  });

  // Upload an image and document to the VLM Run Agents API
  const file1 = await client.files.create({
    file: fs.createReadStream("image.jpg"),
    purpose: "assistants"
  });
  const file2 = await client.files.create({
    file: fs.createReadStream("document.pdf"),
    purpose: "assistants"
  });

  // Perform chat completion (with mixed-modality inputs)
  const chatCompletion = await client.chat.completions.create({
    model: "vlmrun-orion-1:auto",
    messages: [
      { role: "user", content: [
        { type: "text", text: "What's in this image and document?" },
        { type: "input_file", file_id: file1.id },
        { type: "input_file", file_id: file2.id }
      ]}
    ]
  });
  ```
</CodeGroup>

## Extra Body

The `extra_body` field allows you to specify additional request metadata that is used by the VLM Run Agents API (outside of the OpenAI Python SDK), as indicated by the `vlmrun` field. This metadata is used to specify other request metadata such as `allow_training`, `environment` etc.

For example, the following code specifies the request `metadata`:

<CodeGroup>
  ```python Python expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import openai

  client = openai.OpenAI(
      base_url="https://api.vlm.run/v1/openai", api_key="<VLMRUN_API_KEY>"
  )

  chat_completion = client.chat.completions.create(
      model="vlmrun-orion-1:auto",
      messages=[
        { role: "user", content: "What's the capital of France?" }
      ],
      temperature=0,
      extra_body={
          "vlmrun": {
              "metadata": {
                  "environment": "dev",
                  "allow_retention": False,
              },
          }
      }
  )
  ```

  ```typescript Node.js expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import { OpenAI } from "openai";

  const client = new OpenAI({
    baseURL: "https://api.vlm.run/v1/openai",
    apiKey: "<VLMRUN_API_KEY>"
  });

  const chatCompletion = await client.chat.completions.create({
    model: "vlmrun-orion-1:auto",
    messages: [
      { role: "user", content: "What's the capital of France?" }
    ],
    temperature: 0,
    extra_body: {
      vlmrun: {
        metadata: {
          environment: "dev",
          allow_retention: false
        }
      }
    }
  });
  ```
</CodeGroup>

## Request Metadata

<Tip>
  For more details on the request metadata, please refer to the [Request Metadata](/api-reference/v1/post-agent-execute#body-metadata) section of the API reference.
</Tip>

Currently, the VLM Run Agents API supports submitting request metadata along with the chat completions request via the `extra_body` keyword argument. For example, the VLM Run Agents API accepts the following request metadata:

<Accordion title="Request Metadata Fields">
  The VLM Agents API supports the following request `vlmrun.metadata` fields.

  * `environment` (`dev`, `staging`, `prod`): This property specifies the environment in which the request is being made. This can be useful for tracking requests across different environments. By default, this property is set to `prod`.
  * `session_id`: This property is a string UUID for the session, which can be used to track requests across different sessions. This property is required and must be a valid UUID (36 characters long).
  * `allow_training`: This property flags the request as a potential candidate for our training dataset. If set to `true`, the request may be used for training our base models. If set to `false`, the request will be used for inference only. By default, this property is set to `true`.
  * `allow_retention`: This property flags the request as a potential candidate for our retention dataset. If set to `true`, the request may be used for retention of the data. If set to `false`, the request will be used for inference only. By default, this property is set to `true`.
  * `allow_logging`: This property flags the request as a potential candidate for our logging dataset. If set to `true`, the request may be used for logging of the data. If set to `false`, the request will be used for inference only. By default, this property is set to `true`.
  * `extra`: This property is a dictionary of extra metadata that can be used to track the request.
</Accordion>

<CodeGroup>
  ```python Python expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import openai

  client = openai.OpenAI(
      base_url="https://api.vlm.run/v1/openai", api_key="<VLMRUN_API_KEY>"
  )

  chat_completion = client.chat.completions.create(
      model="vlmrun-orion-1:auto",
      messages=[
        { role: "user", content: "What's the capital of France?" }
      ],
      temperature=0,
      extra_body={
          "vlmrun": {
              "domain": "...",
              "metadata": {
                  "environment": "dev",
                  "session_id": "...",
                  "allow_training": False,
              }
          }
      }
  )
  ```

  ```typescript Node.js expandable theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import { OpenAI } from "openai";

  const client = new OpenAI({
    baseURL: "https://api.vlm.run/v1/openai",
    apiKey: "<VLMRUN_API_KEY>"
  });

  const chatCompletion = await client.chat.completions.create({
    model: "vlmrun-orion-1:auto",
    messages: [
      { role: "user", content: "What's the capital of France?" }
    ],
    temperature: 0,
    extra_body: {
      vlmrun: {
        domain: "...",
        metadata: {
          environment: "dev",
          session_id: "...",
          allow_training: false
        }
      }
    }
  });
  ```
</CodeGroup>

## Token Usage

The OpenAI Python SDK provides usage statistics for your account on every API call. This can be useful for monitoring your usage and costs when using the API. We refer the user to the [VLM Run Pricing](https://vlm.run/#pricing) page for more information on pricing and usage.

## Compatibility Differences

Unlike the OpenAI API, the VLM Run Agents API adds support to the following fields:

* messages can now contain `input_file` objects `{"type": "input_file", "file_id": "<file_id>"}` where `file_id` is the id of the file uploaded to the VLM Run Agents API. These are especially useful for processing large files such as videos, images, etc.
* `max_tokens`: The `max_tokens` field in `chat.completions.create` is currently not respected by our server. This means that in case the token outputs exceed the limit, the server will still return the full output.
* `logprobs`, `logit_bias`, `top_logprobs`, `presence_penalty`, `frequency_penalty`, `n`, `stream`, `stop`: These fields are not currently supported by the VLM Run Agents API. We will be adding support for these features in the near future.