System One

POST /typesafe/v1/systemone answers typed questions about a state (text, JSON, or images) and returns a calibrated probability for every answer. Nothing is generated and nothing is parsed, so an answer can never be off-schema. The route is wire-compatible with TypeSafe’s Jev API. Point the typesafe-sdk client at https://gateway.vlm.run/typesafe and it works unchanged. See TypeSafe SDK Compatibility.

Core philosophy

World knowledge, not a trained classifier: the engines are open-weight VLMs, so a question can name any concept they already know. There is no dataset to collect, no model to fine-tune and nothing to redeploy when the label set changes: edit the criteria and the next read answers the new question.
Under 200 ms, with the shape guaranteed: a read is one forward pass, not a decode and a parse, and the budget is held there on purpose. Over a socket the measured p50 is ~44 ms. The answer is read off the model rather than written by it, so it is always one of your labels, with a probability attached, and never a string you have to parse or repair.
Cheap enough to stop rationing: a read is a fraction of a cent, so the right instinct is to point it at every visual question you have: gate an expensive pipeline, route a document, label a frame, sanity-check an upload. The cost of asking should never be the reason you did not.

Quickstart

Ask a scanned invoice page three typed questions in one call. The image goes in as an image_url part carrying a base64 data URL, with detail set to high because the amounts are small print.

import base64, pathlib
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

client = TypeSafeClient(
    api_key="<VLMRUN_API_KEY>",
    base_url="https://gateway.vlm.run/typesafe",
)

page = base64.b64encode(pathlib.Path("invoice-page.jpg").read_bytes()).decode()

result = client.system_one(
    "Answer about the attached page.",
    {
        "kind": Choice(
            instructions="What kind of document is this page from?",
            criteria={"invoice": None, "receipt": None, "contract": None, "other": None},
        ),
        "has_total": Noul(instructions="Is a total amount due visible on the page?"),
        "legibility": Score(
            instructions="How legible is the text?",
            criteria=["unreadable", "partly legible", "clear"],
        ),
    },
    extra_body={
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": f"data:image/jpeg;base64,{page}",
                    "detail": "high",
                },
            }
        ]
    },
    model="google/diffusiongemma-26b-a4b-it",
)

print(result.choices["kind"].choice, result.nouls["has_total"].noul)

import { readFileSync } from "node:fs";
import { choice, noul, score, TypeSafeClient } from "@typesafe-ai/sdk";

const client = new TypeSafeClient({
  apiKey: process.env.VLMRUN_API_KEY,
  baseURL: "https://gateway.vlm.run/typesafe",
  defaultModel: "google/diffusiongemma-26b-a4b-it",
});

const page = readFileSync("invoice-page.jpg").toString("base64");

// Extra fields such as `content` are forwarded from a request variable.
// An inline object literal would trip TypeScript's excess-property check.
const request = {
  state: "Answer about the attached page.",
  questions: {
    kind: choice("What kind of document is this page from?", {
      invoice: null,
      receipt: null,
      contract: null,
      other: null,
    }),
    has_total: noul("Is a total amount due visible on the page?"),
    legibility: score("How legible is the text?", [
      "unreadable",
      "partly legible",
      "clear",
    ]),
  },
  content: [
    { type: "image_url",
      image_url: { url: `data:image/jpeg;base64,${page}`, detail: "high" } },
  ],
};

const { answers } = await client.systemOne(request);
console.log(answers.kind.choice, answers.has_total.noul);

vlmrun gw s1 -s "Answer about the attached page." invoice-page.jpg --detail high \
  -m google/diffusiongemma-26b-a4b-it \
  --choice kind="invoice|receipt|contract|other" \
  --noul has_total="Is a total amount due visible on the page?" \
  --score legibility="unreadable|partly legible|clear"

IMAGE=$(base64 -w0 invoice-page.jpg)

curl https://gateway.vlm.run/typesafe/v1/systemone \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d @- <<JSON
{
  "model": "google/diffusiongemma-26b-a4b-it",
  "state": "Answer about the attached page.",
  "content": [
    {"type": "image_url",
     "image_url": {"url": "data:image/jpeg;base64,$IMAGE", "detail": "high"}}
  ],
  "questions": {
    "kind": {
      "type": "choice",
      "instructions": "What kind of document is this page from?",
      "criteria": {"invoice": null, "receipt": null, "contract": null, "other": null}
    },
    "has_total": {"type": "noul", "instructions": "Is a total amount due visible on the page?"},
    "legibility": {
      "type": "score",
      "instructions": "How legible is the text?",
      "criteria": ["unreadable", "partly legible", "clear"]
    }
  }
}
JSON

Response

{
  "model": "google/diffusiongemma-26b-a4b-it",
  "answers": {
    "kind": {
      "type": "choice",
      "choice": "invoice",
      "probabilities": { "invoice": 0.88, "receipt": 0.07, "contract": 0.02, "other": 0.03 },
      "confidence": 0.64
    },
    "has_total": { "type": "noul", "noul": 0.91 },
    "legibility": {
      "type": "score",
      "score": 1.74,
      "legend": { "0": "unreadable", "1": "partly legible", "2": "clear" },
      "probabilities": { "0": 0.02, "1": 0.22, "2": 0.76 },
      "confidence": 0.47
    }
  },
  "usage": {
    "input_tokens": 486,
    "output_tokens": 0,
    "input_tokens_details": { "cached_tokens": 0, "image_tokens": 280, "text_tokens": 206 },
    "reads": 1,
    "cost": 0.000129
  }
}

Models

Several engines answer these questions, each named explicitly, and a decision reports the id that answered it. See Models.

When to use a read

Ask	Use	Why
”Is this scanned page an invoice?”	System One	One calibrated probability, never off-schema.
”Which of these 6 kinds is this scan?”	System One	Full distribution over labels, plus a confidence.
”Should I run the expensive extraction pass?”	System One	Cheap gate ahead of OCR or structured extraction.
”Transcribe this page to markdown”	Chat Completions	The answer is free-form text, not a label.
”Summarize this video”	Chat Completions	Generation, not a decision.

A read costs the prompt prefill plus one forward pass. Generating the same answer pays a full decode plus parsing, and the parse can still fail.

Question types

Three types, each answering a different shape of question.

`type`	Ask it when	You get back
`noul`	The answer is yes or no.	A probability that the answer is yes.
`choice`	The answer is one label from a fixed set.	The winning label, a probability for every label, and a confidence.
`score`	The answer is a level on an ordered scale.	The expected level, a legend, per-level probabilities, and a confidence.

A score answer is the expected level, so a 3-level question returns a number from 0 to 2 rather than a bucket. For the exact criteria each type takes and its limits, see the request reference.

Confidence

confidence runs from 1 (the model puts all its mass on one label) to 0 (the distribution is uniform). Threshold on it to decide when a human, or a larger model, should look. Document Classification works a full routing example.

Probabilities are the model’s own at temperature 1, not a parsed guess. They are still model-specific: revalidate your thresholds when you change models.

Inputs

Media rides in content, a list of content parts. A part names its own kind, so one list carries images, a document, and text together.

Content part	Limit	Notes
`image_url`	8 per request, 5 MB each	JPEG, PNG, WebP, or GIF, as a `data:` URL.
`file` or `document_url`	1 document per request	A PDF. Its first 8 pages are rasterised for you at 96 DPI.
`text`	any	Read after the `state`.

Media leads, the state follows. A bare string in content is one text part.

Send images or one document, never both. A document’s pages become the read’s images, so the two compete for the same 8 slots. Sending both is a 422.

No roles, no message array. A read denoises a single canvas over a single state, so there is no turn to address and no history to carry.

detail decides how closely an input is read. The default is auto, which reads at full fidelity: 280 vision tokens. Set "detail": "low" to read at 70 instead, which is 4x fewer billed image tokens at the same latency and is usually enough for classifying or routing. The cap applies per request, so one high or auto part lifts the whole read. For the accepted part shapes and the full detail table, see the request reference.

Classify a PDF

The route rasterises the document itself, so one call gets you the same typed answers you would get for images. Send the PDF as a file part, with file_data set to either a base64 data URL as below or an http(s) URL.

import base64, pathlib
from typesafe_sdk import Choice, Noul, TypeSafeClient

client = TypeSafeClient(
    api_key="<VLMRUN_API_KEY>",
    base_url="https://gateway.vlm.run/typesafe",
)

pdf = base64.b64encode(pathlib.Path("invoice.pdf").read_bytes()).decode()

result = client.system_one(
    "Answer about the document.",
    {
        "kind": Choice(
            instructions="What kind of document is this?",
            criteria={"invoice": None, "contract": None, "report": None, "other": None},
        ),
        "has_total": Noul(instructions="Is a total amount due visible?"),
    },
    extra_body={
        "content": [
            {
                "type": "file",
                "file": {
                    "filename": "invoice.pdf",
                    "file_data": f"data:application/pdf;base64,{pdf}",
                    "detail": "high",
                },
            }
        ]
    },
    model="google/diffusiongemma-26b-a4b-it",
)

print(result.choices["kind"].choice)

import { readFileSync } from "node:fs";
import { choice, noul, TypeSafeClient } from "@typesafe-ai/sdk";

const client = new TypeSafeClient({
  apiKey: process.env.VLMRUN_API_KEY,
  baseURL: "https://gateway.vlm.run/typesafe",
  defaultModel: "google/diffusiongemma-26b-a4b-it",
});

const pdf = readFileSync("invoice.pdf").toString("base64");

const request = {
  state: "Answer about the document.",
  questions: {
    kind: choice("What kind of document is this?", {
      invoice: null,
      contract: null,
      report: null,
      other: null,
    }),
    has_total: noul("Is a total amount due visible?"),
  },
  content: [
    { type: "file",
      file: { filename: "invoice.pdf",
              file_data: `data:application/pdf;base64,${pdf}`,
              detail: "high" } },
  ],
};

const { answers } = await client.systemOne(request);
console.log(answers.kind.choice);

vlmrun gw s1 -s "Answer about the document." invoice.pdf --detail high \
  -m google/diffusiongemma-26b-a4b-it \
  --choice kind="invoice|contract|report|other" \
  --noul has_total="Is a total amount due visible?"

PDF=$(base64 -w0 invoice.pdf)

curl https://gateway.vlm.run/typesafe/v1/systemone \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d @- <<JSON
{
  "model": "google/diffusiongemma-26b-a4b-it",
  "state": "Answer about the document.",
  "content": [
    {"type": "file",
     "file": {"filename": "invoice.pdf",
              "file_data": "data:application/pdf;base64,$PDF",
              "detail": "high"}}
  ],
  "questions": {
    "kind": {
      "type": "choice",
      "instructions": "What kind of document is this?",
      "criteria": {"invoice": null, "contract": null, "report": null, "other": null}
    },
    "has_total": {"type": "noul", "instructions": "Is a total amount due visible?"}
  }
}
JSON

A PDF longer than 8 pages is answered from its opening pages, which is what classifying or routing one needs. Reading every page of a long document is a job for the Gateway’s per-page document pipeline, not for a decision.

Streaming

WS /typesafe/ws answers the same questions per frame instead of per request, for a caller reading a video or an image stream. The questions are sent once, each frame is raw binary, and every decision carries the same answers and usage as a posted read. See Streaming.

Gateway extensions

Three optional fields go beyond Jev’s contract.

Field	Applies to	Notes
`reasoning_effort`	Generative engines	`none` (default), `minimal`, or `low`: 0, 32, or 64 tokens of reasoning before the answer.
`steps`	Diffusion engines	Denoise steps per read, 1 to 8. Default `1`, which is Jev’s contract.
`samples`	Any	Noise draws to average, 1 to 32. Every draw is billed.

reasoning_effort is off by default on purpose: a read is a prefill, and thinking is the one thing that turns it into a generation. medium and high (128 and 256 tokens) are declared but return a 422 until an engine is fast enough to hold the latency target under them. A diffusion engine rejects any reasoning budget with a 422. See the request reference for ranges and defaults. Unknown fields are rejected with a 422, so a typo is never silently ignored.

Cost levers, in order of effect

Lever	Effect
Read, do not generate	One forward pass instead of a full decode plus a parse.
Ask the same questions of many states	The schema is prefix-cached, so a fixed question set pays its prefill once per replica.
Batch questions into one request	Around 20 short questions fit one read; more are grouped and read concurrently, one prefill each.
Set `detail: "low"` where the question allows	70 vision tokens instead of the default 280, so 4x fewer billed image tokens at the same latency.
Leave `samples` unset	Averaging over noise draws multiplies cost without steadying a borderline answer. Sharpen `instructions` instead.

Next steps

System One API

Full request and response reference, plus error envelopes.

TypeSafe SDK Compatibility

Point an existing Jev client at the Gateway with three environment variables.

Chat Completions

OCR, VQA, and document inference when you need generated text.

Rate Limits

Per-tier quotas and how they apply to a read.

Streaming

WS /typesafe/ws, for per-frame decisions over video or an image stream.

CLI

vlmrun gw s1 for one-off reads, gates that guard a script, and dry runs.

Get Started

OpenAI-Compatible API

TypeSafe-Compatible API

Guides

API Reference

Tools

Platform

Core philosophy

Quickstart

Models

When to use a read

Question types

Confidence

Inputs

Classify a PDF

Streaming

Gateway extensions

Cost levers, in order of effect

Next steps

System One API

TypeSafe SDK Compatibility

Chat Completions

Rate Limits

Streaming

CLI

​Core philosophy

​Quickstart

​Models

​When to use a read

​Question types

​Confidence

​Inputs

​Classify a PDF

​Streaming

​Gateway extensions

​Cost levers, in order of effect

​Next steps

System One API

TypeSafe SDK Compatibility

Chat Completions

Rate Limits

Streaming

CLI

Core philosophy

Quickstart

Models

When to use a read

Question types

Confidence

Inputs

Classify a PDF

Streaming

Gateway extensions

Cost levers, in order of effect

Next steps