Field Extraction

1. Pick an approach

	System One fields	Chat Completions with `json_schema`	Chat Completions, schema in the prompt
Surface	`POST /typesafe/v1/systemone`	`POST /v1/openai/chat/completions`	`POST /v1/openai/chat/completions`
Decoding	Every field at once	One token at a time, constrained	One token at a time
Output	Each field as its type or `null`	JSON that matches the schema	Text that should contain JSON
Parse can fail	No	No	Yes
Confidence per field	Yes, plus `p_null`	No	No
Where a text value came from	`span`	No	No
Takes a PDF directly	Yes, first 16 pages	Only on the OCR models	Only on the OCR models
Nested objects	No: flat fields and lists	Yes	Yes

Use System One fields for a flat set of values that code acts on: invoice and receipt totals, form fields, identifiers, dates. Use a json_schema completion for nested objects, or when a value is a paragraph of generated text.

Measured

One RTX 5090 serving google/gemma-4-26b-a4b-it through the Gateway. Each row reads the same 10 fields (text, dates, amounts, a list, and an optional PO number) from the same 32 synthetic invoices. For a PDF, the chat approaches send the pages as images, since this model takes no PDF on Chat Completions.

Input	Approach	p50 latency, 1 at a time	Documents/s, 4 at a time	Documents/s, 16 at a time	Output tokens	Accuracy
Text	Schema in the prompt	988 ms	2.9	9.5	174	100.0%
Text	`json_schema`	960 ms	3.0	10.1	169	100.0%
Text	System One fields	272 ms	6.0	5.8	103	100.0%
Image	Schema in the prompt	1065 ms	2.5	6.2	173	100.0%
Image	`json_schema`	878 ms	2.8	7.8	148	100.0%
Image	System One fields	340 ms	3.9	4.0	103	100.0%
PDF, 2 pages	Schema in the prompt	1107 ms	2.2	4.6	174	99.7%
PDF, 2 pages	`json_schema`	1068 ms	2.3	4.6	169	99.7%
PDF, 2 pages	System One fields	410 ms	4.0	3.8	100	97.5%

Latency: System One returns a document 2.6x to 3.6x sooner, at every input type.
Output: 30% to 43% fewer output tokens. Keys and JSON syntax are never generated.
Throughput: System One leads at low and moderate concurrency. With 16 or more documents in flight, one JSON completion per document keeps up better, because each field is one engine call.

On other models:

google/diffusiongemma-26b-a4b-it writes every field on one canvas in a single read. On text it returned a document in 170 ms against 185 ms for a JSON reply, and read 1.5x the documents per second at 4 at a time. Its fields carry no confidence or p_null.
qwen/qwen3.5-0.8b writes a whole JSON reply in about 330 ms, so fields save little time, and they were less accurate: each field is read without the others. Use a json_schema completion on a model this small.

2. Extract from text

Name each field with the id you want back and give it a type. Write instructions for anything the id does not make obvious. A field id reaches the model, so order_number reads better than f1. This example reads a shipping confirmation email. The email has no gift message, so gift_message comes back null.

Python

import os

import httpx

EMAIL = """From: orders@northwind-supply.example
Subject: Your order NW-58213 has shipped

Hi Priya Raman,

Order NW-58213 left our Reno warehouse on March 3, 2026 with UPS Ground,
tracking number 1Z999AA10123456784. It should arrive by March 6, 2026.

In this shipment:
  2 x Standing desk frame, black
  1 x Bamboo desktop, 60 x 30 in
  4 x Under-desk cable tray

Order total: $1,184.97, charged to the Visa ending in 4421.

Northwind Supply Co."""

ORDER_FIELDS = {
    "order_number": {"type": "text", "instructions": "The order number"},
    "customer_name": {"type": "text", "instructions": "The customer the email is addressed to"},
    "ship_date": {"type": "date", "instructions": "The date the order shipped"},
    "delivery_date": {"type": "date", "instructions": "The expected delivery date"},
    "carrier": {"type": "text", "instructions": "The shipping carrier and service"},
    "tracking_number": {"type": "text", "instructions": "The carrier's tracking number"},
    "items": {"type": "list", "items": "text", "instructions": "The name of each item, without the quantity"},
    "total": {"type": "number", "decimals": 2, "instructions": "The order total"},
    "gift_message": {"type": "text", "instructions": "The gift message, if the order has one"},
}

reply = httpx.post(
    "https://gateway.vlm.run/typesafe/v1/systemone",
    headers={"Authorization": f"Bearer {os.environ['VLMRUN_API_KEY']}"},
    json={"model": "qwen/qwen3.8-27b", "state": EMAIL, "questions": ORDER_FIELDS},
    timeout=60,
)
reply.raise_for_status()
for name, answer in reply.json()["answers"].items():
    print(f"{name:<16} {answer['value']!r}")

Output

order_number     'NW-58213'
customer_name    'Priya Raman'
ship_date        '2026-03-03'
delivery_date    '2026-03-06'
carrier          'UPS Ground'
tracking_number  '1Z999AA10123456784'
items            ['Standing desk frame, black', 'Bamboo desktop, 60 x 30 in', 'Under-desk cable tray']
total            1184.97
gift_message     None

The System One quickstart for fields sends fields from Node.js and cURL too, and shows the full reply.

Do	Why
Use `date` and `number` for dates and amounts	The model converts `March 3, 2026` and `$1,184.97` for you. You get `2026-03-03` and `1184.97`.
Keep each field to one value	A field is one line. For several values, use a `list`.
Say what a list item holds	`"The name of each item, without the quantity"` keeps every item the same shape.
Use `span` as provenance	A `text` value with a `span` was copied from the state at those offsets. Highlight it for a reviewer.
Leave nested structure to a JSON completion	Fields are flat. A table with columns is several `list` fields, or a `json_schema` completion.

3. Extract from an image

Send the image as an image_url part, as an http(s) URL or a data: URL. The state says what to read. Keep detail at its default for small print. This example reads a U.S. passport card exemplar.

Python

import os

import httpx

PASSPORT = "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.id-extraction/usa_passport.webp"

PASSPORT_FIELDS = {
    "surname": {"type": "text"},
    "given_names": {"type": "text"},
    "passport_number": {"type": "text", "instructions": "The passport card number"},
    "nationality": {"type": "text"},
    "sex": {"type": "text"},
    "date_of_birth": {"type": "date"},
    "place_of_birth": {"type": "text"},
    "issue_date": {"type": "date", "instructions": "The date the card was issued"},
    "expiry_date": {"type": "date", "instructions": "The date the card expires"},
}

reply = httpx.post(
    "https://gateway.vlm.run/typesafe/v1/systemone",
    headers={"Authorization": f"Bearer {os.environ['VLMRUN_API_KEY']}"},
    json={
        "model": "qwen/qwen3.8-27b",
        "state": "Read the attached passport card.",
        "content": [{"type": "image_url", "image_url": {"url": PASSPORT}}],
        "questions": PASSPORT_FIELDS,
    },
    timeout=60,
)
reply.raise_for_status()
for name, answer in reply.json()["answers"].items():
    print(f"{name:<16} {answer['value']!r}")

Output

surname          'TRAVELER'
given_names      'HAPPY'
passport_number  'C03004786'
nationality      'USA'
sex              'F'
date_of_birth    '1981-01-01'
place_of_birth   'NEW YORK, U.S.A.'
issue_date       '2009-11-30'
expiry_date      '2019-11-29'

A value read from an image has no span. Its confidence and p_null still apply.

4. Extract from a PDF

Send the PDF as a document_url part, or as a file part with the bytes inline. The route renders the first 16 pages at 96 DPI and reads them as one set of images. A field can come from any of them: a total on page 2 is read in the same call as an invoice number on page 1. Three fields control the pages, as on Chat Completions:

Field	Use
`document_pages`	Read only these pages, 0-indexed. `[0, [2, 4]]` reads pages 0, 2, and 3.
`document_max_pages`	Read at most this many pages, 1 to 16. Fewer pages bill fewer image tokens.
`document_dpi`	Render pages at this resolution, 72 to 400. Raise it for small or faint print.

This example reads a one-page Google invoice. Its figures are small and light, so it renders at 144 DPI. The invoice prints no purchase order number, so po_number comes back null.

import os

import httpx

INVOICE = "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/google_invoice.pdf"

INVOICE_FIELDS = {
    "invoice_number": {"type": "text", "instructions": "The invoice number"},
    "bill_to": {"type": "text", "instructions": "The name the invoice is billed to"},
    "issue_date": {"type": "date", "instructions": "The invoice date"},
    "due_date": {"type": "date", "instructions": "The date payment is due"},
    "po_number": {"type": "text", "instructions": "The purchase order number, if one is printed"},
    "items": {"type": "list", "items": "text", "instructions": "The name of each line item"},
    "subtotal": {"type": "number", "decimals": 2},
    "tax": {"type": "number", "decimals": 2},
    "total": {"type": "number", "decimals": 2, "instructions": "The invoice total, after discounts, tax and shipping"},
    "balance_due": {"type": "number", "decimals": 2, "instructions": "The amount still owed"},
}

reply = httpx.post(
    "https://gateway.vlm.run/typesafe/v1/systemone",
    headers={"Authorization": f"Bearer {os.environ['VLMRUN_API_KEY']}"},
    json={
        "model": "qwen/qwen3.8-27b",
        "state": "Extract the fields from the attached invoice.",
        "content": [{"type": "document_url", "document_url": {"url": INVOICE}}],
        "document_dpi": 144,
        "questions": INVOICE_FIELDS,
    },
    timeout=120,
)
reply.raise_for_status()
answers = reply.json()["answers"]
for name, answer in answers.items():
    print(f"{name:<16} {answer['value']!r}")

PDF=$(curl -sL https://storage.googleapis.com/vlm-data-public-prod/hub/examples/document.invoice/google_invoice.pdf | base64 -w0)

curl https://gateway.vlm.run/typesafe/v1/systemone \
  -H "Authorization: Bearer $VLMRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d @- <<JSON
{
  "model": "qwen/qwen3.8-27b",
  "state": "Extract the fields from the attached invoice.",
  "content": [{"type": "file", "file": {"filename": "google_invoice.pdf", "file_data": "data:application/pdf;base64,$PDF"}}],
  "document_dpi": 144,
  "questions": {
    "invoice_number": {"type": "text", "instructions": "The invoice number"},
    "balance_due": {"type": "number", "decimals": 2, "instructions": "The amount still owed"}
  }
}
JSON

Output

invoice_number   '23413561D'
bill_to          'Jane Smith'
issue_date       '2019-09-24'
due_date         '2019-09-30'
po_number        None
items            ['12 ft HDMI cable', '27" Computer Monitor', 'Ergonomic Keyboard', 'Optical mouse', 'Laptop', 'Misc processing fees']
subtotal         22379.39
tax              1767.97
total            19647.68
balance_due      4647.68

Chat Completions takes a PDF only on the OCR models. On a general model such as qwen/qwen3.8-27b, you would rasterise the pages yourself and send them as images. System One does that for you.

Act on the confidence

Accept a field when its confidence clears your threshold. Send the rest to a person, or to a larger model with only those fields.

accepted, review = {}, []
for name, answer in answers.items():
    if answer["value"] is None and answer["p_null"] > 0.9:
        accepted[name] = None  # absent from the document
    elif answer["confidence"] >= 0.9:
        accepted[name] = answer["value"]
    else:
        review.append(name)

Answer	Meaning
A value, high `confidence`	Accept.
`null`, `p_null` near 1	The document does not give the field.
A value, high `p_null`, low `confidence`	A guess. Check it.

Thresholds are model-specific. Revalidate them on your own documents when you change models.

6. Extract and decide in one call

Fields share a request with noul, choice, and score questions. Read the values and a gate together:

QUESTIONS = {
    **INVOICE_FIELDS,
    "needs_approval": {"type": "noul", "instructions": "Is the total over $5,000?"},
    "kind": {"type": "choice", "criteria": {"invoice": None, "receipt": None, "credit_note": None}},
}

Every question is read concurrently. The reply carries a value for each field and a probability for each decision.

System One

Field types, answer shapes, and limits.

System One API

Full request and response reference.

Document Classification

Route documents by kind before you extract.

Response Formats

json_schema on Chat Completions, for nested output.

Gateway

Documents

Images

Video and audio

Field Extraction

1. Pick an approach

Measured

2. Extract from text

3. Extract from an image

4. Extract from a PDF

Act on the confidence

6. Extract and decide in one call

System One

System One API

Document Classification

Response Formats

​1. Pick an approach

​Measured

​2. Extract from text

​3. Extract from an image

​4. Extract from a PDF

​ Act on the confidence

​6. Extract and decide in one call

​Related

System One

System One API

Document Classification

Response Formats

1. Pick an approach

Measured

2. Extract from text

3. Extract from an image

4. Extract from a PDF

Act on the confidence

6. Extract and decide in one call

Related