Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.vlm.run/llms.txt

Use this file to discover all available pages before exploring further.

Analyze and understand user interface elements in screenshots and application images. Perfect for automated testing, design system validation, accessibility auditing, and mobile app analysis.
UI parsing example showing UI element detection and classification with interactive elements
UI VQA & Grounding
UI VQA & Grounding
Web Interface
Web interface UI parsing

​
Usage Example

For best results, we recommend using the Structured Outputs API to get responses in a structured and validated data format.
from vlmrun.client import VLMRun

# Initialize the VLMRun client
client = VLMRun(api_key="<VLMRUN_API_KEY>")

# Parse UI elements in the image
response = client.agent.completions.create(
    model="vlmrun-orion-1:auto",
    messages=[
        {
          "role": "user",
          "content": [
            {"type": "text", "text": "Analyze all UI elements in this mobile app screenshot"},
            {"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/web.ui-automation/win11.jpeg", "detail": "auto"}}
          ]
        }
    ],
)

print(response.choices[0].message.content)

​
FAQ

UI Parsing is the process of analyzing UI elements in screenshots and application images to identify UI elements, buttons, and interactive components for automated testing.
UI VQA & Grounding is the process of asking specific questions about the UI elements in screenshots and application images to identify UI elements, buttons, and interactive components for automated testing. This is different from UI parsing, where all UI elements are returned. In most cases, you should use UI VQA & Grounding to get more accurate results.