> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.
# Pointing
> Detect and predict key anatomical points and structural features in images with sub-pixel accuracy
Detect and localize keypoints of objects, people or faces in images with precise coordinate mapping. Ideal for counting, localization and salience detection.
Object Localization
Person Localization
Face Localization
## Usage Example
For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Predict the keypoints in the image
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Point to all the cars and doors in this image"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto"}}
]
}
],
)
print(response.choices[0].message.content)
```
```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
from vlmrun.client import VLMRun
from pydantic import BaseModel, Field
# Define the response schema
class KeyPoint(BaseModel):
xy: tuple[float, float] = Field(..., description="Normalized keypoint coordinates [x, y]")
label: str = Field(..., description="Label of the keypoint")
class Keypoints(BaseModel):
keypoints: list[KeyPoint] = Field(..., description="List of keypoints")
# Initialize the VLMRun client
client = VLMRun(api_key="")
# Predict the keypoints with structured output
response = client.agent.completions.create(
model="vlmrun-orion-1:auto",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Point to all the cars and doors in this image"},
{"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", "detail": "auto"}}
]
}
],
response_format={"type": "json_schema", "schema": Keypoints.model_json_schema()},
)
# Validate the response
result = Keypoints.model_validate_json(response.choices[0].message.content)
# >>> Keypoints(keypoints=[KeyPoint(xy=(0.5, 0.5), label='car'), ...])
```
```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Point to all the cars and doors in this image" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
]
}
]
});
console.log(response.choices[0].message.content);
```
```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import { VlmRun } from "vlmrun";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
// Define the response schema with Zod
const KeypointsSchema = z.object({
keypoints: z.array(z.object({
xy: z.array(z.number()).describe("Normalized keypoint coordinates [x, y]"),
label: z.string().describe("Label of the keypoint")
})).describe("List of keypoints")
});
// Initialize the VLMRun client
const client = new VlmRun({
apiKey: "",
baseURL: "https://api.vlm.run/v1"
});
// Predict the keypoints with structured output
const response = await client.agent.completions.create({
model: "vlmrun-orion-1:auto",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Point to all the cars and doors in this image" },
{ type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/image.caption/car.jpg", detail: "auto" } }
]
}
],
response_format: {
type: "json_schema",
schema: zodToJsonSchema(KeypointsSchema)
}
});
const result = KeypointsSchema.parse(JSON.parse(response.choices[0].message.content));
```
## FAQ
The keypoints come in the format of a list of objects with their keypoints. The keypoints are in the format of normalized `xy`, where `x` and `y` are the top-left corner of the keypoint. All values are between 0 and 1, and normalized by the image size. `x` and `y` are normalized by the image width and height respectively.
You can extract the following tags for each keypoint:
* **Object Name**: The name of the object that the keypoint belongs to. For example, "car", "door", "person", "face", etc.
* **Confidence Score**: The confidence score of the keypoint detection.