> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.

# Hand Tracking

> Read 2D and 3D hand joints from every frame of a video and detect a pinch

Read the 21 joints of every hand in each frame, over one WebSocket, with `geopavlakos/hamer`. Each hand comes back with 2D joints on the image and 3D joints in meters, so you can measure the hand itself, not only its picture. This guide draws each hand on a 4 s clip of hands tying a box, and detects a pinch from the 3D gap between thumb and index finger.

![Three video frames with hand skeletons drawn over hands tying a cake box](https://mintcdn.com/autonomiai/PRWL1LD_l1Le04c7/images/gateway-live-hands.jpg?fit=max\&auto=format\&n=PRWL1LD_l1Le04c7\&q=85\&s=6e667f8f589cc8e4f02bc9935401f4e5)

Frames 9, 17, and 26 of the output. Red is a right hand, blue is a left hand.

## 1. Cut the clip

```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
curl -sO https://storage.googleapis.com/vlm-data-public-prod/hub/examples/video.transcription/bakery.mp4
mkdir -p hands
ffmpeg -ss 24 -t 4 -i bakery.mp4 -vf "fps=8,scale=640:-2" -q:v 3 hands/%04d.jpg
```

That writes 32 frames of 640 x 360 pixels from 24 s to 28 s of the video. Hands move fast, so this guide reads 8 frames per second. `hamer` allows up to 20.

## 2. Stream and detect a pinch

```bash theme={"theme":{"light":"github-light","dark":"dark-plus"}}
pip install websockets pillow
```

```python Python [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
import asyncio
import json
import math
import pathlib

import websockets
from PIL import Image, ImageDraw

URL = "wss://gateway.vlm.run/v1/ws"
FPS = 8
PINCH_M = 0.03
FINGERS = ["thumb", "index", "middle", "ring", "pinky"]
BONES = [(f"{f}_{a}", f"{f}_{b}") for f in FINGERS for a, b in [("mcp", "pip"), ("pip", "dip"), ("dip", "tip")]]
BONES += [("wrist", f"{f}_mcp") for f in FINGERS]

paths = sorted(pathlib.Path("hands").glob("*.jpg"))
out = pathlib.Path("tracked")
out.mkdir(exist_ok=True)


def draw(path: pathlib.Path, content: dict) -> Image.Image:
    image = Image.open(path).convert("RGB")
    pen = ImageDraw.Draw(image)
    w, h = image.size
    index = {name: i for i, name in enumerate(content["kpts_labels"])}
    for hand in content["items"]:
        xy = [(x * w, y * h) for x, y in hand["kpts_xy"]]
        color = (255, 80, 80) if hand["label"] == "right_hand" else (80, 140, 255)
        for a, b in BONES:
            pen.line([xy[index[a]], xy[index[b]]], fill=color, width=3)
    return image


async def main() -> None:
    async with websockets.connect(
        URL, additional_headers={"Authorization": "Bearer <VLMRUN_API_KEY>"}
    ) as ws:
        await ws.send(json.dumps({
            "type": "session.create",
            "model": "geopavlakos/hamer",
            "method": "pose",
            "max_inflight": 4,
        }))
        created = json.loads(await ws.recv())
        if created["type"] == "error":
            raise RuntimeError(f"{created['error_type']}: {created['message']}")

        async def send_frames() -> None:
            for path in paths:
                await ws.send(path.read_bytes())
                await asyncio.sleep(1 / FPS)

        sender = asyncio.create_task(send_frames())
        for _ in paths:
            event = json.loads(await ws.recv())
            if event["type"] != "result":
                print(event)
                continue
            content = json.loads(event["response"]["choices"][0]["message"]["content"])
            if not content["items"]:
                print(event["frame"], "no hands")
                continue
            draw(paths[event["frame"] - 1], content).save(out / f"{event['frame']:04d}.jpg")
            index = {name: i for i, name in enumerate(content["kpts_labels"])}
            hands = []
            for hand in content["items"]:
                gap = math.dist(hand["kpts_xyz"][index["thumb_tip"]], hand["kpts_xyz"][index["index_tip"]])
                hands.append(f"{hand['label']} {gap * 100:.1f} cm{' pinch' if gap < PINCH_M else ''}")
            print(event["frame"], hands)
        await sender

        await ws.send(json.dumps({"type": "session.close"}))
        print(json.loads(await ws.recv()))


asyncio.run(main())
```

The sample draws each hand's bones on its frame, writes the frame to `tracked/`, and prints the thumb-to-index gap of every hand. A gap under 3 cm counts as a pinch.

```text Output theme={"theme":{"light":"github-light","dark":"dark-plus"}}
1 ['right_hand 3.4 cm', 'left_hand 4.5 cm', 'right_hand 12.1 cm']
2 ['right_hand 3.1 cm', 'left_hand 2.7 cm pinch', 'right_hand 5.2 cm']
...
6 ['right_hand 2.3 cm pinch', 'left_hand 6.2 cm']
7 no hands
8 ['right_hand 3.0 cm']
9 ['right_hand 2.7 cm pinch', 'left_hand 2.9 cm pinch']
...
14 ['right_hand 1.7 cm pinch', 'left_hand 0.9 cm pinch']
...
32 ['left_hand 5.0 cm', 'left_hand 6.0 cm', 'right_hand 3.5 cm']
{'type': 'session.stats', 'session_id': '...', 'id': 'ws-...', 'reads': 32, 'dropped': 0, 'errors': 0, 'cost': 0.003442, 'p50_ms': 30.87, 'p95_ms': 33.13, 'fps': 7.75}
```

All 32 frames were read, with a median read of 31 ms. The session was open about 4 s, which bills $0.0034 at $3 / hour.

## 3. Read a hand

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{
  "object": "world.pose.kpts",
  "kpts_labels": ["wrist", "thumb_mcp", "thumb_pip", "thumb_dip", "thumb_tip", "index_mcp", "...", "pinky_tip"],
  "items": [
    {
      "bbox_xywh": [0.0, 0.2777, 0.0337, 0.0895],
      "label": "right_hand",
      "score": 0.9115,
      "kpts_xy": [[0.0437, 0.283], [0.0335, 0.3339]],
      "kpts_xyz": [[0.1009, 0.0069, 0.0064], [0.0876, 0.0435, 0.0201]]
    }
  ]
}
```

| Field | Meaning |
| - | - |
| `label` | `right_hand` or `left_hand` |
| `score` | Hand detection confidence, from 0 to 1 |
| `kpts_xy` | 21 joints on the image, as fractions of the frame width and height |
| `kpts_xyz` | The same 21 joints in 3D, in meters. Use distances between joints, not their positions. |
| `kpts_labels` | The joint names, in the order of `kpts_xy` and `kpts_xyz` |

Look joints up by name with `kpts_labels`, not by a fixed index. The full schema is in [`world.pose.kpts`](https://docs.vlm.run/gateway/response-types#world-pose-kpts).

> [!WARNING]
>
> A frame with no hands returns `{"object": "world.pose.kpts", "items": []}` with no `kpts_labels`. Check `items` before you read the labels. Hands also have no id: their order in `items` can change between frames, so match hands across frames by `label` and box position.

## 4. The first session after idle

When `hamer` has no running replica, `session.create` is refused while the Gateway starts one. The session gets an `error` event, then close `1013`:

```json theme={"theme":{"light":"github-light","dark":"dark-plus"}}
{"type": "error", "error_type": "model_provisioning_error", "message": "geopavlakos/hamer is starting up. Retry in about 240 s.", "frame": null, "retry_after": 240}
```

Wait `retry_after` seconds, then open a new session. In the first session after start-up, the first 4 reads took about 2 s each and 13 of 32 frames were dropped as `busy`. Later sessions read every frame. To avoid this at a live event, open a short session a few minutes before you need the model.

## Related

> [card] [Webcam from the Browser](https://docs.vlm.run/gateway/guides/live-webcam.md)
>
> Run a session on a user's camera.

> [card] [Masks per Frame](https://docs.vlm.run/gateway/guides/live-masks.md)
>
> Segment every frame by a text prompt.

> [card] [Response Types](https://docs.vlm.run/gateway/response-types.md#world-pose-kpts)
>
> The `world.pose.kpts` schema in JSON, Pydantic, and Zod.

> [card] [Realtime](https://docs.vlm.run/gateway/realtime.md)
>
> Session lifecycle, close codes, and billing.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.