Skip to main content
Stream a user’s camera from a web page to a realtime model, and draw each reply on the video as it arrives. The browser sends frames to a small relay on your server, and the relay holds the API key and the Gateway socket. This guide draws body keypoints with usyd-community/vitpose-plus-large at 5 frames per second.
A street scene with body keypoints drawn as blue dots on a walking man

The page below with a recorded street scene as the camera. Each dot is a keypoint from the latest result.

1. Why a relay

The browser WebSocket API cannot set request headers, so a page cannot send Authorization on the upgrade. Putting the key in the page would also give it to every visitor. The relay solves both: the browser connects to your server, and your server opens the Gateway socket with the key. The relay forwards every message unchanged, in both directions. The page speaks the WS /v1/ws protocol, so session.create, frames, and events are the same as in Video over WebSocket.

2. Run the relay

The relay serves the page at / and relays the socket at /live. It opens one Gateway session for each browser connection, and closes it when the browser leaves.
Messages the browser sends before the Gateway socket opens wait, then go out in order. When either side closes, the relay closes the other.
A session on /v1/ws bills every second it is open, with or without frames. The relay closes the Gateway session when the browser socket closes. Keep that line if you change the relay, or a closed tab bills until the 30 second idle close.

3. Build the page

Save the page as public/index.html. It asks for the camera, opens the relay socket, and sends one JPEG of the video every 200 ms once session.created arrives.
public/index.html
Open http://localhost:8080 and allow the camera. The status line shows the latest frame number and its event, and the overlay redraws on every result. Keypoints are fractions of the frame from 0 to 1, so the page multiplies them by the canvas size. Keypoints with a score below 0.5 are not drawn. getUserMedia runs only on a secure origin: https in production, or localhost while you develop. In production, serve the page over https and connect with wss://.

4. Keep the stream live

The page sends on a timer and does not wait for replies. A frame that arrives while max_inflight reads are outstanding is dropped with reason: "busy", so a slow read costs you one stale frame, never a growing delay. Each event names its frame, so you can always tell how old the drawing is. In a test with a recorded camera feed, the page sent 5 frames per second and drew a result for each one.

5. Before you ship

  • Authenticate your users at the relay. The relay spends your API key. Check the user’s session on the /live upgrade, and refuse it when the check fails.
  • Cap sessions per user. Each browser connection is one Gateway session. Your plan allows 1 at once on Free and up to 5 on paid plans. See Rate Limits.
  • Reconnect on 1013. A full plan or a starting model closes with 1013. Wait retry_after when the error event has one, then open a new socket. See Close codes.
  • Reopen at 10 minutes. A session lasts up to 10 minutes. Open a new one when it closes with the reason expired.

Video over WebSocket

The same socket from a script, with a recorded video.

Masks per Frame

Segment every frame with facebook/sam3.1.

Hand Tracking

2D and 3D hand joints per frame with geopavlakos/hamer.

Realtime

Session lifecycle, close codes, and billing.