---
name: pose-estimation
description: Extract human pose landmarks from video or images (MediaPipe Tasks API) and compare pose sequences with DTW joint-angle scoring. Use for analyzing dance/movement videos, scoring a recreated animation against source footage, or any "what is the body doing at time t" question. Works on real footage AND stylized game characters when a single subject fills the frame.
---

# Pose Estimation

Two tested scripts, one venv. Extract 33-landmark pose sequences from video, then
score two sequences against each other (e.g. source dance video vs game recreation).

## Setup (idempotent, run once per machine)

```bash
bash ~/.claude/skills/pose-estimation/scripts/setup.sh
```

Creates `.venv` (pins python3.12 — MediaPipe does not support 3.13+) and downloads the
`full` pose model. `lite`/`heavy` model variants auto-download on first use via curl
(NOT urllib — macOS python lacks SSL certs; keep the curl download path).

Always run scripts with the venv python: `~/.claude/skills/pose-estimation/.venv/bin/python`.

## Extract poses

```bash
VENV=~/.claude/skills/pose-estimation/.venv/bin/python
SKILL=~/.claude/skills/pose-estimation/scripts

$VENV $SKILL/extract_pose.py input.mp4 --out poses.json --fps 8 \
    --overlay check.mp4 --min-conf 0.3 --model heavy
```

- `--fps N` sample rate; `--times "0.0,0.5,1.0"` instead samples an explicit beat grid
  (pass a musical beat grid for dance work — samples land on choreographically meaningful frames).
- `--overlay out.mp4` draws the skeleton on sampled frames — ALWAYS eyeball this before
  trusting the data.
- Output JSON has both `landmarks` (normalized image coords) and `world_landmarks`
  (meters, hip-centered). Use world landmarks for any comparison — they are camera- and
  scale-invariant.
- The `OK: N/M samples had a detected pose` line is the health signal. Low N/M → see
  framing rules below.

## Compare two pose sequences

```bash
$VENV $SKILL/compare_poses.py source_poses.json recreation_poses.json \
    --a-range 2.0:6.0 --b-range 0:4.0
```

Reduces each frame to 8 joint angles (elbows, shoulders, hips, knees), aligns the two
sequences with DTW (tolerates tempo drift), prints JSON: `score_0_100`,
`mean_angle_error_deg`, and `per_joint_error_deg` sorted worst-first (tells you WHERE
the recreation diverges — e.g. arms right, legs wrong). Verified: identical input → 100.0;
different dance segments → ~70.

## Framing rules (the thing that actually determines success)

MediaPipe pose is **single-person** and needs the subject to fill a large fraction of
the frame. Verified findings:

- Wide shot with many small figures (e.g. a game formation capture at 1600x900 with
  ~80px characters): **0% detection**. This is framing, not the model.
- Same video, one character crop-zoomed to fill frame (`ffmpeg -vf "crop=W:H:X:Y,scale=4x"`):
  **94% detection** — even on stylized low-poly game characters (Quaternius rigs).
- Real photos/footage: works out of the box.

So: for multi-person or wide footage, crop to one subject first:

```bash
ffmpeg -i wide.mp4 -vf "crop=140:180:350:420,scale=560:720:flags=lanczos" -an solo.mp4
```

Knobs when detection is weak: `--min-conf 0.15`, `--model heavy`, bigger crop upscale.

## Typical workflow (video → analysis)

1. Extract a frame, look at it, choose the crop for the subject.
2. Extract with `--overlay`, check the overlay video, check detection %.
3. Compare / analyze from `world_landmarks` or via compare_poses.py.