animation/.claude/skills/pose-estimation/SKILL.md

name, description

name	description
pose-estimation	Extract human pose landmarks from video or images (MediaPipe Tasks API) and compare pose sequences with DTW joint-angle scoring. Use for analyzing dance/movement videos, scoring a recreated animation against source footage, or any "what is the body doing at time t" question. Works on real footage AND stylized game characters when a single subject fills the frame.

Pose Estimation

Two tested scripts, one venv. Extract 33-landmark pose sequences from video, then score two sequences against each other (e.g. source dance video vs game recreation).

Setup (idempotent, run once per machine)

bash ~/.claude/skills/pose-estimation/scripts/setup.sh

Creates .venv (pins python3.12 — MediaPipe does not support 3.13+) and downloads the full pose model. lite/heavy model variants auto-download on first use via curl (NOT urllib — macOS python lacks SSL certs; keep the curl download path).

Always run scripts with the venv python: ~/.claude/skills/pose-estimation/.venv/bin/python.

Extract poses

VENV=~/.claude/skills/pose-estimation/.venv/bin/python
SKILL=~/.claude/skills/pose-estimation/scripts

$VENV $SKILL/extract_pose.py input.mp4 --out poses.json --fps 8 \
    --overlay check.mp4 --min-conf 0.3 --model heavy

• --fps N sample rate; --times "0.0,0.5,1.0" instead samples an explicit beat grid (pass a musical beat grid for dance work — samples land on choreographically meaningful frames).
• --overlay out.mp4 draws the skeleton on sampled frames — ALWAYS eyeball this before trusting the data.
• Output JSON has both landmarks (normalized image coords) and world_landmarks (meters, hip-centered). Use world landmarks for any comparison — they are camera- and scale-invariant.
• The OK: N/M samples had a detected pose line is the health signal. Low N/M → see framing rules below.

Compare two pose sequences

$VENV $SKILL/compare_poses.py source_poses.json recreation_poses.json \
    --a-range 2.0:6.0 --b-range 0:4.0

Reduces each frame to 8 joint angles (elbows, shoulders, hips, knees), aligns the two sequences with DTW (tolerates tempo drift), prints JSON: score_0_100, mean_angle_error_deg, and per_joint_error_deg sorted worst-first (tells you WHERE the recreation diverges — e.g. arms right, legs wrong). Verified: identical input → 100.0; different dance segments → ~70.

Framing rules (the thing that actually determines success)

MediaPipe pose is single-person and needs the subject to fill a large fraction of the frame. Verified findings:

• Wide shot with many small figures (e.g. a game formation capture at 1600x900 with ~80px characters): 0% detection. This is framing, not the model.
• Same video, one character crop-zoomed to fill frame (ffmpeg -vf "crop=W:H:X:Y,scale=4x"): 94% detection — even on stylized low-poly game characters (Quaternius rigs).
• Real photos/footage: works out of the box.

So: for multi-person or wide footage, crop to one subject first:

ffmpeg -i wide.mp4 -vf "crop=140:180:350:420,scale=560:720:flags=lanczos" -an solo.mp4

Knobs when detection is weak: --min-conf 0.15, --model heavy, bigger crop upscale.

Typical workflow (video → analysis)

1. Extract a frame, look at it, choose the crop for the subject.
2. Extract with --overlay, check the overlay video, check detection %.
3. Compare / analyze from world_landmarks or via compare_poses.py.