80 lines
3.4 KiB
Markdown
80 lines
3.4 KiB
Markdown
|
---
|
||
|
|
name: pose-estimation
|
||
|
|
description: Extract human pose landmarks from video or images (MediaPipe Tasks API) and compare pose sequences with DTW joint-angle scoring. Use for analyzing dance/movement videos, scoring a recreated animation against source footage, or any "what is the body doing at time t" question. Works on real footage AND stylized game characters when a single subject fills the frame.
|
||
|
|
---
|
||
|
|
|
||
|
|
# Pose Estimation
|
||
|
|
|
||
|
|
Two tested scripts, one venv. Extract 33-landmark pose sequences from video, then
|
||
|
|
score two sequences against each other (e.g. source dance video vs game recreation).
|
||
|
|
|
||
|
|
## Setup (idempotent, run once per machine)
|
||
|
|
|
||
|
|
```bash
|
||
|
|
bash ~/.claude/skills/pose-estimation/scripts/setup.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
Creates `.venv` (pins python3.12 — MediaPipe does not support 3.13+) and downloads the
|
||
|
|
`full` pose model. `lite`/`heavy` model variants auto-download on first use via curl
|
||
|
|
(NOT urllib — macOS python lacks SSL certs; keep the curl download path).
|
||
|
|
|
||
|
|
Always run scripts with the venv python: `~/.claude/skills/pose-estimation/.venv/bin/python`.
|
||
|
|
|
||
|
|
## Extract poses
|
||
|
|
|
||
|
|
```bash
|
||
|
|
VENV=~/.claude/skills/pose-estimation/.venv/bin/python
|
||
|
|
SKILL=~/.claude/skills/pose-estimation/scripts
|
||
|
|
|
||
|
|
$VENV $SKILL/extract_pose.py input.mp4 --out poses.json --fps 8 \
|
||
|
|
--overlay check.mp4 --min-conf 0.3 --model heavy
|
||
|
|
```
|
||
|
|
|
||
|
|
- `--fps N` sample rate; `--times "0.0,0.5,1.0"` instead samples an explicit beat grid
|
||
|
|
(pass a musical beat grid for dance work — samples land on choreographically meaningful frames).
|
||
|
|
- `--overlay out.mp4` draws the skeleton on sampled frames — ALWAYS eyeball this before
|
||
|
|
trusting the data.
|
||
|
|
- Output JSON has both `landmarks` (normalized image coords) and `world_landmarks`
|
||
|
|
(meters, hip-centered). Use world landmarks for any comparison — they are camera- and
|
||
|
|
scale-invariant.
|
||
|
|
- The `OK: N/M samples had a detected pose` line is the health signal. Low N/M → see
|
||
|
|
framing rules below.
|
||
|
|
|
||
|
|
## Compare two pose sequences
|
||
|
|
|
||
|
|
```bash
|
||
|
|
$VENV $SKILL/compare_poses.py source_poses.json recreation_poses.json \
|
||
|
|
--a-range 2.0:6.0 --b-range 0:4.0
|
||
|
|
```
|
||
|
|
|
||
|
|
Reduces each frame to 8 joint angles (elbows, shoulders, hips, knees), aligns the two
|
||
|
|
sequences with DTW (tolerates tempo drift), prints JSON: `score_0_100`,
|
||
|
|
`mean_angle_error_deg`, and `per_joint_error_deg` sorted worst-first (tells you WHERE
|
||
|
|
the recreation diverges — e.g. arms right, legs wrong). Verified: identical input → 100.0;
|
||
|
|
different dance segments → ~70.
|
||
|
|
|
||
|
|
## Framing rules (the thing that actually determines success)
|
||
|
|
|
||
|
|
MediaPipe pose is **single-person** and needs the subject to fill a large fraction of
|
||
|
|
the frame. Verified findings:
|
||
|
|
|
||
|
|
- Wide shot with many small figures (e.g. a game formation capture at 1600x900 with
|
||
|
|
~80px characters): **0% detection**. This is framing, not the model.
|
||
|
|
- Same video, one character crop-zoomed to fill frame (`ffmpeg -vf "crop=W:H:X:Y,scale=4x"`):
|
||
|
|
**94% detection** — even on stylized low-poly game characters (Quaternius rigs).
|
||
|
|
- Real photos/footage: works out of the box.
|
||
|
|
|
||
|
|
So: for multi-person or wide footage, crop to one subject first:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
ffmpeg -i wide.mp4 -vf "crop=140:180:350:420,scale=560:720:flags=lanczos" -an solo.mp4
|
||
|
|
```
|
||
|
|
|
||
|
|
Knobs when detection is weak: `--min-conf 0.15`, `--model heavy`, bigger crop upscale.
|
||
|
|
|
||
|
|
## Typical workflow (video → analysis)
|
||
|
|
|
||
|
|
1. Extract a frame, look at it, choose the crop for the subject.
|
||
|
|
2. Extract with `--overlay`, check the overlay video, check detection %.
|
||
|
|
3. Compare / analyze from `world_landmarks` or via compare_poses.py.
|