init: animation pipeline hub — Mac↔PC bridge for converting and implementing animations
Skills (.claude/skills/): animation-creation, iclone-video-mocap, pose-estimation (MediaPipe models via LFS), retarget-animations (deprecated). Tools: cc/mixamo/kevin/mocap retargeters + composite baker. Plans: animation-gen-pipeline + skills-adoption. exchange/: incoming-fbx, converted-glb, reference-video (LFS for fbx/glb/mp4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,79 @@
|
||||
---
|
||||
name: pose-estimation
|
||||
description: Extract human pose landmarks from video or images (MediaPipe Tasks API) and compare pose sequences with DTW joint-angle scoring. Use for analyzing dance/movement videos, scoring a recreated animation against source footage, or any "what is the body doing at time t" question. Works on real footage AND stylized game characters when a single subject fills the frame.
|
||||
---
|
||||
|
||||
# Pose Estimation
|
||||
|
||||
Two tested scripts, one venv. Extract 33-landmark pose sequences from video, then
|
||||
score two sequences against each other (e.g. source dance video vs game recreation).
|
||||
|
||||
## Setup (idempotent, run once per machine)
|
||||
|
||||
```bash
|
||||
bash ~/.claude/skills/pose-estimation/scripts/setup.sh
|
||||
```
|
||||
|
||||
Creates `.venv` (pins python3.12 — MediaPipe does not support 3.13+) and downloads the
|
||||
`full` pose model. `lite`/`heavy` model variants auto-download on first use via curl
|
||||
(NOT urllib — macOS python lacks SSL certs; keep the curl download path).
|
||||
|
||||
Always run scripts with the venv python: `~/.claude/skills/pose-estimation/.venv/bin/python`.
|
||||
|
||||
## Extract poses
|
||||
|
||||
```bash
|
||||
VENV=~/.claude/skills/pose-estimation/.venv/bin/python
|
||||
SKILL=~/.claude/skills/pose-estimation/scripts
|
||||
|
||||
$VENV $SKILL/extract_pose.py input.mp4 --out poses.json --fps 8 \
|
||||
--overlay check.mp4 --min-conf 0.3 --model heavy
|
||||
```
|
||||
|
||||
- `--fps N` sample rate; `--times "0.0,0.5,1.0"` instead samples an explicit beat grid
|
||||
(pass a musical beat grid for dance work — samples land on choreographically meaningful frames).
|
||||
- `--overlay out.mp4` draws the skeleton on sampled frames — ALWAYS eyeball this before
|
||||
trusting the data.
|
||||
- Output JSON has both `landmarks` (normalized image coords) and `world_landmarks`
|
||||
(meters, hip-centered). Use world landmarks for any comparison — they are camera- and
|
||||
scale-invariant.
|
||||
- The `OK: N/M samples had a detected pose` line is the health signal. Low N/M → see
|
||||
framing rules below.
|
||||
|
||||
## Compare two pose sequences
|
||||
|
||||
```bash
|
||||
$VENV $SKILL/compare_poses.py source_poses.json recreation_poses.json \
|
||||
--a-range 2.0:6.0 --b-range 0:4.0
|
||||
```
|
||||
|
||||
Reduces each frame to 8 joint angles (elbows, shoulders, hips, knees), aligns the two
|
||||
sequences with DTW (tolerates tempo drift), prints JSON: `score_0_100`,
|
||||
`mean_angle_error_deg`, and `per_joint_error_deg` sorted worst-first (tells you WHERE
|
||||
the recreation diverges — e.g. arms right, legs wrong). Verified: identical input → 100.0;
|
||||
different dance segments → ~70.
|
||||
|
||||
## Framing rules (the thing that actually determines success)
|
||||
|
||||
MediaPipe pose is **single-person** and needs the subject to fill a large fraction of
|
||||
the frame. Verified findings:
|
||||
|
||||
- Wide shot with many small figures (e.g. a game formation capture at 1600x900 with
|
||||
~80px characters): **0% detection**. This is framing, not the model.
|
||||
- Same video, one character crop-zoomed to fill frame (`ffmpeg -vf "crop=W:H:X:Y,scale=4x"`):
|
||||
**94% detection** — even on stylized low-poly game characters (Quaternius rigs).
|
||||
- Real photos/footage: works out of the box.
|
||||
|
||||
So: for multi-person or wide footage, crop to one subject first:
|
||||
|
||||
```bash
|
||||
ffmpeg -i wide.mp4 -vf "crop=140:180:350:420,scale=560:720:flags=lanczos" -an solo.mp4
|
||||
```
|
||||
|
||||
Knobs when detection is weak: `--min-conf 0.15`, `--model heavy`, bigger crop upscale.
|
||||
|
||||
## Typical workflow (video → analysis)
|
||||
|
||||
1. Extract a frame, look at it, choose the crop for the subject.
|
||||
2. Extract with `--overlay`, check the overlay video, check detection %.
|
||||
3. Compare / analyze from `world_landmarks` or via compare_poses.py.
|
||||
Reference in New Issue
Block a user