# Animation Gen Pipeline — Video → Recreated Dance

**Goal:** give an agent a video of people dancing; the agent breaks the choreography
down and recreates it in-game as a dance JSON, using Kevin Iglesias (KI) clips, UAL
clips, composites, and Mixamo retargets — then verifies its own recreation against the
source and iterates.

**Honest framing:** this produces a *cover version*, not motion capture. Continuous
human motion gets quantized into a discrete vocabulary of clips + composites. For game
choreography (kapa haka teams, dance floor content) that is the desired output.

## Verified infrastructure (all exists, all tested 2026-07-13)

| Piece | Where | Status |
|---|---|---|
| Dance runtime + hot-reload JSON | `src/Testing/Dance/`, `assets/dances/*.json` | SHIPPED — test bed plays dances, reloads in ~0.5s |
| Composite baking (upper/lower masks) | `CompositeClipBaker.cs` | SHIPPED |
| BPM beat clock + quantized transitions | `BeatClock.cs`, `bpm` in dance JSON | SHIPPED |
| Viewport → mp4 recording | `VideoRecorder.cs` → `~/Downloads/dance_*.mp4` | SHIPPED |
| Mixamo FBX → Quaternius GLB | `tools/mixamo_retarget.py` (Blender 5.1.2 headless) | PROVEN (`northern_soul.glb`) |
| Clip vocabulary | 12 KI packs + UAL1/2 + mixamo dir, ~810 clips; KI social = Dance01–18, DancePoses, claps, cheers | SHIPPED |
| Pose extraction from video | `pose-estimation` skill (user-level, MediaPipe Tasks) | BUILT + TESTED today |
| Pose sequence comparison (DTW, 8 joint angles) | `compare_poses.py` in same skill | BUILT + TESTED (identity=100.0, cross-segment≈71) |
| Animation creation recipes | `.claude/skills/animation-creation/SKILL.md` | BUILT today |

**Key experimental finding (2026-07-13):** MediaPipe tracks the stylized Quaternius
characters at **94% detection** when ONE character is crop-zoomed to fill the frame
(`crop=140:180:350:420,scale=560:720:flags=lanczos` on a test bed recording). Wide
formation shots detect **0%**. Every capture step below therefore frames a single dancer.

## Pipeline stages

### Stage 1 — Ingest the source video
1. `ffprobe` duration/fps; extract audio (`ffmpeg -vn audio.wav`).
2. **BPM + beat grid**: `librosa.beat.beat_track` (add librosa to the pose-estimation
   venv when first needed). Beat times become the dance's `bpm` and the sampling grid.
3. Extract pose: `extract_pose.py video.mp4 --times "<beat grid>" --overlay check.mp4`.
   If multi-person/wide: crop-zoom the clearest dancer first. Eyeball the overlay.
4. Extract frames on the beat grid for the vision pass (`ffmpeg -vf select=...`).

### Stage 2 — Choreography breakdown
- **Programmatic move segmentation**: the 8-angle signature (elbows/shoulders/hips/knees)
  from the pose JSON gives a per-beat pose vector; a spike in angle-space distance
  between consecutive beats = likely move boundary. Cheap, deterministic.
- **Vision pass**: agent reads beat-grid frames and writes a per-segment description
  ("beats 1–8: crouched stomp, arms slapping thighs..."). Combines with the programmatic
  boundaries — vision names the moves, angles locate them.
- Output: `breakdown.json` — segments with beat ranges, text description, pose signature.

### Stage 3 — Clip card catalog (the one missing build item)
One-time batch job; the enabler for matching. For each dance-relevant clip
(KI Dance/DancePose/claps/cheers + UAL emotes + mixamo pack):
1. Test bed plays the clip on ONE dancer, camera framed close (solo framing per the
   94% finding). VideoRecorder captures ~2 loops.
2. `extract_pose.py` on the capture → **pose signature** stored per clip.
3. Delegated vision (pi `nova_describe` / cheap model) writes a **text card** per clip.
4. Output: `assets/dances/catalog/clip_cards.json` — `{clipRef, text, bpm-ish tempo,
   energy, pose_signature_file}`. Rebuild only when packs change.

Build item: a `CatalogMode` in the dance test bed (iterate clips → frame solo dancer →
record → next). Everything else is scripting.

### Stage 4 — Matching (segment → clip or composite)
1. **Numeric**: DTW each video segment's angle sequence against every catalog clip
   signature (`compare_poses.py` logic). Rank by score.
2. **Split-mask matching for composites**: score arms (4 arm angles) and legs (4 leg
   angles) *independently*. If no whole clip scores well but clip A wins arms and clip
   B wins legs → propose `{layers:[{mask:"upper",clip:A},{mask:"lower",clip:B}]}`.
3. **Vision tiebreak**: agent compares top-3 candidates' text cards against the
   segment description; picks; records rationale.

### Stage 5 — Gap-fill via Mixamo
Segment with no acceptable match → agent emits a **shopping list** (search terms for
mixamo.com; downloads are behind Adobe login, so Jeremy grabs FBXs manually — ~5 min)
→ `mixamo_retarget.py` → new pack in `assets/quaternius/mixamo/` → add to catalog
(Stage 3 for just that pack) → re-match.

### Stage 6 — Emit the dance
Write `assets/dances/<name>.json`: `bpm` from Stage 1, one move per segment, `loops`
from beat counts. Hot-reload shows it immediately if the test bed is running.

### Stage 7 — Verify loop (closes the circle)
1. Test bed plays the dance; camera solo-framed on one dancer; VideoRecorder captures.
2. `extract_pose.py` on the capture → `compare_poses.py` vs source, per segment
   (`--a-range/--b-range` from the beat grid).
3. `score_0_100` per segment + `per_joint_error_deg` says WHERE it diverges (arms vs
   legs → swap just that layer of a composite). Iterate moves until segments clear a
   threshold (start ~65; calibrate).
4. Final deliverable: side-by-side mp4 (`ffmpeg hstack`) + per-segment score table.

## Deliverables ledger

| Item | Status |
|---|---|
| `pose-estimation` skill (extract + compare, venv, models) | DONE 2026-07-13 |
| `animation-creation` skill (project-level recipes) | DONE 2026-07-13 |
| This pipeline doc | DONE 2026-07-13 |
| Test bed `CatalogMode` (solo-frame + record each clip) | TODO — first build item |
| Catalog batch script + `clip_cards.json` | TODO |
| librosa beat grid helper | TODO (trivial, add when needed) |
| Segment/match/emit agent workflow (`/dance-from-video`) | TODO — after catalog |

## Risks / open questions
- **Camera angle mismatch**: source video vs game capture viewpoint differ; world
  landmarks + joint angles are view-invariant in principle, but MediaPipe's 3D lift is
  weaker at oblique angles. Mitigation: match game camera height/angle roughly to source.
- **Multi-dancer sources**: pick ONE dancer (ideally the leader) and track them; group
  formation is authored separately (formations are already a test bed parameter).
- **Catalog scale**: ~810 clips × record+describe is hours of batch time. Start with
  the ~40 obviously dance-relevant clips; expand on demand.
- **Score threshold semantics**: 71 = "different segments of the same dance", so
  same-move recreations should land higher; calibrate on known pairs before trusting.
- KI clips are polished loops; Mixamo retargets can have foot slide — acceptable for
  dance (no locomotion), watch for it.