animation/.agents/plans/animation-gen-pipeline-2026-07-13.md

Animation Gen Pipeline — Video → Recreated Dance

Goal: give an agent a video of people dancing; the agent breaks the choreography down and recreates it in-game as a dance JSON, using Kevin Iglesias (KI) clips, UAL clips, composites, and Mixamo retargets — then verifies its own recreation against the source and iterates.

Honest framing: this produces a cover version, not motion capture. Continuous human motion gets quantized into a discrete vocabulary of clips + composites. For game choreography (kapa haka teams, dance floor content) that is the desired output.

Verified infrastructure (all exists, all tested 2026-07-13)

Piece	Where	Status
Dance runtime + hot-reload JSON	src/Testing/Dance/, assets/dances/*.json	SHIPPED — test bed plays dances, reloads in ~0.5s
Composite baking (upper/lower masks)	CompositeClipBaker.cs	SHIPPED
BPM beat clock + quantized transitions	BeatClock.cs, bpm in dance JSON	SHIPPED
Viewport → mp4 recording	VideoRecorder.cs → ~/Downloads/dance_*.mp4	SHIPPED
Mixamo FBX → Quaternius GLB	tools/mixamo_retarget.py (Blender 5.1.2 headless)	PROVEN (northern_soul.glb)
Clip vocabulary	12 KI packs + UAL1/2 + mixamo dir, ~810 clips; KI social = Dance01–18, DancePoses, claps, cheers	SHIPPED
Pose extraction from video	pose-estimation skill (user-level, MediaPipe Tasks)	BUILT + TESTED today
Pose sequence comparison (DTW, 8 joint angles)	compare_poses.py in same skill	BUILT + TESTED (identity=100.0, cross-segment≈71)
Animation creation recipes	.claude/skills/animation-creation/SKILL.md	BUILT today

Key experimental finding (2026-07-13): MediaPipe tracks the stylized Quaternius characters at 94% detection when ONE character is crop-zoomed to fill the frame (crop=140:180:350:420,scale=560:720:flags=lanczos on a test bed recording). Wide formation shots detect 0%. Every capture step below therefore frames a single dancer.

Pipeline stages

Stage 1 — Ingest the source video

1. ffprobe duration/fps; extract audio (ffmpeg -vn audio.wav).
2. BPM + beat grid: librosa.beat.beat_track (add librosa to the pose-estimation venv when first needed). Beat times become the dance's bpm and the sampling grid.
3. Extract pose: extract_pose.py video.mp4 --times "<beat grid>" --overlay check.mp4. If multi-person/wide: crop-zoom the clearest dancer first. Eyeball the overlay.
4. Extract frames on the beat grid for the vision pass (ffmpeg -vf select=...).

Stage 2 — Choreography breakdown

• Programmatic move segmentation: the 8-angle signature (elbows/shoulders/hips/knees) from the pose JSON gives a per-beat pose vector; a spike in angle-space distance between consecutive beats = likely move boundary. Cheap, deterministic.
• Vision pass: agent reads beat-grid frames and writes a per-segment description ("beats 1–8: crouched stomp, arms slapping thighs..."). Combines with the programmatic boundaries — vision names the moves, angles locate them.
• Output: breakdown.json — segments with beat ranges, text description, pose signature.

Stage 3 — Clip card catalog (the one missing build item)

One-time batch job; the enabler for matching. For each dance-relevant clip (KI Dance/DancePose/claps/cheers + UAL emotes + mixamo pack):

1. Test bed plays the clip on ONE dancer, camera framed close (solo framing per the 94% finding). VideoRecorder captures ~2 loops.
2. extract_pose.py on the capture → pose signature stored per clip.
3. Delegated vision (pi nova_describe / cheap model) writes a text card per clip.
4. Output: assets/dances/catalog/clip_cards.json — {clipRef, text, bpm-ish tempo, energy, pose_signature_file}. Rebuild only when packs change.

Build item: a CatalogMode in the dance test bed (iterate clips → frame solo dancer → record → next). Everything else is scripting.

Stage 4 — Matching (segment → clip or composite)

1. Numeric: DTW each video segment's angle sequence against every catalog clip signature (compare_poses.py logic). Rank by score.
2. Split-mask matching for composites: score arms (4 arm angles) and legs (4 leg angles) independently. If no whole clip scores well but clip A wins arms and clip B wins legs → propose {layers:[{mask:"upper",clip:A},{mask:"lower",clip:B}]}.
3. Vision tiebreak: agent compares top-3 candidates' text cards against the segment description; picks; records rationale.

Stage 5 — Gap-fill via Mixamo

Segment with no acceptable match → agent emits a shopping list (search terms for mixamo.com; downloads are behind Adobe login, so Jeremy grabs FBXs manually — ~5 min) → mixamo_retarget.py → new pack in assets/quaternius/mixamo/ → add to catalog (Stage 3 for just that pack) → re-match.

Stage 6 — Emit the dance

Write assets/dances/<name>.json: bpm from Stage 1, one move per segment, loops from beat counts. Hot-reload shows it immediately if the test bed is running.

Stage 7 — Verify loop (closes the circle)

1. Test bed plays the dance; camera solo-framed on one dancer; VideoRecorder captures.
2. extract_pose.py on the capture → compare_poses.py vs source, per segment (--a-range/--b-range from the beat grid).
3. score_0_100 per segment + per_joint_error_deg says WHERE it diverges (arms vs legs → swap just that layer of a composite). Iterate moves until segments clear a threshold (start ~65; calibrate).
4. Final deliverable: side-by-side mp4 (ffmpeg hstack) + per-segment score table.

Deliverables ledger

Item	Status
pose-estimation skill (extract + compare, venv, models)	DONE 2026-07-13
animation-creation skill (project-level recipes)	DONE 2026-07-13
This pipeline doc	DONE 2026-07-13
Test bed CatalogMode (solo-frame + record each clip)	TODO — first build item
Catalog batch script + clip_cards.json	TODO
librosa beat grid helper	TODO (trivial, add when needed)
Segment/match/emit agent workflow (/dance-from-video)	TODO — after catalog

Risks / open questions

• Camera angle mismatch: source video vs game capture viewpoint differ; world landmarks + joint angles are view-invariant in principle, but MediaPipe's 3D lift is weaker at oblique angles. Mitigation: match game camera height/angle roughly to source.
• Multi-dancer sources: pick ONE dancer (ideally the leader) and track them; group formation is authored separately (formations are already a test bed parameter).
• Catalog scale: ~810 clips × record+describe is hours of batch time. Start with the ~40 obviously dance-relevant clips; expand on demand.
• Score threshold semantics: 71 = "different segments of the same dance", so same-move recreations should land higher; calibrate on known pairs before trusting.
• KI clips are polished loops; Mixamo retargets can have foot slide — acceptable for dance (no locomotion), watch for it.