jeremy
31ba2911df
Skills (.claude/skills/): animation-creation, iclone-video-mocap, pose-estimation (MediaPipe models via LFS), retarget-animations (deprecated). Tools: cc/mixamo/kevin/mocap retargeters + composite baker. Plans: animation-gen-pipeline + skills-adoption. exchange/: incoming-fbx, converted-glb, reference-video (LFS for fbx/glb/mp4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
115 lines
6.9 KiB
Markdown
115 lines
6.9 KiB
Markdown
# Animation Gen Pipeline — Video → Recreated Dance
|
||
|
||
**Goal:** give an agent a video of people dancing; the agent breaks the choreography
|
||
down and recreates it in-game as a dance JSON, using Kevin Iglesias (KI) clips, UAL
|
||
clips, composites, and Mixamo retargets — then verifies its own recreation against the
|
||
source and iterates.
|
||
|
||
**Honest framing:** this produces a *cover version*, not motion capture. Continuous
|
||
human motion gets quantized into a discrete vocabulary of clips + composites. For game
|
||
choreography (kapa haka teams, dance floor content) that is the desired output.
|
||
|
||
## Verified infrastructure (all exists, all tested 2026-07-13)
|
||
|
||
| Piece | Where | Status |
|
||
|---|---|---|
|
||
| Dance runtime + hot-reload JSON | `src/Testing/Dance/`, `assets/dances/*.json` | SHIPPED — test bed plays dances, reloads in ~0.5s |
|
||
| Composite baking (upper/lower masks) | `CompositeClipBaker.cs` | SHIPPED |
|
||
| BPM beat clock + quantized transitions | `BeatClock.cs`, `bpm` in dance JSON | SHIPPED |
|
||
| Viewport → mp4 recording | `VideoRecorder.cs` → `~/Downloads/dance_*.mp4` | SHIPPED |
|
||
| Mixamo FBX → Quaternius GLB | `tools/mixamo_retarget.py` (Blender 5.1.2 headless) | PROVEN (`northern_soul.glb`) |
|
||
| Clip vocabulary | 12 KI packs + UAL1/2 + mixamo dir, ~810 clips; KI social = Dance01–18, DancePoses, claps, cheers | SHIPPED |
|
||
| Pose extraction from video | `pose-estimation` skill (user-level, MediaPipe Tasks) | BUILT + TESTED today |
|
||
| Pose sequence comparison (DTW, 8 joint angles) | `compare_poses.py` in same skill | BUILT + TESTED (identity=100.0, cross-segment≈71) |
|
||
| Animation creation recipes | `.claude/skills/animation-creation/SKILL.md` | BUILT today |
|
||
|
||
**Key experimental finding (2026-07-13):** MediaPipe tracks the stylized Quaternius
|
||
characters at **94% detection** when ONE character is crop-zoomed to fill the frame
|
||
(`crop=140:180:350:420,scale=560:720:flags=lanczos` on a test bed recording). Wide
|
||
formation shots detect **0%**. Every capture step below therefore frames a single dancer.
|
||
|
||
## Pipeline stages
|
||
|
||
### Stage 1 — Ingest the source video
|
||
1. `ffprobe` duration/fps; extract audio (`ffmpeg -vn audio.wav`).
|
||
2. **BPM + beat grid**: `librosa.beat.beat_track` (add librosa to the pose-estimation
|
||
venv when first needed). Beat times become the dance's `bpm` and the sampling grid.
|
||
3. Extract pose: `extract_pose.py video.mp4 --times "<beat grid>" --overlay check.mp4`.
|
||
If multi-person/wide: crop-zoom the clearest dancer first. Eyeball the overlay.
|
||
4. Extract frames on the beat grid for the vision pass (`ffmpeg -vf select=...`).
|
||
|
||
### Stage 2 — Choreography breakdown
|
||
- **Programmatic move segmentation**: the 8-angle signature (elbows/shoulders/hips/knees)
|
||
from the pose JSON gives a per-beat pose vector; a spike in angle-space distance
|
||
between consecutive beats = likely move boundary. Cheap, deterministic.
|
||
- **Vision pass**: agent reads beat-grid frames and writes a per-segment description
|
||
("beats 1–8: crouched stomp, arms slapping thighs..."). Combines with the programmatic
|
||
boundaries — vision names the moves, angles locate them.
|
||
- Output: `breakdown.json` — segments with beat ranges, text description, pose signature.
|
||
|
||
### Stage 3 — Clip card catalog (the one missing build item)
|
||
One-time batch job; the enabler for matching. For each dance-relevant clip
|
||
(KI Dance/DancePose/claps/cheers + UAL emotes + mixamo pack):
|
||
1. Test bed plays the clip on ONE dancer, camera framed close (solo framing per the
|
||
94% finding). VideoRecorder captures ~2 loops.
|
||
2. `extract_pose.py` on the capture → **pose signature** stored per clip.
|
||
3. Delegated vision (pi `nova_describe` / cheap model) writes a **text card** per clip.
|
||
4. Output: `assets/dances/catalog/clip_cards.json` — `{clipRef, text, bpm-ish tempo,
|
||
energy, pose_signature_file}`. Rebuild only when packs change.
|
||
|
||
Build item: a `CatalogMode` in the dance test bed (iterate clips → frame solo dancer →
|
||
record → next). Everything else is scripting.
|
||
|
||
### Stage 4 — Matching (segment → clip or composite)
|
||
1. **Numeric**: DTW each video segment's angle sequence against every catalog clip
|
||
signature (`compare_poses.py` logic). Rank by score.
|
||
2. **Split-mask matching for composites**: score arms (4 arm angles) and legs (4 leg
|
||
angles) *independently*. If no whole clip scores well but clip A wins arms and clip
|
||
B wins legs → propose `{layers:[{mask:"upper",clip:A},{mask:"lower",clip:B}]}`.
|
||
3. **Vision tiebreak**: agent compares top-3 candidates' text cards against the
|
||
segment description; picks; records rationale.
|
||
|
||
### Stage 5 — Gap-fill via Mixamo
|
||
Segment with no acceptable match → agent emits a **shopping list** (search terms for
|
||
mixamo.com; downloads are behind Adobe login, so Jeremy grabs FBXs manually — ~5 min)
|
||
→ `mixamo_retarget.py` → new pack in `assets/quaternius/mixamo/` → add to catalog
|
||
(Stage 3 for just that pack) → re-match.
|
||
|
||
### Stage 6 — Emit the dance
|
||
Write `assets/dances/<name>.json`: `bpm` from Stage 1, one move per segment, `loops`
|
||
from beat counts. Hot-reload shows it immediately if the test bed is running.
|
||
|
||
### Stage 7 — Verify loop (closes the circle)
|
||
1. Test bed plays the dance; camera solo-framed on one dancer; VideoRecorder captures.
|
||
2. `extract_pose.py` on the capture → `compare_poses.py` vs source, per segment
|
||
(`--a-range/--b-range` from the beat grid).
|
||
3. `score_0_100` per segment + `per_joint_error_deg` says WHERE it diverges (arms vs
|
||
legs → swap just that layer of a composite). Iterate moves until segments clear a
|
||
threshold (start ~65; calibrate).
|
||
4. Final deliverable: side-by-side mp4 (`ffmpeg hstack`) + per-segment score table.
|
||
|
||
## Deliverables ledger
|
||
|
||
| Item | Status |
|
||
|---|---|
|
||
| `pose-estimation` skill (extract + compare, venv, models) | DONE 2026-07-13 |
|
||
| `animation-creation` skill (project-level recipes) | DONE 2026-07-13 |
|
||
| This pipeline doc | DONE 2026-07-13 |
|
||
| Test bed `CatalogMode` (solo-frame + record each clip) | TODO — first build item |
|
||
| Catalog batch script + `clip_cards.json` | TODO |
|
||
| librosa beat grid helper | TODO (trivial, add when needed) |
|
||
| Segment/match/emit agent workflow (`/dance-from-video`) | TODO — after catalog |
|
||
|
||
## Risks / open questions
|
||
- **Camera angle mismatch**: source video vs game capture viewpoint differ; world
|
||
landmarks + joint angles are view-invariant in principle, but MediaPipe's 3D lift is
|
||
weaker at oblique angles. Mitigation: match game camera height/angle roughly to source.
|
||
- **Multi-dancer sources**: pick ONE dancer (ideally the leader) and track them; group
|
||
formation is authored separately (formations are already a test bed parameter).
|
||
- **Catalog scale**: ~810 clips × record+describe is hours of batch time. Start with
|
||
the ~40 obviously dance-relevant clips; expand on demand.
|
||
- **Score threshold semantics**: 71 = "different segments of the same dance", so
|
||
same-move recreations should land higher; calibrate on known pairs before trusting.
|
||
- KI clips are polished loops; Mixamo retargets can have foot slide — acceptable for
|
||
dance (no locomotion), watch for it.
|