init: animation pipeline hub — Mac↔PC bridge for converting and implementing animations

Skills (.claude/skills/): animation-creation, iclone-video-mocap,
pose-estimation (MediaPipe models via LFS), retarget-animations (deprecated).
Tools: cc/mixamo/kevin/mocap retargeters + composite baker.
Plans: animation-gen-pipeline + skills-adoption.
exchange/: incoming-fbx, converted-glb, reference-video (LFS for fbx/glb/mp4).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

plans/animation-gen-pipeline-2026-07-13.md

@@ -0,0 +1,114 @@
+# Animation Gen Pipeline — Video → Recreated Dance
+
+**Goal:** give an agent a video of people dancing; the agent breaks the choreography
+down and recreates it in-game as a dance JSON, using Kevin Iglesias (KI) clips, UAL
+clips, composites, and Mixamo retargets — then verifies its own recreation against the
+source and iterates.
+
+**Honest framing:** this produces a *cover version*, not motion capture. Continuous
+human motion gets quantized into a discrete vocabulary of clips + composites. For game
+choreography (kapa haka teams, dance floor content) that is the desired output.
+
+## Verified infrastructure (all exists, all tested 2026-07-13)
+
+| Piece | Where | Status |
+|---|---|---|
+| Dance runtime + hot-reload JSON | `src/Testing/Dance/`, `assets/dances/*.json` | SHIPPED — test bed plays dances, reloads in ~0.5s |
+| Composite baking (upper/lower masks) | `CompositeClipBaker.cs` | SHIPPED |
+| BPM beat clock + quantized transitions | `BeatClock.cs`, `bpm` in dance JSON | SHIPPED |
+| Viewport → mp4 recording | `VideoRecorder.cs` → `~/Downloads/dance_*.mp4` | SHIPPED |
+| Mixamo FBX → Quaternius GLB | `tools/mixamo_retarget.py` (Blender 5.1.2 headless) | PROVEN (`northern_soul.glb`) |
+| Clip vocabulary | 12 KI packs + UAL1/2 + mixamo dir, ~810 clips; KI social = Dance01–18, DancePoses, claps, cheers | SHIPPED |
+| Pose extraction from video | `pose-estimation` skill (user-level, MediaPipe Tasks) | BUILT + TESTED today |
+| Pose sequence comparison (DTW, 8 joint angles) | `compare_poses.py` in same skill | BUILT + TESTED (identity=100.0, cross-segment≈71) |
+| Animation creation recipes | `.claude/skills/animation-creation/SKILL.md` | BUILT today |
+
+**Key experimental finding (2026-07-13):** MediaPipe tracks the stylized Quaternius
+characters at **94% detection** when ONE character is crop-zoomed to fill the frame
+(`crop=140:180:350:420,scale=560:720:flags=lanczos` on a test bed recording). Wide
+formation shots detect **0%**. Every capture step below therefore frames a single dancer.
+
+## Pipeline stages
+
+### Stage 1 — Ingest the source video
+1. `ffprobe` duration/fps; extract audio (`ffmpeg -vn audio.wav`).
+2. **BPM + beat grid**: `librosa.beat.beat_track` (add librosa to the pose-estimation
+   venv when first needed). Beat times become the dance's `bpm` and the sampling grid.
+3. Extract pose: `extract_pose.py video.mp4 --times "<beat grid>" --overlay check.mp4`.
+   If multi-person/wide: crop-zoom the clearest dancer first. Eyeball the overlay.
+4. Extract frames on the beat grid for the vision pass (`ffmpeg -vf select=...`).
+
+### Stage 2 — Choreography breakdown
+- **Programmatic move segmentation**: the 8-angle signature (elbows/shoulders/hips/knees)
+  from the pose JSON gives a per-beat pose vector; a spike in angle-space distance
+  between consecutive beats = likely move boundary. Cheap, deterministic.
+- **Vision pass**: agent reads beat-grid frames and writes a per-segment description
+  ("beats 1–8: crouched stomp, arms slapping thighs..."). Combines with the programmatic
+  boundaries — vision names the moves, angles locate them.
+- Output: `breakdown.json` — segments with beat ranges, text description, pose signature.
+
+### Stage 3 — Clip card catalog (the one missing build item)
+One-time batch job; the enabler for matching. For each dance-relevant clip
+(KI Dance/DancePose/claps/cheers + UAL emotes + mixamo pack):
+1. Test bed plays the clip on ONE dancer, camera framed close (solo framing per the
+   94% finding). VideoRecorder captures ~2 loops.
+2. `extract_pose.py` on the capture → **pose signature** stored per clip.
+3. Delegated vision (pi `nova_describe` / cheap model) writes a **text card** per clip.
+4. Output: `assets/dances/catalog/clip_cards.json` — `{clipRef, text, bpm-ish tempo,
+   energy, pose_signature_file}`. Rebuild only when packs change.
+
+Build item: a `CatalogMode` in the dance test bed (iterate clips → frame solo dancer →
+record → next). Everything else is scripting.
+
+### Stage 4 — Matching (segment → clip or composite)
+1. **Numeric**: DTW each video segment's angle sequence against every catalog clip
+   signature (`compare_poses.py` logic). Rank by score.
+2. **Split-mask matching for composites**: score arms (4 arm angles) and legs (4 leg
+   angles) *independently*. If no whole clip scores well but clip A wins arms and clip
+   B wins legs → propose `{layers:[{mask:"upper",clip:A},{mask:"lower",clip:B}]}`.
+3. **Vision tiebreak**: agent compares top-3 candidates' text cards against the
+   segment description; picks; records rationale.
+
+### Stage 5 — Gap-fill via Mixamo
+Segment with no acceptable match → agent emits a **shopping list** (search terms for
+mixamo.com; downloads are behind Adobe login, so Jeremy grabs FBXs manually — ~5 min)
+→ `mixamo_retarget.py` → new pack in `assets/quaternius/mixamo/` → add to catalog
+(Stage 3 for just that pack) → re-match.
+
+### Stage 6 — Emit the dance
+Write `assets/dances/<name>.json`: `bpm` from Stage 1, one move per segment, `loops`
+from beat counts. Hot-reload shows it immediately if the test bed is running.
+
+### Stage 7 — Verify loop (closes the circle)
+1. Test bed plays the dance; camera solo-framed on one dancer; VideoRecorder captures.
+2. `extract_pose.py` on the capture → `compare_poses.py` vs source, per segment
+   (`--a-range/--b-range` from the beat grid).
+3. `score_0_100` per segment + `per_joint_error_deg` says WHERE it diverges (arms vs
+   legs → swap just that layer of a composite). Iterate moves until segments clear a
+   threshold (start ~65; calibrate).
+4. Final deliverable: side-by-side mp4 (`ffmpeg hstack`) + per-segment score table.
+
+## Deliverables ledger
+
+| Item | Status |
+|---|---|
+| `pose-estimation` skill (extract + compare, venv, models) | DONE 2026-07-13 |
+| `animation-creation` skill (project-level recipes) | DONE 2026-07-13 |
+| This pipeline doc | DONE 2026-07-13 |
+| Test bed `CatalogMode` (solo-frame + record each clip) | TODO — first build item |
+| Catalog batch script + `clip_cards.json` | TODO |
+| librosa beat grid helper | TODO (trivial, add when needed) |
+| Segment/match/emit agent workflow (`/dance-from-video`) | TODO — after catalog |
+
+## Risks / open questions
+- **Camera angle mismatch**: source video vs game capture viewpoint differ; world
+  landmarks + joint angles are view-invariant in principle, but MediaPipe's 3D lift is
+  weaker at oblique angles. Mitigation: match game camera height/angle roughly to source.
+- **Multi-dancer sources**: pick ONE dancer (ideally the leader) and track them; group
+  formation is authored separately (formations are already a test bed parameter).
+- **Catalog scale**: ~810 clips × record+describe is hours of batch time. Start with
+  the ~40 obviously dance-relevant clips; expand on demand.
+- **Score threshold semantics**: 71 = "different segments of the same dance", so
+  same-move recreations should land higher; calibrate on known pairs before trusting.
+- KI clips are polished loops; Mixamo retargets can have foot slide — acceptable for
+  dance (no locomotion), watch for it.

plans/animation-skills-adoption-2026-07-13.md

@@ -0,0 +1,112 @@
+# Animation skills adoption — 2026-07-13
+
+Goal: let agents **create** animations and rig them onto GLBs for ariki-game, not just
+retarget existing packs. All animation runs on the one shared Quaternius skeleton (65
+bones, UE names), so the bottleneck is tooling that gets new motion onto that rig. This
+report records the research tiering, what this pass integrated (file authoring only), and
+the orchestrator runbook for the install/launch steps an agent must not do.
+
+## Research findings
+
+**Tier 1 — adopt now (integrated or queued for trial this pass):**
+
+- **blender-mcp** (github.com/ahujasid/blender-mcp, MIT, 16k+ stars) — MCP server bridging
+  the agent to a live Blender via a socket addon; the agent sends arbitrary `bpy` to build
+  armatures, keyframe, set up IK/NLA, and export GLB. *Why Tier 1:* the one capability the
+  project was missing — direct programmatic Blender authoring from the agent loop.
+- **Blender Toolkit skill** (majiayu000) — Mixamo→custom-rig retarget with fuzzy bone
+  matching over a WebSocket bridge. *Why Tier 1:* directly comparable to the project's own
+  `tools/mixamo_retarget.py`; trial it head-to-head and keep the winner.
+- **Blender Animation & Rigging Expert skill** — programmatic armatures, IK/FK, drivers,
+  NLA via MCP. *Why Tier 1:* the deep-blender companion to blender-mcp for non-trivial rigs.
+- **Extend our own project skill** (this pass) — the existing `.claude/skills/animation-creation/`
+  is the cheapest surface; adding Path 5 + gltf-transform keeps everything in one place.
+
+**Tier 2 — trial before committing:**
+
+- **gltf-transform** (gltf-transform.dev, donmccurdy) — headless GLB surgery: inspect,
+  merge, resample, prune, dedup; same-skeleton clip transplants need no Blender. *Why Tier
+  2:* documented here and usable via `npx`, but the project's clip-editing volume is
+  currently low; adopt the recipes, verify the v4 API on first real use.
+- **PoseCap** (github.com/CorridorTech/PoseCap) — markerless mocap, video → PEAR model →
+  SMPL-X poses → Blender keyframes; works on pre-recorded clips; custom-rig retarget is
+  roadmap-only; UI extension, not headless. *Why Tier 2:* promising video-to-motion path,
+  but SMPL-X→Quaternius retarget is unbuilt — eval before investing.
+- **OpenClaw / freshtechbro Blender pipeline skills** — engine export-settings knowledge.
+  *Why Tier 2:* useful reference for engine-specific gotchas, redundant once our export
+  block is canonical.
+
+**Tier 3 — skip for now:**
+
+- **godot-mcp servers** — redundant: the game already has an agent API on port 4329 plus an
+  in-scene `VideoRecorder`; a Godot MCP adds nothing.
+- **DavinciDreams 3D Design Team plugin and other knowledge-only skill packs** — broad and
+  shallow; the targeted recipes above cover the same ground more concretely.
+
+## Integrated in this pass
+
+File authoring only (no installs, no commits):
+
+- **A — `.mcp.json`**: registered the `blender` MCP server (`uvx blender-mcp`).
+- **B — `.claude/skills/animation-creation/SKILL.md`**: added **Path 5 — Author in live
+  Blender (blender-mcp)** and a **GLB surgery without Blender (gltf-transform)** section,
+  both pointing at the new reference docs.
+- **C — `references/blender-mcp-recipes.md`** (new): `bpy` recipes for setup, keyframing,
+  IK setup + bake, NLA layering, and an **export block mirrored exactly from
+  `tools/mixamo_retarget.py`** (`NLA_TRACKS` mode, `export_force_sampling`, size-opt off),
+  plus gotchas (`*RM` variants, `RESET` clip, one armature/GLB, fps=30, quaternion mode).
+- **D — `references/gltf-transform-recipes.md`** (new): CLI/SDK inspect, same-skeleton clip
+  transplant sketch (flagged `verify against gltf-transform v4 API before first use`),
+  resample/prune/dedup shrink pass, and when-to-prefer-over-Blender guidance.
+
+## Orchestrator runbook
+
+**Not run by the authoring agent.** The orchestrator (or Jeremy) executes these:
+
+1. **blender-mcp smoke test.** From a shell: `uvx blender-mcp` — confirm the server starts
+   and binds the socket (Ctrl-C to stop; this only checks installability).
+2. **Install the Blender addon.** Download `addon.py` from
+   github.com/ahujasid/blender-mcp and in Blender 5.1.2
+   (`/Applications/Blender.app`): Edit → Preferences → Add-ons → Install → select
+   `addon.py` → enable. Then 3D View sidebar → BlenderMCP → **Connect**.
+3. **Reload MCP config.** Restart Claude Code so the `blender` server in `.mcp.json` is
+   picked up; confirm the `blender` MCP tools (`execute-code`, etc.) are now available.
+4. **blender-mcp agent smoke test.** Through the MCP tools: import a game GLB
+   (`assets/quaternius/kevin/kevin_male_social.glb`), keyframe a wave on `hand_l`
+   (see `blender-mcp-recipes.md` keyframing recipe), export GLB into
+   `assets/quaternius/mixamo/`, and confirm the new clip appears in
+   `scenes/animation_showcase.tscn`.
+5. **gltf-transform smoke test.** `npx @gltf-transform/cli inspect
+   assets/quaternius/kevin/kevin_male_social.glb` — confirm it lists the animations and
+   their durations. (No install step; `npx` fetches the CLI on demand.)
+6. **Skill trials (user-level install at `~/.claude/skills/`).** Install **Blender Toolkit**
+   (majiayu000) and **Blender Animation & Rigging Expert**. Run one Mixamo clip → Quaternius
+   retarget through Blender Toolkit and compare the output against
+   `tools/mixamo_retarget.py` in the dance test bed — keep whichever loses as
+   reference-only.
+7. **PoseCap eval.** Install the PoseCap Blender extension; run one short dance clip →
+   SMPL-X keyframes; assess quality vs the existing "match video to shipped clips via the
+   pose-estimation skill" approach **before** investing in SMPL-X→Quaternius retarget.
+
+> **Always ASK Jeremy before any game launch.** Run `tinqs pull` before launching the game
+> so the working tree is current.
+
+## Verification
+
+- **Retarget round-trip:** a clip authored via blender-mcp (or retargeted through Blender
+  Toolkit) plays correctly in the dance test bed (`SCENE=dance_test_bed
+  bash tools/game.sh spawn` — ask Jeremy first), recorded with the in-scene
+  `VideoRecorder` to `~/Downloads/*.mp4` and scored against source footage with the
+  `pose-estimation` skill.
+- **gltf-transform clip transplant:** a clip moved between two same-skeleton GLBs via the
+  SDK recipe plays in `scenes/animation_showcase.tscn` at the expected `<pack>/<ClipName>`,
+  confirming the channel-target re-pointing worked.
+
+## Open items
+
+- Confirm the gltf-transform v4 SDK call for cross-document animation copy on first real
+  use (the `copyToDocument` sketch in `gltf-transform-recipes.md` is flagged for this).
+- Decide Blender Toolkit vs `tools/mixamo_retarget.py` after the head-to-head retarget
+  trial (runbook step 6).
+- Decide whether to invest in SMPL-X→Quaternius retarget after the PoseCap quality eval
+  (runbook step 7).