init: animation pipeline hub — Mac↔PC bridge for converting and implementing animations
Skills (.claude/skills/): animation-creation, iclone-video-mocap, pose-estimation (MediaPipe models via LFS), retarget-animations (deprecated). Tools: cc/mixamo/kevin/mocap retargeters + composite baker. Plans: animation-gen-pipeline + skills-adoption. exchange/: incoming-fbx, converted-glb, reference-video (LFS for fbx/glb/mp4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,124 @@
|
||||
---
|
||||
name: animation-creation
|
||||
description: Create new skeletal animations for ariki-game on the shared Quaternius skeleton — retarget Mixamo/Kevin-Iglesias FBX to game-ready GLB packs, bake composite clips (upper body of one clip over lower body of another), and author dance sequences as hot-reload JSON. Use when the user wants a new animation, dance, or move that doesn't exist in the shipped packs.
|
||||
---
|
||||
|
||||
# Animation Creation (ariki-game)
|
||||
|
||||
Every animation in the game runs on the **one shared Quaternius skeleton** (65 bones,
|
||||
UE names: `pelvis`, `spine_01..03`, `Head` capital-H, `thigh_l`, ...). Creating an
|
||||
animation means getting motion onto that skeleton. Four proven paths, cheapest first.
|
||||
|
||||
## Path 1 — It probably already exists (~810 clips)
|
||||
|
||||
Check before creating. Packs: `assets/quaternius/kevin/*.glb` (12 packs, male+female ×
|
||||
combat/idles/misc/movement/social/work — social has `Dance01`–`Dance18`,
|
||||
`DancePose01_Loop`–`07` with `_Begin`/`_Stop`, claps, waves, cheers), UAL1/UAL2
|
||||
standard libraries, `assets/quaternius/mixamo/*.glb`. List a pack's clips:
|
||||
|
||||
```bash
|
||||
# clip names live in the GLB's JSON chunk
|
||||
node -e "const b=require('fs').readFileSync('assets/quaternius/kevin/kevin_male_social.glb');const len=b.readUInt32LE(12);const j=JSON.parse(b.slice(20,20+len));console.log(j.animations.map(a=>a.name).join('\n'))"
|
||||
```
|
||||
|
||||
## Path 2 — Retarget a Mixamo FBX (new motion from Mixamo's huge library)
|
||||
|
||||
Download FBX(s) from mixamo.com (any character, "Without Skin" is fine), then:
|
||||
|
||||
```bash
|
||||
/Applications/Blender.app/Contents/MacOS/Blender --background \
|
||||
--python tools/mixamo_retarget.py -- \
|
||||
--src "<dir-or-fbx>[,<more>...]" --out assets/quaternius/mixamo/<pack_name>.glb
|
||||
```
|
||||
|
||||
- World-rotation-delta method; handles `mixamorig:` prefix variants automatically.
|
||||
- Proven end-to-end: `northern_soul.glb`, `run_to_dive.glb`.
|
||||
- Output GLBs in `assets/quaternius/mixamo/` are auto-discovered by ClipCatalog /
|
||||
AnimationShowcase — referenceable immediately as `<pack_name>/<ClipName>`.
|
||||
- Kevin Iglesias FBX sources use the sibling `tools/kevin_retarget.py` (different rig
|
||||
root handling — Kevin has a demo-scene offset bone).
|
||||
|
||||
## Path 3 — Composite clips (new move from two existing clips, no Blender)
|
||||
|
||||
The dance system bakes "upper body of clip A over lower body of clip B" at load time.
|
||||
Author it as a move in `assets/dances/*.json` — hot-reloads in ~0.5s while the dance
|
||||
test bed runs:
|
||||
|
||||
```json
|
||||
{ "layers": [
|
||||
{ "mask": "upper", "clip": "kevin_male_social/HandClap01" },
|
||||
{ "mask": "lower", "clip": "kevin_male_movement/Crouch01_Walk_Forward" } ],
|
||||
"duration": 4.0 }
|
||||
```
|
||||
|
||||
- Masks: `lower` = `root|pelvis|thigh*|calf*|foot*|ball*` (owns hip position AND
|
||||
rotation — always); `upper` = everything else. Authoritative predicate:
|
||||
`tools/bake_run_punch.mjs`.
|
||||
- For a composite needed **outside** the dance system, pre-bake a merged pack with a
|
||||
`bake_run_punch.mjs`-style node script instead (same mask logic, writes a GLB).
|
||||
|
||||
## Path 4 — Dance sequences (choreography = ordered moves + BPM)
|
||||
|
||||
A full "animation" at the choreography level is a JSON file in `assets/dances/`:
|
||||
`{ "name", "bpm", "moves": [ {"clip": "pack/Clip", "loops": 2}, {composite...} ] }`.
|
||||
Move transitions quantize to the beat clock. View in the test bed:
|
||||
`SCENE=dance_test_bed bash tools/game.sh spawn` (ASK Jeremy before launching the game).
|
||||
See `.agents/plans/dance-test-bed-2026-07-13.md` for full format + class specs.
|
||||
|
||||
## Path 5 — Author in live Blender (blender-mcp)
|
||||
|
||||
For motion that exists nowhere and can't be composited from existing clips — hand-keyed
|
||||
poses, IK-assisted tweaks, NLA layering of retargeted clips. Prerequisites: Blender 5.1.2
|
||||
open with the BlenderMCP addon connected (sidebar → BlenderMCP → Connect) and the `blender`
|
||||
MCP server registered in `.mcp.json`. Workflow: import an existing game GLB to pull in the
|
||||
real Quaternius armature → author (keyframes, IK, NLA) → export GLB into
|
||||
`assets/quaternius/mixamo/` so ClipCatalog auto-discovers it as `<pack>/<ClipName>`.
|
||||
Recipes: `references/blender-mcp-recipes.md`.
|
||||
|
||||
## GLB surgery without Blender (gltf-transform)
|
||||
|
||||
Inspect clip names, transplant or merge clips between same-skeleton GLBs, and
|
||||
resample/prune/dedup to shrink packs — no Blender required. Runs via
|
||||
`npx @gltf-transform/cli`, or as a JS SDK from a node script like the existing
|
||||
`tools/bake_run_punch.mjs`. Complements Path 3's node-script compositing when the job is a
|
||||
clip move or pack merge, not new motion. Recipes: `references/gltf-transform-recipes.md`.
|
||||
|
||||
## Hard rules (violations = invisible or broken animation)
|
||||
|
||||
- **Never use `*RM` root-motion clip variants** in dances/composites — they translate the rig.
|
||||
- **Skeleton path remap is mandatory** when playing raw GLB clips on a game rig: GLB
|
||||
tracks target `Armature/Skeleton3D:bone`; derive the rig's real prefix from an
|
||||
already-remapped clip (`ap.GetAnimation("idle").TrackGetPath(0)`), remap once, share.
|
||||
- Clip name fallback: exact → `+"_Loop"` → `-"_Loop"` (Godot import sometimes strips it).
|
||||
- Skip `RESET` clips; set `LoopMode.Linear` on extracted clips.
|
||||
- `.uid` files auto-generate — never hand-author.
|
||||
|
||||
## Verify what you created
|
||||
|
||||
- Single clips / packs: `scenes/animation_showcase.tscn` pages through every shipped clip.
|
||||
- Dances / composites: dance test bed; `VideoRecorder` (in-scene) records the viewport
|
||||
to `~/Downloads/*.mp4` for review — feed that to the `pose-estimation` skill to score
|
||||
a recreation against source footage (crop-zoom one dancer first; wide shots don't detect).
|
||||
|
||||
## Path 6 — Generate from video (pose-driven mocap, BUILT 2026-07-13)
|
||||
|
||||
`tools/mocap_retarget.py` turns `pose-estimation` skill output (extract_pose.py JSON,
|
||||
world landmarks) into a GLB clip on the Quaternius skeleton:
|
||||
|
||||
```bash
|
||||
/Applications/Blender.app/Contents/MacOS/Blender --background \
|
||||
--python tools/mocap_retarget.py -- \
|
||||
--pose <pose.json> --out assets/quaternius/mixamo/<pack>.glb --name <ClipName> \
|
||||
--range 166:192 --smooth 5 --render-check /tmp/check # renders frames to eyeball
|
||||
```
|
||||
|
||||
- Limb bones: shortest-arc aim from rest direction (preserves rest roll); pelvis/spine:
|
||||
full-basis deltas conjugated through char-frame alignment (same convention as
|
||||
mixamo_retarget.py — landmark space is subject-left=+X, up=+Z, facing=-Y, NOT identity).
|
||||
- MediaPipe world landmarks are hip-centered → no root motion; pelvis height is
|
||||
reconstructed from pelvis-above-ankle extent vs standing (carries squats).
|
||||
- Known v1 limits: no twist control, no fingers, no horizontal root motion.
|
||||
- Proven: `haka_mocap.glb` / `HakaFull` (26s, 208 keys) generated from a YouTube haka
|
||||
tutorial via `~/.claude/skills/pose-estimation`.
|
||||
|
||||
Full pipeline context: `.agents/plans/animation-gen-pipeline-2026-07-13.md`.
|
||||
@@ -0,0 +1,159 @@
|
||||
# blender-mcp recipes
|
||||
|
||||
Practical `bpy` snippets the agent pastes through blender-mcp's `execute-code` tool to
|
||||
author animations on the shared Quaternius skeleton and export them as game-ready GLB
|
||||
packs. These mirror the proven conventions in `tools/mixamo_retarget.py` — keep them in
|
||||
lockstep.
|
||||
|
||||
## Setup assumptions
|
||||
|
||||
- Blender **5.1.2** at `/Applications/Blender.app` (call headless as
|
||||
`/Applications/Blender.app/Contents/MacOS/Blender`).
|
||||
- The BlenderMCP addon is installed and **connected** (3D View → sidebar → BlenderMCP →
|
||||
Connect). If the `blender` MCP tool isn't available, the addon isn't connected.
|
||||
- **One game GLB imported** so the scene holds the real Quaternius armature. The game
|
||||
ships these as animations-only GLBs; import one to reuse its exact rig:
|
||||
|
||||
```python
|
||||
bpy.ops.import_scene.gltf(filepath="/Users/jeremykashkett/Tinqs/local.repo/ariki-game/assets/quaternius/kevin/kevin_male_social.glb")
|
||||
arm = next(o for o in bpy.data.objects if o.type == "ARMATURE")
|
||||
arm.name = "Armature" # match the convention mixamo_retarget.py enforces
|
||||
bpy.context.scene.render.fps = 30
|
||||
```
|
||||
|
||||
## Skeleton orientation
|
||||
|
||||
65 bones, UE-style names: `pelvis`, `spine_01`, `spine_02`, `spine_03`, `neck_01`,
|
||||
`Head` (**capital H**), `clavicle_l`, `upperarm_l`, `lowerarm_l`, `hand_l`, `thigh_l`,
|
||||
`calf_l`, `foot_l`, `ball_l`, and the `_r` twins, plus 3-segment fingers
|
||||
(`index_0{l,r}` … `pinky_03`, `thumb_0{l,r}` … `thumb_03`). The armature object is named
|
||||
`Armature`; work in pose mode via `bpy.context.object.pose.bones["<name>"]`.
|
||||
|
||||
## Keyframing recipe
|
||||
|
||||
Build an action from an explicit list of poses (name, frame, per-bone quaternions).
|
||||
Quaternius bones are authored in **quaternion** rotation mode, so set every pose bone's
|
||||
`rotation_mode = "QUATERNION"` first (this is what `mixamo_retarget.py` does):
|
||||
|
||||
```python
|
||||
import bpy
|
||||
from mathutils import Quaternion
|
||||
|
||||
arm = bpy.data.objects["Armature"]
|
||||
for pb in arm.pose.bones:
|
||||
pb.rotation_mode = "QUATERNION"
|
||||
|
||||
act = bpy.data.actions.new("Wave01")
|
||||
arm.animation_data_create()
|
||||
arm.animation_data.action = act
|
||||
bpy.context.scene.frame_start, bpy.context.scene.frame_end = 1, 30
|
||||
|
||||
# poses: (frame, {bone: (w,x,y,z)})
|
||||
poses = [
|
||||
(1, {"hand_l": (1,0,0,0), "lowerarm_l": (1,0,0,0)}),
|
||||
(15, {"hand_l": (0.7,0,0,0.7), "lowerarm_l": (0.9,0.3,0,0)}), # raise hand
|
||||
(30, {"hand_l": (1,0,0,0), "lowerarm_l": (1,0,0,0)}),
|
||||
]
|
||||
for frame, bones in poses:
|
||||
bpy.context.scene.frame_set(frame)
|
||||
for name, q in bones.items():
|
||||
pb = arm.pose.bones[name]
|
||||
pb.rotation_quaternion = Quaternion(q)
|
||||
pb.keyframe_insert(data_path="rotation_quaternion", frame=frame)
|
||||
```
|
||||
|
||||
## IK setup recipe
|
||||
|
||||
Drive `calf_l` (or `lowerarm_l`) toward a target empty with a 2-bone chain, then bake
|
||||
the solved motion to real keyframes and drop the constraint — the game never needs IK:
|
||||
|
||||
```python
|
||||
import bpy
|
||||
|
||||
arm = bpy.data.objects["Armature"]
|
||||
target = bpy.data.objects.new("IK_target_l", None) # empty
|
||||
bpy.context.collection.objects.link(target)
|
||||
target.location = arm.matrix_world @ arm.pose.bones["foot_l"].head + (0.0, 0.0, 0.2)
|
||||
|
||||
pb = arm.pose.bones["calf_l"]
|
||||
ik = pb.constraints.new("IK")
|
||||
ik.target = target
|
||||
ik.chain_count = 2 # thigh_l + calf_l
|
||||
ik.use_tail = True
|
||||
|
||||
# Bake the visual (constraint-solved) result to plain keyframes, then remove the constraint.
|
||||
bpy.ops.object.select_all(action="DESELECT")
|
||||
arm.select_set(True)
|
||||
bpy.context.view_layer.objects.active = arm
|
||||
bpy.ops.nla.bake(frame_start=1, frame_end=30, visual_keying=True,
|
||||
bake_types={"POSE"}, clean_curves=True)
|
||||
for c in list(pb.constraints):
|
||||
pb.constraints.remove(c)
|
||||
```
|
||||
|
||||
## NLA layering recipe
|
||||
|
||||
The glTF exporter emits each **NLA track** as a separate animation clip. To ship multiple
|
||||
clips in one GLB (or layer retargeted clips), push each action onto its own NLA track —
|
||||
this is the exact pattern `tools/mixamo_retarget.py` uses to export a pack:
|
||||
|
||||
```python
|
||||
import bpy
|
||||
arm = bpy.data.objects["Armature"]
|
||||
if arm.animation_data is None:
|
||||
arm.animation_data_create()
|
||||
for act in bpy.data.actions: # every action you want to ship
|
||||
tr = arm.animation_data.nla_tracks.new()
|
||||
tr.name = act.name # clip name = track name
|
||||
tr.strips.new(act.name, 1, act)
|
||||
arm.animation_data.action = None # avoid a stray duplicate clip
|
||||
```
|
||||
|
||||
To **blend** two clips on the stack (e.g. a base loop under a gesture), set the upper
|
||||
strip's `blend_type` (`"ADD"`, `"REPLACE"`) and `blend_in`/`blend_out` frames, then bake
|
||||
the combined result down to a single action (visual keying) and re-export that.
|
||||
|
||||
## Export recipe
|
||||
|
||||
The canonical export block lives in `tools/mixamo_retarget.py` and **must be mirrored
|
||||
exactly** — the game relies on these flags (`NLA_TRACKS` mode = one clip per track;
|
||||
`export_force_sampling` for clean per-bone channels; size optimization off to preserve
|
||||
retargeted curves). Select only the armature and drop meshes first so the GLB is
|
||||
animations-only (the game binds clips to its own body):
|
||||
|
||||
```python
|
||||
import bpy
|
||||
out = "/Users/jeremykashkett/Tinqs/local.repo/ariki-game/assets/quaternius/mixamo/wave_pack.glb"
|
||||
|
||||
# Drop meshes — animations-only GLB (mirrors mixamo_retarget.py).
|
||||
for o in [o for o in bpy.data.objects if o.type == "MESH"]:
|
||||
bpy.data.objects.remove(o, do_unlink=True)
|
||||
|
||||
bpy.ops.object.select_all(action="DESELECT")
|
||||
arm = bpy.data.objects["Armature"]
|
||||
arm.select_set(True)
|
||||
bpy.context.view_layer.objects.active = arm
|
||||
|
||||
bpy.ops.export_scene.gltf(
|
||||
filepath=out, use_selection=True,
|
||||
export_animations=True, export_animation_mode="NLA_TRACKS",
|
||||
export_force_sampling=True, export_optimize_animation_size=False)
|
||||
print("[blender-mcp] EXPORTED →", out)
|
||||
```
|
||||
|
||||
Output GLBs in `assets/quaternius/mixamo/` are auto-discovered by ClipCatalog /
|
||||
AnimationShowcase — referenceable immediately as `<pack_name>/<ClipName>` where
|
||||
`<ClipName>` is the NLA track (Blender action) name.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Never export `*RM` root-motion variants** for dances/composites — they translate the
|
||||
rig and break compositing. Author clips in place.
|
||||
- **Skip the `RESET`/rest action** — don't push it to an NLA track or it ships as a clip.
|
||||
- **Clip name = Blender action/NLA-track name.** The game refers to clips as
|
||||
`<pack>/<ClipName>` (e.g. `wave_pack/Wave01`).
|
||||
- **One armature per exported GLB.** If you imported a GLB that brought its own body
|
||||
meshes, delete them before export (see the export block).
|
||||
- **Match `fps=30`** so frame counts line up with shipped packs and the dance beat clock.
|
||||
- **Quaternion mode** for all pose bones before keyframing (matches the retargeter).
|
||||
- `.uid` files auto-generate in Godot — never hand-author them.
|
||||
@@ -0,0 +1,101 @@
|
||||
# gltf-transform recipes
|
||||
|
||||
Headless GLB surgery with gltf-transform (donmccurdy, gltf-transform.dev) — inspect,
|
||||
merge, transplant, and shrink animation packs with no Blender required. Same Quaternius
|
||||
skeleton across all game GLBs, so node names line up 1:1 and clip moves are mechanical.
|
||||
|
||||
## Install / run
|
||||
|
||||
CLI (no global install — `npx` pulls it on demand):
|
||||
|
||||
```bash
|
||||
npx @gltf-transform/cli --help
|
||||
npx @gltf-transform/cli inspect assets/quaternius/kevin/kevin_male_social.glb
|
||||
```
|
||||
|
||||
Or as a JS SDK from a node script (mirror the layout of the existing
|
||||
`tools/bake_run_punch.mjs`):
|
||||
|
||||
```js
|
||||
import { NodeIO } from "@gltf-transform/core";
|
||||
import { prune, dedup, resample } from "@gltf-transform/functions";
|
||||
const io = new NodeIO();
|
||||
const doc = io.read("path/to/file.glb");
|
||||
```
|
||||
|
||||
Install once into the repo's node tooling: `npm i -D @gltf-transform/core @gltf-transform/functions`
|
||||
(the orchestrator installs — you author the script only).
|
||||
|
||||
## Inspect clips
|
||||
|
||||
```bash
|
||||
# Full report: meshes, animations, channels, duration.
|
||||
npx @gltf-transform/cli inspect assets/quaternius/kevin/kevin_male_social.glb
|
||||
```
|
||||
|
||||
For a quick clip-name list without the SDK, the pure-node JSON-chunk one-liner already in
|
||||
`SKILL.md` (Path 1) reads the GLB's JSON chunk and prints `animations[].name` — use it to
|
||||
confirm a clip exists before importing/transplanting.
|
||||
|
||||
## Clip transplant between same-skeleton GLBs
|
||||
|
||||
Both files share the Quaternius skeleton, so every channel's target node matches by name.
|
||||
Sketch: read source + target, copy the source `Animation` into the target document, then
|
||||
re-point each channel's `targetNode` to the target-doc node with the same name.
|
||||
|
||||
```js
|
||||
// verify against gltf-transform v4 API before first use
|
||||
import { NodeIO } from "@gltf-transform/core";
|
||||
import { copyToDocument } from "@gltf-transform/functions";
|
||||
|
||||
const io = new NodeIO();
|
||||
const target = io.read("assets/quaternius/mixamo/northern_soul.glb"); // pack to extend
|
||||
const source = io.read("downloads/clip_to_add.glb"); // same skeleton
|
||||
|
||||
const srcAnim = source.getRoot().listAnimations()
|
||||
.find(a => a.getName() === "NewClip");
|
||||
const targetNodes = new Map(target.getRoot().listNodes().map(n => [n.getName(), n]));
|
||||
|
||||
copyToDocument(target, source, [srcAnim]); // copy the Animation property
|
||||
const dstAnim = target.getRoot().listAnimations().at(-1);
|
||||
for (const ch of dstAnim.listChannels()) {
|
||||
const srcNode = ch.getTargetNode();
|
||||
if (srcNode) ch.setTargetNode(targetNodes.get(srcNode.getName())); // re-point by name
|
||||
}
|
||||
io.write("assets/quaternius/mixamo/northern_soul.glb", target); // overwrites — back up first
|
||||
```
|
||||
|
||||
Common variations: copy **all** source animations (iterate `listAnimations()`), rename on
|
||||
the way in (`dstAnim.setName("Wave01")`), or strip a clip (remove its channels then
|
||||
`prune`). If `copyToDocument` isn't the right entry point in your installed version, fall
|
||||
back to walking `srcAnim.listChannels()` and rebuilding each (sampler interpolation +
|
||||
input/output accessors + targetNode/path) on a fresh `target.createAnimation()`.
|
||||
|
||||
## Shrink / clean
|
||||
|
||||
Standard transforms, applied in this order for pack size:
|
||||
|
||||
```bash
|
||||
npx @gltf-transform/cli optimize <in.glb> <out.glb> # resample + prune + dedup combined
|
||||
# or individually:
|
||||
npx @gltf-transform/cli <in.glb> <out.glb> resample # drop redundant keyframes
|
||||
npx @gltf-transform/cli <in.glb> <out.glb> prune # remove unreferenced data
|
||||
npx @gltf-transform/cli <in.glb> <out.glb> dedup # merge identical accessors/meshes
|
||||
```
|
||||
|
||||
In the SDK: `await doc.transform(prune(), dedup(), resample())`. `resample` takes an
|
||||
optional `tolerance`/`ratio` to trade fidelity for size — retargeted clips have clean
|
||||
curves and tolerate aggressive resampling; mocap-sourced clips do not.
|
||||
|
||||
## When to prefer this over Blender
|
||||
|
||||
Prefer gltf-transform when there's **no new motion** to author:
|
||||
|
||||
- Same-skeleton clip moves (one clip from a downloaded GLB into a shipped pack).
|
||||
- Batch renames (fix a pack of `*_Loop` suffixes, normalize clip names).
|
||||
- Pack merges / splits (combine two `assets/quaternius/mixamo/` packs).
|
||||
- Headless size passes before shipping (resample/prune/dedup).
|
||||
|
||||
Reach for **Blender (Path 2 retarget / Path 5 authoring)** when the source uses a
|
||||
*different* skeleton (needs the world-rotation-delta retarget), or when you must author,
|
||||
IK-solve, or layer new motion that exists nowhere.
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
name: iclone-video-mocap
|
||||
description: Advise on Reallusion iClone 8 and the Video Mocap plugin (AI body mocap from video) — filming rules, generation settings, mocap cleanup (Motion Correction, Curve Editor), face layering with AccuFACE, FBX export, and integrating the result into ariki-game via tools/cc_retarget.py. Use when the user asks how to do anything in iClone/Character Creator, wants to record video for mocap, or wants an iClone export playing in the game.
|
||||
---
|
||||
|
||||
# iClone 8 + Video Mocap (advisory + ariki-game integration)
|
||||
|
||||
Jeremy uses **iClone 8** (Windows) with the **iClone Video Mocap plugin** — Reallusion's cloud AI body-mocap from ordinary video (QuickMagic-powered, launched Oct 2025, requires iClone 8.63+). This skill answers "how do I do X in iClone" and "how do I get it into the game".
|
||||
|
||||
**Reference files (read the one that matches the question):**
|
||||
- `references/video-mocap-plugin.md` — Video Mocap product facts, pricing (~$2.50–3/generation, points), full 5-stage workflow, filming rules, known bugs, AccuFACE.
|
||||
- `references/iclone8-core-animation.md` — motion clips/layer keys, Edit Motion Layer (IK/pins), Reach Target, timeline ops, Curve Editor filters, Motion Correction, face animation, characterization, FBX dialog options, hotkeys.
|
||||
- `references/export-pipeline-to-blender-godot.md` — iClone/CC4 FBX+BVH export details, soupday Blender Tools, retargeting to simple rigs, FBX→GLB→Godot, pitfalls checklist.
|
||||
- `references/ariki-integration.md` — **the game-side path**: `tools/cc_retarget.py` (CC_Base_→Quaternius), ClipCatalog discovery, dance JSON, end-to-end recipe.
|
||||
|
||||
## The 60-second mental model
|
||||
|
||||
- iClone animation = **Motion Clip + Layer Keys** (layer keys live inside the clip and can't leave it). Pose edits go through **Edit Motion Layer** (hotkey N, HumanIK IK/FK, pinning). Foot/hand locking via **Reach Target**; foot-slide fix via right-click clip > **Motion Correction** (flatten between iterations). Jitter fix: right-click clip > **Sample Motion Clip to New Layer**, then Curve Editor (F12) **Smooth / Butterworth / Optimize** filters.
|
||||
- Video Mocap = separate plugin panel: import video → trim to ≤60 s region → **Full Body / Upper Body** (+ Includes Fingers) → cloud generate (5–8 min, costs points) → **Apply Motion** to selected character (check "Apply reference video" for rotoscope cleanup) → clip lands in Content Manager > Motion > VideoMocap as rlMotion. **Body only — no face**; layer AccuFACE/LIVE FACE for talking.
|
||||
- System: iClone is **60 fps internal, cm units, Z-up**; CC3+ characters bind in **A-pose**; FBX axis conversion comes from the **Target Tool Preset**. Constraint keys (Reach/Link/Look At/Path) need **Animation > Flatten All Motion with Constraint** before export.
|
||||
|
||||
## Filming rules that make or break Video Mocap (advise BEFORE recording)
|
||||
|
||||
Eye-level static camera, single continuous shot; whole actor in frame the entire time; one person; textured clothing that contrasts the background; avoid flips, jumps, fast spins, hand-occluding poses. Money-saver: **stack several motions into one ≤60 s take with ~2 s gaps** — one generation fee, split into clips in iClone.
|
||||
|
||||
## Getting it into ariki-game (short version)
|
||||
|
||||
```bash
|
||||
/Applications/Blender.app/Contents/MacOS/Blender --background \
|
||||
--python tools/cc_retarget.py -- \
|
||||
--src "<iclone-export.fbx>" --out assets/quaternius/dancegen/<pack>.glb --name <ClipName>
|
||||
```
|
||||
|
||||
Export from iClone with preset **Blender**, range **All**, **Preserve Bone Names (CC Base)** ON. The tool maps `CC_Base_*` → the shared 65-bone Quaternius skeleton, strips the mesh, and the GLB is auto-discovered by ClipCatalog as `<pack>/<ClipName>` (dirs: kevin/, mixamo/, dancegen/). Details, dance-JSON format, and the **unvalidated A-pose caveat** (first export needs an arm-orientation QA; workaround: FBX Advanced > Use T-Pose As Bind Pose): `references/ariki-integration.md`.
|
||||
|
||||
## Cross-cutting gotchas
|
||||
|
||||
- iClone 8 has **no animation-only FBX export** (CC4 does: FBX Options > Motion) and **no BVH export** (CC4 4.4+ does). No native glTF anywhere — route through Blender or Godot's ufbx.
|
||||
- Marketplace content needs a per-pack **Export License** to leave via FBX/USD; your own recordings/characters don't.
|
||||
- Video Mocap: keep plugin ≥1.02 (pre-1.02 double-charged points on re-open); Loop-enabled clips silently discard motion-layer edits — Flatten first.
|
||||
- MixMoves is gone in iClone 8 (Motion Director replaced it); AccuLips only works on CC-Standard/Gamebase characters.
|
||||
- ariki side: ASK Jeremy before launching the game; never use `*RM`/root-motion clip variants in dances.
|
||||
@@ -0,0 +1,75 @@
|
||||
# ariki-game — Integrating iClone Animation Exports
|
||||
|
||||
How motion authored in iClone 8 (incl. Video Mocap output) lands on ariki-game's shared Quaternius skeleton. Mapped 2026-07-15 from the repo.
|
||||
|
||||
**Headline: the retargeter already exists — `tools/cc_retarget.py` (created 2026-07-14) handles `CC_Base_*` (Reallusion CC/iClone) FBX → Quaternius GLB.** The pipeline is: iClone FBX export → `cc_retarget.py` → mesh-stripped GLB in an auto-discovered pack dir → referenceable as `<pack>/<ClipName>` in dances/showcase.
|
||||
|
||||
## 1. The shared Quaternius skeleton (retarget target)
|
||||
|
||||
- **Canonical rig:** `assets/quaternius/base-characters/Universal Base Characters[Standard]/Base Characters/Godot - UE/Superhero_Male_FullBody.gltf` — all retargeters default to it (`--target` overrides). `kevin_retarget.py:26` is the exception (hardcoded absolute path, non-portable).
|
||||
- **65 bones, UE-style names, T-pose rest.** Spine: `root, pelvis, spine_01..03, neck_01, Head` (capital H). Per side: `clavicle_, upperarm_, lowerarm_, hand_, thigh_, calf_, foot_, ball_, ball_leaf_` + fingers `thumb/index/middle/ring/pinky_01..03_` with `_04_leaf_` tips.
|
||||
|
||||
## 2. ClipCatalog discovery (`src/Testing/Dance/ClipCatalog.cs`)
|
||||
|
||||
- Hardcoded packs (lines 20-25): UAL1, UAL2, Run_Punch.
|
||||
- Auto-discovered dirs (lines 37-45): `res://assets/quaternius/kevin`, `res://assets/quaternius/mixamo`, `res://assets/quaternius/dancegen` — drop a GLB in one and it's live; **pack name = GLB basename**.
|
||||
- Clip ref format: `"<pack>/<ClipName>"` (e.g. `dancegen/HakaFull`).
|
||||
- Lazy extraction (lines 80-118): first `AnimationPlayer` in GLB scene, skips `RESET`, forces `LoopMode.Linear`, caches. Name fallback: `exact → +"_Loop" → -"_Loop"`.
|
||||
- Gotcha baked into the code (lines 63-64): `DirAccess.GetNext()` returns `""` at end, not null — null check pegs CPU.
|
||||
|
||||
## 3. `tools/cc_retarget.py` — the iClone/CC ingestion tool
|
||||
|
||||
Blender 5 headless, same world-rotation-delta method as `mixamo_retarget.py`:
|
||||
|
||||
```bash
|
||||
/Applications/Blender.app/Contents/MacOS/Blender --background \
|
||||
--python tools/cc_retarget.py -- \
|
||||
--src "<dir-or-fbx>[,<more>...]" --out assets/quaternius/dancegen/<pack>.glb \
|
||||
[--name <ClipName>] [--target <gltf>]
|
||||
```
|
||||
|
||||
- **BONE_MAP (lines 25-48):** `CC_Base_Hip→pelvis`, `CC_Base_Waist→spine_01`, `CC_Base_Spine01→spine_02`, `CC_Base_Spine02→spine_03`, `CC_Base_NeckTwist01→neck_01`, `CC_Base_Head→Head`; limbs `CC_Base_L_Clavicle/Upperarm/Forearm/Hand/Thigh/Calf/Foot/ToeBase → clavicle_l/upperarm_l/lowerarm_l/hand_l/thigh_l/calf_l/foot_l/ball_l`. **Thumbs 3-seg, other fingers 1-seg** (`CC_Base_L_Index1→index_01_l` etc. — game-optimized CC rigs have one segment per finger). Twist helpers (`UpperarmTwist01…`) unmapped → ride parent rigidly.
|
||||
- **Char-frame conjugation:** `q_delta = C @ (q_src·q_src_rest⁻¹) @ C⁻¹` with `C = char_frame(target) @ char_frame(source)⁻¹`; source frame from `CC_Base_Hip`/`CC_Base_Head`/`CC_Base_L_Upperarm` (lines 128-129).
|
||||
- **Units/scale:** `unit = src_arm.scale.x` captured BEFORE `transform_apply` (importer's cm→m 0.01); pelvis = world-space delta `× tscale × unit` (lines 176-177). Handles iClone's cm natively.
|
||||
- **fps 60** (only tool at 60 — "iClone exports at 60; glTF export resamples fine").
|
||||
- **Motion-presence guard** (`pick_action`, lines 89-105): picks the longest action whose fcurves reference `CC_Base_` bones; **rejects `TempMotion` <5-frame placeholders** with error "FBX exported WITHOUT motion — Re-export with Include Motion".
|
||||
- Export tail (shared with all retargeters): drop MESH objects, one NLA track per action (track name = clip name = glTF animation name), `export_scene.gltf(export_animation_mode="NLA_TRACKS", export_force_sampling=True, export_optimize_animation_size=False)`.
|
||||
|
||||
### ⚠ Open validation item: A-pose
|
||||
The tool's comments claim "CC rigs rest in T-pose, verified on ActorBuild exports", but iClone/CC3+ characters bind in **A-pose**. The char-frame delta math measures off actual rest and should self-cancel — but the left-arm char-frame axis (`CC_Base_L_Upperarm` bone-Y) points down-and-out in A-pose, not straight +X. **QA the first A-pose export**: if arms bake rotated, fix the char-frame or add a per-arm rest-alignment quaternion. This is the one genuine divergence vs the proven Mixamo (T-pose) path. Alternatively force T-pose on the iClone side: FBX Advanced Settings > **Use T-Pose As Bind Pose**.
|
||||
|
||||
## 4. Recommended iClone export settings for this pipeline
|
||||
|
||||
- iClone: `File > Export > Export Fbx...`, Target Tool Preset **Blender** (or Unity 3D), Export Range **All**/**Range**, FPS `Project(60)` fine (tool handles it). Advanced: **Preserve Bone Names (CC Base)** ON (default — the bone map depends on it), consider **Use T-Pose As Bind Pose** (see §3 caveat), **Reset Bone Scale** ON.
|
||||
- Run **Menu > Animation > Flatten All Motion with Constraint** first if the take uses Reach/Link/Look At/Path — those keys don't survive FBX.
|
||||
- iClone 8 always exports mesh+animation (no motion-only mode); `cc_retarget.py` strips the mesh anyway, so don't fight it — but uncheck Embed Textures / set Max Texture Size low to keep the FBX small.
|
||||
- Animation-only alternative: export via CC4 (`FBX Options: Motion`) — the retargeter still needs a rest pose, and Blender only gets bind pose from a weighted mesh, so prefer the iClone mesh+motion export.
|
||||
- Store-content license wall: Reallusion marketplace motions/characters need a per-pack **Export License** to leave via FBX; your own Video Mocap recordings and characters are unrestricted.
|
||||
|
||||
## 5. Runtime consumers
|
||||
|
||||
- **AnimationShowcase** (`src/Testing/AnimationShowcase.cs`, `scenes/animation_showcase.tscn`): pages all clips. `SCENE=animation_showcase bash tools/game.sh spawn` (game.sh:926). ASK Jeremy before launching the game.
|
||||
- **Dance test bed** (`scenes/dance_test_bed.tscn`, `src/Testing/Dance/DanceTestBed.cs:118-124`): hot-reload dances via `DanceLibrary.Poll`; in-scene `VideoRecorder` → `~/Downloads/*.mp4` at 30fps. `SCENE=dance_test_bed bash tools/game.sh spawn` (game.sh:927).
|
||||
- **Dance JSON** (`assets/dances/*.json`): `{ "name", "bpm", "moves": [...] }`; move = `{ "clip": "pack/Clip", "loops": N }` or composite `{ "layers": [{"mask":"upper","clip":...},{"mask":"lower","clip":...}], "duration": s }`. Transitions quantize to the BPM beat clock. Examples: `haka_mocap.json` (uses `haka_mocap/HakaFull`), `clap_stomp.json` (110), `war_challenge.json` (120).
|
||||
- **CompositeClipBaker.cs**: lower mask = `root|pelvis|thigh*|calf*|foot*|ball*` (always owns hip pos+rot — prevents hip-fight/foot-slide); upper = rest; first-writer-wins per track. Cache keyed by definition → JSON edits need no invalidation.
|
||||
- **Hard rules:** skeleton path remap mandatory for raw GLB clips (GLB tracks target `Armature/Skeleton3D:bone`; derive real prefix from an already-remapped clip — `PlayerController.LoadUalLibrary` is canonical). **Never use `*RM`/`*RootMotion` clip variants** — everything is in-place.
|
||||
|
||||
## 6. Godot import conventions
|
||||
|
||||
- `.import` sidecars: `importer="scene"`, **`animation/fps=30`** (even 60fps cc output resamples to 30 at import), `animation/remove_immutable_tracks=true`, `animation/import_rest_as_RESET=false`, `meshes/generate_lods=false`. `.uid` files auto-generate — never hand-author.
|
||||
- Commit GLB + let Godot import → immediately referenceable as `<pack>/<ClipName>`.
|
||||
|
||||
## 7. Precedents
|
||||
|
||||
- **Kevin Iglesias**: 199 Unity FBX (`B-*` rig) → 12 packs in `assets/quaternius/kevin/` via `kevin_retarget.py` batch — the batch-directory template.
|
||||
- **Video-to-mocap (free, local)**: `extract_pose.py` (MediaPipe) → `mocap_retarget.py` (rotation-solving, no source armature) → `dancegen/haka_mocap.glb` / `HakaFull`. iClone Video Mocap is the paid, higher-quality alternative feeding the same `cc_retarget.py` path.
|
||||
- Four bone-map dicts exist: `kevin_retarget.py:29`, `mixamo_retarget.py:26`, `cc_retarget.py:25`, `mocap_retarget.py` AIM_BONES.
|
||||
|
||||
## 8. End-to-end recipe (iClone Video Mocap → in-game dance)
|
||||
|
||||
1. Film per the Video Mocap guidelines (eye-level static camera, full body, single actor, textured clothing; stack takes with 2s gaps into one ≤60s region).
|
||||
2. Generate in the plugin (Full Body [+ Includes Fingers]), apply to a CC character, clean up in iClone (Motion Correction for feet, Curve Editor smooth/Butterworth for jitter — see the other references).
|
||||
3. Export FBX per §4.
|
||||
4. `cc_retarget.py --src <fbx> --out assets/quaternius/dancegen/<pack>.glb --name <ClipName>`.
|
||||
5. Verify: open in animation_showcase or reference `<pack>/<ClipName>` from a dance JSON in `assets/dances/` (hot-reloads in the dance test bed).
|
||||
6. First A-pose export: eyeball arm orientation (§3 caveat) before batching.
|
||||
@@ -0,0 +1,108 @@
|
||||
# iClone 8 / CC4 → Blender → Godot Animation Export Pipeline Reference
|
||||
|
||||
Compiled 2026-07-15 from Reallusion manuals, soupday docs, Godot docs, Reallusion forum. Claims not confirmed against a primary source are marked [UNVERIFIED].
|
||||
|
||||
## 1. iClone 8 FBX Export
|
||||
|
||||
**Menu path:** `Menu > File > Export > Export Fbx...` (select one or more avatars/props/accessories/cameras first). iClone 8 ships as a single edition — no separate "Pipeline" SKU and no 3DXchange needed; FBX/OBJ export is built in.
|
||||
|
||||
**Export dialog options (exact names, iClone 8 manual):**
|
||||
- **Target Tool Preset** — dropdown, auto-configures Advanced Settings per DCC. Supported-tools list is rendered as an image in the manual page (not text-extractable); known presets include 3ds Max, Blender, Cinema 4D, Maya, MotionBuilder, Unity, Unreal [UNVERIFIED exact/full list].
|
||||
- **Export Range**: `T-Pose` | `Current Frame` | `All` | `Range` — this is the mesh-vs-animation control. `T-Pose`/`Current Frame` = static export; `All`/`Range` = baked animation.
|
||||
- **Export Animation FPS** — free field / dropdown to resample export frame rate.
|
||||
- **Embed Textures** — embed in FBX vs. export as separate files.
|
||||
- **Max Image Size** — cap texture resolution (default: original size).
|
||||
- **Convert Image Format to** — force texture format (default JPG).
|
||||
- **Delete Hidden Face** — strips occluded mesh.
|
||||
- **Use Smooth Mesh** — export subdivided/smoothed surface.
|
||||
- **Merge Face Hair to One Object**, **Tear Ducts UV Adjustment** — face-mesh consolidation options.
|
||||
- **Embed Timeline Timecode**.
|
||||
- **Convert Skinned Expressions to Morphs** — bakes skinned (bone-driven) facial expressions into morph targets so faces survive in non-Reallusion tools. Important for facial anim in Godot.
|
||||
- **Delete Unused Morphs** — drops unkeyed blendshapes (smaller files; disable if you need the full ARKit/viseme set at runtime).
|
||||
- **Advanced Settings** panel — per-tool fixups; for the Blender template this includes **Rename Duplicated Nodes** and **Reset Bone Offset** (needed so bones behave in Blender Object/Pose/Edit modes).
|
||||
|
||||
**Animation-only export:** iClone 8 has **no** "export animation only" FBX mode — mesh/materials always come along (confirmed by forum threads and Feedback Tracker "IClone 8 has limited FBX export options compared to CC4"). Workarounds: export via CC4 (which has `FBX Options: Motion`), or import the full FBX in Blender/soupday tools and discard meshes. For constraint-based animation (Link/Look At/Attach/Path/Reach), run `Menu > Animation > Flatten All Motion with Constraint` before export or those keys are lost.
|
||||
|
||||
**Not exportable as FBX:** iLight, iEffect, iParticle, iTree, iGrass, heightmap iTerrain, iWater, iSky, aml, iTalk, iPath.
|
||||
|
||||
**Export License gating:** The *software* does not gate FBX export; the *content* does. Reallusion marketplace/Content Store items require a purchased **Export License** per pack to leave the ecosystem; without it you get "You must first purchase the content licenses in order to export FBX and USD files." Your own characters and content flagged export-ready export freely. The Content EULA restricts exporting iContent to FBX/OBJ etc. without that license.
|
||||
|
||||
Sources: https://manual.reallusion.com/iClone-8/Content/ENU/8.0/83-FBX/FBX_UI_Intro.htm , https://manual.reallusion.com/iClone-8/Content/ENU/8.0/83-FBX/Exporting_FBX_from_iClone.htm , https://forum.reallusion.com/516723/export-animation-only-to-fbx-with-iclone-8/Forum445.aspx , https://www.reallusion.com/license/content.html , https://forum.reallusion.com/556427/
|
||||
|
||||
## 2. CC4 FBX Export (the better animation exit door)
|
||||
|
||||
**Menu path:** `File > Export > FBX (Clothed Character)`.
|
||||
|
||||
**Dialog options:**
|
||||
- **Target Tool Preset** dropdown (auto-sets Advanced Settings): Blender, Unity, Unreal, Maya, 3ds Max, etc.
|
||||
- **FBX Options**: `Mesh` | `Motion` | `Mesh and Motion` — `Motion` is the true animation-only export (armature + animation, no character mesh). Separate checkbox **Export Mesh and Motion Individually** writes mesh FBX + one FBX per motion.
|
||||
- **Include Motion**: `Calibration` (T-pose calibration motion) | `Current Pose` | `Custom` (queue .iMotion/.iMotionPlus clips; MotionPlus carries facial + body). Disabled when Export Type = Mesh only.
|
||||
- **Frame rate**: dedicated page "Setting Frame Rate for Exporting FBX" — resample dropdown/manual entry.
|
||||
- **First Frame in Bind-Pose** — inserts bind pose (T-pose for CC1/CC3 Base, **A-pose for CC3+ characters**) one frame ahead of the motion. Critical for retargeting tools that read frame 0 as rest pose.
|
||||
- Texture size/format, embed toggle; **Delete Hidden Mesh**; **Merge Beard and Brows into one object**; **Bake diffuse and specular maps from Digital Human Hair Shader**; **Bake diffuse maps from skin color**; **Use Subdivided Mesh**; InstaLOD/Remesher/Merge Material for game LODs.
|
||||
|
||||
Sources: https://manual.reallusion.com/Character-Creator-4/Content/ENU/4.1/17-Export/Export-FBX.htm , https://manual.reallusion.com/Character-Creator-4/Content/ENU/4.0/17_Export/Exporting_Mesh_and_Motion_Individually.htm , https://manual.reallusion.com/Character-Creator-4/Content/ENU/4.0/17_Export/Setting_Frame_Rate_for_FBX.htm , https://manual.reallusion.com/Character-Creator-4/Content/ENU/4.0/17_Export/Using-First-Fram-in-Bind-Pose-Feature.htm
|
||||
|
||||
## 3. BVH Export
|
||||
|
||||
- **CC4 4.4+ has native BVH export**: `File > Export > BVH`. Options: Prefix/Suffix; bone selection with designated **motion root** (map to Hips); **Target Tool Preset**; **Rename Duplicated Nodes** + **Reset Bone Offset** (Blender compat); **Frame Rate** (sampling); Include Motion: `Calibration` | `Current Pose` | `Current Animation` | `Custom` (batch via Load Perform/Open File). Supports .iMotion/.iMotionPlus/.rlMotion/.rlPose. Known failures exporting motions retargeted onto non-standard (e.g. Daz G8) skeletons.
|
||||
- **iClone 8:** no BVH exporter documented in the iClone 8 manual [UNVERIFIED — historical `File > Export to Other 3D Format > Export BVH` docs are 3DXchange5-era]. Practical path: send motion to CC4 and BVH-export there, or export FBX and convert in Blender.
|
||||
- Both tools **import** BVH (Convert External Motions → iMotion).
|
||||
|
||||
Sources: https://manual.reallusion.com/Character-Creator-4/Content/ENU/4.4/17-Export/Exporting-Motions-to-BVH.htm , https://discussions.reallusion.com/t/bvh-export-fails-from-cc4/13056 , https://manual.reallusion.com/iClone-8/Content/ENU/8.4/50-Animation/Import-External-Motions/Convert-External-Motions.htm
|
||||
|
||||
## 4. USD / glTF
|
||||
|
||||
- **USD:** via the free **Omniverse Connector plugin** (iClone must be 8.3+). After install, `Plugins > Omniverse` exposes **Export USD** plus Live Sync (one-click whole-project or per-item two-way transfer). CC4 has an equivalent USD/Omniverse export [UNVERIFIED exact CC4 menu]. USD export of store content is also Export-License-gated.
|
||||
- **glTF/GLB:** **no native glTF export** in iClone 8 or CC4 [UNVERIFIED as absolute, but no manual page exists]. Standard route: FBX → Blender → glTF exporter, or FBX → FBX2glTF/ufbx inside Godot. Community tutorials (Freedom Arts "CC4 Export to GLB and GLTF with Blender") all route through Blender.
|
||||
|
||||
Sources: https://manual.reallusion.com/Omniverse-Plug-in/Content/ENU/iC-8.3/01-Installation/For-IC-Users.htm , https://docs.omniverse.nvidia.com/connect/iclone.html , https://www.classcentral.com/course/youtube-character-creator-4-export-to-glb-gltf-with-blender-cc4-pipeline-tutorial-blender-310648
|
||||
|
||||
## 5. soupday CC/iC Blender Tools (cc_blender_tools)
|
||||
|
||||
Add-on (a.k.a. "CC/iC Pipeline") for importing/auto-setup of CC3/CC4 + iClone 7/8 exports into Blender. Docs explicitly warn: **"Do Not use the standard blender FBX importer"** for CC/iC characters.
|
||||
|
||||
**Recommended export settings** — CC4: preset `Blender`, `FBX (Clothed Character)`, FBX Options `Mesh and Motion`, Include Motion `Current Pose`, check `Delete Hidden Faces`, textures NOT embedded. iClone: `File > Export > Export FBX`, target `Blender`, range `All`, check `Embed Textures` + `Delete Hidden Faces`.
|
||||
|
||||
**Features:**
|
||||
- **Import Character** button: auto-builds full material setup (from the sidecar `.json`), optional toggles for Lighting, Physics, Wrinkles, Rigify; `Import Animation` checkbox brings motions in as Blender Actions.
|
||||
- **Import Animations** button (Retargeting section): imports armature + actions **including shape-key actions** from extra FBX files, discarding meshes/materials.
|
||||
- **Rigify** button (Rigging and Animation panel): one-click Rigify control rig, optional **Full Face Rig**; original CC skeleton is kept (hidden) as deform rig.
|
||||
- **Retargeting**: sources supported — Character Creator, ActorCore, GameBase, iClone, **Mixamo (X/Y Bots)**, "General Humans". Workflow: pick Source Armature + action → **Preview Retarget** (live tweak arm spread, leg spacing, heel/toe angle, root height) → **Bake Retarget** (new action named `RigifiedChar | <Type> | ClipName`) → **Retarget Shape Keys** copies facial/viseme shape-key animation onto every mesh with matching shape keys. iClone facial performances arrive as "iCTM" temp-motion actions. NLA editor recommended for stacking body + shape-key actions.
|
||||
- **Round-trip:** `Export to CC3/4` button requires the **FbxKey** from the original CC export; writes back JSON material params/textures for lossless re-import into CC. **Export to Unity** (.fbx or .blend); after Rigify, three export modes: mesh-only, armature+animation-only, or both.
|
||||
|
||||
Sources: https://github.com/soupday/cc_blender_tools , https://soupday.github.io/cc_blender_tools/usage.html , https://soupday.github.io/cc_blender_tools/animation.html , https://soupday.github.io/cc_blender_tools/adv-animation.html , https://soupday.github.io/cc_blender_tools/pipeline.html
|
||||
|
||||
## 6. CC Skeleton & Retargeting to Other Rigs
|
||||
|
||||
- **Naming convention:** every bone prefixed `CC_Base_` — e.g. `CC_Base_BoneRoot`, `CC_Base_Hip`, `CC_Base_Waist`, `CC_Base_Spine01/02`, `CC_Base_NeckTwist01/02`, `CC_Base_Head`, `CC_Base_L_Clavicle/L_Upperarm/L_Forearm/L_Hand`, `CC_Base_L_Thigh/L_Calf/L_Foot/L_ToeBase`, plus twist chains (`CC_Base_L_UpperarmTwist01/02`, `ForearmTwist`, `ThighTwist`, `CalfTwist`) and full finger chains [names verified via CC4 Bone List/Bone Manager docs and community presets; exact full hierarchy: see CC4 Bone List F3 panel]. GameBase conversion produces a simplified skeleton with different names (breaks soupday Rigify — see issue #60).
|
||||
- **Not Mixamo-named** (`mixamorig:` ≠ `CC_Base_`); retargeting is by humanoid mapping, not name matching. soupday retarget handles Mixamo→CC-Rigify directly.
|
||||
- **Blender retarget options:** (a) soupday Rigify retarget (best for CC targets); (b) **Auto-Rig Pro** Remap — community bone-list preset "Character Creator 4 to Auto-Rig Pro IK" exists (gist by muratagawa); ARP is the commonly recommended robust option; (c) **Rokoko Studio Live** addon Retargeting tab — select source/target armature → **Build Bone List** → fix mappings → retarget; a ready-made CC3 bone-list JSON is sold/shared on Gumroad; addon has Blender-version lag issues.
|
||||
- **To a Quaternius-style simple rig:** map only the ~20 core humanoid bones (Hips←CC_Base_Hip, Spine←Spine01/02 collapsed, Head, UpperArm/LowerArm/Hand, UpperLeg/LowerLeg/Foot); ignore twist bones and finger chains, or bake twist rotation into parent limb bones before transfer. Rest-pose difference (CC A-pose vs Quaternius T-pose) must be corrected with a rest-pose alignment step (ARP "Redefine Rest Pose" / Rokoko auto rest-pose matching) or shoulders will bake ~45° off.
|
||||
|
||||
Sources: https://manual.reallusion.com/Character-Creator-4/Content/ENU/4.0/04_Introducing_the_User_Interface/Bone-List.htm , https://gist.github.com/muratagawa/ed07632bdd07bdec442fa194821b92b9 , https://github.com/Rokoko/rokoko-studio-live-blender , https://support.rokoko.com/hc/en-us/articles/4410463481489 , https://github.com/soupday/cc_blender_tools/issues/60
|
||||
|
||||
## 7. FBX → GLB → Godot
|
||||
|
||||
**Blender leg:**
|
||||
- CC FBX is authored in **centimeters**; Blender's FBX importer compensates with **armature object scale 0.01** (and Edit Mode shows 0.01 scaling). Fix: use soupday importer, or apply transforms (select armature+meshes → `Object > Apply > All Transforms`) before doing anything else; the classic manual fix is scale ×100 → Apply Scale → ×0.01 dance when round-tripping to engines.
|
||||
- Standard importer bone options: `Automatic Bone Orientation` + `Force Connect Children` make bones viewable but **alter bone rolls/joint rotations** — avoid if you will export animation back out; CC bones have zero-length/leaf-bone issues that iClone's "Reset Bone Offset" advanced option mitigates.
|
||||
- **GLB export (File > Export > glTF 2.0):** glTF is Godot's recommended format. Key settings: `Data > Armature > Export Deformation Bones Only` = ON (required when meshes have shape keys, prevents shading errors; also strips Rigify control bones), Include `Shape Keys` + `Shape Key Animations` [UNVERIFIED exact checkbox names per Blender version], Animation mode `Actions` or push actions to **NLA tracks** so every clip exports as a named animation; `+Y Up` handled automatically; enable Backface Culling on materials. If exporting a Rigify character, bake action to the deform (CC_Base_) bones first or export with "Export Deformation Bones Only".
|
||||
|
||||
**Godot leg:**
|
||||
- **Godot 4.3+ imports .fbx natively via ufbx** — no external converter; handles unit conversion (cm→m applied to mesh data, not node scale), no extraneous RootNode. Pre-4.3 / legacy files use **FBX2glTF** (requires downloaded binary; per-file importer switch in Import dock; switching importers changes node hierarchy, rest poses, surface order). Direct iClone-FBX → Godot-ufbx works but skips Blender-side cleanup (material rebuild, morph naming) [UNVERIFIED quality for CC characters specifically].
|
||||
- GLB path is still the most predictable: animations arrive in an `AnimationPlayer`, blendshapes as mesh blend-shape tracks. Import dock: set `Animation > FPS` (default 30 — raise to 60 for iClone-sourced clips to avoid resampling loss [Godot 4 samples at import]), loop suffixes (`-loop`/`-cycle` name suffix or import options), and use **Root Motion** by assigning the hip/root bone as `root_motion_track` in AnimationTree.
|
||||
|
||||
Sources: https://godotengine.org/article/introducing-the-improved-ufbx-importer-in-godot-4-3/ , https://docs.godotengine.org/en/stable/tutorials/assets_pipeline/importing_3d_scenes/available_formats.html , https://docs.godotengine.org/en/stable/classes/class_editorsceneformatimporterfbx2gltf.html , https://blenderartists.org/t/bringing-armature-into-edit-mode-scales-it-to-0-01-no-matter-what-i-do/1485346
|
||||
|
||||
## 8. Common Pitfalls Checklist
|
||||
|
||||
1. **60 fps:** iClone's realtime engine/timeline runs at 60 fps [UNVERIFIED whether iClone 8 allows changing project fps]; set **Export Animation FPS** deliberately and match Godot's import bake FPS, or clips get resampled twice.
|
||||
2. **cm units / 0.01 scale:** CC/iClone FBX = centimeters. Blender import → 0.01-scaled armature; apply transforms before retarget/export or child meshes, physics, and GLB scale drift. Godot ufbx converts units into mesh data automatically.
|
||||
3. **A-pose vs T-pose:** CC3+ (CC4 default) characters' **bind pose is A-pose**; CC1/CC3 Base is T-pose. The Export dialog's T-pose option only *prepends a T-pose frame* — the skin bind pose stays A-pose (Feedback Tracker: "Exported FBX character stays in A pose when I select T pose"). Use **First Frame in Bind-Pose** so retargeters can read frame 0; align rest poses before baking onto T-pose target rigs.
|
||||
4. **Bind pose only appears with animation:** importing CC FBX into Blender *without* animation shows bind pose, not the current pose you saw in iClone. Conversely, Blender's FBX exporter writes bind poses only when a weighted mesh is exported — armature-only motion FBX loses bind pose.
|
||||
5. **Twist bones:** CC twist chains (`...Twist01/02`) carry real rotation; dropping them without baking their rotation into parent bones causes candy-wrapper forearms/thighs on simple rigs.
|
||||
6. **Facial blendshapes:** enable **Convert Skinned Expressions to Morphs** (iClone) and don't over-prune with **Delete Unused Morphs**; in Blender keep shape keys + use `Export Deformation Bones Only` in glTF export; shape-key actions need **Retarget Shape Keys** (soupday) to follow a retarget; verify blend-shape tracks in Godot's imported AnimationPlayer.
|
||||
7. **Root motion:** iClone motions move `CC_Base_BoneRoot`/Hip in world space; decide in-place vs root-motion before export (flatten constraints first), then configure Godot AnimationTree root-motion track; Rigify retarget offers root-height adjustment at preview time.
|
||||
8. **Store-content license wall:** FBX/USD export silently blocked per-asset without Export License — audit content packs before building a pipeline around them.
|
||||
9. **GameBase/non-standard skeletons:** break both soupday Rigify and CC4 BVH export; keep characters on the standard CC_Base_ skeleton until final export.
|
||||
@@ -0,0 +1,170 @@
|
||||
# iClone 8 Core Animation System — Technical Reference
|
||||
|
||||
Compiled 2026-07-15 from manual.reallusion.com (iClone 8 Online Manual, sections 8.0–8.5), Reallusion KB, and Reallusion forums. Excludes the Video Mocap plugin (see video-mocap-plugin.md). `[UNVERIFIED]` marks low-confidence items.
|
||||
|
||||
## 1. Character Animation Architecture
|
||||
|
||||
### 1.1 Core model: Motion Clips + Layer Keys
|
||||
- "A **Motion** or **Animation** is the combination of the **Motion Clip** and the **Layer Keys**." Since iClone 1.x-era redesign, **layer keys are encased inside motion/animation clips**: dragging a clip drags its layer keys; a clip's layer key **cannot be moved outside the range of the clip**.
|
||||
- Motion track has sub-tracks: **Motion Layer** keys (per body part: e.g. `R Arm`, `LFingers`, `RFingers`) and **gesture layer clips** for both hands (LHand/RHand). Gesture data can be **Sampled** (extracted into LFingers/RFingers Motion Layer tracks) or **Flattened** (merged back into the motion clip).
|
||||
- **Relative Motion Layer Keys** blend an offset into the underlying motion. Old "absolute" motion layer keys were removed — the equivalent is removing the motion and posing via Edit Motion Layer.
|
||||
- A motion layer key's effect **persists to end of clip until another key is set**; transitions between consecutive layer keys are auto-generated.
|
||||
|
||||
### 1.2 Edit Motion Layer panel (Modify > Animation tab > Edit Motion Layer; menu Animation > Edit Motion Layer; hotkey **N**)
|
||||
Engine is **HumanIK** ("automatic floor contact while the intuitive real-time IK motion control keeps the body balanced"). Panel sections:
|
||||
1. **Set as Default / Reset to Default** (base pose retrieval; Default button adds a layer key standing the character straight in default pose).
|
||||
2. Mode tabs: **IK / Mask / Detail**.
|
||||
3. **Body pictogram** with effector-point states: Selected / Free / Move-locked / Rotate-locked / fully Locked / blend-weight-activated joints; quick lock toggle.
|
||||
4. **Pin presets**: lock joint translation, joint rotation, Both Feet, Both Hands, Hands+Feet, Both Toes, Unlock All.
|
||||
5. **Mirror controls**: edit bilateral bones simultaneously, copy bone status to opposite side, mirror entire pose.
|
||||
6. Selection modes: **Full Body** (torso may drag along when an effector moves; Ctrl+Click multi-select; keys inserted on ALL Motion Layer sub-tracks except fingers) vs **Body Part** (limbs/head only; key lands only in the relevant sub-track, e.g. R Arm).
|
||||
7. **Set Key** and **Reset Pose** buttons.
|
||||
8. **Viewport Controller** (colored on-screen bone pickers), **Bone Display Mode** (bones/bounding boxes), Bone Settings (Size, Affect all Bones, Color, Opacity).
|
||||
9. Palm pictogram for **hand gestures**.
|
||||
10. **Effector Blend Weight** activate/deactivate.
|
||||
11. Three **Mirror Pose** buttons (incl. Pose Mirror Right to Left, Full Body Pose Mirror).
|
||||
12. **Vertical Pose Alignment** toggle — includes Z-axis values (normally only X and Y are aligned → confirms Z = vertical axis in iClone).
|
||||
13. **Properties** dropdown for numeric translate/rotate entry.
|
||||
14. **Copy Pose / Paste Pose**.
|
||||
- IK posing: pick an effector (e.g. hand), use Move tool (W) and drag in viewport — 2D mouse motion drives the limb; manual says "make sure your target body parts face the camera" for precise 2D→3D control. Rotate tool (E) = FK-style joint angle edits. **Foot Contact / Hand Contact** checkboxes prevent ground penetration. Recommended: deactivate sub-joint blend weights and pin (lock) bones you don't want dragged before editing.
|
||||
- **Pose Mixer / Mask Mode** (inside Edit Motion Layer): pick body parts on pictogram (Select All / Select None / Invert Selection), then apply `*.rlPose` files from Content Manager — only masked parts take the pose; repeat with other pose files to compose partial poses; masked mirroring via Pose Mirror buttons. Custom poses saved via Add/save to library (`.rlPose`; iClone 8.4 added save-custom-poses page).
|
||||
- **Edit Pose vs motion layer keys**: "Edit Pose" panel is the Character Creator 4 equivalent (CC manual: "Using the Edit Pose Panel", same Body Key Editor); inside iClone all posing goes through Edit Motion Layer and always lands as **Motion Layer keys** relative to the current clip (or on an empty motion as pose-to-pose keyframing).
|
||||
- Layering workflow onto existing motion: apply clip → Edit Motion Layer → go to frame → adjust effectors → key auto-added to Motion Layer track and blended into the Motion-track pose.
|
||||
|
||||
### 1.3 Animation Layer panel (F11; iClone 8.x, distinct from the single Motion Layer track)
|
||||
- Stores pose keys in stackable named layers to "set their weight together". Toolbar: **Add New Layer, Duplicate Layer, Delete Layer, Rename Layer, Merge** (merge as **optimized keys** or **sampled key per frame**), reorder up/down, **Lock**, layer **Color**, **activation toggle** (mute), **Weight slider** with **weight keyframing** over time. Supports per-body-part layering (manual: "Animation Layers for Body Parts").
|
||||
|
||||
### 1.4 Reach Target (Modify > Animation tab > Reach Target; menu Animation > Reach Target)
|
||||
Panel: (1) **Bone Edit Mode** checkbox (show bones); (2) **Bone Settings** (Bone Size / Affect to all bones / Color / Opacity); (3) **Pictogram** effector states: Selected / Free / **Reaching** (contacts target) / Motion-compelling (keeps original motion); (4) **Reach Target** section incl. **Select Target and Keep Current Pose** button (reach-key effect without snapping to the target); (5) **Reach Mode** — **Position** checkbox (hand snaps/aligns to target pivot) and **Rotation** checkbox (also follows target orientation); (6) **Reach Offset** (position and angle offsets from the target); (7) **Lock to Origin** (pin effector to original position); (8) **Release** (sets a release key on the timeline).
|
||||
- **Lock to Original workflow** (protecting existing pose while one part reaches): lock affected effectors **one at a time** (multi-selection not allowed for locking); secondary drift may require additional locks; verify by playback.
|
||||
- Reach keys/constraints are **not** exported directly to FBX — use **Menu > Animation > Flatten All Motion with Constraint** to bake Link / Look At / Attach / Path / Reach keys into motion clips first.
|
||||
- Effector **dummy props** can be used as reach targets to fix motions ("Fixing Motions with Effector" page); Look At / Reach / Path control also via dummy props (Set > Prop > Using Dummy Props).
|
||||
|
||||
Sources: manual pages `50-Animation/Motion-Layer/Introducing_the_Edit_Motion_Layer_Panel.htm`, `Using_Body_Key_Editor.htm`, `How_to_Use_IK.htm`, `Layering_Motion_Layer_Keys_to_Exising_Motions.htm`, `Using-Mask-Mode.htm`, `50-Animation/Animation-Layer/Introducing-the-Animation-Layer-Panel.htm`, `50-Animation/Reach_Target/*.htm`, `51-Animation-Timeline-Editing/Clips_and_Keys.htm`.
|
||||
|
||||
## 2. Timeline (F3; Shift+F3 focuses it)
|
||||
|
||||
### 2.1 Tracks
|
||||
Per-object track stack includes: **Transform**, **Motion** (with Motion Layer + gesture sub-tracks), **Facial layer tracks**, **Viseme** (expand triangle → **Voice** and **Lips** sub-tracks), Expression tracks (see §5), **Visible**, Effect/HDR/IBL property tracks, **Audio/Sound + Lips** sync tracks, **Part tracks** (bone-skinned objects) and **Bone tracks** (individual bones + IK effectors), **Collect Clip** track, and the **Project track** (flags/markers). Track show/hide hotkeys: O, Q, T, M, R, Z, E, L, V, S, D, I.
|
||||
|
||||
### 2.2 Clip operations (toolbar / right-click / hotkeys)
|
||||
- **Add Key** `A`; **Break** `Ctrl+B` (non-destructive split, hidden data preserved); **Break Flatten** (destructive split, discards data beyond boundaries); **Auto Extend** (expands clip when keys fall outside; also Menu > Animation > **Clip Auto Extend**); **Loop** `1` (drag right clip edge); **Speed** `2` (drag edge; shown as % on clip title bar); **Resize Clip** `3`; **Motion Direction Control** `4`; **Motion Modifier** `5`; Cut/Copy/Paste `Ctrl+X/C/V`; duplicate = `Ctrl+LMB drag`; Delete key. Prev/Next key: `Ctrl+Left / Ctrl+Right` (toolbar also lists Tab/Shift+Tab). Playback range on/off: `'` / `;`. Zoom `+`/`-`, Fit `Ctrl+*`, **Actual Size** `Ctrl+/` (1 cell = 1 frame at 60fps). Insert/Delete frames for project or single object; Mark In/Out; flags on Project track.
|
||||
- **Reversing Clips**: right-click clip > Reverse (page `Reversing_Clips.htm`).
|
||||
- **Time Warp** (right-click motion clip > Time Warp): default = **Linear**; presets **Ease In / Ease In & Out / Ease Out / Ease Out & In**; **Custom** panel = curve-type dropdown + variation slider (left ≈ linear, right = more dramatic). Changes speed *within* the clip.
|
||||
- **Transitions between clips**: transition range sits ahead of the latter clip; each clip has 4 draggable triangles: **Fade In Start** (weight 0) / **Fade In End** (100) / **Fade Out Start** (100) / **Fade Out End** (0); blend progression between lower triangles is always **linear**. Weights do nothing if only one clip is on the track.
|
||||
- **Transition Curves between two keys** (Transform and other key tracks): set 2 keys, right-click the **later** key > **Transition Curve** > pick from **16 curves + Default** (Ease In, Ease Out, Ease In & Out, Pause and Move, Gain momentum before Move, Damping, Gain momentum and Damp, Stutter Start, End in a Bounce, Stuttering Start and End, Elastic Start and Sudden End, Sudden Start with Elastic end, Elastic Start and End, Step, Linear, Default). 4 basic presets have a **Strength** slider; **Default** curve comes from **Preference panel > Default Key Tangent**. Live viewport preview when hovering choices.
|
||||
- **Hollow vs solid keys**: hollow = partially keyed frame (not all axes keyed — common after Sampling due to optimization); solid = all axes keyed. Editing hollow keys in detail requires the **Curve Editor**.
|
||||
|
||||
### 2.3 Sampling & Flattening (right-click on motion/animation clip)
|
||||
- **Sample Motion Clip to New Layer** >> **Sample All Parts - Optimized** | **Sample All Parts - Per Frame** | **Sample Selected Parts** — exposes clip contents as **Layer Keys** (standard CC characters). Props/accessories: **Sample Animation Clip**.
|
||||
- **Flatten Motion Clip** >> **Flatten All Layers** | **Flatten And Keep All Keys** — merges Layer Keys into the clip. For clips with Speed/Time Warp applied, keys are **recalculated and optimized** rather than merged, and cells re-match timeline units (standard characters only). Menu names differ for standard vs non-standard characters vs props.
|
||||
- **Menu > Animation > Flatten All Motion with Constraint** bakes constraint keys (Link, Look At, Attach, Path, Reach) — required before FBX export of constrained animation.
|
||||
|
||||
### 2.4 Collect Clip track
|
||||
- Open Collect Clip track → click-drag a range covering the desired clips/keys → right-click: **Add Motion to Library** (actors; merges/compacts Transform + Motion + Hands + Motion Layer tracks into one motion, save to Custom library), **Add Gesture to Library** (hand motions), **Add to Perform List** (iProps: Transform + Animation tracks), **Add to Perform** (props: Transform track). Re-import via right-click Motion track > **Import**, or the Perform menu. This is the standard way to consolidate a mocap-plus-layer-edit result into a reusable `.iMotion`/MotionPlus asset.
|
||||
|
||||
### 2.5 Motion Direction Control (hotkey 4)
|
||||
Fixes the "transform keys fight motion clips" problem: select clip > Motion Direction Control button → leading line(s) show clip travel direction; **Rotate** fields (exact degrees), **Translate** fields (initialize clip start position), or use Object Rotate tool visually. With multiple clips, adjust the second clip while keeping adjacent motions aligned/connected. Related: **Menu > Animation > Auto Motion Alignment** >> **No Alignment / Align Position Only / Align Position and Direction**.
|
||||
|
||||
Sources: `51-Animation-Timeline-Editing/Timeline-Operation-Basic.htm`, `Clips_and_Keys.htm`, `Sampling_and_Flattening.htm`, `Transition_Methods_in_Tracks.htm`, `Creating-Transition-between-Clips.htm`, `Time_Warp_for_Motion_Clips.htm`, `Collect_Clip_Track.htm`, `Using-Motion-Direction-Control.htm`, `90_Keyboard_Shortcuts/Timeline_Shortcuts.htm`, `02-Main-Menu/Menu_Animation.htm`.
|
||||
|
||||
## 3. Curve Editor (built-in plugin; F12, Shift+F12 focuses; separate manual section 58-Curve-Editor)
|
||||
|
||||
- Purpose: classic-animation curve control over iClone keys ("time and space adjustment", ease in/out, arcs, smooth/linear camera moves). Select motion keys in Timeline, edit their curves.
|
||||
- UI: **Menu bar** (Key, Tangent, Navigation, Zoom, Optimize/Filter, Curve menus); **Toolbar** areas: Outliner Display Options, **Tangent Tools**, **Handle Tools**, **Key Editing Tools**, Navigation Tools; **Search bar**; **Outliner** (structure nodes + curve nodes of selected object); **Edit mode** dropdown: **Layer Keys** | **Transformation**; **Time mode**: **Project Time** | **Clip Time**; Current Frame field; **Graph view**; **FK Mode** button (launches motion-layer editing for bone selection); Status bar with Frame/Value numeric fields.
|
||||
- Key Editing Tools: **Move Key** (Shift+Drag = single-axis constraint), **Offset Key** (shifts entire sequence before/after to preserve surrounding animation), **Add Keyframe** (directly on curve; toggle "Add Keyframes for All Curves"), **Remove Keys**, **Scale Keys** (bounding-box: horizontal = speed, vertical = magnitude).
|
||||
- Filter menu: **Optimize** (Threshold 0–1 curve-fitting; higher = smoother/fewer keys; options: Optimize Selected Keys only / Selected Curves only / Selected Keys on Selected Curves; Preview shows before/after key counts — for densely keyed mocap curves), **Smooth** (Smooth Factor strength, **Falloff** to avoid cliffs at range edges, **Keep Custom Tangent**, Preview), **Butterworth** smoothing (**Cut-off (Hz)**, **Sample Rate (fps)**, Falloff, **Align to frames** = float keys → integer frames, Preview). These are the primary **mocap jitter cleanup** tools — sample the imported mocap clip to layer keys first, then filter.
|
||||
- Tangent menu/tools include standard tangent types (Auto/Linear/Step/Ease etc.) `[UNVERIFIED — exact list not confirmed]`. Dedicated Curve Editor keyboard shortcut page exists (`58-Curve-Editor/Keyboard_Shortcuts.htm`).
|
||||
- Gotcha: to curve-edit inside a motion clip you must expose keys (Sample) — clip contents themselves aren't directly curve-editable; hollow keys are adjusted here.
|
||||
|
||||
Sources: `58-Curve-Editor/Welcome_to_Curve_Editor.htm`, `UI_Intro_for_Curve_Editor.htm`, `UI_Intro_for_Curve_Editor_Key_Editing_Tools.htm`, `UI_Intro_for_Curve_Editor_Menu_Optimize.htm`.
|
||||
|
||||
## 4. Motion Correction / Cleanup Tools
|
||||
|
||||
### 4.1 Motion Correction (foot/hand sliding fix)
|
||||
- Right-click motion clip > **Motion Correction**. Derived from Reach: calculates ground-contact times, auto-creates **hand/foot prints**, sets Reach keys to feet/hands so they stick to prints.
|
||||
- Panel: **Presets** (pre-optimized body part sets), **Body Parts** (manual per-part activation), **Parts Settings**: **Threshold** (higher = severe sliding, lower = minor sliding — the main dial), **Transition** (frames for Reach/Release key transition), **Reach Offset** / **Release Offset** (shift Reach/Release keys left/right), **Reach Mode** checkboxes (touch prints / align to print directions), **Sync Threshold with all Parts** checkbox. Click **Correct**.
|
||||
- Iteration gotcha: after each Correct, run **Flatten Motion Clip > Flatten All Layers**, deactivate the already-fixed parts, then correct remaining parts; repeat.
|
||||
- This is the primary post-import cleanup pass for external/video mocap clips.
|
||||
|
||||
### 4.2 Motion Modifier (Timeline button, hotkey 5)
|
||||
- Slider-based whole-clip body-part adjustment ("fix posture issues or alter overall look"). **Base Pose** options: **Default Pose** (motions that don't change much), **Auto Generate** (averages pose for highly varying clips), **Load** (use first pose of a loaded iMotion). **Part Modify** section with **Stand Mode** / **Move Mode**. Converts entire clip into an altered one. Not recommended for dramatic motions (big turns/heavy transforms).
|
||||
|
||||
### 4.3 Motion Director (iClone 8 flagship; replaces MixMoves conceptually)
|
||||
- Behavior files (**iMD**) drive characters: each character can have multiple **Behaviors**; each behavior has **Modes** (currently **Casual** and **Athletic**); modes contain speeds ('Stroll', 'Walk', 'Jog'); each speed has perform-motions and **Mix-motions** ('Main Mixer' + sub-mixers).
|
||||
- Control: keyboard/gamepad (WASD-style; **Ctrl+R** start/stop playback-record; Shift = accelerate a level, Ctrl = decelerate; customizable via **Hotkey Manager**, Edit > Hotkey Manager, Ctrl+`); waypoints via **Alt+left-click** (queue) and **Alt+right-click** (absolute, cancels others). Cameras: **MD Camera** (system), **Free Camera**, **Follow Camera** (8 presets + 5 customizable) on MD Triggers panel. **Snap to Surface** for terrain-following characters (8.1). **Menu > Animation > Create iMD Cache** pre-retargets actors for MD performance. MD Props with **Reach Effect** settings exist (8.5).
|
||||
- Recording writes normal motion clips to the timeline, editable with everything in §1–§3.
|
||||
|
||||
### 4.4 MixMoves — REMOVED in iClone 8
|
||||
- The iClone 7 MixMoves system (automatic motion-graph transitions) is **not in iClone 8**; legacy MixMoves animations still usable as regular motions (Reallusion forum confirmation). Motion Director is the successor.
|
||||
|
||||
### 4.5 Motion Puppet / Direct Puppet / Prop Puppet / Morph Animation (Menu > Animation)
|
||||
- **Motion Puppet**: record motions from puppet templates onto the selected character (mask body parts, mouse-driven). **Direct Puppet**: simultaneously move+record body to generate motions. **Prop Puppet**: puppet-record prop animation. **Morph Animation**: Morph Animator panel — keyed morphing sliders (also how morph-based prop/character shape animation is keyed). **Edit Animation Layer**: layer keys on existing prop animations.
|
||||
|
||||
### 4.6 Smoothing
|
||||
- Body: Curve Editor **Smooth / Butterworth** filters (§3), Optimize for key reduction.
|
||||
- HumanIK foot floor contact: automatic in Edit Motion Layer (Foot/Hand Contact checkboxes). Character-level contact geometry comes from CC's **Floor Contact Planes** (soles/palms; toes/fingers bend to avoid penetration) — set in Character Creator, honored by iClone terrain snapping.
|
||||
|
||||
Sources: `51-Animation-Timeline-Editing/Motion-Correction-for-Sliding-Issues.htm`, `Modifying_and_Fixing_Motions.htm`, `50-Animation/Motion-Director/Basic-Concepts.htm`, `Setting-Hotkeys-with-Hotkey-Manager.htm`, `Snap-to-Surface.htm` (8.1), `02-Main-Menu/Menu_Animation.htm`, forum.reallusion.com/519260 (MixMoves), CC manual `06_Body/Setting_the_Floor_Contact_Planes.htm`.
|
||||
|
||||
## 5. Face Animation
|
||||
|
||||
### 5.1 Track structure
|
||||
- **Viseme track** → sub-tracks **Voice** (audio clip) and **Lips** (auto-generated viseme keys). **Expression** timeline = 4 tracks: **Muscle** (morph data), **Eye Blink**, **Eyeball**, **Head** (orient/tilt). **Facial Layer** tracks hold Face Key output. Face clips support Auto-Extend, Loop, Speed; layer keys copy/paste only within their originating clip. **Sample/Flatten** available to convert expression clips ↔ three-track layer keys. **Menu > Animation > Expression Clip Auto Crop** trims overlapping expression clips.
|
||||
|
||||
### 5.2 AccuLips
|
||||
- Open: **Modify > Animation tab > Motion section > Facial group > Create Script**, or **Menu > Animation > Create Script > AccuLips** (other Create Script options: Record Voice, Text-To-Speech, Wave File).
|
||||
- Works only for **CC G1–G3+ Standard and CC Gamebase** characters. Audio: wav, mp3, m4a, aac, wma.
|
||||
- Workflow: Open Audio File / Record Voice → **Generate Text** (speech-to-text; fix red-highlighted words) → **Align** (word-by-word sync; or **Update Selected** for a range) → adjust word durations → **Apply** → visemes generated on Lips track. Optional: load a TXT/SRT script with identical filename to skip transcription; custom viseme-word pairs saveable to **dictionary**. TTS input auto-aligns. Old iTalk scripts convertible.
|
||||
|
||||
### 5.3 Viseme editing & strength (3 levels)
|
||||
1. **Project/character level**: Modify > Animation tab > **Facial Animation Settings > Viseme Strength** slider. 2. **Clip level**: right-click voice clip on Viseme track > **Talking Style Editor** (hotkey 6) — per-clip viseme strength sliders. 3. **Individual key**: double-click viseme key (or empty cell to add) → **Lips Editor** (hotkey 7) — phoneme choice (e.g. "Phoneme: Oh") + **Expressiveness** slider (result visible after slider release).
|
||||
- **Smoothing mouth**: right-click Lips/Lip Options track range > **Lip Sync Options** → radio **Full Mouth** (+Smooth checkbox + smooth value) or **Parts** (independent lips / tongue / jaw sliders).
|
||||
|
||||
### 5.4 Face Key panel (Menu > Animation > Face Key, or Modify > Animation tab)
|
||||
- Tabs: **Expression** (preset expression style sets via dropdown), **Modify** (detail morph sliders grouped: **Brow, Eye, Nose, Cheek, Mouth, Custom**), **Morph** (**Expressiveness** slider + **Revert**). **Muscle** editing: click highlighted face regions and drag in viewport to sculpt an expression; **Muscle Symmetrical Selection** for mirrored edits; **Reset to Zero** button. Keys land on **Facial Layer** tracks; blend sequential keys by placing the playhead between them. Muscle vs Detail distinction: Muscle = region-drag posing, Detail = per-morph sliders.
|
||||
|
||||
### 5.5 Face Puppet (Menu > Animation > Face Puppet)
|
||||
- Real-time mouse-driven expression recording. Profiles: **Male, Female, Stylized** ("00_"-prefixed enhanced versions), **Universal** (mouth shut + six mouth expressions). **Facial Profile Strength** slider exaggerates/smooths. **Solo Feature Profile** states: None / Active / Selected. During Preview/Record: **left-click = blink**, hotkeys **Q W E R T Y** trigger Full Face Control Profile commands. **Profile Details** (Edit Property) table: 60 standard + 24 custom morphs. Preference > Control > "Confine Face Puppet area to 3D view-port". Attribute tab **Auto-Blink** toggle (disable during puppeteering).
|
||||
|
||||
Sources: `50-Animation/Facial-Animation/Creating-Voice-Script-with-Accurate-Lipsync.htm`, `Three-Levels-of-Setting-Strength-for-Visemes.htm`, `50-Animation/Pro_Facial_Animation/Face_Puppet.htm`, `Face_Key_Editing.htm`, `Facial_Timeline.htm`, `51-Animation-Timeline-Editing/Adding_or_Modifying_Lip_Synching_Keys.htm`, `Smoothing_Mouth_Movements.htm`.
|
||||
|
||||
## 6. Characters & Characterization
|
||||
|
||||
- **CC4 Standard characters** (CC3+/CC Gamebase etc.): full feature support — Edit Motion Layer/HumanIK, facial system (AccuLips requires CC G1–G3+ Standard/Gamebase), Motion Director, sampling with "All Parts - Optimized". Created/edited in Character Creator 4; CC4's **Characterization panel** data (bone-mapping, bone-rotation, floor contact, **HIK property data**) is what iClone uses for retargeting.
|
||||
- **ActorCore characters/motions**: Reallusion's store rigs (AccuRig-rigged); load directly in iClone and accept iClone/ActorCore motions; AccuRig converts arbitrary humanoid meshes → compatible rigs.
|
||||
- **Non-standard/Humanoid characters**: custom FBX rigs are converted via CC4's **Convert to Non-Standard / Humanoid** (Characterization mode: map custom bones to standard iClone bones, adjust to **T-pose** — all `.iMotion` are T-pose-based; skipping T-pose causes body-part offsets when motions apply). 3DXchange's old conversion role now lives in CC4/AccuRig. Feature availability in iClone differs (e.g. sampling menu items differ for non-standard characters; AccuLips unsupported).
|
||||
- **Imported mocap interaction (core side)**: **File > Import > Import External Motion** (iClone 8.4+) accepts **FBX or BVH** motion for standard/humanoid characters. Options: **Motion Profile** (from CC Characterization data), optional **T-pose** file from the same source, **Sample Per Second** (higher = more detail), **Root Bone** dropdown (choose source root node), **Auto-generate to Perform List**; **Convert All** → clips stored under custom **Motion > External Motion** library. Once converted, mocap clips are ordinary motion clips: trim/loop/speed/time-warp, Motion Correction, sample-to-layer + Curve Editor filtering, Edit Motion Layer layering, Reach Target locking.
|
||||
- **Props/Accessories**: animate via **Transform track** keys, **Prop Puppet**, **Edit Animation Layer**, Morph Animator; clip form = Animation clips ("Sample/Flatten Animation Clip"); collect via Collect Clip > Add to Perform (List). Constraint animation via Link/Attach/Path/Look At dummy workflows, flattened before export.
|
||||
|
||||
Sources: `8.4/50-Animation/Import-External-Motions/Import-External-Motions.htm`, CC4 manual `09-Import-Humanoid/*`, 3DXchange 6 legacy `Converting_Models_to_Non_Standard_Characters.htm`, actorcore.reallusion.com learn-and-support.
|
||||
|
||||
## 7. Import / Export
|
||||
|
||||
- **Menu paths**: `File > Export > Export FBX...` | `Export Alembic...` | `USD (Omniverse)`. `File > Import > Import External Motion` (mocap in). 3DXchange is **discontinued**; all conversion/export features merged into iClone 8 / CC4 (kb.reallusion.com/Product/53036) — iClone 8 ships pipeline-capable (FBX/OBJ export built in, no separate Pipeline SKU).
|
||||
- **Export FBX dialog** (verbatim, from manual screenshots): **Target Tool Preset**: Maya, 3ds Max, Unity 3D, Unreal (UE4 Skeleton), Unreal (UE5 Skeleton), Motion Builder, Blender, Cinema 4D, **Maya (Motion Pipeline)**, **Motion Builder (Motion Pipeline)**, Marmoset Toolbag. Preset changes downstream settings incl. axis conversion (no manual Y-up/Z-up radio in iClone 8; legacy 3DXchange had explicit Y-Up/Z-Up choice). **FPS** dropdown (default `Project(60)`). **Export Range**: T-Pose (no animation, avatar moved to origin (0,0,0)) / Current Frame / All / Range (From/To). Checkboxes: **Embed Textures**, **Embed Timeline Timecode**, **Max Texture Size** (default on, 2048), **Convert Image Format to** (default Bmp shown; JPG if unset), **Delete Hidden Face**, **Use Smooth Mesh**, **Convert Skinned Expressions to Morphs**, **Merge Face Hair to One Object**, **Delete Unused Morphs**, **Tear Ducts UV Adjustment** (character can't round-trip to CC if on).
|
||||
- **Export FBX Advanced Settings** (gear icon): **Merge Opacity to Diffuse Texture**, **Preserve Bone Names (CC Base)** (default on), **Mouth Open as Morph**, **Epic Skeleton** (UE4 Skeleton / UE5 Skeleton radio), **Epic Skeleton Hand and Foot IK**, **Reset Motion Root**, **Export Weight-map to Vertex Color** (default on), **Use T-Pose As Bind Pose**, **Reset Bone Scale** (default on), **Add Reference Pose** (default on) with radio: T-Pose for Unity / A-Pose for UE4 Skeleton / A-Pose for UE5 Skeleton / A-Pose for UEFN. Also **Export JSON for Auto Material Setup** (default on, for Auto Setup plugins) `[per Auto Setup manual]`.
|
||||
- **Root motion**: **Reset Motion Root** — since 7.6, for Unity/Unreal game-engine presets the root node **follows the hip** instead of staying at origin; needed for root-motion scripting/blending in engines. Constraint animation must be flattened first (**Animation > Flatten All Motion with Constraint**). Motion-only export: use the **(Motion Pipeline)** presets.
|
||||
- **Batch export**: select multiple objects; with Target Tool Preset = **Unreal** or **Unity 3D** each object exports to a **separate FBX** named after the iClone object; any non-game-engine preset merges all picked objects into **one** FBX.
|
||||
- **Non-exportable types**: iLight, iEffect, iParticle, iTree, iGrass, iTerrain, iWater, iSky, aml, iTalk, iPath.
|
||||
- **USD**: via free **Omniverse connector plugin** — Menu > Plugin > Omniverse > Export USD, File > Export, or toolbar. Options: Export All/Selected; Material Render Mode (RTX Real-Time = OV PBR shader; RTX Interactive Path Tracing = SSS MDL for digital humans); Export Range > **Motion FPS: 60/30/24**; Send to Server (Nucleus). Outputs main USD + light/material/motion dependency folders.
|
||||
- **BVH**: **import** supported (Import External Motion); **BVH export is NOT available from iClone 8** — legacy BVH export was 3DXchange Pipeline; CC4 4.4+ has BVH export (see export-pipeline-to-blender-godot.md §3).
|
||||
- **Blender/Godot**: Blender via Target Tool Preset "Blender" + free **CC/iC Blender Tools** (soupday) importer; Godot has **no preset** — use Unity 3D or Blender preset FBX, or round-trip through Blender→glTF `[workflow inference, not Reallusion-documented]`.
|
||||
- **Licensing**: Reallusion content items require an **Export License** to leave the Reallusion ecosystem in FBX/OBJ/USD ("You must first purchase the content licenses in order to export FBX and USD files" error otherwise); iClone-only content (legacy "maroon" items) is blocked without it. Your own imported/created assets are unrestricted. FBX-key/`.fbxkey` applies to CC character mesh round-trips. Export License purchase deducts already-owned iContent cost.
|
||||
- **Alembic** export exists for vertex-cache workflows (`Export Alembic...`).
|
||||
|
||||
Sources: `83-FBX/FBX_UI_Intro.htm` (+ dialog screenshots), `Exporting_FBX_from_iClone.htm`, `Batch_Exporting_FBX_Files.htm`, `Resetting_Motion_Root.htm`, kb.reallusion.com/Product/53036, reallusion.com/ContentStore/Export_License.pdf, Omniverse plugin manual (iC-8.3), forum.reallusion.com/556427.
|
||||
|
||||
## 8. System Fundamentals
|
||||
|
||||
- **Frame rate**: Project Settings > **Time Unit** section — FPS dropdown per project; **default project length 1800 frames, maximum 54000**; timeline "Actual Size" = 1 cell : 1 frame **at 60fps** (iClone's native/internal sampling is 60fps; FBX FPS default `Project(60)`; render/export offers 12/24/25/30/60). Time Mode: Realtime vs **By Frame** (physics). Time Unit display: time-based or frame-based.
|
||||
- **Units**: **1 iClone unit = 1 cm**; default grid 100×100 units = 1×1 m. CC characters ≈175 units tall. Matches Unreal cm scale directly.
|
||||
- **Axes**: iClone is **Z-up** (Reallusion: "iClone and 3DXchange are Z-Up based 3D tools"); corroborated by Edit Motion Layer "Vertical Pose Alignment includes Z-axis values". FBX export axis conversion is handled by the Target Tool Preset (Y-up conversion for Maya/Unity etc.). Character bones do **not** share a common up-vector (spine Z-up, legs point down, feet back) — a known pain point when retargeting exported skeletons. A literal `Z_UpAxis` node/track name: not found in iClone 8 docs `[UNVERIFIED — likely a 3DXchange-era conversion dummy]`.
|
||||
- **Key global shortcuts**: Space play/pause; ,/. start/end frame; ←/→ frame step; **F3 Timeline**, **F11 Animation Layer panel**, **F12 Curve Editor** (Shift+ variants focus); **N = Edit Motion Layer**; Q/W/E/R = Select/Move/Rotate/Scale; Ctrl+Z/Ctrl+Y undo/redo; X/Z/C/V camera Pan/Zoom/Orbit/Roll; Home = 45° perspective; F = front view of selection. Timeline-specific keys in §2.2. All remappable: **Edit > Hotkey Manager (Ctrl+`)** — Device: Keyboard, expand item, press new key.
|
||||
- **Rotation order**: editable per object via **Menu > Animation > Rotation Order** (Euler order — matters for curve editing and export). **Remove Object Animation** / **Remove Scene Animation** clear animation (scene removal offers frame-position options).
|
||||
|
||||
Sources: `03-Introducing-the-User-Interface/Project_Settings_Time_Unit.htm`, `90_Keyboard_Shortcuts/Global_Shortcuts.htm`, `Timeline_Shortcuts.htm`, `Hotkey-Manager.htm`, 3DXchange5 `Y_Up_or_Z_Up.htm`, forum.reallusion.com/109018 & /326528 (cm units), forum.reallusion.com/525257 (bone axes).
|
||||
|
||||
---
|
||||
|
||||
Key gotchas worth surfacing: layer keys can't leave their clip; flatten-between-iterations for Motion Correction; Flatten All Motion with Constraint before FBX export; MixMoves removed in iC8; AccuLips is CC-Standard-only; game-engine presets (Unreal/Unity) are the only batch/per-file + Reset-Motion-Root paths; 60fps internal / cm / Z-up when planning Godot import.
|
||||
@@ -0,0 +1,134 @@
|
||||
# Reallusion AI Video Mocap for iClone 8 — Reference (researched 2026-07-15)
|
||||
|
||||
## 1. Product Disambiguation (Reallusion mocap ecosystem, mid-2026)
|
||||
|
||||
| Product | What it is | Body/Face | Input | Pricing model |
|
||||
|---|---|---|---|---|
|
||||
| **iClone Video Mocap** (plug-in) | AI body mocap from ordinary video, powered by **QuickMagic** cloud AI. THE "video mocap plugin". Launched **Oct 29–31, 2025** | Body + fingers; **NO face** | Recorded video file | Free plugin; pay-per-use points per generation (cloud) |
|
||||
| **AccuFACE** (Motion LIVE gear) | AI **facial** tracker from webcam (live) or video file (offline). Local NVIDIA RTX inference | Face only | Webcam or video file | Perpetual plugin, $499 MSRP (launch promo $250, Nov 2023) |
|
||||
| **Motion LIVE** | Hub plugin aggregating real-time mocap hardware/gear (Xsens, Rokoko, Perception Neuron, Leap Motion, iPhone LIVE FACE, Faceware, AccuFACE) into one unified real-time recording panel | Body+face+hands (via gear) | Live hardware streams | Hub + per-gear profile purchases |
|
||||
| **Motion Director** | Gameplay-style interactive character locomotion/control system inside iClone (WASD-style driving, not video mocap) | Body locomotion | Keyboard/gamepad | Built into iClone 8 |
|
||||
|
||||
Video Mocap deliberately does NOT do faces — official workflow layers AccuFACE (or iPhone LIVE FACE) on top for talking animations.
|
||||
Sources: https://www.cgchannel.com/2025/10/reallusion-launches-ai-motion-capture-service-video-mocap/ , https://www.reallusion.com/iclone/motion-capture/default.html
|
||||
|
||||
## 2. Video Mocap — Identity, Versions, Requirements
|
||||
|
||||
- Exact name: **"iClone Video Mocap"** / **"Video Mocap Plug-in for iClone"**. Tagline: "Precise AI Mocap from Any Video".
|
||||
- Technology partner: **QuickMagic** (AI mocap provider); processing is **cloud/online** (upload video → AI conversion on QuickMagic cloud → download motion). Not local; no GPU requirement for generation itself.
|
||||
- Requirements: **iClone 8.63 or above**, **Windows 10 & 11 only**. Plugin itself is a **free download**, installed/updated via **Reallusion Hub**.
|
||||
- Version history: launch ~v1.0 (Oct 2025, alongside iClone 8.63); **v1.02 (announced 2026-02-26)** — urgent fix for task-status-refresh bug that caused re-submission and **redundant point charges** after closing the app; also added diagnostic logs and auto-recovery of task statuses from local video/motion files. Users on iClone 8.71 + plugin 1.02 reported in forum (Mar 2026+). Manual page shows revision date June 8, 2026.
|
||||
- Official manual root: https://manual.reallusion.com/iclone-video-mocap-plug-in/content/welcome.htm (sections: Workflow, Full Body, Upper Body, Import, Edit, Generate (purchase.htm), Apply, Save, Adding Facial Expression (talking-animation.htm), Video Guidelines (rule.htm)).
|
||||
|
||||
Sources: https://www.reallusion.com/iclone/video-mocap/download.html , https://discussions.reallusion.com/t/urgent-update-video-mocap-1-02-fixes-task-status-refresh-bug-to-prevent-redundant-charges/16659
|
||||
|
||||
## 3. Pricing
|
||||
|
||||
- Current metering (product page + manual, mid-2026): **5 points per second of processed video** (AI / Bonus / DA Points), **100 points = US$1.00** → **$0.05/sec, max $3.00 for a 60 s task**. Deduction order: **AI Points → Bonus Points → DA Points**.
|
||||
- At launch (Oct 2025) press reported a flat **250 DA Points (US$2.50) per task** up to 60 s; the Apr 2026 beginner article still says "$2.50 per motion". Flat-250 vs 5/sec discrepancy: pricing appears to have moved to per-second metering; treat exact figure as ~$2.50–3.00/generation [UNVERIFIED which applies at any given moment].
|
||||
- Points sources: **DA Points** ($10 / $50 / $100 packs, min $10 spend) or **Reallusion AI Cloud subscription** AI Points (FREE $0 = 500 pts; STARTER $11/mo = 1,100; STANDARD $35/mo = 3,500; PRO $95/mo = 9,500; annual −20%). AI Points are universal across AI Studio, Video Mocap, Headshot image gen. DA:AI = 1:1 value.
|
||||
- Fully licensed iClone users get complimentary Video Mocap generation coupon(s); one coupon = one task regardless of duration. (Launch promo advertised "100 free AI mocap generations" limited-time; one forum user mentions losing "5 complimentary generations" — exact freebie count varies by promo [UNVERIFIED].)
|
||||
- Failed tasks: Reallusion support has refunded points for failed tasks case-by-case (forum evidence). Verify tasks/transactions: https://kb.reallusion.com/Print53250.aspx
|
||||
|
||||
Sources: https://manual.reallusion.com/iclone-video-mocap-plug-in/content/purchase.htm , https://ai.reallusion.com/plans.html , https://www.cgchannel.com/2025/10/reallusion-launches-ai-motion-capture-service-video-mocap/
|
||||
|
||||
## 4. Workflow (official 5 stages: Import Video | Edit Video | Generate Motion | Apply Motion | Save Motion)
|
||||
|
||||
### 4.1 Import Video
|
||||
- Open the plugin from iClone's Plugins menu after Hub install [menu path UNVERIFIED, standard for RL plugins].
|
||||
- Import via **"+ Add Video"** button or drag-and-drop into the panel.
|
||||
- **13 supported formats**: `.avi .flv .m4v .mkv .mov .mp4 .mpeg .ogv .webm .wmv .3gp .asf .dv .mxf`.
|
||||
- Source video duration: **1 s – 15 min** accepted; **processing region must be 1–60 s** (longer videos auto-trimmed or must be trimmed). Max file size **400 MB** (else reduce bitrate/rescale/compress/trim). Resolution: **≥720p recommended; 4K accepted but may hurt performance**. FPS not specified.
|
||||
- Task queue: up to **100 videos**; batch generation: up to **10 tasks at once**. Statuses: Unprocessed → Trimmed → (processing) → Applicable / Error.
|
||||
|
||||
### 4.2 Edit Video panel
|
||||
- Drag playhead to preview; drag **Start Time** / **End Time** handles to set the processing region (timecode MM:SS:FF); drag the whole region to shift it with fixed duration; **Reset Range** restores full span; **Apply Settings** confirms → status "Trimmed".
|
||||
- **Detect Actors** button: when multiple people are in frame, click it and pick the single target actor in a frame where they are clearly visible. **One actor per task**; for group scenes, duplicate the task per performer and select a different detection each time.
|
||||
- Trim out segments where no person appears (hurts AI quality).
|
||||
|
||||
### 4.3 Generate Motion
|
||||
- Choose tracking mode: **Full Body** (default) or **Upper Body**; optional **"Includes Fingers"** checkbox in both modes for finger capture.
|
||||
- Full Body: whole body must be clearly visible throughout. Upper Body: designed for framing cut off below the pelvis (sign language, instrument playing, speech); everything below the hip is ignored/not generated.
|
||||
- Submit → upload → cloud processing, typically **5–8 minutes per task**; progress monitored in the task list. Insufficient points → "Add Funds" / "Cancel" dialog.
|
||||
|
||||
### 4.4 Apply Motion
|
||||
- Select completed task → **Apply Motion** button (bottom-right), or double-click the task, or right-click → **Apply Motion**.
|
||||
- Motion is saved as **rlMotion** to a preset folder and applied to the **currently selected character**; if none selected, **a dummy character is auto-loaded**. Auto-retargeting adapts the motion to any iClone/CC4/AccuRIG standard character body type.
|
||||
- **"Apply reference video"** checkbox: loads the source video as a 2D background object in the viewport behind the character for frame-by-frame comparison during cleanup.
|
||||
|
||||
### 4.5 Save Motion
|
||||
- Auto-saved to Content Manager **Animation > Motion > VideoMocap**; default disk path `C:\Users\Public\Documents\Reallusion\Reallusion Custom\Animation\Motion\VideoMocap`. Folder button changes path; Reset restores default. Right-click task → **Find Motion File**. Thumbnail from Edit Video frame (Quick Thumbnail on) or posed character (off).
|
||||
|
||||
Sources: https://manual.reallusion.com/iclone-video-mocap-plug-in/content/video-mocap-workflow.htm , .../import.htm , .../edit.htm , .../purchase.htm , .../video-mocap-full-body.htm , .../video-mocap-upper-body.htm , .../apply.htm , .../save.htm
|
||||
|
||||
## 5. Official Video Guidelines (rule.htm) — filming rules
|
||||
|
||||
- **Camera angle**: flat/eye-level. Overhead or elevated angles make the result lean forward/backward.
|
||||
- **Camera motion**: static camera, single continuous shot, no cuts/switched shots; only minor shake tolerated ("Counteracts camera shakes" is marketing — keep it steady).
|
||||
- **Framing**: entire actor (or pelvis-up for Upper Body) visible for the whole clip; "any body part that moves out of the frame will present a problem". Actor not too small in frame; size consistent (no zooms/dollies).
|
||||
- **Clothing**: avoid solid colors and colors matching the background; patterns/texture are fine.
|
||||
- **Background/lighting**: clean and bright preferred, though the engine tolerates dim light, busy backdrops, patterned clothing.
|
||||
- **Motion restrictions**: avoid very fast motion, violent shaking, acrobatic flips, vertical actions (pull-ups, jumps), and ungrounded-feet activity (swimming).
|
||||
- **Occlusion**: single person only; avoid limbs intersecting other actors; avoid hand-occluding poses ("running fingers through hair").
|
||||
|
||||
Source: https://manual.reallusion.com/iclone-video-mocap-plug-in/content/rule.htm
|
||||
|
||||
## 6. Cleanup / Post-Processing in iClone (official recommended tools)
|
||||
|
||||
Artifact → tool mapping (from Reallusion Magazine + 80.lv + tutorials):
|
||||
- **Foot sliding** → **Motion Correction** tool (Motion Correction for Sliding Issues, iClone 8 manual) + **Reach Target** to lock/release foot positions; also Foot Contact tool + iClone footstep visualization. Official tutorial: "Mocap Motion Fix – Eliminating Foot Sliding | iClone 8 Tutorial" (https://www.youtube.com/watch?v=t8yeA-DJ0ds).
|
||||
- **Sudden elevation changes / abrupt vertical pops** → dedicated fix tutorial "Fixing Sudden Elevation Change"; edit hip height in Curve Editor.
|
||||
- **Depth misalignment / motion offset** → **Auto Motion Alignment** / Root Alignment.
|
||||
- **Jitter/noise** → **Curve Editor** smoothing.
|
||||
- **Sudden body twists / rotating limbs** → Motion Layer editing with HumanIK controls.
|
||||
- **Hand/gesture errors** → iClone gesture tools / hand pose library.
|
||||
- **Missing or garbage frames** → rotoscope against the 2D reference video and bridge with transition curves.
|
||||
- The auto-loaded reference-video background is the key rotoscoping aid — compare character to footage in real time.
|
||||
- **Gotcha**: if the applied clip has **Loop** enabled, motion-layer edits "spring back" (looped frames reference the source); **Flatten** the motion clip (or disable Loop) before keyframe editing (forum-confirmed staff answer).
|
||||
- Tutorial course hub: https://courses.reallusion.com/home/iclone/pipelines/video-mocap-plug-in
|
||||
|
||||
Sources: https://magazine.reallusion.com/2025/10/29/iclone-video-mocap-converts-videos-into-editable-3d-motion/ , https://80.lv/articles/getting-ai-mocap-ready-for-production-with-reallusion-s-iclone-8 , https://discussions.reallusion.com/t/cant-adjust-motions-after-applying-from-video-mocap/16711 , https://manual.reallusion.com/iClone-8/Content/ENU/8.0/51-Animation-Timeline-Editing/Motion-Correction-for-Sliding-Issues.htm
|
||||
|
||||
## 7. Adding Face (talking animation) — official 7-step layering with AccuFACE
|
||||
|
||||
1. Apply Video Mocap body motion + reference video. 2. In **AccuFACE VIDEO mode**, load a speech video framed on the actor's head (RL strongly recommends **separate recordings for body and face**). 3. Connect AccuFACE in **Motion LIVE**, enable **"Audio for Viseme Track"** (auto voice clip). 4. Preview + record facial clips via Motion LIVE. 5. Align viseme/expression clips to the motion clip on the timeline using the reference video. 6. Verify sync in playback. 7. Save to Content Manager custom library.
|
||||
Source: https://manual.reallusion.com/iclone-video-mocap-plug-in/content/talking-animation.htm
|
||||
|
||||
## 8. AccuFACE (face-from-video) — key facts
|
||||
|
||||
- Motion LIVE gear plugin; **iClone 8.6+**, Windows 10/11 64-bit, 8 GB RAM, **NVIDIA RTX only** (20/30/40/50 series + RTX Pro; driver 520.46+ or 572+ for RTX 50) — local GPU inference, no cloud, no AMD/Intel support. $499 MSRP (frequent discounts).
|
||||
- Input: live webcam (min 1280×720@30fps; 720p/30 optimal) or video files (**AVI, WMV, MP4, MKV, MOV**). Captures up to 60 fps facial data; single-pass audio + face capture; **AccuLIPS** refines lip/tongue; output framerate-independent (24/30/60 fps).
|
||||
- Calibration: guided process (live mode) or pick calibration poses across video frames (video mode). Refinement controls: **Strength Balance** (per-region), **Smooth Filter**, **Denoise**, **Anti-Interference** (cross-region triggering).
|
||||
Source: https://www.reallusion.com/iclone/motion-capture/accuface.html
|
||||
|
||||
## 9. Known Limitations & Community Issues
|
||||
|
||||
- Hard limits: 60 s/task, 1 actor/task, no face capture, cloud-only (needs internet + points), 400 MB file cap, 10-task batch cap.
|
||||
- AI-typical artifacts acknowledged by Reallusion itself: foot sliding, depth misalignment, unnatural twists, jitter — cleanup is an expected part of the workflow, not an edge case.
|
||||
- Failure modes: subject leaving frame, background-matching clothing, fast/occluded motion, spins with limb occlusion, airborne/ungrounded feet.
|
||||
- Forum-reported bugs: (a) pre-1.02 task-status refresh bug → duplicate charges (fixed in 1.02); (b) failed tasks bricking the queue — Error tasks with retry/delete greyed out and Generate Motion disabled for all new videos; support refunded points but the user had to hand-edit plugin task files to recover (https://discussions.reallusion.com/t/video-mocap-problem/17173); (c) Loop-enabled clips blocking motion-layer edits (see §6).
|
||||
- Community consensus: results are solid for grounded, single-person, moderate-speed motion; twitchy feet and torso rotation remain the main manual-cleanup costs, consistent with other AI mocap (DeepMotion, Plask).
|
||||
|
||||
## 10. Best-Practice Tips (Reallusion Magazine "Video Mocap for Beginners", Apr 2026)
|
||||
|
||||
- A mid-range phone (Samsung A54) or 1080p webcam (Logitech C922) on a tripod is fully sufficient; ordinary home lighting worked.
|
||||
- Rehearse, then **overact** mundane motions; record multiple takes.
|
||||
- **Stack multiple motions in one 60 s clip with ~2 s gaps** between them — one generation fee yields several clips to split in iClone.
|
||||
- Pre-trim in an external editor; use real props to anchor hand positions and prevent hand drift.
|
||||
- Use Upper Body mode + Includes Fingers for seated/talking/instrument performances.
|
||||
Source: https://magazine.reallusion.com/2026/04/13/video-mocap-for-beginners/
|
||||
|
||||
## 11. Brief Comparison
|
||||
|
||||
- **vs Motion LIVE hardware**: Motion LIVE = real-time, suit/camera hardware ($$$–$$$$), no per-use fee, best fidelity and long takes; Video Mocap = offline, zero hardware, ~$2.50–3/clip, 60 s cap, cleanup required. Reallusion positions them as hybrid-complementary (ActorCore packs + hardware for hero shots, Video Mocap for bulk/reference-driven motion).
|
||||
- **vs free/DIY (Rokoko Video free tier, DeepMotion, Plask, open-source VIBE/4DHumans pipelines)**: those export FBX/BVH requiring manual retarget; Video Mocap's differentiator is native rlMotion output, auto-retargeting to CC4/AccuRIG characters, the in-viewport reference-video rotoscope, and direct access to iClone's correction toolset — then FBX/USD export from iClone to Unreal, Unity, Blender, Maya, MotionBuilder.
|
||||
|
||||
## 12. Primary Sources
|
||||
|
||||
- Product: https://www.reallusion.com/iclone/video-mocap/ | Download: https://www.reallusion.com/iclone/video-mocap/download.html
|
||||
- Manual: https://manual.reallusion.com/iclone-video-mocap-plug-in/content/welcome.htm (+ import/edit/purchase/apply/save/rule/talking-animation/video-mocap-full-body/video-mocap-upper-body .htm)
|
||||
- Magazine: https://magazine.reallusion.com/2025/10/29/iclone-video-mocap-converts-videos-into-editable-3d-motion/ , https://magazine.reallusion.com/2026/04/13/video-mocap-for-beginners/
|
||||
- Press: https://www.cgchannel.com/2025/10/reallusion-launches-ai-motion-capture-service-video-mocap/ , https://beforesandafters.com/2025/11/08/iclone-video-mocap-converts-videos-into-editable-3d-motions/
|
||||
- AI plans: https://ai.reallusion.com/plans.html | AccuFACE: https://www.reallusion.com/iclone/motion-capture/accuface.html
|
||||
- Forum: https://discussions.reallusion.com/t/urgent-update-video-mocap-1-02-fixes-task-status-refresh-bug-to-prevent-redundant-charges/16659 , /t/video-mocap-problem/17173 , /t/cant-adjust-motions-after-applying-from-video-mocap/16711
|
||||
- Courses: https://courses.reallusion.com/home/iclone/pipelines/video-mocap-plug-in | 80.lv: https://80.lv/articles/getting-ai-mocap-ready-for-production-with-reallusion-s-iclone-8
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
name: pose-estimation
|
||||
description: Extract human pose landmarks from video or images (MediaPipe Tasks API) and compare pose sequences with DTW joint-angle scoring. Use for analyzing dance/movement videos, scoring a recreated animation against source footage, or any "what is the body doing at time t" question. Works on real footage AND stylized game characters when a single subject fills the frame.
|
||||
---
|
||||
|
||||
# Pose Estimation
|
||||
|
||||
Two tested scripts, one venv. Extract 33-landmark pose sequences from video, then
|
||||
score two sequences against each other (e.g. source dance video vs game recreation).
|
||||
|
||||
## Setup (idempotent, run once per machine)
|
||||
|
||||
```bash
|
||||
bash ~/.claude/skills/pose-estimation/scripts/setup.sh
|
||||
```
|
||||
|
||||
Creates `.venv` (pins python3.12 — MediaPipe does not support 3.13+) and downloads the
|
||||
`full` pose model. `lite`/`heavy` model variants auto-download on first use via curl
|
||||
(NOT urllib — macOS python lacks SSL certs; keep the curl download path).
|
||||
|
||||
Always run scripts with the venv python: `~/.claude/skills/pose-estimation/.venv/bin/python`.
|
||||
|
||||
## Extract poses
|
||||
|
||||
```bash
|
||||
VENV=~/.claude/skills/pose-estimation/.venv/bin/python
|
||||
SKILL=~/.claude/skills/pose-estimation/scripts
|
||||
|
||||
$VENV $SKILL/extract_pose.py input.mp4 --out poses.json --fps 8 \
|
||||
--overlay check.mp4 --min-conf 0.3 --model heavy
|
||||
```
|
||||
|
||||
- `--fps N` sample rate; `--times "0.0,0.5,1.0"` instead samples an explicit beat grid
|
||||
(pass a musical beat grid for dance work — samples land on choreographically meaningful frames).
|
||||
- `--overlay out.mp4` draws the skeleton on sampled frames — ALWAYS eyeball this before
|
||||
trusting the data.
|
||||
- Output JSON has both `landmarks` (normalized image coords) and `world_landmarks`
|
||||
(meters, hip-centered). Use world landmarks for any comparison — they are camera- and
|
||||
scale-invariant.
|
||||
- The `OK: N/M samples had a detected pose` line is the health signal. Low N/M → see
|
||||
framing rules below.
|
||||
|
||||
## Compare two pose sequences
|
||||
|
||||
```bash
|
||||
$VENV $SKILL/compare_poses.py source_poses.json recreation_poses.json \
|
||||
--a-range 2.0:6.0 --b-range 0:4.0
|
||||
```
|
||||
|
||||
Reduces each frame to 8 joint angles (elbows, shoulders, hips, knees), aligns the two
|
||||
sequences with DTW (tolerates tempo drift), prints JSON: `score_0_100`,
|
||||
`mean_angle_error_deg`, and `per_joint_error_deg` sorted worst-first (tells you WHERE
|
||||
the recreation diverges — e.g. arms right, legs wrong). Verified: identical input → 100.0;
|
||||
different dance segments → ~70.
|
||||
|
||||
## Framing rules (the thing that actually determines success)
|
||||
|
||||
MediaPipe pose is **single-person** and needs the subject to fill a large fraction of
|
||||
the frame. Verified findings:
|
||||
|
||||
- Wide shot with many small figures (e.g. a game formation capture at 1600x900 with
|
||||
~80px characters): **0% detection**. This is framing, not the model.
|
||||
- Same video, one character crop-zoomed to fill frame (`ffmpeg -vf "crop=W:H:X:Y,scale=4x"`):
|
||||
**94% detection** — even on stylized low-poly game characters (Quaternius rigs).
|
||||
- Real photos/footage: works out of the box.
|
||||
|
||||
So: for multi-person or wide footage, crop to one subject first:
|
||||
|
||||
```bash
|
||||
ffmpeg -i wide.mp4 -vf "crop=140:180:350:420,scale=560:720:flags=lanczos" -an solo.mp4
|
||||
```
|
||||
|
||||
Knobs when detection is weak: `--min-conf 0.15`, `--model heavy`, bigger crop upscale.
|
||||
|
||||
## Typical workflow (video → analysis)
|
||||
|
||||
1. Extract a frame, look at it, choose the crop for the subject.
|
||||
2. Extract with `--overlay`, check the overlay video, check detection %.
|
||||
3. Compare / analyze from `world_landmarks` or via compare_poses.py.
|
||||
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare two pose sequences (extract_pose.py JSON) and score their similarity.
|
||||
|
||||
Method: each detected frame is reduced to a joint-angle signature (8 angles:
|
||||
elbows, shoulders, hips, knees) computed from 3D world landmarks — scale- and
|
||||
translation-invariant, so a phone video and a game capture compare fairly.
|
||||
The two angle sequences are aligned with dynamic time warping (handles tempo
|
||||
drift / offset), then scored:
|
||||
|
||||
mean_angle_error_deg average per-joint angular error along the DTW path
|
||||
per_joint_error_deg which joints diverge most (arms vs legs etc.)
|
||||
score_0_100 100 * max(0, 1 - mean_error/90) (rough but comparable)
|
||||
|
||||
Typical use: score a recreated game dance against the source video, per move
|
||||
segment (--a-range/--b-range trim by time before aligning).
|
||||
"""
|
||||
import argparse, json, sys
|
||||
|
||||
import numpy as np
|
||||
|
||||
# (name, a, b, c) -> angle at b between vectors (a-b) and (c-b), landmark indices
|
||||
ANGLE_DEFS = [
|
||||
("l_elbow", 11, 13, 15), ("r_elbow", 12, 14, 16),
|
||||
("l_shoulder", 13, 11, 23), ("r_shoulder", 14, 12, 24),
|
||||
("l_hip", 11, 23, 25), ("r_hip", 12, 24, 26),
|
||||
("l_knee", 23, 25, 27), ("r_knee", 24, 26, 28),
|
||||
]
|
||||
|
||||
|
||||
def load_angles(path, t_min, t_max):
|
||||
with open(path) as f:
|
||||
doc = json.load(f)
|
||||
ts, sigs = [], []
|
||||
for fr in doc["frames"]:
|
||||
if not fr.get("detected"):
|
||||
continue
|
||||
t = fr["t"]
|
||||
if t < t_min or (t_max is not None and t > t_max):
|
||||
continue
|
||||
lms = fr.get("world_landmarks") or fr.get("landmarks")
|
||||
pts = np.array([[p["x"], p["y"], p["z"]] for p in lms])
|
||||
sig = []
|
||||
for _, a, b, c in ANGLE_DEFS:
|
||||
v1, v2 = pts[a] - pts[b], pts[c] - pts[b]
|
||||
n1, n2 = np.linalg.norm(v1), np.linalg.norm(v2)
|
||||
if n1 < 1e-8 or n2 < 1e-8:
|
||||
sig.append(0.0)
|
||||
continue
|
||||
cosang = np.clip(np.dot(v1, v2) / (n1 * n2), -1.0, 1.0)
|
||||
sig.append(float(np.degrees(np.arccos(cosang))))
|
||||
ts.append(t)
|
||||
sigs.append(sig)
|
||||
if not sigs:
|
||||
sys.exit(f"ERROR: no detected frames in range in {path}")
|
||||
return np.array(ts), np.array(sigs)
|
||||
|
||||
|
||||
def dtw_path(A, B):
|
||||
"""O(n*m) DTW on mean-abs-angle-diff cost. Returns (path, cost_matrix)."""
|
||||
n, m = len(A), len(B)
|
||||
cost = np.mean(np.abs(A[:, None, :] - B[None, :, :]), axis=2) # n x m, degrees
|
||||
acc = np.full((n + 1, m + 1), np.inf)
|
||||
acc[0, 0] = 0.0
|
||||
for i in range(1, n + 1):
|
||||
for j in range(1, m + 1):
|
||||
acc[i, j] = cost[i - 1, j - 1] + min(acc[i - 1, j], acc[i, j - 1], acc[i - 1, j - 1])
|
||||
path = []
|
||||
i, j = n, m
|
||||
while i > 0 and j > 0:
|
||||
path.append((i - 1, j - 1))
|
||||
step = np.argmin([acc[i - 1, j - 1], acc[i - 1, j], acc[i, j - 1]])
|
||||
if step == 0: i, j = i - 1, j - 1
|
||||
elif step == 1: i -= 1
|
||||
else: j -= 1
|
||||
path.reverse()
|
||||
return path, cost
|
||||
|
||||
|
||||
def parse_range(s):
|
||||
if not s:
|
||||
return 0.0, None
|
||||
lo, _, hi = s.partition(":")
|
||||
return float(lo or 0.0), (float(hi) if hi else None)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("pose_a", help="pose JSON (e.g. source video)")
|
||||
ap.add_argument("pose_b", help="pose JSON (e.g. recreation)")
|
||||
ap.add_argument("--a-range", default=None, help="time window in A, 'start:end' seconds")
|
||||
ap.add_argument("--b-range", default=None, help="time window in B, 'start:end' seconds")
|
||||
ap.add_argument("--out", default=None, help="optional JSON report path")
|
||||
args = ap.parse_args()
|
||||
|
||||
a_lo, a_hi = parse_range(args.a_range)
|
||||
b_lo, b_hi = parse_range(args.b_range)
|
||||
ts_a, A = load_angles(args.pose_a, a_lo, a_hi)
|
||||
ts_b, B = load_angles(args.pose_b, b_lo, b_hi)
|
||||
|
||||
path, cost = dtw_path(A, B)
|
||||
path_errs = np.array([np.abs(A[i] - B[j]) for i, j in path]) # steps x joints
|
||||
mean_err = float(path_errs.mean())
|
||||
per_joint = {ANGLE_DEFS[k][0]: round(float(path_errs[:, k].mean()), 1)
|
||||
for k in range(len(ANGLE_DEFS))}
|
||||
score = round(100.0 * max(0.0, 1.0 - mean_err / 90.0), 1)
|
||||
|
||||
report = {
|
||||
"a": {"file": args.pose_a, "frames": len(A), "span_s": [float(ts_a[0]), float(ts_a[-1])]},
|
||||
"b": {"file": args.pose_b, "frames": len(B), "span_s": [float(ts_b[0]), float(ts_b[-1])]},
|
||||
"mean_angle_error_deg": round(mean_err, 2),
|
||||
"per_joint_error_deg": dict(sorted(per_joint.items(), key=lambda kv: -kv[1])),
|
||||
"score_0_100": score,
|
||||
"dtw_path_len": len(path),
|
||||
}
|
||||
print(json.dumps(report, indent=2))
|
||||
if args.out:
|
||||
with open(args.out, "w") as f:
|
||||
json.dump(report, f, indent=2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,179 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Extract human pose landmarks from a video (or single image) with MediaPipe Pose
|
||||
(Tasks API, mediapipe >= 0.10.30 — the legacy `solutions` API no longer exists).
|
||||
|
||||
Outputs one JSON document with per-sample 2D landmarks (normalized image coords)
|
||||
and 3D world landmarks (meters, hip-centered) — the world landmarks are what you
|
||||
want for pose comparison because they are camera-scale invariant.
|
||||
|
||||
Sampling: --fps N samples the video at N Hz (default 10). --times "0.0,0.5,1.0"
|
||||
samples at explicit timestamps instead (e.g. a musical beat grid). --overlay
|
||||
writes the sampled frames with the skeleton drawn to an .mp4 for eyeball checks.
|
||||
|
||||
Works on real footage and (usually) on stylized humanoid game characters, but
|
||||
detection confidence drops on non-photoreal rigs — lower --min-conf and try
|
||||
--model heavy if frames come back undetected.
|
||||
"""
|
||||
import argparse, json, os, sys
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
import mediapipe as mp
|
||||
from mediapipe.tasks import python as mp_tasks
|
||||
from mediapipe.tasks.python import vision
|
||||
|
||||
SKILL_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
# The 33 pose landmarks, in model output order (fixed contract of the pose model).
|
||||
POSE_LANDMARKS = [
|
||||
"nose", "left_eye_inner", "left_eye", "left_eye_outer", "right_eye_inner",
|
||||
"right_eye", "right_eye_outer", "left_ear", "right_ear", "mouth_left",
|
||||
"mouth_right", "left_shoulder", "right_shoulder", "left_elbow", "right_elbow",
|
||||
"left_wrist", "right_wrist", "left_pinky", "right_pinky", "left_index",
|
||||
"right_index", "left_thumb", "right_thumb", "left_hip", "right_hip",
|
||||
"left_knee", "right_knee", "left_ankle", "right_ankle", "left_heel",
|
||||
"right_heel", "left_foot_index", "right_foot_index",
|
||||
]
|
||||
|
||||
# Skeleton edges for overlay drawing (standard MediaPipe pose connections, torso+limbs).
|
||||
CONNECTIONS = [
|
||||
(11, 12), (11, 13), (13, 15), (12, 14), (14, 16), # shoulders + arms
|
||||
(11, 23), (12, 24), (23, 24), # torso
|
||||
(23, 25), (25, 27), (27, 29), (27, 31), # left leg
|
||||
(24, 26), (26, 28), (28, 30), (28, 32), # right leg
|
||||
(15, 17), (15, 19), (15, 21), (16, 18), (16, 20), (16, 22), # hands
|
||||
]
|
||||
|
||||
|
||||
def lms_to_list(lms):
|
||||
return [
|
||||
{"name": POSE_LANDMARKS[i],
|
||||
"x": round(p.x, 5), "y": round(p.y, 5), "z": round(p.z, 5),
|
||||
"visibility": round(p.visibility or 0.0, 3)}
|
||||
for i, p in enumerate(lms)
|
||||
]
|
||||
|
||||
|
||||
def draw_skeleton(frame_bgr, lms, t, detected):
|
||||
h, w = frame_bgr.shape[:2]
|
||||
if detected:
|
||||
pts = [(int(p.x * w), int(p.y * h)) for p in lms]
|
||||
for a, b in CONNECTIONS:
|
||||
cv2.line(frame_bgr, pts[a], pts[b], (0, 220, 255), 2)
|
||||
for x, y in pts:
|
||||
cv2.circle(frame_bgr, (x, y), 3, (0, 255, 0), -1)
|
||||
cv2.putText(frame_bgr, f"t={t:.2f}s {'OK' if detected else 'NO POSE'}",
|
||||
(10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.8,
|
||||
(0, 255, 0) if detected else (0, 0, 255), 2)
|
||||
return frame_bgr
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("input", help="video file (.mp4/.mov/...) or single image")
|
||||
ap.add_argument("--out", required=True, help="output JSON path")
|
||||
ap.add_argument("--fps", type=float, default=10.0, help="sample rate in Hz (default 10)")
|
||||
ap.add_argument("--times", default=None, help="comma-separated timestamps in seconds (overrides --fps)")
|
||||
ap.add_argument("--overlay", default=None, help="optional .mp4 path: sampled frames with skeleton drawn")
|
||||
ap.add_argument("--model", default="full", choices=["lite", "full", "heavy"],
|
||||
help="pose model variant (heavy = most accurate; auto-downloads)")
|
||||
ap.add_argument("--min-conf", type=float, default=0.5,
|
||||
help="min detection/tracking confidence (lower for stylized characters)")
|
||||
args = ap.parse_args()
|
||||
|
||||
model_path = os.path.join(SKILL_DIR, "models", f"pose_landmarker_{args.model}.task")
|
||||
if not os.path.exists(model_path):
|
||||
url = (f"https://storage.googleapis.com/mediapipe-models/pose_landmarker/"
|
||||
f"pose_landmarker_{args.model}/float16/latest/pose_landmarker_{args.model}.task")
|
||||
os.makedirs(os.path.dirname(model_path), exist_ok=True)
|
||||
print(f"downloading model {args.model}...", file=sys.stderr)
|
||||
import subprocess
|
||||
subprocess.run(["curl", "-sL", "-o", model_path, url], check=True)
|
||||
|
||||
is_image = os.path.splitext(args.input)[1].lower() in (".png", ".jpg", ".jpeg", ".webp", ".bmp")
|
||||
|
||||
options = vision.PoseLandmarkerOptions(
|
||||
base_options=mp_tasks.BaseOptions(model_asset_path=model_path),
|
||||
running_mode=vision.RunningMode.IMAGE if is_image else vision.RunningMode.VIDEO,
|
||||
min_pose_detection_confidence=args.min_conf,
|
||||
min_pose_presence_confidence=args.min_conf,
|
||||
min_tracking_confidence=args.min_conf,
|
||||
)
|
||||
landmarker = vision.PoseLandmarker.create_from_options(options)
|
||||
|
||||
frames_out, overlay_frames = [], []
|
||||
|
||||
def process(frame_bgr, t, idx):
|
||||
mp_img = mp.Image(image_format=mp.ImageFormat.SRGB,
|
||||
data=cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB))
|
||||
if is_image:
|
||||
res = landmarker.detect(mp_img)
|
||||
else:
|
||||
res = landmarker.detect_for_video(mp_img, int(t * 1000))
|
||||
detected = bool(res.pose_landmarks)
|
||||
entry = {"t": round(t, 4), "index": idx, "detected": detected}
|
||||
if detected:
|
||||
entry["landmarks"] = lms_to_list(res.pose_landmarks[0])
|
||||
if res.pose_world_landmarks:
|
||||
entry["world_landmarks"] = lms_to_list(res.pose_world_landmarks[0])
|
||||
frames_out.append(entry)
|
||||
if args.overlay is not None:
|
||||
overlay_frames.append(draw_skeleton(
|
||||
frame_bgr.copy(), res.pose_landmarks[0] if detected else None, t, detected))
|
||||
|
||||
if is_image:
|
||||
img = cv2.imread(args.input)
|
||||
if img is None:
|
||||
sys.exit(f"ERROR: cannot read image {args.input}")
|
||||
process(img, 0.0, 0)
|
||||
src_fps, duration = 0.0, 0.0
|
||||
else:
|
||||
cap = cv2.VideoCapture(args.input)
|
||||
if not cap.isOpened():
|
||||
sys.exit(f"ERROR: cannot open video {args.input}")
|
||||
src_fps = cap.get(cv2.CAP_PROP_FPS) or 30.0
|
||||
duration = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) / src_fps if src_fps else 0.0
|
||||
|
||||
if args.times:
|
||||
sample_ts = sorted(float(s) for s in args.times.split(",") if s.strip())
|
||||
else:
|
||||
sample_ts = list(np.arange(0.0, max(duration - 1e-6, 0.0), 1.0 / args.fps))
|
||||
|
||||
for i, t in enumerate(sample_ts): # ascending — VIDEO mode needs monotonic timestamps
|
||||
cap.set(cv2.CAP_PROP_POS_MSEC, t * 1000.0)
|
||||
ok, frame = cap.read()
|
||||
if not ok:
|
||||
frames_out.append({"t": round(t, 4), "index": i, "detected": False, "error": "read failed"})
|
||||
continue
|
||||
process(frame, t, i)
|
||||
cap.release()
|
||||
|
||||
landmarker.close()
|
||||
|
||||
if args.overlay and overlay_frames:
|
||||
h, w = overlay_frames[0].shape[:2]
|
||||
vw = cv2.VideoWriter(args.overlay, cv2.VideoWriter_fourcc(*"mp4v"),
|
||||
args.fps if not args.times else 4.0, (w, h))
|
||||
for f in overlay_frames:
|
||||
vw.write(f)
|
||||
vw.release()
|
||||
|
||||
detected = sum(1 for f in frames_out if f.get("detected"))
|
||||
doc = {
|
||||
"source": os.path.abspath(args.input),
|
||||
"source_fps": round(src_fps, 3),
|
||||
"duration_s": round(duration, 3),
|
||||
"samples": len(frames_out),
|
||||
"detected": detected,
|
||||
"landmark_names": POSE_LANDMARKS,
|
||||
"frames": frames_out,
|
||||
}
|
||||
os.makedirs(os.path.dirname(os.path.abspath(args.out)) or ".", exist_ok=True)
|
||||
with open(args.out, "w") as f:
|
||||
json.dump(doc, f)
|
||||
print(f"OK: {detected}/{len(frames_out)} samples had a detected pose -> {args.out}"
|
||||
+ (f" (overlay: {args.overlay})" if args.overlay else ""))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,33 @@
|
||||
#!/bin/bash
|
||||
# Bootstrap the pose-estimation venv + model. Idempotent — safe to run every time.
|
||||
# MediaPipe supports Python <= 3.12, so we pin the interpreter explicitly.
|
||||
set -euo pipefail
|
||||
|
||||
SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
VENV="$SKILL_DIR/.venv"
|
||||
MODEL="$SKILL_DIR/models/pose_landmarker_full.task"
|
||||
MODEL_URL="https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_full/float16/latest/pose_landmarker_full.task"
|
||||
|
||||
PY=""
|
||||
for cand in python3.12 python3.11 python3.10; do
|
||||
if command -v "$cand" >/dev/null 2>&1; then PY="$cand"; break; fi
|
||||
done
|
||||
if [ -z "$PY" ]; then
|
||||
echo "ERROR: need python 3.10-3.12 for mediapipe (found none). brew install python@3.12" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -x "$VENV/bin/python" ]; then
|
||||
"$PY" -m venv "$VENV"
|
||||
fi
|
||||
|
||||
"$VENV/bin/pip" install --quiet --upgrade pip
|
||||
"$VENV/bin/pip" install --quiet "mediapipe>=0.10.9" "opencv-python>=4.8" "numpy"
|
||||
|
||||
if [ ! -f "$MODEL" ]; then
|
||||
mkdir -p "$SKILL_DIR/models"
|
||||
curl -sL -o "$MODEL" "$MODEL_URL"
|
||||
fi
|
||||
|
||||
echo "OK: venv at $VENV, model at $MODEL"
|
||||
echo "Run: $VENV/bin/python $SKILL_DIR/scripts/extract_pose.py --help"
|
||||
@@ -0,0 +1,131 @@
|
||||
# Skill: Animation Retargeting (Kevin → BoZo) — DEPRECATED
|
||||
|
||||
**DEPRECATED May 19, 2026.** BoZo and Kevin characters retired. Quaternius character system replaces them — no retargeting needed. This skill is kept for engine reference (Godot fork rest fixer fix).
|
||||
|
||||
```bash
|
||||
# 1. Clean import (creates default .import files)
|
||||
rm -f assets/animations/kevin/*.import assets/models/characters/Body/Body_BasicBody.fbx.import
|
||||
rm -f .godot/imported/Archer*.scn .godot/imported/Body_BasicBody*.scn .godot/imported/Villager*.scn .godot/imported/Throwing*.scn
|
||||
godot --headless --path . --import
|
||||
|
||||
# 2. Patch .import files with BoneMap + retarget config
|
||||
godot --headless --path . -s setup_retarget.gd
|
||||
|
||||
# 3. Delete cache + reimport with retarget config
|
||||
rm -f .godot/imported/Archer*.scn .godot/imported/Body_BasicBody*.scn .godot/imported/Villager*.scn .godot/imported/Throwing*.scn
|
||||
godot --headless --path . --import
|
||||
```
|
||||
|
||||
IMPORTANT: Must delete `.godot/imported/*.scn` cache between steps 2→3, otherwise Godot skips reimport.
|
||||
|
||||
## What the pipeline does
|
||||
|
||||
1. **Bone renamer:** Renames Kevin bones (`B-hips` → `Hips`) and BoZo bones (`pelvis` → `Hips`) to Humanoid standard
|
||||
2. **Overwrite Axis (method 1):** Changes skeleton rest poses to match SkeletonProfileHumanoid reference
|
||||
3. **Animation conversion:** Transforms keyframe values from old rest coordinate system to new rest coordinate system using formula from `post_import_plugin_skeleton_rest_fixer.cpp:777`:
|
||||
```
|
||||
new_q = new_pg_q.inv * old_pg_q * qt * old_rest_q.inv * old_pg_q.inv * new_pg_q * new_rest_q
|
||||
```
|
||||
|
||||
## Known Bug (patched in tinqs/godot fork)
|
||||
|
||||
**Commit:** `4e16c3f5bd` on `main`
|
||||
|
||||
The `%GeneralSkeleton` unique node resolution via `get_node()` fails in headless import mode. Original code at line 733:
|
||||
```cpp
|
||||
Node *node = (ap->get_node(ap->get_root_node()))->get_node(NodePath(track_path));
|
||||
ERR_CONTINUE(!node); // Silently skips ALL animation tracks!
|
||||
```
|
||||
|
||||
**Fix:** Falls back to `find_child()` when unique name resolution fails. Without this fix, rest poses change but animation keyframe values stay in the old coordinate system.
|
||||
|
||||
## Current Status (2026-05-18)
|
||||
|
||||
**Working:**
|
||||
- Bone renaming (Kevin → Humanoid, BoZo → Humanoid) ✓
|
||||
- Rest pose overwrite to Humanoid reference ✓
|
||||
- Skeleton node resolution fix (find_child fallback) ✓
|
||||
- Animation conversion runs for spine/chest bones ✓
|
||||
|
||||
**NOT working:**
|
||||
- Animation conversion appears to be a NO-OP for limb bones (arms, legs)
|
||||
- Spine keys → near identity (correct). Arm/leg keys → near rest (unconverted)
|
||||
- Root cause: unknown. Possibly old_rest captured AFTER partial overwrite, or formula output matches input for bones with specific rest orientations
|
||||
|
||||
**Visual symptoms:**
|
||||
- Arms cross behind back during animation
|
||||
- Legs may be double-rotated (rest * rest)
|
||||
- Spine animates correctly
|
||||
|
||||
## Bone Mapping
|
||||
|
||||
### Kevin → Humanoid
|
||||
| Humanoid | Kevin |
|
||||
|----------|-------|
|
||||
| Hips | B-hips |
|
||||
| Spine | B-spine |
|
||||
| Chest | B-chest |
|
||||
| UpperChest | B-upperChest |
|
||||
| Neck | B-neck |
|
||||
| Head | B-head |
|
||||
| LeftShoulder | B-shoulder.L |
|
||||
| LeftUpperArm | B-upper_arm.L |
|
||||
| LeftLowerArm | B-forearm.L |
|
||||
| LeftHand | B-hand.L |
|
||||
| Right* | B-*.R (mirror) |
|
||||
| LeftUpperLeg | B-thigh.L |
|
||||
| LeftLowerLeg | B-shin.L |
|
||||
| LeftFoot | B-foot.L |
|
||||
| LeftToes | B-toe.L |
|
||||
| Right* | B-*.R (mirror) |
|
||||
|
||||
### BoZo → Humanoid
|
||||
| Humanoid | BoZo |
|
||||
|----------|------|
|
||||
| Hips | pelvis |
|
||||
| Spine | spine_01 |
|
||||
| Chest | spine_03 |
|
||||
| UpperChest | spine_05 |
|
||||
| Neck | neck_01 |
|
||||
| Head | head |
|
||||
| LeftShoulder | clavicle_l |
|
||||
| LeftUpperArm | upperarm_l |
|
||||
| LeftLowerArm | lowerarm_l |
|
||||
| LeftHand | hand_l |
|
||||
| Right* | *_r (mirror) |
|
||||
| LeftUpperLeg | thigh_l |
|
||||
| LeftLowerLeg | calf_l |
|
||||
| LeftFoot | foot_l |
|
||||
| LeftToes | ball_l |
|
||||
| Right* | *_r (mirror) |
|
||||
|
||||
BoZo intermediate bones (spine_02, spine_04, neck_02) have no Kevin equivalent and keep global rest.
|
||||
|
||||
## Key Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `setup_retarget.gd` | Headless script: patches .import files with BoneMap + retarget config |
|
||||
| `run_retarget.gd` | EditorScript: same but runs inside Godot editor GUI |
|
||||
| `addons/retarget_setup/plugin.gd` | EditorPlugin: auto-runs on editor start |
|
||||
| `assets/retarget/kevin_bonemap.tres` | Kevin → Humanoid BoneMap resource |
|
||||
| `assets/retarget/bozo_bonemap.tres` | BoZo → Humanoid BoneMap resource |
|
||||
| `inspect_fbx.gd` | Debug: prints bone names, rest poses, animation values |
|
||||
| `src/Viewer/PlayerController.cs` | Character: loads animation, remaps %GeneralSkeleton paths |
|
||||
|
||||
## Godot Engine Source (for debugging)
|
||||
|
||||
| File | What |
|
||||
|------|------|
|
||||
| `editor/import/3d/post_import_plugin_skeleton_rest_fixer.cpp` | Overwrite Axis + animation conversion |
|
||||
| `editor/import/3d/post_import_plugin_skeleton_renamer.cpp` | Bone renaming |
|
||||
| `scene/3d/retarget_modifier_3d.cpp` | Runtime retargeting (RetargetModifier3D) |
|
||||
| `scene/resources/bone_map.cpp` | BoneMap resource |
|
||||
| `scene/resources/skeleton_profile.cpp` | SkeletonProfile + Humanoid preset |
|
||||
|
||||
## What NOT to do
|
||||
|
||||
1. **Manual quaternion retargeting in C#** — tried 5 different formulas, all failed. The formula requires parent GLOBAL rest transforms for the entire bone chain. Only Godot's engine code handles this correctly.
|
||||
2. **Rename BoZo bones at runtime** — breaks outfit/clothing FBX loading (bone name mismatch).
|
||||
3. **BoneAttachment3D for head without inverse rest compensation** — double-transforms the head.
|
||||
4. **Headless import without deleting .scn cache** — Godot skips reimport if source FBX unchanged.
|
||||
Reference in New Issue
Block a user