🦾

KrumpRobot: Neural Physical AI Dance Motion Generator

Full 1:38 Choreography Β· Unitree H1 & G1 Β· Adobe Mixamo Rig Β· Modal Cloud Trained
● HARDWARE: UNITREE H1 / G1 ● FULL SONG: 1:38 (58 BARS) ● 140 BPM AUDIO SYNC ● LATENCY: 2.77 MS (p50) ● STATUS: VERIFIED (PASS)
Arun Krump Session Soundtrack (FL Studio 21 Master Β· 1:38)
140 BPM Phase: Intro Stance (0:00 - 0:14)
0:00 / 1:38
🦾 3D Humanoid Robotics Studio [FULL 1:38 CHOREOGRAPHY · 140 BPM]
🎬 Section: Intro Stance (Bar 1/58)
πŸ’₯ GRF Live: 1.00Γ— BW
πŸ“ Knee Flex: 48.2Β° (Singularity Free)
Model: Unitree H1 (Bipedal 360 NΒ·m)
Time: 0.0s / 98.5s
FPS: 60
βš™οΈ 1:38 Musical Choreography Routines
58 BARS TOTAL
Jump to Song Section:
Motion Playback Speed 1.00Γ— (Real-Time)
Impact Energy Scaling 1.00Γ—
Ground Reaction Force (Fz)
1.00Γ— BW
Inference Latency
2.77 ms (p50)
Sub-3ms Unitree MPC
Zero Moment Point (ZMP) Footprint x: +0.00, z: +0.00
3D Model & Avatar Presentation:
πŸŽ₯ Video Motion Capture & Mixamo Retargeting
MOTION2MIXAMORIG
Arun Performance vs KrumpRobot Y-Bot Retargeting (4 Panels)
Top-L: Live Video (f90)
Top-R: Mixamo 3D Rig
Bot-L: SMPL-X Skeleton
Bot-R: Rigged Y-Bot
πŸ“ 4-Stage Motion2MixamoRig Pipeline
GitHub Repo β†—
01. Person Detection & Tracking: YOLOv8x detects continuous bounding boxes across all frames of Arun's video without tracking drift.
02. 2D Anatomical Keypoints: RTMPose-m (ONNX runtime) extracts 17 COCO keypoints with left/right limb temporal stabilization.
03. 3D Body Reconstruction: GVHMR (HMR4D + ViT) reconstructs metric 3D SMPL-X mesh rotations and root trajectory at 30 fps CFR.
04. Rest-Relative Retargeting: Calibrates 22 Mixamo bones with rest basis transform, achieving 8.765 Γ— 10⁻¹¹ forward kinematics error.
Rest Forward Kinematics Error
8.765 Γ— 10⁻¹¹
Rest Skinning Error
1.42 Γ— 10⁻⁡ cm
🎬 Mixamo Character (.GLB) 🎡 Soundtrack (.MP3) πŸ“¦ 300-Frame Motion (.NPZ) πŸ“Ή 4-Panel Video (.MP4)
πŸ’» Standalone Robotics & MPC Controller Code
PYTORCH & NUMPY ONLY
#!/usr/bin/env python3
"""
KrumpRobot Standalone PyTorch Inference
Loads KrumpRobot weights and generates 3D joint rotations autoregressively.
No quantum dependencies required.
"""

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

class KrumpRobotPolicy(nn.Module):
    def __init__(self, state_dim=182, cond_dim=9, hidden_dim=256, n_bones=22):
        super().__init__()
        self.n_bones = n_bones
        self.cond_proj = nn.Linear(cond_dim, 64)
        self.state_proj = nn.Linear(state_dim, 192)
        self.fusion_norm = nn.LayerNorm(hidden_dim)
        self.gru = nn.GRU(hidden_dim, hidden_dim, num_layers=2, batch_first=True, dropout=0.1)
        self.res1 = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.LayerNorm(hidden_dim),
            nn.GELU(),
            nn.Linear(hidden_dim, hidden_dim)
        )
        self.head_quat = nn.Linear(hidden_dim, n_bones * 4)
        self.head_vel = nn.Linear(hidden_dim, n_bones * 4)
        self.head_root = nn.Linear(hidden_dim, 3)
        self.head_dyn = nn.Linear(hidden_dim, 4)
        self.head_zmp = nn.Linear(hidden_dim, 2)

    def forward(self, x_seq, cond):
        B, T, _ = x_seq.shape
        s_proj = self.state_proj(x_seq)
        c_proj = self.cond_proj(cond).unsqueeze(1).expand(-1, T, -1)
        h_in = self.fusion_norm(torch.cat([s_proj, c_proj], dim=-1))
        gru_out, _ = self.gru(h_in)
        last_feat = gru_out[:, -1, :]
        h = last_feat + self.res1(last_feat)
        raw_quat = self.head_quat(h).view(B, self.n_bones, 4)
        norm_quat = F.normalize(raw_quat, p=2, dim=-1).view(B, -1)
        return {
            "quat": norm_quat,
            "vel": self.head_vel(h),
            "root": self.head_root(h),
            "dyn": self.head_dyn(h),
            "zmp": self.head_zmp(h)
        }

if __name__ == "__main__":
    model = KrumpRobotPolicy()
    state_dict = torch.load("krumprobot_weights.pt", map_location="cpu")
    model.load_state_dict(state_dict)
    model.eval()
    print("KrumpRobot Policy Loaded Successfully!")

    # Generate 1 step conditioned on STOMP primitive at 140 BPM
    dummy_window = torch.zeros(1, 24, 182)
    # Conditioning: [STOMP(1), JAB(0), SWING(0), POP(0), KILL(0), sin(ph), cos(ph), Energy, Fz]
    cond = torch.tensor([[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.90, 3.82]], dtype=torch.float32)
    with torch.no_grad():
        out = model(dummy_window, cond)
    print("Generated 22-bone quaternions shape:", out["quat"].shape)
    print("Predicted Vertical Ground Reaction Force (Fz):", f"{out['dyn'][0, 2].item():.2f}x BW")
    print("Predicted Zero Moment Point (ZMP):", out["zmp"][0].numpy())
πŸ’Ύ PyTorch Weights (krumprobot_weights.pt) 🌐 WebGL Weights (krumprobot_weights.json) 🦾 Generated Motion (krumprobot_generated_motion.npz) πŸ“œ Training Receipt (receipt.json)