Preserve hand–object interaction.
Interaction-graph seeds train a reusable retargeter, refined with object-aware contact losses.
DexOAKObject-Aware Kinematic Retargeting
Teach the interaction.
Transfer the motion.
Different hands. The same intent. DexOAK turns human hand–object motion into object-aware robot references, then refines them into physical interactions.
Improve RL. Transfer without re-solving.
Better kinematic references improve residual RL and transfer to unseen data without per-frame optimization.
From two hands to hardware.
State-of-the-art results on OakInk-v2 and ARCTIC; 46/56 open-loop hardware replays across five tasks.
Video
The complete story.
Method, simulation comparisons, and real-robot experiments.
01 / Motion, from every angle
Same task.
Same task.
Different embodiments.
Orbit the scene. Follow the trajectories. Compare the kinematic reference with the motion actually achieved after reinforcement learning.
Recorded failure
01 / Reference
Preparing live geometry…
02 / After residual RLRecorded policy rollout
Preparing live geometry…
Views and playback are synchronized.
Loading recorded trajectories…
02 / On the real robot
From reference
From reference
to reality.
A UR5 and Allegro hand replay selected successful simulation rollouts. Seven tasks, including interactions held out from retargeter training.
REAL HARDWARE
UR5 + ALLEGRO
Cut fruit
4 / 8 successful replays
The method
Learn the interaction.
Learn the interaction.
Reuse the retargeter.
Optimize training references once. Retarget new demonstrations with a forward pass.
Optimize seed references
Match hand–object graph Laplacian coordinates by optimizing robot wrist pose and joints.
Learn the retargeting function
Train a feed-forward network on the optimized references, conditioned on the human, object, and previous robot state.
Refine contact
Fine-tune with surface-contact, object-part, contact-direction, collision, and temporal losses.
Refine dynamics
Residual RL corrects the kinematic reference to execute the interaction under physics.
Results · Contribution 03
Object-aware references.
Object-aware references.
Stronger dynamic retargeting.
DexOAK improves each reported state-of-the-art comparison across both hands and datasets.
DexOAK + RLManipTransSPIDERDexMachina
SRMT object and hand error thresholdsSRobj object-only error thresholdsADD-AUC object surface-point tracking
Higher is better. — not reported. Each method uses its own dynamic refinement. * SPIDER uses more OakInk-v2 sequences; that comparison is indicative.
Contributions 01 & 02
Contact-aware kinematics improve the same RL backend.
On Allegro right-hand OakInk-v2 tasks, SRMT rises from 20.6% to 48.2%. Fine-tuning adds 5.0 points over the interaction-graph seed.
Effect of the kinematic reference
SRMT (%) ↑ · Same ManipTrans residual RL
Contribution 02
Transfer to unseen data without re-solving.
OakInk-v2 → ARCTIC, Allegro: 87.9 ADD-AUC zero-shot; 90.0 after object-aware fine-tuning, near the 91.3 in-domain result.
108.9 HzZero-shot reference generation
Cross-dataset transfer
ADD-AUC (%) ↑ · Same DexMachina RL
Reference-generation speed includes target-data optimization, training, or fine-tuning where required; excludes RL. This experiment uses a different RL backend from the end-to-end comparison above.
Contribution 03
Replay learned interactions on hardware.
46 / 56Successful open-loop replays · five tasks
Successful simulation policies replayed on UR5 + Allegro. Held-out laptop closure and button press each achieve 10/12.
DexOAKManipTrans
Successful hardware replays
05 / Citation
Build on DexOAK.
Preprint · arXiv:2610.26762