TELEDEXTER

Beyond Pick-and-Place

Humans don't just pick things up. They spin pens, regrasp screwdrivers mid-task, reorient objects across their fingers — orchestrating dozens of contact transitions per second¹Trends and Challenges in Robot Manipulation (Billard & Kragic, Science 2019) ↩. Most of robotic manipulation today lives in pick-and-place: close the fingers, lift, move, release. That problem is largely solved. Even today's flagship demos — the three systems below, foundation policies trained on tens of thousands of hours of human and robot data — are, watched closely, inventive variations on the static grasp, not the dexterous capability of a human hand. The moment a task demands the object to move inside the hand — reorientation, finger-gaiting, functional grasp switches for tool use — the entire stack falls apart.

Current dexterous-hand systems — GENE-26.5²GENE-26.5: Advancing Robotic Manipulation to Human Level (Genesis AI, 2026) ↩, EgoScale³EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data (NVIDIA GEAR Lab, 2026) ↩, and LDA-1B⁴LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestion (Galbot, 2026) ↩ (left to right) — remain in quasi-static grasping and interaction, lacking in-hand dexterous capability.

What makes these tasks hard is not the kinematics — it is the dynamics. Every grasp transition demands precise coordination of forces, friction, and contact timing across fingers. Current teleoperation⁵AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System (Qin et al., RSS 2023) ↩^,6DOGlove: Dexterous Manipulation with a Low-Cost Open-Source Haptic Force Feedback Glove (Zhang et al., RSS 2025) ↩^,7DexOp: A Device for Robotic Transfer of Dexterous Human Manipulation (Fang et al., 2025) ↩^,8DexterityGen: Foundation Controller for Unprecedented Dexterity (Yin et al., RSS 2025) ↩ maps the operator's finger joints onto the robot, a faithful mirror of pose that discards the underlying physics. At the first finger-gaiting moment the object slips and drops. No reliable dexterous teleoperation means no dexterous demonstrations, and no demonstrations means no autonomous policies. Something is missing between the operator's intent and the fingers' execution.

World models and VLAs can plan from video at scale, but lack the dexterous data to act on contact-rich physics. Our long-term goal is to build the tip of the dexterous-data pyramid and give these models the physical dexterity to interact with the real world.

The Missing Cerebellum

The cerebellum learns internal models of body and object dynamics, turning intended movement goals into fast, physically grounded motor commands.

— after Ito, Nature Reviews Neuroscience 2008 · Wolpert, Miall & Kawato, Trends in Cognitive Sciences 1998

When you spin a pen or flip a coin, your brain does not plan every finger joint. It issues a goal — move the pen tip here, rotate the coin there — and your cerebellum translates that goal into the millisecond-level contact coordination that makes it physically happen. You never think about which finger to lift, when to re-establish contact, or how much force to apply. The cerebellum handles all of it, in real time, reflexively.

Current dexterous hands have no equivalent. The operator's intent goes straight to joint commands with nothing in between — no layer that understands contact, no layer that can improvise a finger sequence when the first one fails. The missing cerebellum.

TeleDexter fills this gap. The operator specifies synchronized geometric targets for both the fingertips and the object pose. A learned co-tracking controller — trained entirely in simulation via reinforcement learning — discovers the joint commands, contact sequences, and timing to physically realize them. The operator prescribes the goal; TeleDexter computes the motor commands needed to achieve it.

TeleDexter: Building the Cerebellum

You don't write a cerebellum. You grow one — from human demonstrations, through millions of simulated failures, to real hardware with no rehearsal.

Learning material: human demonstrations

We capture unscripted human hand–object interactions using a motion-capture system — translations, rotations, finger gaiting, tool-use sequences — covering the full range of contact modes the controller may encounter at deployment.

We run a two-stage geometry-aware retargeting to produce physically grounded reference motions for the robot hand. The first stage aligns hand kinematics; the second refines against the object mesh to enforce surface contact and prevent interpenetration.

Practice: learning to track in simulation

The controller is formulated as a hand–object co-tracking problem. The operator specifies synchronized targets for fingertip positions and object pose; the controller learns the joint commands, contact sequences, and timing to physically realize them. It is trained via RL in simulation on the reference motions.

• Consecutive subgoal tracking. Each reference trajectory is decomposed into a sequence of waypoints — synchronized fingertip and object-pose targets sampled at varying intervals. The policy must reach each sampled waypoint but freely discovers its own contact strategy between them. This operational slack is what turns RL from a trajectory copier into a contact strategist, allowing it to discover finger gaiting, rolling, and regrasping sequences that emerge only through dynamic simulation.
• Hybrid reward. A sparse reward fires each time the policy reaches a waypoint, weighted by the temporal distance from the previous one so that larger jumps earn proportionally larger bonuses. A dense tracking reward provides a small continuous signal at every timestep, shaping early exploration before any waypoint is reached.
• Curriculum learning. Training progressively hardens along three axes: gravity ramps to full, tracking tolerances tighten, and inter-waypoint distances grow. The policy builds a repertoire of stable contact primitives before the full difficulty kicks in.
• Random action masking. Random subsets of joints are frozen for short durations during training, forcing the policy to succeed with partially stale commands. This simple regularizer is the single most impactful intervention for zero-shot sim-to-real transfer.
• Single-stage RL. One policy, one reward function, one training run across all contact modes. Scaling to new objects requires only human demonstrations; the training recipe stays the same.

Going live: zero-shot deployment for teleoperation

The learned controller deploys directly to the real dexterous hand with no fine-tuning. A motion-capture system streams the operator's wrist, fingertip, and object poses in real time. Kinematic retargeting handles the initial grasp approach, and once stable contact is established, the operator switches to the co-tracking controller for dexterous manipulation.

What TeleDexter Can Do

Challenging tasks, two embodiments, all in one framework. Every clip below is a real-world rollout.

Multi-stage Tool Use

Human intent on the right, learned contact execution on the left. The operator streams fingertip and object targets; the co-tracking controller improvises the contact sequence that realizes them.

7 stages: pick up → rotate face-down → drive nails → rotate claw-down → pull nails → rotate → place

7 stages: pick up → rotate → sweep forward → rotate bristles-right → sweep right → rotate → place

5 stages: pick up → rotate → tighten screw → rotate → place

6 stages: pick up → rotate → screw in → unscrew → rotate → place

In-Hand Dexterity

The co-tracking controller generalizes beyond the training trajectories. Deployed zero-shot, it tracks a continuous stream of randomly sampled hand-object targets, reaching dozens of arbitrary goal transitions in sequence before failure.

Continuous pen spinning on the SharpaWave: a long uninterrupted take (left), a mid-spin recovery (right).

The same training recipe runs on a 4-finger, 16-DoF LeapHand⁹LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning (Shaw et al., RSS 2023) ↩. Only the retargeting stage adapts to the target morphology. The human demonstrations stay the same.

From Teleoperation to Autonomy

TeleDexter serves as a data engine for dexterous manipulation that no existing teleoperation system can produce. With 50 demonstrations per task, we train RGB autonomous policies via behavioral cloning¹⁰Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (Chi et al., IJRR 2025) ↩ — closing the loop from human dexterity to robot autonomy.

The Numbers

Limitations

TeleDexter is a reflex layer that survives contact-rich execution — a "cerebellum" for both human teleoperation and autonomous policies. It turns human dexterity into a renewable supply of demonstrations for autonomous manipulation, across hands. However, the role the human cerebellum plays in dexterous manipulation intelligence runs far deeper than what TeleDexter achieves today, and closing that distance remains a long road.

The learned controller is object-specific — adapting to a new object requires new human demonstrations and a dedicated training run. The system relies on a heavy motion-capture setup for real-time hand and object pose. Scaling to an object-general controller and replacing MoCap with vision-based tracking are the next steps.

Citation

@article{li2026teledexter,
  title   = {Towards Human-level Dexterous Teleoperation},
  author  = {Li, Puhao and Chen, Zeyuan and Wu, Yingying and Wei, Pengkun and Li, Yuyang and Wang, Tianyu and Shi, Jiaxiao and Yu, Mingrui and Jia, Baoxiong and Zhu, Song-Chun and Liu, Tengyu and Huang, Siyuan},
  journal = {arXiv preprint arXiv:2607.11481},
  year    = {2026}
}

Full Video Demo