Hand-in-the-Loop: Improving VLA Policies for Dexterous Manipulation via Seamless Hand-Arm Intervention

Abstract

Vision-Language-Action (VLA) models are prone to compounding errors in dexterous manipulation, where high-dimensional action spaces and contact-rich dynamics amplify small policy deviations over long horizons. While Interactive Imitation Learning (IIL) can refine policies through human correction data, applying it to high-degree-of-freedom (DoF) robotic hands remains challenging due to a command mismatch between human teleoperation and policy execution at the intervention moment, which causes abrupt robot-hand configuration changes, or “gesture jumps”. We present Hand-in-the-Loop (HandITL), a seamless human-in-the-loop intervention method that blends human corrective intent with autonomous policy execution to avoid gesture jumps during bimanual dexterous manipulation. Compared with taking over control using direct teleoperation, HandITL reduces intervention jitter by 99.8% and preserves robust post-intervention manipulation, reducing grasp failures by 87.5% and mean completion time by 19.1%. We validate HandITL on tasks requiring bimanual coordination, tool use, and fine-grained long-horizon manipulation. When used to collect correction data for policy refinement, HandITL yields policies that outperform those trained with standard teleoperation data by 19% on average across three long-horizon dexterous tasks.

Method

Hand-Arm Seamless Intervention

HandITL enables high-dimensional human correction during ongoing dexterous VLA rollouts without disrupting autonomous execution. At each intervention step, the operator’s corrective intent is fused with the policy command to generate the final executed action:

\mathbf{a}_{t}^{exec} = \alpha \mathbf{a}_{t}^{\pi} + \beta \mathbf{a}_{t}^{h}

where $\alpha$ and $\beta$ control the relative authority of the policy and the human operator. This formulation supports both full takeover for substantial recovery and copilot shared control for small residual corrections while preserving the policy’s ongoing behavior.

Optimization-Based Relative Hand Retargeting

A key challenge in dexterous intervention is the command mismatch between human teleoperation and policy execution at the intervention moment. Directly switching to absolute pose retargeting can cause abrupt robot-hand configuration changes, or gesture jumps, which may destabilize ongoing grasps.

To address this, HandITL performs optimization-based relative hand retargeting anchored at the intervention timestamp. Instead of matching absolute human hand poses, the robot hand follows the operator’s incremental fingertip motions from the intervention onset while preserving its own current grasp configuration. This enables smooth multi-finger correction without requiring initial pose alignment between the human hand and the robot hand.

The retargeting objective combines global hand-shape tracking, precision grasping, structural safety, and temporal regularization. Together, these terms enable smooth, safe, and contact-preserving dexterous correction during intervention.

Velocity-Based Shared Arm Control

For arm correction, HandITL uses a velocity-based shared-control interface. Human wrist motions are converted into residual end-effector twists, smoothed with an exponential moving average, and injected into the policy-predicted arm commands. Because the correction is derived from relative motion, the residual naturally decays to zero when the operator stops moving, avoiding persistent drift without requiring a manually defined neutral position.

Experiments

We evaluate HandITL from three perspectives: intervention command discontinuity, post-intervention manipulation capability, and policy evaluation on long-horizon tasks. The experiments are conducted on real bimanual dexterous manipulation tasks involving tool use, fine-grained grasping, and multi-stage task execution.

Intervention Command Discontinuity

We first measure whether different intervention methods introduce abrupt hand-command changes at the intervention moment. Direct teleoperation switching often causes large gesture jumps, leading to tool drops or unstable grasps. In contrast, HandITL preserves the robot’s current grasp and only applies relative corrective motion.

data_pyramid — Intervention command discontinuity on the Drill (top) and Bread Clip (bottom) tasks.

On the Bread Clip task, HandITL reduces the mean command discontinuity from approximately $4.38 \times 10^{-2}$ to $6.8 \times 10^{-5}$ , achieving a 99.8% reduction. On the Drill task, it also reduces command discontinuity while maintaining stable trigger contact during intervention.

Post-Intervention Manipulation Capability

We further test whether the robot can still perform precise manipulation after intervention. Compared with direct teleoperation and differential baselines, HandITL achieves smoother finger control, fewer grasp failures, and better cross-operator consistency.

makeup_result — Post-intervention manipulation capability on the Pick Up and Place the Parts and Pick Up the Drill tasks.

On the Pick Up and Place the Parts task, HandITL achieves the fastest mean completion time ( $42.8\,\mathrm{s}$ ), improves efficiency by $19.1\%$ , and reduces grasp failures by $87.5\%$ compared with teleoperation. These results show that relative retargeting not only makes intervention smoother, but also preserves practical dexterous manipulation capability after intervention.

Policy Evaluation on Long-Horizon Tasks

Finally, we study whether on-policy correction data can improve downstream VLA policy performance. Starting from a base policy fine-tuned on a 20-hour teleoperation dataset, we compare additional post-training with pure teleoperation data, full-takeover correction data, and copilot shared-control correction data. For a fair comparison, all additional datasets are 1 hour long and are mixed with the base dataset at the same additional-to-base sampling ratio of 0.5:1 during post-training. All runs use identical training schedules and hyperparameters.

Comparing the five policies reveals three critical insights regarding long-horizon performance:

Limited Improvements from Pure Teleoperation: Simply increasing pure teleoperation data brings limited and inconsistent gains. Both Teleop_old and Teleop_new show only marginal improvements, and their effects vary across tasks. This suggests that additional off-policy demonstrations may not adequately cover policy-induced states where compounding errors occur, especially in late phases requiring precise contact-rich manipulation.
Effectiveness of Intervention Data: In contrast, policies fine-tuned with intervention data achieve higher average normalized completion scores. Since Copilot and Full Takeover data are recorded from deployed policy rollouts and include human corrections at failure-prone moments, they provide targeted supervision on policy-induced states and recovery behaviors. This makes intervention data more effective than standard teleoperation demonstrations for improving long-horizon robustness.
Copilot vs. Full Takeover: Among the two intervention strategies, Copilot generally yields the strongest overall performance. Unlike Full Takeover, where human commands dominate the executed trajectory, Copilot keeps the policy as the primary controller while injecting local residual corrections. The resulting data stays closer to the policy’s rollout distribution, leading to more stable downstream improvements.

Citation

  @misc{li2026handintheloopimprovingvlapolicies,
      title={Hand-in-the-Loop: Improving VLA Policies for Dexterous Manipulation via Seamless Hand-Arm Intervention}, 
      author={Zhuohang Li and Liqun Huang and Wei Xu and Zhengming Zhu and Nie Lin and Xiao Ma and Xinjun Sheng and Ruoshi Wen},
      year={2026},
      eprint={2605.15157},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2605.15157}, 
  }