VTAP Gripper

This paper presents a tactile-reactive gripper that integrates a Visuo-Tactile Active Palm (VTAP) and compliant, reconfigurable fingers equipped with tactile array sensors. The design exploits structured finger-palm synergy and multi-modal perception to achieve both robust grasping and fine manipulation. The actuated bi-modal palm seamlessly combines long-range visual localization with contact-rich tactile feedback, substantially extending the system's manipulation capability. To bridge the embodiment gap between human hand motion and the heterogeneous three-finger structure, we further propose a staged, gesture-conditioned retargeting framework for dexterous teleoperation. Extensive experiments validate the system across a range of challenging tasks: reactive grasping of YCB and fragile objects, in-hand syringe reorientation and plunger actuation, singulation of clustered objects down to 3 mm in diameter, and vision-tactile peg-in-hole insertion. Results demonstrate that high manipulation performance can be achieved through coordinated finger-palm interaction and multi-modal sensing, without resorting to high degrees of freedom anthropomorphic designs. The VTAP gripper and its retargeting framework offer a practical reference architecture for dexterous gripper design, manipulation, and contact-rich data collection in support of learning-based approaches.

Overview of the proposed retargeting framework for teleoperated dexterous manipulation. (a) Experimental setup. (b) Gesture recognition for selecting grasp primitives of the reconfigurable three-finger gripper (cage, power, and pinch). (c) Key optimization objectives for human-to-gripper retargeting. (d) Mapping of thumb-index rolling motions to the gripper's adduction/abduction DOF for fine in-hand singulation.

Videos of in-hand singulation across five representative objects with varying size, geometry, and stiffness. Each video shows the transition from an initial multi-object grasp with multiple tactile contact patterns, through object removal during in-hand manipulation, to successful isolation of a single object verified by a distinct tactile contact pattern on each fingertip.

In-hand syringe reorientation and plunger actuation via teleoperation using the proposed gripper. The sequence illustrates pinch-based grasping, in-hand reorientation to align the syringe outlet, and coordinated palm actuation for controlled plunger depression.

Peg-in-hole insertion enabled by the bi-modal actuated palm. Vision mode is used for initial pose estimation, followed by tactile-based hole localization through edge extraction and circle fitting. Finally, the robotic arm adjusts the end-effector pose based on the estimated hole center and executes peg insertion with a 1 mm tolerance.

VTAP Gripper:

Synergizing Fingertip Sensing and a Visuo-Tactile Active Palm for Dexterous In-Hand Manipulation

Yuhao Zhou¹, Sheeraz Athar^1, Zhixian Hu^1, Binghao Huang², Yunzhu Li², Juan Wachs¹, and Yu She^1✉

Summary

Design

Mechanical

Sensing

Retargeting Framework

Experiments

Tactile-Reactive Grasping

In-Hand Object Reorientation

Singulation

Dexterous In-Hand Manipulation

Autonomous Peg-in-Hole Insertion

Bibtex

Acknowledgements

Fun Facts