Paper Walkthrough: CosmoH2G — Transferring a Human Hand's Flips and Rotations to a Two-Finger Gripper

Turning a video of a human hand into a two-finger gripper motion has mostly worked for flat, simple tasks like pushing and placing. CosmoH2G (arXiv:2609.07498) collects 6,189 paired hand–gripper episodes focused on rotations and flips, then fixes the start and end poses before generating the motion in between. A walkthrough grounded only in the paper's text.

CosmoH2G: A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements

The problem this paper tackles

The paper's full title is "CosmoH2G: A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements" (Zhao, Xu, Jin, Hao, Cui, Han; arXiv:2609.07498, September 2026). H2G stands for Hand-to-Gripper: converting a human hand's motion into the motion of a robot's two-finger gripper.

Here is the abstract's argument in plain terms. Transferring human hand demonstrations to robot grippers has become a cheap way to teach robots, but existing methods are largely confined to simple, planar tasks and cannot handle complex spatial movements such as trajectories with rotations or flips. The authors take an implicit, data-driven approach guided by fine-grained hand-pose motion. They build a scalable pipeline that collects paired hand and gripper demonstrations using a handheld gripper, under a protocol that prioritizes motion complexity, yielding 6,189 episodes across 1,254 objects. Learning the mapping naively, end to end, turns out to be insufficient: small trajectory deviations compound quickly under intricate dynamics. So they propose a two-stage framework — Stage I predicts sparse keyframes (the initial and terminal gripper poses), Stage II generates the full continuous sequence conditioned on them. To limit cumulative drift, orientation is learned while translation is post-optimized using a grasping heuristic and kinematic consistency. In both simulation and on a real robot, the framework significantly outperforms traditional baselines.

An analogy: watch a hand, reproduce it with tongs

Imagine a video of someone picking up an egg tart, flipping it over above a plate, and setting it down. Now do the same thing with a pair of tongs — a tool with only two fingers. You cannot copy the five-finger grip directly: the joint count, the contact area, and the grasp freedom are all different. The paper calls this the embodiment gap and sorts prior work into two camps (§1, §2.1).

1. Rule-based retargeting: a hand-written table maps hand keypoints to gripper poses every frame — for instance "the grasp point is the midpoint between thumb and index finger." Works for pinch grasps; breaks on natural grips and large rotations (Fig. 4).
2. Trajectory-conditioned methods: ignore the hand, track only the object's path in the video, and have the robot follow that path. This sidesteps the embodiment gap, but tracking is fragile in cluttered scenes, and the fine-grained information carried by the hand pose — keeping a tool oriented, re-gripping to avoid a collision — is thrown away.

Fig. 3 shows the second failure concretely. When the object rotates a lot, the tracked points get occluded behind the object itself and the tracker drifts onto whatever is still visible. Flipping is the worst case for anything that follows tracked points.

The authors' stance is therefore to stop writing correspondence rules by hand and instead learn the correspondence from paired data in which a hand and a gripper perform the same task (§1). Which means the data has to exist first.

The data: "do it again, with a handheld gripper"

Half of this paper is about the dataset. The collection protocol has four parts (§3.1).

• Hand side: collectors vary their grip and contact regions freely, not just pinching. Every task includes in-plane rotation and vertical flipping.
• Gripper side: a handheld gripper called UMI is used to imitate the same task on the spot, matching the contact area, grasp orientation, and motion of the hand demonstration. This is far faster than teleoperating a robot (§2.2) and captures complex motion.
• Recording: only fixed-viewpoint RGB-D (color plus depth) video, which avoids recalibrating whenever the environment changes.
• Verification: after recording, the object poses at the first and last frames are checked for consistency between the hand and UMI demonstrations; mismatched pairs are discarded.

The result is 1,254 objects and 6,189 episodes, 3–5 pairs per object. By category: food 23.03%, decor 27.88%, beauty 3.64%, necessities 12.12%, toys 33.33% (Table 1). From the videos, the pipeline reconstructs the object mesh, registers the object in the first and last frames, reconstructs hand meshes per frame, and tracks the UMI using its CAD model, producing object point clouds, hand mesh sequences, and 6-DoF gripper pose sequences (§3.2).

Pair quality is also quantified. A trajectory similarity score between the hand and UMI paths is computed and only pairs above 0.9 are kept; the retained pairs average 0.957 (median 0.958) (§3.2, Fig. 13). Per Appendix D.1, the score is the DTW (dynamic time warping) distance between the two 3D trajectories, normalized by length and mapped to 0–1 with a tolerance of 20% of the table width. DTW aligns two motions that have the same shape but different timing, and it is a dynamic-programming algorithm in the same spirit as the edit-distance table below: fill the grid one cell at a time to find the best alignment.

The paper claims the dataset's orientation variation far exceeds that of Robosuite, MimicPlay, and RT-1 (§3.3, Fig. 2). The comparison is a histogram only; no numbers appear in the text.

Why generating everything in one shot failed

With the data in hand, the obvious plan is to train one model that maps the hand point-cloud sequence to a gripper pose sequence. The paper reports that this naive approach did not reach the precision needed for reliable grasping. Three single-stage models — a denoising Transformer, Diffusion Policy, and ACT — all landed at 58–65% success rate and 47–53° placement orientation error (§5.3, Table 3). The stated reason: under complex motion, small trajectory deviations compound rapidly, so an early error corrupts the grasp position and final orientation.

The fix is to split the problem in two (§4). Stage I generates 6-DoF gripper poses for just two frames — the grasp and the placement. Stage II generates the motion between them, conditioned on those two poses. But Stage II generates orientation only; position is computed geometrically from the hand trajectory and then post-optimized. Pin both ends of a hard mapping first, then fill in the middle.