DragMesh-2: Dexterous Articulated Manipulation Through Contact

Quick answer

DragMesh-2 manipulates articulated objects (dishwasher doors, storage drawers, a microwave) with a 51-DoF SMPL-X hand whose policy controls only the hand. The target joint has no action channel, so the door or drawer moves only when sustained hand-handle contact drags it. The headline number: its PICA training scheme reaches 0.89 task success at nominal damping and holds 0.56 at 4x damping on seven GAPartNet objects, against 0.58 / 0.27 for a state-only PPO baseline, all without RGB, depth, force, or tactile input. The gain comes from a contact-history encoder plus auxiliary contact-prediction losses, not from the base RL alone.

The problem: articulated motion you cannot command

Most manipulation policies cheat a little. The object’s joint gets its own controller, or a scripted trajectory drives it open, and the hand just rides along. DragMesh-2 removes that crutch. The drawer joint is passive. If the hand loses grip, the drawer stops. So the policy has to learn the one thing that is hard to learn blind: keep enough contact force on the handle, in the right direction, for the whole pull, while only seeing kinematics.

This is why the paper is worth reading even though the absolute numbers are modest. It separates the manipulation problem from the actuation shortcut, and that makes the failure modes honest. The name “DragMesh-2” signals the lineage: it extends an object-centric mesh-dragging generator into a hand-driven interaction policy.

How PICA injects physics without a force sensor

The observation has no force or tactile channel. PICA (Physically Informed Contact-Aware training) reconstructs a sense of contact from kinematics through three pieces.

A contact-history temporal encoder packs the current PD tracking error and the previous action into history tokens, then runs a gated linear attention encoder over recent interaction blocks. That gives the policy a short memory of how contact has been behaving.

On top of that, a causal-window auxiliary head predicts four contact-response signals from those temporal features: the recent object-joint response, the maximum palm-handle distance, a detachment-risk proxy, and the maximum tracking stress. Predicting over K-step windows rather than single-step differences smooths out the noise the contact solver injects.

The reward then adds explicit physical terms: contact maintenance, action regularization, a detachment penalty, action-boundary saturation suppression, and contact-distance regularization. Training also randomizes the joint damping over a [1.0, 2.0] multiplier so the policy does not overfit one dynamics setting. The same idea of pushing physical structure into a learned controller shows up in other embodied work like Playful Agentic Robot Learning.

Key results

• Success vs damping (deterministic, 7 objects, 20 episodes each): PICA reaches 0.89 at 1x, 0.80 at 2x, 0.56 at 4x. State-only PPO trails at 0.58 / 0.44 / 0.27, and a Transformer-PPO variant is worse still at 0.35 / 0.23 / 0.09. The contact machinery roughly doubles OOD success.
• Ablation, who earns the gain: GLA encoder alone (no PICA signals) gets 0.65 / 0.56 / 0.36; PICA signals alone (no GLA) get 0.75 / 0.71 / 0.43; the full system gets 0.89 / 0.80 / 0.56. Neither half reproduces the whole, so the lift is a joint effect of memory plus contact supervision.
• Stochastic execution holds up: sampling actions instead of taking the mean, PICA still scores 0.82 / 0.72 / 0.43, well above state-only PPO at 0.44 / 0.35 / 0.26.
• Nominal success hides collapse: extending training from 150 to 500 epochs raised 1x success from 0.90 to a perfect 1.00 while 4x success fell from 0.55 to 0.10, and action-saturation (fraction of steps at near-max action) climbed to 0.99. The pretty in-distribution number predicted the opposite of robustness.
• Baseline honesty: a scripted trajectory-tracking baseline actually beats PICA in-distribution (1.00 at 1x) but only ties it on average under load (0.71 at 4x), and a parallel-jaw primitive flatlines at 0.14 across all damping, which tells you the dexterous formulation is doing real work the gripper cannot.

Read this before you over-read the numbers

The easy misread is “0.89 success, articulated manipulation is solved.” It is not. That 0.89 is at nominal damping on seven objects with privileged state observation and no perception stack. Drop to 4x damping, the regime the authors call out-of-distribution, and even the best system lands at 0.56, with detachment becoming the dominant failure mode. The paper’s own diagnostic is the point: it shows that the in-distribution score is a misleading headline, and that contact inferred only from kinematic error is not enough for stable light pulling when damping is high.

For a builder, the takeaway is structural. If you are training a contact-heavy policy, do not select checkpoints on nominal-damping success, because that metric climbs while real robustness drops. Watch action-saturation and a detachment proxy instead. The four auxiliary signals here are cheap to add and double as early-warning instrumentation, which may be more transferable than the specific architecture.

Limits and open questions

The evidence is narrow: seven GAPartNet objects, all simulated, with privileged kinematic state and no vision, force, or tactile input. The authors are upfront that contact can only be inferred indirectly from tracking error, and that this is insufficient for stable light pulling at high damping. Extended fine-tuning past 50 epochs on the combined modules yields no further OOD gains, suggesting a robustness ceiling without richer contact sensing. There is also strong per-object heterogeneity at 4x damping, so the average hides instance-specific difficulty. A real-robot demonstration appears, but the quantitative story is sim-only. Whether adding even a coarse tactile channel breaks the ceiling is the obvious next question, and the closest reference points are perception-grounded embodied agents like ABot Earth-0.5.

FAQ

What is DragMesh-2 and how is it different from a normal grasping policy?

DragMesh-2 is a contact-driven framework for dexterous manipulation of articulated objects. The difference from a normal grasping or articulation policy is that the object’s joint has no actuator: the drawer or door only moves through sustained hand-handle contact. The 51-DoF hand has to maintain the right contact force and direction for the whole motion using only kinematic observations, with no force or tactile sensor.

Does DragMesh-2 work under out-of-distribution damping, or only in easy conditions?

Partly. At nominal (1x) damping PICA reaches 0.89 success, but under 4x damping, which the paper treats as out-of-distribution, the best system drops to 0.56 with detachment becoming the dominant failure mode. That is still well above the 0.27 of a state-only PPO baseline at 4x, so the contact-aware training clearly helps under load, but it does not solve the high-damping regime.

Why did training DragMesh-2 longer make it worse, and what does that mean for evaluation?

Extending training from 150 to 500 epochs pushed 1x success from 0.90 to 1.00 while 4x success collapsed from 0.55 to 0.10 and action-saturation climbed to 0.99. The policy was overfitting to a single dynamics setting and slamming actions to their limits. The lesson is that nominal-damping success is a misleading checkpoint-selection metric for contact-heavy tasks, and saturation or detachment proxies are safer to watch.

One line: DragMesh-2 shows you can drag articulated objects open with a dexterous hand and no tactile sensor, reaching 0.89 success in-distribution and 0.56 under 4x damping, but its real contribution is proving the in-distribution score lies about robustness. Read the original paper on arXiv.