Paper: Reconstructing Hands in 3D with Transformers
Venue: CVPR 2024
Authors: Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, Jitendra Malik
Affiliations: UC Berkeley, NYU
arXiv: 2312.05251
GitHub: geopavlakos/hamer
One-Line Summary
A fully transformer-based 3D hand mesh recovery model built on a ViT-H backbone and Transformer decoder, achieving 2–3× better in-the-wild generalization over prior methods via 2.7M training examples and a new benchmark (HInt).
1. Background and Problem Definition
Monocular 3D Hand Mesh Recovery
The task is estimating the full 3D shape and pose of a hand from a single RGB image. Complete hand meshes are a core input for AR/VR, human-computer interaction, robotics, and medical analysis.
Common failure modes of prior methods:
- Brittle CNN backbones: Limited receptive fields and inductive biases cause failure on in-the-wild images
- Small studio datasets: Training on controlled, small-scale data that does not reflect real environments
- Inability to handle occlusions and interactions: Performance collapses under hand-hand, hand-object interaction, or heavy occlusion
- Limited diversity: Robust only to specific skin tones, lighting, and viewpoints
Scaling Philosophy
HaMeR’s approach rests on a simple premise:
“Recent developments in computer vision and NLP point to the direction where advances are achieved by simple, high capacity models, powered by huge amounts of data.”
Rather than complex architectural design or domain-specific inductive biases, HaMeR tests the hypothesis that scaling both model capacity and data simultaneously works for 3D hand reconstruction as well.
2. Output Representation: MANO Hand Model
HaMeR uses MANO, a parametric hand model, as its output space.
MANO parameters:
- Pose \(\theta \in \mathbb{R}^{48}\): Finger joint rotations (PCA-based)
- Shape \(\beta \in \mathbb{R}^{10}\): Per-identity hand shape variables
- Camera \(\pi\): Weak-perspective camera translation
The full output \(\Theta = \{\theta, \beta, \pi\}\) deterministically yields a 778-vertex mesh and 21 joint locations.
Two reasons for using MANO as output: a compact parameter space eases optimization, and only physically plausible hand shapes can be produced.
3. Architecture
Full Pipeline
Single RGB image (hand bounding box crop)
→ ViT-H image encoder → patch token sequence
→ Transformer decoder (single query token, cross-attends to all patch tokens)
→ MANO parameter regression (θ, β, π)
→ MANO layer → 3D mesh + joint coordinates
- Splits the image into fixed-size patches and produces a token sequence
- Global self-attention captures full image context simultaneously
- Fine-tuned from ImageNet-21K pretrained weights
The key advantage of ViT-H over CNN backbones is the global receptive field. Every layer attends to the full image, making it easier to reason about occluded or truncated hand regions.
A single query token performs cross-attention over all ViT-H output patch tokens. The query aggregates the full image information and regresses MANO parameters.
The design is deliberately simple: a single forward pass produces the final output without iterative refinement or multi-stage regression.
4. Loss Functions
Three losses are jointly optimized.
3D Loss (datasets with 3D ground truth)
\[\mathcal{L}_{3D} = \|\theta - \theta^*\|_2^2 + \|\beta - \beta^*\|_2^2 + \|X - X^*\|_1\]
L2 loss on pose and shape parameters plus L1 loss on 3D joint coordinates.
2D Reprojection Loss
\[\mathcal{L}_{2D} = \|x - x^*\|_1\]
L1 loss between projected 2D joint coordinates and 2D keypoint annotations. Enables training on datasets that have only 2D annotations and no 3D ground truth.
Adversarial Loss (for 2D-only data)
\[\mathcal{L}_{adv} = \sum_k (D_k(\Theta) - 1)^2\]
Three discriminators are used:
- Full shape discriminator: Judges whether the overall MANO parameters correspond to a natural hand
- Full pose discriminator: Judges the plausibility of the full hand pose
- Per-joint discriminator: Judges individual finger joint angle naturalness
The adversarial loss suppresses unnatural hand poses that arise when training without 3D supervision.
5. Training Data Scaling
2.7M Training Examples
4× larger than the FrankMocap baseline. Ten heterogeneous datasets are combined.
Datasets with 3D annotations:
| Dataset |
Characteristics |
| FreiHAND |
Studio, single hand |
| HO-3D |
Hand-object interaction |
| MTC (Panoptic Studio) |
Multi-camera capture |
| RHD |
Synthetic data |
| InterHand2.6M |
Two-hand interaction |
| H2O3D |
Hand-object interaction |
| DexYCB |
Hand-object manipulation |
Datasets with 2D annotations only:
| Dataset |
Characteristics |
| COCO WholeBody |
Natural environments |
| Halpe |
Person photography |
| MPII NZSL |
Sign language |
For 2D-only datasets, only the reprojection and adversarial losses are applied — no 3D loss. This allows in-the-wild data that lacks 3D ground truth to contribute to training.
6. HInt Dataset: A New In-the-Wild Benchmark
Limitations of Existing Benchmarks
Benchmarks like FreiHAND and HO3Dv2 are collected in controlled environments. They cannot adequately measure generalization to real-world conditions (egocentric video, hand-object interaction, varied lighting).
HInt (Hand Interactions in the Wild)
A new in-the-wild benchmark with 40,400 annotated hands.
Key features:
- 2D keypoints for 21 joints + per-keypoint occlusion labels (first dataset to provide these)
- 86.7% of hands are in contact scenarios
- 90.5% inter-annotator agreement on occlusion labels
- 94.6% of visible keypoints within 0.25× palm length across annotators
Three sources:
| Source |
Count |
Characteristics |
| Hands23 (New Days of Hands) |
12.0K |
Third-person, natural environments |
| Epic-Kitchens VISOR |
5.3K |
Egocentric, kitchen settings |
| Ego4D |
23.2K |
Egocentric, diverse activities |
Being the first large-scale in-the-wild hand dataset to provide occlusion labels is significant — it enables separate measurement of performance on occluded vs. visible joints.
7. Experimental Results
FreiHAND Benchmark (Table 1)
| Method |
PA-MPJPE (mm) ↓ |
PA-MPVPE (mm) ↓ |
F@5mm ↑ |
F@15mm ↑ |
| I2L-MeshNet |
7.4 |
7.6 |
0.681 |
0.973 |
| MobRecon |
5.7 |
5.8 |
0.784 |
0.987 |
| HaMeR |
6.0 |
5.7 |
0.785 |
0.990 |
HaMeR achieves state-of-the-art on FreiHAND, on par with or marginally above prior methods on this studio benchmark.
HO3Dv2 Benchmark (Table 2)
| Method |
AUCⱼ ↑ |
PA-MPJPE (mm) ↓ |
AUCᵥ ↑ |
| HandOccNet |
0.831 |
8.8 |
— |
| AMVUR |
0.835 |
8.3 |
0.836 |
| HaMeR |
0.846 |
7.7 |
0.841 |
Best on all metrics for HO3Dv2, which includes hand-object interaction.
HInt Benchmark: PCK@0.05 (Table 3) — Core Result
| Method |
New Days |
VISOR |
Ego4D |
| FrankMocap |
16.1% |
16.8% |
13.1% |
| HandOccNet (param) |
9.1% |
8.1% |
7.7% |
| HaMeR |
48.0% |
43.0% |
38.9% |
HaMeR shows 2–3× improvement over all prior methods on in-the-wild data. This is the paper’s strongest claim.
Breakdown by occlusion status (VISOR):
| Split |
HaMeR |
| Visible joints |
56.6% |
| Occluded joints |
25.9% |
8. Ablation: Data Scale vs. Model Scale
Independent Contributions and Synergy (Table 5)
| Config |
Large Data |
Large Model |
New Days |
VISOR |
Ego4D |
| FrankMocap |
✗ |
✗ |
16.1% |
16.8% |
13.1% |
| Base (ResNet50) |
✗ |
✗ |
16.9% |
17.5% |
13.9% |
| + Large data only |
✓ |
✗ |
31.3% |
29.9% |
24.7% |
| + Large model only |
✗ |
✓ |
25.9% |
24.1% |
19.4% |
| HaMeR (both) |
✓ |
✓ |
48.0% |
43.0% |
38.9% |
Key observation: large data alone gives +14.4%p, large model alone gives +9.8%p, but together they give +31.1%p — a synergistic effect larger than the sum of independent contributions. Data scale and model capacity amplify each other.
Effect of HInt Training Data (Table 4)
After fine-tuning with HInt’s training split:
| Dataset |
Without HInt |
With HInt |
Improvement |
| VISOR (all) |
43.0% |
56.5% |
+13.5%p |
| VISOR (visible) |
56.6% |
66.5% |
+9.9%p |
| VISOR (occluded) |
25.9% |
42.6% |
+16.7%p |
| Ego4D (all) |
38.9% |
46.9% |
+8.0%p |
The larger gain on occluded joints (+16.7%p) compared to visible joints (+9.9%p) demonstrates that HInt’s occlusion labels directly improve occlusion handling.
9. Qualitative Generalization
Scenarios where HaMeR demonstrates robustness:
- Egocentric and third-person viewpoints
- Hand-hand and hand-object interactions with occlusion
- Motion blur, diverse lighting conditions
- Diverse skin tones
- Non-standard appearances (gloves, robotic hands, illustrations)
- Temporally smooth video output from per-frame inference (no temporal smoothing applied)
10. Limitations
- Spurious detections: False positives from the upstream hand detector propagate through the pipeline
- Left/right classification errors: Occasional misclassification of hand side
- Extreme poses: Performance degrades on highly unusual finger configurations
- Severe occlusion: Improved by HInt training but still challenging under complete occlusion
- No temporal modeling: Single-frame approach with no explicit temporal consistency
- No 3D GT for in-the-wild data: Only 2D PCK evaluation is possible; 3D quantification is unavailable in-the-wild
11. Summary
HaMeR’s central claim is one: in 3D hand reconstruction, model and data scale matter more than architectural complexity.
Concretely:
- A simple pipeline: ViT-H + Transformer decoder
- 2.7M training examples from 10 heterogeneous datasets
- A new in-the-wild benchmark, HInt (40.4K hands with occlusion labels)
These three elements together produce 2–3× better in-the-wild performance over prior methods. The synergy between data scale and model capacity is particularly striking — combining them outperforms the sum of their individual contributions.
HaMeR demonstrates that LLM-style scaling laws hold in the domain of 3D human reconstruction.
“Instead of complex inductive biases — a large enough model with enough data. Hand reconstruction is no exception.”
]]>
