mmHand: 3D hand pose estimation using millimeter-wave radar

Abstract

Hand pose recognition provides attractive applications, including smart homes, AR/VR, etc, which can facilitate HumanComputer Interaction to solve problems. Traditional vision-based hand pose recognition fail in poor lighting, and also have privacy issues, while mmWave signals can effectively solve these problems, while protecting privacy. In this paper, we propose mmHand, a system for high-precision and fine-grained gesture recognition using commercial millimeter-wave devices. Specifically, we use millimeter-wave radar to collect data, generate point clouds of the environment, and use camera for cross-modal supervised training to process and enhance the point clouds. Finally, cross-attention mechanism is used to achieve recognition of hand joints from the enhanced point clouds data. Extensive experimental results show that the MPJPE (Mean Per Joint Position Error) of mmHand is 0.45cm, and when the normalization error is 0.2, the recognition accuracy can reach 94%. The MPJPE in different environments and distances are both at low level, which demonstrates the superior robustness and effectiveness of our system.