Weakly Supervised Segmentation Guided Hand Pose Estimation During Interaction with Unknown Objects

Hand pose estimation is important for human computer interaction, but the performance is not satisfying when the hand is interacting with objects. To alleviate the influence of unknown objects, we propose a novel weakly supervised segmentation guided scheme to estimate hand poses. Approximate hand masks generated from annotations of sparse hand joints are used to supervise the segmentation task. Better features can be extracted since they are shared between the two tasks of hand segmentation and hand pose estimation. With the guidance of weakly supervised segmentation, the network can learn intermediate features balanced between focusing on the foreground and preserving contextual information. Finally the xy and z coordinates are estimated in different branches but utilizing shared feature maps. Experimental results of three different tasks on the publicly available FHAD dataset demonstrate the effectiveness of the proposed architecture.