100Days Of Hands Video Dataset 131 Days boardgame diy drink food - PowerPoint PPT Presentation

Dandan Shan 1 , Jiaqi Geng 1 *, Michelle Shu 2 *, David F. Fouhey 1 University of Michigan 1 , Johns Hopkins University 2 Sponsored by Procter & Gamble and Nokia Networks

100Days Of Hands Video Dataset 131 Days boardgame diy drink food 12 Categories 27.3K Videos furniture gardening housework packing 19.2K Uploaders puzzle repair study vlog

100K Frame-level Annotations • Box around hand • Side (left / right) • Contact (no / self / other / portable / furniture) • Box around object in contact • Association

Method Hand Standard Detection Box Side Classification (if hand) State Classification (if hand) Offset Regression (if in contact) Faster-RCNN simple greedy matching Object Standard Detection on hands and objects Box (another class) [1] Ren et al. Faster R-CNN: Towards real-time object detection with region proposal networks . NIPS 2015. [2] Yang et al. A Faster Pytorch Implementation of Faster R-CNN (https://github.com/jwyang/faster-rcnn.pytorch) .

Cross-Dataset Hand Detection Faster RCNN with Resnet-101 backbone, AP (TP = IoU > 0.5) 100DOH VLOG VIVA Ego VGG TV+Co 100DOH 90.1 86.4 86.5 90.8 73.9 65.4 VLOG 78.6 77.5 76.6 83.2 64.6 59.2 VIVA 23.6 27.7 90.8 56.8 21.5 10.1 Train on Ego 40.7 32.6 44.9 90.7 17.4 8.0 VGG 61.4 61.7 78.8 63.0 79.6 66.6 TV+Co 56.2 61.5 74.9 62.4 77.4 69.9 >80 >70 >60 >40 <40

Qualitative Results

Method Hand Box L Hasson Side et al. State R Offset Given location and side , can predict Object low-dimensional parameterization of hand Box [1] Hasson et al. Learning joint reconstruction of hands and manipulated objects . CVPR 2019. [2] Romero et al. Embodied Hands: Modeling and Capturing Hands and Bodies Together. SIGGRAPH Asia 2017.

Enabling MANO at Scale [1] Hasson et al. Learning joint reconstruction of hands and manipulated objects . CVPR 2019. [2] Romero et al. Embodied Hands: Modeling and Capturing Hands and Bodies Together. SIGGRAPH Asia 2017.

Hand Predictions Input Prediction Input Prediction

Code and Data Available! 100 Days Of Hands Mesh Quality Assessment 100DOH Video Dataset 100K labeled Frames Full Hand State Detection Hand Detection Full Hand State Detection Model (egocentric) Model (detectron2) Model