SLIDE 1
Where are they looking?
Adria Recasens*, Aditya Khosla*, Carl Vondrick, Antonio Torralba
Where are they looking?
Follow the gaze of the person and identify the object being looked at
Where are they looking? Adria Recasens*, Aditya Khosla*, Carl - - PowerPoint PPT Presentation
Where are they looking? Adria Recasens*, Aditya Khosla*, Carl - - PowerPoint PPT Presentation
Where are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick, Antonio Torralba Presented by: Surbhi Goel Where are they looking? Follow the gaze of the person and identify the object being looked at Demo:
SLIDE 2
SLIDE 3
Demo: http://gazefollow.csail.mit.edu/demo.html
SLIDE 4
Experiments
- Dataset Visualizations
○ Images in the Dataset ○ Head Locations ○ Gaze Locations/Length
- Model Experiments
○ Qualitative Evaluation ○ Visualizing Gaze Mask and Saliency Map ○ Animal Gaze Following ○ Extending to Short Video
SLIDE 5
Dataset Visualizations
SLIDE 6
Training Set Images
SLIDE 7
Training Set Images
SLIDE 8
Training Set Images
SLIDE 9
Heatmaps for Head Location
Train Test
SLIDE 10
Heatmaps for Gaze Location
Train Test
SLIDE 11
Heatmaps for Relative Gaze Location
Train Test
SLIDE 12
Histogram for Length of Gaze
Train Test
SLIDE 13
Observations
- Head/Gaze are concentrated for train and scattered for test
- Relative gaze is concentrated for both
- Gaze length relatively short (0.2 peak)
SLIDE 14
Model Evaluation
SLIDE 15
Good Cases
SLIDE 16
Good Cases
SLIDE 17
Bad Cases
Head fully tilted but missed
SLIDE 18
Bad Cases
Face forward but eyes tilted No object of attention
SLIDE 19
Bad Cases
Back facing
SLIDE 20
Observations
- Handle groups well
- Gaze location is very accurate, head location often not
- Unable to capture eye movement independent of face orientation
- Fails at a lot of back facing cases
SLIDE 21
Gaze Mask and Saliency Map
SLIDE 22
Gaze Mask and Saliency Map
- Gaze Mask incorporates the general direction of gaze
- Saliency Map incorporates the salient objects in image
- Element-wise product captures locations that satisfy both
SLIDE 23
Gaze Mask and Saliency Map
Image with Gaze Gaze Mask Saliency Map
SLIDE 24
Animal Gaze Follow
SLIDE 25
Animal Gaze Follow
SLIDE 26
Animal Gaze Follow
Works (almost) for even birds
SLIDE 27
Animal Gaze Follow
Works even when more than one salient object
SLIDE 28
Animal Gaze Follow
- Model generalizes to animals
○
Initialized with ImageNet which has animal data
- Able to learn properties based on orientation of head
- Point of gaze is not always correct
SLIDE 29
Extension to a Short Video
Apply model per frame of video
SLIDE 30
Extension to a Short Video
Head detector often fails, could use temporal context to improve
SLIDE 31
Conclusions
- Can be confused with mixed orientations and back-facing
- Model generalizes well to animals
- Could be potentially extended to videos
- Could be applied to other domains?
SLIDE 32