the experiments for
play

The experiments for You -Do, I- Learn Presenter: Wenguang Mao - PowerPoint PPT Presentation

Oct 18, 2023 •180 likes •597 views

The experiments for You -Do, I- Learn Presenter: Wenguang Mao Instructor: Kristen Grauman Author for the paper: Dima Damen Recap of the Paper Gaze attention Gaze point Clustering position Clustering TRO MOI Gaze area appearance

  1. The experiments for “You -Do, I- Learn” Presenter: Wenguang Mao Instructor: Kristen Grauman Author for the paper: Dima Damen

  2. Recap of the Paper Gaze attention Gaze point Clustering position Clustering TRO MOI Gaze area appearance Neighbor frames

  3. Experiment Setup • Dataset: Bristol Egocentric Object Interactions Dataset

  4. Experiment Setup • Dataset: Bristol Egocentric Object Interactions Dataset • Egocentric videos at 6 locations • Gaze point on each frame • Gaze positions in 3D space • Gaze fixation on each frame • Ground truth positions of TROs • 3D map for each location, 3D positions of the camera for each frame, …… • Code: VLFeat, Matlab toolboxes, and programs written by myself

  5. Why Need Gaze Info • Given an egocentric image, which part of the image do you think I am focusing on? • Center of image? • Blue point: center of image • Red point: gaze point

  6. Why Need Gaze Info • The distance between the center and the gaze point (a) Desk (b) Door

  7. Why Need Gaze Info • The distance between the center and the gaze point Center of image is not good approximation for the gaze point (a) Desk (b) Door

  8. Why Need Gaze Info • The distance between the center and the gaze point ( during gaze fixation ) (a) Desk (b) Door

  9. Why Need Gaze Info • The distance between the center and the gaze point ( during gaze fixation ) Center of image is not good approximation for the gaze point Even during attention period (a) Desk (b) Door

  10. How Gaze Fixation Helps • Do you think there is any TRO in the video clips • Red dot: gaze point

  11. How Gaze Fixation Helps • Do you think there is any TRO in the video clips • Red dot: gaze point Gaze fixation helps identify a TRO

  12. How Gaze Fixation Helps • Do you think there is any TRO in the video clips • Red dot: gaze point

  13. How Gaze Fixation Helps • Do you think there is any TRO in the video clips • Red dot: gaze point Gaze fixation alone is far from enough to find TROs

  14. How 3D Positions of Gaze Help • Blue circles: 3D positions of gazes in a video • Red cross: ground truth positions of TRO (a) Without gaze fixation filtering (a) With gaze fixation filtering

  15. How 3D Positions of Gaze Help • Blue circles: 3D positions of gazes in a video • Red cross: ground truth positions of TRO 3D gaze positions are very helpful to identify TROs (a) Without gaze fixation filtering (a) With gaze fixation filtering

  16. Clustering for Gaze 3D positions • Right number of clusters (kmeans) • Yellow square: cluster center (a) Without gaze fixation filtering (b) With gaze fixation filtering

  17. Clustering for Gaze 3D positions • Right number of clusters (kmeans) • Yellow square: cluster center With the knowledge of right number of TROs, they can be easily identified using 3D gaze positions (a) Without gaze fixation filtering (b) With gaze fixation filtering

  18. Clustering for Gaze 3D positions • Too less clusters • Yellow square: cluster center (a) Without gaze fixation filtering (b) With gaze fixation filtering

  19. Clustering for Gaze 3D positions • Too less clusters • Yellow square: cluster center If underestimating the number, low precision and low recall for identifying TROs (a) Without gaze fixation filtering (b) With gaze fixation filtering

  20. Clustering for Gaze 3D positions • Too much clusters • Yellow square: cluster center (a) Without gaze fixation filtering (b) With gaze fixation filtering

  21. Clustering for Gaze 3D positions • Too much clusters • Yellow square: cluster center If overestimating the number, high recall and low precision (a) Without gaze fixation filtering (b) With gaze fixation filtering

  22. Spectral Clustering • Right number of clusters (a) kmeans (b) spectral

  23. Spectral Clustering • Right number of clusters Same with K-means (a) kmeans (b) spectral

  24. Spectral Clustering • Too less clusters (a) kmeans (b) spectral

  25. Spectral Clustering • Too less clusters Same with k-means (a) kmeans (b) spectral

  26. Spectral Clustering • Too much clusters (a) kmeans (b) spectral

  27. Spectral Clustering • Too much clusters Outperform k-means, high precision and high recall. (a) kmeans (b) spectral

  28. What is the Limitation of Gaze Positions • Can we only use 3D gaze positions? • No, because of moving TRO • How to solve this problem? • Appearance

  29. Appearance • How HoG features represent an image

  30. Appearance • How HoG features represent an image HoG is good to describe the boundary

  31. Identify TROs based on Appearance • Extract HoG from the region near the gaze point for each frame • Generate BoW representation for each frame • Perform clustering on frames • Use the frame closest to the center to represent each cluster • Compare the appearance of center frames with the ground truth

  32. Appearance • Five TROs around the desk tape socket screwdriver charger box

  33. Results Success (box) Success (tape) Duplicated (box) Success (charger) Failure

  34. Results Success (box) Success (tape) Duplicated (box) Missing two TROs, the appearance is not as effective as the position Success (charger) Failure

  35. Using Neighbor frames Failure Success (charger) Success (box) Success (driver) Success (tape)

  36. Using Neighbor frames Failure Success (charger) Success (box) Missing one TRO, using neighbor frames is helpful to improve performance Success (driver) Success (tape)

  37. Over-Estimating No. of Clusters Failure Success (box) Success (driver) Success (charger) Success (tape) Duplicated (box) Duplicated (box) Duplicated (driver)

  38. Over-Estimating No. of Clusters Failure Success (box) Success (driver) Success (charger) Missing one TROs, over-estimating is helpful to identify more TROs Success (tape) Duplicated (box) Duplicated (box) Duplicated (driver)

  39. Also Using Neighbor frames Failure Success (tape) Success (socket) Duplicated (socket) Success (driver) Success (box) Success (charger) Duplicated (box)

  40. Also Using Neighbor frames Failure Success (tape) Success (socket) Duplicated (socket) Finding all TROs Success (driver) Success (box) Success (charger) Duplicated (box)

  41. Conclusion • Gaze information is important and necessary for egocentric videos, and the center of image is not a good approximation • Gaze fixation is helpful for identifying TROs, but itself is not enough • 3D positions of gaze give rich information for TROs, but clustering method and the estimation on the number of TROs is critical • Use spectral clustering and do not worry about overestimating • Appearance is another important feature for identifying TROs • Using neighbor frames is beneficial to improve performance • Over-estimating No. of TROs is helpful to reduce false negative

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Experiments Philosophy of Economics University of Virginia Matthias Brinkmann Contents 1.

Experiments Philosophy of Economics University of Virginia Matthias Brinkmann Contents 1.

Experiments Philosophy of Economics University of Virginia Matthias Brinkmann Contents 1. Public Goods Experiments 2. Types of Experiments 3. The Role of Experiments 29/10/2018 Experiments 2 A Standard Finding (Guala 2005, 22) Two

460 views • 14 slides
Hagner experiments status after 22 years Per-Erik Wikberg SLU Ume Hagner experiments

Hagner experiments status after 22 years Per-Erik Wikberg SLU Ume Hagner experiments

Hagner experiments status after 22 years Per-Erik Wikberg SLU Ume Hagner experiments Liberich liberation thinning, enrichment planting (in Swedish naturkultur) Principle: optimizing the net present value at each site of the

601 views • 34 slides
Collider Experiments and India Sunanda Banerjee January, 2019 Experiments in High Energy Physics

Collider Experiments and India Sunanda Banerjee January, 2019 Experiments in High Energy Physics

Collider Experiments and India Sunanda Banerjee January, 2019 Experiments in High Energy Physics Particle physics experiments in the world started with No accelerator Visual detection technique In absence of accelerators what was

472 views • 46 slides
Experiments designed to help the participants Maximilian Kasy 7/10/2019 Why experiments? Are

Experiments designed to help the participants Maximilian Kasy 7/10/2019 Why experiments? Are

Experiments designed to help the participants Maximilian Kasy 7/10/2019 Why experiments? Are your programs effective in helping refugees? How to find out? Possibility 1: Compare the outcomes of those who got the programs to others who

330 views • 13 slides
Planned OBT Experiments at CRL Planned OBT Experiments at CRL September 14, 2011 Sang Bog Kim

Planned OBT Experiments at CRL Planned OBT Experiments at CRL September 14, 2011 Sang Bog Kim

Planned OBT Experiments at CRL Planned OBT Experiments at CRL September 14, 2011 Sang Bog Kim and Vladimir Korolevych Research Scientist Environmental Technologies Branch Chalk River Laboratories Chalk River, Ontario Canada 1 UNRESTRICTED

594 views • 17 slides
Key Experiments Key Experiments and Simulation: and Simulation: Status Report Status Report

Key Experiments Key Experiments and Simulation: and Simulation: Status Report Status Report

Key Experiments Key Experiments and Simulation: and Simulation: Status Report Status Report E.Farnea INFN Sezione di Padova, Italy AGATA Week, Strasbourg, November 24 th , 2005 Outline Outline Performance of the Demonstrator Some

317 views • 19 slides
WISP searches by Tokyo tabletop experiments group UTokyo tabletop experiments group Toshio

WISP searches by Tokyo tabletop experiments group UTokyo tabletop experiments group Toshio

WISP searches by Tokyo tabletop experiments group UTokyo tabletop experiments group Toshio NAMBA 0 UT tabletop experiments group (only related to todays talk) UTokyo Physics & ICEPP : S. Asai, T. Inada, T. Yamaji, T. Yamazaki(KEK),

624 views • 49 slides
Lecture 5/Chapter 5 Experiments Variables Roles Outside Variables Single- or

Lecture 5/Chapter 5 Experiments Variables Roles Outside Variables Single- or

Lecture 5/Chapter 5 Experiments Variables Roles Outside Variables Single- or Double-Blind Experiments Flaws and Remedies in Experiments Roles of variables Most statistical studies of relationships attempt to establish evidence

763 views • 32 slides
Designs Chapter 11 Quasi-Experimentation Quasi-experiments resemble experiments, but lack

Designs Chapter 11 Quasi-Experimentation Quasi-experiments resemble experiments, but lack

Quasi-Experimental Designs Chapter 11 Quasi-Experimentation Quasi-experiments resemble experiments, but lack experimental control lack of random assignment is the key point of distinction between quasi- experiments and true

888 views • 31 slides
Experimental Design and the Search for Quasi-Experiments Department of Government London School

Experimental Design and the Search for Quasi-Experiments Department of Government London School

Review Randomized Experiments Quasi-Experiments Experimental Design and the Search for Quasi-Experiments Department of Government London School of Economics and Political Science Review Randomized Experiments Quasi-Experiments 1 A Review

823 views • 58 slides
Survey Experiments April 8 1 / 16 Outline Guest presentation Overview of survey experiments

Survey Experiments April 8 1 / 16 Outline Guest presentation Overview of survey experiments

Survey Experiments April 8 1 / 16 Outline Guest presentation Overview of survey experiments Discussion of readings Synopsis presentations 2 / 16 Guest Presentation Rune Slothuus, Professor in IFSK Discussing survey experiment by Druckman,

496 views • 16 slides
Remote Participation in in Remote Participation physical experiments physical experiments

Remote Participation in in Remote Participation physical experiments physical experiments

Remote Participation in in Remote Participation physical experiments physical experiments Piroska Giese KFKI RMKI e-mail : giese@rmki.kfki.hu Yahoo ID : kfki_Piroska_Giese April 18-19 2002, Cluj-Napoca RoEduNet Conference Slide 1 Overview

543 views • 40 slides
Social Experiments Handbook of Econometrics James J. Heckman and Edward J. Vytlacil The

Social Experiments Handbook of Econometrics James J. Heckman and Edward J. Vytlacil The

Social Experiments Handbook of Econometrics James J. Heckman and Edward J. Vytlacil The University of Chicago and Stanford University Handout Draft, July 28, 2004 1 1 Social Experiments 1.1 Introduction Consider ideal experiments with no

1.06k views • 79 slides
Non-LHC experiments in HEPDATA Matthew Wing (UCL) Initial comments. Data / experiments

Non-LHC experiments in HEPDATA Matthew Wing (UCL) Initial comments. Data / experiments

Non-LHC experiments in HEPDATA Matthew Wing (UCL) Initial comments. Data / experiments to be included ? More exotic data. British Museum HEPDATA Advisory Board Meeting 16 October 2015 Initial comments It would be

465 views • 5 slides
ACMS 20340 Statistics for Life Sciences Chapter 8: Designing Experiments Fishers Experiments

ACMS 20340 Statistics for Life Sciences Chapter 8: Designing Experiments Fishers Experiments

ACMS 20340 Statistics for Life Sciences Chapter 8: Designing Experiments Fishers Experiments Experiment Terminology individuals (if human) subjects explanatory variable factors In conducting an experiment, we try to keep as

340 views • 31 slides
THE ELECTRIC DISCHARGE EXPERIMENTS A Theorists Exploration of First Laboratory

THE ELECTRIC DISCHARGE EXPERIMENTS A Theorists Exploration of First Laboratory

THE ELECTRIC DISCHARGE EXPERIMENTS A Theorists Exploration of First Laboratory Plasmas I. INTRODUCTION Who? Experimentalists interested in electrical properties of matter What? Experiments with electromagnetic

160 views • 14 slides
On Improving the Efficiency and Robustness of Table Storage Mechanisms for Tabled Evaluation

On Improving the Efficiency and Robustness of Table Storage Mechanisms for Tabled Evaluation

On Improving the Efficiency and Robustness of Table Storage Mechanisms for Tabled Evaluation Ricardo Rocha DCC-FC & LIACC University of Porto, Portugal ricroc@ncc.up.pt PADL 2007, Nice, France, January 2007 On Improving the Efficiency

821 views • 29 slides
Acknowledgements Krzysztof Gajos Corin Anderson Mary Czerwinski Pedro Domingos

Acknowledgements Krzysztof Gajos Corin Anderson Mary Czerwinski Pedro Domingos

Gu Guid idelines for or In Intellig igent t In Interfaces Daniel Weld University of Washington Acknowledgements Krzysztof Gajos Corin Anderson Mary Czerwinski Pedro Domingos Oren Etzioni Raphael Hoffman Tessa

498 views • 8 slides
Matthew Series Lesson #191 March 11, 2018 Dean Bible Ministries www.deanbibleministries.org Dr.

Matthew Series Lesson #191 March 11, 2018 Dean Bible Ministries www.deanbibleministries.org Dr.

Matthew Series Lesson #191 March 11, 2018 Dean Bible Ministries www.deanbibleministries.org Dr. Robert L. Dean, Jr. The Accomplishments of Christs Death: Redemption 1 Pet 1:1819 Interlude: Accomplishments of Messiahs Death 1.

501 views • 32 slides
Five Star Quality Rating System Design For Nursing Home Compare Nathan Shaw RN, BSN, MBA, LHRM,

Five Star Quality Rating System Design For Nursing Home Compare Nathan Shaw RN, BSN, MBA, LHRM,

Understanding the Five Star Quality Rating System Design For Nursing Home Compare Nathan Shaw RN, BSN, MBA, LHRM, RAC CT 3.0 Director of Clinical Reimbursement March 23rd, 2015 Objectives Objectives Provide a background of the five-star

863 views • 59 slides
Photon Detection System (PDS) and SN triggering Pierre Lasorak 1 Introduction Outline

Photon Detection System (PDS) and SN triggering Pierre Lasorak 1 Introduction Outline

Photon Detection System (PDS) and SN triggering Pierre Lasorak 1 Introduction Outline Introduction Final aim: What to expect from the PDS for SN triggering? Can we motivate from SN/DAQ point of view? Addition of reflective

608 views • 11 slides
Coat of arms Knights couldnt tell who was on their side and who the enemies were so they

Coat of arms Knights couldnt tell who was on their side and who the enemies were so they

Coat of arms Knights couldnt tell who was on their side and who the enemies were so they painted patterns on their shields. It became fashionable to have a Coat of Arms. People hired artists to design them. Some famous coat of

315 views • 7 slides
Some thoughts on safety of machine learning Fabio Roli University of Cagliari, Italy HUML

Some thoughts on safety of machine learning Fabio Roli University of Cagliari, Italy HUML

Pattern Recognition and Applications Lab Some thoughts on safety of machine learning Fabio Roli University of Cagliari, Italy HUML 2016, Venice, December 16th, 2016 Department of Electrical and Electronic Engineering The black cloud

629 views • 30 slides
Convergence and error estimates for the compressible Navier-Stokes equations Antonin Novotny ( 1 )

Convergence and error estimates for the compressible Navier-Stokes equations Antonin Novotny ( 1 )

Convergence and error estimates for the compressible Navier-Stokes equations Antonin Novotny ( 1 ) , IMATH ( 2 ) , Universit e du Sud Toulon Var ( 1 ) http://myweb.labscinet.com/novotny ( 2 ) http://imath.univ-tln.fr 13.09-18.09, 2015 Part 1 :

395 views • 27 slides

More recommend