Users in Need Computer Science Millions of people with Severe - PDF document

10/17/2014 Camera-based Interfaces for People with Severe Motion Impairments John Magee Margrit Betke Boston University Computer Science Users in Need Computer Science Millions of people with  Severe cerebral palsy  Traumatic brain injury  Stroke  Multiple sclerosis, muscular dystrophy  ALS (Lou Gehrig’s) cannot communicate with traditional means:  Often nonverbal  Limited voluntary motion Lack of communication ability ≠ lack of active minds! Communication technology often not available or too expensive, inefficient, difficult, etc. Intelligent assistive interfaces can greatly improve the lives of people with severe paralysis 1

10/17/2014 Research Team Computer Science Prof. James Gips at Boston College My students at Boston University: O. Al-Hinai, W. Akram, E. Cansizoglu, H. Chennamaneni, M. Chau, R. Cloud, C. Connor, S. Deshpande, S. Epstein, C. Fagiani, P. Fleming, I. Fedyuk, M. Gorman, O. Gusyatin, K. Grauman, W.-B. Kim, C. Kwan, J. Lombardi, J. Magee, W. Mullally, M. Paquette , M. Scott, M. Shugrina, L. Tiberii, M. Urinson, B. Waber, E. Yu Outline Computer Science  Motivation  Camera Mouse demo  Impact -- Camera Mouse users word-wide  How does the Camera Mouse work?  Assistive software  Recent research  Take-home message: Tasks for Future 2

10/17/2014 Traditional Approaches Computer Science Binary Switch  Touch Switches (blue button)  Hit plate  Wobble stick  Grip handle  Pinch  Pull string  Photocell Switches  Sip or Puff Switches  Voice activated Switches Traditional Approaches Computer Science EagleEyes by J. Gips at Boston College Severe paralysis may leave the eyes as the only muscles that a person can control Gaze direction is detected through electro-oculography 3

10/17/2014 Traditional Approaches Computer Science Active Infrared Lighting for Gaze Detection Non-commercial custom- made hardware (e.g., IBM’s Blue Eyes) Relatively expensive commercial hardware (e.g., Applied Science Laboratories)  Calibration procedure?  Long term effect of infrared light on eyes?  Costs? Gestures of MS/CP Patients Computer Science 4

10/17/2014 The Camera Mouse Computer Science The Camera Mouse Computer Science  Camera-based tracking of body User with ALS to enable control of a mouse pointer  Has been commercialized and sold to individuals, schools, and hospitals in the US and Europe 5

10/17/2014 Computer Science Camera Mouse Demo on Eagle Aliens Free download at www.cameramouse.org Camera Mouse Users (2001) Computer Science Age Gen- Condition Continuing der to Use? 2 M CP Y Obtaining a system for home. 3 F CP Y First user with home system. 6 F CP Y Spelled name. Obtaining a home system. 8 M CP Y Spells naughty words and laughs. 11 M CP Y Obtaining a home system. 14 M CP Y Spells words. Obtaining a home system. 15 M CP N Close, but could not control reliably. 19 M CP N Does not have sufficient muscle control. 23 M TBI N Does not have sufficient muscle control. 31 M TBI Y Spelled "TAKE OFF DAD" 37 M CP Y Spelled "MERRY CHRISTMAS" 58 M CP Y Spells, explores internet on home system. 6

10/17/2014 Impact Computer Science  Numerous camera mouse users with cerebral palsy, multiple sclerosis, ALS, traumatic brain injuries  Camera Mouse currently used In Schools in Australia, England, In Hospitals Indonesia, Ireland, In Nursing Homes Turkey, USA, Uzbekistan At Home  26 schools in Northern Ireland obtained the Camera Mouse in 2003  Free download from www.cameramouse.org since April 2007  Now 2,500 downloads per month; 35,000 in 2008 cameramouse.org Traffic Statistics Computer Science November 1, 2007, 3 pm Now about 2,500 down- loads per month 7

10/17/2014 Thank-you email from a user Computer Science From: Mesutulbe [mailto:mesutulbe@gmail.com] Sent: Thursday, July 27, 2006 1:46 AM Subject: I'm grateful to you I'm so grateful to you. Because I'm a MS(Multpl Sclerosis) patient since 20 years. I can't move my finger. However I'm in internet fr 10 hour at this time. Thank you very much for everything. Sincerly Dr.Mesut Thank-you email from a user Computer Science 8

10/17/2014 Thank-you email from a user Computer Science http://www.jacquirogers.id.au Tracked Features Computer Science 9

10/17/2014 Correlation-based Tracking Computer Science Method: Template Matching Computer Science Normalized Correlation Coefficient (NCC):     A s ( x , y ) t ( x , y ) s ( x , y ) t ( x , y )    ( , ) r s t s t            2 2 number of 2 2 A ( , ) ( ( , )) A t ( x , y ) ( t ( x , y )) A s x y s x y t s pixels in template Template origin Search area: Template: 50x50 15x15 possible new pixels origin points 10

10/17/2014 Correlation Surface Computer Science Peak Tracking Methods Computer Science  Correlation (NCC)  Optical flow (Horn-Schunk) I (x+dx,y+dy,t+dt) = I (x,y,t)  Weighted optical flow (Lucas-Kanade) W(x,y) I(x,y,t)  Kalman filtering state s(t|t): Position, Velocity, Acceleration Kalman filter minimizes Bayesian mean square error E[(s true (t)-s(t|t)) 2 ]: Estimate update: s(t|t) = s(t|t-1) + K(t) (x(t)- s(t|t-1)) Prediction s(t|t-1) reduces search space for measurement x(t) 11

10/17/2014 Applications Computer Science Lucas/ Kanade Tracker: Ideal Estimated Feature Location Summary of Tracking Results Computer Science  For slow, smooth motions, NCC with Kalman filtering worked best:  Tracking accuracy  Processing time  For erratic movements, NCC without Kalman filtering worked better (LK tracker performed 2 nd best)  Horizontal motion easier to track than vertical motion  Any time-consuming processing that results in skipping input frames reduces overall tracking accuracy 12

10/17/2014 Evaluation Methods Computer Science 1. Computer Vision: Measure Tracking Accuracy: True feature trajectory versus detected trajectory across different users and different tasks 2. Human-Computer Interaction:  Is the interface accurate enough to be operational?  What is the communication bandwidth? Fitts Law (accuracy/speed tradeoff)  What are the directions of movement most and least comfortable for users with disabilities?  Motivating test software for human subjects Applications Computer Science 13

10/17/2014 Painting a Line Computer Science Subject was asked to draw a straight horizontal line from left to right Painting a Line Computer Science User with Disabilities Nose Lip Eye Horizontal Line Trial 1 9 s Horizontal Line Trial 2 21 s Horizontal Line Trial 3 20 s Horizontal Line Trial 4 4 s Horizontal Line Trial 5 17 s Vertical Line Trial 1 11 s Vertical Line Trial 2 6 s Vertical Line Trial 3 28 s Vertical Line Trial 4 55 s Average: 19 s, Std Deviation: 15.6 s 14

10/17/2014 Two-level Text Entry Computer Science Experience Computer Science Each user was asked to spell 3 words: “RAINING” “MINIMAL” “POOR” User with disabilities: “OPOTOTTR” instead of “POOR.” 15

10/17/2014 Applications Computer Science Movement Evaluation Interface Computer Science 16

10/17/2014 Movement Evaluation Interface Computer Science Some Lessons Learned Computer Science Designing Human-Computer Interface:  Text entry applications need large, strategically placed areas for letters or words to reduce the problem of false selection  “Rest” areas needed (“Midas Touch Problem”)  Trajectories most comfortable for a user with severe cerebral palsy were along diagonal axes Using Human-Computer Interface:  Choice of tracking method should depend on application used  Nose is most reliable feature for non-disabled users; allows fast and smooth motion  No clear winning feature for users with disabilities  Automatic re-initialization needed when features lost 17

10/17/2014 Assistive Software Computer Science  Text Entry  Games  Web mediators  GUI of browser  RefLink: automatic entity extraction  Software for navigating music and videos  Image editor “Camera Canvas”  Animate! Making an anthropomorphic figure to dance Rick Hoydt Speller Computer Science 18

10/17/2014 Applications Computer Science Computer Science 19

10/17/2014 Slime Volleyball Computer Science Animate! Computer Science 20

10/17/2014 Animate! Computer Science Camera Canvas Computer Science  Image editing software package  Works with Camera Mouse as mouse- substitution input system  Sliding menu bar  Selection area 21

10/17/2014 Computer Science Camera Canvas Computer Science 22

10/17/2014 Camera Canvas Computer Science Camera Canvas Computer Science 23

10/17/2014 An Interface that Enables People with Motion Impairments to Analyze Web Content and Dynamically Link to References Navigate to the article Computer Science 24

10/17/2014 Find a Term Computer Science Find a Term Computer Science 25

10/17/2014 Copy and paste on search bar Computer Science Click on Search Computer Science 26

10/17/2014 Click on Result Computer Science Wikipedia Page Computer Science 27