Gestix: A Doctor-Computer Sterile Gesture Interface for Dynamic - PowerPoint PPT Presentation

Gestix: A Doctor-Computer Sterile Gesture Interface for Dynamic Environments Juan Wachs, Helman Stern, Yael Edan Ben Gurion University of the Negev, Be’er Sheeva, Israel Michael Gillam, Craig Feied, Mark Smith, Jon Handler Institute for Medical Informatics, Washington Hospital Center Washington DC, US

Presentation Overview � Introduction and Motivation � Gesture interfaces in Medicine � “Gestix” - System Overview � Detecting and Tracking the Hand � Operation Modes � Conclusions Wachs et al. 2006 WSC 11 2

Motivation: Sterile Environments Toilets Cleaner Toilets Cleaner Than Mice Than Keyboards Maintain Sterile Study by Clorox Environments Wachs et al. 2006 WSC 11 3

Hand Gestures in Medical Environment • Access to critical inform ation in hospitals, in case of epidem ic crisis eruption, w hile m aintaining total sterilization. • Surgeon can control the different tools and visualization devices by sim ply m aking the appropriate gesture. • Portable gesture recognition system s can be used to control hom e devices, and w heelchair transportation vehicles for the handicapped population. • Sterile exploration and m anipulation of high-quality im ages of the hum an anatom y, produced by 3 D scanning techniques ( MRI , CT) . Wachs et al. 2006 WSC 11 4

Gesture Interfaces in Medicine A robotic wheelchair based on the integration of human and environmental observations [Kuno et. al 2003] Wachs et al. 2006 WSC 11 5

Gesture Interfaces in Medicine Interactive Manipulation of Real time Visualization from Medical Volume Data by using 2-Handed VR-Techniques [Köchy et al. 98] Wachs et al. 2006 WSC 11 6

Gesture Interfaces in Medicine The Gesture Pendant: A Self-illuminating, Wearable, Infrared Computer Vision System for Home Automation Control and Medical Monitoring [Starner et. al 2000] Wachs et al. 2006 WSC 11 7

Gesture Interfaces in Medicine Staying Alive: A Virtual Reality Visualization Tool for Cancer Patients [Becker and Pentland 96] Wachs et al. 2006 WSC 11 8

Gesture Interfaces in Medicine Facilitating User Interaction with Complex Systems via Hand Gesture Recognition [New et. al 2003] Wachs et al. 2006 WSC 11 9

Gesture Interfaces in Medicine FAce MOUSe: A Novel Human–Machine Interface for Controlling the Position of a Laparoscope [Nishikawa et. al 2003] Wachs et al. 2006 WSC 11 10

Gesture Interfaces in Medicine FAce MOUSe: A Novel Human–Machine Interface for Controlling the Position of a Laparoscope (Cont’d) Wachs et al. 2006 WSC 11 11

Summary: Why Hand Gestures? • Sterile • Rapid Reaction • Unencum bered • Navigational Richness • Noisy Environm ents • Support of m ultim odal inputs • Natural com m unication Wachs et al. 2006 WSC 11 12

GESTIX: System Overview Hand Gesture Control Screen Navigation Map Wachs et al. 2006 WSC 11 13

System Overview - Architecture (Cont’d) State machine for the gesture-based medical browser Start Stop Calibration Mode Detector Browse Rotate Zoom Flick Sweep Track SSI Gestures Gesture Left Right Up Down Closer Further Rotate Wachs et al. 2006 WSC 11 14

Hand Detection Calibration The background is updated using ‘Background Differencing’ = − α ∗ + α ∗ Bcc ( i , j ) ( 1 ) Bcc ( i , j ) f ( i , j ) − k k 1 When a significant change occurs, the background is not updated anymore, and the CDM is created Wachs et al. 2006 WSC 11 15

Camshift Algorithm to track skin color Building the color model Sat Hue Sat Wachs et al. 2006 WSC 11 16 Hue

Color Cue (The Back Projection Image) Wachs et al. 2006 WSC 11 17

Motion Cue (Frame Differencing) Wachs et al. 2006 WSC 11 18

Fusion of Color and Motion Cues = α ∗ φ + − α I ( i , j ) min{ 1 , p ( i , j ). d } ( i , j ) ( 1 ) * p ( i , j ) k k k k k k I k ( i , j ) Fused intensity image p k ( i , j ) Skin color probability (color cue) at frame k φ ( i , j ) Motion image (motion cue) at frame k k d Amplifying factor α Motion assessment variable k Wachs et al. 2006 WSC 11 19

Operation Modes - Browsing When a doctor wishes to browse the image database, he/she moves the hand rapidly out from a ‘neutral area’ to any of the 4 directions, and then back to the neutral area, hence evoking a ‘flick’ gesture. Wachs et al. 2006 WSC 11 20

Switching to Zoom Mode Hand Orientation – Major Axis f1 f2 f3 f4 180 α ( f , f ) α i j ( f , f ) i j 90 − ∗ ε ≤ α − α ≤ + ∗ ε 90 2 90 2 (3) Frame number ( f , f ) ( f , f ) 3 4 1 2 α ≤ ∗ ε ∨ α ≤ ∗ ε std ( ) 1 . 5 std ( ) 1 . 5 (4) Average angle ( f , f ) ( f , f ) 3 4 1 2 Instant angle Wachs et al. 2006 WSC 11 21

Operation Modes – Zoom Mode Wachs et al. 2006 WSC 11 22

Operation Modes - Rotation Straight lines are detected using the Probabilistic Hough Transform and the straight sterile instrument is detected by Weighted Rule Based Algorithm * Wachs et al. 2006 WSC 11 23

Operation Modes – Rotation (Cont’d) Wachs et al. 2006 WSC 11 24

Conclusions � ‘Flicking’ gestures for interface control. � Rotation of images are controlled by a straight sterile instrument type of objects. � Dynamic gestures are detected by a color- motion fusion approach � The gesture recognition system was implemented in a sterile medical data- browser. Wachs et al. 2006 WSC 11 25

Thanks! juan@bgu.ac.il http://www.bgu.ac.il/~juan Wachs et al. 2006 WSC 11 26