Gestix: A Doctor-Computer Sterile Gesture Interface for Dynamic - - PowerPoint PPT Presentation

Gestix: A Doctor-Computer Sterile Gesture Interface for Dynamic Environments

Juan Wachs, Helman Stern, Yael Edan Ben Gurion University of the Negev, Be’er Sheeva, Israel Michael Gillam, Craig Feied, Mark Smith, Jon Handler Institute for Medical Informatics, Washington Hospital Center Washington DC, US

Wachs et al. 2006 WSC 11 2

Presentation Overview

Introduction and Motivation Gesture interfaces in Medicine “Gestix” - System Overview Detecting and Tracking the Hand Operation Modes Conclusions

Wachs et al. 2006 WSC 11 3

Maintain Sterile Environments

Motivation: Sterile Environments

Study by Clorox Toilets Cleaner Than Keyboards Toilets Cleaner Than Mice

Wachs et al. 2006 WSC 11 4

Hand Gestures in Medical Environment

• Access to critical inform ation in hospitals, in case of

epidem ic crisis eruption, w hile m aintaining total sterilization.

• Surgeon can control the different tools and

visualization devices by sim ply m aking the appropriate gesture.

• Portable gesture recognition system s can be used to

control hom e devices, and w heelchair transportation vehicles for the handicapped population.

• Sterile exploration and m anipulation of high-quality

im ages of the hum an anatom y, produced by 3 D scanning techniques ( MRI , CT) .

Wachs et al. 2006 WSC 11 5

A robotic wheelchair based on the integration of human and environmental observations [Kuno et. al 2003]

Gesture Interfaces in Medicine

Wachs et al. 2006 WSC 11 6

Interactive Manipulation of Real time Visualization from Medical Volume Data by using 2-Handed VR-Techniques [Köchy et al. 98]

Gesture Interfaces in Medicine

Wachs et al. 2006 WSC 11 7

The Gesture Pendant: A Self-illuminating, Wearable, Infrared Computer Vision System for Home Automation Control and Medical Monitoring [Starner et. al 2000]

Gesture Interfaces in Medicine

Wachs et al. 2006 WSC 11 8

Staying Alive: A Virtual Reality Visualization Tool for Cancer Patients [Becker and Pentland 96]

Gesture Interfaces in Medicine

Wachs et al. 2006 WSC 11 9

Facilitating User Interaction with Complex Systems via Hand Gesture Recognition [New et. al 2003]

Gesture Interfaces in Medicine

Wachs et al. 2006 WSC 11 10

FAce MOUSe: A Novel Human–Machine Interface for Controlling the Position of a Laparoscope [Nishikawa et. al 2003]

Gesture Interfaces in Medicine

Wachs et al. 2006 WSC 11 11

FAce MOUSe: A Novel Human–Machine Interface for Controlling the Position of a Laparoscope (Cont’d)

Gesture Interfaces in Medicine

Wachs et al. 2006 WSC 11 12

Summary: Why Hand Gestures?

• Sterile
• Rapid Reaction
• Unencum bered
• Navigational Richness
• Noisy Environm ents
• Support of m ultim odal inputs
• Natural com m unication

Wachs et al. 2006 WSC 11 13

GESTIX: System Overview

Screen Navigation Map Hand Gesture Control

Wachs et al. 2006 WSC 11 14

System Overview - Architecture (Cont’d)

State machine for the gesture-based medical browser Start Calibration Mode Detector Stop

Flick Gestures Left Right Up Down Sweep Gesture Closer Further Track SSI Rotate

Browse Zoom Rotate

Wachs et al. 2006 WSC 11 15

Hand Detection

Calibration

The background is updated using ‘Background Differencing’

) , ( ) , ( ) 1 ( ) , (

1

j i f j i Bcc j i Bcc

k k

∗ + ∗ − =

−

α α

When a significant change occurs, the background is not updated anymore, and the CDM is created

Wachs et al. 2006 WSC 11 16

Camshift Algorithm to track skin color

Building the color model

Hue Sat Sat Hue

Wachs et al. 2006 WSC 11 17

Color Cue (The Back Projection Image)

Wachs et al. 2006 WSC 11 18

Motion Cue (Frame Differencing)

Wachs et al. 2006 WSC 11 19

Fusion of Color and Motion Cues

) , ( * ) 1 ( ) , ( } ). , ( , 1 min{ ) , ( j i p j i d j i p j i I

k k k k k k

α φ α − + ∗ =

) , ( j i Ik

k

α

) , ( j i pk d ) , ( j i

k

φ

Skin color probability (color cue) at frame k Motion image (motion cue) at frame k Amplifying factor Motion assessment variable Fused intensity image

Wachs et al. 2006 WSC 11 20

Operation Modes - Browsing

When a doctor wishes to browse the image database, he/she moves the hand rapidly out from a ‘neutral area’ to any of the 4 directions, and then back to the neutral area, hence evoking a ‘flick’ gesture.

Wachs et al. 2006 WSC 11 21

Switching to Zoom Mode

f1 f2 f3 f4 ) f , f (

j i

α

) f , f (

j i

α

ε α α ε ∗ + ≤ − ≤ ∗ − 2 90 2 90

) f , f ( ) f , f (

2 1 4 3

ε α ε α ∗ ≤ ∨ ∗ ≤ 5 . 1 ) ( std 5 . 1 ) ( std

) f , f ( ) f , f (

2 1 4 3

(3) (4)

90 180

Frame number Average angle Instant angle

Hand Orientation – Major Axis

Wachs et al. 2006 WSC 11 22

Operation Modes – Zoom Mode

Wachs et al. 2006 WSC 11 23

Operation Modes - Rotation

Straight lines are detected using the Probabilistic Hough Transform and the straight sterile instrument is detected by Weighted Rule Based Algorithm

*

Wachs et al. 2006 WSC 11 24

Operation Modes – Rotation (Cont’d)

Wachs et al. 2006 WSC 11 25

Conclusions

‘Flicking’ gestures for interface control. Rotation of images are controlled by a

straight sterile instrument type of objects.

Dynamic gestures are detected by a color-

motion fusion approach

The gesture recognition system was

implemented in a sterile medical data- browser.

Wachs et al. 2006 WSC 11 26

Thanks!

juan@bgu.ac.il http://www.bgu.ac.il/~juan