1 7/20/2014 3D sensing: Kinect TM Kinect IR projection CMU Summer - PDF document

7/20/2014 Virtual Fixtures in Robotic III. Intelligent Augmentation Surgery ● Hawkeye King, Sina Nia Kosari, Fredrik Ryden, Howard Chizeck ● Goals ● Sensor-based real-time guidance ● Shared human-machine control ● Improved safety CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Virtual Fixture Fixtures: Ruler Example Task: Draw a straight line CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab Surgical Virtual Fixture: guidance: Target Vital Structure Remote Remote operator operator CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab 1

7/20/2014 3D sensing: Kinect TM Kinect IR projection CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington wikipedia.org Bio Robotics Lab Bio Robotics Lab Medical Virtual Medical Virtual Sensing based virtual Sensing based virtual fixture fixture Fixture: Fixture: “Hole” in VF Target Target Vital Structure Vital Structure Remote Remote operator operator VF based VF based On organ shape On organ shape CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington And Motion Bio Robotics Lab Bio Robotics Lab Laser Resection Laser Tumor Resection Demo: Simulated Tumor Laser VF Tip Real Time VF based On organ shape (butcher shop liver) Blake Hannaford, University of Washington CMU Summer School 7.2014 CMU Summer School 7.2014 Blake Hannaford, University of Washington Google 13-Sept-2013 Blake Hannaford, University of Washington Bio Robotics Lab 2

7/20/2014 Haptic Rendering using Kinect • Haptic Rendering based data from non-contact devices (Kinect) Haptics via Kinect • Potentially useful in many tele-operation tasks. CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab Comparison with a Traditional Haptic Algorithm The Main Challenges We need an algorithm for moving the proxy in the point cloud environment. The proxy should be constrained by the points as if the points represented surfaces. We must be able to interact with time varying point clouds. CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab New Definition of Proxy New Definition of Proxy Our proxy consists of Our proxy consists of three regions to detect three regions to detect states of free motion , states of free motion , contact and contact and entrenchment. entrenchment. CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab 3

7/20/2014 New Definition of Proxy New Definition of Proxy Our proxy consists of Our proxy consists of three regions to detect three regions to detect states of free motion , states of free motion , contact and contact and entrenchment . entrenchment. CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab Proxy in Free Motion Proxy in Free Motion CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab Proxy in Free Motion Proxy in Free Motion CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab 4

7/20/2014 Proxy in Free Motion Proxy in Contact CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab Proxy in Contact Proxy in Contact CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab Proxy in Contact Proxy in Contact CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab 5

7/20/2014 Proxy in Contact Proxy in Contact CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab Proxy in Contact Proxy in Contact CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab Proxy in Contact Proxy in Contact CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab 6

7/20/2014 Proxy in Contact Entrenched Proxy CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab Entrenched Proxy Entrenched Proxy CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab Blake Hannaford, University of Washington Blake Hannaford, University of Washington CMU Summer School 7.2014 CMU Summer School 7.2014 7

7/20/2014 CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Blake Hannaford, University of Washington Blake Hannaford, University of Washington CMU Summer School 7.2014 CMU Summer School 7.2014 8

7/20/2014 CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington FLS ● Basic curriculum from Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) ● http://www.flsprogram.org Advanced Virtual Fixtures ● Block Transfer: ● RPI Video of Block transfer Task CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab • “Virtual Fixtures” Putting Green VF – Tactile Perceptual Overlays • “Mixed Autonomy” – Automation of sub-tasks CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab 9

7/20/2014 CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab Bio Robotics Lab Ender's Game CMU Summer School 7.2014 Blake Hannaford, University of Washington CMU Summer School 7.2014 Blake Hannaford, University of Washington Bio Robotics Lab 10