in AR-ready Environm nments ts Gudrun Klinker Fachgebiet Augmented - - PowerPoint PPT Presentation

Ubiqu quitou tous Augmented Reality in AR-ready Environm nments ts

Gudrun Klinker Fachgebiet Augmented Reality, Technische Universität München, Germany May 30, 2011

Many of the videos shown in this talk can be found in our YouTube Channel Augmen ented tedReal Reality ityTUM

Augmented Reali lity y (AR)

Provides users with computer information within their real environment ” Three-dimensional ” Immersive ” Interactive ” In real-time ” Mobile

[ECRC 96] [TU München, ForLog]

Vision

“ We are surrounded by information.

” Information is real, but we cannot always memorize all of it and/or notice it with our limited senses. ” Information has many dimensions. ” Information can relate to many aspects of an object. ” Information has many kinds of reference points. ” Different people perceive information differently. ” Information is not static ” we need and want to interact with it.

“ Computers can help us analyze, explore and understand information. “ Critical technologies:

3

3D 2D stationary mobile AR VR

Wearable, Ubicomp

Mobile AR ” integrated in stationary, wide-area, multi-media environments (ubiquitous / pervasive / ambient computing) “ Ubiquitous Tracking

” Mobile: head-mounted camera, gyroscope, … ” Stationary: room-based cameras, RFIDs, GPS, …

“ Ubiquitous Information Presentation

” Mobile: head-mounted displays, portable displays, displays attached to mobile tools, head-mounted projectors… ” Stationary: wall-sized displays, desktops, projectors, …

“ Ubiquitous Interaction

” Mobile: tracked persons/objects (tangibles), wearable devices (buttons), gestures ” Stationary: context (spatial cells, time): ‚modes‛, devices at fixed positions

Requires flexible, standardizing system architecture (middleware)

4

Ubiquit itous s Augmented d Realit ity

Our Approach (techn hnology-dri driven)

5

Ubiquit itous s Augmented d Realit ity

VR versus us AR: the human side

3D 2D stationary mobile AR VR

Wearable, Ubicomp

VR VR AR AR

Properties

Engineered (safe) physical reality Cluttered (dangerous) physical reality

Goal

Immerse in a secondary reality Add secondary information to the primary (physical) reality

Problems

Lack of presence

• Degree of realism
• Simulator sickness

Lack of situation awareness

• Perceptual tunnelling, information
• verload, cognitive capture.
• Tracking, occlusion of physical objects

Approach

Exploit human sensing limitations

• e.g. Change blindness

Analyze potential distractions

• e.g. Eye tracking

Convergence

Use AR to enhance VR Use VR to evaluate AR

Overvi rview w of this s talk

“ Ubiquitous Information Presentation Schemes “ Usability in Real Life (Dangerous Situations) “ Ubiquitous Tracking

Ubiquit itous s Informatio ion Prese sentatio ion Schemes

• Various

s Approac aches s -

Head-based Desktop-based Hand-based Hybrid On real surfaces

Youtube Channel: AutmentedRealityTUM

Split Prese sentatio ion Schemes

• Head-Mo

Mounted Laser ser Projec ector + Monitor -

Industrial application: quality assurance “ Information ‚what to do‛ “ Information ‚where to do it‛

Schwerdtfeger and Klinker: Hybrid Information Presentation: Combining a Portable Augmented Reality Laser Projector with a Conventional Computer Display, (short paper) IPT-EGVE Symposium, July 2007

In ergonomics: compensation vs. pursuit Six parameters, 26 = 64 variations

9

Ubiquit itous s Informatio ion Prese sentatio ion Schemes Ego-ce centric ic vs. Exoce centric ic Presen sentati ation

X X X X x … … x Y X X x x … … x Z X X x x … … x Rx x … x Ry … x Rz … x

Overvi rview w of this s talk

“ Ubiquitous Information Presentation Schemes “ Usability in Real Life “ Ubiquitous Tracking

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Car Driving

ing (Senso sor Visualizat alizatio ion) ) -

Situation: “ Increasing number of sensors in the car “ What can be recognized? How reliably? “ How can drivers understand the automatic data interpretation process in a car ” and the resulting autonomous actions?

Walchshäusl, Lindl, and Tatschke: Detection of Road Users in Fused Sensor Data Streams for Collision Mitigation. In Proceedings of 10th International Forum of Advanced Microsystems and Automobile Applications, Berlin, Apr. 25-27, 2006.

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Car Driving

ing (Senso sor Visualizat alizatio ion) ) -

Goals “ Support for sensor research (debugging using live visualizations) “ Support of MMI research (appreciation of live sensor data) Augmented Reality for Co-Drivers and Service Technicians “ Head-mounted Display “ Portable TFT Display “ Multi-touch Display in the lab

Tönnis, Lindl, Walchshäusl, and Klinker: Visualization of Spatial Sensor Data in the Context of Automotive Environment Perception Systems. ISMAR 07, Nov. 2007.

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Car Driving

ing (Danger er Ahead) ad) -

Goal: “ Avoidance of rear-end accidents (e.g.: using ACC) Approach: “ Visualization of the breaking path in the HUD as ” Bar [Bubb 76] ” Drive path

Tönnis, Lange and Klinker: Visual Longitudinal and Lateral Driving Assistance in the Head-Up Display of Cars, ISMAR 07, Nov 2007. Tönnis, Lange, Klinker , Bubb: Transfer von Flugschlauchanzeigen in das Head-Up Display von Kraftfahrzeugen. VDI/VW Tagung “Integri.Sicherheit und Fahrerassistenzsysteme, Wolfsburg, Okt.’06.

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Car Driving

ing (Danger er Ahead) ad) -

Evaluations in a Fixed-Base Driving Simulator: “ Test participants drive faster with a drive-path assistant. “ The drive-path assistant generates higher speed oscillations. “ Steering is improved. “ Test participants preferred the bar. “ Drive path reduced concentration on the driving task.

Tönnis, Lange and Klinker: Visual Longitudinal and Lateral Driving Assistance in the Head-Up Display of Cars, ISMAR 07, Nov 2007. Tönnis, Lange, Klinker , Bubb: Transfer von Flugschlauchanzeigen in das Head-Up Display von Kraftfahrzeugen. VDI/VW Tagung “Integri.Sicherheit und Fahrerassistenzsysteme, Wolfsburg, Okt.’06.

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Car Driving

ing (Danger er Ahead) ad) -

Important issues “ Information overload Too much information, too rapid changes “ Perceptual tunneling Focusing on a single stimulus “ Cognitive capture Absentmindedness, reduced situational awareness “ Occlusion of real objects

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Car Driving

ing (Danger er from Behind) ) -

Goal: “ Guide a user’s attention to danger from behind Approach: “ Visualization by sound and in the HUD as ” 2D overview (bird’s eye view) ” 3D arrow

Tönnis and Klinker: Effective Control of a Car Driver’s Attention for Viaual and Acoustic Guidance towards the Direction of Imminent Danger, ISMAR 06, Oct. 2006.

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Car Driving

ing (Danger er from the e Side) e) -

Evaluations in a Fixed-Base Driving Simulator: “ 3D arrow (with or without sound) better than 2D overview. ” Faster recognition of danger (shorter reaction times). ” Smaller lane deviations. “ Multi-modal warnings (picture and sound) better than graphical warning alone. “ Important: ” Use of a real HUD in the driving simulator. ” Model of the 3D arrow.

Tönnis and Klinker: Effective Control of a Car Driver’s Attention for Viaual and Acoustic Guidance towards the Direction of Imminent Danger, ISMAR 06, Oct. 2006.c

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Car Driving

ing (Simulat latio ion Environmen ents) s) -

Goal: “ Analysis and evaluation of new MMIs in realistic, dynamic scenes Alternatives: “ Experiments using real cars in real traffic ” Dangerous ” Situations cannot be selected systematically “ Driving simulators ” Significant modeling effort ” Partially insufficient realism (esp. w.r.t. dynamic traffic behavior)

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Car Driving

ing (Simulat latio ion Environmen ents) s) -

“ CAR Simulation Environment “ Parent Simulation Environment

Sandor and Klinker: Lessons Learned in Designing Ubiquitous Augmented Reality User Interfaces. In: Emerging Technologies of Augmented Reality. Interfaces & Design, Idea Group Inc, 2006. Tönnis: The Tangible Car: Rapid Intuitive Traffic Scenario Generation in a Hybrid Table-Top and Virtual Environment. 4th Int. Workshop on the Tangible Space Initiative, in Conjunction with ISMAR 07, N

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Professi

essional Diving Operat atio ions s -

Stress due to environmental and working conditions “ Pressure “ Visibility “ Weightlessness “ Current Restrictions on divers‘ sensory input cause “ Diminished spatial position and orientation awareness “ Reduced cognition and memory “ Reduced confidence and safety, increased fear (even panic) Improve conditions by adding elements in divers‘ views to increase “ Awareness “ Safety

Morales, Keitler, Maier and Klinker: An Underwater Augmented Reality System for Professional Diving Operations. OCEANS 2009, MTS/IEEE, Biloxi, Mississippi, Nov. 2009.

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Professi

essional Diving Operat atio ions s -

“ Hardware ” Watertight housing for web-camera and HMD (eMagin Z800 3D Visor) ” Watertight housing for HMD driver and umbilical connections to surface “ Tracking setup (Ubitrack) ” Underwater camera calibration ” Underwater HMD calibration (SPAAM) ” Laminated marker

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Professi

essional Diving Operat atio ions s -

Underwater maintenance “ Provide information relevant to ” Orientation and position awareness “ Artificial horizon (red grid) ” Navigation aids “ Red arrow + rubber band ” Context-dependent memory “ 3D model (e.g. of an assembling task)

Morales, Keitler, Maier and Klinker: An Underwater Augmented Reality System for Professional Diving Operations. OCEANS 2009, MTS/IEEE, Biloxi, Mississippi, Nov. 2009.

Usabi bili lity y in Real l Life (Dangero rous s Situations) s)

• Professi

essional Diving Operat atio ions s -

Underwater vessel inspection “ Provide information relevant to ” Navigation aids “ Red arrow + rubber band ” Context-dependent memory “ 3D model of vessel

Morales, Keitler, Maier and Klinker: An Underwater Augmented Reality System for Professional Diving Operations. OCEANS 2009, MTS/IEEE, Biloxi, Mississippi, Nov. 2009.

“ Collection of large assortments in series production

” Large storage area, many objects, varying size ” Flexible arrangement in the storage area ” Changing staff

“ Several phases

” Course navigation ” Fine navigation

“ Existing systems

” Mobile data terminal with scanner ” Pick-by-light ” Pick-by-voice

24

Usabi bili lity y in Real l Applicatio ions

• Navigati

ation Support in Picki king Tasks ks -

Schwerdtfeger, Reif, Günthner, Klinker: Pick-by-Vision – There is Something to Pick at the End of the Augmented Tunnel. Submitted to: Special Issue on “Autmented Reality” of the Springer Journal on Virtual Reality (M. Billinghurst and D. Schmalstieg, eds.), Springer Verlag, 2010.

Pick-by-Vision “ Omni-directional support of object selection “ HMD-based “ Visualisations ” Direct: rectangle ” Meta: tunnel “ Ubiquitous tracking

25

Usabi bili lity y in Real l Applicatio ions

• Navigati

ation Support in Picki king Tasks ks -

Schwerdtfeger, Reif, Günthner, Klinker: Pick-by-Vision – There is Something to Pick at the End of the Augmented Tunnel. Submitted to: Special Issue on “Autmented Reality” of the Springer Journal on Virtual Reality (M. Billinghurst and D. Schmalstieg, eds.), Springer Verlag, 2010.

Important issues “ Quality and understandability of the 3D display: uniqueness (Arrow vs. frame) “ Support for estimating appropriate speed of motion (rotation) (Arrow/rubber band vs. tunnel) “ Occlusions

26

Usabi bili lity y in Real l Applicatio ions

• Navigati

ation Support in Picki king Tasks ks -

Schwerdtfeger, Reif, Günthner, Klinker: Pick-by-Vision – There is Something to Pick at the End of the Augmented Tunnel. Submitted to: Special Issue on “Autmented Reality” of the Springer Journal on Virtual Reality (M. Billinghurst and D. Schmalstieg, eds.), Springer Verlag, 2010.

Long series of user tests “ Several evaluations (objective and subjective data, plus informal comments) “ Informal comments from exhibits, fairs, open houses. “ Collaboration with occupational scientists (analysis of heart rate variablilty) Experiences “ Special (non-standard) initial learning phase („try-and-ask‚) “ Optional interruptions by the supervisor “ Initially many „formative‚ tests, expert interrogations with few people

27

Usabi bili lity y in Real l Applicatio ions

• Navigati

ation Support in Picki king Tasks ks -

Schwerdtfeger, Reif, Günthner, Klinker: Pick-by-Vision – There is Something to Pick at the End of the Augmented Tunnel. Submitted to: Special Issue on “Autmented Reality” of the Springer Journal on Virtual Reality (M. Billinghurst and D. Schmalstieg, eds.), Springer Verlag, 2010.

Usabi bili lity y in Real l Applicatio ions Intel elli ligen ence ce Amplificati ication

Augmented chemical reactions “ Dynamic 3D visualization of potential molecular bonds ” Investigation of spatial impact (is there enough space?) ” AR-based interactive exploration of options “ Teaching tool for students: ” Interactively controlled simulation and visualization of atomic forces

Maier, Tönnis, Klinker, Raith, Drees, Kühn: What Do you Do When Two Hands Are not Enough? Interactive Selection of Bonds Between Pairs of Tangible Molecules International Symposium On 3D User Interfaces (3DUI), Waltham, MA, March 2010.

Usabi bili lity y in Real l Applicatio ions Intel elli ligen ence ce Amplificati ication

Augmented chemical reactions “ Example: catalyst design for metal organic reactions ” Catalyst: organomolybdenum complex ” Educt 1: peroxide molecule (front), bond with central metal of the catalyst ” Educt 2: ethen molecule (right) “ Problem: generally more than two hands required ” Bond selection “ Distance-based selection “ Selection by shaking (toggling through a list) ” Confirmation

Maier, Tönnis, Klinker, Raith, Drees, Kühn: What Do you Do When Two Hands Are not Enough? Interactive Selection of Bonds Between Pairs of Tangible Molecules International Symposium On 3D User Interfaces (3DUI), Waltham, MA, March 2010.

First evaluation

“ 19 persons (7f, 12m, 20-51 y) “ Most of the without prior exposure to AR “ Within-subject design “ 2 test phases (24 bonds each, permuted)

30

Usabi bili lity y in Real l Applicatio ions Intel elli ligen ence ce Amplificati ication

Maier, Tönnis, Klinker, Raith, Drees, Kühn: What Do you Do When Two Hands Are not Enough? Interactive Selection of Bonds Between Pairs of Tangible Molecules International Symposium On 3D User Interfaces (3DUI), Waltham, MA, March 2010.

Overvi rview w of this s talk

“ Ubiquitous Information Presentation Schemes “ Usability in Real Life (Dangerous Situations) “ Ubiquitous Tracking

Ubiquit itous s Trackin king

• Goals

s -

Creation and implementation of a framework independently of specific applications and devices that is capable of providing context-dependent tracking services in heterogeneous, wide areas. “ Trackers partially wearable and partially stationary “ Trackers available across several applications “ Applications independent of tracker installations “ Tracking arrangements configured dynamically, establishing real-time connections between trackers and applications “ Critical aspects:

” Explicit description of spatial relationships (Spatial Relationship Graph) ” Automatic compilation of spatial relationships into data flows between sensors, fusion components, and applications (Data Flow Network) ” Efficient, robust and precise use in real applications

Ubiquit itous s Trackin king

• Spatial

ial Relat latio ionship ship Graph (SRG) G) -

3D World SRG

ART HMD* Door*

6DoF 6DoF

Eye, HMD Door

6DoF 6DoF 6DoF static (calibration) static (registration) dynamic (tracking) dynamic (tracking)

Ubiquitous itous Trackin cking

• Automatic

ic Compilat latio ion into Data Flow Networ works ks -

• Spatial Relationship Graph (SRG)
• Data Flow Network (DFN): SRG edges = DFN nodes

ARTObj ARTHMD Invert Multiply Application ART HMD Door

6DoF 6DoF 6DoF

Integrat ration Concept for Applica icatio ions

Plant-wide Installations of Sensors

Sensor Sensor Sensor Client

Application

S UbiTrack Server Plant-wide-SRG Database Server trackframe Manager Sensor Client Application Client

Application-SRG

Sensor Client

Huber, Pustka, Keitler, Echtler and Klinker: A System Architecture for Ubiquitous Tracking Environments, ISMAR 07, Nov. 2007. Becker, Huber and Klinker: Utilizing RFIDs for Location-Aware Computing, UIC, June 08.

Spatial ial Relat atio ionship ship Patterns erns

• Inversion Concatenation Sensor Fusion
• 3D-3D Pose Estimation 2D-3D Pose Estimation Hand-Eye Calibration

A B C

6DoF 6DoF 6DoF

A B

6DoF 6DoF

A B

6DoF 6DoF 2..*

T O F

3DoF 3DoF 6DoF 3..*

C I

3D2D

F O

3DoF 2DoF 6DoF 4..*

R H

6DoF

C O

6DoF 6DoF 6DoF 3..*

Pustka, Huber, Bauer and Klinker: Spatial Relationship Patterns: Elements of Reusable Tracking and Calibration Systems, ISMAR 06, Oct. 2006. AWARD.

Automated Senso sor r Fusion

Patterns/algorithms for “ Calibration

” Camera estimation ” Hand-eye calibration ” Display calibration (SPAAM)

“ Registration

” 3D-3D pose estimation

“ Sensor fusion ” Complementary

“ Functionally complementary “ Temporally/spatially complementary

” Competitive

“ Binary competitive („winner take all‚) “ Mixed competitive (error reducing: Kalman)

” Co-operative

“ Independent co-operation (concatenation, correction) “ Dependent co-operation (initialization)

Error Estimation and Propagation in SRGs

“ Computation of sensor error

” Gaussian errors ” High-precision ART tracking ” Simple marker tracking

“ Fusion algorithm

” Kalman filter ” Fully integrated into framework

Bauer, Schlegel, Pustka, and Klinker: Predicting and Estimating the Accuracy of N-Ocular Optical Tracking Systems, ISMAR 06, Oct. 2006.

Dynamic ic Reconfig igurat ration

“ Scenario: AR-equipped person enters room with high-precision tracking “ Client and application should automatically connect “ Idea: Use low-precision tracking to change SRG

A A A

Summary y / Conclusi lusions

“ Augmented/Mixed Reality has the potential to significantly help computer users integrate and understand a wealth of digital information in their real surroundings “ AR won’t be used in isolation “ Our approach: AR → Ubiquitous AR “ Research issues under investigation ” Concepts/middleware for integrating AR into a wider, ubiquitous information presentation scheme ” Usability “ Particular emphasis: User interfaces for critical or complex situations Recently:

• AR-Tunnel for picking
• .Visualization concepts for traffic anticipation
• Augmented Chemistry
• AR for divers

Thank you

“ Ubiquitous interaction in real environments

” AR-Tunnel for picking ” Visualization concepts for traffic anticipation („just beyond the horizon‚) ” Augmented Chemistry ” AR for divers ” Navigation metaphors in large nD-environments („virtual Arabia‚) ” 2D/3D UIs for use in catastrophic events ” AR on mobile devices

“ Multi-touch environments: TischLib (OpenSource) “ Tracking cart for wide area tracking (ISMAR tracking contest)

” People tracking ” Sensor synchronization ” Less precise tracking (wide area) ” ToF based tracking ” Trackman

“ Outreach / Teaching:

” Itüpferl ” Mädchen machen Technik ” Rappelkiste ” Bachelor program „Informatik: Games Engineering‚ 42

Recent / n new/ w/ other r things ings we do