Handheld Augmented Reality Reto Lindegger lreto@ethz.ch Tuesday, - - PowerPoint PPT Presentation

▶
handheld augmented reality
SMART_READER_LITE
LIVE PREVIEW

Handheld Augmented Reality Reto Lindegger lreto@ethz.ch Tuesday, - - PowerPoint PPT Presentation

Jan 15, 2023 361 likes •876 views

Handheld Augmented Reality Reto Lindegger lreto@ethz.ch Tuesday, March 26, 2013 Distributed Systems Seminar 1 Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary AUGMENTED REALITY

slide-1
SLIDE 1

Tuesday, March 26, 2013 Distributed Systems Seminar

Reto Lindegger

lreto@ethz.ch

Handheld Augmented Reality

1

slide-2
SLIDE 2

Tuesday, March 26, 2013 Distributed Systems Seminar

AUGMENTED REALITY

2

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-3
SLIDE 3

Tuesday, March 26, 2013 Distributed Systems Seminar

A Definition

Three important characteristics:

  • Combines real and virtual environment
  • Interactive in real-time
  • Registered in 3D

3

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

Definition from: A Survey of Augmented Reality by Ronald T. Azuma (1997)

slide-4
SLIDE 4

Tuesday, March 26, 2013 Distributed Systems Seminar

A Definition

What this excludes:

4

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

  • 2D Overlay
  • 3D Objects rendered in a movie
slide-5
SLIDE 5

Tuesday, March 26, 2013 Distributed Systems Seminar

A Definition

5

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

What this includes:

Virtual objects in an rendered real- world reconstruction Interaction between real-world

  • bjects and virtual objects

Information in a video feed

slide-6
SLIDE 6

Tuesday, March 26, 2013 Distributed Systems Seminar

Handheld Augmented Reality

  • Devices which fit into a user‘s hand
  • Portable and (ideally) not infrastructure dependent
  • Smartphones as augmented reality displays
  • Commodity hardware
  • Widely used
  • But:
  • Limited computation power
  • User has to hold device all the time

6

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-7
SLIDE 7

Tuesday, March 26, 2013 Distributed Systems Seminar

4 Problems and 4 Solutions

  • Indoor Navigation
  • One Handed Mobile Device Interaction
  • Situated Visualization
  • 3D Surface Reconstruction

7

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-8
SLIDE 8

Tuesday, March 26, 2013 Distributed Systems Seminar

INDOOR NAVIGATION

WITH ACTIVITY-BASED INSTRUCTIONS

8

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-9
SLIDE 9

Tuesday, March 26, 2013 Distributed Systems Seminar

Problem Statement

  • Navigation in unknown, complex building
  • No GPS
  • Low effort for installation, low cost
  • Sparse localization
  • As accurate as possible

9

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-10
SLIDE 10

Tuesday, March 26, 2013 Distributed Systems Seminar

Previous Work

Sensing Infrastructure

  • Cyberguide Project, infrared
  • BAT system, ultrasonic waves
  • Chittaro and Nadalutti, RFID

➡ Require instrumentation of the environment

10

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-11
SLIDE 11

Tuesday, March 26, 2013 Distributed Systems Seminar

Previous Work

Sparse infrastructure

  • „You-are-here“ maps
  • Information at checkpoints
  • Way between checkpoints completely up to the user

➡ Finding next checkpoint not assisted

11

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-12
SLIDE 12

Tuesday, March 26, 2013 Distributed Systems Seminar

Previous Work

Measuring movement

  • Start point known
  • Measure movement with sensors and camera
  • Instruction based on movement and estimated location

➡ Inaccurate over time ➡ Phone movement indistinguishable from user movement

12

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-13
SLIDE 13

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

Alessandro Mulloni et al. :

Handheld Augmented Reality Indoor Navigation with Activity- Based Instructions

  • Combine egocentric and exocentric navigation
  • egocentric: turn-by-turn navigation
  • exocentric: „you-are-here“ maps
  • Main requirements:
  • Robustness to user failures
  • Minimal instrumentation of the environment
  • Adaptiveness to localization accuracy
  • Interactive flow of activities with minimal user input

13

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-14
SLIDE 14

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

Sparse localization: Info points

  • floor-mounted posters
  • Recognized by camera
  • Act as checkpoint for user and software
  • Change user interface: more information
  • Recalculate path

14

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-15
SLIDE 15

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

Between info points: Activities

  • Navigation between info points activity-based
  • Total route described in sequence of activities
  • „Go 5 steps“, „Turn left“, „Go 8 steps“
  • Accelerometer used to count steps
  • User can actively activate next activity
  • Scrolling through list and selecting any activity also possible

15

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-16
SLIDE 16

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

16

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

At Info points Info point, recognized by camera Between Info points

slide-17
SLIDE 17

Tuesday, March 26, 2013 Distributed Systems Seminar

Evaluation

  • User study: navigation in an unfamiliar building
  • With and without information point
  • Compare used time, navigation errors and user feedback
  • Navigation works well, info points improve performance and

usability

  • Step counter just hint, but not used to switch to next activity

17

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-18
SLIDE 18

Tuesday, March 26, 2013 Distributed Systems Seminar

Personal Opinion

  • Application could be used in a museum or in an airport
  • Guide visitors through building
  • Find shortest way to an exhibit
  • Useful for people with visual impairment
  • Extended with voice in- and output
  • Has to be quite accurate
  • Not realistic for daily use as it is now

18

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-19
SLIDE 19

Tuesday, March 26, 2013 Distributed Systems Seminar

LOOKING AT YOU

FUSED GYRO AND FACE TRACKING FOR VIEWING LARGE IMAGERY ON MOBILE DEVICES

19

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-20
SLIDE 20

Tuesday, March 26, 2013 Distributed Systems Seminar

Problem Statement

  • Photos are getting bigger:
  • High resolution
  • Panoramic images
  • Multi-Perspective images
  • Screens are getting smaller:
  • Smartphones
  • Tablets
  • Many sensors on mobile devices available:
  • Accelerometer
  • Gyroscope
  • Compass
  • Camera

20

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-21
SLIDE 21

Tuesday, March 26, 2013 Distributed Systems Seminar

Previous Work

  • Scrolling speed coupled with zoom level
  • Google Street View (compass and gyro)
  • Tourwrist (360° panoramas)
  • Gaze-enhanced scrolling techniques
  • Glasses-free 3D displays (face tracking)

http://iihm.imag.fr/en/demo/hcpmobile

21

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-22
SLIDE 22

Tuesday, March 26, 2013 Distributed Systems Seminar

Previous Work

Glasses-free 3D displays (face tracking)

22

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-23
SLIDE 23

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

Neel Joshi et al. :

Fused Gyro and Face Tracking for Viewing Large Imagery on Mobie Devices

  • Different applications
  • Large field of view
  • Wide multi-perspective panorama
  • Multi-views
  • Combinations
  • Touch-based interaction has drawbacks
  • Hand obscures part of the picture
  • Difficult to distinguish between navigation and other interaction
  • Requires 2 hands

23

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-24
SLIDE 24

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

Using only gyroscope data is not enough

24

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-25
SLIDE 25

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

  • In the real world we
  • move our gaze relatively to a scene
  • move an object relatively to our gaze
  • In both cases, head moves relatively to the scene/object
  • Exploiting the relative position of the head to the screen
  • Face tracking can provide input in 3 dimensions
  • position, based on face location (x,y)
  • depth, based on face size (z)
  • but suffers from noise, high latency and limited field of view

25

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-26
SLIDE 26

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

  • Combine gyroscope data and face tracking!
  • Features:
  • Navigation by moving head and/or device
  • Natural and smooth navigation
  • Don‘t have to spin 360° in place
  • Zooming by changing distance between face and device

26

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-27
SLIDE 27

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

The application in action

27

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-28
SLIDE 28

Tuesday, March 26, 2013 Distributed Systems Seminar

Evaluation

  • User study: Find a mark placed somewhere on a large image
  • Comparison of different input techniques
  • Finger based, combined gyro and face tracking
  • 50% of the participants indicated that touch control is easier

than gesture control

  • Motion based methods not significantly faster than finger

based

28

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-29
SLIDE 29

Tuesday, March 26, 2013 Distributed Systems Seminar

Personal Opinion

  • Very interesting technology for viewing multi-view images
  • 3D images look impressive
  • Useful to view large panoramic images
  • Applications like StreetView
  • Get impression of the location where picture was taken
  • Not useful for „flat“ images
  • Maps
  • Large high-resolution non-panoramic images

29

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-30
SLIDE 30

Tuesday, March 26, 2013 Distributed Systems Seminar

SITE LENS

SITUATED VISUALIZATION TECHNIQUES FOR URBAN SITE VISITS

30

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-31
SLIDE 31

Tuesday, March 26, 2013 Distributed Systems Seminar

Problem Statement

  • Urban planner, urban designer or architects visit a site before

the design activity

  • Interested in different information and facts about the

location

  • Carbon monoxide measurement
  • demographics
  • traffic flows
  • congregation of people

➡ Much data from different sources, has to be processed and analyzed after the visit, with maps, videos and photos etc.

31

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-32
SLIDE 32

Tuesday, March 26, 2013 Distributed Systems Seminar

Previous Work

  • ArcGIS
  • Vidente Project: visualization of subsurface features

(pipelines, power cables) for utility field workers

32

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-33
SLIDE 33

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

Sean White et al. :

SiteLens: Situated Visualization Techniques for Urban Site Visits

  • Preparation: Acquire environmental data
  • collecting data by measurements
  • get data from EPA (Environmental Protection Agency)
  • Three different modes for presenting data:
  • screen fixed display
  • world-fixed, augmented reality display

33

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-34
SLIDE 34

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

Visual Representation

  • Spheres
  • Value is mapped to altitude and

color

  • Cylinders
  • Value is mapped to length and color
  • Smoke
  • Value is mapped to density
  • Position indicates, where the

data was collected

34

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-35
SLIDE 35

Tuesday, March 26, 2013 Distributed Systems Seminar

Evaluation

  • Feedback from urban designers and planner
  • Good to have measured data combined with visual

inspection

  • Map data alone could not explain high CO level at the end of a street,

visual inspection revealed that there were cars waiting to enter the highway

  • Different preferences for representation:
  • Spheres vs cylinders
  • Clouds: phycological impact more important than location accuracy
  • Live sensor data as improvement

35

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-36
SLIDE 36

Tuesday, March 26, 2013 Distributed Systems Seminar

Personal Opinion

  • Great idea to visualize measurements in the location where

they were taken

  • Importance/usefulness depends on need of urban planner/

designer

  • Only few measurements available
  • User of the application could collect and share data
  • Accuracy of data might not be guaranteed

36

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-37
SLIDE 37

Tuesday, March 26, 2013 Distributed Systems Seminar

KINECT FUSION

REAL-TIME DYNAMIC 3D SURFACE RECONSTRUCTION AND INTERACTION

37

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-38
SLIDE 38

Tuesday, March 26, 2013 Distributed Systems Seminar

Problem Statement

  • Reconstructing an real-world object or scene
  • Moving camera freely
  • Process data in real time

38

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-39
SLIDE 39

Tuesday, March 26, 2013 Distributed Systems Seminar

Goals

  • Real-time camera tracking and 3D reconstruction
  • Capture detailed 3D models with exact geometry
  • Dynamic interaction
  • Infrastructure-less
  • Support whole room construction and interaction

39

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-40
SLIDE 40

Tuesday, March 26, 2013 Distributed Systems Seminar

Previous Work

  • The Digital Michelangelo Project [2000]
  • Laser rangefinder and cameras
  • Large and quite immobile

40

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-41
SLIDE 41

Tuesday, March 26, 2013 Distributed Systems Seminar

Previous Work

  • Real-Time Visibility-Based Fusion of Depth Maps [2007]
  • Reconstruction by combining multiple depth maps
  • Depth maps computed from images captured by moving camera

41

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-42
SLIDE 42

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

  • Generate 3D reconstruction in 4 phases
  • Get live depth map from Kinect, convert into 3D points an normals
  • Calculate the movement of the camera relative to the scene
  • Derive global camera position, update consistent 3D model
  • Render view of the volume and 3D surface with raytracing

42

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-43
SLIDE 43

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

Shahram Izadi et al. :

KinectFusion: Real-Time Dynamic 3D Surface Reconstruction and Interaction

  • Use commodity hardware: Kinect
  • Cheap, Portable
  • Provides depth map
  • Provides RGB camera

43

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-44
SLIDE 44

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

  • Moving the camera leads to
  • New viewpoints
  • More details of the scene
  • Reconstruction becomes more complete over time
  • RGB camera not used for 3D model, but can be used generate

texture for the model

44

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-45
SLIDE 45

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

  • Object segmentation through direct interaction
  • Scan complete scene first
  • Move an object around
  • System monitors 3D reconstruction and observes rapid change in the

model

  • Repositioned item is segmented from the background
  • Augmented reality: interaction with the model
  • Place (virtual) objects in this 3D reconstruction
  • Shadow from virtual objects on real-world objects
  • Reflection of real-world objects on virtual objects
  • Collision detection

45

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-46
SLIDE 46

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

  • Static background, moving foreground
  • Algorithm can distinguish between moving foreground in the scene

and camera movement

  • Background movement use for camera tracking
  • Foreground movement tracked separately
  • Intersection between foreground and background can be

determined

  • Intersection is recognized as touch
  • Allows multi-touch on every surface

46

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-47
SLIDE 47

Tuesday, March 26, 2013 Distributed Systems Seminar

Solution

47

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-48
SLIDE 48

Tuesday, March 26, 2013 Distributed Systems Seminar

Personal Opinion

  • Great success, fascinating technique
  • Opens new possibilities for applications
  • Robotic
  • Entertainment
  • Designing, planning
  • Handheld camera, but visualization on screen
  • People in the scene not necessarily see the augmented reality
  • Combination with other techniques lead to new possibilities

(projection)

  • Good to see interesting and useful application of Kinect

48

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

slide-49
SLIDE 49

Tuesday, March 26, 2013 Distributed Systems Seminar 49

Many applications for handheld augmented reality:

  • Indoor Navigation with Activity-Based Instructions
  • Fused Gyro & Face Tracking for Touch-free interaction
  • Situated Visualization Techniques for Urban Site Visits
  • Real-Time Dynamic 3D Surface Reconstruction and Interaction

Potential for improvement, but most techniques seem advanced

Introduction - Indoor Navigation - Touch-free interface - Visualization - Surface Reconstruction - Summary

SUMMARY