VU Augmented Reality on Mobile Devices VU Augmented Reality on - - PowerPoint PPT Presentation

▶
vu augmented reality on mobile devices vu augmented
SMART_READER_LITE
LIVE PREVIEW

VU Augmented Reality on Mobile Devices VU Augmented Reality on - - PowerPoint PPT Presentation

Sep 14, 2023 176 likes •732 views

VU Augmented Reality on Mobile Devices VU Augmented Reality on Mobile Devices Introduction Introduction What is AR What is AR Interaction Techniques Navigation Collaboration Navigation, Collaboration Visualization

slide-1
SLIDE 1

VU Augmented Reality on Mobile Devices VU Augmented Reality on Mobile Devices

  • Introduction

What is AR

  • Introduction – What is AR
  • Interaction Techniques
  • Navigation Collaboration

Navigation, Collaboration

  • Visualization Techniques
  • Visual Coherence

Visual Coherence

  • Tracking
  • Gudrun Klinker

1

slide-2
SLIDE 2

Ubiquitous Augmented Reality in AR-ready Environments in AR-ready Environments

Prof Gudrun Klinker Ph D

  • Prof. Gudrun Klinker Ph.D

Technische Universität München Fachgebiet Augmented Reality (FAR)

Date: May 30th 2011 Time: 2:15pm -3:45pm Location: TU Vienna 1040 Wien, Karlsplatz 13 Hauptgebäude HS 7, Ground Floor, Stiege VII

Abstract

In this talk, I will present some of our recent work as well as our vision towards providing ubiquitous Augmented Reality services in AR-ready environments. I will elaborate on the concepts of ubiquitous tracking, ubiquitous information presentation and ubiquitous manipulation leading to the vision that users will use AR as one of their means to experience and manipulate a mixed physical and virtual reality. I will report on some observations we made when we tried these concepts in real applications in which

2

users need to keep a keen eye on primary tasks in their real environment while also using/exploring an associated virtual information space.

slide-3
SLIDE 3

Internship at Qualcomm Austria Research Center Qualcomm Austria Research Center

Offering

Skills/Experience

Offering

  • Be part of a team developing

mobile Augmented Reality enabling technology

  • Design, implement, and verify

Skills/Experience

  • Outstanding C++ and/or Java

programming skills,

  • expertise in object-oriented design,

and

Design, implement, and verify algorithms for Augmented Reality

  • Support development of new

computer vision approaches f

and

  • a good foundation in one of the

following areas

  • Augmented Reality &

Computer Vision

p pp for mobile devices

  • 3-4 month, employment

Computer Vision

  • Integration & Verification
  • Software Tools
  • Please apply directly on our website at

www.qualcomm.com/careers/ under requisition

3

number E1879350.

slide-4
SLIDE 4

Tracking Tracking

  • Requirements
  • Requirements
  • Provide position and orientation
  • Untethered

l ki l

  • large working volume
  • Robustness !
  • Indoor vs. outdoor
  • Indoor: Can instrument environment
  • Outdoor: self-contained or large scale infrastructure
  • No single sensor provides 6DoF
  • Tracking vs. Initialization

4

slide-5
SLIDE 5

Indoor Tracking Indoor Tracking

  • Ultrasonic beacon array
  • Ultrasonic beacon array
  • AT&T Bat, Intersense IS900
  • Magnetic Trackers

g

  • Infrared LED array
  • UNC HiBall, MIT‘s locust swarm

O t id i t i i

  • Outside-in computer vision
  • Observer cameras + passive IR

targets (e.g., ART Track - roomsize)

  • Inside-out computer vision
  • Fiducials (e.g. ARToolKit)

Dead reckoning techniques

5

  • Dead-reckoning techniques
slide-6
SLIDE 6

Example: AT&T Sentient AR Example: AT&T Sentient AR

6

slide-7
SLIDE 7

Signpost 2003 Signpost 2003

  • Guides a user trough an unfamiliar building
  • Guides a user trough an unfamiliar building
  • Requires markers on the walls and 3D model of the

building

7

slide-8
SLIDE 8

Outdoor Outdoor

  • Position
  • Position
  • GPS, UWB, Omnisense
  • WiFi, Cell information
  • Orientation
  • Gyroscopes, magnetometers, linear accelerometers
  • Inside out computer vision
  • Inside-out computer vision
  • Natural features
  • Image databases
  • Requires sensor fusion

8

slide-9
SLIDE 9

GPS GPS

  • Pseudo range to satellite
  • Pseudo range to satellite
  • Robust, always & everywhere available
  • many error sources along the way
  • Consumer level
  • 5 - 50m
  • Urban canyons, multi-path, shadowing

y , p , g

  • DGPS
  • local correction of atmospheric errors

0 5

  • >0.5m
  • RTK
  • Phase information

9

  • >2.5cm horizontally, >10cm height
slide-10
SLIDE 10

Other RF methods Other RF methods

  • WiFi Cell tower
  • WiFi, Cell tower
  • Skyhook, large database of WiFi and

cell tower

  • Used in mobile phones
  • Used in mobile phones
  • Ubisense
  • Very short pulse signal
  • Range and direction measurements
  • 30m range, 15cm accuracy
  • Omnisense
  • Omnisense
  • Coded signal
  • 4km range, 0.5m accuracy

10

slide-11
SLIDE 11

Orientation Orientation

  • Inertial measurement units
  • Inertial measurement units
  • Gyroscopes
  • linear accelerometers ~ gravity

t t 3D

  • magnetometer ~ 3D compass
  • laser gyroscopes
  • Townwear, Satoh ’99

,

  • Magnetometer ~ 3D

compass

  • Deviations a common problem

11

slide-12
SLIDE 12

Example: Smart Vidente Example: Smart Vidente

  • Tablet PC J3400
  • Tablet PC J3400
  • C2Duo (SU9600 1,60GHz ,1.6 kg)
  • Inertial Sensor XSENS MTx
  • VRMagic VRM FC-6 COB COLOR
  • RTK GPS
  • Novatel OEMV1 – L1
  • Novatel OEMV2 – L1/L2
  • Differential data from network

Differential data from network

  • Kalmanfilter
  • Validated to: 5cm 2D location,

12

Validated to: 5cm 2D location, 10cm height

Tablet PC based AR System

slide-13
SLIDE 13

Reprojection error Reprojection error

13

slide-14
SLIDE 14

Handheld Information Browsers Handheld Information Browsers

Peak ar Wikitude Geo-referenced Wikipedia information Peak.ar

  • verlays names of peaks

14

Layar Dedicated content layers

slide-15
SLIDE 15

Navigation Navigation

acrossair acrossair Nearest Tube Wikitude Drive Navigationsinformation

15

slide-16
SLIDE 16

Image-based Tracking Image-based Tracking

  • Many devices have cameras !
  • Many devices have cameras !
  • Pixel accuracy -> computer vision

Marker Natural Features Image-based Localization Visual Search

16

slide-17
SLIDE 17

Image-based Tracking Image-based Tracking

  • Many devices have cameras !
  • Many devices have cameras !
  • Pixel accuracy -> computer vision
  • Model-based tracking

Model based tracking

  • Natural features
  • Computer graphic models

R i I iti li ti

  • Requires Initialization
  • Not robust
  • Sensor fusion
  • Sensor fusion
  • Recovery methods

17

slide-18
SLIDE 18

Many different methods and combinations Many different methods and combinations

  • Features
  • Features
  • Points
  • Edges

H i

  • Horizon
  • • Sensor fusion
  • Gyroscopes

Azuma, ’99

y p

Kretschmer et al., ’02

18

Behringer, ’98 Ribo et.al., ’02

slide-19
SLIDE 19

Starting Simple: Marker Tracking Starting Simple: Marker Tracking

  • Has been done for more than 10 years
  • Has been done for more than 10 years
  • Some mobile phones today are faster than computers of that time
  • Several open source solutions exist

p

  • Fairly simple to implement
  • – Standard computer vision methods

A t l k id 4 i t

  • A rectangular marker provides 4 corner points
  • -> enough for pose estimation!

19

slide-20
SLIDE 20

Marker Tracking Pipeline Overview Marker Tracking Pipeline Overview

G l D ll thi i l th 20 illi d bil h

20

Goal: Do all this in less than 20 milliseconds on a mobile phone…

slide-21
SLIDE 21

Marker Tracking – Overview Marker Tracking – Overview

21

slide-22
SLIDE 22

Marker Tracking – Fiducial Detection Marker Tracking – Fiducial Detection

  • Threshold the whole image
  • Search scan line per scan line for edges
  • Search scan-line per scan-line for edges

(white to black steps)

  • Follow edge until either
  • Back to starting pixel
  • Back to starting pixel
  • Image border
  • Check for size
  • Reject fiducials early that are too small

Reject fiducials early that are too small

22

slide-23
SLIDE 23

Marker Tracking – Rectangle Fitting Marker Tracking – Rectangle Fitting

  • Start with an arbitrary point “x”

Start with an arbitrary point x

  • The point with maximum distance must be a corner c0
  • Create a diagonal through the center
  • Find points c1 & c2 with maximum distance left and right of
  • Find points c1 & c2 with maximum distance left and right of

diagonal

  • New diagonal from c1 to c2
  • Find point c3 right of diagonal with maximum distance
  • Find point c3 right of diagonal with maximum distance
  • Repeat to check if no more corners exist

23

slide-24
SLIDE 24

Marker Tracking – Pattern checking Marker Tracking – Pattern checking

  • Calculate homography using the 4 corner points
  • “Direct Linear Transform” algorithm
  • Direct Linear Transform algorithm
  • Maps normalized coordinates to marker coordinates (simple

perspective projection, no camera model)

  • Extract pattern by sampling
  • Check pattern
  • Id (implicit encoding)
  • Template (normalized cross correlation)

F 2D 3D d ti ti

  • Four 2D-3D correspondences ~ pose estimation

24

slide-25
SLIDE 25

Natural Feature Tracking Natural Feature Tracking

  • More difficult than marker tracking
  • More difficult than marker tracking
  • Markers are designed for their purpose
  • The natural environment is not…
  • • Less well established methods
  • Every year new ideas are proposed
  • • Usually much slower than marker tracking
  • • Usually much slower than marker tracking

25

slide-26
SLIDE 26

Detection in every Frame Detection in every Frame

  • This is what most trackers“
  • This is what most „trackers

do…

  • Targets are detected every

g y frame

  • Popular because detection

d ti ti and pose estimation are solved simultaneously

26

slide-27
SLIDE 27

Natural Feature Tracking by Detection Natural Feature Tracking by Detection

  • SIFT
  • State of the art for object recognition
  • Known to be slow (best implementation
  • Ferns
  • State of the art for fast pose tracking
  • Memory intensive (requires ~10x too

( p for phones is ~10-100x too slow for real- time use)

  • Typically used off-line

y ( q much memory for phones)

  • Long training phase

27

SIFT: [Lowe, 2004] Ferns: [Ozuysal, 2007] See [Wagner, 2008]

slide-28
SLIDE 28

Overview on SIFT descriptor Overview on SIFT descriptor

  • key point localization
  • key-point localization
  • Searching for minima and maxima in scale space (also provides

scale estimation for scale invariance)

  • Requires building pyramid of Difference of Gaussian
  • Requires building pyramid of Difference of Gaussian
  • Major performance bottleneck of SIFT
  • Feature description
  • Estimation of dominant 2D feature orientations
  • Orientation histogram of 4x4 sub-regions (128 bins)
  • Feature matching
  • Feature matching
  • Feature database stored as k-d tree for sub-linear search time

28

slide-29
SLIDE 29

PhonySIFT – key-point Detection PhonySIFT – key-point Detection

  • Scale space search way
  • Scale-space search way

too slow…

  • Replaced with FAST*

p corner detector

  • Robustness by database

lti l l

  • n multiple scales

* [Rosten, 2005]

30

slide-30
SLIDE 30

PhonySIFT – Descriptor Creation PhonySIFT – Descriptor Creation

  • Estimate dominant key point
  • Estimate dominant key-point
  • rientation using gradient

histogram

  • Compensate for detected
  • rientation

C t d i t f 3 3

  • Create descriptor from 3x3

subregions with 4 directions = 36 bins instead of 128

  • – All weights pre-computed!

31

slide-31
SLIDE 31

PhonySIFT – Descriptor Matching PhonySIFT – Descriptor Matching

  • Brute force matching not an option
  • Brute force matching not an option
  • K-d tree still not efficient enough
  • New approach: Spill forest

New approach: Spill forest

  • Multiple spill trees (k-d tree with overlap)
  • Randomized dimensions for pivoting
  • Voting to merge
  • Voting to merge

results from different trees

41

slide-32
SLIDE 32

PhonySIFT – Outlier Removal PhonySIFT – Outlier Removal

  • Cascade of removal techniques
  • Cascade of removal techniques
  • • Start with cheapest, finish with most expensive…
  • – Overall rotation check
  • – Line tests
  • – RANSAC for homography estimation

43

slide-33
SLIDE 33

Overall rotation check Overall rotation check

  • SIFT provides key point rotation for free
  • SIFT provides key-point rotation for free
  • – All key-points must have same relative rotation
  • – Look at histogram and keep only peaks

2π

44

slide-34
SLIDE 34

Overall rotation check Overall rotation check

  • SIFT provides key point rotation for free
  • SIFT provides key-point rotation for free
  • – All key-points must have same relative rotation
  • – Look at histogram and keep only peaks

2π

45

slide-35
SLIDE 35

Performance of SIFT and Ferns modified for mobile phone tracking mobile phone tracking

53

slide-36
SLIDE 36

Speed Analysis for PhoneSIFT Speed Analysis for PhoneSIFT

54

slide-37
SLIDE 37

NFT with SIFT on a Mobile Phone NFT with SIFT on a Mobile Phone

55

slide-38
SLIDE 38

Doing it better: Dedicated Detection and Tracking Dedicated Detection and Tracking

56

slide-39
SLIDE 39

PatchTracker Workflow (1) PatchTracker Workflow (1)

  • Target Recognition:
  • Target Recognition:
  • find feature in reference image
  • Target Tracking:

g g

  • Take previous pose and apply motion model
  • Get estimate for what we are looking for
  • Create affine warped patches of reference features
  • Create affine warped patches of reference features
  • Closely resemble how the feature should look in the

camera image g

  • Project patches into camera image and use normalized cross

correlation (NCC) to match

57

slide-40
SLIDE 40

Patch Prediction and Warping Patch Prediction and Warping

58

slide-41
SLIDE 41

Patch Prediction and Warping Patch Prediction and Warping

patch template

59

slide-42
SLIDE 42

Patch Prediction and Warping Patch Prediction and Warping

patch template

60

slide-43
SLIDE 43

Patch motion measurements Patch motion measurements

  • Search half res with low number of feature
  • Search half-res with low number of feature
  • E.g. 25 features, search radius of 5 pixels
  • Refine estimated pose

p

  • Search full-res with many features
  • E.g. 100 features, search radius of 2 pixels

R fi f fi l

  • Refine for final pose

61

slide-44
SLIDE 44

PatchTracker Workflow Analysis PatchTracker Workflow Analysis

  • Affine warped patches allow very strong affine
  • Affine warped patches allow very strong affine

transformations (tilt close to 90°)

  • NCC allows severe lighting changes

g g g

  • 5 pixel search radius at half-res allows “wide” base-

line

  • 5 x 2 x 20Hz = 200 pixels/sec
  • Tracking 100 features at full-res strongly reduces

jitter jitter

62

slide-45
SLIDE 45

Detection and Tracking Detection and Tracking

66

slide-46
SLIDE 46

PatchTracker in Action (2) PatchTracker in Action (2)

67

slide-47
SLIDE 47

When does it break? When does it break?

68

slide-48
SLIDE 48

When does it break? When does it break?

69

slide-49
SLIDE 49

How fast is it? How fast is it?

70

slide-50
SLIDE 50

Orthogonal Strengths and Weaknesses Orthogonal Strengths and Weaknesses

71

slide-51
SLIDE 51

High Robustness at fixed camera frame rate High Robustness at fixed camera frame rate

72

slide-52
SLIDE 52

No need to stay in 2D No need to stay in 2D

  • Patches don’t need to come from a single plane
  • Patches don t need to come from a single plane
  • Per feature
  • 3D position

p

  • Plane normal for affine warp
  • Multiple reference images
  • Continue as before
  • Continue as before ...

73

slide-53
SLIDE 53

Features in 3D Features in 3D

  • Different geometry per feature

Different geometry per feature

  • Defined by 3D shape

74

slide-54
SLIDE 54

Tracking 3D Objects Tracking 3D Objects

75