DepthLab: Real-time 3D Interaction with Depth Maps for Mobile - - PowerPoint PPT Presentation

▶
depthlab real time 3d interaction with depth maps for
SMART_READER_LITE
LIVE PREVIEW

DepthLab: Real-time 3D Interaction with Depth Maps for Mobile - - PowerPoint PPT Presentation

Jun 05, 2023 136 likes •955 views

DepthLab: Real-time 3D Interaction with Depth Maps for Mobile Augmented Reality Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarue, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh

slide-1
SLIDE 1

DepthLab: Real-time 3D Interaction with Depth Maps for Mobile Augmented Reality

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarue, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, David Kim Google | ACM UIST 2020

slide-2
SLIDE 2

Introduction

slide-3
SLIDE 3

Introduction

slide-4
SLIDE 4

Introduction

slide-5
SLIDE 5

Introduction

slide-6
SLIDE 6

Introduction

Is direct placement and rendering of 3D

  • bjects suffjcient for realistic AR experiences?
slide-7
SLIDE 7

Introduction

Not always!

slide-8
SLIDE 8

Introduction

Viruual content looks like it’s “pasted on the screen” rather than “in the world”!

slide-9
SLIDE 9

Introduction

slide-10
SLIDE 10

Introduction

slide-11
SLIDE 11

Introduction

How can we bring these advanced features to mobile AR experiences without relying on dedicated sensors or the need for computationally expensive surgace reconstruction?

slide-12
SLIDE 12

Introduction

slide-13
SLIDE 13

Introduction

Google

  • Pixel 2, Pixel 2 XL, Pixel 3, Pixel 3 XL, Pixel 3a, Pixel 3a XL, Pixel 4, Pixel 4 XL

Huawei

  • Honor 10, Honor V20, Mate 20 Lite, Mate 20, Mate 20 X, Nova 3, Nova 4, P20, P30,

P30 Pro LG

  • G8X ThinQ, V35 ThinQ, V50S ThinQ, V60 ThinQ 5G

OnePlus

  • OnePlus 6, OnePlus 6T, OnePlus 7, OnePlus 7 Pro, OnePlus 7 Pro 5G, OnePlus 7T,

OnePlus 7T Pro Oppo

  • Reno Ace

Samsung

  • Galaxy A80, Galaxy Note8, Galaxy Note9, Galaxy Note10, Galaxy Note10 5G, Galaxy

Note10+, Galaxy Note10+ 5G, Galaxy S8, Galaxy S8+, Galaxy S9, Galaxy S9+, Galaxy S10e, Galaxy S10, Galaxy S10+, Galaxy S10 5G, Galaxy S20, Galaxy S20+ 5G, Galaxy S20 Ultra 5G Sony

  • Xperia XZ2, Xperia XZ2 Compact, Xperia XZ2 Premium, Xperia XZ3

Xiaomi

  • Pocophone F1

And growing…

https://developers.google.com/ar/discover/supported-devices

slide-14
SLIDE 14

Introduction

Is there more to realism than occlusion?

slide-15
SLIDE 15

Introduction

Surgace interaction?

slide-16
SLIDE 16

Introduction

Realistic Physics?

slide-17
SLIDE 17

Introduction

Path Planning?

slide-18
SLIDE 18
slide-19
SLIDE 19

Introduction

slide-20
SLIDE 20

Related Work

slide-21
SLIDE 21

Introduction

slide-22
SLIDE 22

Introduction

slide-23
SLIDE 23

Introduction

slide-24
SLIDE 24

Introduction

slide-25
SLIDE 25

Related Work

slide-26
SLIDE 26

Introduction

slide-27
SLIDE 27

Introduction

Up to 8 meters, with the best within 0.5m to 5m

slide-28
SLIDE 28

Motivation

slide-29
SLIDE 29

Introduction

ARCore Depth API DepthLab

Mobile AR developers

slide-30
SLIDE 30

Design Process

3 brainstorming sessions 18 participants 39 aggregated ideas

slide-31
SLIDE 31

Design Process

slide-32
SLIDE 32

System

slide-33
SLIDE 33

Data Structure

2D array (160x120 and above) of 16-bit integers

slide-34
SLIDE 34

Data Structure

slide-35
SLIDE 35

Data Structure

slide-36
SLIDE 36

System

slide-37
SLIDE 37

Localized Depth

slide-38
SLIDE 38

Localized Depth

slide-39
SLIDE 39

Localized Depth

slide-40
SLIDE 40

Localized Depth

slide-41
SLIDE 41

Localized Depth

slide-42
SLIDE 42

Localized Depth

slide-43
SLIDE 43

Surface Depth

slide-44
SLIDE 44

Surface Depth

Physics with depth mesh.

slide-45
SLIDE 45

Surface Depth

Texture decals with depth mesh.

slide-46
SLIDE 46

Surface Depth

Projection mapping with depth mesh.

slide-47
SLIDE 47

Dense Depth

slide-48
SLIDE 48

Dense Depth

θ N L

slide-49
SLIDE 49

Dense Depth

slide-50
SLIDE 50

Dense Depth

slide-51
SLIDE 51

Dense Depth

slide-52
SLIDE 52

Dense Depth

go/realtime-relighting, go/relit

slide-53
SLIDE 53

Dense Depth

slide-54
SLIDE 54

Dense Depth

slide-55
SLIDE 55

Experiments

slide-56
SLIDE 56

Experiments

slide-57
SLIDE 57

Experiments

slide-58
SLIDE 58

Experiments

slide-59
SLIDE 59

Discussion

slide-60
SLIDE 60

Discussion

slide-61
SLIDE 61

Discussion

slide-62
SLIDE 62

Limitations

Dynamic Depth? HoloDesk, HyperDepth, Digits, Holoportation for mobile AR?

slide-63
SLIDE 63

Envision

slide-64
SLIDE 64

GitHub

slide-65
SLIDE 65

Play Store

slide-66
SLIDE 66

DepthLab: Real-time 3D Interaction with Depth Maps for Mobile Augmented Reality

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarue, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, David Kim Google | ACM UIST 2020

slide-67
SLIDE 67

Thank you!

slide-68
SLIDE 68

Demo

slide-69
SLIDE 69

Introduction Introduction

slide-70
SLIDE 70

Introduction

slide-71
SLIDE 71

Thank you!

slide-72
SLIDE 72

Introduction

Occlusion is a critical component for AR realism! Correct occlusion helps ground content in reality, and makes viruual objects feel as if they are actually in your space.

slide-73
SLIDE 73

Introduction

slide-74
SLIDE 74

Depth Mesh

slide-75
SLIDE 75

Localized Depth

slide-76
SLIDE 76

Dense Depth

slide-77
SLIDE 77

Introduction

slide-78
SLIDE 78

Taxonomy

slide-79
SLIDE 79

Introduction

slide-80
SLIDE 80

Introduction