F LASH B ACK Immersive Virtual Reality on Mobile Devices via - - PowerPoint PPT Presentation

FLASHBACK

Immersive Virtual Reality 


• n Mobile Devices 


via Rendering Memoization

Kevin Boos David Chu Eduardo Cuervo

MobiSys 2016

2

3

high graphic complexity
 photo-realistic, wide FOV low latency
 < 25ms high framerate
 60+ FPS

mobile devices cannot 
 meet these demands

Tethered HMDs

Current VR Landscape

5

Responsiveness Mobility Affordability Energy Efficiency Graphical Quality

Te

Tethered HMDs

Current VR Landscape

6

Responsiveness Mobility Affordability Energy Efficiency Graphical Quality

Te

Responsiveness Mobility Affordability Energy Efficiency Graphical Quality

7

Mo

TeTethered HMDs

Mobile HMDs

Current VR Landscape

Responsiveness Mobility Affordability Energy Efficiency Graphical Quality

8

Te

Mo

Fl

Tethered HMDs Mobile HMDs FLASHBACK

Current VR Landscape

FLASHBACK objectives

9

mobility self-contained on mobile device graphic quality desktop-level image quality responsiveness low end-to-end latency energy efficiency long battery life, low thermal output affordability no specialized hardware

Design

Mobile GPUs are constrained Mobile storage is abundant

key idea: pre-render all possible views for fast real-time replay

FLASHBACK design & challenges

• Pre-rendering [offline]
• infinite input space

• Live playback [runtime]
• huge cache, fast retrieval
• inexact query matching 

• Dynamic objects

13

FLASHBACK design & challenges

• Pre-rendering [offline]
• infinite input space

• Live playback [runtime]
• huge cache, fast retrieval
• inexact query matching 

• Dynamic objects

14

Pre-rendering

15

pose

(key)

frame

(value)

: map pose to frame

infinite input space

Pre-rendering

16

input =

3D position 3D orientation

Pre-rendering: megaframes

17

RIGHT

LEFT EYE RIGHT EYE

LEFT TOP BOTTOM FRONT REAR

Pre-rendering: megaframes

18

position

(key)

megaframe

(value)

RIGHT

LEFT EYE RIGHT EYE

LEFT TOP BOTTOM FRONT REAR

Further reducing the input space

• Automate iteration over possible inputs
• Prune unreachable player positions 

• Configurable quantization granularity

19

Full-coverage frame cache is available at runtime

FLASHBACK design & challenges

• Pre-rendering [offline]
• infinite input space

• Live playback [runtime]
• huge cache, fast retrieval
• inexact query matching 

• Dynamic objects

20

GPU RAM (Flash/SSD)

frame cache

Live playback overview

21

HMD

pose

Cube Warp

final frame megaframe(s)

Building the cache

22

GPU RAM (Flash/SSD)

frame cache

fast retrieval from huge cache

L3: secondary storage – 9.2 ms L2: system RAM – 8.7ms L1: GPU VRAM – 0.35 ms

• raw, decoded megaframes

Spatially indexing the cache

23

• R-trees: fast, n-D

nearest-neighbor search algorithm


• Two cache indices:
• Universal
• GPU-only

inexact 
 query matching

FLASHBACK design & challenges

• Pre-rendering [offline]
• infinite input space

• Live playback [runtime]
• huge cache, fast retrieval
• inexact query matching 

• Dynamic objects

24

Pre-rendering dynamic objects

• Extension of static scene
• 7D input space:
• 3D relative position
• 3D object rotation
• 1D animation sequence

25

Automated megaframe capture

• Supports arbitrary motion


paths and animations

• but most are periodic

26

CK.orientation CK.pos

• Anim. List
• Anim. List

Top-level R-tree, position-indexed

Dynamic

CacheKey

Mid-level R-tree,

• rientation-indexed

Last-level animation list, timestamp-indexed

• Anim. List

Retrieved megaframe

• A 7D query is not meaningful
• Decompose into chain of queries


= Faster queries

• Can prune search space 


at each level

Dynamic object cache indexing

Dynamic + static compositing

27

sta$c& dynamic& composite& megaframe&

Evaluation

• HP Pavilion Mini + Oculus Rift DK2
• Small weak computer approximates mobile device
• Underperforms Samsung Galaxy S6 Gear VR

Evaluation setup

29 0.2 0.4 0.6 0.8 1 1.2 Manhattan T-Rex ALU Texturing

Benchmark Score (normalized)

HP Pavilion Mini Samsung Galaxy S6 Desktop

HP Pavilion Mini Samsung Galaxy S6 6 Desktop

30

FLASHBACK

31

Local Rendering

32

10 20 30 40 50 60 70 80 Mobile Device Strong Desktop FlashBack (Decoding) FlashBack (GPU)

Framerate (FPS)

Static 1 Dynamic Object 2 Dynamic Objects

15x higher framerate

50 100 150 200 250 Mobile HMD Strong Desktop FlashBack (Decoding) FlashBack (GPU)

End-to-End Latency (ms)

8x lower latency

97x more energy efficient

33

1 2 3 4 5 6 Strong Desktop Mobile Device FlashBack (Decoding) FlashBack (GPU)

Energy Per Displayed Frame (J)

Static 1 Dynamic Object 2 Dynamic Objects

longer battery life less thermal discomfort

FLASHBACK maintains image quality

• Measure perceived visual quality via SSIM
• Compares rendered scene against a pristine image

34

0.93 FLASHBACK 0.81 local rendering

• n mobile device

poor quality good quality

0.75 1.0 0.5

Limitations

• Dynamic object scalability is moderate
• con: per-pixel megaframe compositing is slow
• pro: object complexity is irrelevant 

• Lighting models are limited

• Restricted by hardware decoder

35

Related work

36

precaching

web search results database queries data types, behavior, and design choices 
 are not applicable to VR domain.

• ffloading

compute offload wearable AR on Glass rendering

• prelim. work on HMDs

requires good network connection.
 latency/quality less demanding.
 ignores local device storage.

caching objects as rendered images

QuickTime VR reuse past renderings caching with impostors static video playback only. focused on desktop environments. inaccuracies in object representation. very limited dynamic object support.

warping cubemaps

VR address recalculation requires specialized hardware 
 added to high-end GPUs.

FLASHBACK in conclusion

• Avoids real-time rendering by pre-generating frames
• flattens complex VR app behavior into data structures
• Supports static scene and dynamic animated objects

37

framerate ⬆ 
 latency ⬇ 
 energy ⬇

kevinaboos.web.rice.edu

38

Backup Slides

References

[3] D. Lymberopoulos, et al., “Pocketweb: Instant web browsing for mobile devices,” ASPLOS 2012. [4] D. Barbara, et al., “Sleepers and workaholics: Caching strategies in mobile environments,” SIGMOD 1994. [5] E. Cuervo, et al., “Maui: Making smartphones last longer with code offload,” MobiSys 2010. [6] B. Chun, et al., “Clonecloud: Elastic execution between mobile device and cloud,” EuroSys 2011.
 [7] M. Gordon, et al., “Comet: Code offload by migrating execution transparently,” OSDI 2012. [8] K. Ha, et al., “Towards wearable cognitive assistance,” MobiSys 2014. 
 [9] E. Cuervo, et al., “Kahawai: High-quality mobile gaming using gpu offload,” MobiSys 2015. [10] Y. Degtyarev, et al., “Demo: Irides, attaining quality, responsiveness, and mobility for VR HMDs,” MobiSys 2015. [11] S. Chen, “Quicktime VR: An image-based approach to virtual environment navigation,” SIGGRAPH 1995. [12] G. Schaufler, “Exploiting frame-to-frame coherence in a virtual reality system,” IEEE VR AIS 1996. [13] G. Schaufler and W. Sturzlinger, “A Three Dimensional Image Cache for Virtual Reality,” CG Forum 1996. [14] J. Shade, et al., “Hierarchical image caching for accelerated walkthroughs of complex environments,” SIGGRAPH 1996. [15] M. Regan and R. Pose, “Priority rendering with a virtual reality address recalculation pipeline,” SIGGRAPH 1994. [16] M. Regan and R. Pose,” An interactive graphics display architeture,” IEEE VR AIS 1993.

40

41

Guttman, A. "R-Trees: A Dynamic Index Structure for Spatial Searching". ACM SIGMOD ‘84.

Cache lookup with R-trees

• create minimally-overlapping

bounding boxes around 3D points


• characteristics fit our needs
• good insertion and deletion
• fast lookup is priority
• better querying semantics

VR system in a nutshell

• Head-Mounted Display (HMD)
• Smartphone-class hardware
• Internal sensors and external trackers
• Combine sensor readings → player pose
• 3D position
• 3D orientation

42

Microbenchmarks

43

2 4 6 8 10 12 14 16 18 20 100 1000 10000 100000 1e+06

Cache Query Time (µs) Cache Size (number of frames)

0.3 0.4 from GPU from RAM from Disk 3 6 9 12 15 18

Cache Retrieval Time (ms)

44

car interior 115 MB bedroom 730 MB living room 2.8 GB two-story house 8.7 GB basketball arena 29 GB Viking Village 54 GB can be compressed 
 using video codec for 
 efficient deployment

Typical cache sizes (uncompressed)