SLIDE 1
Rouslan Dimitrov, NVIDIA 3/19/2019
2
S9156: STREAMING VR CONTENT FROM THE CLOUD Rouslan Dimitrov, NVIDIA - - PowerPoint PPT Presentation
S9156: STREAMING VR CONTENT FROM THE CLOUD Rouslan Dimitrov, NVIDIA - - PowerPoint PPT Presentation
S9156: STREAMING VR CONTENT FROM THE CLOUD Rouslan Dimitrov, NVIDIA 3/19/2019 Workstation Graphics For Low-Power Devices 2 VR STREAMING TO ANY DEVICE The Projects Vision Content Cloud Servers Internet Clients 3 ARCHITECTURE Cloud
SLIDE 2
SLIDE 3
3
VR STREAMING TO ANY DEVICE
Cloud Servers Content Internet Clients The Project’s Vision
SLIDE 4
4
ARCHITECTURE
Cloud Server to HMD
OpenVR Runtime VR App Virtual HMD Driver GeForce Now Late & Lens Warp VR Runtime Video Decoder GeForce Now HMD Video Encoder Virtual Audio Driver Audio Encoder Audio Decoder Internet Driver App
SLIDE 5
5
ARCHITECTURE
WIN SERVER CLIENT
INSTALLER
NVSS NVSC Audio/Video Data (sensors, inputs, QoS) Encode RGB Frame
Video SDK GFN Stack
Display
VR App
Decode Video Params (e.g. ROI, optional) HEVC Frame HEVC Frame VR Runtime Controller & inputs Sensor, Pose & Controller Data RGB Frame Sensor & input data Sensor & input data Audio
(e.g., OpenVR)
App Client app Server driver
Time Warp Lens Warp
RGB Frame Driver API
VR Runtime
Runtime API OpenVR Interfaces
HMD
Audio Playback Audio Capture / Encode
SLIDE 6
6
QOS
Relevant Internet problems:
- Jitter
- Missed packets
- Bandwidth variation
Reliability & Resilience
SLIDE 7
7
QOS
Need to dynamically adjust:
- Jitter absorption queue size (for jitter)
- Forward error correction strength (for dropped packets)
- Video compression rate (for bandwidth variation)
SLIDE 8
8
WIN SERVER
VR App
Video Params (e.g. ROI, optional)
App
Driver API VR Runtime Runtime API
ARCHITECTURE
INSTALLER
NVSS NVSC Audio / Video / Data (sensors, inputs, QoS) Encode RGB Frame
Video SDK GFN Stack
Decode HEVC Frame Sensor, Pose & Controller data Sensor & input data Audio
Client app Server driver
Audio Capture / Encode Audio Playback Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller & inputs
(e.g., OpenVR) Time Warp Lens Warp HMD
OpenVR Interfaces
SLIDE 9
9
LATENCY
Using:
- HelloVR sample modified to change color on button press
- Latency meter (instrumented switch + photodiode)
- Local Cloud with Client on WiFi (average ping time 4 ms)
Button to Photons
SLIDE 10
10
LATENCY
Results: Server Driver + Client App latency = 12.1 ms. The Jitter FIFO can be downsized at the cost of occasional missed frames.
Button to Photons
Latency [ms] Stdev Local VR 38.3 3.4 CloudVR 81.6 3.3 Diff 43.3
- 0.1
Jitter FIFO + Ping 31.2 Diff – Above 12.1
*Average of 40 clicks
SLIDE 11
11
WIN SERVER
VR App
Video Params (e.g. ROI, optional)
App
Driver API VR Runtime Runtime API
LATENCY
INSTALLER
NVSS NVSC Audio / Video / Data (sensors, inputs, QoS) Encode RGB Frame
GFN Stack
Decode HEVC Frame RGB Frame Sensor & input data Audio
Client app Server driver
Audio Capture / Encode Audio Playback Frame Pacing Jitter FIFO CLIENT Display VR Runtime Controller & inputs
(e.g., OpenVR) Time Warp Lens Warp HMD
3-10 ms 10-50 ms (depending on network) 1 ms 1-3 ms 3-10 ms
OpenVR Interfaces < 0.5 ms
SLIDE 12
12
DELIVERED FRAMES
No Motion, 1440x1600x2 @ 50 Mbps
SLIDE 13
13
DELIVERED FRAMES
No Motion, 1440x1600x2 @ 10 Mbps
SLIDE 14
14
SLIDE 15
15
DELIVERED FRAMES
Fast Motion, 1440x1600x2 @ 50 Mbps
SLIDE 16
16
DELIVERED FRAMES
Fast Motion, 1440x1600x2 @ 10 Mbps
SLIDE 17
17
VR VILLAGE
Explore the VR Village to get hands-on with the latest advances in virtual reality (including Cloud VR Streaming)
VR THEATER
Go to the VR Theater to see and experience narrated VR demos built by our partners
VR PARTNERS
Explore a great lineup of VR partners around the VR Village showcasing their groundbreaking technology
COME EXPLORE ALL THINGS VR AT GTC 2019
VR VILLAGE HOURS Tuesday: 12:00pm - 7:00pm Wednesday: 12:00pm - 7:00pm Thursday: 11:00am - 2:00pm
See More VR on the Exhibition Floor Expo Hall 3, Concourse Level
SLIDE 18