Level elUp: A Thin-cl cloud Appr pproac ach h to Game e Lives - - PowerPoint PPT Presentation

Level elUp: A Thin-cl cloud Appr pproac ach h to Game e Lives estrea eaming

Landon Cox (Microsoft Research) Lixiang Ao (UC San Diego)

Game Livestreaming

• Twitch
• Average >2m concurrent viewers and >90k concurrent channels
• >65m hours streamed and >1.5b hours watched per month
• Market size $40b, expected to grow 18% per year

Game Livestreaming

Cloud-based Video Transcoding Is Expensive

• $300+ to transcode 100 hours of video on Azure
• <$20 to livestream 100 hours of single-bitrate video on Wowza
• Reason: Video transcoding is resource demanding, usually requires

hardware accelerators in the cloud

Mobile Hardware Trends

GPU video codec ML

Mobile Hardware Trends

• As of May 2018, hundreds of millions devices equipped with powerful

GPU/hardware codec/ML accelerators

• Most smartphones are expected to have ML accelerators in few years
• Combined capabilities of hardware accelerators on mobile devices are

greater than the cloud

• The edge is ready to play a more central role in video livestreaming

LevelUp: A Thin-cloud Approach to Game Livestreaming

• Offload cloud-based transcoding by encoding multi-bitrate videos on

broadcasters’ smartphones

• In case of bandwidth constraints, viewers boost reduced-resolution

video quality with super-resolution using ML

• Adopt game-specific CNN models to improve quality

LevelUp Design

What if bandwidth is not enough?

LevelUp Design – bandwidth constrained

+

Single-image super resolution (SR)

• CNNs upscale resolution of reduced-resolution images
• Mobile ML accelerators enable fast NN computation
• A lightweight CNN model of 4 layers is sufficient for LevelUp
• Different games have visual features, require different models
• Models are trained offline, downloaded to viewers before streaming

Capture screen Resize to 3 resolutions

Broadcaster’s video pipeline

Send to HW encoder queue Encode segments Upload to server

Viewer’s video pipeline – without SR

Download video segment Decode Display

Viewer’s video pipeline – with SR

Download video segment Decode Separate grayscale, chroma SR Bicubic interpolate Merge, display

Evaluation

• Can super resolution improve the visual quality of reduced-resolution

game streams?

• Can broadcasters perform multi-bitrate encoding in realtime?
• Can viewers super-resolve reduced-resolution video streams in

realtime?

• What is LevelUp’s energy overhead?

Game stream super-resolution

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% CSGO Dota 2 Fallout 4 GTAV Rust Starcraft 2 Witcher 3

Percent change

PSNR SSIM VMAF 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% CSGO Dota 2 Fallout 4 GTAV Rust Starcraft 2 Witcher 3

Percent change

PSNR SSIM VMAF

480p + SR 270p + SR PSNR: per-pixel, SSIM: structural, VMAF[1]: ML model of human perception [1] https://github.com/Netflix/vmaf # of improvements with SR

Game stream super-resolution

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% CSGO Dota 2 Fallout 4 GTAV Rust Starcraft 2 Witcher 3

Percent change

PSNR SSIM VMAF 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% CSGO Dota 2 Fallout 4 GTAV Rust Starcraft 2 Witcher 3

Percent change

PSNR SSIM VMAF

VMAF improved 15-20% VMAF improved 61-88% [1] https://github.com/Netflix/vmaf 480p + SR 270p + SR

Game stream super-resolution

Interpolated, 480x270 LevelUp, 480x270

Game stream super-resolution

Interpolated, 480x270 LevelUp, 480x270

Quality vs. Bitrate

10 20 30 40 50 60 70 80 90 100 0.1 1 10 100 VMAF Bitrate (Mbs)

GTAV CSGO Starcraft Dota 2 Rust Witcher 3 Fallout 4 GTAV (SR) CSGO (SR) Starcraft (SR) Dota 2 (SR) Rust (SR) Witcher 3 (SR) Fallout 4 (SR)

Quality vs. Bitrate

10 20 30 40 50 60 70 80 90 100 0.1 1 10 100 VMAF Bitrate (Mbs)

GTAV CSGO Starcraft Dota 2 Rust Witcher 3 Fallout 4 GTAV (SR) CSGO (SR) Starcraft (SR) Dota 2 (SR) Rust (SR) Witcher 3 (SR) Fallout 4 (SR)

VMAF boost by LevelUp VMAF boost by LevelUp

Quality vs. Bitrate

10 20 30 40 50 60 70 80 90 100 0.1 1 10 100 VMAF Bitrate (Mbs)

GTAV CSGO Starcraft Dota 2 Rust Witcher 3 Fallout 4 GTAV (SR) CSGO (SR) Starcraft (SR) Dota 2 (SR) Rust (SR) Witcher 3 (SR) Fallout 4 (SR)

Low+SR vs. Medium Bitrate: 70% lower bandwidth, 3-20% VMAF loss Medium+SR vs. High Bitrate: 90% lower bandwidth, 10-20% VMAF loss

Super-resolution can greatly improve game stream quality w/o extra bandwidth consumption

Can broadcaster encode multi-bitrate streams?

• Encode 3 2-second segments (1080p, 480p, 270p) at the same time
• Test on iPhone 11 Pro, the iPhone Xs, iPhone 8, iPhone 7, and iPhone

6s

• All devices can encode in realtime
• Multi-bitrate encoding is feasible on modern smartphones

Can viewers super-resolve video?

0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2

iPhone 11 (A13) iPhone Xs (A12) iPhone 8 (A11) iPhone 7 (A10) iPhone 6s (A9) Seconds

30FPS-capable

Recent smartphones equipped with ML accelerators can super-resolve gaming streams in realtime.

Energy overhead

10 20 30 40 50 60 70 80 90 100 Low (iPhone Xs) Med (iPhone Xs) Low-SR (iPhone Xs) Low-SR (iPhone 11 Pro)

% charge remaining

} -5% after 30 min streaming

LevelUp has small energy overheads even with super-resolution enabled.

Conclusion

• Game livestreaming is expensive due to realtime transcoding
• LevelUp can greatly reduces game livestreaming costs by leveraging

smartphones for transcoding

• LevelUp uses super-resolution to boost quality for reduced-resolution

videos by up to 88%

• LevelUp can transcode and super-resolve game streams in realtime

using recent smartphone hardware accelerators

Thank you! Q & A

Landon Cox (Microsoft Research) landon.cox@microsoft.com Lixiang Ao (UC San Diego) liao@ucsd.edu