SVE: Distributed Video Processing at Facebook Scale Qi Huang - PowerPoint PPT Presentation

SVE: Distributed Video Processing at Facebook Scale Qi Huang Petchean Ang, Peter Knowles, Tomasz Nykiel, Iaroslav Tverdokhlib, Amit Yajurvedi, Paul Dapolito IV, Xifan Yan, Maxim Bykov, Chuen Liang, Mohit Talwar, Abhishek Mathur, Sachin Kulkarni, Matthew Burke, Wyatt Lloyd Facebook , University of Southern California, Cornell, Princeton

Video is growing across Facebook • FB: 500M users watch 100M hours video daily (Mar. 16) • Instagram: 250M daily active users for stories (Jun. 17) • All: many tens of millions of daily uploads, 3X NYE spike 01

Processing is diverse and demanding Processing Re-encoding Video Input Classification video Thumbnail Pt. 1 Pt. 2 Legacy System SVE Scaling Challenges Impact of Design 02

Legacy: upload video file to web server Client Web Server She is having so much fun with #MSQRD 03

Legacy: preserve original for reliability Original Client Web Server Storage She is having so much fun with #MSQRD 04

Legacy: process after upload completes Original Processing Client Web Server Storage She is having so much fun with #MSQRD 05

Legacy: encode w/ varying bitrates Original Processing Client Web Server Storage 720P She is having so much fun with #MSQRD 4Mbps 1080P 16Mbps 480P 1.5Mbps 06

Legacy: store encodings before sharing Original Final Processing Client Web Server Storage Storage 720P She is having so much fun with #MSQRD 4Mbps 1080P 16Mbps 480P 1.5Mbps 07

Sharing with adaptive streaming Final Client Web Server Storage 720p 480p FBCDN 08

Focus: pre-sharing pipeline Original Final Client Web Server Processing Storage Storage All steps from when a user starts an upload until a video is ready to be shared 09

Serial pipeline leads to slow processing Original Final Client Web Server Processing Storage Storage 10

Monolithic script slows development “Let’s experiment speech recognition, add a logic to extract audio and analysis” “Change color coding at “We need to change the thumbnail different time” Original Final Blob Client Web Server Processing generation logic for videos > x Storage Storage minutes to create scene-based scrubber preview” “We want to experiment AI-based “Pass-through for encodings to spend 10x CPU for 30% small and well- compression improvement on formatted videos” popular videos” 11

Challenges for video processing @ FB Speedy Users can share videos quickly Flexible Thousands of engineers can write pipelines for tens of apps Robust Handle faults and overload that is inevitable at scale 12

Our Streaming Video Engine (SVE) is speedy, flexible, and robust 13

Speedy: harness parallelism Users can share videos quickly • Overlap fault tolerance and processing • Overlap upload and processing • Parallel processing 14

Architectural changes for parallelism Original Final Client Web Server Processing Storage Storage 15

Architectural changes for parallelism Scheduler Worker Final Client Web Server Preprocessor Storage Worker Worker Original Storage 16

Overlap fault tolerance and processing Scheduler Worker Final Write-through Client Web Server Preprocessor Storage Cache Worker Worker Original Storage 17

Overlap upload and processing Scheduler Preprocessor Final Worker Client Web Server Storage Worker Split into segments Worker Original Storage 18

Overlap upload and processing Scheduler Preprocessor Final Worker Client Web Server Storage Worker ...upload in progress Worker Original Storage 19

Parallel processing w/ many workers Scheduler Preprocessor Final Worker 720P Encode Client Web Server Storage Worker 480P Encode ...upload in progress Worker Thumbnail Original Storage 20

Parallel processing w/ many workers Scheduler Preprocessor Final 720P Encode Client Web Server Storage 480P Encode ...upload in progress Thumbnail Original Storage 21

Parallel processing w/ many workers Scheduler Preprocessor Final 720P Encode Client Web Server Storage 480P Encode ...upload in progress Thumbnail Original Storage 22

Parallel processing w/ many workers Scheduler Preprocessor Final Worker Client Web Server Storage Worker Worker Original Storage 23

Three sources of parallelism Scheduler Preprocessor Final Worker Client Web Server Storage Worker Overlap fault tolerance and processing Overlap upload and processing Worker Original Parallel processing Storage 24

Results: 2.3x ~ 9.3x speedup 9.3 10 Relative speedup 6.1 3.7 3 2.3 0 < 3M 3M ~ 10M 10M ~ 100M 100M ~ 1G >1G Video size buckets 25

Results: 2.3x ~ 9.3x speedup 9.3 10 Overlap upload & processing Relative speedup 6.1 3.7 3 2.3 0 < 3M 3M ~ 10M 10M ~ 100M 100M ~ 1G >1G Video size buckets 26

Results: 2.3x ~ 9.3x speedup Parallel Processing 9.3 10 Relative speedup 6.1 3.7 3 2.3 0 < 3M 3M ~ 10M 10M ~ 100M 100M ~ 1G >1G Video size buckets 27

Challenges for video processing @ FB Speedy Users can share videos quickly 2.3x ~ 9.3x speedup Flexible Thousands of engineers can write pipelines for tens of apps Robust Handle faults and overload that is inevitable at scale 28

Flexible: build DAG framework Thousands of engineers can write pipelines for tens of apps • DAG of computation on the stream-of-tracks abstraction • Engineers write only sequential tasks in a familiar language • Dynamic DAG generation per video 29

DAG on stream-of-tracks abstraction $pipeline = Pipeline.build() Images $video_track=$pipeline>addTrack(IMG_TYPE) ->addTask() $audio_track=$pipeline>addTrack(AUD_TYPE) Sound ->addTask() Input video $meta_track=$pipeline>addTrack(META_TYPE) ->addTask() Metadata Track 30

DAG on stream-of-tracks abstraction Encode(HD) Encode(SD) $pipeline = Pipeline.build() Images Thumbnail $video_track=$pipeline>addTrack(IMG_TYPE) ->addTask(Encode(HD), Encode(SD), Thumb) ->addTask(Encode(HD, 10s), ->addTask() Encode(SD, 10s), Thumb(10s)) Encode(AAC) $audio_track=$pipeline>addTrack(AUD_TYPE) Sound ->addTask(Encode(AAC)) ->addTask() $meta_track=$pipeline>addTrack(META_TYPE) ->addTask(Analysis) ->addTask() Analysis Metadata Tasks Track 31

DAG on stream-of-tracks interface Cnt Segments Encode(HD, 10sec) Encode(HD) Cnt Segments Encode(SD, 10sec) Encode(SD) Combine Tracks Images Cnt Segments Thumbnail(10sec) Thumbnail Notification Encode(AAC) Sound Video Classification Analysis Metadata Tasks Sync Point Tasks Track 32

Dynamic DAG Generation DAG Structure Preprocessor Scheduler $pipeline = Pipeline.build() DAG $video_track=$pipeline>addTrack(IMG_TYPE) ->addTask() Web Server Generation $audio_track=$pipeline>addTrack(AUD_TYPE) ->addTask() Worker Code $meta_track=$pipeline>addTrack(META_TYPE) ->addTask() Worker Worker Worker Cache 33

Dynamic DAG Generation DAG Structure Preprocessor Scheduler DAG Web Server Generation Encode(HD) Code Encode(SD) Encode(AAC) Analysis Cache 34

Dynamic DAG Generation DAG Structure Preprocessor Scheduler $pipeline = Pipeline.build() DAG $video_track=$pipeline>addTrack(IMG_TYPE) ->addTask() Web Server Generation $audio_track=$pipeline>addTrack(AUD_TYPE) ->addTask() Worker Code $meta_track=$pipeline>addTrack(META_TYPE) ->addTask() Worker Worker Worker Cache 35

One system for 15+ applications • Generate billions of tasks per day • Varying DAG size • 360 video has thousands of tasks per upload • Newsfeed post averages at 153 tasks per upload • Instagram averages at 22 tasks per upload • Messenger averages at 18 tasks per upload 36

Challenges for video processing @ FB Speedy 2.3x ~ 9.3x speedup Flexible One system for 15+ applications Thousands of engineers can write pipelines for tens of apps Robust Handle faults and overload that is inevitable at scale 37

Robust: tolerate overload Handle faults and overload that is inevitable at scale • Rely on priority to degrade non-latency-sensitive tasks • Defer full video processing for some new uploads • Load-shedding across global deployments 38

3X peak load during New Year Eve NYE Xmas Upload volume 3X Date 39

Prepare for overload Scheduler Worker Worker Final Blob Client Web Server Preprocessor Preprocessor Storage Worker Worker Worker Worker Original Storage 40

Use priority for worker overload Worker Worker Worker Worker Worker Worker Scheduler Hi-priority queue Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Low-priority queue Worker Worker Worker Worker Worker Worker Only assign hi-pri tasks Worker Worker Worker Worker Worker Worker under overload 41

Defer full video processing Preprocessor Scheduler DAG Web Server Generation Code Hi-priority queue Original Storage Cache 42

Regional redirection Scheduler Worker Web Server Preprocessor Scheduler Worker Traffic: Worker Traffic: Local distribution → 70% Local distribution → 100% Worker Remote distribution → 30% Preprocessor Worker Worker 43