Salsify: Low-Latency Network Video Through Tighter Integration - - PowerPoint PPT Presentation

▶
salsify low latency network video through tighter
SMART_READER_LITE
LIVE PREVIEW

Salsify: Low-Latency Network Video Through Tighter Integration - - PowerPoint PPT Presentation

Mar 19, 2024 29 likes •563 views

Salsify: Low-Latency Network Video Through Tighter Integration Between a Video Codec and a Transport Protocol Sadjad Fouladi, John Emmons, Emre Orbay, Catherine Wu + , Riad S. Wahby, Keith Winstein Stanford University, + Saratoga High School

slide-1
SLIDE 1

Salsify: Low-Latency Network Video Through Tighter Integration Between a Video Codec and a Transport Protocol

Sadjad Fouladi, John Emmons, Emre Orbay, Catherine Wu+, Riad S. Wahby, Keith Winstein

Stanford University, +Saratoga High School

https://snr.stanford.edu/salsify

slide-2
SLIDE 2

Outline

  • Introduction
  • Salsify's New Architecture
  • Measurement Testbed
  • Evaluation
  • Conclusions

2

slide-3
SLIDE 3

NSDI’18

AT&T-MOBILE

Teleoperation of Robots and Vehicles Remote Surgery Cloud Video Gaming

slide-4
SLIDE 4

Video Conferencing

slide-5
SLIDE 5

Video Conferencing (reality)

slide-6
SLIDE 6

WebRTC (Chrome 65)

slide-7
SLIDE 7

Current systems do not react fast enough to network variations, end up congesting the network, causing stalls and glitches.

slide-8
SLIDE 8

Enter Salsify

8

  • Salsify is a new architecture for real-time Internet video.
  • Salsify tightly integrates a video-aware transport protocol, with a

functional video codec, allowing it to respond quickly to changing network conditions.

  • Salsify achieves 4.6⨉ lower p95-delay and 2.1 dB SSIM higher visual

quality on average when compared with FaceTime, Hangouts, Skype, and WebRTC.

slide-9
SLIDE 9

Outline

  • Introduction
  • Salsify’s New Architecture
  • Measurement Testbed
  • Evaluation
  • Conclusions

9

slide-10
SLIDE 10

video codec transport protocol

Today's systems combine two (loosely-coupled) components

10

slide-11
SLIDE 11

Two distinct modules, two separate control loops

11

target bit rate video codec transport protocol

300 packets/s 24 frames/s

compressed frames

slide-12
SLIDE 12

Shortcomings of the conventional design

  • The codec can only achieve the bit rate on average.
  • Individual frames can still congest the network.
  • The resulting system is slow to react to network variations.

12

slide-13
SLIDE 13

Salsify explores a more tightly-integrated design

13

transport protocol & video codec

slide-14
SLIDE 14

Brand-new architecture based on components we know and love!

  • Individual component of Salsify are not exactly new:
  • The transport protocol is inspired by “packet pair” and “Sprout-EWMA”.
  • The video format, VP8, was finalized in 2008.
  • The functional video codec was described at NSDI’17.
  • Salsify is a new architecture for real-time video that integrates these

components in a way that responds quickly to network variations.

14

slide-15
SLIDE 15

Salsify’s architecture:

Video-aware transport protocol

15

transport protocol & video codec

slide-16
SLIDE 16
  • There’s no notion of bit rate, only the next frame size!
  • Transport uses packet inter-arrival time, reported by the receiver.

Video-aware transport protocol

16

“What should be the size of the next frame?”

* without causing excessive delay

slide-17
SLIDE 17
  • Pauses between frames give

the receiver a “pessimistic” view of the network.

  • Receiver treats each frame
  • f the video as a separate

packet train.

The sender does not transmit continuously

17

Receiver

t₁ t₂ t₃ t₄ t₅

grace period

frame i frame i+1 Sender

slide-18
SLIDE 18

Salsify’s architecture:

Functional video codec

18

transport protocol & video codec

slide-19
SLIDE 19

Transport tells us how big the next frame should be, but...

It’s challenging for any codec to choose the appropriate
 quality settings upfront to meet a target size—they tend to

  • ver-/undershoot the target.

19

slide-20
SLIDE 20

How to get an accurate frame out of an inaccurate codec

  • Trial and error: Encode with different quality settings, pick the one that fits.
  • Not possible with existing codecs.

20

slide-21
SLIDE 21

frame frame frame frame

After encoding a frame, the encoder goes through a state transition that is impossible to undo

21

slide-22
SLIDE 22

There’s no way to undo an encoded frame in current codecs

22

encode(🏟,🏟,...) → frames...

The state is internal to the encoder—no way to save/restore the state.

slide-23
SLIDE 23

Functional video codec to the rescue

encode(state, 🏟) → state′, frame

23

Salsify’s functional video codec exposes the state that can be saved/restored.

slide-24
SLIDE 24

Order two, pick the one that fits!

  • Salsify’s functional video codec can explore different execution paths

without committing to them.

  • For each frame, codec presents the transport with three options:

A slightly-higher-quality version, A slightly-lower-quality version, Discarding the frame.

24

b e t t e r w

  • r

s e

5 K B 1 K B

slide-25
SLIDE 25

Salsify’s architecture:

Unified control loop

25

transport protocol & video codec

slide-26
SLIDE 26

Codec → Transport
 “Here’s two versions of the current frame.”

26

b e t t e r w

  • r

s e

5 K B 2 5 K B

30 KB

target frame size

slide-27
SLIDE 27

Transport → Codec
 “I picked option 2. Base the next frame on its exiting state.”

27

2 5 K B

30 KB

target frame size

slide-28
SLIDE 28

Codec → Transport
 “Here’s two versions of the latest frame.”

28

b e t t e r w

  • r

s e

5 K B 2 5 K B

55 KB

target frame size

slide-29
SLIDE 29

Transport → Codec
 “I picked option 1. Base the next frame on its exiting state.”

29

5 K B

55 KB

target frame size

slide-30
SLIDE 30

Codec → Transport
 “Here’s two versions of the latest frame.”

30

b e t t e r w

  • r

s e

7 K B 2 5 K B 5 K B

5 KB

target frame size

slide-31
SLIDE 31

Transport → Codec
 “I cannot send any frames right now. Sorry, but discard them.”

31

5 KB

target frame size

slide-32
SLIDE 32

Codec → Transport
 “Fine. Here’s two versions of the latest frame.”

32

better worse

45 KB 20 KB

50 KB

target frame size

slide-33
SLIDE 33

Transport → Codec
 “I picked option 1. Base the next frame on its exiting state.”

33

50 KB

45 KB

target frame size

slide-34
SLIDE 34

There’s no notion of frame rate or bit rate in the system.
 Frames are sent when the network can accommodate them.

slide-35
SLIDE 35

Outline

  • Introduction
  • Salsify's New Architecture
  • Measurement Testbed
  • Evaluation
  • Conclusions

35

slide-36
SLIDE 36

Goals for the measurement testbed

  • A system with


reproducible input video and
 reproducible network traces that runs
 unmodified version of the system-under-test.

  • Target QoE metrics: per-frame quality and delay.

36

slide-37
SLIDE 37

barcoded video video in/out (HDMI) HDMI to USB camera emulated network receiver HDMI output

slide-38
SLIDE 38

Sent Image Timestamp: T+0.000s Received Image Timestamp: T+0.765s Quality: 9.76 dB SSIM

slide-39
SLIDE 39

Outline

  • Introduction
  • Salsify's New Architecture
  • Measurement Testbed
  • Evaluation
  • Conclusions

39

slide-40
SLIDE 40

Evaluation results: Verizon LTE Trace

40

8 10 12 14 16 18 500 700 1000 2000 5000 7000

Video Quality (SSIM dB) Video Delay (95th percentile ms) WebRTC (VP9-SVC) Skype FaceTime Hangouts WebRTC

B e t t e r

slide-41
SLIDE 41

Evaluation results: Verizon LTE Trace

41

8 10 12 14 16 18 500 700 1000 2000 5000 7000

Video Quality (SSIM dB) Video Delay (95th percentile ms) WebRTC (VP9-SVC) Skype FaceTime Hangouts WebRTC Status Quo

(conventional transport and codec)

slide-42
SLIDE 42

Evaluation results: Verizon LTE Trace

42

8 10 12 14 16 18 500 700 1000 2000 5000 7000

Video Quality (SSIM dB) Video Delay (95th percentile ms) WebRTC (VP9-SVC) Skype FaceTime Hangouts WebRTC Status Quo

(conventional transport and codec)

Salsify (conventional codec)

slide-43
SLIDE 43

Evaluation results: Verizon LTE Trace

43

8 10 12 14 16 18 500 700 1000 2000 5000 7000

Video Quality (SSIM dB) Video Delay (95th percentile ms) Salsify WebRTC (VP9-SVC) Skype FaceTime Hangouts WebRTC Status Quo

(conventional transport and codec)

Salsify (conventional codec)

slide-44
SLIDE 44

Evaluation results: AT&T LTE Trace

44

8 9 10 11 12 13 14 15 16 200 300 500 700 1000 2000 5000

Video Quality (SSIM dB) Video Delay (95th percentile ms) WebRTC (VP9-SVC) Skype FaceTime Hangouts Salsify WebRTC

Better

slide-45
SLIDE 45

Evaluation results: T-Mobile UMTS Trace

45

9 10 11 12 13 14 3500 5000 7000 10000 14000 18000

Video Quality (SSIM dB) Video Delay (95th percentile ms) WebRTC (VP9-SVC) Skype FaceTime Hangouts Salsify WebRTC

Better

slide-46
SLIDE 46

Evaluation results: Emulated Wi-Fi (no variations, only loss)

46

7 8 9 10 11 12 300 500 700 1000 2000 5000 15000

Video Quality (SSIM dB) Video Delay (95th percentile ms) WebRTC (VP9-SVC) Skype Hangouts Salsify FaceTime WebRTC

Better

slide-47
SLIDE 47

WebRTC (Google Chrome 65.0 dev) Salsify

Check out the demo videos at https://snr.stanford.edu/salsify

slide-48
SLIDE 48

Outline

  • Introduction
  • Salsify's New Architecture
  • Measurement Testbed
  • Evaluation
  • Conclusions

48

slide-49
SLIDE 49

Codecs have been treated as black boxes in video systems for a long time.

slide-50
SLIDE 50

50

  • NSDI’17: ExCamera
  • Using the functional codec to do massively-parallel video compression on

AWS Lambda.

  • NSDI’18: Salsify
  • Using the functional codec to compress frames to the right size, at the right

time.

  • Same interface, two different applications.

New systems have emerged from this functional interface

slide-51
SLIDE 51

We encourage the codec designer and implementors to include save/restore state in the codecs—even if it’s large or opaque.

slide-52
SLIDE 52

Improvements to video codecs may have reached the point of diminishing returns, but changes to the architecture of video systems can still yield significant benefits.

slide-53
SLIDE 53

Takeaways

53

  • Salsify is a new architecture for real-time Internet video.
  • Salsify tightly integrates a video-aware transport protocol, with a

functional video codec, allowing it to respond quickly to changing network conditions.

  • Salsify achieves 4.6⨉ lower p95-delay and 2.1 dB SSIM higher visual quality
  • n average when compared with FaceTime, Hangouts, Skype, and WebRTC.
  • The code is open-source, and the paper and raw data are open-access:


https://snr.stanford.edu/salsify

Thank you: NSF, DARPA, Google, Dropbox, VMware, Huawei, Facebook, Stanford Platform Lab.