IEEE ComSoc Distinguished Lectures – Portland State University, Dec. 2019
Make Streaming Great Again
Ali C. Begen, PhD
http://ali.begen.net
Make Streaming Great Again IEEE ComSoc Distinguished Lectures - - PowerPoint PPT Presentation
Make Streaming Great Again IEEE ComSoc Distinguished Lectures Portland State University, Dec. 2019 Ali C. Begen, PhD http://ali.begen.net Ankara BS in EE Toronto, ON Istanbul San Jose, CA Adv. Res. and Dev. Prof. in CS Dept. Atlanta,
IEEE ComSoc Distinguished Lectures – Portland State University, Dec. 2019
Ali C. Begen, PhD
http://ali.begen.net
Istanbul
Ankara
BS in EE
Atlanta, GA
PhD in ECE w/ CS
San Jose, CA
Toronto, ON San Diego, CA
Intern
Konya
2200 guests
Konya
Born, raised and living
3
AMAAAS*
We will Scratch the Surface on the Red Colored Items
Capture/ Ingest Encoding Encryption Packaging Storage Analytics
Consumption Reliable and Scalable Delivery Production
CDNs Access Core
IEEE ComSoc Distinguished Lecture - Dec. 2019
AMAAAS: Ask me almost anything about streaming
Download vs. Streaming, Why HTTP, Manifests, etc.
One Request à One Response
Progressive Download
HTTP Request HTTP Response
IEEE ComSoc Distinguished Lecture - Dec. 2019 5
Playback starts only after there is several seconds of data in the playback buffer Download will continue as fast as possible Fetched content will be wasted if the viewer clicks away Can seek only throughout the fetched content
IEEE ComSoc Distinguished Lecture - Dec. 2019 6
What is Streaming?
Streaming Sounds Cooler!
Client consumption rate is also limited by real-time constraints as
That is, client cannot fetch content not available yet Server transmission rate (loosely or tightly) matches to client consumption rate That is, no buffer overrun or underrun is acceptable
Streaming is transmission of a continuous content from a server to a client and its simultaneous consumption by the client
Streaming is More Viewer Friendly
IEEE ComSoc Distinguished Lecture - Dec. 2019 7
Playback starts when there is just few seconds of data Download rate will match the encoding bitrate and downloading pauses if the player pauses à Less waste Can seek to anywhere in the entire content
Video Delivery over HTTP
playback while still downloading
file as fast as possible
Progressive Download
via media indexing
transmission based
Pseudo Streaming
into short-duration chunks
streaming and ad insertion
Chunked Streaming
created
to network and device conditions
Adaptive Streaming
IEEE ComSoc Distinguished Lecture - Dec. 2019 8
Adaptive Streaming over HTTP
Decoding and Presentation Streaming Client Media Buffer Content Ingest (Live or Pre-captured) Multi-rate Encoder Packager Origin (HTTP) Server … … … … Server Storage HTTP GET Request Response
IEEE ComSoc Distinguished Lecture - Dec. 2019 9
Adapt Video to Web rather than Changing the Web
HTTP Adaptive Streaming (HAS)
–
Features well-understood naming/addressing and authentication/authorization infrastructure
–
Provides easy traversal for all kinds of middleboxes (e.g., NATs, firewalls)
–
Enables cloud access, leverages the existing (cheap) HTTP caching infrastructure in the CDNs
–
Downloads small chunks to minimize waste
–
Enables monitoring consumption and tracking streaming clients
–
Enables shorter stream start time upon zapping or seeking
–
Reduces skips and freezes
IEEE ComSoc Distinguished Lecture - Dec. 2019 10
The cure may be worse than the disease if you are not careful enough
Dead, Surviving, Maturing and Newborn Technologies
–
http://www.movenetworks.com
–
http://www.iis.net/expand/SmoothStreaming
–
http://www.adobe.com/products/flashplayer.html
–
http://www.adobe.com/products/httpdynamicstreaming
–
https://tools.ietf.org/html/rfc8216
–
https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis
–
http://mpeg.chiariglione.org/standards/mpeg-dash
–
http://mpeg.chiariglione.org/standards/mpeg-a/common-media-application-format
IEEE ComSoc Distinguished Lecture - Dec. 2019 11
DASH MPD: Template Based and Extensible
Manifests: List of Accessible Segments and Their Timings
MPD Period id = 1 start = 0 s Period id = 3 start = 300 s Period id = 4 start = 850 s Period id = 2 start = 100 s Adaptation Set 0 subtitle turkish Adaptation Set 2 audio english Adaptation Set 1
BaseURL=http://abr.rocks.com/
Representation 2 Rate = 1 Mbps Representation 4 Rate = 3 Mbps Representation 1 Rate = 500 Kbps
Representation 3
Rate = 2 Mbps Resolution = 720p Segment Info Duration = 10 s Template: 3/$Number$.mp4
Segment Access
Initialization Segment
http://abr.rocks.com/3/0.mp4
Media Segment 1 start = 0 s
http://abr.rocks.com/3/1.mp4
Media Segment 2 start = 10 s
http://abr.rocks.com/3/2.mp4
Adaptation Set 3 audio italian Adaptation Set 1 video Period id = 2 start = 100 s
Representation 3 Rate = 2 Mbps
Selection of components/tracks Well-defined media format Selection of representations Splicing of arbitrary content like ads Chunks with addresses and timing
IEEE ComSoc Distinguished Lecture - Dec. 2019 12
IEEE ComSoc Distinguished Lecture - Dec. 2019 13
HLS Playlist: Text Based and Verbose
Manifests: List of Accessible Segments and Their Timings
#EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=232370,CODECS="mp4a.40.2, avc1.4d4015" gear1/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=649879,CODECS="mp4a.40.2, avc1.4d401e" gear2/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=41457,CODECS="mp4a.40.2" gear0/prog_index.m3u8
master.m3u8
Source: https://developer.apple.com/streaming/examples/ and https://www.gpac-licensing.com/2014/12/01/apple-hls-comparing-versions/
#EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-PLAYLIST-TYPE:VOD #EXTINF:9.97667, fileSequence0.ts #EXTINF:9.97667, fileSequence1.ts #EXTINF:9.97667, fileSequence2.ts . . . #EXT-X-ENDLIST
gear1/prog_index.m3u8
Example Representations
Encoding Bitrate Resolution
3.45 Mbps 1280 x 720
2.2 Mbps 960 x 540
1.4 Mbps 960 x 540
900 Kbps 512 x 288
600 Kbps 512 x 288
400 Kbps 340 x 192
200 Kbps 340 x 192
Source: Vertigo MIX10, Alex Zambelli’s Streaming Media Blog, Akamai, Comcast
Vancouver 2010 Sochi 2014
Encoding Bitrate Resolution
3.45 Mbps 1280 x 720
1.95 Mbps 848 x 480
1.25 Mbps 640 x 360
900 Kbps 512 x 288
600 Kbps 400 x 224
400 Kbps 312 x 176 Encoding Bitrate Resolution
18 Mbps 4K (60p)
12.2 Mbps 2560x1440 (60p)
4.7 Mbps 2K (60p)
3.5 Mbps 1280x720 (60p)
2 Mbps 1280 x 720
1.2 Mbps 768 x 432
750 Kbps 640 x 360
500 Kbps 512 x 288
300 Kbps 320 x 180
320 x 180
PyeongChang 2018
IEEE ComSoc Distinguished Lecture - Dec. 2019 14
Smart and Selfish Clients
HAS Working Principle
(HTTP/1.1/2 runs over TCP, HTTP/2/3 runs over QUIC)
protected content Client monitors and measures
Client performs adaptation
Request Response HTTP Server Client
Client measures and reports metrics for analytics
(One can also multicast media segments)
IEEE ComSoc Distinguished Lecture - Dec. 2019 15
Tradeoffs in Adaptive Streaming
User experience Overall quality Quality stability Stalls Zapping/seeking time Live latency
IEEE ComSoc Distinguished Lecture - Dec. 2019 16
Scaling Issues
A Single Microsoft Smooth Streaming Client under a Controlled Environment
Demystifying a Streaming Client
1 2 3 4 5 50 100 150 200 250 300 350 400 450 500 Bitrate (Mbps) Time (s) Available Bandwidth Requests Chunk Tput Average Tput
Reading: “An experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP,” ACM MMSys 2011
Buffer-filling State Back-to-back requests Steady State Periodic requests
IEEE ComSoc Distinguished Lecture - Dec. 2019 18
– Video is just ordinary Web content
– Congestion avoidance – Reliable transport – No need for special QoS for video
Does it just work?
Mostly yes, when streaming clients compete with other types of traffic Not really, when streaming clients compete with each
Streaming clients form an “accidental” distributed control-feedback system, causing all kinds of issues in
Streaming over HTTP – The Promise
IEEE ComSoc Distinguished Lecture - Dec. 2019 19
10 Streaming Clients Sharing a 10 Mbps Link
Selfishness Hurts Everyone
400 800 1200 1600 100 200 300 400 500 Requested Bitrate (Kbps) Time (s) Alice Bob Lisa John Jane
IEEE ComSoc Distinguished Lecture - Dec. 2019 20
Two Competing Clients
Understanding the Root Cause
– Unfairness, underutilization and/or instability may occur – Clients may grossly overestimate their fair share of the available bandwidth
Clients cannot figure out how much bandwidth to use until they use too much (Just like TCP)
Reading: “What happens when HTTP adaptive streaming players compete for bandwidth?,” ACM NOSSDAV 2012
IEEE ComSoc Distinguished Lecture - Dec. 2019 21
How to Solve the Issues?
Fix the clients and/or the transport
Get support from the network
Enable a control plane
IEEE ComSoc Distinguished Lecture - Dec. 2019 22
Review of over 100 Adaptation Schemes
IEEE ComSoc Distinguished Lecture - Dec. 2019 23
Looking for a Permanent head Damage (Degree)?
Bitrate Adaptation Schemes Client- based Adaptation Bandwidth- based Buffer- based Mixed adaptation Proprietary solutions MDP-based Server- based Adaptation Network- assisted Adaptation Hybrid Adaptation SDN-based Server and network- assisted
Reading: “A survey on bitrate adaptation schemes for streaming media over HTTP,” IEEE Commun. Surveys Tuts., 2019
Conflicting Goals in the Media Delivery Chain
Consumer Content Provider ISP CDN Provider Advertiser Device/ App
IEEE ComSoc Distinguished Lecture - Dec. 2019 24
A Control Plane Approach
IEEE ComSoc Distinguished Lecture - Dec. 2019 25
Origin (HTTP) Server Encoder/ Transcoder Packager CDN RG Clients
Media Parameters for enhancing reception (PER) Metrics and status messages Parameters for enhancing delivery (PED)
Analytics Server
Experience Metrics (almost done)
Media Client Data
Service Provider: “Your video or CDN provider must be slow” Video/CDN Provider: “Your home network must suck” Consumer: “The device
slow” Device/App Vendor: “It must be the OS” OS Vendor: “Your Internet connection must be bad”
IEEE ComSoc Distinguished Lecture - Dec. 2019 26
2066) across players
“Overabundance of information implies a scarcity of user attention”
Yes, at the Source, Encoder, Packager, Origin, Cache, GW, Player, etc.
Do We Need Analytics?
Linear Delivery under Three Seconds
Reading: “Bandwidth prediction in low-latency chunked streaming,” ACM NOSSDAV 2019
–
Buffering
–
Playhead positioning
–
Resilience
Contributors to the Latency
IEEE ComSoc Distinguished Lecture - Dec. 2019 28
IEEE ComSoc Distinguished Lecture - Dec. 2019 29
Low Latency is Always a Trade-Off against Playback Robustness
Stream Start Time ≠ Latency
Time Live encoder producing 2-second segments iOS (3 segments) Last fully available segment Lowest latency 1 2 3 4 Start Now 2 3 4 4 5 Latency: 7 s Latency: 3 s Latency: 2 s 6 seconds of buffer ~0 seconds of startup* 2 seconds of buffer ~0 seconds of startup* 2 seconds of buffer 1 second of startup*
* Segment fetching time is assumed to be negligible in this example
5
CMAF Chunks are One or More Frames
Refresher on CMAF
RAP
…
RAP
…
RAP
…
RAP
…
Fragment Fragment Fragment Fragment
Segment Segment
Track File
… … …
Encoding Packaging Encryption CMAF Header
Seamless switching can only happen at fragment boundaries
30 IEEE ComSoc Distinguished Lecture - Dec. 2019
Encoding and Streaming in a Quality-Aware Fashion
Reading: “Spending quality time with the Web video,” IEEE Internet Comput., 2016, and “Quality-aware HTTP adaptive streaming,” IBC 2015
Nomenclature of Rate Control
– Misnomer, refers to adaptive streaming over HTTP
– The decoder’s buffer is filled at a constant rate
– The decoder’s buffer is filled at a non-constant rate
– The decoder’s buffer is filled at a non-constant rate with strict min and max bounds
– Choosing the bitrate ladder based on the content
– Advanced optimizations based on viewer, display and viewing conditions
P a s t
CBR encoding with fixed bitrate ladders
Today
Slowly moving to cVBR and custom bitrate ladders Future
C
t e x t
w a r e e n c
i n g ?
IEEE ComSoc Distinguished Lecture - Dec. 2019 32
Picking the Bitrate Ladder Based on the Content
Content-Based (Content-Aware) Encoding
Content-aware encoding gives us fairness in quality as
IEEE ComSoc Distinguished Lecture - Dec. 2019 33
– Segments are ~CBR encoded – Client fetches segments based on bitrate
information only
because of
– Low-motion/complexity vs. high-
motion/complexity scenes
– Upshifts and downshifts dictated by the
adaptation logic
Segments Have Different Complexities
Bitrate Quality Video Segment #1 Equal Bitrate Allocation among Segments Consistent Quality Video Segment #2
IEEE ComSoc Distinguished Lecture - Dec. 2019 34
Guidelines Limited Bitrate Variability to (Mostly) 10% So Far
Adaptation Feature Delivers Inconsistent Quality
Easy Moderate Easy Easy Easy Difficult Difficult Difficult Moderate Moderate Moderate Moderate
…
S Time (s) Segment Size 0 2 4 6 8 10 12 14 16 18 20 22 24 Segment Quality QCBR Small variation in encoding bitrate Large variation in quality
If there is something worse than having to watch a video at a lousy quality, it is to watch that video with varying quality
IEEE ComSoc Distinguished Lecture - Dec. 2019 35
What If We Encode in a More Subtle Fashion?
Easy Moderate Easy Easy Easy Difficult Difficult Difficult Moderate Moderate Moderate Moderate
…
Time (s) Segment Size 2 4 6 8 10 12 14 16 18 20 22 24 Segment Quality QVBR
While we spend the same total amount of bits, we not only increase average quality but also reduce quality variation
Large variation in encoding bitrate Low variation in quality S
HLS authoring spec for ATV allows 2x capping rate for VoD. For linear content, variability is limited to 10-25% range.
IEEE ComSoc Distinguished Lecture - Dec. 2019 36
Content-aware Encoding Content-aware Streaming
IEEE ComSoc Distinguished Lecture - Dec. 2019 37
We can Still Save Bandwidth and/or Improve Quality
What If the Content is Already CBR Encoded
Reading: “Streaming video over HTTP with consistent quality,” ACM MMSys 2014
2.8 Mbps Network HTTP Server
…
k+1 k+2
4 Mbps
k+3 k k+1 k+2 k+3 k
…
3 Mbps
…
2 Mbps
…
1 Mbps
Representations (4 bitrate levels) Smart Clients Naive (Most) Clients A Bit Smarter (Few) Clients
… … …
IEEE ComSoc Distinguished Lecture - Dec. 2019 38
We can Still Save Bandwidth
What If There is No Smartness in the Client
3.2 Mbps Network Regular HTTP Server
…
k+1 k+2
4 Mbps
k+3 k k+1 k+2 k+3 k
…
3 Mbps
…
2 Mbps
…
1 Mbps
Representations (4 bitrate levels) Naive Clients 3.2 Mbps Network Quality-aware HTTP Server (or Packager) Naive Clients
… …
Reading: “More juice less bits: content aware streaming,” ACM MMSys 2016
Bandwidth Savings
The server/packager replaces some of the 3 Mbps segments with the 2 Mbps ones since delta quality is insignificant
IEEE ComSoc Distinguished Lecture - Dec. 2019 39
What If the Content is VBR Encoded
2.8 Mbps Network HTTP Server
k+1 k+2 k+3 k
…
QL_3
…
QL_2
…
QL_1
Representations (3 quality levels) Smart Clients
…
The client streams the highest consistent-quality video without draining its buffer while respecting the available bandwidth (Solved using dynamic programming) The resolution stays the same but the encoding rate varies in a given representation (per quality level)
IEEE ComSoc Distinguished Lecture - Dec. 2019 40
Dynamic Programming Solution
à Dynamic programming reduces processing time to polynomial time Time Buffer Size
B[0] BHIGH BLOW B[N] B[n]
IEEE ComSoc Distinguished Lecture - Dec. 2019 41
average quality (or total quality)
– Pick Q=1, then – Pick Q=4
case quality
– Pick Q=2, then – Pick Q=2
suboptimal
infeasible
A Toy Example
Buffer size (s) Increase in buffered data Decrease in buffered data Quality gain
Q
Steps Buffer (s) 1 1.5 0.5 1.9 0.9 0.8
1 2 Q = 1 Q = 2 Q = 2 Q = 4 Q = 2 Q = 4
IEEE ComSoc Distinguished Lecture - Dec. 2019 42
Reading: “Streaming video over HTTP with consistent quality,” ACM MMSys 2014
DASH Toolset for Content-Aware Streaming
– Set minBufferTime to allow for degree of VBRness – Provide the client information about effective segment bitrates and qualities
aux file signaled in the manifest (ISO/IEC 23009-8 in progress)
– Decouple switching and random access
– Estimate download times
– Take segment size and quality values into account in the adaptation logic
IEEE ComSoc Distinguished Lecture - Dec. 2019 43
– CBR encoders and naive streaming clients – Capped VBR encoders (small caps) and
naive clients
– Capped VBR encoders (small cap) and a
bit smarter clients
– Capped VBR encoders (large cap) and
smart clients
Deployment Scenarios
What If We Have Both CBR and VBR Encoded Content
If you do not control both ends, follow Postel’s Law (RFC 1122) Be liberal in what you accept, conservative in what you send If there are unknown clients:
large
Otherwise, the clients might get confused, break, stall or even crash
IEEE ComSoc Distinguished Lecture - Dec. 2019 44
– In-stream: Temporal bit shifting between segments – Across-streams: Bit shifting between streams sharing a bottleneck link
Extending the Idea to Optimization across Streams
Bitrate Quality Stream 1 (News) Equal Bitrate Allocation among Streams Consistent Quality Stream 2 (Sports)
Reading: “Spending quality time with the Web video,” IEEE Internet Comput., 2016, and “Quality-aware HTTP adaptive streaming,” IBC 2015
IEEE ComSoc Distinguished Lecture - Dec. 2019 45
Temporal Pooling Viewers react differently to glitches for different content types, and they forgive in different time scales
watching sports on a big screen vs.
watching news on a smaller screen Spatial Pooling We want controlled unfairness (which is fairness in quality not bitrate) among viewers
and your wife is watching a food show on a tablet
two others are watching news on a 48” screen
– Each asset is copied multiple times
– Huge cost for encoding/packaging/storage – Inefficiencies in CDN caching/distribution
– Lack of consistent app behavior – Varying video features, APIs and semantics
– Partial profile support – Switching glitches – Audio discontinuities – Ad splicing problems – Long-term playback instability – Request protocol deficiencies – Memory problems, CPU weaknesses – Scaling (display) issues – Variable HDR support – Unknown capabilities, …
– Piracy and restreaming – Account sharing, use of VPNs/proxies, …
Houston, We Have So Many Problems!
IEEE ComSoc Distinguished Lecture - Dec. 2019 46
IEEE ComSoc Distinguished Lecture - Dec. 2019 47
Recommendations
– Package in CMAF/DASH segments (fmp4 only, no TS) à Less packaging/storage capacity
– Use common encryption for multi-DRM support à Simpler packaging, less storage, flexibility to
switch DRM in case of a breach
– For iOS devices, create an HLS playlist (version 7+) – For everything else, create a DASH MPD
– Encrypt using the CBCS mode
– If Widevine and PlayReady do not support CBCS
–
Improving open-source HEVC encoding
–
Low-latency live streaming
–
Machine learning and statistical modeling for video streaming
–
Volumetric media: from capture to consumption
–
Fake media and tools for preventing illegal broadcasts
Important Dates Submit by Research Track
Demo Track
Open Source/Dataset
Workshops
Conference June 8-11
NEW
Visit http://acmmmsys.org today!
NEW
Visit ht http://ali li.begen.net fo for tutorials and papers Also AM AMAAAS AAAS al ali.begen@ozyegin.edu.tr