LIVE STREAMING AT SCALE Jordi Cenzano | Director of engineering - - PowerPoint PPT Presentation

▶
live streaming at scale
SMART_READER_LITE
LIVE PREVIEW

LIVE STREAMING AT SCALE Jordi Cenzano | Director of engineering - - PowerPoint PPT Presentation

Apr 20, 2023 186 likes •520 views

LIVE STREAMING AT SCALE Jordi Cenzano | Director of engineering mmsys2019 jcenzano@brightcove.com 2019/06/19 jordicenzano Agenda Simple live streaming Real (complex) live streaming blocks Some challenges Ingest

slide-1
SLIDE 1

LIVE STREAMING AT SCALE

Jordi Cenzano | Director of engineering

jcenzano@brightcove.com jordicenzano

mmsys2019 2019/06/19

slide-2
SLIDE 2

Agenda

  • Simple live streaming
  • Real (complex) live streaming blocks
  • Some challenges
  • Ingest
  • Personalized live streams (SSAI)
  • Monitoring
  • Latency
  • Redundancy
  • Future
  • Conclusions
slide-3
SLIDE 3

WHAT IS LIVE STREAMING?

slide-4
SLIDE 4

Simple live streaming

slide-5
SLIDE 5

Simple live streaming demo

WebServer (nodeJS): https://gist.github.com/jordicenzano/a943bda656ad1e0a0568fbaad311af05 RTMP + transcode + HLS (ffmpeg): https://gist.github.com/jordicenzano/c5516b5854eb046bd1c27076fe4ba917

#!/usr/bin/env bash ffmpeg -hide_banner \

  • listen 1 -i "rtmp://0.0.0.0:1935/live/stream" \
  • c:v libx264 -b:v 900k -g 60 -profile:v baseline -preset veryfast \
  • c:a aac -b:a 48k \
  • hls_flags delete_segments -hls_time 6 -hls_segment_filename ~/480p_%05d.ts ~/480p.m3u8

#!/usr/bin/env bash http-server ~ -p 8080 --cors -c -1

slide-6
SLIDE 6

WHAT IS REALLY LIVE STREAMING?

slide-7
SLIDE 7

Live streaming blocks

RTMP TS + ?

(h264+AAC)

h264 AAC Nxh264 NxAAC TS fMP4 Txt XML HLS DASH HTTP[s] HLS DASH HTTP[s]

slide-8
SLIDE 8

LIVE INGEST

slide-9
SLIDE 9

Live ingest

From: https://en.wikipedia.org/wiki/Internet_service_provider
  • Data loss
  • Latency
  • Jitter
  • Reordering
  • Corruption
  • Duplication

S L A

slide-10
SLIDE 10

Live ingest options: RTMP

  • Pros
  • Reliable (TCP based)
  • Popular
  • Cons (for live streaming)
  • Only 1 video + 1 audio + data
  • Connection oriented
  • Retries consume BW
  • Slow packet loss recovery
  • No real time guaranteed
slide-11
SLIDE 11

Live ingest options

  • Pros
  • Can transport everything
  • Not connection oriented
  • Designed for live
  • Very common in the broadcast world
  • Cons
  • Not reliable
  • Complexity
slide-12
SLIDE 12

Live ingest RTMP vs SRT

Source code: https://github.com/jordicenzano/live-streaming-ingest-advance d-simulation
  • Comparison RTMP vs SRT
  • 5 % packet loss
  • 50ms delay
  • 11 Mbps BW
  • Test stream
  • 8Mbps VBR (9Mbps max)
  • h264 1 keyframe every 3s
slide-13
SLIDE 13

PERSONALIZATION

(SSAI: Server Side ad insertion)

slide-14
SLIDE 14

Live streaming Live SSAI

slide-15
SLIDE 15
  • Considerations
  • One manifest per viewer = 1:1
  • Same for content
  • Different for adbreaks
  • Track every viewer in a “DB”

Live stream personalization

1M CCU HLS 6s TD 2.5 Tbps(@2.5Mbps) 333K rps

(1 m3u8 1 redir ts/fmp4)

166K r-w/s

  • Ultimate solution: Edge?
slide-16
SLIDE 16

Live streaming SSAI

SCTE 35 SCTE35 info Session info Job info Ad-123c.mp4 ad-123c-0001.ts

slide-17
SLIDE 17

Live streaming SSAI

$

X

... #EXTINF:8.0, A-0000.ts #EXTINF:6.40, A-0001.ts #AD-BREAK #EXTINF:8.0, A-0002.ts #EXTINF:4.0, A-0003.ts ... ... #EXTINF:8.0, A-0000.ts #EXTINF:6.40, A-0001.ts #EXT-X-DISCONTINUITY #EXTINF:8.0, ad-123c-0001.ts #EXT-X-DISCONTINUITY #EXTINF:4.0, A-0003.ts ...

slide-18
SLIDE 18

Live streaming SSAI impressions

1M CCU 6 ads / adbreak 20 imp / ad 120 M imp / adbreak

slide-19
SLIDE 19

MONITORING

slide-20
SLIDE 20
  • Considerations
  • Measure all you can
  • Impact of logging / metrics
  • All real time
  • Metrics
  • Logs
  • Aggregate and alarm wisely
  • Automate

Monitoring

slide-21
SLIDE 21

LATENCY Latency ≠ BAD

slide-22
SLIDE 22

HLS Latency sources

slide-23
SLIDE 23
  • LHLS (periscope approach)
  • Chunk transfer + advanced segments

Low latency “simple” demo

Source code: https://github.com/jordicenzano/go-ts-segmenter https://github.com/mjneil/go-chunked-streaming-server (From Matthew Neil)
slide-24
SLIDE 24

REDUNDANCY

slide-25
SLIDE 25

Redundancy blocks

slide-26
SLIDE 26

Redundancy blocks

#EXTM3U #Example of HLS REDUNDANT STREAMS #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000, RESOLUTION=720x480 http://ALPHA.mycompany.com/lo/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000, RESOLUTION=720x480 http://BETA.mycompany.com/lo/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000, RESOLUTION=1920x1080 http://ALPHA.mycompany.com/md/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000, RESOLUTION=1920x1080 http://BETA.mycompany.com/md/prog_index.m3u8

slide-27
SLIDE 27

Redundancy blocks: Full sync approach

slide-28
SLIDE 28

Redundancy blocks: non-sync approach

... #EXTINF:4.000000, from-A-0000.ts #EXTINF:4.000000, from-A-0001.ts #EXT-X-DISCONTINUITY #EXTINF:4.000000, from-B-0001.ts ....

A B

slide-29
SLIDE 29

Future

  • Do more at the edge with multi CDN
  • Personalized streams
  • Cohorts approach (hotstar 2019)
  • Low latency deployments
  • ABR
  • Codecs
  • More, faster, deeper pixels
  • Context aware encoding
slide-30
SLIDE 30

Conclusions: Experience based

  • Everything fails, and we are live
  • Plan B, plan C, ,,,, plan Z
  • Better hot - hot
  • Be sure the involved 3rd party are ready
  • CDNs, AdServers
  • Load test (not easy)
  • There are limits everywhere
  • Individually
  • End to end
  • Enjoy the challenge
slide-31
SLIDE 31

Q & A

slide-32
SLIDE 32

Thanks!

Jordi Cenzano jcenzano@brightcove.com jordicenzano