Impact of Duration on Active Video testing Saba Ahsan - PowerPoint PPT Presentation

Impact of Duration on Active Video testing ¡ ¡ Saba ¡Ahsan ¡ ¡ Department ¡of ¡Communica5ons ¡and ¡Networking ¡ Aalto ¡University ¡ ¡ Varun ¡Singh ¡(callstat.io), ¡Jörg ¡OG ¡( Technische Universität München ) ¡

2

Motivation • Actively measure performance of Internet video (YouTube) from a user’s perspective • Finish the test as quickly as possible but still get reliable results

Instantaneous video bit rate of Full HD (1080p) YouTube videos Talkshow Trailer 6 4 Mbps 2 0 0 50 100 150 0 50 100 150 Time(s)

Problem Statement • When conducting active video tests in a time constrained environment, what is the minimum that the test should run for … – Can we represent the bit rate magnitude and variations in Internet video with a short clip? 5

Dataset • Popular YouTube videos by geographical area • Mp4 videos (frame sizes and timing) – Non-adaptive 720p > 32k – Non-adaptive 360p > 62k – DASH 1080p > 8k – DASH 720p > 19k 6

Selecting a cut-off value Clip − S2 2.0 1.5 1.0 0.5 Mbps 0 20 40 60 Full − S1 2.0 1.5 1.0 0.5 0.0 0 100 200 300 Time(s) 7

Selecting a cut-off value Clip − S2 2.0 1.5 1.0 0.5 Mbps 0 20 40 60 Full − S1 2.0 1.5 1.0 0.5 0.0 0 100 200 300 Time(s) 8

Selecting a cut-off value Clip − S2 2.0 1.5 1.0 0.5 Mbps 0 20 40 60 Full − S1 2.0 1.5 1.0 0.5 0.0 0 100 200 300 Time(s) 9

Selecting a cut-off value Clip − S2 2.0 1.5 1.0 0.5 Take from the Mbps 0 20 40 60 beginning, and startup delay is Full − S1 unchanged 2.0 1.5 1.0 0.5 0.0 0 100 200 300 Time(s) 10

Kolmogorov-Smirnov Test Cutoff (s) 10 60 120 180 240 300 DASH 1080p DASH 720p Non − adapt 360p Non − adapt 720p 1.0 0.8 0.6 CDF 0.4 0.2 0.0 0.0 0.4 0.8 0.0 0.4 0.8 0.0 0.4 0.8 0.0 0.4 0.8 KS Test p − value 11

Autocorrelation Function-based dissimilarity ● DASH 1080p DASH 720p Non − adapt 360p Non − adapt 720p 3 ACF dissimilarity ● 2 90 th percentile ● 1 ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● 0 0 200 400 600 Cutoff length (s) 12

Does it really work? • Do we get similar user experience metrics even if we cut-off the test before the video ends? – YES 13

Does it really work? • Do we get similar user experience metrics even if we cut-off the test before the video ends? – YES (We tested with our own client in emulated networks) we measured stall ratio= total stall duration/total playable video 14

Testing in Experiment network with 10 YouTube videos of different lengths ● 2Mbps Network rate 3Mbps 4Mbps ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● 0.10 Stall ratio 0.01 0 200 400 600 Cutoff length(s) 15

Testing in Experiment network with 10 YouTube videos longer than 10 min (600 s) ● 14Mbps Network rate 15Mbps 16Mbps 17Mbps ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● 0.075 Stall ratio ●●● ●● 0.050 ● ● 0.025 ● 0.000 ● 0 200 400 600 Cutoff length(s) 16

Summary and Future work … • Test durations should run at least for 1 min, but 3 min gives more reliable results. 17

Summary and Future work … • Test durations should run at least for 1 min, but 3 min gives more reliable results. • What about DASH related metrics? 18

Summary and Future work … • Test durations should run at least for 1 min, but 3 min gives more reliable results. • What about DASH related metrics? • Results “probably” not suitable for feature-length movies 19

Thank you!