System Considerations in Real Time Video QoE Assessment Amy - PowerPoint PPT Presentation

System Considerations in Real Time Video QoE Assessment Amy Csizmar Dalal Department of Computer Science Carleton College adalal@carleton.edu

Outline  Architectural overview  Design tradeoffs and scenarios  Results  Conclusions and future work

QoE assessment architecture Goals: Improve system performance, better protocol/network support for Internet video

QoE rating architecture Sampled every second Goal: Examine the data-related design tradeoffs at various points in the rating architecture

Design tradeoffs Tradeoff Description Considerations Range Sampling rate Time between data Missing key congestive 1-5 sec samples events vs. resource utilization Interrating time How much of a False positives/ 10-60 sec stream’s data to negatives vs. missing examine before key congestive events assigning a rating Stream state data How many pieces of “Noisy” data vs. 1, 2, 3, all 4 combos data to use at once, inaccurate data and in what configurations Training set Whether to target the Better chance of a See “scenarios” composition training set to the match vs. resource stream to rate or use utilization all data in the training set Timing concerns Time to train the Flexibility vs. accuracy Fix sample rate system before rating at 1 sec, vary commences interrating time

Scenarios A A A A A Fine-tuned VOD A A A A A A A A A B B B B B General VOD B B B B B B B B B Training set videos General video

Experimental data Name Time Description Action level (MM:SS) cow 1:57 dialog moderate: frequent scene shifts okgo 3:06 music video moderate: stable scene, heavy action up 4:40 animated movie high: frequent scene shifts, short heavy action

Results: Top individual scenarios Scenario Video Stream Sample Time (s) Accuracy state data rate (s) (%) cow TP , BW 2 60 82 Fine-tuned okgo TP , BW 1 20 84 VOD up TP , BW 1 20 80 cow FR 5 50 88 General okgo TP , BW 1 50 86 VOD up TP , BW 1 20 81 cow FR 1 50 83 General okgo TP , BW 2 20 79 video up TP , BW 1 50 75 TP = received packets BW = bandwidth

Results: Top combinations Stream Accuracy (%) Sample Time Scenario state Cow Okgo Up rate (s) (s) Data Fine- Bandwidth + 1 20 77.83 84.10 79.85 tuned received VOD packets General Bandwidth + 1 20 84.73 83.61 80.60 VOD received packets General Bandwidth + 1 20 78.82 78.80 75.19 video received packets

Timing results, general VOD and general video

Conclusions: Tradeoffs summary Tradeoff Best choice Discussion Sampling rate 1 sec Allows maximum detection of congestive events Interrating time 20 sec Stream state Bandwidth + data combos received (Mostly) stream-independent packets Training set All available Fine-tuning does not improve performance composition videos here Training time < 10 minutes Off-line; short enough to allow for worst case retraining flexibility

Timing results, fine-tuned VOD