Its Not the Cost, Its the Quality! Ion Stoica Conviva Networks and - - PowerPoint PPT Presentation

It’s Not the Cost, It’s the Quality!

Ion Stoica Conviva Networks and UC Berkeley

1

A Brief History

Fall, 2006: Started Conviva with Hui Zhang (CMU) Initial goal: use p2p technologies to reduce

distribution costs and improve the scale

Slowly, realized our customers (content premium

producers & aggregators) value more quality than cost

Today: maximize distribution quality, distribution

management, and provide real-time analytics

2

Where is the Data Coming From?

Content Providers and Aggregators CDNs

3

4

Trends

Trends: CDN Pricing

CDN pricing has decreased x1.5-2 every year

• ver the last 5 year

5

5 10 15 20 25 30 35 40 45 2006 2007 2008 2009 2010

cents/GB

Trends: Streaming Rate for Premium Content

Average streaming rate has increased 20-40%

every year

6

200 400 600 800 1000 1200 1400 1600 2006 2007 2008 2009 2010

Kbps

Trends: Per-hour Streaming Cost

Per-hour streaming cost has decreased 15-35%

every year

7

1 2 3 4 5 6 7 8 9 10 2006 2007 2008 2009 2010

cents/hour

HTTP Chunking

Trend accelerated by switching from proprietary

streaming technologies (e.g., Adobe’s FMS) to HTTP Chunking:

• Move Networks (2005)
• Apple (2008)
• Microsoft (2008/2009)
• Adobe (2010, 2nd half)

8

How Does HTTP Chunking Work?

9 Viewer

CDN1

Viewer Viewer

CDN2 ISP A ISP B

…

• rigin

http cache

HTTP Chunking Advantages

Chunks: immutable, relative large objects

(hundreds of KB)

• Great for caching

Leverage existing HTTP infrastructure

• CDNs
• ISP deployed caches
• Enterprise http proxies

Low cost and high scale

10

What Does this Mean?

Ad supported premium content

• CPM (cost per thousand of ad impressions) for

premium content has reached: $20-$40

• One ad covers one hour of streaming!

Paid content

• $0.99 episode, distribution cost < 3%

Subscription based premium content

• Distribution, usually a few percents of total cost
• It costs $1.6 per month to stream content to an user

watching 2 hours per day

Production & rights costs dominate

11

12

Quality Matters

Quality Matters

Better quality

• Increase viewing time  more ad opportunities
• Increase retention rate
• Protect brand

Quality

• Join time
• Buffering ratio
• Rendering quality
• Streaming rate

13

Analysis

Load:

• Four channels of a premier video-on demand (VoD)

content producer

• Four days
• Number of sessions (views): 1,176,049
• A large live event: ~250,000 concurrent viewers

Metrics

• Content length distribution
• Viewer Hour Loss (VHL): number of viewer hours lost

due to quality issues

VoD Object Length Distribution

15

50 100 150 200 250 300 10 40 70 100 130 160 190 220 250 280 310 340 370 400 430 460 490 520 550 580 610 640 680 750 880 1240 1330 2540 2570 2600 2640 4040 5410 5550 Object Length

Short clips: [2min, 3min] Medium clips: [9min, 11min] Full Episodes: [42min, 45min]

Number of Objects

Quality Metrics

Buffering Quality (BQ):

PlayingTime/(PlayingTime + BufferingTime)

• Rendering Quality (RQ):

RenderingRate/EncoderRate

• Good session
• BQ > 95%
• RQ > 60%

Analysis Underestimates Quality Impact

For most analysis use BQ only

• RQ only a small part of quality issues due to low

bit rate (500-700Kbps)

Ignore connection failures

Short Clip (2-3min) Analysis

18

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Average Playing Time (Minutes) Quality

Short Clip (2-3min) Analysis

19

50000 100000 150000 200000 250000 300000 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Number of Sessions Quality

Viewer Hour Gain

D : Average duration of sessions with high quality

(0.98 <= quality < 1)

Dq : Average duration of sessions with quality = q Nq : Number of sessions with quality = q Viewer hour gain for sessions with quality q

Nq x (D – Dq)

Total viewer hour gain

∑q Nq x (D – Dq)

20

Viewer hour loss for 1-2 minute clips:

Medium Clip (9-11min) Analysis

21

1 2 3 4 5 6 7 8 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Average Playing Time (Minutes) Quality

Viewer hour loss for 9-11min clips:

Full Episodes (42-45min) Analysis

22

5 10 15 20 25 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Average Playing Time (Minutes) Quality

• Viewer hour loss for episodes:
• Viewer hour loss for all content:

Large Scale Live Event

23

5 10 15 20 25 30 35 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Average Playing Time (Minutes) Quality

Viewer hour loss:

Large Scale Live Event: Engagement Funnel

24

1 0.99 0.8624 0.7364 0.6602 0.5509 0.4357 0.3639 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 All 0-0 0-1 1-3 3-5 5-10 10-20 20-30 Fraction of Sesions Session Duration (Mins) 0<=Q<=0.5 0.5<=Q<=0.75 0.75<=Q<=0.9 0.9 <= Q <= 0.95 0.95<=Q<=0.98 0.98<=Q<=1.0 Quality = 1.0 Continuing

Half people leave due to quality issues

Another Case Study: Live Event

25 Total sessions

151,980

Unique viewers

73,942

Sessions per viewer

1.9

Total viewer hours

58,436

5000 10000 15000 20000 7:00 PM 7:15 PM 7:30 PM 7:45 PM 8:00 PM 8:15 PM 8:30 PM 8:45 PM 9:00 PM 9:15 PM 9:30 PM 9:45 PM 10:00 PM 10:15 PM 10:30 PM 10:45 PM 11:00 PM 11:15 PM 11:30 PM 11:45 PM 12:00 AM 12:15 AM 12:30 AM 12:45 AM 1:00 AM

Peak Concurrent Views

Quality Engagement Total views 151,980 25 minutes Failed views 13,815 (9%) 0 minutes Quality impacted views 21,584 (14%) 16 minutes Good views 116,581 (77%) 27 minutes Unique viewers 75,328 48 minutes Failed viewers 1,386 (2%) 0 minutes Quality impacted viewers 14,309 (19%) 30 minutes Good viewers 59,633 (79%) 51 minutes Total viewer hours 58,436 hours Lost viewer hours 5,134 hours (9%)

Viewer with poor quality watch 41% less minutes!

Does High Bit Rate Video Help?

Comparing Engagement of low and high bitrates

• Viewers watch longer on average on 1500Kbps

5 10 15 20 25 30 35 minutes

Average Play Time (minutes)

500 kbps 1500 kbps

10-15% increase

Summary

Quality impact:

• BQ can impact viewer engagement by up to 40%
• Higher bit-rates can increase viewer engagement by

up to 15%

Engagement loss due to quality issues: between

4 and 30%

• Even a 4% improvement, may offset distribution

costs

• Ignore other quality issues, like connectivity and

media failures

27

28

Root Cause Analysis

Viewers vs. Buffering Quality

29

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% 50000 100000 150000 200000 250000 0:00:00 0:15:00 0:30:00 0:45:00 1:00:00 1:15:00 1:30:00 1:45:00 2:00:00 2:15:00 2:30:00 2:45:00 3:00:00 3:15:00 3:30:00 3:45:00 4:00:00 4:15:00 4:30:00 4:45:00 5:00:00 5:15:00 5:30:00 5:45:00 6:00:00 6:15:00 Viewers Average Buffering Quality %

Light Period Heavy Period

What causes quality issues?

Root Cause Analysis

Root cause a quality issue to:

• Viewer machine (CPU)
• Last mile + ISP (Autonomous System Number)
• CDN

Note:

• Cannot differentiate between edge and core ISPs
• Use only passive measurements, no IP traceroute

Viewer

Last Mile + ISP

CDN

Peering ISP

Metrics and Definitions

Quality metrics

• Buffering quality (BQ)
• playing time/(playing time + buffering time)
• Rendering quality (RQ)
• rendering frame rate/encoded frame rate

Session classification:

• Good: (BQ >= 95%) AND (RQ >= 60%)
• Low BQ: (BQ < 95%)
• Low RQ: (BQ >= 95%) AND (RQ < 60%)

Methodology: Root Causing Viewer Machine

CPU likely to be the issue when:

• Rendering quality low
• Buffering quality high

Conclude CPU is the issue when session’s

• RQ < 60%
• BQ > 95%

Good Low BQ Low RQ

Quality Issues: Light Period

74% 21.5% 4% (CPU issue) Network/CDN issues

Good Low BQ Low RQ

Quality Issues: High Period

62% 6.2% (CPU issue) 31.5% Network/CDN issues

Explaining Buffering Issues

Assume buffering quality issues are either due to:

• CDN, or
• ISP

Recall: a session has buffering quality issues if

• BQ < 95%

Methodology: Root Causing CDN (1/2)

Viewers connected to same ASN but using two CDNs Intuition: if quality experienced by CDN 1 viewers is

significantly lower than of CDN 2 viewers for same ASN, CDN 1 has quality issues

Last Mile + ISP

CDN 1

Peering ISP

CDN 2

Peering ISP

Methodology: Root Causing CDN (2/2)

Select all ASNs who have more than 50

sessions for each CDN

• If difference between quality of viewers in CDN1 and

CDN2 for same ASN is > 10%

• Lower quality CDN is root cause at current time

Methodology: Root Causing ASN/ISP

Two CDNs:

• Conclude ASN A has quality issues if ASN A’s viewers

connected to either CDN1 or CDN2 experience “bad quality”

• Average quality of viewers connected to other ASNs

higher

One CDN

• ASN A’s viewers connected to CDN have much lower

quality than the average quality of viewers connected to CDN

Buffering Quality: Light Period

Unqualified CDN ISP Uknown

46% 33% 16% 5%

Buffering Quality: Heavy Period

40 Unqualified CDN ISP Uknown

36.7% 39.7% 15.4% 8.2%

Some Findings

Most of ASNs who had quality issues were

enterprise ASNs

• Expected given that the large scale event was during

the workday

• One ASN had 44% buffering quality!

No CDN was uniformly bad

• (see next)

CDN Comparison

• Quantify quality difference between CDNs
• Methodology:

1.

Select all ASNs which have more than 50 sessions

• n both CDNs

2.

Compute average quality for CDN1 and CDN2 viewers per ASN

3.

Order ASNs by difference in quality between CDN1 and CDN2

Internet delivery is more variable than realized…

Content Delivery

Networks all have problems sometime

Even in the same

viewer session the best quality changed many times during the event

CDN 1 was best CDNs were even CDN 2 was best

Summary

24-38% of Total Sessions have Quality Issues Quality Issues Classification Solution CDN

(7-12% of total sessions)

Resource switching End-Host CPU

(4-6% of total sessions)

Bit-rate switching ISP

(2-3% of total sessions)

Localize traffic, bit-rate switching Unqualified

(9-11% of total sessions)

Mitigated by above Unknown

(1-4% of total sessions)

N/A

Conclusions

At least for premium content

• Reducing cost is important, but…
• … improving quality is even more important

P2P can play an important role

• Localize traffic
• Highly robust to source failures

Great opportunity

• Adobe has announced full p2p support for Flash

Player 10.1

• No need for client download!

45