Science of f Significance How an e-commerce company went from Wild - - PowerPoint PPT Presentation

How an e-commerce company went from Wild West testing rules to wildly successful results through variance testing

Science of f Significance

Em Emil ily Em Emmer Senior Interactive Marketing Manager Extra Space Storage Alli lison Ban Banko Reporter MECLABS

Session speaker

2

Emil ily Emmer Sen enior In Interactiv ive Marketi ting g Manager Extra Space Storage

• At Extra Space Storage, Emmer has increased the conversion rate

by 130% over the past four years.

• Her group has increased Web performance 1,300% and delivered

more than $50 million of incremental revenue annually.

• Extra Space Storage tests and optimizes across channels including

desktop, mobile, email, call center and online advertising.

• Emmer holds her MBA from W.P. Carey School of Business at

Arizona State University and a B.A. from the University of Utah.

@emilyemmer

About Ext xtra Space Storage

• Publicly traded company founded in 1977
• Nationwide with 1,100 self-storage properties
• 650,000 customers with 500,000 new customers annually
• $1 billon in revenue annually

Wild West testing rules

Homepage Treatment 15% Lift Contr trol

A deeper lo look in into the test

Control Treatment

475 Conversions 413 Conversions 12% Traffic 3 Weeks Testing 12% Traffic 3 Weeks Testing 15% Lift

Actual dif ifference or varia iance?

What is variance testing? A/A test of f identical experiences.

Instead of absolute difference in conversion rate, this test measures the natural variability

• f a website.

Varia iance vs. . conversion testing

A/B Testing A/A Testing

Difference (lift) Variance (noise)

Results based on difference between treatment and control Results used to determine testing rules

15% Lift

Wildly successful testing

Varia iance testin ing help lped us understand:

• How long do we run our tests?
• How much traffic do we need for each treatment?
• How many treatments do we run in our tests?
• How big must the lift be to call a winner?
• How many conversions do we need per treatment?

Extr xtra Space Storage varia iance testin ing

Loc Location: ExtraSpace.com Over erview: 5 exp xperiments

Bac Background: Company needed testing rules to determine a significant result. Ob Objective: Determine the baseline noise of the website for reading future tests. Prim rimary ry Res esearch Qu Question: What conditions make our tests confident? Tes est t Des Desig ign: Same experiences tested across different amounts of traffic.

Res esea earch Notes:

Exp xperim iment 1: : 35% tr traffic ic per r tr treatment (2 tr treatments)

Week 2 Week 3 Control 2.03 2.02 Control B (Same Experience) 1.98 2.02 Rel elative Dif Difference 2.08 2.08 1.04 1.04

1% difference with 3 weeks traffic

Relatively small amount of variance with 35% traffic per treatment.

Exp xperim iment 2: : 18% tr traffi fic per r tr treatment (4 (4 tr treatments)

Week 2 Week 3 Control 1.94 1.98 Control B (Same Experience) 2.62 2.05 Rel elative Dif Difference 4.65 4.65 3.67 3.67

4% difference with 3 weeks traffic

Higher natural variance with 18% traffic per experience.

Exp xperim iment 3: : 12% tr traffi fic per r tr treatment (6 (6 tr treatments)

Week 2 Week 3 Control 1.55 2.02 Control B (Same Experience) 1.91 1.76 Rel elative Dif Difference 23 23.56 .56 14 14.44 .44

14% difference with 3 weeks traffic

Significant variance.

Exp xperim iment 4: : 7% tr traffic ic per r tr treatment (6 (6 tr treatments)

Week 3 Week 4 Control 2.02 1.78 Control B (Same Experience) 1.77 2.13 Rel elative Dif Difference 18 18.70 .70 19 19.69 .69

20% difference with 4 weeks traffic

Substantial noise at this volume of testing.

Exp xperim iment 5: : 3% traffic ic per r treatment (6 treatments)

Week 4 Week 5 Control 2.03 2.17 Control B (Same Experience) 1.77 1.85 Rel elative Dif Difference 15 15.11 .11 16 16.89 .89

17% difference with 5 weeks traffic

Large difference at this volume of testing traffic.

Summary: Varia iance by y traffic ic and tim ime perio iod

Traffic Time A Time B Experiment 1 35% 2% 1% Experiment 2 18% 5% 4% Experiment 3 12% 24% 14% Experiment 4 7% 19% 20% Experiment 5 3% 15% 17%

Our testing rules

• Need to test at least 15% of volume for A/B tests
• Lift needs to be more than 4% to be called a winner
• Need 500 conversions per experience
• Run tests for at least two weeks

Our testing rules

• Need to test at least 15% of volume for A/B tests
• Lift needs to be over 4% to be called a winner
• Need 500 conversions per experience
• Run tests for at least two weeks

45% increase in conversion rates

How can I I start?

When should I conduct varia iance testin ing?

• At different times of the year if your website has

seasonal differences (i.e., high/low season variance)

• Repeat variance testing if you notice dramatic

changes to your traffic

• Repeat variance testing after any site changes are

made that affect visitor behavior

27

Test 50% Traffic

2 Experiences

Test 33% Traffic

3 Experiences

Test 25% Traffic

4 Experiences

Test 20% Traffic

5 Experiences

Test 10% Traffic

10 Experiences

5 Sim imple le Tests: C Complete these varia iance tests sim imultaneously to develop testing rule les for your website

Analy lyze Results: Compare result lts wit ithin in each test and across tests

Absolute Variance Experience Visitor Conversions Conv. Rate

• Vs. A
• Vs. B
• Vs. C
• Vs. D

A - Default 24,241 482 1.99%

• 1.98%

2.55% 2.53% B - Default 24,208 472 1.95% 1.94%

• 4.45%

4.43% C - Default 24,210 494 2.04% 2.62% 4.65%

• 0.02%

D - Default 24,264 495 2.04% 2.60% 4.63% 0.02%

Alt lternatively: In Integrate varia iance in into testing

A/B Test

Control Treatment

A/A/B Test

Control Treatment Control (V2) Treatment

Key takeaways

• Variance testing is EASY and has long-term benefits
• Just test setup is required, no develo

lopment

• Gain consensus about test setup, duration and results
• Get guidance for significant results through testing

Thank you

Emil ily Emmer Senior Interactive Marketing Manager Extra Space Storage @em emil ilyemmer Allis llison Ba Banko Reporter MECLABS @alli allisonbanko