Case Study: How Consumer Source, Inc. uses multivariate testing to - - PowerPoint PPT Presentation

▶
case study how consumer source inc uses multivariate
SMART_READER_LITE
LIVE PREVIEW

Case Study: How Consumer Source, Inc. uses multivariate testing to - - PowerPoint PPT Presentation

Jan 20, 2023 109 likes •507 views

Case Study: How Consumer Source, Inc. uses multivariate testing to make intelligent business decisions and ramp up conversions Paul Terry Sr. Manager - Optimization Consumer Source A Primedia Company Case Study Background Case Study

slide-1
SLIDE 1

Paul Terry

  • Sr. Manager - Optimization

Consumer Source – A Primedia Company Case Study: How Consumer Source, Inc. uses multivariate testing to make intelligent business decisions and ramp up conversions

slide-2
SLIDE 2

Case Study Background

slide-3
SLIDE 3
  • Who we are: Consumer Source, a Primedia company

Business established in 1975 with the print publication Apartment Guide – a free directory of apartment communities.

Case Study Background

3

  • Primary Business: Lead generation for

subscription based advertisers on the internet and in print

slide-4
SLIDE 4
  • ApartmentGuide.com, Rentals.com, NewHomeGuide.com,

AmericanHomeGuides.com, RentalHouses.com, Rentlist.com

Case Study Background

4

  • Advertisers increasingly want more internet presence
  • Early site improvements were HIPPO driven
  • Development cycles were long
  • Analytics were fuzzy
  • Results were obscure
slide-5
SLIDE 5
  • In 2008, incoming executive management began to transform

Consumer Source into an internet company.

  • Efforts & budgets for SEO and SEM stepped up

Case Study Background

5

  • Agile development

methodology adopted

  • A Website Optimization

team was created, and SiteSpect was selected to implement multivariate testing

slide-6
SLIDE 6

Case Study Background

6

  • Most testing involves no involvement

from IT; some graphic design

  • Testing objectives typically to increase

lead conversion and advertising revenue

  • Run A/B or multivariate tests on several

websites; 5-6 concurrent per site.

  • Winners become Agile stories for

implementation

  • Some winners are rolled out to other sites
slide-7
SLIDE 7

Tactics: Test (nearly) Everything

slide-8
SLIDE 8

Tactics: Test (nearly) Everything

8

slide-9
SLIDE 9

9

Tactic #1: Hone Calls to Action Original : “Contact Property” Users prefer: Why do I want to do that? Check Availability

Current Rent Specials

Tactics: Test (nearly) Everything

slide-10
SLIDE 10

Tactics: Test (nearly) Everything

10

Tactic #2: Improve the location of the Lead Form Expert Opinion: Main photo should be “front and center” Users prefer: Make it

  • bvious how to proceed.
slide-11
SLIDE 11

Tactics: Test (nearly) Everything

11

Tactic #3: Improve the lead form Original: First and Last name Users prefer: One name field

slide-12
SLIDE 12

Tactics: Test (nearly) Everything

12

Tactic #3: Improve the lead form Require Phone Number Some Clients prefer us to require the phone number

slide-13
SLIDE 13

Tactics: Test (nearly) Everything

13

Original: Move-in Date Calendar control Users prefer: Immediately, Next week, Next month, etc. Tactic #3: Improve the lead form

slide-14
SLIDE 14

Tactics: Test (nearly) Everything

14

Original: Small black “Submit” button Users prefer: Large red “Send” button. Tactic #3: Improve the lead form

slide-15
SLIDE 15

Tactics: Test (nearly) Everything

15

Tactic #4: Search Header

slide-16
SLIDE 16

Tactics: Test (nearly) Everything

16

Original: Page headers varied Users prefer: Consistent treatment promotes ease of use Tactic #4: Search Header

slide-17
SLIDE 17

Tactics: Test (nearly) Everything

17

Running series of tests to optimize how the type ahead functions Tactic #5: Search Type Ahead

slide-18
SLIDE 18

The Challenge – Increase value to ALL advertisers

slide-19
SLIDE 19

Increase value to ALL advertisers

19

MVT Lab Challenge: Create a tide to lift all boats!

fold

slide-20
SLIDE 20

Increase value to ALL advertisers

20

fold

Goal #1: Increase leads to primary advertisers Important!

slide-21
SLIDE 21

Increase value to ALL advertisers

21

Goal #2: Maintain or Increase clicks to secondary advertisers

fold

Also Important! Express value of Content A in terms of Content B

slide-22
SLIDE 22

Increase value to ALL advertisers

22

Goal #3: Draw users below the fold

fold

Rich content shouldn’t be missed Small intra document links

slide-23
SLIDE 23

Increase value to ALL advertisers

23

Opportunity : Photos are hidden

fold

slide-24
SLIDE 24

Increase value to ALL advertisers

24

  • Four main designs
  • Three multivariate and 8 A/B tests
  • Twenty Variations
slide-25
SLIDE 25

Increase value to ALL advertisers

25

  • Four main designs
  • Three multivariate and 8 A/B tests
  • Twenty Variations
  • Conventional wisdom challenged
slide-26
SLIDE 26

Increase value to ALL advertisers

26

  • Four main designs
  • Three multivariate and 8 A/B tests
  • Twenty Variations
  • Conventional wisdom challenged
  • Three quarters of a million tested users selected the

winning solution for our advertisers

slide-27
SLIDE 27

Increase value to ALL advertisers

27

fold

slide-28
SLIDE 28

Increase value to ALL advertisers

28

Community Carousel

fold

  • Auto advance slideshow
  • Mouse wheel scrolling
  • 10 initial photos shown
  • Ajax driven
  • Click to enlarge photo
  • Click to load all photos
slide-29
SLIDE 29

Increase value to ALL advertisers - Results

29

fold

  • 21% more visitors send

a lead

  • 26% more leads per visit

Community Carousel

slide-30
SLIDE 30

Increase value to ALL advertisers - Results

30

fold

  • 15% more Credit Score

clicks

  • 21% more visitors send

a lead

  • 26% more leads per visit

Community Carousel

slide-31
SLIDE 31

Increase value to ALL advertisers - Results

31

fold

  • 45% more users scroll to

the bottom of the page.

  • 21% more visitors send

a lead

  • 26% more leads per visit
  • 15% more Credit Score

clicks Community Carousel

slide-32
SLIDE 32

Test Analytics

slide-33
SLIDE 33

Test Analytics

33

The final Community Carousel test iteration converging to a 26% increase in leads was verified when the feature went to production

Time – Identifying normal cycles helps you to know how long to run a test

slide-34
SLIDE 34

Test Analytics

34

Community Carousel was even more effective on IE8, our highest converting browser. Lead conversion +35%

Time – Identifying normal cycles helps you to know how long to run a test Segmentation –Understand and eliminate outlying data. Specific browsers, succeeding users, failing users

slide-35
SLIDE 35

Test Analytics

35

Confidence ratings above 90% indicate high statistical significance.

Time – Identifying normal cycles helps you to know how long to run a test Segmentation –Understand and eliminate outlying data. Specific browsers, succeeding users, failing users Confidence – Statistical calculation “Two Tailed Test” expresses statistical significance

  • f results
slide-36
SLIDE 36

Test Analytics – The Story

36

Every user tells a story Define a common set of metrics across all tests See how your test changes the story for each segment

The Community Carousel caused the nearby lead call to action to more than triple its conversion

slide-37
SLIDE 37

Summary & Key Takeaways

slide-38
SLIDE 38

Summary & Key Takeaways

38

 Test the small things - Page layout, positioning and simplicity of forms,

colors, breaking below the fold

 Hone calls to action – Know what your users want, and discover how best to

lead them there

 Test the big things - Bold features and new designs can be fleshed out,

refined, and put into production with confidence

 Establish a framework of common metrics  Measure competing content in terms of each other  Testing practice should work well with development process – Agile!  There are NO failures. Always gain better understanding of your site and

users

slide-39
SLIDE 39

39

Question-and-Answer Session