The Biggest Optimization Mistake I Ever Made Case Study: Air Canada - - PowerPoint PPT Presentation

Tim Burgess Justin Bridegan Web Analytics Manager Senior Marketing Manager Air Canada MECLABS

Case Study: Air Canada

The Biggest Optimization Mistake I Ever Made

Presenters

Tim Burgess Web Analytics Manager Air Canada Justin Bridegan Senior Marketing Manager MECLABS

2

3

Where I Came From…

 Agile  Selling primarily to new visitors  Experimental with both software and marketing  Failure was accepted as part of the learning process  Don’t ask about the product quality

4

Testing at Air Canada

 Unknown or brand new to most individuals and groups  Measurement is done through

• ld database architecture

 Company structure is separated into silos  Web analytics and optimization are positioned in IT

5

Web Analytics Maturity Model

2 1.5 2 2 2 3

Resources Methodology Tools Management Objectives Scope

Tools 1 Management 1 Objectives 1 Scope Resources 1 Methodology 1

Jan 2011 Mar 2012

***This has been stolen from Stephan Hammel

6

7

Before I tell you about the worst test I ever had….

…don’t feel bad for me…

• We are installing an enterprise-level

testing tool

• We are gaining ground in the marketing

community

• My boss still likes me (as far as I know)

8

5 steps for developing a test within Air Canada

1. Develop a strategy – where is the lowest-hanging fruit for profitability? 2. Use the MarketingExperiments concepts of Friction and Value to develop mockups 3. Find a stakeholder whose interest I will work in, and sell them on the concept 4. Set up and run the test 5. Report to stakeholders 1 2 3 4 5

9

Step 1: Strategy

• Test to improve seat selection

– Air Canada’s largest source of ancillary revenue Develop a strategy – Where is the lowest-hanging fruit for profitability?

1

10

Use the MarketingExperiments concepts of Friction and Value to develop mockups

Step 2: Analyze Friction and Value

2

Perceived Cost

BPY=Pc(VF>CF) BPY=Pc[Cl+Cr](VF[Ap+Ex]>CF[Rc+Ef])

Perceived Value WHY DO PEOPLE SAY “YES”?

11

The Original: Seat selection

The original variation is good from a navigational point of view, but does nothing to emphasize the value

• f “Advanced” or

“Preferred” seat selection.

12

Variation A

This variation focused

• n bringing Value to the

top of the page through bullets, while maintaining clear navigation.

13

Variation B

The goal of Variation B was to bring navigation further above the fold, while still featuring the bullet points.

14

Variation C

Answered the question, “What if they really don’t know the difference between Preferred and Advanced seat selection?” by promoting description in text form above navigation.

15

Find a stakeholder whose interest I will work in, and sell them on the concept

Step 3: Find a stakeholder

3

16

Stakeholder – Revenue Management

The group that manages pricing, loads and network and also responsible for Seat Selection revenue. Loved the idea!

17

Testing

Step 4: Set up and run the test

4

Control Variable A Variable B Variable C

18

Step 5: Report to stakeholders

5

Report on findings

We ran the test for one week to 95% significance, with a follow-up test for three weeks, that showed that variation C was generating 48.78% more revenue than the original. Control Variable C

19

We have a winner…

So we celebrated!

Variable C

20

One small hitch…

The numbers in the months following the experiment did not reflect the experiment results

My Mistake: By communicating the results the way I did, I led the stakeholders to believe that we would see those numbers once the test went live.

• Revenue Management did not understand how this could happen
• My boss was disappointed
• I was confused

21

So, what happened?

• Visitors on

Aircanada.com are

• ften repeat visitors
• There are many points

in the process where visitors can purchase seat selection (see the email below)

• We cannibalized on
• ther channels

22

Analytics data

Preferred Seat Revenue/flight order

1 2 3 4 5 6 7 Jul 1, 2011 Jul 6, 2011 Jul 11, 2011 Jul 16, 2011 Jul 21, 2011 Jul 26, 2011 Jul 31, 2011 Aug 5, 2011 Aug 10, 2011 Aug 15, 2011 Aug 20, 2011 Aug 25, 2011 Aug 30, 2011

Advanced Seat Selection Revenue/Order

0.5 1 1.5 2 2.5 3 3.5 4 Jul 1, 2011 Jul 6, 2011 Jul 11, 2011 Jul 16, 2011 Jul 21, 2011 Jul 26, 2011 Jul 31, 2011 Aug 5, 2011 Aug 10, 2011 Aug 15, 2011 Aug 20, 2011 Aug 25, 2011 Aug 30, 2011

Even though analytics data showed a change in the amount of revenue for Preferred Seat selection, Revenue Management was used to looking at its own data, which is not slice and trend-able like analytics data. Using conservative numbers the actual impact of the test was an approximate increase of 5% on Preferred Seat selection revenue.

23

The irony…

…is that this is the most successful test I have run to date, in terms of profitability.

24

Key takeaways

 Remember the big picture  Manage expectations  Predict scenarios, not results  Keep testing Remember… The way in which hypotheses and results are presented is pivotal to building support and making optimization a long-term strategy.

25

Thank you!

Questions?