Dieudonn Kantu Ipsos Ian Durbach Ipsos & Department of - - PowerPoint PPT Presentation

Early warning systems for detecting changes in marketing metrics over time

Dieudonné Kantu Ipsos Ian Durbach Ipsos & Department of Statistical Sciences, University of Cape Town

1

50 60 70 80 90 100 1/1/10 3/1/10 5/1/10 7/1/10 9/1/10 11/1/10 1/1/11 3/1/11 5/1/11 7/1/11 9/1/11 11/1/11 Index of search popularity

Woolworths

When can we be sure a change has happened?

Overview

• The limits of prediction: why “early detection” is the aim
• Can we do early detection?

3

Limits of prediction

• Marketing lore is full of examples of unexpected

successes and failures: E.T., Paramount studio, Honda, etc

• Why do we think we can easily predict the future?

– Watts’ analogy of the future as a “bundle of threads”

• Only two conditions in which predictions of the future

can be trusted:

– When regular feedback allows repetitive testing of theories – When a large number of smaller predictions can be aggregated

4

Limits of prediction

• Many marketing decisions are unique, one-off decisions

– Prediction in advance difficult or impossible

• Small improvements in predictive accuracy: large “business

value” does not mean a better understanding of behavior

• Lessons from Netflix

– 2006: Recommendation system outperforms naïve model by 10% – Another 10%? $1 million prize, >3 years, worldwide interest – Solution never implemented!

• Goal: to detect change as quickly as possible after it occurs

5

Limits of prediction

• Two-year purchase history for 1738 shoppers, 16 brands

6 (a) (b) 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 0 96 3 0 86 12 2 1 40 47 8 2 1 1 1 32 35 21 9 3 1 2 13 34 27 12 6 4 1 1 1 2 12 24 26 20 11 5 2 3 8 17 23 24 11 8 4 1 1 2 3 5 13 20 22 18 12 7 3 1 4 7 8 14 18 18 16 9 4 3 2 2 4 2 7 13 18 19 17 12 8 4 1 5 4 4 6 14 19 14 16 12 5 5 1 5 1 3 8 13 17 18 17 13 8 3 1 6 5 1 6 8 12 16 18 18 9 5 2 6 1 4 8 12 17 19 18 13 7 2 7 4 2 1 3 5 14 18 20 16 8 9 7 1 3 7 12 18 22 20 13 5 8 6 3 1 3 3 5 7 13 25 19 14 8 2 5 11 20 26 24 12 9 2 1 1 2 4 4 7 16 28 37 9 1 3 9 21 35 32 10 2 1 1 1 1 2 2 3 8 79 10 2 12 86 Times bought in NEXT 10 purchases Times bought in LAST 10 purchases Times bought in NEXT 10 purchases Times bought in LAST 10 purchases

Actual behaviour Dirichlet model

Early detection

• What is needed is a change in mind-set: False positive and

false negatives can both be evaluated easily enough.

7

Diagnosed as true Diagnosed as false True in reality True positive (correct) False negative (Type 2 error) False in reality False positive (Type 1 error) True negative (correct) Table 1: Classification of model error types

Chow test

• A traditional statistical method

– Partition data into two parts – Test whether the means (or slopes of Regression lines) differ in the two parts

8

EWMA models

• A more “heuristic” control method

– uses a known “control period” to set up bounds of an “in control” process. – Future data will trigger an alarm if outside the bounds

9

Bayesian change point models

• Simultaneously estimates where change-points lie and

how process changes

• Partitions data into “blocks”, each block defined by its
• wn statistical model

10

Bayesian change point models

• Example: Hypothetical purchase histories

11

Bayesian change point models

• Example: Hypothetical purchase histories

12

Dynamic monitoring

13

Using the Bayesian change-point model

3 periods after apparent change, not sure at all it was “real” After 8 periods, much more confident But now also detecting signs of a second change here (incorrectly, it turns out)

Can our models do early detection?

• Simulation: we generate data for 2 time periods using different

parameters for each time period (a true change)

• We vary the following parameters:
• The number of periods before the change-point (3 to 20)
• The number of periods after the change-point (3 to 10)
• The change in the average level of the time series that occurs at

the change-point (0 to 5 units or standard deviations).

• Check how many times, out of 100, our methods detect the true

change

14

Simulation study

• We apply two models for each approach: Conservative

and Aggressive 1. Chow test: 5% vs. 20% significance level 2. EWMA model: at least 75% vs. 50% of subsequent data points outside specified bounds. 3. Bayesian model: 67% vs. 33% posterior probability that a change has occurred.

15

Simulation results

• Results show probability of detecting change (EWMA and Chow models)

16

Simulation results

• Results show probability of false positives (EWMA and Chow

models)

17

Simulation results

• Results show probability of detecting change (Bayesian model)

18

Simulation results

• Results show probability of false positives (Bayesian model)

19

Guidelines for practice

• Markets tend to be highly complex systems consisting of many

interacting and inter-related components.

• Predicting what will happen next with is an exceedingly difficult

task.

• Analysis goal: shifts from predicting change to detecting it as

quickly as possible – contribute in real-time to crafting overall corporate strategy. – allow managers can respond to change quickly

20

Guidelines for practice

• No clear winner among methods for detecting change
• All methods needed at least 5 to 10 observations before they

detect all but the very largest changes.

• Marketers therefore have two possible choices, each with
• wn pro’s and con’s:
• collect data more often
• use statistical methods in an “aggressive” mode

21

Thank you!

22