Dieudonn Kantu Ipsos Ian Durbach Ipsos & Department of - PowerPoint PPT Presentation

Early warning systems for detecting changes in marketing metrics over time Dieudonné Kantu Ipsos Ian Durbach Ipsos & Department of Statistical Sciences, University of Cape Town 1

When can we be sure a change has happened? Index of search popularity 100 50 60 70 80 90 1/1/10 3/1/10 5/1/10 7/1/10 Woolworths 9/1/10 11/1/10 1/1/11 3/1/11 5/1/11 7/1/11 9/1/11 11/1/11

Overview • The limits of prediction: why “early detection” is the aim • Can we do early detection? 3

Limits of prediction • Marketing lore is full of examples of unexpected successes and failures: E.T., Paramount studio, Honda, etc • Why do we think we can easily predict the future? – Watts’ analogy of the future as a “bundle of threads” • Only two conditions in which predictions of the future can be trusted: – When regular feedback allows repetitive testing of theories – When a large number of smaller predictions can be aggregated 4

Limits of prediction • Many marketing decisions are unique, one-off decisions – Prediction in advance difficult or impossible • Small improvements in predictive accuracy: large “business value” does not mean a better understanding of behavior • Lessons from Netflix – 2006: Recommendation system outperforms naïve model by 10% – Another 10%? $1 million prize, >3 years, worldwide interest – Solution never implemented! • Goal: to detect change as quickly as possible after it occurs 5

Limits of prediction • Two-year purchase history for 1738 shoppers, 16 brands (a) Times bought in NEXT 10 purchases (b) Times bought in NEXT 10 purchases 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 96 3 0 0 0 0 0 0 0 0 0 0 86 12 2 0 0 0 0 0 0 0 0 Times bought in LAST 10 purchases Times bought in LAST 10 purchases 1 40 47 8 2 1 1 0 0 0 0 0 1 32 35 21 9 3 1 0 0 0 0 0 2 13 34 27 12 6 4 1 0 1 0 1 2 12 24 26 20 11 5 2 0 0 0 0 3 8 17 23 24 11 8 4 1 1 0 2 3 5 13 20 22 18 12 7 3 1 0 0 4 7 8 14 18 18 16 9 4 3 2 2 4 2 7 13 18 19 17 12 8 4 1 0 5 4 4 6 14 19 14 16 12 5 5 1 5 1 3 8 13 17 18 17 13 8 3 1 6 5 1 6 8 12 16 18 18 9 5 2 6 0 1 4 8 12 17 19 18 13 7 2 7 4 2 1 3 5 14 18 20 16 8 9 7 0 0 1 3 7 12 18 22 20 13 5 8 6 3 1 3 3 5 7 13 25 19 14 8 0 0 0 0 2 5 11 20 26 24 12 9 2 1 1 0 2 4 4 7 16 28 37 9 0 0 0 0 0 1 3 9 21 35 32 10 2 1 0 1 1 1 2 2 3 8 79 10 0 0 0 0 0 0 0 0 2 12 86 Actual behaviour Dirichlet model 6

Early detection Diagnosed as true Diagnosed as false True in reality True positive False negative (correct) (Type 2 error) False in reality False positive True negative (Type 1 error) (correct) Table 1: Classification of model error types • What is needed is a change in mind-set: False positive and false negatives can both be evaluated easily enough. 7

Chow test • A traditional statistical method – Partition data into two parts – Test whether the means (or slopes of Regression lines) differ in the two parts 8

EWMA models • A more “heuristic” control method – uses a known “ control period ” to set up bounds of an “in control” process. – Future data will trigger an alarm if outside the bounds 9

Bayesian change point models • Simultaneously estimates where change-points lie and how process changes • Partitions data into “blocks”, each block defined by its own statistical model 10

Bayesian change point models • Example: Hypothetical purchase histories 11

Bayesian change point models • Example: Hypothetical purchase histories 12

Dynamic monitoring Using the Bayesian change-point model 3 periods after apparent change, not sure at all it was “real” After 8 periods, much more confident But now also detecting signs of a second change here (incorrectly, it turns out) 13

Can our models do early detection? • Simulation: we generate data for 2 time periods using different parameters for each time period (a true change) • We vary the following parameters: - The number of periods before the change-point (3 to 20) - The number of periods after the change-point (3 to 10) - The change in the average level of the time series that occurs at the change-point (0 to 5 units or standard deviations). • Check how many times, out of 100, our methods detect the true change 14

Simulation study • We apply two models for each approach: Conservative and Aggressive 1. Chow test: 5% vs. 20% significance level 2. EWMA model: at least 75% vs. 50% of subsequent data points outside specified bounds. 3. Bayesian model: 67% vs. 33% posterior probability that a change has occurred. 15

Simulation results • Results show probability of detecting change (EWMA and Chow models) 16

Simulation results • Results show probability of false positives (EWMA and Chow models) 17

Simulation results • Results show probability of detecting change (Bayesian model) 18

Simulation results • Results show probability of false positives (Bayesian model) 19

Guidelines for practice • Markets tend to be highly complex systems consisting of many interacting and inter-related components. • Predicting what will happen next with is an exceedingly difficult task. • Analysis goal: shifts from predicting change to detecting it as quickly as possible – contribute in real-time to crafting overall corporate strategy. – allow managers can respond to change quickly 20

Guidelines for practice • No clear winner among methods for detecting change • All methods needed at least 5 to 10 observations before they detect all but the very largest changes. • Marketers therefore have two possible choices, each with own pro’s and con’s: - collect data more often - use statistical methods in an “aggressive” mode 21

Thank you! 22