Deep Data Analytics for Pricing: Uses, Issues, and Solutions Walter - - PowerPoint PPT Presentation

Deep Data Analytics for Pricing: Uses, Issues, and Solutions

Walter R. Paczkowski, Ph.D.

Data Analytics Corp. and Rutgers University

October, 2018

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 1 / 181

Workshop Presented At:

The Professional Pricing Society: 29th Fall Conference and Workshops October, 2018 Dallas, TX

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 2 / 181

Workshop Overview

The workshop has five parts:

1 Introduction to Deep Data Analytics (DDA) 2 The Distinction Between Data and Information 3 The Role of DDA for Pricing 4 DDA Drill-down and Case Study 5 Organizing for DDA for Pricing Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 3 / 181

Workshop Objectives

The objectives for this workshop are:

1 Discuss the critical need for Rich Information for effective pricing. 2 Argue that Rich Information can only come from Deep Data

Analytics.

3 Illustrate that Deep Data Analytics is concerned not only with

statistical/econometric methods but also with data structures.

4 Highlight requirements for implementing Deep Data Analytics for

pricing. A case study will be used to meet the objectives.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 4 / 181

Workshop Appendix

An Appendix contains material for your review outside the workshop: Statistical highlights; Software recommendations; and References.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 5 / 181

Workshop Central Message

Effective Pricing Analysis

Effective pricing requires Rich Information based on Deep Data Analytics.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 6 / 181

Part I Introduction to Deep Data Analytics (DDA)

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 7 / 181

Introduction to Deep Data Analytics (DDA)

Pricing Development

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 8 / 181

Introduction to Deep Data Analytics (DDA)

Pricing Development

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 9 / 181

Introduction to Deep Data Analytics (DDA)

Pricing Development

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 10 / 181

Introduction to Deep Data Analytics (DDA)

Pricing strategy has two parts:

1 Price Structure: How prices are delivered – uniformly or

discriminatorily.

Example

A high price at the beginning of a fashion season and a low price at the end to target consumers based on their price sensitivities or elasticities.

2 Price Level: The price actually charged. Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 11 / 181

Introduction to Deep Data Analytics (DDA)

The Components of a Pricing Strategy

Pricing Structure

• Uniform
• Discriminatory

Pricing Strategy Pricing Level

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 12 / 181

Introduction to Deep Data Analytics (DDA)

Just as important as the price level is the price effect. All too often, managers just think of ”stimulating demand” by lowering the price. This is too narrow and simplistic.

Think More Broadly

What is the effect of a different price structure and/or level on a key business metric (KBM) such as. . . Revenue? Bid win? Contribution? Customer acquisition? Customer retention? Market share? This is where elasticities are important.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 13 / 181

Introduction to Deep Data Analytics (DDA)

The Components of a Pricing Strategy

Pricing Structure

• Uniform
• Discriminatory

Pricing Strategy Pricing Level Effect Assessment

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 14 / 181

Introduction to Deep Data Analytics (DDA)

Pricing decisions about strategies and levels are complex. These decisions must pass business case gates and overcome hurdles established by executive management. The quality of a decision depends upon the input into that decision. That input is information based on data. At the heart of pricing is Deep Data Analytics: a paradigm for converting raw data into actionable, insightful, and useful Rich Information.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 15 / 181

Introduction to Deep Data Analytics (DDA)

Deep Data Analytics for Pricing

Deep Data Analytics provides Rich Information through three critical, interrelated parts. Structure

• Non-nested Data, Nested/Multilevel

Data Visualize

• Relationships, Trends, Patterns,

Anomalies Model

• Elasticities, Take Rates, Willingness-

to-Pay

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 16 / 181

Introduction to Deep Data Analytics (DDA)

The Composition of DDA

Deep Data Analytics

Make Sense of Data Model Potential Outcomes Distill Key Insights

Skill Sets Collaboration Analytical Toolset

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 17 / 181

Part II The Distinction Between Data and Information

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 18 / 181

The Distinction Between Data and Information

Effective pricing decisions about structure, level, and effects cannot be made in a vacuum, no matter how ”street-smart” some think they are.

What is Needed?

Information Without information, you have to approximate (i.e., guess) what would succeed – guessing is costly.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 19 / 181

The Distinction Between Data and Information

There’s No Such Thing as a Free Lunch

Information Cost Cost of Approximation Base Approximation Cost

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 20 / 181

The Distinction Between Data and Information

Information Continuum1

Poor Information Rich Information Poor Information Analytical Toolset Rich Information

• Raw
• Cleansing
• Insightful
• Disorganized
• Organizing
• Organized
• Fuzzy
• Modeling
• Clear
• Unfiltered
• Reporting
• Filtered

1Based on Zahay et al. (2004) Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 21 / 181

The Distinction Between Data and Information

A common misconception, especially in the modern age of Big Data, is that data per se are information. Data are raw, unfiltered, disorganized pieces of material (i.e., ”stuff”).

”Stuff” is what we collect – and store in a closet.

Modern day data closets are computer files/folders, data marts, and data warehouses – and data lakes!

Data are like Lego bricks that have to be assembled.

Like the bricks, they can be assembled in infinite ways reflecting our creativity and questions.

Rich Information is insight built and extracted from data by creatively manipulating the data bricks.

Raw data as ”stuff” are, at best, Poor Information.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 22 / 181

The Distinction Between Data and Information

Data Sources for ”Bricks” Experiments

• Controlled
• Possibilities
• New products

Observations

• Transactions
• Past Behavior
• Existing products

Surveys

• Opinions
• Past/Current Behavior
• New concepts

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 23 / 181

The Distinction Between Data and Information

Guiding Principle

Data bricks, regardless of source, have to be analyzed to assemble them into useful, insightful, and actionable Rich Information.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 24 / 181

The Distinction Between Data and Information

What is analysis? To analyze means to break into parts.

Greek root: analusis

”a breaking up, a loosening, releasing.” a

ahttp://www.etymonline.com/index.php?term=analysis

Analyzing survey data means looking for

• relationships
• patterns
• trends
• anomalies

beneath the surface of the obvious and then to assemble the pieces from the analysis into a report. The report merely encapsulates the information.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 25 / 181

The Distinction Between Data and Information

Examples of the Four Parts Of Analysis

Example

1 Relationships

Correlations between products purchased and distribution channels Satisfaction or purchase intent key drivers

2 Trends

Developments or changes over time (e.g., a tracking study) Before/after treatment

3 Patterns

Segments

4 Anomalies

Outliers

Not all outliers are created equal Some are innocuous; others are pernicious

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 26 / 181

Part III The Role of DDA for Pricing

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 27 / 181

The Role of DDA for Pricing

Two Types of Data Analysis

Data Analysis Shallow

• Means & Std. Errors
• Simple Charts
• Much Left Hidden

Deep

• Data Structure
• Statistics/ML
• Reveals Hidden

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 28 / 181

The Role of DDA for Pricing

Information Continuum

Poor Information Rich Information

• Means
• Graphs
• Regressions
• Machine Learning
• Proportions
• Tabs
• Elasticities
• Predictive Modeling

OLS Regression Shallow Data Analysis Deep Data Analysis

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 29 / 181

The Role of DDA for Pricing

Deep Data Analytics

Deep Data Analytics is the process of taking raw data bricks/Poor Information, regardless of source, and assembling/converting them into Rich Information using advanced statistical, econometric, and machine learning methods applicable to a data set’s structure.

Advantage of Deep Data Analytics

Shallow Data Analyses leave untapped information. Deep Data Analyses reveal Rich Information.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 30 / 181

The Role of DDA for Pricing

There’s No Such Thing as a Free Lunch

Rich Poor Information Continuum Cost Cost of Approximation Base Approximation Cost Base Analysis Cost Cost of Analytics Cost of Analytics: DDA

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 31 / 181

Part IV DDA Drill-down and Case Study

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 32 / 181

DDA Drill-down and Case Study

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 33 / 181

DDA Drill-down and Case Study Identify Data Structure

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 34 / 181

DDA Drill-down and Case Study

Data structure and variables are often (most of the time?) overlooked. Both have major implications.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 35 / 181

DDA Drill-down and Case Study

Data structure is not multiple data tables. Customer DB Order DB Product DB SQL Engine Pricing Data Mart (PDM) This is data organization, not structure. It’s a convenience for storage and parsing – not analytics.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 36 / 181

DDA Drill-down and Case Study

This database framework is important, but not my view of data structure. The PDM from the database structure is part of the Analytical Toolset. Data structure is deeper although, on the one hand, it is shallow or simplistic but, on the other hand, it is complex and subtle.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 37 / 181

DDA Drill-down and Case Study

Some variables impose a structure, explicitly or implicitly. Dummy variables impose structure by dividing the data.

This is an explicit structural definition.

Example

Gender, age grouping, commercial/residential, large/small

Clustering of cases or variables also divides the data.

This is an implicit structural definition. The structure is hidden or latent and must be revealed.

Example

Segmentation (not a priori)

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 38 / 181

DDA Drill-down and Case Study Case Study: Overview

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 39 / 181

DDA Drill-down and Case Study

Stores come in different sizes (i.e., selling surfaces). Small storefronts (e.g., Mom & Pops) to ”Big Box” stores (e.g., warehouse clubs). Even within one chain, store sizes vary, perhaps due to geography.

Example

Grocery stores in Manhattan vs. suburbs of Central Jersey.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 40 / 181

DDA Drill-down and Case Study

Store size impacts ”sales lift” forecasting from price promotions. If ηQ

P is the price elasticity, then ηTR P

= 1 + ηQ

P .

Sales lift: LIFT = ηTR

P

× (%∆P) = (1 + ηQ

P ) × (%∆P).

Example

If ηQ

P = −1.8, then ηTR P

= −0.8. If %∆P = −0.25, then LIFT = −0.8 × (−0.25) = 0.2 or 20%.

Overestimate lift: Storage costs or perishable losses. Underestimate lift: Stock-outs and lost sales, good-will.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 41 / 181

DDA Drill-down and Case Study

Question

What is the elasticity allowing for store size?

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 42 / 181

DDA Drill-down and Case Study

Retailers are moving to ”customized pricing practices . . . in which pricing depends on store size and clientele.” Evidence shows that retailers ”price promote more intensely in their large stores” but the evidence is shaky.2

Large stores tend to be in suburban areas; small in urban. Available real estate is the issue.

Implication: more elastic in large stores, but not clear-cut.

More elastic in urban areas because of intense competition. More inelastic in suburban areas because of value of time for shopping.

Would have to drive to next shopping mall which is time consuming. If value of time out-weighs a price saving, then more inelastic.

2See Haans and Gijsbrechts (2011, 428). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 43 / 181

DDA Drill-down and Case Study

Side Effects by Store Size

Large Stores Small Stores Benefits Benefits

• Increased parking
• Personal treatment
• Additional services
• More competition
• Wider variety
• Neighborhood focus
• One-stop shopping
• Wider variety of stores

Costs Costs

• Longer distance to travel
• Smaller variety
• More/longer aisles
• Fewer products
• Longer checkout time
• Frequent store entry/exit
• Higher in-store search
• Higher store search

These will affect price elasticities, but impact is unclear.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 44 / 181

DDA Drill-down and Case Study Case Study: Background

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 45 / 181

DDA Drill-down and Case Study

Fictional data on a retail chain in New England states. One consumer product Six stores

3 Urban (Small) 3 Suburban (Large)

600 consumers

Each consumer’s purchases and prices averaged to one annual number so n = 600. Will discuss aggregation later.

Consumer data:

Average price paid Household income Average purchase size

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 46 / 181

DDA Drill-down and Case Study Case Study: Pooled Model

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 47 / 181

DDA Drill-down and Case Study

A simple, naive model is a pooled regression based on a Stat 101 data structure. The structure is a simple rectangular array of rows and columns.

A simple ”tidy” structure in R terminology.3

1

Each variable in the data set is placed in its own column.

2

Each observation is placed in its own row.

3

Each value is placed in its own cell.

A Stat 101 – Tidy Data Structure

3See Wickham and Grolemund (2017). The chart and rules are on p. 149. Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 48 / 181

DDA Drill-down and Case Study

The model based on this simple data structure is:4 Qi = eβ0 × Pβ1

i

× I β2

i

× eǫi

• r

ln Qi = β0 + β1 × ln Pi + β2 × ln Ii + ǫi where Pi is the price paid by the ith consumer and Ii is that consumer’s income. The model is ”pooled” since the structure is simple, naive. The price elasticity is simply β1.

4See the Appendix for a discussion of this model. Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 49 / 181

DDA Drill-down and Case Study

Price elasticity: −2.4

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 50 / 181

The Components of a Pricing Strategy: Stores

Pricing Structure

• Uniform

Pricing Strategy Pricing Level Effect Assessment η = −2.4

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 51 / 181

DDA Drill-down and Case Study

This is an example of Shallow Data Analysis.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 52 / 181

DDA Drill-down and Case Study Case Study: Dummy Variable Model

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 53 / 181

DDA Drill-down and Case Study

Analysts often impose structure with dummies.

Example

Segment consumers into homogeneous groups, say J segments.

Groups could be a priori or derived.

Pool all consumers into one model with J − 1 dummies to identify the groups. Let: Yi = Purchase Intent Xi = Individual trait or price. Then for J = 4 segments: Yi = β0 + β1Xi + γ1D1 + γ2D2 + γ3D3 + ǫi Effect is to shift intercept for each segment, but maintain the same slope.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 54 / 181

DDA Drill-down and Case Study

You could interact dummies with the independent variable(s) to change the slope(s): Yi = β0 + β1Xi + γ1D1 × Xi + γ2D2 × Xi + γ3D3 × Xi + ǫi Or, you could do both: Yi = β0 + β1Xi + γ1D1 + γ2D2 + γ3D3+ γ4D1 × Xi + γ5D2 × Xi + γ6D3 × Xi + ǫi Regardless of model specification, you can calculate elasticities.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 55 / 181

DDA Drill-down and Case Study

Location dummy as proxy for store size can be added to the basic store

• model. A model is:

Qi = eβ0+γ1×Locationi × Pβ1+γ2×Locationi

i

× I β2

i

× eǫi

• r

ln Qi = β0 + γ1 × Locationi + β1 × ln Pi + γ2 × Locationi × ln Pi + ǫi where Locationi =

• 1 if Suburban

0 if Urban

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 56 / 181

DDA Drill-down and Case Study

Store price elasticities:5 β1 : Urban β1 + γ2 : Suburban

5See the Appendix for the elaticities. Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 57 / 181

DDA Drill-down and Case Study

Urban elasticity: −1.2 Suburban: −0.6

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 58 / 181

The Components of a Pricing Strategy: Stores

Pricing Structure

• 3rd Discrimination

Pricing Strategy Pricing Level Effect Assessment

• Urban: η = −1.2
• Suburban: η = −0.6
• Urban: Low Price
• Suburban: High Price

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 59 / 181

DDA Drill-down and Case Study

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 60 / 181

DDA Drill-down and Case Study

Problems with dummy variable approach:

1 Proliferation of parameters from dummies. 2 Inefficient use of data – ignores a hierarchical data structure. 3 Shifts in parameters (intercepts and slopes) are fixed effects. 4 Does not allow for key drivers for the dummies.

Everyone in a segment behaves the same way – but what drives that behavior?

For the stores example, not all stores (or customers) are included; only a sample of stores is used. Random effects due to sampling are excluded.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 61 / 181

DDA Drill-down and Case Study Latent Regression Analysis: Another View of Structure

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 62 / 181

DDA Drill-down and Case Study

A data structure may be hidden or latent. No explicit variables such as location or size. Need to uncover or reveal a latent structure.

This structure is implicit in the data.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 63 / 181

DDA Drill-down and Case Study

You can estimate segments and elasticities simultaneously using latent class regression.6 This makes more efficient use of the data. This class of models tries to find a latent variable that explains or determines or drives variables we can measure.

Example

You cannot measure or observe religious preference, a latent variable, but this may determine consumption patterns of some products.

6See Paczkowski (2018) for an extensive discussion. Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 64 / 181

DDA Drill-down and Case Study

7

7Based on Collins and Lanza (2010, 5). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 65 / 181

DDA Drill-down and Case Study

There is a plethora of (sometimes confusing) types. Latent Class Analysis (LCA)

Dependent variable is discrete/categorical.

Latent Profile Analysis (LPA)

Latent Regression Analysis (LRA)

All have a similar characteristic – there is a hidden, unknown factor or class or segment or group that drives or determines the results we see. What we see or observe is sometimes said to be realized. The results point to or indicate the latent class.

Observed behaviors provide clues to the latent variables that drive those behaviors.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 66 / 181

DDA Drill-down and Case Study

The Latent Regression model recognizes that a plain regression model on a super-population without allowing for classes involves estimating a single set of parameters across all observations. Estimate a pooled model This may be misleading if observations come from a number of unknown heterogeneous groups with different parameter values, θS. Latent Regression has been developed consistent with the ordinary OLS family:

normal; binary; and count.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 67 / 181

DDA Drill-down and Case Study

Key Assumption

There is one discrete latent variable with several classes or segments. Each individual belongs to a class but which one is unknown.

The indicator variables point to a class but they could just as well point to several classes at once. The error associated with each indicator variable helps to hide which class. So there is a probability that an individual belongs to a class with sum

• f probabilities over the classes equaling 1.0.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 68 / 181

DDA Drill-down and Case Study

Assume we have n objects (e.g., people or firms in a survey or database) and that the ith object has Ti records or observations where the number may differ for each object. The total number of observations is T = n

i=1 Ti.

Each object has one response for each record, yit, t = 1, . . . , Ti.

Example

A store i has Ti = 12 responses for 12 months. These responses can be continuous, nominal, or counts. Each response appears in its own record with an ID variable connecting them. This is a repeated measures format.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 69 / 181

DDA Drill-down and Case Study

There are two types of independent variables.

1 Predictors

Q predictors: zpred

itq , 1 ≤ q ≤ Q.

The predictors may vary by object i and repeated measure t so they are used to predict the response.

Example

Price is a predictor.

2 Covariates

R covariates: zcov

ir

, 1 ≤ r ≤ R. The covariates do not vary by repeated measure for an object, but they do vary by objects. The covariates help predict class membership.

Example

Firm size and other firmographics are covariates.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 70 / 181

DDA Drill-down and Case Study

This is a two-level data structure. Low-level replications within a high-level object. The predictor variables are for the low-level replications. The covariates are for the high-level objects.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 71 / 181

DDA Drill-down and Case Study

For the latent variable, there is one variable x with K categories or levels

• r classes or segments, 1 ≤ k ≤ K.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 72 / 181

DDA Drill-down and Case Study

The regression model is from the GLM family with parameters differing across the latent classes. This is how we get price elasticities by segments or classes. The GLM family includes a number of members. In statistics, the generalized linear model (GLM) is a flexible generalization of ordinary linear regression that allows for response variables that have error distribution models other than a normal distribution. The GLM generalizes linear regression by allowing the linear model to be related to the response variable via a link function and by allowing the magnitude of the variance

• f each measurement to be a function of its predicted value.8

8https://en.wikipedia.org/wiki/Generalized linear model. Last accessed August 11,

2015

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 73 / 181

DDA Drill-down and Case Study

The model structure is not simple – there are two parts.

1 General probability structure.

Explains how the responses (i.e., dependent variable) are generated. This is a general mixture model probability structure that defines the relationships between the exogenous, latent, and response variables. This is a probability density corresponding to a particular set of yi values given a particular set of exogenous values.

2 Conditional distributions.

An assumed distributional form for the response variables, which depends on the scale types of the variables concerned.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 74 / 181

DDA Drill-down and Case Study

The general probability structure is: f (yit | zi) =

K

• x=1

Pr(x | zcov

i

) × f (yit | x, zpred

it

) =

K

• x=1

Pr(x | zcov

i

) ×

Ti

• t=1

f (yit | x, zpred

it

) where the responses are assumed to be independent and f is a probability density function. The Pr(x | zcov

i

) is a mixture weight.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 75 / 181

DDA Drill-down and Case Study

Since the basic probability structure is conditional, we need the conditional distributions. For continuous dependent variables, we use the normal distribution.

For nominal data we use a Binomial. For count data we use a Poisson.

The normal distribution can be a truncated normal if yit > 0 or a censored normal if yit ≥ 0 but with many yit = 0.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 76 / 181

DDA Drill-down and Case Study

The latent variable probabilities (i.e., the mixture weights) are multinomial: Pr(x | zcov

it

) = eηx|zcov

it

• x′=1 Ke

ηx′ |zcov

it

where the η functions are linear in the parameters combinations of the covariates.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 77 / 181

DDA Drill-down and Case Study

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 78 / 181

DDA Drill-down and Case Study

Model estimation is based on maximum likelihood.9 Estimation requires knowing how many classes exist. We don’t know this – this is part of the problem to solve. Procedure is to specify the number of classes, K, and examine basic fit statistics to determine which models do better. Typical measures are AIC and BIC. AIC = −2 × ln L + 2 × (Number of Parameters) BIC = −2 × ln L + (ln N) × (Number of Parameters) where N is the total sample size. Choose the model with the lowest AIC or BIC.

9Use an EM algorithm for computation (sometimes augmented by Newton-Raphson). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 79 / 181

DDA Drill-down and Case Study

Once the parameters have been estimated, posterior class membership probabilities can be calculated. These give the probability of each object belonging to a class. For each object, the probabilities sum to 1.0 and give the chance of the object belonging to each class.

This means that we have a ”fuzzy” solution unlike for hierarchical clustering and decision trees where the object is assigned to a single, unique class.10

10There are fuzzy clustering approaches but not widely used. Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 80 / 181

DDA Drill-down and Case Study

The predictor variables are used to estimate the parameters for the latent classes – we still need to predict or profile or characterize the classes. In an explanatory study, we want to predict class membership. In a descriptive study, we want to profile the classes based on a set of variables. Price segmentation is really a combination since profiling is just as important as predicting considering the marketing mix concept.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 81 / 181

DDA Drill-down and Case Study

There are two ways to handle the covariates.

1 One-step Approach

Estimate a model with predictor and covariate variables. Involves estimating the class model and the mixing probabilities simultaneously.

This is the framework I described above.

Most software in this area allows this.

2 Three-step Approach 1

Estimate class model with just predictor variables

2

Assign objects to a class.

3

Estimate separate (usually logit) model of class membership using the covariates.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 82 / 181

DDA Drill-down and Case Study

The one-step approach has an advantage — it’s one step!. But it has major issues:

1 It is impractical when the number of covariates is large as is typical in

many explanatory studies.

Each time a covariate is added or deleted, the model must be reestimated.

2 You have to decide on the type of model: with or without covariates. 3 Most researchers view modeling as adding covariates after the classes

are developed – profiling from the old cluster/discriminant approach.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 83 / 181

DDA Drill-down and Case Study

The three-step approach involves:

1 Estimating the latent classes. 2 Assign subjects to a class using the posterior probabilities based on

the observed responses and the estimated parameters from the first

• step. There is one posterior probability per class and the probabilities

sum to 1.0. This holds for each object.

Modal approach: assign to the class with the largest posterior probability. Proportional assignment.

3 Regress the estimated class memberships on the covariates.

Assumes that the assignments are actual memberships.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 84 / 181

DDA Drill-down and Case Study

Case Study: Several models were estimated. Models with more than four segments had very small segment sizes (< 1%) so dropped anything with more than four. Three segments did well, but didn’t seem practical for this market.

So stayed with four segments.

Estimated with and without covariates.

Without covariates, segment four was < 1%, so kept covariates.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 85 / 181

DDA Drill-down and Case Study

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 86 / 181

DDA Drill-down and Case Study

Elasticity Summary

Segment Elasticity 1

• 1.04

2

• 0.46

3

• 2.10

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 87 / 181

The Components of a Pricing Strategy: Stores Extended

Pricing Structure

• Price Segmentation

Pricing Strategy Pricing Level Effect Assessment η = −1.04, −0.46, −2.10

• By Segment

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 88 / 181

DDA Drill-down and Case Study

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 89 / 181

DDA Drill-down and Case Study

The LR model reveals latent structures. Have different parameter estimates by groups. But the parameter estimates are not themselves functions of variables

• r key drivers.

There is no context for the parameters or the groups.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 90 / 181

DDA Drill-down and Case Study Multilevel Structure

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 91 / 181

DDA Drill-down and Case Study

There are two data structures:

1 Non-Nested 2 Nested or Multilevel

Definition

Non-nested Data Structure: The data in the population are at the same

• level. The sample data are from a SRS process.

Definition

Nested/Multilevel Data Structure: The data in the population are

• hierarchical. The sample data are from a multistage sampling process.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 92 / 181

DDA Drill-down and Case Study

For a non-nested data structure, the central statistical model is successive sampling from one level only. For one level, there is no context for the behavior of the measurement units in the sample.

Example

All consumers in a random sample are the same. Their behavior is driven solely by their traits – and the prices they see. Variables can, of course, be aggregated or disggregated to a different level.

Aggregation or disaggregation are sometimes done to hide/avoid data complexity.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 93 / 181

DDA Drill-down and Case Study

Not all data are created equal. In simple statistical analysis, say Stat 101 descriptive statistics and basic OLS, all data are at the same level. This means that some data must be aggregated and others disaggregated to put them on the same level.

Definition

Aggregation: Taking data at a low level and redefining its values to be used at a higher level. For example, averaging income at the individual household level and using the mean for marketing region average household income.

Definition

Disaggregation: Taking data at a high level and redefining it to be used at a lower level. For example, dividing quarterly income by 3 and using the result as average monthly income or taking marketing region sales and dividing by the number of states in a region to get average state sales.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 94 / 181

DDA Drill-down and Case Study

Example

Time Series: Convert from one frequency to another.a

aFrequency: Number of measurements per year.

The Case Study weekly sales (high frequency data) of 600 consumers were aggregated to an annual number (low frequency data).

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 95 / 181

DDA Drill-down and Case Study

Example

Sales: Convert from store to region and vice versa.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 96 / 181

DDA Drill-down and Case Study

Aggregation and disaggregation are very common. Data at one level are ”moved” to another so all the data are at one level. Standard statistical/econometric methods (e.g., OLS, ANOVA) are then applied. But there are problems with aggregation and disaggregation. Aggregation may make the problem ”easier.” Less data to manage. Avoid problems such as autocorrelation with time series.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 97 / 181

DDA Drill-down and Case Study

Aggregation problems: Information, what is needed for decision making, is made more hidden/obscure.

Recall that information is buried inside the data and must be extracted.

Statistically, there is a loss of power for statistical tests and procedures. Disaggregation problems: Data are ”blown-up”. Statistical tests assume they are independent draws from a distribution, but they are not since they have a common base thus violating this key assumption. Also, sample size is affected since measures are at a higher level than what the sampling was designed for.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 98 / 181

DDA Drill-down and Case Study

There are also two subtle issues associated with converting to a single level:

1 Ecological Fallacy; and 2 Atomistic Fallacy.

Simpson’s Paradox could also be an issue.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 99 / 181

DDA Drill-down and Case Study

Definition

Ecological Fallacy: Aggregated data are used to draw conclusions about disaggregated units.

Example

Model sales and estimate price elasticities at the marketing region and use these elasticities to price at the store level. What holds at the region level may not hold at the store level.

Each store has its own defining characteristics

Clientele SES Size Local preferences

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 100 / 181

DDA Drill-down and Case Study

Definition

Atomistic Fallacy: Disaggregated data are used to draw conclusions about aggregate units.

Example

Model individual consumers from Big Data and apply the results to a whole market.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 101 / 181

DDA Drill-down and Case Study

Many problems involve nested or multilevel or hierarchical data structures not at one level.11 Data are measured at a lower level but within the context of a higher level.

Terminology

Low Level: Level 1 or micro level High Level: Level 2 or macro level. The macro level gives context or meaning to the micro level: the macro influences the micro.

11The terms ”nested”, ”multilevel”, ”hierarchical” are used interchangeably. I prefer

”multilevel.”

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 102 / 181

DDA Drill-down and Case Study

Multilevel Data Structure: Two Levels

Macro (Level 2) Micro 2 (Level 1) Micro 1 (Level 1) Micro 3 (Level 1)

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 103 / 181

DDA Drill-down and Case Study

Connection to Two Main Fallacies

Micro Level Macro Level Atomistic Fallacy Draw Conclusions About Higher Level Ecological Fallacy Draw Conclusions About Lower Level Disaggregation Aggregation

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 104 / 181

DDA Drill-down and Case Study

Non-Nested Determinants of Purchasing: Consumers

Adapted from Luke (2004, 5). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 105 / 181

DDA Drill-down and Case Study

Non-Nested Determinants of Purchasing: Businesses

Adapted from Luke (2004, 5). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 106 / 181

DDA Drill-down and Case Study

Definition

Data are hierarchical if they naturally form levels proceeding from micro or individual level data to more macro or aggregate levels.

Example

Consumers shopping in a store. Households in a marketing region. Purchasing managers in a company business unit or in an industry.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 107 / 181

DDA Drill-down and Case Study

In a multilevel structure, the statistical model is successive multi-stage sampling. There is a hierarchy of levels that give context for the behavior of the measurement units in the sample. Behavior is driven by their traits – and prices – as well as their context (i.e., environment).

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 108 / 181

DDA Drill-down and Case Study

Nested/Multilevel Determinants of Purchasing: Consumers

Adapted from Luke (2004, 5). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 109 / 181

DDA Drill-down and Case Study

Nested/Multilevel Determinants of Purchasing: Businesses

Adapted from Luke (2004, 5). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 110 / 181

DDA Drill-down and Case Study

Definition

Global variables are those that refer only to the level they are defined for. Example: Gender at the individual level.

Definition

Relational variables are those at one level but describe the relationship across units at that level. For example, an index, perhaps derived from a principal components analysis (PCA) of demographic variables, would be relational because it would show the relationship among all the units at

• ne level.

Definition

Contextual variables are those for higher levels so that all units at lower levels have the same values for these variables. For example, Real GDP is contextual because all consumers and businesses would be exposed to the exact same values for Real GDP.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 111 / 181

DDA Drill-down and Case Study

Examples of hierarchical structures are more common in marketing and pricing than thought:12 Segments Stores Marketing regions States Neighborhoods Organization membership Brand loyalty Many more could be listed.

12See Oakley et al. (2005) and Ray and Ray (2008). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 112 / 181

DDA Drill-down and Case Study

Some data are naturally hierarchical or nested: Family Neighborhood Store City Segment You need to account for the nesting and interactions between the nests.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 113 / 181

DDA Drill-down and Case Study Data Visualization

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 114 / 181

DDA Drill-down and Case Study

It is well known that you must graph your data to fully understand any messages inside the data: relations, trends, patterns, anomalies. Typical scatterplots used with hierarchical data will not suffice.

The data sets are usually very large

The groups would be obscured/hidden.13

13Chart source: Bell (2001, 3). Also see Finch, et al. (2014). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 115 / 181

DDA Drill-down and Case Study

Explore the data using: Lattice/Panel/trellis plots for:

Dot plots Scatterplots (Yes! But in trellis form)

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 116 / 181

DDA Drill-down and Case Study

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 117 / 181

DDA Drill-down and Case Study

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 118 / 181

DDA Drill-down and Case Study

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 119 / 181

DDA Drill-down and Case Study

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 120 / 181

DDA Drill-down and Case Study Modeling Hierarchically Structured Data

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 121 / 181

DDA Drill-down and Case Study

A more efficient model structure is needed that reflects the data structure. There is a two-stage system of equations.14 Stage I Equation: Yij = β0j + β1jXij + ǫij where: ǫij ∼ N(0, σ2); Yij is the outcome variable for individual i in group j; Xij is the individual-level variable for individual i in group j; β0j is the group-specific intercept; and β1j is the group-specific effect or slope of the individual-level variable. Notice that the parameters vary by group.

14The following draws from Roux (2002). Also see Gelman and Hill (2007). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 122 / 181

DDA Drill-down and Case Study

Stage II Equations: β0j = γ00 + γ01Gj + U0j β1j = γ10 + γ11Gj + U1j where: Gj is a group-level variable; γ00 is the common intercept across groups; γ01 is the effect of the group-level predictor on the group-specific intercept; γ10 is the common slope associated with the individual-level variable across groups; and γ11 is the effect of the group-level predictor on the group-specific slopes. These γ parameters are called hyperparameters.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 123 / 181

DDA Drill-down and Case Study

The error terms in the Stage II Equations are called macro errors. U0j ∼ N(0, τ 2

00)

U1j ∼ N(0, τ 2

11)

There could be a covariance between the intercepts and slope which is represented by τ01.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 124 / 181

DDA Drill-down and Case Study

The group-specific parameters have two parts:

1 A ”fixed” part that is common across groups: γ00, γ01 for the

intercept and γ10 and γ11 for the slope; and

2 A ”random” part that varies by group: U0j for the intercept and U1j

for the slope. The underlying assumption is that the group-specific intercepts and slopes are random samples from a normally distributed population of intercepts and slopes.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 125 / 181

DDA Drill-down and Case Study

A reduced form of the model is:15 Yij = γ00 + γ01Gj + γ10Xij + γ11Gj × Xij

• Fixed Component

+ U1j × Xij + U0j + ǫij

• Random Component

15See Roux (2002). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 126 / 181

DDA Drill-down and Case Study

Many variations are possible: Null Model: no explanatory variables; Intercept varying, slope constant; Intercept constant, slope varying; and Intercept varying, slope varying. The Null Model is particularly important because it acts as a baseline model – no individual effects.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 127 / 181

DDA Drill-down and Case Study

This is a more complicated, and richer, model. The random component for the error is a composite of terms, not just

• ne term as in a Stat 101 OLS model.

A dummy variable approach to modeling the hierarchical structure would not include this composite error term. The dummy variable approach is incorrect – there is a model misspecification The correct specification has to reflect random variations at the Stage I level as well as at the Stage II level and, of course, any correlations between the two.

The composite error term contains an interaction between an error and the Stage I predictor variable which violates that OLS Classical Assumptions.

A dummy variable OLS specification would not do this.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 128 / 181

DDA Drill-down and Case Study

Definition

Intraclass Correlation Coefficient (ICC) ρI = τ 2

00

τ 2

00 + σ2

with 0 ≤ ρI ≤ 1. τ 2

00 is the between-group variability and σ2 is the

within-group variability. This result comes from the Null Model.16 It measures the association of data within a cluster/group/segment/level.

16See the Appendix for details. Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 129 / 181

DDA Drill-down and Case Study

If the within-group variance approaches 0, ρI approaches 1. When ρI = 1, all cases within a group are identical – they have the same responses. If ρI → 0, then the within-group variance dominates and there is no correlation among cases within groups so no need to deal with a multilevel structure.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 130 / 181

DDA Drill-down and Case Study

The size of ρI has implications for the effective sample size for analysis and hypothesis testing. The effective sample size (ESS) is given by:17 ESS = m × k 1 + ρI × (m − 1) where m is the number of subjects within a group, k is the number of groups, and n = m × k is the total sample size. If ρI = 0, then ESS = m × k. If ρI = 1, then ESS = k. So if ρI = 1 and k = 5, then all cases within a group are identical and so it is as if there is

• nly one case per group. With 5 groups, this is tantamount to having only

5 cases or observations.

• 17S. Killip et al. (2004)

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 131 / 181

DDA Drill-down and Case Study

The implication is that the probability of an incorrect decision is greater the larger is the ICC. If ρI → 1 then ESS → k and the standard error of an estimate gets larger.18 Sample Size Square Root

• Std. Error

100 10 0.1 5 2.2 0.4 The larger the standard error, the smaller the value of a test statistic. The smaller the value of a test statistic, the larger the probability of a false negative: not rejecting the Null Hypothesis when it should be rejected.

18The standard error is proportional to the inverse of the square root of the sample

size.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 132 / 181

DDA Drill-down and Case Study

Hierarchical Structure: Retail Store

Store Customer Macro or Level 2 Micro or Level 1 Stores are a random sample of a population of stores in the retail chain.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 133 / 181

DDA Drill-down and Case Study

A dummy variable approach would classify store size into small/large or urban/suburban. But store size, as a macro or level 2 context or ecological factor may by a proxy for hard-to-measure factors: customer characteristics, types of shopping trips (i.e., goals), shopping time constraints and so should be used. A product (or product category such as cereal) is nested in a store which is characterized by size.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 134 / 181

DDA Drill-down and Case Study

Multilevel Model with Varying Intercepts

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 135 / 181

DDA Drill-down and Case Study

P-Value Summary

Covariance Parameter Subject Estimate Std Error t-stat p-Value Var(Intercept) store 0.0608 0.0395 1.5387 0.0619 Cov(Intercept,log Price) store

• 0.1271

0.0907

• 1.4014

0.9195 Var(log Price) store 0.2108 0.2314 0.9112 0.1811 Residual 0.0208 0.0012 17.1230 <0.0001

Variation in log quantity across different sizes is marginally significant while the covariance between log price and size is insignificant.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 136 / 181

DDA Drill-down and Case Study

Estimated parameters for store 1 (397 square feet): Net Intercept = 2.47 + 0.11 = 2.58 Price Effect = −0.91 + 0.02 = −0.89 Model: ln Quantity = 2.58 − 0.89 × ln Price + 0.62 × ln Income Price Elasticity: -0.89

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 137 / 181

DDA Drill-down and Case Study

Price Elasticity Summary

Store Net Intercept Net Price Elasticity 1 2.58

• 0.89

2 2.68

• 1.40

3 2.78

• 1.80

4 2.36

• 0.38

5 2.15

• 0.41

6 2.28

• 0.57

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 138 / 181

DDA Drill-down and Case Study

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 139 / 181

DDA Drill-down and Case Study Modeling Extensions for Multilevel Data

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 140 / 181

DDA Drill-down and Case Study

You are not restricted to a linear model of sales. Suppose the PDM contains data on bids as well as completed sales. The company could operate on a contract basis so bids are a normal way to make a sale.

Contractors, light and heavy construction equipment suppliers, plumbing/electrical/materials companies are some examples.

What should also be maintained is whether or not the bid was won or loss.

In many instances, if the bid was won, then it is entered into a PDM, but not if it is lost. A loss is considered a loss and there is no reason to record it.

A win-loss variable would be a dummy variable: 1 if the bid was won and 0 if lost. The remaining data are the same. Competitive data would be very valuable because this could help account for the loss and be used in modeling.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 141 / 181

DDA Drill-down and Case Study

A linear model such as Yi = α + βXi is inappropriate when Yi is binary because the disturbance variances are heteroskedastic and there is a chance of predicting outside the range of Yi which is just 0 and 1.19 The logistic cumulative distribution function overcomes these issues and gives rise to. Other distribution functions can be used, but this is the most common in practice.

19See Gujarati (2003) on the inappropriateness of a linear model for this problem. Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 142 / 181

DDA Drill-down and Case Study

A logistic model specifies the probability of a win as a function of explanatory variables. If the win-loss variable is Yi for the ith bid, then Logistic = Pr(Yi = 1) = eα+βXi 1 + eα+βXi

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 143 / 181

DDA Drill-down and Case Study

The logistic model is nonlinear. Estimation is simplified by a transformation involving odds which are defined as the ratio of the probability of a win over the probability of a loss. The probability is estimated using the log of these odds, called the log

• dds or logit (short for logarithmic transformation).

The logit is the linear function of the independent variables which is much easier to work with.20 The logistic statement and the logit statement are inverses of each

• ther so:

Logistic = Pr(Yi = 1) = logit−1(α + βXi).

20For this model, odds =

eX 1 + eX × 1 + eX 1 = eX where the second ratio is the inverse

• f the probability of a loss.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 144 / 181

DDA Drill-down and Case Study

The logistic model can be easily extended to a multilevel situation for varying-intercepts as Logistic = Pr(Yi = 1) = logit−1(αij + βXi) for i = 1, . . . , nj and j = 1, . . . , J. A varying-intercepts, varying-slopes model is Logistic = Pr(Yi = 1) = logit−1(αij + βijXi) with αj = γα

0 + γα 1 Zj + ǫα j

βj = γβ

0 + γβ 1 Zj + ǫβ j

where Zj term is an independent variable for the macro level that affects the intercepts and slopes but not the win-loss.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 145 / 181

DDA Drill-down and Case Study

You may have just counts of events.

Example

The number of orders – a count – taken by a sales rep in one year. The

• rders would be a function of the price but you cannot ignore the

characteristics of the rep. The number of orders would be different by rep with different personalities and network. The orders are nested in the rep. The orders as a count would follow a Poisson Process.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 146 / 181

DDA Drill-down and Case Study

The basic Poisson Process is yi ∼ Poisson(θi) θi = exp(β0 + β1Xi) where Xi could be a price point.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 147 / 181

DDA Drill-down and Case Study

In Poisson models, the variance equals the mean so there is no independent variance parameter, σ2

i .

The implication is that you could have a variance larger than what is predicted by the model. The Poisson model is said to exhibit overdispersion because there is no variance parameter ”to capture the variation in the data.”21

21Gelman and Hill (2007, 325). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 148 / 181

DDA Drill-down and Case Study

We can extend the Poisson to multilevel data as yi ∼ Poisson(µieβXi+ǫi) ǫi ∼ N0, σ2

ǫ

The σ2

ǫ captures the overdispersion; σ2 ǫ = 0 is the classical Poisson.22

22See Gelman and Hill (2007) and Snijders and Bosker (2012). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 149 / 181

DDA Drill-down and Case Study

One final extension: panel data.23 Recall the store example: data were total expenditures for customers by store for one year. This is the case where we have multiple repeated measures of the same variable for the same unit where the units are a sample. For the store example, there would be a three-level model:

1 time measures within an individual; 2 individuals nested within stores; and 3 stores. 23Also known as longitudinal data or time series-cross sectional data or repeated

measures data.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 150 / 181

DDA Drill-down and Case Study

Panel models are far more complicated because of potential autocorrelation and the presence of the additional level of nesting.24

24See Snijders and Bosker (2012) for a good discussion. Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 151 / 181

DDA Drill-down and Case Study

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 152 / 181

Part V Organizing for DDA for Pricing

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 153 / 181

Organizing for DDA for Pricing

Most discussions about pricing involve organizing around narrow pricing issues: motivating the sales force; having convincing messages about levels and changes; how to motivate customers; and the list goes on. I refer to these as the Management Culture for pricing.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 154 / 181

Organizing for DDA for Pricing

My focus is organizing for DDA that is behind the price number. There must be a Data Culture for pricing. The two cultures are needed and they must work in unison to achieve the business goals. The Data Culture with DDA at its core has been downplayed or ignore (maybe purposely?).

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 155 / 181

Organizing for DDA for Pricing

Pricing Development

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 156 / 181

Organizing for DDA for Pricing

The Components of a Pricing Strategy

Pricing Structure

• Uniform
• Discriminatory

Pricing Strategy Pricing Level Effect Assessment Management Culture Data Culture (DDA Core)

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 157 / 181

Organizing for DDA for Pricing

A Data Culture must be developed - you cannot:25 import a culture; impose or order a culture. You develop a Data Culture by working at the organization level: moving beyond specialists and ”skunkworks”; motivating a deep business engagement and collaboration; creating employee pull; having DDA support the entire organization.

25See Diaz et al. (2018). Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 158 / 181

Organizing for DDA for Pricing

Data Culture

Data culture can accelerate the application of analytics, amplify its power, and steer companies away from risky outcomes.a

aBased on Diaz et al. (2018, 1).

The ”risky outcome” is the cost of making a mistake because of Poor Information.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 159 / 181

Organizing for DDA for Pricing

There’s No Such Thing as a Free Lunch

Rich Poor Information Continuum Cost Cost of Approximation Base Approximation Cost Base Analysis Cost Cost of Analytics:

• Mgmt. Culture

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 160 / 181

Organizing for DDA for Pricing

There’s No Such Thing as a Free Lunch

Rich Poor Information Continuum Cost Cost of Approximation Base Approximation Cost Base Analysis Cost Cost of Analytics:

• Mgmt. Culture

Cost of Analytics:

• Mgmt. + Data Cultures

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 161 / 181

Organizing for DDA for Pricing

Not all businesses embrace a Data Culture so Pricing Strategy (and all business decisions) flow from a Management Culture only. Rather than the gap between those who do and those who do not use data analytics in general is not shrinking – it is growing!

The Data Use Gap

The gap between leaders and laggards in adopting analytics, within and among industry sectors, is growing.a

aDiaz et al. (2018, 1).

Some businesses are overwhelmed with executives and employees questioning the benefits of a role for data analytics, let alone DDA.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 162 / 181

Organizing for DDA for Pricing

A Data Ecosystem is needed. This consists of: data collection and organization; data accessibility; data analytic skill sets; data analytic tool sets; and data collaboration across organizations.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 163 / 181

Organizing for DDA for Pricing

Collaboration across multiple disciplines and functions, especially when repeated and well orchestrated, reduces the Costs of Deep Data Analytics.

Sharing Benefits

”The characteristics of information — be it software, text or even biotech research — make it an economically obvious thing to share. It is a ”non-rival” good: ie, your use of it does not interfere with my use. Better still, there are network effects: i.e., the more people who use it, the more useful it is to any individual user. Best of all, the existence of the internet means that the costs of sharing are remarkably low. The cost of distribution is negligible, and co-ordination is easy because people can easily find others with similar goals and can contribute when convenient.”a

aBased on http://www.economist.com/node/3623762.

This is the classic returns to Division of Labor.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 164 / 181

Organizing for DDA for Pricing

Pricing Should Be Multicollaborative

Pricing

PD/I&S26 Marketing Procurement Legal IT Global Supply Chain

26Product Development and Implementation & Support Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 165 / 181

Organizing for DDA for Pricing

Three Stages of DDA and Functional Areas

Structure

• Market Research, Marketing, Sales,

Management Visualize

• Data Science, Stakeholders

Model

• Data Science, Stakeholders, Manage-

ment

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 166 / 181

Organizing for DDA for Pricing

Skill Sets for DDA

Structure

• Market Knowledge, Customer Behav-

ior Visualize

• Data Science Expertise, Complex

Graph Understanding Model

• Data Science, Statistics, Econometrics

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 167 / 181

Part VI Summary

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 168 / 181

Summary

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 169 / 181

Part VII Contact Information

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 170 / 181

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 171 / 181

Part VIII Appendix

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 172 / 181

Intraclass Correlation

The Intraclass Correlation comes from the Null Model – the model without and explanatory variables at either the individual or group levels. So the model is simply Yij = γ00 + U0j + ǫij from above. Assuming V (ǫij) = σ2, V (U0j) = τ 2

00, then

V (Yij) = V (U0j) + V (ǫij) = τ 2

00 + σ2

It is then easy to show that COV (Yij, Yi′j) = τ 2

• 00. The correlation is then

COV (Yij, Yi′j) = COV (Yij, Yi′j)

• V (Yij) ×
• V (Yi′j)

= τ 2

00

τ 2

00 + σ2

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 173 / 181

Inherently Linear Demand Model

We want constant elasticity: ηQ

P = d ln Q

d ln P = c. Or: d ln Q = c × d ln P. Integrating both sides and adding the integration constant, we get:

• d ln Q

= ln Q = c ×

• d ln P

So: ln Q = ln A + c × ln P = ln A + ln Pc = ln A × Pc Exponentiating both sides, the demand model is then: Q = A × Pc

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 174 / 181

Inherently Linear Demand Model: Elasticities

Model in log form: ln Q = ln A + c × ln P Take first derivative: 1 Q = c × 1 P × dQ dP Solve for c: c = P Q × dQ dP = ηQ

P

So the estimated coefficient is the elasticity.

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 175 / 181

Software

Free

R27 Python28 Gretl29

Commercial

OLS

Any stat/econometric package

Latent Class Modeling

JMP (not latent regression) Latent Class Gold Stata (Stata 15)

Multilevel Modeling

JMP SAS Stata

27See Finch et al. (2014) and Gelman & Hill (2007) for extensive discussion on using

R for multilevel modeling.

28With Pandas and Statsmodel packages for OLS. 29For OLS and econometric methods. Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 176 / 181

References

Collins, Linda M. and Stephanie T. Lanza Latent Class Analysis and Latent Transition Analysis: With Applications in the Social, Behavioral, and Health Sciences John Wiley & Sons (2010) Finch, W. Holmes; Jocelyn E. Bolin; and Ken Kelley Multilevel Modeling Using R CRC Press (2014) Gelman, Andrew and Jennifer Hill Data Analysis Using Regression and Multilevel/Hierarchical Models Cambridge University Press (2007) Gujarati, D.N. Basic Econometrics McGraw-Hill/Irwin (2003)

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 177 / 181

References

Luke, Douglas A. Multilevel Modeling Sage Publications (2004) Paczkowski, W.R. Pricing Analytics: Models and Advanced Quantitative Techniques for Product Pricing Routledge (2018) Snijders, Tom A.B. and Roel J. Bosker Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling 2nd Edition Sage Publications (2012) Wickham, Hadley and Garrett Grolemund R for Data Science: Import, Tidy, Transform, Visualize, and Model Data O’Reilly Media (2017)

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 178 / 181

References

Bell, John F Visualizing Multilevel Models: the Initial Analysis of Data http://www.cambridgeassessment.org.uk/Images/109679-visualising- multilevel-models-the-initial-analysis-of-data.pdf (2001) Diaz, A., Kayvaum Rowshankish, and Tamim Saleh. ”Data culture: Where organization meets analytics.” McKinsey Quarterly (Posted online on 9/6/18). Haans, Hans and Els Gijsbrechts “One-deal-fits-all?” On Category Sales Promotion Effectiveness in Smaller versus Larger Supermarkets Journal of Retailing. 87 (4, 2011) 427 – 443

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 179 / 181

References

Killip, Shersten; Ziyad Mahfoud; and Kevin Pearce ”What Is an Intracluster Correlation Coefficient? Crucial Concepts for Primary Care Researchers” Annals of Family Medicine. 2 (2004) 204 – 208 Oakley, James A.; Dawn Iacobucci; and Adam Duhachek Multilevel, Hierarchical Linear Models and Marketing: This is not your adviser’s OLS model in Review of Marketing Research, Vol. 2; N.K. Malhotra (2005) Ray, Jean-Claude and Daniel Ray ”Multilevel Modeling for Marketing: a Primer” Recherche et Applications en Marketing. 23 (2008) 55 – 77 Roux, A.V. Diez ”A Glossary for Multilevel Analysis” Journal of Epidemiology & Community Health. 56 (2002) 588 – 594

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 180 / 181

References

Zahay, D. A. Griffin and E. Fredicks ”Sources, uses, and forms of data in the new product development process” Industrial Marketing Management. 33 (2004) 657-666

Walter R. Paczkowski, Ph.D. Deep Data Analytics for Pricing October, 2018 181 / 181