Constructing High Frequency Price Indexes Data Daniel Melser Using - - PowerPoint PPT Presentation

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Constructing High Frequency Price Indexes Using Scanner Data

Daniel Melser

School of Finance & Economics University of Technology, Sydney Haymarket, NSW 2007 Australia Email: daniel.melser@uts.edu.au

May 4, 2011

Prepared for the 12th Meeting of the Ottawa Group, Wellington, NZ

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Introduction

The use of scanner data has been disappointing. Why?

Various ‘reasonable’ approaches to index construction can lead to wildly different results. There is concern about bias but also the high variance of the resulting price indexes. Part of the problem stems from the large changes in prices and quantities caused by the sales cycle and stockpiling.

We think about some new ideas, within a hybrid economic-stochastic framework, that reflect the sales cycle and builds indexes which encompass the possibility that purchase and consumption may not take place simultaneously. We use a large publicly available US scanner data set from IRI to illustrate.

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

The Wacky World of Chained Indexes

Figure: Weekly Chained Indexes for Laundry Detergent from 2001–6

50 100 150 200 250 300 350 −20 −15 −10 −5 5 10 15 20 Week Log Index Chained Geometric Laspeyres Chained Geometric Paasche Chained Tornqvist

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Various Approaches to Constructing Price Indexes with Scanner Data

The rolling year GEKS approach of Diewert, Fox and Ivancic (2011) looks good.

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Various Approaches to Constructing Price Indexes with Scanner Data (cont.)

Average basket methods may also be worth some

• investigation. Why?

It is consistent with standard agency practice which compares prices to a reference period. It can be configured to respect the fact that purchase and consumption decisions are made over a time-span (‘budget horizon’) rather than contained within a single period. In some circumstances it may provide a sounder measure

• f price change.

Our approach to measuring prices in period c is to ask a question like... What would have been the cost of purchasing the reference bundle across a ‘budgeting horizon’ Ar facing period c’s price distribution?

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

An Example of an Average Basket-Type Approach

Let’s look at a Laspeyres/Lowe-type index. Notation: pist=price, qist=quantity, i=product, s=store, t=time (weeks). Take the prices and quantities from a reference period Ar as the base (Ar is a span of periods such as a year, e.g. Ar = {1, 2, . . . , 52}). We want to compare the prices in the span of periods Ar with the prices in an individual period c. How? Estimate the distribution of prices in period c and use this information in some sense to create a pseudo-sample spread over a span of dimension |Ar|. Call this ˜ pc.

Could simply insert the actual prices pisc for each of the time periods or sample from a distribution.

˜ PEL

r,c|Ar =

• t∈Ar,s∈St,i∈Ist ˜

pc

istqist

• t∈Ar,s∈St,i∈Ist pistqist

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

A Mixture Model for Prices

In order to model the distribution of prices in a given period we propose a mixture model for log-prices: log pist ∼ zistN

• αist − βi, σ2

i

• + (1 − zist)N
• αist, σ2

i

• , βi ≫ 0

zist ∼ B(ωit)

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Estimating the Parameters of the Mixture Model

To estimate the parameters of the model a likelihood-based approach, the EM (Expectation Maximization) algorithm, is used. This iterates between estimating the sales labels (the z’s) conditional on the parameters (the α’s, β’s, ω’s and σ’s), and estimating the parameters conditional on the sales labels. In theory the likelihood improves at every step. In practice convergence can be slow, though we did not have this problem.

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Some Results from Estimating the Model

Table: Summary Statistics for IRI Data — Boston

Product Number of: Expenditures, Probability

• Avg. Sales

Items Stores Obs. $ (% on Sale)

• f Sale, %

Discount, % Beer 558 33 175,658 10,773,410 (26.29) 8.46 12.87 Soft Drinks 863 9 355,071 14,401,327 (38.93) 21.79 25.51 Coffee 460 14 304,416 9,782,830 (34.59) 21.99 27.65 Deodorant 551 14 313,495 1,873,691 (17.61) 15.70 36.55 Diapers 260 37 248,830 8,624,339 (24.70) 20.04 24.32 Laundry Detergent 325 16 210,504 8,748,413 (36.82) 17.24 31.80 Milk 282 17 321,594 48,326,059 (17.16) 19.52 19.56 Mustard and Ketch. 225 23 245,953 6,051,455 (21.29) 17.71 25.97 Paper Towels 219 38 234,434 21,202,529 (28.65) 12.25 28.98 Peanut Butter 116 35 270,964 9,242,605 (26.52) 17.45 25.23 Salty Snacks 963 9 307,746 8,788,534 (27.01) 18.35 27.55 Shampoo 551 14 299,967 2,326,703 (19.74) 15.70 30.27 Soup 552 9 264,729 4,612,889 (36.60) 22.41 34.43 Spaghetti Sauce 422 12 288,320 5,465,383 (36.57) 23.67 28.22 Sugar Substitute 52 51 191,022 4,435,557 (14.81) 13.31 22.85 Toilet Tissue 163 32 241,574 22,487,510 (28.11) 13.42 27.97 Toothpaste 403 14 246,966 2,952,563 (19.95) 15.70 31.27 Yoghurt 550 6 272,707 10,179,578 (22.92) 17.28 28.59 NOTE: Results are for IRI data discussed in detail in Bronnenberg, Kruger and Mela (2008). The data covers 313 weeks (6 years) and records weekly average prices at the store-level by product barcode.

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Some Examples for Deodorants in Chicago...

20 40 60 80 100 120 140 2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4 Weeks Price 20 40 60 80 100 120 140 2 2.5 3 3.5 4 Weeks Price 20 40 60 80 100 120 140 1.5 2 2.5 3 3.5 4 4.5 5 Weeks Price 120 130 140 150 160 170 180 190 200 210 1.5 2 2.5 3 3.5 4 Weeks Price

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Checking for Stockpiling

If stockpiling does not occur then a classical model of demand, such as the CES cost function, should explain consumers’ expenditures. We look at expenditure shares (vist) in the period immediately before and immediately after a sale. Our expectation is that, controlling for price, expenditure shares will be lower after the sale than before as consumers will have built up inventories during the sale. log visr/λr visu/λu

• = γ0 + (1 − σ) log

pisr pisu /Pur

• + γ1displayisru + γ2featureisru + eisru

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

The Results of the Stockpiling Regression

Table: Results of Stockpiling Regression

Product Obs. R2 Coefficients: Intercept Elasticity (σ) Display Feature Beer 250 0.0167 –0.1362∗∗ 1.2652∗∗∗ 0.4737 –0.8335 Soft Drinks 10,019 0.0067 –0.0732∗∗∗ 1.4242∗∗∗ –0.0398 –0.0044 Coffee 7,587 0.0056 –0.0502∗∗∗ 0.5494∗∗∗ 0.0458 –0.0201 Deodorant 8,718 0.0339 –0.0350∗∗∗ 0.4911∗∗∗ –0.0055 0.0541 Diapers 9,471 0.0475 –0.0196∗∗ 0.6477∗∗∗ 0.3432∗∗ 0.0039 Laundry Detergent 4,770 0.0005 –0.1061∗∗∗ 0.9916∗∗∗ 0.1792 –0.0534 Milk 8,005 0.0016 0.0016 1.4885∗∗∗ 0.1837 0.0015 Mustard and Ketch. 2,257 0.0116 0.0086 0.1245 0.2216 0.0469 Paper Towels 3,789 0.0617 –0.0356∗∗∗ 1.4993∗∗∗ –0.2069 0.2553∗∗ Peanut Butter 5,644 0.0155 –0.0910∗∗∗ 0.3122∗∗∗ –0.0528 0.0708 Salty Snacks 5,765 0.0128 –0.0485∗∗∗ 1.7570∗∗∗ –0.0508 –0.0014 Shampoo 9,377 0.0261 –0.0536∗∗∗ 0.6271∗∗∗ –0.0744 0.0126 Soup 6,416 0.0087 –0.0837∗∗∗ 0.5901∗∗∗ –0.0942 0.0061 Spaghetti Sauce 11,493 0.0019 –0.0599∗∗∗ 1.1128∗∗∗ 0.1101 0.1117 Sugar Substitute 935 0.0793 –0.0532 1.5789∗∗∗ 0.9133 0.23466 Toilet Tissue 5,559 0.0625 –0.0943∗∗∗ 1.4801∗∗∗ 0.0423 0.0655 Toothpaste 9,864 0.0072 –0.0400∗∗∗ 0.5713∗∗∗ 0.2121 0.0163 Yoghurt 14,983 0.0092 –0.0653∗∗∗ 1.6283∗∗∗ 0.0994 –0.0289 Note: ∗ denotes significance at the 10% confidence level, ∗∗=5%, and ∗∗∗=1%.

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Sales, Stockpiling and Index Bias

Using our model of prices we can decompose a link in the chained Laspeyres index as, log PGL

b,c =

• s∈S,i∈Is

visb log pisc pisb

• =
• s∈S,i∈Is

visb(αisc − αisb) −

• s∈S,i∈Is

βivisb(zisc − zisb) +

• s∈S,i∈Is

visb(eisc − eisb) Where, visb = pisbqisb

• s∈S,i∈Is pisbqisb

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Sales, Stockpiling and Index Bias (cont.)

Using our model of prices we can decompose a link in the chained T¨

• rnqvist index as,

log PT

b,c =

• s∈S,i∈Is

(visc + visb) 2 log pisc pisb

• =
• s∈S,i∈Is

(visc + visb) 2 (αisc − αisb) −

• s∈S,i∈Is

βi (visc + visb) 2 (zisc − zisb) +

• s∈S,i∈Is

(visc + visb) 2 (eisc − eisb)

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Sales, Stockpiling and Index Bias (cont.)

Using our model of prices we can decompose the GEKS T¨

• rnqvist index as,

log PMT

b,c =

• s∈S,i∈Is

(visc + ¯ vis) 2 (αisc − ¯ αis) − (visb + ¯ vis) 2 (αisb − ¯ αis)

• −
• s∈S,i∈Is

βi (visc + ¯ vis) 2 (zisc − ¯ zis) − (visb + ¯ vis) 2 (zisb − ¯ zis)

• +
• s∈S,i∈Is

(visc + ¯ vis) 2 (eisc − ¯ eis) − (visb + ¯ vis) 2 (eisb − ¯ eis)

• ,

¯ xis = 1 |AT|

• a∈AT

xisa

A feature of this is that price change can potentially be recorded even when no prices change. Is the multiperiod identity test strong enough: P(p1, p2, q1, q2)P(p2, p3, q2, q3)P(p3, p1, q3, q1) = 1

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

A Suggestion on the Calculation of Rolling Year Indexes

One approach to constructing the rolling year GEKS index for month t, with a 13-month window, is:

(a) Estimate ‘new’ multilateral indexes φt, φt−1, φt−2, . . . , φt−12. (b) The existing ‘published’ indexes are fixed at some previously calculated values δt−1, δt−2 . . . , δt−12. (c) Calculate δt (all in logs) as, ˆ δt = δt−1 + (φt − φt−1).

What about choosing δt such that it minimises the squared distance between φ and δ but fixing all δ other than δt (with φ and δ normalized on the same period).

50 100 150 200 250 300 350 −0.08 −0.06 −0.04 −0.02 0.02 0.04 0.06 0.08 0.1 0.12 Week Log Index GEKS Tornqvist GEKS Tornqvist (proposed)

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Summary and Conclusions

It is quite feasible to calculate fixed based rolling year price indexes using scanner data. These indexes are conceptually quite compelling when thinking about the way that households purchase for inventory and later consumption. Understanding the sales cycle is important for thinking about the bias from chaining and for constructing high frequency index numbers with low variance. A mixture model is a very natural way to incorporate sales into a model of prices.

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Comparing the Various Indexes — Laundry Detergent

50 100 150 200 250 300 350 −1.6 −1.4 −1.2 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 Week Log Index Chained Tornqvist GEKS Tornqvist Expected Stochastic Expected Laspeyres 50 100 150 200 250 300 350 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 Week Probability of Sale 50 100 150 200 250 300 350 −0.08 −0.06 −0.04 −0.02 0.02 0.04 0.06 0.08 0.1 Week Log Index GEKS Tornqvist Expected Geometric Expected Laspeyres 50 100 150 200 250 300 350 −0.08 −0.06 −0.04 −0.02 0.02 0.04 0.06 0.08 0.1 0.12 Week Log Index GEKS Tornqvist Expected Geometric Expected Laspeyres

Constructing High Frequency Price Indexes Using Scanner Data Daniel Melser

Comparing the Various Indexes

Table: Variance of Weekly Changes Relative to GEKS T¨

• rnqvist (%)

Product Expected Laspeyres Expected Geometric Coffee 22.12 8.60 Carbonated Beverages 27.29 10.38 Laundry Detergent 37.62 7.28 Milk 39.68 20.74 Peanut Butter 14.08 7.24 Salty Snacks 31.08 11.93 Shampoo 15.33 6.19 Toothpaste 20.06 8.81