Official Statistics in the New Data Ecosystem David J. Hand Imperial - - PowerPoint PPT Presentation

1

Official Statistics in the New Data Ecosystem

David J. Hand Imperial College, London

March 2015

2

Pascal’s wager on the existence of God: “You must wager. It is not optional. You are embarked.”

3

Pascal’s wager on the existence of God: “You must wager. It is not optional. You are embarked.” Same is true for official statistics and the “new data ecosystem”

4

Pascal’s wager on the existence of God: “You must wager. It is not optional. You are embarked.” Same is true for official statistics and the “new data ecosystem”. It’s out there and

• ne either ignores it, believing it’s inconsequential,
• r one engages with it.

Official statisticians have to bet one way or the other

5

 Ignore it and you risk becoming irrelevant  Engage and you are leaping aboard a treadmill which is getting faster and faster

‐ tomorrow’s IT will be different from today’s (think Twitter, YouTube, etc) ‐ “The future you have tomorrow won't be the same future you had yesterday.” Chuck Palahniuk ‐ “In times of change, learners inherit the Earth, while the learned find themselves beautifully equipped to deal with a world that no longer exists.” Eric Hoffer

6

What’s new in the world of data?

‐ source of data: automatic acquisition of data ‐ speed of acquisition of data: “streaming data” ‐ size of data sets: “big data” ‐ diversity of data ‐ complexity of data

7

Source: Modern data capture technologies

Automatic data collection:

‐ electronic measurements: point of sale credit card terminals, petrol pumps, contactless travel cards, phone records, emails, GPS, CCTV cameras, ...

Internet of things Administrative data Social media data – data directly from the web “Properties” of automatic data collection:

‐ immediate ‐ complete ??? ‐ untouched by human hands ???

8

Speed: Realtime data collection – and analysis

This has several major implications, threatening the position of NSIs ?

1: timeliness

The balance of timeliness against accuracy: Example: UK GDP

 1st estimate: 44% of the data available by 25 days,  2nd estimate: 88% by 55 days,  3rd estimate: 85 days

9

What about inflation rate?: Elaborate procedure to collect sample data Contrast with direct recording from transactions:

“Google has created a new inflation measure – the Google Price Index – based on the cost of goods sold online which could prove more accurate and up‐to‐date than official statistics. Google's mountain of web shopping data could also be used by the

• nline group for economic forecasting ahead of the publication of
• fficial statistics, the Financial Times reported.

Google's chief economist Hal Varian said that he is working on ‘predicting the present’ by using real‐time search data to forecast

• fficial figures which often are published at least a month after the

period they cover.” http://www.theguardian.com/business/2010/oct/12/google-create-new-inflation-measure

10

2: new kind of analytic tools needed

“Streaming data”: the data keep on coming, like water from a hose Permanently executing analytic tools, processing the data as it arrives

‐ anomalies ‐ changes ‐ summaries (trends, averages, variability, maxima, ...)

Realtime → automatic analysis Contrast the more familiar: a fixed database

11

Note: perhaps we cannot store the data once it has been processed In that case we need to know what questions we will ask We cannot later ask arbitrary questions, but only those that can be answered from our summary statistics

Summarising a stream Subsetting a stream: sampling, but requires different approaches from classical survey sampling Filtering a stream: accept only those cases which meet some criterion

12

Size: Large data sets

Census: the largest set of data collected by NSIs until relatively recently ? But now administrative data, register‐based data

‐ some countries (e.g. Scandinavian) ahead of the field

transaction data

even administrative data sets can be tiny compared with those arising from automatic data collection (e.g. social media, Google searches, twitter messages,email transaction logs, phone logs, transport logs, ...)

13

Diversity of data

Survey, census, administrative, transaction, experimental, ... Numerical tables, image, text, signal, networks, ... Different kinds of data have different properties

e.g. survey data: answers to the questions you choose but slow and expensive to collect, response bias? e.g. transaction data: fine granularity, both spatial and temporal, immediate, but may not address the question you want, complete population coverage – in principle, but rarely in practice → an opportunity: Perhaps data of different kinds can be combined synergistically, to

• vercome the problems of each individual kind

14

Stitching different kinds of data together Linking Matching Merging Technical challenges have begun to be addressed in different fields e.g. medical combining information from scans with traditional numeric and text files Great potential benefit from cross‐disciplinary collaboration and awareness

15

“survey and census data is what they say: administrative and transaction data is what they do”

16

“survey and census data is what they say: administrative and transaction data is what they do” New forms of data are closer to social reality

17

Complexity of data

Especially, networks and linked data ‐ Social networks ‐ Cybersecurity ‐ Fraud detection

18

Ping‐Pong Fraud Ring Model (HBOS Data Mining Team, 2007)

Within the Mortgage Industry, in addition to individual fraudulent applications, fraud can also occur in groups often referred to as Fraud

• Rings. These groups can consist of applicants, brokers and/or other

professionals that get together to cheat the system. The purpose of the Ping‐Pong Fraud Ring Model is to assist the Mortgage Fraud team in identifying these Fraud Rings. This is a 3 step approach:

• 1. Ping‐Pong clustering: aggregating applications that have

common names, addresses, telephone numbers, employers, etc

• 2. Ranking: inconsistency rules are used to rank the clusters in order
• f severity.
• 3. Linkage analysis: identifying links between customers, brokers,

employers, and/or solicitors.

19

OTHER CRITICAL ISSUES Data quality

“most big data that have received popular attention are not the output of instruments designed to produce valid and reliable data amenable for scientific analysis”

Lazer et al 2014

Caution: No data set is without potential quality issues Role and reputation of NSIs, as standard bearers of quality

20

Example: Mistaken idea that admin data/transaction data has no quality issues

Quality Assurance of Administrative Data Administrative Data Quality Assurance Toolkit UK Statistics Authority, January 2015

21

Top Tips UKSA Administrative Data Quality Assurance Toolkit, 2015 These five tips summarise the main pointers for statistical producers to develop a good understanding of the quality issues of administrative data:

Don’t trust the safeguards. Check if safeguards are functioning

effectively

Get involved: work and share with suppliers, such as through

secondments and webinars, to develop a common understanding

Raise a red flag: identify potential data quality concerns using input and

• utput quality indicators and investigate anomalies

See the big picture: identify what investigations and audits have been

conducted and what they found

Corroborate the evidence: confirm the levels and the trends shown by

the stats derived from the admin data

22

Example: Even transaction data implies a complex sociological selection process

e.g. reject inference in credit scoring

23

Example: Instrumental failure

Wind speed in metres per second, against time

24

Example: Proportion of homeowner missing values vs time

25

The rate of change of technology

Implications: ‐ for series based on a particular technology, what happens when it changes, or disappears altogether (Windows XP) ‐ surveys have been around forever, but Google Trends?

26

Statistical subtleties

The danger of “the data speak for themselves” ‐ selection bias ‐ regression to the mean ‐ individual behaviour vs population behaviour

Are wages improving or remaining constant? A1: calculate the median wage at time 1 and time 2 A2: calculate the change of individuals in the population

27

Google flu trends “In February 2013, ... Nature reported that [Google Flu Trends] was predicting more than double the proportion of doctor visits for influenza‐like illness than the Centers for Disease Control and Prevention, which bases its estimates on surveillance reports from laboratories across the United States ... despite the fact that GFT was built to predict CDC reports.”

Lazer et al 2014

28

Initial version: find best matches from 50 million search terms for 1152 data points Overfitting! → part flu detector, part winter detector Updated, 2009: still consistently overestimated flu prevalence (100 out of 108 consecutive weeks); autocorrelated errors, etc Some understanding of classical time series modelling could help

29 Correlation = 0.992

30

Predicting the financial markets Preis T, Moat H.S., and Stanley H.E. (2013) ‘Quantifying trading behavior in financial markets using Google trends.’ Scientific Reports, 3, 1684. ‘By analyzing changes in Google query volumes for search terms related to finance, we find patterns that may be interpreted as ‘‘early warning signs’’ of stock market moves.’

31

Chose 98 search terms related to stock markets Implemented a strategy which ‐ sold DJIA following an increase in searches ‐ and bought following a decrease in searches Concluded: "We find that returns from the Google Trends strategies are significantly higher overall than returns from the random strategies (<R>=0.60; t=8.65, df=97, p<0.001, one sample t‐ test)." But this is an inappropriate test Telling us nothing about the effectiveness of the strategy

32

1) the 98 terms were purposively sampled, not a random sample from a population of possible terms: the authors confused fixed and random effects; 2) the standard deviation used in the denominator of their t‐ test is irrelevant to the variability of the mean return: it’s the wrong statistic; 3) they ignored the correlation between the random aspect

• f the different words’ returns: anyone working in the

financial sector will be aware of the critical importance of correlation!

33

Google Price Index My earlier quote about the Google Price index was from 2010 Where is the GPI now? “[Hal Varian, Google’s Chief Economist] responded that the GPI was never intended to be a public project, data source, etc. It was simply a project internal to Google that got hyped up by the press”

http://money.stackexchange.com/questions/21629/what‐happened‐to‐the‐google‐price‐index

34

Data not designed to answer the research or policy question, may not be well‐suited to answer that question ‐ administrative data to replace the census ?

35

Crime maps “For many years, all forces have mapped crimes and incidents to help them focus investigations, analyse hot spots and tackle crime vigorously. The information now on the forces’ websites has a different more community‐ focused perspective and means the public can now look at crime levels in their community simply by putting their postcode into their local police force’s website.”

www.crimemaps.org.uk

36

“More than 5.2 million people have not reported crimes for fear of deterring home buyers or renters since the online crime map was launched in February 2011” “A quarter (24 per cent) of people would not report a crime for fear it would harm their chances of selling or renting their property”

http://www.directline.com/media/archive‐2011/news‐11072011

37

Trust, privacy, ...

‐ people readily give data to supermarkets, travel companies, phone companies, credit card companies,.. ‐ government misuse, targeting subgroups. Formal separation of statistical offices from government? Anonymisation De‐identification Trusted Third Party linkage methods → need for stascians to learn new skills → broader data scientists

38

Interaction with the commercial world

The credit bureau model: everyone benefits NSI takes fine granularity commercial (e.g. transaction) data, combines it with public data, analyses it, and feeds back the results to the commercial body – for a price NSI retains control, protects public

39

The skills gap

What skills are needed? More than the current skills How many NSI statisticians are at ease with realtime transaction data?

40

Big data and open data are only useful if you have the skills to extract information

When the judge told barrister Frederick Edwin Smith: ‘I’ve listened to you for an hour and I’m none the wiser’

41

Smith replied: ‘None the wiser, perhaps, my lord, but certainly better informed.’

42

But being better informed is not much use if you cannot make good use of that better information

43

But being better informed is not much use if you cannot make good use of that better information

Wisdom is also necessary

44

But being better informed is not much use if you cannot make good use of that better information

Wisdom is also necessary We need the skills to extract wisdom from

information

45

Conclusions

‐ huge opportunities ‐ technical and social challenges ‐ which must be grasped: Pascal’s wager: we are embarked; we cannot bury our heads in the sand ‐ but we need the capability to grasp the opportunities ‐ we need to train people with the skills

46

thank you