A Wisdom of the Crowd Approach to Forecasting Funded by the - - PowerPoint PPT Presentation

▶

a wisdom of the crowd approach to forecasting

A Wisdom of the Crowd Approach to Forecasting Funded by the - - PowerPoint PPT Presentation

Oct 25, 2022 102 likes •395 views

A Wisdom of the Crowd Approach to Forecasting Funded by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior National Business Center contract number D11PC20059 Brandon Turner and Mark Steyvers UC, Irvine

slide-1

SLIDE 1

A Wisdom of the Crowd Approach to Forecasting

Funded by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior National Business Center contract number D11PC20059

Brandon Turner and Mark Steyvers

UC, Irvine

December 17th, 2011

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 1 / 18

slide-2

SLIDE 2

The Research

UCI is one member of Team ARA, along with six other universities:

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 2 / 18

slide-3

SLIDE 3

The Research

We work together to Investigate good elicitation methods Build models that use this information to predict the future

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 2 / 18

slide-4

SLIDE 4

The Research

Everyday people log on to a website They make predictions about items (IFPs) they are interested in We record lots of data and analyze it

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 2 / 18

slide-5

SLIDE 5

The Research

The goal is to beat MITRE, a data collection company, at making predictions MITRE uses the unweighted linear average on their own data Team ARA competes against four other teams to beat MITRE’s ULinOP

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 2 / 18

slide-6

SLIDE 6

The Research

The data comes in a variety of forms Binary IFPs Multi-Choice IFPs Continuous IFPs

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 2 / 18

slide-7

SLIDE 7

The Research

We currently have over 50 models To evaluate them, we compare them to our own ULinOp We are now past the burn-in period

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 2 / 18

slide-8

SLIDE 8

Outline

1

Wisdom of the Crowd

2

Data

3

Two Aggregation Models

4

Results

5

Conclusions/Future Directions

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 3 / 18

slide-9

SLIDE 9

Wisdom of the Crowd

Outline

1

Wisdom of the Crowd

2

Data

3

Two Aggregation Models

4

Results

5

Conclusions/Future Directions

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 4 / 18

slide-10

SLIDE 10

Wisdom of the Crowd

Motivation

The Wisdom of the Crowd Effect Groups of people make an estimate about a quantity The “correctness” of these participants will vary The mean of the estimates is better than the majority of the group

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 5 / 18

slide-11

SLIDE 11

Wisdom of the Crowd

Motivation

WoC effects have been found in a variety of interesting problems Static judgments Rank-ordering tasks Event recall Scene reconstruction Combinatorial problems

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 5 / 18

slide-12

SLIDE 12

Wisdom of the Crowd

Motivation

Can the WoC effect be harnessed to predict the future? Build on previous “shared truth” models Build on classic JDM confidence literature

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 5 / 18

slide-13

SLIDE 13

Data

Outline

1

Wisdom of the Crowd

2

Data

3

Two Aggregation Models

4

Results

5

Conclusions/Future Directions

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 6 / 18

slide-14

SLIDE 14

Data

Data

817 participants (general public) Provided estimates of the probability of the

ccurrence of future events

51 (binary) questions Judgments made over a one-month period

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 7 / 18

slide-15

SLIDE 15

Data

Complications

At first, there are no known answers Questions are designed to eventually resolve

“Who will win the January 2012 Taiwan Presidential election?” “By 1 January 2012 will the Iraqi government sign a security agreement that allows US troops to remain in Iraq?”

18 questions resolved during the one-month period Focused on binary items only

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 8 / 18

slide-16

SLIDE 16

Two Aggregation Models

Outline

1

Wisdom of the Crowd

2

Data

3

Two Aggregation Models

4

Results

5

Conclusions/Future Directions

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 9 / 18

slide-17

SLIDE 17

Two Aggregation Models

Modeling Approach

Assume some latent shared truth (CCT) Model the aggregate of the judgments (WoC) Assume the shared truth is systematically inaccurate Assume a distortion occurs, prohibiting accurate forecasting

By Question By Subject

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 10 / 18

slide-18

SLIDE 18

Two Aggregation Models

Modeling Approach

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 11 / 18

slide-19

SLIDE 19

Two Aggregation Models

Modeling Approach

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 11 / 18

slide-20

SLIDE 20

Two Aggregation Models

Modeling Approach

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 11 / 18

slide-21

SLIDE 21

Two Aggregation Models

New Modeling Attempts

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 12 / 18

slide-22

SLIDE 22

Results

Outline

1

Wisdom of the Crowd

2

Data

3

Two Aggregation Models

4

Results

5

Conclusions/Future Directions

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 13 / 18

slide-23

SLIDE 23

Results

Results

Distortion by Question

Performed 4.7% better than unweighted average Mean predictive error was 0.337

Distortion by Subject

Performed 9.6% better than unweighted average Mean predictive error was 0.320

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 14 / 18

slide-24

SLIDE 24

Results

Posterior Predictive Distributions

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 15 / 18

slide-25

SLIDE 25

Conclusions/Future Directions

Outline

1

Wisdom of the Crowd

2

Data

3

Two Aggregation Models

4

Results

5

Conclusions/Future Directions

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 16 / 18

slide-26

SLIDE 26

Conclusions/Future Directions

Conclusions

An accurate shared truth does not perform well A distorted version of the shared truth does well Distortion by subject is better than by question

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 17 / 18

slide-27

SLIDE 27

Conclusions/Future Directions

Future (Current) Directions

Exploit non-stationarity

Judgments might change over time and recent judgment might be more accurate; track opinions over time

Recalibrate judgments

Recalibrate individual judgments before aggregating Recalibrate the aggregate

Exploit individual differences

Estimate expertise from resolved IFPs and user profiles Match between user profile and IFP profile

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 18 / 18

slide-28

SLIDE 28

Conclusions/Future Directions

Future (Current) Directions

Model missingness

Incorporate information about the specific IFPs a user chooses to forecast, along with information about the number of IFPs that a user forecasts

Supervised learning algorithms

Enter a large number of features in various supervised learning algorithms, determine which are related to individuals Brier scores

Bayesian nonparametrics

Isolate subgroups of users with different forecasts/opinions, aggregate based on these subgroups

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 18 / 18