Outline The Path of Inclusion Identity and Cognitive Diversity - - PowerPoint PPT Presentation

▶
outline
SMART_READER_LITE
LIVE PREVIEW

Outline The Path of Inclusion Identity and Cognitive Diversity - - PowerPoint PPT Presentation

Dec 03, 2023 251 likes •1.37k views

Scott E Page University of Michigan Santa Fe Institute Leveraging Diversity Outline The Path of Inclusion Identity and Cognitive Diversity Prediction Problems Solving Case: The Netflix Prize Takeaways Framework Tools: Diverse Perspectives,

slide-1
SLIDE 1

Leveraging Diversity

Scott E Page

University of Michigan Santa Fe Institute

slide-2
SLIDE 2

Outline

The Path of Inclusion Identity and Cognitive Diversity Prediction Problems Solving Case: The Netflix Prize Takeaways

slide-3
SLIDE 3

Framework

Tools: Diverse Perspectives, Heuristics, and Interpretations Tasks: Problem Solving and Prediction

slide-4
SLIDE 4
slide-5
SLIDE 5
slide-6
SLIDE 6

The Path of Inclusion

slide-7
SLIDE 7

Hiring diverse people is the right thing to do.

slide-8
SLIDE 8

Hiring diverse people is the required by law.

slide-9
SLIDE 9

Seeking diversity enlarges the pool and results in better employees.

slide-10
SLIDE 10

Diversity

Ability

slide-11
SLIDE 11

Diversity is a strategic advantage. It makes

  • rganizations more productive and more

innovative on cognitive tasks.

slide-12
SLIDE 12

Diversity Ability

slide-13
SLIDE 13

Identity and Cognitive Diversity

slide-14
SLIDE 14
slide-15
SLIDE 15

Gunter Blobel: The exception

slide-16
SLIDE 16

Gunter Blobel: The exception

slide-17
SLIDE 17
slide-18
SLIDE 18

Prediction

slide-19
SLIDE 19
slide-20
SLIDE 20
slide-21
SLIDE 21

Iowa Electronic Markets

IEM Prices Obama 0.535 McCain 0.464 Final Gallup Poll Obama 0.55 McCain 0.44 Actual Outcome Obama 0.531 McCain 0.469

slide-22
SLIDE 22
slide-23
SLIDE 23

Methods of Divination

Stars and Planets (astrology) Rolling Dice Tarot Cards Palm Reading Crystal Balls Head Shape (Phrenology) Atmospheric Conditions Dreams Animal Entrails Moles on the body

David Orrell “The Future of Everything.

Lightning Smoke and Fire Flight of Birds Neighing of Horses Tea Leaves and Coffee Grounds Passages of Sacred Texts Numbers I Ching Guessing MODELS

slide-24
SLIDE 24
slide-25
SLIDE 25
slide-26
SLIDE 26

West Virginia

Congressional District

District 1 District 2 District 3

slide-27
SLIDE 27

West Virginia

Slaw available on request Slaw standard on hot dogs Slaw not available No data available

slide-28
SLIDE 28

Interpretations: Pile Sort

Place the following food items in piles Broccoli Carrots Canned Beets Fresh Salmon Arugula Fennel Spam Ahi Tuna Canned Posole Niman Pork Sea Bass Canned Salmon

slide-29
SLIDE 29

BOBO Sort

Veggie Organic Canned Broccoli Fresh Salmon Canned Beets Arugula Sea Bass Spam Carrots Niman Pork Canned Salmon Fennel Ahi Tuna Canned Posole

slide-30
SLIDE 30

Airstream Sort

Veggie Meat/Fish Weird? Broccoli Fresh Salmon Canned Posole Fennel Spam Sea Bass Carrots Niman Pork Arugula Canned Beets Canned Salmon Ahi Tuna

slide-31
SLIDE 31

Crowd Error = Average Error - Diversity

Diversity Prediction Theorem

slide-32
SLIDE 32

Crowd Error = Average Error – Diversity

0.6 = 2,956.0 - 2955.4

Galton’s Steer

slide-33
SLIDE 33

2005 NFL Draft

Player A B C D E F G CROWD Alex Smith 1 1 1 1 1 1 1 1 Ronnie Brown 2 2 4 2 2 5 2 2.7 Braylon Edwards 3 3 2 7 3 2 3 3.3 Cedric Benson 4 4 13 4 8 4 8 5.9 Carnell Williams 8 5 5 5 4 13 4 6.4 Adam Jones 16 9 6 8 6 6 9 8.1

slide-34
SLIDE 34

2005 NFL Draft

Predictor A B C D E F G CROWD Squared Error 158 89 210 235 112 82 75 34.4

slide-35
SLIDE 35

NFL Experts

Average Error: 137.3 Diversity: 102.9 Crowd Error: 34.4

Predictor A B C D E F G CROWD Squared Error 158 89 210 235 112 82 75 34.4

slide-36
SLIDE 36

Problem Solving

slide-37
SLIDE 37

Gunter Blobel: The exception

slide-38
SLIDE 38

Perspectives

slide-39
SLIDE 39

The Technocratic Ideal

Frederick Winslow Taylor 1856-1915

http://www.resourcesystemsconsulting

slide-40
SLIDE 40

Simple: Shovel Landscape

Efficiency Size

slide-41
SLIDE 41
slide-42
SLIDE 42

Caloric Landscape

slide-43
SLIDE 43

Masticity Landscape

slide-44
SLIDE 44

Ben and Jerry’s Perspective

chunk size number of chunks

slide-45
SLIDE 45

Consultant’s Perspective

caloric rank

slide-46
SLIDE 46

Ben and Jerry’s Local Optima: Ave = 90

chunk size number of chunks

86 91 92 91

slide-47
SLIDE 47

Consultant’s Local Optima: Ave = 80

caloric rank

78 92 76 74

slide-48
SLIDE 48

Ben and Jerry’s Perspective

chunk size

Y

number of chunks

X Z

slide-49
SLIDE 49

Consultant’s Perspective

caloric rank

Z X Y

slide-50
SLIDE 50

Different Peaks

X Z

slide-51
SLIDE 51

Heuristics

slide-52
SLIDE 52

IQ Question: Fill in the Blank: 1 2 3 5 _ 13

slide-53
SLIDE 53

1 2 3 5 8 13

xi+2 - xi+1 = x

slide-54
SLIDE 54

IQ Question: 1 4 9 16 _ 36

slide-55
SLIDE 55

1

4 9 16 25 36

xi

2

slide-56
SLIDE 56

IQ Question: 1 2 6 _ 1806

slide-57
SLIDE 57

1

2 6 42 1806

xi+1 – xi = xi

2

2 - 1 = 12

6 – 2 = 22 42 – 6 = 62 1806 – 42 = 422

slide-58
SLIDE 58

xi+1 – xi = xi

2

A combination of the first two heuristics

slide-59
SLIDE 59

1 + 1 = 3

Superadditivity

slide-60
SLIDE 60
slide-61
SLIDE 61
slide-62
SLIDE 62

Network + Electrical Engineers

slide-63
SLIDE 63
slide-64
SLIDE 64
slide-65
SLIDE 65

A Test

  • Create a bunch of agents with diverse perspectives and

heuristics

  • Rank them by their

performance on a problem.

  • Note: all of the agents must be “smart”
slide-66
SLIDE 66

Experiment

Group 1: Best 20 agents Group 2: Random 20 agents Have each group work collectively - when one agent gets stuck at a point, another agent tries to find a further improvement. Group stops when no one can find a better solution.

slide-67
SLIDE 67

The IQ View

75 121 84 135 111 9

Alpha Group

138 137 139 140 136 132

Diverse Group

slide-68
SLIDE 68

The diverse group almost always outperforms the group of the best by a substantial margin.

See Lu Hong and Scott Page Proceedings of the National Academy of Sciences (2002)

slide-69
SLIDE 69

The Toolbox View

EZ AHK FD BCD AEG IL ADE BCD BCD ABC ACD BDE

Alpha Group Diverse Group

slide-70
SLIDE 70

Calculus Condition: Problem solvers must all be smart-

  • we must be able to list their local optima

Diversity Condition: Problem solvers must have diverse heuristics and perspectives Hard Problem Condition: Problem itself must be difficult

What Must be True?

slide-71
SLIDE 71

Case: Netflix Prize

slide-72
SLIDE 72

Outline

Netflix Prize: Background Predictive Models

Factor Models

Ensembles of Models Ensembles of Teams The Value of Diversity

slide-73
SLIDE 73

Netflix Prize

November 2006, Netflix offers a prize of $1 million to anyone who can defeat their Cinematch recommender system by 10% of more.

slide-74
SLIDE 74

Some Details

Netflix users rank movies from 1 to 5 Six years of data Half million users 17,700 movies Data divided into (training, testing) Testing Data dived into (probe, quiz, test)

slide-75
SLIDE 75

Interesting Asides

Lost in Translation and The Royal Tenenbaums had the highest variance Shawshank Redemption had the highest rating Miss Congeniality had the most ratings.

slide-76
SLIDE 76

Singular Value Decomposition

Each movie represented by a vector: (p1,p2,p3,p4…pn) Each person represented by a vector: (q1,q2,q3,q4…qn) Rating: rij = mi + aj + pq Training: choose p,q to minimiize (actualij –rij)2

+ c( ||p||2+ ||q||2)

slide-77
SLIDE 77

BellKor’s Initial Models

Approximately 50 dimensions Best Model: 6.8% improvement Combination of Models: 8.4% improvement

slide-78
SLIDE 78

Two Questions

Q1: Why more than one model? Q2: Why do more work better than one?

slide-79
SLIDE 79

Q1: Why More than one Model

This question has two answers. A1: they used different variables A2: their stochastic optimization technique got stuck in different places

slide-80
SLIDE 80

Different Tuning Parameters and Initial Points Lead to Different Peaks on a Rugged Landscape

slide-81
SLIDE 81

UCSC

A2: Diversity Prediction Theorem

SqE(c) = SqE(s) - PDiv(s)

(c −θ)

2 = 1

n (si −θ)

2 i=1 n

∑

− 1 n (si − c)

2 i=1 n

∑

slide-82
SLIDE 82

BellKor’s Pragmatic Chaos

More is Better: Seven person team created combining top two teams Now over 800 predictor sets (sets of variables). Difficult be build a “grand” model but possible to build lots of “huge” models

slide-83
SLIDE 83

Ensemble Effects

Best Model 8.4% Ensemble: 10.1% Rules: Once someone breaks 10%, then the contest ends in 30 days.

slide-84
SLIDE 84

Enter ``The Ensemble’’

23 teams from 30 countries who blended their predictive models who tried in the last moments to defeat BellKor’s Pragamatic Chaos

slide-85
SLIDE 85

The Ensemble

“The contest was almost a race to agglomerate as many teams as possible,” said David Weiss, a Ph.D. candidate in computer science at the University of Pennsylvania and a member of the Ensemble. “The surprise was that the collaborative approach works so well, that trying all the algorithms, coding them up and putting them together far exceeded our expectations.”

New York Times 6/27/09

slide-86
SLIDE 86

And The Winner is…

RMSE for The Ensemble: 0.856714 RMSE for Bellkor's Pragmatic Chaos: 0.856704 By the rules of the competition the scores are rounded to four decimal places so it was a tie. However, BellKor’s Pragmatic Chaos submitted 20 minutes earlier so they

  • won. (and they had the lower error)
slide-87
SLIDE 87

Oh, by the way..

BellKor’s Pragmatic Chaos 10.06% The Ensemble 10.06% 50/50 Blend 10.19%

slide-88
SLIDE 88

Takeaways

slide-89
SLIDE 89
  • 1. Value of Diversity Depends on Extent of

Collaboration.

slide-90
SLIDE 90

Holedigging

slide-91
SLIDE 91

Boosting

slide-92
SLIDE 92

Collective Problem Solving

slide-93
SLIDE 93
  • 2. Create Oracles
slide-94
SLIDE 94
slide-95
SLIDE 95
  • 3. Create Perspectives/Skills

Spreadsheets

name engineer sales physics statistics A x x B x x C x

slide-96
SLIDE 96
  • 4. Listen to Others But Avoid Group Think

Haacked.com

slide-97
SLIDE 97

Learning

Average individual squared error of seven experts who made forecasts about the NBA draft from May 23rd through June 25th.

May 23rd : 213.17 May 30th : 86.33 June 13th: 114.5 June 18th : 139.67 June 22nd : 109 June 25th: 69.67

slide-98
SLIDE 98

Avoiding Group Think

Date Individual Diversity Collective Error May 23rd : 213.17 168.03 45.14 May 30th : 86.33 81.41 28.57 June 13th: 114.5 70.31 44.19 June 18th : 139.67 113.3 26.34 June 22nd : 109.0 84.0 25.0 June 25th: 69.67 35.58 33.58

slide-99
SLIDE 99

Avoiding Group Think

Date Individual Diversity Collective Error May 23rd : 213.17 168.03 45.14 May 30th : 86.33 81.41 28.57 June 13th: 114.5 70.31 44.19 June 18th : 139.67 113.3 26.34 June 22nd : 109.0 84.0 25.0 June 25th: 69.67 35.58 33.58

slide-100
SLIDE 100

Encourage Dissent

If everyone agrees, then either the predictive task was easy and everyone has the correct forecast (in which case the meeting was a waste of time) or the the task was challenging and everyone has the same, wrong forecast.

slide-101
SLIDE 101
slide-102
SLIDE 102
  • 5. Technology Can Supplement Hierarchy

www.healys.eu

slide-103
SLIDE 103

www.encefalus.com

slide-104
SLIDE 104
slide-105
SLIDE 105

Goldcorp Challenge

March 6, 2000, Goldcorp offers $575k to participants who would help find gold at its Red Lake Mine in Ontario, Canada 110 targets identified, over 50% were new, over 80% were successful. Company value up from $100 Million to $9 Billion.

slide-106
SLIDE 106

Prediction Markets

slide-107
SLIDE 107

The Math Tells What’s Possible

slide-108
SLIDE 108

The Parable of the Bike

50m 50m

x

E E

x

Run Bike

slide-109
SLIDE 109

The Need for Leadership

50m 50m

x

E E

x

homogeneous Cognitively diverse

slide-110
SLIDE 110

?