DECISION MAKING WITH INCOMPLETE INFORMATION MARC C. CANELLAS PH.D. - - PowerPoint PPT Presentation

DECISION MAKING WITH INCOMPLETE INFORMATION

MARC C. CANELLAS

PH.D. DISSERTATION DEFENSE SCHOOL OF AEROSPACE ENGINEERING

ADVISOR: DR. KAREN FEIGH, AEROSPACE ENGINEERING, GEORGIA TECH COMMITTEE: DR. BRIAN GERMAN, AEROSPACE ENGINEERING, GEORGIA TECH

• DR. AMY PRITCHETT, AEROSPACE ENGINEERING, GEORGIA TECH
• DR. STEVE CROSS, INDUSTRIAL & SYSTEMS ENGINEERING, GEORGIA TECH
• DR. JUAN ROGERS, PUBLIC POLICY, GEORGIA TECH

PEOPLE ARE CONSISTENTLY REQUIRED TO MAKE DECISIONS TO ACHIEVE MISSION SUCCESS REGARDLESS OF THE TIME PRESSURE AND QUANTITY AND QUALITY OF INFORMATION

1Orasanu and Connolly, 1993; 2Green and Mehr, 1997;3Tversky and Kahneman, 1974; 4Katsikopoulos and Fasolo, 2006; 5Elwyn et al., 2001; 6General result of naturalistic decision making and fast-and-

frugal research programs

• Necessary information often not available,2 or too costly (in time, money) to gather
• r process1
• Probabilities, relative importance of cues, and cue values are often unreliable3
• People tend to use simple rules like heuristics without gathering and processing all

information (due to time, cost, experience, etc.) 1,4,5,6

2

COMMAND AND CONTROL IN THESE DEGRADED AND DENIED INFORMATION ENVIRONMENTS IS A FUNDAMENTAL ISSUE IN MODERN MILITARY OPERATIONS

3

Modern warfighters are becoming technology and information dependent. Need to prioritize information for defense and offense (e.g., radar jamming, cybersecurity attacks)2

• 1Lt. Kessler, US Navy, 2010 2 Conversation with Lt. Cmdr. Beris, MORS METSM 2016

Need for new tactical training and decision support tools.1

TRANSLATING C2D2E INTO DECISION THEORY FOR MODELING AND SIMULATION

4

Command and Control within Degraded and Denied Information Environments (C2D2E) Decision tasks with incomplete information Range of decision making strategies

Representing fundamental task attributes Representing decision makers

• Represents unavailable information and

time pressure

• Well-studied as a driver of decision making

strategy selection and performance

• Has clear mapping to interface design: can

alter saliency and accessibility of information1 Experts tend to use naturalistic decision making or heuristics (fewer, more important cues)2 Novices tend to use normative and analytic strategies (more cues but less important ones) but sometimes also resort to heuristics when stressed2

1Feigh et al., 2012; 2Garcia-Retamero and Dhami, 2009; Gigerenzer and Gaissmaier, 2011

DECISION TASKS FOR THIS THESIS

5

Judgment (Classifying single object) Decision Making (Choice among options) Dynamic Static Sequential Single Analytics Heuristics Do the cue values change during the decision making process?

Decision making for this thesis will examine how analytic and heuristic strategies perform in static, single decision tasks with incomplete information.

Yes No Are options presented one at a time? Yes No Is all information processed by the strategy? Yes No

EXEMPLAR DECISION TASK: SINGLE, STATIC, DECISION MAKING WITH INCOMPLETE INFORMATION

6

Naval Defense Task – Operator of Phalanx CIWS

• Radar guided 20mm M61 Vulcan cannon gatling gun capable of

firing 4,500 rounds per minute

• Last line of defense for every surface combat ship in the US Navy
• Can be controlled by a human-operator

Altitude Speed Distance Size Target A 5,000 ft 1,200 kts 5 nmi 0.2 m2 Target B 1,500 ft 1,500 kts 7 nmi 0.2 m2 Operator’s Decision Attributes1 Decision Task Engage the most dangerous of the known simultaneous targets Single, static decision Attributes of targets are altitude, speed, distance, and size of object 3-5 cues, max for 15 sec Sensors degraded by environment density, countermeasures, etc. Incomplete information 15 seconds from effective horizon (seeing the target) to ship impact Heuristic strategies

1Prengaman et al., 2001

? ? ? ?

CURRENT FOUNDATIONS FOR DECISION SUPPORT FOR INCOMPLETE INFORMATION ENVIRONMENTS

1Gigerenzer et al., 1999; Todd, et al., 2011; 2Martignon and Hoffrage, 2002; Garcia-Retamero and Reiskamp, 2008; 3Slovic and MacPhillamy, 1974; Kivetz and Simonson, 2000; 4Nelson, 2005; 5Katsikopoulos and Fasolo, 2006

Behavioral Decision Making Consumer and Marketing Research Probabilistic Evidence Acquisition

+ Mathematical, computational, and human-subjects studies

• f both analytic and heuristic

decision making1

• Only measure of incomplete

information is total information2 + How the information is distributed and presented to a decision maker does matter3

• Only small scale human-

subjects studies and not focused on time-stressed decision making + Probabilistic frameworks for how decision makers or decision support systems should acquire information4

• Require assessments of

probabilities that are untenable in real-world environments5

+ + +

7

There is a need for a comprehensive investigation of incomplete information (mathematical, computational, and human-subjects studies) with a focus on providing decision support without requiring often-unreliable information about probabilities, etc.

MOTIVATION, RESEARCH GAPS, AND RESEARCH QUESTIONS

8

• 1. Does the distribution of incomplete information

affect decision making performance? If so, when and how?

• 2. Can information acquisition and restriction

methods be defined which do not rely on probabilities, etc.? When are they effective? Incomplete understanding of the effect of distributions of incomplete information on decision making performance. Analytic techniques rely on information that is

• ften not available or reliable.

Need to prioritize information for defensive and

• ffensive capabilities

Need for new tactical training and decision support tools for environments with degraded and denied environments

Define new measures of incomplete information

• 1. Does the distribution of incomplete information

affect decision making performance? If so, when and how?

ORGANIZATION OF THE REST OF THIS DEFENSE PRESENTATION

9

Define new measures of incomplete information Characterize the impact

• f distribution of

information on decision making strategies

• 2. Can information acquisition and restriction

methods be defined which do not rely on probabilities, etc.? When are they effective?

Define heuristic information acquisition and restriction rules Characterize the effectiveness of the rules for decision making strategies Define new measures of incomplete information

DECISION MAKING ACCURACY IS DETERMINED BY THE ENVIRONMENT, STRATEGIES, & INCOMPLETE INFORMATION

10

Environmental Parameters

(types of, and relationships between, cues and

• ptions)

Strategies

(information processing and integration)

Incomplete Information

(distribution of missing information)

Accuracy

(Percent of correct decisions made) Characteristics of the Environment Characteristics of Strategies Measures of Accuracy Can be affected by training and decision support tools.

1Payne et al., 1990; 2Gigerenzer and Gaissmaier, 2011; 3Karelaia, 2006; 4Dieckmann and Rieskamp, 2007; 5Gigerenzer and Goldstein, 1996; 6Park, 2004; 7Garcia-Retamero and Dhami, 2009; 8Dawes, 1979; 9von Helversen and Rieskamp, 2009; 10Kattah et al., 2009; 11Kohli and Jeddi

RANGE OF DECISION MAKING STRATEGIES

11

Weighted Additive (WADD)1 Equal Weighting (EW)1 Tallying2 Take-Two3,4 Take-the-Best (TTB)5 Linear model with weights Linear model without weights Linear model counting “good” cues Choose the first

• ption to discriminate

with two “good” cues Choose the first

• ption to discriminate

with one “good” cue Multi-attribute decision making6; Novice decision makers7 When weights cannot be determined or agreed upon8 Criminal sentencing decisions9; Medical decision making10 People often search for a second, confirming attribute3,4 Consumer choice11; Expert decision makers7 Analytic: Gather as much information as possible to use in a linear model (generally uses continuous data) Heuristic: Uses only a subset of “necessary” information (generally uses discretized data into “good/bad” categories)

Measure Definition Task A Task B Total information1 Amount of cue values known Total information is 4 or 50% Total information is 4 or 50% Option Imbalance (Canellas, Feigh, and Chua, 2014, IEEE CogSIMA) Difference in amount of cue values known between options Option Imbalance is 4 or 100%, i.e., an operator is familiar with Target 2 and unfamiliar with Target 1. Option Imbalance is 0 or 0%, i.e., an operator has equal amounts of information for both options. Cue balance (Canellas, Feigh, and Chua, 2014, IEEE SMC) Number of cues which have cue values known for all options Cue Balance is 0 or 0%: For no cue does the operator have comparable information. Cue Balance is 2 or 50%: An operator has equal information about both options for 2 out of 4 cues.

DEFINING NEW MEASURES OF DISTRIBUTIONS OF INCOMPLETE INFORMATION

12

Altitude Speed Distance Size Target 1 ? ? ? ? Target 2 K K K K Altitude Speed Distance Size Target 1 K K ? ? Target 2 K K ? ?

Task A Task B

1Martignon and Hoffrage, 2002; Garcia-Retamero and Rieskamp, 2008)

Measure Definition Task C Task D Total information1 Amount of cue values known 6 (75%) 4 (50%) Option Imbalance (Canellas, Feigh, and Chua, 2014, IEEE CogSIMA) Difference in amount of cue values known between options 2 (50%) 0 (0%) Cue balance (Canellas, Feigh, and Chua, 2014, IEEE SMC) Number of cues which have cue values known for all options 2 (50%) 0 (0%)

DEFINING NEW MEASURES OF DISTRIBUTIONS OF INCOMPLETE INFORMATION

13

Altitude Speed Distance Size Target 1 K K ? ? Target 2 K K K K Altitude Speed Distance Size Target 1 ? K ? K Target 2 K ? K ?

Task C Task D

1Martignon and Hoffrage, 2002; Garcia-Retamero and Rieskamp, 2008)

Characterize the impact

• f distribution of

information on decision making strategies Characterize the impact

• f distribution of

information on decision making strategies Defined new measures

• f incomplete

information

• 1. Does the distribution of incomplete information

affect decision making performance? If so, when and how?

COMPUTER STUDY 1: CHARACTERIZING THE IMPACT OF DISTRIBUTION OF INFORMATION ON DECISION STRATEGIES

14

Defined new measures

• f incomplete

information

• 2. Can information acquisition and restriction

methods be defined which do not rely on probabilities, etc.? When are they effective?

Define heuristic information acquisition and restriction rules Characterize the effectiveness of the rules for decision making strategies

K K ? ? K K ? ? ? ? ? ? K K K K

Option imbalance: difference in information between options Cue balance: cues with information for all options

METHOD

15

Environment Artificial environment in which accuracy was determined by a linear model Strategies (type)

• Weighted-additive (analytic)
• Equal-weighting (analytic)
• Tallying (heuristic)
• Take-the-best (heuristic)

Incomplete Information Total information, option imbalance, cue balance

Cue 1 Cue 2 Cue 3 Cue 4 Option 1 40 20 80 40 Option 2 40 60 40 60 Cue 1 Cue 2 Cue 3 Cue 4 Option 1 ? ? K K Option 2 K K K K

28 combinations of incomplete information 48 combinations of cue values

Cue 1 Cue 2 Cue 3 Cue 4 Option 1 ? ? 80 40 Option 2 40 60 40 60

68 (16M) combinations of cue values XX

SINGLE-VARIABLE RESULTS (EXCLUDING RANDOM DECISIONS)

16

50% 60% 70% 80% 90% 100% 0% 25% 50% 75% 100% Option Imbalance 50% 60% 70% 80% 90% 100% 0% 25% 50% 75% 100% Accuracy Total Information 50% 60% 70% 80% 90% 100% 0% 25% 50% 75% 100% Cue Balance

Decision Support: Provide more information. Decision Support: Provide decision makers with cue values for both options. Decision Support: For heuristics, provide equal information about both options. For analytic strategies, there is no effect. Analytic: gather as much information as possible Heuristic: Use subset of information Weighted-Additive Equal-Weighting Tallying Take-the-best

OPTION IMBALANCE – TOTAL INFORMATION TRADEOFF

17

Total Information Option Imbalance Option Imbalance Take-the-Best (TTB) Weighted-Additive Total Information

Cells are empty when that combination cannot occur in a 2-option, 4-attribute task.

• 1. Total information is increased.

100% 77 75% 73 81 50% 69 77 85 25% 64 73 81 91 0% 69 77 86 100 13% 25% 38% 50% 63% 75% 88% 100% 100% 55 75% 56 64 50% 58 66 73 25% 58 67 74 80 0% 65 72 78 86 13% 25% 38% 50% 63% 75% 88% 100%

• 1. Total information is increased.

Accuracy is increased when,

• 2. Option imbalance is decreased.

CUE BALANCE – TOTAL INFORMATION TRADEOFF

18

Total Information Cue Balance Cue Balance Total Information

Cells are empty when that combination cannot occur in a 2-option, 4-attribute task.

Take-the-Best (TTB) Weighted-Additive

100% 100 75% 88 91 50% 78 81 85 25% 71 73 77 81 0% 62 68 73 78 13% 25% 38% 50% 63% 75% 88% 100% 100% 88 75% 84 82 50% 79 77 77 25% 75 72 72 72 0% 64 65 67 68 13% 25% 38% 50% 63% 75% 88% 100%

• 1. Total information is increased.
• 1. Total information is reduced while

maintaining cue balance. Accuracy is increased when,

• 2. Cue balance is increased.

• Heuristic and analytic strategies are differentially affected by distributions of incomplete

information

• To increase the accuracy of analytic strategies, increase total information
• To increase the accuracy of heuristic strategies, decrease option imbalance and increase cue

balance.

• Therefore, total information is not sufficient for understanding heuristic decision making with

incomplete information.

• Tradeoffs between total information, option imbalance, and cue balance suggest new

information acquisition and restriction rules

• For heuristics, in particular, acquire and restrict information to decrease option imbalance and

increase cue balance

IMPLICATIONS (CANELLAS, FEIGH, & CHUA, 2015, IEEE THMS)

19

Characterized the impact

• f distribution of

information on decision making strategies Characterized the impact

• f distribution of

information on decision making strategies Define heuristic information acquisition and restriction rules Characterize the effectiveness of the rules for decision making strategies

• 1. Does the distribution of incomplete information

affect decision making performance? If so, when and how?

COMPUTER STUDY 2: DEFINING AND CHARACTERIZING THE EFFECT OF HEURISTIC INFORMATION ACQUISITION AND RESTRICTION RULES

20

Defined new measures

• f incomplete

information

• 2. Can information acquisition and restriction

methods be defined which do not rely on probabilities, etc.? When are they effective?

Define heuristic information acquisition and restriction rules Characterize the effectiveness of the rules for decision making strategies

K K ? ? K K ? ? ? ? ? ? K K K K

Option imbalance: difference in information between options Cue balance: cues with information for all options

50% 80% 0% 100% Option Imbalance

Distribution affects heuristics accuracy. Tradeoffs between total information, option imbalance, and cue balance.

Acc.

FOUR HEURISTIC INFORMATION ACQUISITION AND RESTRICTION RULES

21

Restriction (Remove 1 piece of information) Acquisition (Add 1 piece of information) Cue Balance (CB-R) (Maintain cue balance) Option Imbalance (OI-R) (Reduce option imbalance) Cue Balance (CB-A) (Increase cue balance) Option Imbalance (OI-A) (Decrease option imbalance) Altitude Speed Distance Size Target A K K ? ? Target B K K K ?

A S D S A K K ? ? B K K ? ? A S D S A K K ? ? B K ? K ? A S D S A K K K ? B K K K ? A S D S A K K ? K B K K K ?

Not enough time to process all known information. Have resources to acquire one more cue value.

TESTING THE HEURISTIC INFORMATION ACQUISITION AND RESTRICTION RULES

22

Environment Artificial environment 15 statistics benchmarking simulation environments that represent “real-world” non-linear environments1 Strategies (type)

• Weighted-additive (analytic)
• Equal-weighting (analytic)
• Tallying (heuristic)
• Take-Two (heuristic)
• Take-the-best (heuristic)

Incomplete Information Total information, option imbalance, cue balance

Environments from 1Czerlinski et al., 1999, used in numerous simulations (e.g. Hogarth and Karelaia, 2006; and Katsikopoulos, 2013);

Measured important dimensions of the environments:

• Cues – Task size (3-5 cues used)
• Predictability – R2 of multiple-linear regression
• Distribution – were the cue weights compensatory (e.g.,

WADD), equal (e.g., EW), or non-compensatory (e.g., TTB)

• Redundancy – average of inter-cue correlations
• Variability – difference in maximum and minimum cue weights

100% 73.8 67% 74.4 83.4 33% 75.7 84.5 86.8 0% 87.4 89.9 96.7 17% 33% 50% 67% 83% 100%

Total Information Option Imbalance

2.9% 0.6% 1.3% 1.1% 2.0%

Measure the average accuracy change per acquisition or restriction at the aggregate-level Take-the-best in the “House” environment with 3 cues Option Imbalance – Restriction = 1.58%

HEURISTIC INFORMATION ACQUISITION RULES

23

Restriction (Remove 1 piece of information) Acquisition (Add 1 piece of information) Cue Balance (CB-R) (Maintain cue balance) Option Imbalance (OI-R) (Reduce option imbalance) Cue Balance (CB-A) (Increase cue balance) Option Imbalance (OI-A) (Decrease option imbalance)

5 10

Analytic: gather as much information as possible Heuristic: Use subset of information Weighted-Additive Equal-Weighting Tallying Take-the-best Take Two

The only mediator of the effectiveness of the rules was whether the strategies were heuristic or analytic. The environmental parameters were not significant (predictability, distribution, redundancy, variability).

Average Accuracy Change per Acq/Restr

• Defined heuristic information acquisition and restriction rules which avoid the

psychological and environmental issues that make analytic information acquisition challenging

• Acquire information to decrease option imbalance or increase cue balance, or both
• Restrict information to decrease option imbalance and maintain or decrease cue balance
• Computer simulations showed that the rules are
• Generally effective: especially for acquisition, and especially for heuristics
• Transparent and easy to communicate: create a balance of information between options and

within cues

• Simple to implement: requiring only knowledge of the distribution; no probabilities, cue

weights, etc.

HEURISTIC DECISION SUPPORT METHODS & DECISION SUPPORT FOR HEURISTICS (CANELLAS AND FEIGH, 2017, IEEE THMS)

24

• 1. Does the distribution of incomplete information

affect decision making performance? If so, when and how?

RESULTS SUMMARY

25

Defined new measures

• f incomplete

information Characterize the impact

• f distribution of

information on decision making strategies

• 2. Can information acquisition and restriction

methods be defined which do not rely on probabilities, etc.? When are they effective?

Define heuristic information acquisition and restriction rules Characterize the effectiveness of the rules for decision making strategies

K K ? ? K K ? ? ? ? ? ? K K K K Option imbalance: difference in information between options Cue balance: cues with information for all options

50% 80% 0% 100% Option Imbalance

Distribution affects heuristics accuracy. Tradeoffs exist between total information, option imbalance, and cue balance.

Acc.

Acquire information to decrease option imbalance

• r increase cue balance, or

both Restrict information to decrease option imbalance and maintain or decrease cue balance Rules are generally effective, especially for acquisition and heuristics. Only mediated by heuristic

• vs. analytic; none of the

environmental parameters.

5 10

• Avg. acc change

Define heuristic information acquisition and restriction rules Characterize the effectiveness of the rules for decision making strategies

HEURISTICS VS. ANALYTICS IS INSUFFICIENT FOUNDATION FOR UNDERSTANDING DECISION MAKING STRATEGIES

26

Heuristics vs. analytics is only a linguistic distinction. To truly explain the differences in results, there must be a functional or mathematical distinction.

Analytics “gather and process all information” Heuristics “ignore some subset of the information”

50% 60% 70% 80% 0% 25% 50% 75% 100% Option Imbalance

Total Information

5 10

Average Accuracy Change Option Imbalance – Acquisition

CONSTRUCTING THE GENERAL LINEAR MODEL OF DECISION MAKING, PT. 1 (CANELLAS AND FEIGH, 2016, JCEDM)

27

𝐷𝑗 = ෍

𝑘=1 𝑜

𝑥

𝑘 ∙ 𝑉 𝑘 𝑏𝑗,𝑘 𝑤

𝐷𝑗 = ෍

𝑘=1 𝑜

𝑥

𝑘 ∙ 𝑉 𝑘 𝒇𝒌 +

𝒃𝒋,𝒌

𝒘 − 𝒇𝒌 𝒜𝒋,𝒌

Cue weight (w): Relative importance of the cues Estimate of missing info (e): Assumed value of a missing piece of information (same range as cue values) Incomplete information (z): 1 if known, 0 if unknown

Standard weighted utility model: Calculate weighted utility (or criterion, C) of a specific option (i) defined by j cue values (𝑏𝑗,𝑘

𝑤 )

Utility function (U): Maps the cue values of the option in its natural units to a general utility for comparison across cues

𝑉

𝑘 𝒜𝒋,𝒌 = 𝟏 = 𝑉 𝑘 𝒇𝒌

𝑉

𝑘 𝒜𝒋,𝒌 = 𝟐 = 𝑉 𝑘 𝒃𝒋,𝒌 𝒘

General linear model of decision making

CONSTRUCTING THE GENERAL LINEAR MODEL OF DECISION MAKING, PT. 2 – FOR BINARY CUE VALUES

28

𝐷𝑗 = ෍

𝑘=1 𝑜

𝑥

𝑘 ∙ 𝑉 𝑘 𝒇𝒌 +

𝒃𝒋,𝒌

𝒘 − 𝒇𝒌 𝒜𝒋,𝒌

𝑫𝒋 = ෍

𝒌=𝟐 𝒐

𝒙𝒌 ∙ 𝑰𝒌 𝒆𝒌 𝒇𝒌 + 𝒃𝒋,𝒌

𝒘 − 𝒇𝒌 𝒜𝒋,𝒌

− 𝒆𝒌𝒅 + 𝚬𝒌

Cue direction (d): Sign of the correlation between the scores and criterion Cutoff value (c): Threshold comparison for binary cues Threshold (𝚬): The difference between cue values is large enough to be meaningfully different

𝐷 = 𝐼 𝒴𝐸 ⊙ 𝒴𝐹 + 𝐵 − 𝒴𝐹 ⊙ 𝑎 − 𝒴 𝐸 ⊙ 𝐷 + 𝒴Δ 𝑋𝑈 Matrix Form

Heaviside utility function (H): Convert cue values into 1’s or 0’s

Strategy Cue Weights (𝑥

𝑘)

Utility Function Estimates of Missing Information Additional Binary Components Final GLM Form Weighted- Additive Compensatory (No restriction): 𝑥

𝑘

Standard Median: ෥ 𝑏𝑘 ~ Equal- weighting Equal: 1 Standard Median: ෥ 𝑏𝑘 ~ Tallying Equal: 1 Binary “Bad”: 0 Threshold is 0 Δj = 0 and Cue Directions are positive 𝑒 = 0 Take-the- Best Non-compensatory (Rank order):

1 4 𝑘−1

Binary “Bad”: 0 Threshold is 0 Δj = 0 and Cue Directions are positive 𝑒 = 0

RECONSTRUCTING THE FOUR MAJOR DECISION STRATEGIES

෍

𝑘=1 𝑜

𝐼 𝑏𝑗,𝑘

𝑤 ∙ 𝑨𝑗,𝑘 − ෥

𝑏𝑘 ෍

𝑘=1 𝑜

𝑓

𝑘 + 𝑏𝑗,𝑘 𝑤 − 𝑓 𝑘 𝑨𝑗,𝑘

෍

𝑘=1 𝑜

1 4

𝑘−1

𝐼 𝑏𝑗,𝑘

𝑤 ∙ 𝑨𝑗,𝑘 − ෥

𝑏𝑘

29

෍

𝑘=1 𝑜

𝑥

𝑘 𝑓 𝑘 + 𝑏𝑗,𝑘 𝑤 − 𝑓 𝑘 𝑨𝑗,𝑘

These are the first mathematical representations of Tallying and Take-the-Best which account for estimates of missing information, thresholds, and cue directions.

Single unifying framework for representing, modeling, and simulating a wide range of judgment* and decision making strategies

• 1. Simple: Look-up and use. Not an algorithm, nor a specialty code.
• 2. Fast: Matrix representation allows for rapid, parallelizable computing.
• 3. Transparent: Allows for specificity in model selection and description. Explicitly models the individual

components of strategies (weights, estimates, utility functions) and the environment (incomplete information).

• 4. Representative: Can approximate “expertise” by changing how much prior information is used to set

the components, specifically weights, estimates, cue directions, cutoffs, and thresholds.

• 5. Applied: Can model any combination of components based on the domain – not restricted to

established models.

• 6. General: Can be fit through human-subjects studies and potentially through machine learning

techniques

BENEFITS OF THE GENERAL LINEAR MODEL

*The GLM can model judgment as classification trees with 3-edges per node 30

A FULL CONTEXTUAL MODEL OF THE DETERMINANTS OF DECISION MAKING PERFORMANCE

Environmental Parameters

(types of, and relationships between, cues and

• ptions)

Weights

(relative importance of attributes)

Estimates

(estimated values of missing information)

Incomplete Information

(distribution of missing information)

Full Information Accuracy

(Accuracy with complete information)

Achievement

(Relative accuracy bounded by [min] random decision accuracy and [max] full information accuracy)

Accuracy

(Percent of correct decisions made) Characteristics of the Environment Characteristics of Strategies Measures of Accuracy Can be affected by training and decision support tools.

More discussion of these in the dissertation document.

Strategies

(information processing and integration)

31

Characterize the impact

• f distribution of

information on decision making strategies Characterize the effectiveness of the rules for decision making strategies Characterize the impact

• f distribution of

information on decision making strategies Characterize the effectiveness of the rules for decision making strategies

• 1. Does the distribution of incomplete information

affect decision making performance? If so, when and how?

STUDY 3: A COMPREHENSIVE EXAMINATION OF STRATEGY COMPONENTS AND INCOMPLETE INFORMATION

32

Defined new measures

• f incomplete

information

• 2. Can information acquisition and restriction

methods be defined which do not rely on probabilities, etc.? When are they effective?

Define heuristic information acquisition and restriction rules

K K ? ? K K ? ? ? ? ? ? K K K K Option imbalance: difference in information between options Cue balance: cues with information for all options

50% 80% 0% 100% Option Imbalance

Distribution affects heuristics accuracy. Tradeoffs exist between total information, option imbalance, and cue balance.

Acc.

Acquire information to decrease option imbalance

• r increase cue balance, or

both Restrict information to decrease option imbalance and maintain or decrease cue balance Rules are generally effective, especially for acquisition and heuristics. Only mediated by heuristic

• vs. analytic; none of the

environmental parameters.

5 10

• Avg. acc change

Objective: Use the general linear model to comprehensively examine the interaction of strategy components (not “heuristic and analytic”) and incomplete information, and the effectiveness of the acq. and restr. rules.

EXAMINING STRATEGY COMPONENTS, INCOMPLETE INFORMATION, AND RULES FOR INFORMATION ACQ. & RESTR.

33

Environment Study 1: Artificial environment Study 2: 15 statistics environments Study 3: 7 machine-learning datasets (306 environments, 70M decision tasks) Strategies (type) 18 variations of strategies

• 2 cutoff values (prior or relative)
• 3 cue weights (Non-compensatory,

Equal, Compensatory)

• 3 estimates of missing information

[Average (accurate), Positive (inaccurate), Negative (inaccurate)] Incomplete Information Total information, option imbalance, cue balance

Environments from the UC Irvine Machine-Learning Database

100% 73.8 67% 74.4 83.4 33% 75.7 84.5 86.8 0% 87.4 89.9 96.7 17% 33% 50% 67% 83% 100%

Total Information Option Imbalance

2.9% 0.6% 1.3% 1.1% 2.0%

In this study, measured the average accuracy change per acquisition

• r restriction at the

decision task level

A S D S A K K ? ? B K ? K ? A S D S A K K ? ? B K K K ?

In Study 2, Measured the average accuracy change per acquisition

• r restriction at the

aggregate level 4.5%

50% 60% 70% 0% 20% 40% 60% 80% 100% 50% 60% 70% 0% 20% 40% 60% 80% 100%

EFFECT OF OPTION IMBALANCE DETERMINED BY ESTIMATES

Estimate missing information accurately (average) Estimate missing information inaccurately (positive or negative) Option Imbalance Accuracy

Altitude Speed Distance Size Target 1 ? ? ? ? Target 2 K K K K Altitude Speed Distance Size Target 1 K K ? ? Target 2 K K ? ?

Weighted-Additive Equal-Weighted Tallying Take-the-Best

34

TOTAL INFORMATION – OPTION IMBALANCE TRADEOFF

100% 53 80% 53 60 Option Imbalance 60% 54 61 65 40% 55 61 65 67 20% 54 61 65 67 69 0% 59 64 67 69 71 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Total Information

Inaccurate estimates

• f missing information

100% 66 80% 65 66 Option Imbalance 60% 64 66 67 40% 62 65 66 68 20% 59 63 66 68 69 0% 60 65 67 69 71 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Total Information

Accurate estimates

• f missing information

35

• 1. Total information is increased.
• 1. Total information is increased.

Accuracy is increased when,

• 2. Option imbalance is decreased.

Cells are empty when that combination cannot occur in a 2-option, 5-cue task.

TOTAL INFORMATION – CUE BALANCE TRADEOFF

100% 71 80% 70 70 Cue Balance 60% 67 68 68 40% 64 65 66 66 20% 59 61 63 63 64 0% 54 58 58 60 60 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Total Information

Inaccurate estimates

• f missing information

100% 72 80% 70 70 Cue Balance 60% 67 68 68 40% 64 66 67 67 20% 59 63 65 66 66 0% 59 62 64 65 65 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Total Information

Accurate estimates

• f missing information

Inaccurate estimates

• f missing information

Accurate estimates

• f missing information

36

• 1. Total information is increased.
• 1. Total information is increased.

Accuracy is increased when,

• 2. Cue balance is increased.

Cells are empty when that combination cannot occur in a 2-option, 5-cue task.

HEURISTIC INFORMATION ACQUISITION AND RESTRICTION RULES

Accurate Estimates Inaccurate Estimates

Restriction (Remove 1 piece of information) Acquisition (Add 1 piece of information) Cue Balance (CB-R) (Maintain cue balance) Option Imbalance (OI-R) (Reduce option imbalance) Cue Balance (CB-A) (Increase cue balance) Option Imbalance (OI-A) (Decrease option imbalance)

• 1.2
• 0.8

0.4 0.8

• 1.4

0.1 1.1 2.6

• 2
• 1

1 2 3 Average Accuracy Change per Acq/Restr

• Restrict information to reduce option imbalance
• Acquire information to reduce option imbalance
• Don’t restrict information
• Acquire information to reduce option imbalance

37

* +/- 0.5% change in accuracy can be considered significant

Accurate estimates

• f missing information

Inaccurate estimates

• f missing information

Effect of distributions of incomplete information Only total information matters Option imbalance and cue balance dominate Heuristic information restriction BAD Reduces accuracy (-1%) OK Maintain accuracy by reducing

• ption imbalance (0.1%)

Heuristic information acquisition GOOD Increase accuracy by reducing option imbalance (0.8%) or Increasing cue balance (0.4%) GREAT Increase accuracy by reducing option imbalance (2.6%) or Increasing cue balance (1.1%)

COMPUTATIONAL RESULTS SUMMARY

38

* +/- 0.5% change in accuracy can be considered significant

IMPLICATIONS FOR DECISION SUPPORT BASED ON COMPUTATIONAL RESULTS

• Focus on improving estimates of missing information:
• Leverage and develop algorithms and interfaces for improving accuracy of

estimates of missing information

• Train participants to better estimate missing information – likely based on

expertise and historical trends

• However, the heuristic rules might be limited:
• People are known to successfully adapt their estimates of missing information1
• The rules don’t really matter when estimates are accurate

1Garcia-Retamero and Rieskamp, 2008

How well do the mathematical results represent human decision making with incomplete information? Specifically, are participants with accurate estimates robust to option imbalance and cue balance?

39

• 1. Does the distribution of incomplete information

affect decision making performance? If so, when and how?

HUMAN-SUBJECTS STUDY: DO DISTRIBUTIONS AFFECT DECISION MAKERS WITH ACCURATE ESTIMATES?

40

Defined new measures

• f incomplete

information Characterized the impact

• f distribution of

information on decision making strategies

• 2. Can information acquisition and restriction

methods be defined which do not rely on probabilities, etc.? When are they effective?

Defined heuristic information acquisition and restriction rules Characterized the effectiveness of the rules for decision making strategies

K K ? ? K K ? ? ? ? ? ? K K K K Option imbalance: difference in information between options Cue balance: cues with information for all options For inaccurate estimates, distribution matters and tradeoffs exist between total information, option imbalance, and cue balance.

Acc.

Acquire information to decrease option imbalance

• r increase cue balance, or

both Restrict information to decrease option imbalance and maintain or decrease cue balance Restriction maintains accuracy for incorrect

• estimates. Acquisition

works for both estimates. Not mediated by environmental parameters.

• Avg. acc change

Human-subjects study: Participants with accurate estimates were still affected by distributions as predicted by strategies with inaccurate estimates. Better distributions increased accuracy and reduced time.

50% 60% 70% 0% 20% 40% 60% 80% 100%

• 2
• 1

1 2 3

• 30 participants in a 2-option, 4-cue naval

defense task with 20 second task limits

• Independent variables (156 tasks)
• 13 levels of incomplete information
• 2 accuracy/estimates pairs
• 2 levels of difficulty
• 3 repetitions
• Dependent variables
• Estimates of missing information
• Correct estimates (average)
• Incorrect estimates

(positive/negative)

• Correct or incorrect decisions
• Time required

ARE DECISION MAKERS WITH ACCURATE ESTIMATES REALLY ROBUST TO DISTRIBUTIONS?

Participants quickly adapted and averaged 94% accuracy after Block 2... But what explains the 6% incorrect decisions? After Block 2, order and difficulty were not significant

41

Predicted Responses

50% 60% 70% 80% 90% 100% 1 2 3 4 5 6 Correct (Average) Incorrect (Positive/Negative)

Using correct/average estimates would result in the correct decision for every decision task (so the red line is equivalent to accuracy).

If only correct estimates were used, the predicted responses would be correct (100%) and incorrect (64%) due to task coupling.

DECISION MAKERS WITH ACCURATE ESTIMATES ARE STILL AFFECTED BY DISTRIBUTIONS

Ways to increase accuracy and reduce time: 1) Reduce option imbalance; 2) Increase cue balance while maintaining total information; 3) Maintain cue balance while reducing total information

100% 95 50% 96 91 80 0% 99 97 98 99 25% 50% 75% 100% Total Information 100% 4.7 50% 4.8 5.1 5.4 0% 3.4 4.4 4.5 4.1 25% 50% 75% 100% Total Information 100% 99 75% 99 50% 99 88 25% 99 91 0% 92 95 25% 50% 75% 100% Total Information 100% 4.1 75% 3.8 50% 3.5 5.3 25% 3.1 5.0 0% 4.3 4.9 25% 50% 75% 100% Total Information

Participant Accuracy Participant Time

Option Imbalance 100% 50% 0% 25% 50% 75% 100% Total Information Cue Balance 100% 75% 50% 25% 0% 25% 50% 75% 100% Total Information

Computer with Accurate Estimates

42

IMPLICATIONS OF THE HUMAN-SUBJECTS STUDY

• Identified a potential reality gap in the understanding of decision making with

incomplete information

• The mathematical models (based on prior human-subjects studies) showed that accurate

estimates should make decision makers unaffected by how information is distributed.

• In practice, participants with accurate estimates were still affected by distributions. In some

cases, almost as if they had inaccurate estimates.

• Some distributions may just be difficult for human decision makers regardless of the

estimates

• Friendly distributions – low option imbalance and high cue balance
• Unfriendly distributions – high option imbalance and low cue balance

43

• 1. Does the distribution of incomplete information

affect decision making performance? If so, when and how?

THESIS RESULTS

44

Defined new measures

• f incomplete

information Characterized the impact

• f distribution of

information on decision making strategies

• 2. Can information acquisition and restriction

methods be defined which do not rely on probabilities, etc.? When are they effective?

Defined heuristic information acquisition and restriction rules Characterized the effectiveness of the rules for decision making strategies

K K ? ? K K ? ? ? ? ? ? K K K K Option imbalance: difference in information between options Cue balance: cues with information for all options For inaccurate estimates, distribution matters and tradeoffs exist between total information, option imbalance, and cue balance.

Acc.

Acquire information to decrease option imbalance

• r increase cue balance, or

both Restrict information to decrease option imbalance and maintain or decrease cue balance Restriction maintains accuracy for incorrect

• estimates. Acquisition

works for both estimates. Not mediated by environmental parameters.

• Avg. acc change

Human-subjects study: Participants with accurate estimates were still affected by distributions as predicted by strategies with inaccurate estimates. Better distributions increased accuracy and reduced time.

50% 60% 70% 0% 20% 40% 60% 80% 100%

• 2
• 1

1 2 3

𝑫𝒋 = ෍

𝒌=𝟐 𝒐

𝒙𝒌 ∙ 𝑰𝒌 𝒆𝒌 𝒇𝒌 + 𝒃𝒋,𝒌

𝒘 − 𝒇𝒌 𝒜𝒋,𝒌

− 𝒆𝒌𝒅 + 𝚬𝒌

CONTRIBUTIONS: WHAT CAN WE DO NOW THAT WE COULDN’T DO BEFORE?

45

1 . A P P L I E D D E C I S I O N S U P P O R T 2 . D E C I S I O N T H E O R Y 3 . G E N E R A L P U B L I C

Gather qualitative and quantitative information about the operator cues, processes, decisions, outcomes Actively select parameters for the GLM or train the GLM using machine learning techniques Test the GLM in various environments, with varying types of incomplete information Determine the most robust combinations of cue weights, estimates, etc. Translate those combinations into training tools for how to estimate and integrate information Use distributions of incomplete information to inform decision about saliency and availability

• f information

WHAT CAN WE DO NOW THAT WE COULDN’T DO BEFORE?

• - FOR APPLIED DECISION SUPPORT --

Information

Task: Design training and decision support tools for operators within degraded and denied information environments.

Altitude > 5000 ft Speed < 500 kts Non- hostile

No Yes No Yes

Hostile Non- hostile

46

WHAT CAN WE DO NOW THAT WE COULDN’T DO BEFORE?

• - FOR DECISION THEORY --

47

How do we integrate judgment and decision making?2

𝑫𝒋 = ෍

𝒌=𝟐 𝒐

𝒙𝒌 ∙ 𝑽𝒌 𝒇𝒌 + 𝒃𝒋,𝒌

𝒘 − 𝒇𝒌 𝒜𝒋,𝒌

Environment

Weights Estimates Incomplete Information Full Information Accuracy Achievement

Accuracy

Use the GLM to integrate mathematically, along with aspects of naturalistic decision making (reducing model complexity) and fast-and-frugal heuristics (integrating their diverse components) How does incomplete information affect decision making performance?1 Measure distributions of information using option imbalance and cue balance. Estimates of missing information and distributions interact to determine accuracy. How do we overcome our methodological bias toward examining only “well- studied” strategies like WADD/TTB?3 Deconstruct strategies into their components and test those components as independent variables.

𝑈𝑈𝐶 = 𝐷 𝑥, 𝑓, 𝑨, 𝑑, 𝑒, Δ 𝐷𝑈𝑈𝐶 = ෍

𝑘=1 𝑜

1 4

𝑘−1

𝐼 𝑒𝑘 ∙ 𝑏𝑗,𝑘

𝑤 ∙ 𝑨𝑗,𝑘 − 𝑒𝑘 ∙ ෥

𝑏𝑘 + Δj

47

1Martignon and Hoffrage, 2002; Garcia-Retamero and Rieskamp, 2008; 2Katsikopoulos, 2011, 2013; Martignon et al., 2008; Hogarth and Karelaia, 2005a, 2007 3Hilbig, 2010; Katsikopoulos, et al., 2010; Luan, et al., 2014;

Judgment (Classifying single object) Decision Making (Choice among options)

WHAT CAN WE DO NOW THAT WE COULDN’T DO BEFORE?

• - FOR THE GENERAL PUBLIC --

48

Task: Decide between two routes when driving to work.

If you are confident you have good estimates of the missing information and have the resources to process the information… Then, input those estimates and make the decision. If you are unconfident in your estimates, know they’re inaccurate,

• r don’t have enough resources to process the information…

Then, ignore unbalanced cues to focus on the cues with information for all options

Start B 10 miles Traffic = ? Street roads Route A Distance = ? Heavy traffic Highway Destination

Distance Traffic Road Type Route A ? = 5 miles Heavy Highway Route B 10 miles ? = Light Street Distance Traffic Road Type Route A ? Heavy Highway Route B 10 miles ? Street

49

Thank you

Acknowledgements

• My advisor, Dr. Feigh
• My committee, Dr. German, Dr. Pritchett, Dr. Cross, and Dr. Rogers
• The Office of Naval Research and National Science Foundation
• Members of the Cognitive Engineering Center
• Ladies and Gentlemen of the German Research Group
• Friends of Student Government Association
• Rachel, Orca, and the rest of my family

WHERE DO WE GO FROM HERE?

T W O B R A N C H E S : 1 ) A P P L I E D 2 ) T H E O R E T I C A L

50

APPLIED: TRANSLATE THE CURRENT ACHIEVEMENTS INTO DECISION SUPPORT THROUGH HITLS

51

Compare the heuristic information acquisition and restriction rules to Bayesian optimal information acquisition methods

Alt Speed Dist. Size Target A 5,000 ft Heuristic 5 nmi Bayesian Target B 1,500 ft 1500 kts ? ?

Test implementation schemes for modifying information according to the rules (saliency and availability) Further examine the estimates and incomplete information - repeat HITL with new methods for identifying participants estimates within each task

Alt Speed Dist. Size Target A 5,000 ft Input estimate 5 nmi Input estimate Target B 1,500 ft 1500 kts Input estimate Input estimate Alt Speed Dist. Size Target A 5,000 ft f(Size) 5 nmi f(Dist,Alt) Target B 1,500 ft 1500 kts f(Alt,Speed) f(Dist,Alt)

Test methods for supporting accurate estimates within bad distributions of incomplete information by inputting estimates.

APPLIED: IMPLEMENT A MIXED-METHOD APPROACH FOR DEVELOPING FAST-AND-FRUGAL TREES FOR DECISION SUPPORT

52

Example from Keller and Katsikopoulos (2014) Task: Identification of friend or foe at military checkpoint in Iraq1 Test: 1053 incidents Baseline: 204 civilian casualties Fast-and-Frugal Tree: 78 (theory)

• 1. Identify cues, actions, and

constraints

• 2. Determine relevant ordering of cues.
• 3. Construct and simulate FFTs
• 4. Signal-detection analysis and selection

Are there multiple

• ccupants?

Does the vehicle comply (slow/stop)? Are there no further threat attributes? Non- hostile

No Yes No Yes Yes

Hostile Non- hostile Hostile

No

• FFTs have binary predictors and outcomes with one exit at each level, two

at final level

• Derived from data and qualitative methods1 or from simplifying random

forests2

• Accurate and robust3 in military, finance, and medical domains
• Transparency, consistency, and communicability lead to more acceptance

than actuarial methods4

1Katsikopoulos et al., 2008; Martignon et al., 2008; Woike (2008); 2Deng, 2014; 4Katsikopoulos et al., 2008; Elwyn et al., 2001; Green & Mehr, 1997;

THEORETICAL: LINK JUDGMENT AND DECISION MAKING TO STUDY SEQUENTIAL AND CASCADING DECISION EVENTS

53

Environmental Values Judgment 𝐷 = 𝑔 𝑥, 𝑓, 𝑉, 𝑨 𝐻 = 𝑈

𝑘 𝐷

AE, ZE AJ, ZJ Decision Making 𝐷 = 𝑔 𝑥, 𝑓, 𝑉, 𝑨 𝐻 = max

𝑗

𝐷𝑗 i: Selected Option Action

Decision Choice Environmental Values, Context, Operator Experience, Biases, etc.

Judgment Decision Making

Single Decision Event

Action

Cascading Decision Events Inputs Decision Event Simulation Output

Conceptual Form Mathematical Form

DECISION TASKS FOR THIS THESIS

54

Judgment (Classifying single object) Decision Making (Choice among options) Dynamic Static Sequential Single Analytics Heuristics Do the cue values change during the decision making process?

The GLM enables the study of sequential, dynamic decision tasks and judgment tasks

Yes No Are options presented one at a time? Yes No Is all information processed by the strategy? Yes No

𝑫𝒋 = ෍

𝒌=𝟐 𝒐

𝒙𝒌 ∙ 𝑽𝒌 𝒇𝒌 + 𝒃𝒋,𝒌

𝒘 − 𝒇𝒌 𝒜𝒋,𝒌

Journal 1. M.C. Canellas and K.M. Feigh, “Heuristic Information Acquisition and Restriction for Decision Support,” IEEE Transactions on Human-Machine Systems. (In Press). 2. M.C. Canellas and K.M. Feigh, “Toward Simple Representative Mathematical Models of Naturalistic Decision Making through Fast-and-Frugal Heuristics,” Journal of Cognitive Engineering and Decision Making, vol. 10, 2016,

• pp. 255–267.

3. M.C. Canellas, K.M. Feigh, and Z.K. Chua, “Accuracy and Effort of Decision-Making Strategies With Incomplete Information: Implications for Decision Support System Design,” IEEE Transactions on Human-Machine Systems,

• vol. 45, Dec. 2015, pp. 686–701.

Conference 1. M.C. Canellas and K.M. Feigh, “Heuristic decision making with incomplete information: Conditions for ecological rationality,” Systems, Man and Cybernetics (SMC), 2014 IEEE International Conference on, 2014, pp. 1963–1970. 2. M.C. Canellas, K.M. Feigh, and Z.K. Chua, “Accuracy and effort of decision making strategies with incomplete information,” Cognitive Methods in Situation Awareness and Decision Support (CogSIMA), 2014 IEEE International Inter-Disciplinary Conference on, 2014, pp. 7–13.

PUBLICATIONS RELATED TO THIS WORK

• 1. A general linear model of

judgment and decision making.

Mathematical, computational, and theoretical contributions of the model. Planned for Psychological Review.

• 2. Determinants of decision making

accuracy with incomplete information.

Decision making simulation extended to decision tasks with three options instead of two. Planned for Judgment and Decision Making

PAPERS - PLANNED

• 3. Incomplete information and the reality gap:

Disagreements between human decision makers and their computational representations

Contrasts the computer results with a the human subjects study. Planned for Psychological Science.

• 4. As they are: Representing and supporting

the heuristics of military decision makers.

Review of the capabilities of the general linear model and military heuristics for modeling and simulation of military decision making. Planned for Military Operations Research Journal.

WHAT’S NEXT?

• 2017-2018/19: Selected as IEEE Congressional Fellow to serve as a Science and

Technology Advisor in Congress

• 2018/19: Law school
• Have become very interested in robot and technology law
• Published a few articles about governing robotics and autonomous weapons
• M. C. Canellas, M. J. Miller, Y. Razin, R. A. Haga, D. Minotra, and R. Battacharrya, “Framing Human-

Robot Regulation: A New Modus Operandi from Cognitive Engineering,” WeRobot, 2017

• M. C. Canellas and R. A. Haga, “Toward Meaningful Human Control of Autonomous Weapons

Systems through Function Allocation,” in IEEE International Symposium on Technology and Society (ISTAS 2015), 2015.

• M. C. Canellas and R. A. Haga, “Toward Technical International Regulations of Autonomous

Weapons,” in IEEE Technology and Society Magazine (Sept, 2016).

HOW DOES THE TAKE-THE-BEST MATH WORK?

• Margitnon and Hoffrage (2002) proved that that Take-the-Best decisions with can be represented by a

linear model with non-compensatory weights with binary cues {0, 1}

• Find weights [0,1] such that the following condition is satisfied:

Condition 1: 𝐵 1,0, … , 0 preferred to 𝐶 0,1, … , 1

𝑥

𝑘 > σ𝑙>𝑘 ∞

𝑥𝑙 𝑥

𝑘 =

1 2

j−1

= 21−j

58

Modeling TTB with full information

Cue 1 Cue 2 Cue 3 Cue … Cue n Target 1 1 Target 2 1 1 1 1

HOW DOES THE TAKE-THE-BEST MATH WORK?

• To account for estimates of missing information, I solved for three separate conditions allowing missing

information to have three states {0, .5, 1} Condition 1: 𝐵 1,0, … , 0 preferred to 𝐶 0,1, … , 1 Condition 2: 𝐵 1,0, … , 0 preferred to 𝐶 .5,1, … , 1 Condition 3: 𝐵 .5,0, … , 0 preferred to 𝐶 0,1, … , 1

59

Cue 1 Cue 2 Cue 3 Cue … Cue n Target 1 1 Target 2 1 1 1 1 Cue 1 Cue 2 Cue 3 Cue … Cue n Target 1 1 Target 2 ? = 0.5 1 1 1 1 Cue 1 Cue 2 Cue 3 Cue … Cue n Target 1 ? = 0.5 Target 2 1 1 1 1

𝑥1 > ෍

𝑙=2 ∞

𝑥

𝑘

𝑥1 > 0.5 ⋅ 𝑥

𝑘 + ෍ 𝑙=2 ∞

𝑥

𝑘

1 > 0.5 > ෍

𝑙=2 ∞

𝑥

𝑘

0.5 ⋅ 𝑥1 > ෍

𝑙=2 ∞

𝑥

𝑘

𝑥

𝑘 =

1 4

j−1

= 41−j

MATHEMATICAL DEFINITIONS OF OPTION IMBALANCE AND CUE BALANCE 𝑈𝐽 = ෍

𝑗=1 𝑛

෍

𝑘=1 𝑜

𝑨𝑗,𝑘 ⋅ 100 𝑛 ⋅ 𝑜

• Fixed (n)
• Percentages calculated in

reference to the total amount of cues in the task

• Relative (n’)
• Percentages calculated in

reference to the total amount of cues that have some information known

60

𝑃𝐽 = max

𝑗

෍

𝑘=1 𝑜

𝑨𝑗,𝑘 − min

𝑗

෍

𝑘=1 𝑜

𝑨𝑗,𝑘 ⋅ 100 𝑜 𝐷𝐶 = ෍

𝑘=1 𝑜

1𝑜 ෍

𝑗=1 𝑛

𝑨𝑗,𝑘 ⋅ 100 𝑜

Cue 1 Cue 2 Cue 3 Cue 4 Target 1 1 ? Target 2 1 1 ?

n = 4 but n’ = 3

• Analyzes each object/option independently of others.
• More work required to measure effort or time.
• Cannot model strategies like Take Two.
• A method for mathematical fitting has yet to be developed.

LIMITATIONS OF GENERAL LINEAR MODEL

• The lack of formal specification of commonly used strategies in the literature leads to confusion as to what

ways certain strategies are the same or different

• Tallying: “simply adds binary cue values”(Katsikopoulos, 2010)
• Equal-Weighting: “simply summing the values for each attribute” (Payne et al., 1990)
• Furthermore, authors typically do not specific how missing information was estimated or what thresholds

are being used, just referencing strategies by proper names.

MATHEMATICAL FORMS OF WELL-KNOWN DECISION MAKING STRATEGIES

𝐷𝑈𝐵𝑀 = ෍

𝑘=1 𝑜

𝐼 𝑒𝑘 ∙ 𝑏𝑗,𝑘

𝑤 ∙ 𝑨𝑗,𝑘 − 𝑒𝑘 ∙ ෥

𝑏𝑘 + Δj 𝐷𝐹𝑋 = ෍

𝑘=1 𝑜

𝑓

𝑘 + 𝑏𝑗,𝑘 𝑤 − 𝑓 𝑘 𝑨𝑗,𝑘

Final Cue Value: If known (z = 1), use cue value. If unknown (z= 0), estimate as 0. Threshold for the Heaviside function: Tallying converts values to binary using the median of possible cue values General linear model for non-binary cues without weights

PROOF THAT THE ESTIMATES MATTER

• Start with the general linear model with one cue, binary

cue values, such that

• ν is the probability that an option is correct when its cue

score is 1 and the other cue score is 0

63

Cue 1 A (Target 1) 1 B (Target 2) 𝑉 𝐵 > 𝑉 𝐶 𝑓 + 𝑏 𝐵 − 𝑓 𝑨 𝐵 > 𝑓 + 𝑏 𝐶 − 𝑓 𝑨 𝐶

z(A) z(B) Option Imbalance Cue Balance Decision Accuracy Equation e = 1 e = 0 e=.5 1 1 0% 100% 1>0 ν ν ν 1 100% 0% 1>e 50% ν ν 1 100% 0% e>0 ν 50% ν 0% 0% e>e 50% 50% 50%

Let’s say that Cue 1 is a perfect cue (v = 100%). Therefore, the average accuracy when estimating missing information as 1 or 0 is 75%; but when estimating missing information as 0.5, the accuracy is 87.5%.

JUDGMENT

𝑏1 𝑏1 > 𝑑1 𝑏2 > 𝑑2 𝑏2 < 𝑑2 𝑏1 𝑏1 > 𝑑1 𝑏2 > 𝑑2 𝑏2 < 𝑑2 𝑏1 > 𝑑1 𝑏2 > 𝑑2 𝑏2 = 𝑑2 𝑏2 < 𝑑2 𝑏1 = 𝑑1 𝑏2 > 𝑑2 𝑏2 = 𝑑2 𝑏2 < 𝑑2 𝑏1 < 𝑑1 𝑏2 > 𝑑2 𝑏2 = 𝑑2 𝑏2 < 𝑑2 𝒙𝟐 = 𝟐 𝒙𝟑 = 𝟐 𝟓 𝒙𝟑 = 𝟐 𝟓 0.5 1 0.5 1 0.5 1 1 0.5 𝒙𝟐 = 𝟐 𝟏 𝟐 𝟓 𝟐 𝟔 𝟓 𝟏 𝟔 𝟓 𝟐 𝟓 1 1 1 𝟐 𝟗 𝟘 𝟗 𝟐 𝟒 𝟓 𝟔 𝟗 𝟐 𝟑 𝑫 𝑫

Non-compensatory cue weights with 3 cue score states {0,0.5,1} Non-compensatory cue weights with 2 cue score states {0,1} Weights Prior model (2 cue states) New model (3 cue states) Equal (e.g. Tallying) 𝑜 + 1 2𝑜 + 1 Non-Compensatory (e.g. TTB) 2𝑜 3𝑜

HOW DOES THE MATRIX GLM WORK? TEST.

65

Variables Description Example 𝐵 Cue values [𝑜_𝑑ℎ𝑗𝑚𝑒, 𝑓𝑒𝑣, 𝑥𝑝𝑠𝑙] 𝑋 Cue weights (non-CF) [1, 0.25, 0.0625] 𝐹 Estimates [1, 1, 1] 𝑎 Information states [1, 1, 1; 1, 1, 1] 𝑌 Transformation matrix [1; 1; 1] 𝐸 Cue directions [−1, 1, −1] 𝐷 Cutoff values [1.5, 2.5, 0.5] Δ Threshold values [0, 0, 0] Statistics Adaptive Toolbox Online Matrix Results True Positive 673 673 True Negative 280 280 False Positive 349 349 False Negative 171 171 Sensitivity Index (d') 0.694 0.6944 Frugality 2.003 2.0034

Categorize a sample of 1,473 cases of married women as either using (1, or signal) or not using (0, or noise) contraceptives):

• 1. Number of children ever born (0 to 12)
• 2. Wife’s education (1-low to 4-high)
• 3. Is wife working? (0-yes, 1-no)

The Adaptive Toolbox Online is a online tool for creating and testing fast-and-frugal trees. It is maintained by the Center for Adaptive Behavior in Berlin, who developed fast-and-frugal trees.

ACCURACY, FULL INFORMATION ACCURACY, AND ACHIEVEMENT

66

𝐻 𝐵 = 𝐵 − 𝑆 𝐺 − 𝑆

Achievement (G) measures the relative accuracy bounded by the minimum accuracy of random selection (G(R) = 0%) and the maximum of full information accuracy (G(F) = 100%)

OBJECTIVE: BETTER MODELING, SIMULATION, AND SUPPORT OF MILITARY OPERATORS

• 1. Military operators use naturalistic heuristics (simple decision algorithms, pattern

recognition, etc.) based on their experiences and expertise to make quick and accurate decisions – especially when faced with limited time, information, resources, or cognition.

• 2. Our new general linear model of judgment and decision making can mathematically

and transparently represent these types of strategies (and more) while accounting for expertise and incomplete information.

• 3. Leveraging the perspectives of naturalistic heuristics and the mathematics of the GLM

will enable new ways to

• Model and simulate military operators
• Develop prescriptive strategies for better performance
• Design support tools and interfaces

REPRESENTING EXPERIENCE/EXPERTISE 1 Model Calibration Amount/type of experience should be proportional to the amount/type of data used to calculate components:

Cue weights and cue order (w)3; Cue directions (d)4; Estimates of missing information (e)5; Cutoff values (c); and Thresholds (Δ)

Experience Experts tend to use naturalistic heuristics (fewer, more important cues)2 Novices tend to use normative and analytic strategies (more, less important cues)2

1Canellas and Feigh, 2016 2Gigerenzer and Gaissmaier (2011); Todd et al., (2012); 3Czerlinski et al., 1999; 4Katsikopoulos et al, 2010; 5Garcia-Retamero and Rieskamp, 2009

http://snagfilms.s3.amazonaws.com/56/ee/cc0b41404c81ad05c2a4ed7aebfc/635933100424826639-sealsjpg

REPRESENTING TIME PRESSURE & DEGRADED AND DENIED INFORMATION ENVIRONMENTS

1Canellas and Feigh, 2016; 2Rieskamp and Hoffrage, 1999, 2008; Maule, 1994; Payne et al., 1988; 3Garcia-Retamero and Rieskamp, 2008, 2009; 4Canellas and Feigh, 2015, 2016 http://navsource.org/archives/08/750/0876711.jpg

High time pressure results in:

• More use of heuristics and

selective information search2

• More incomplete information

How be robust degraded & denied information environments?

• Well-calibrated estimates of

missing information3

• Focus on the balanced

information (option-wise and cue-wise)4

DEVELOPING FAST-AND-FRUGAL TREES FOR CHECKPOINTS IN IRAQ1

Task: Classify incoming vehicle as hostile or non-hostile

ISAF-NATO Forces in Afghanistan 2004-2009

1Keller and Katsikopoulos, 2014; Keller et al., 2014

Identify cues and categories, and their quantifications Construction and simulation of FFTs Analysis and final selection of FFT for implementation in the field

FAST-AND-FRUGAL TREE FOR CHECKPOINTS 1

Are there multiple

• ccupants?

Does the vehicle comply (slow/stop)? Are there no further threat attributes? Non- hostile

No Yes No Yes Yes

Hostile Non- hostile Hostile

No

Development

• Constructed prior to testing on 1053

incidents Results

• Civilian casualties reduced from

204 to 78 (-60%)

• For attacks, only occupants measured:

Never multiple occupants in attack (agreement with FFT)

• Average of 1.2 cues used
• Sequential and deterministic information

search How can the GLM help address the “challenges” faced during development of FFTs?

1Keller and Katsikopoulos, 2014; Keller et al., 2014

• 1. IDENTIFY CUES, CATEGORIES, & STRATEGIES:

QUAL/QUANT STRATEGY IDENTIFICATION

Identify cues and categories (actions) and their related constraints Quantify the cues and categories. Determine relevant

• rdering of cues.

Challenges GLM Solutions

Too many cues/Difficult ordering: Need to down-select cues and find relevant order Fit operators to the GLM: Construct representative tasks for use in human-subjects studies; then, statistically fit the GLM

Occupants Visible weapons Single/ Multiple Visible or not? Slowing down? Yes/No Occupants: Single/ Multiple Slowing down? Yes/No Visible weapons

• r not?

Speed

2/3. SIMULATION OF FFTS AND FINAL SELECTION: FAST-SIMULATION/PERFORMANCE PREDICTIONS

Accuracy

Challenges GLM Solutions

Computational resource constraints: Had to limit the number of FFTs and cases studied. Matrix-form: In a single matrix multiplication, a single FFT can classify m-objects. Artificiality of simulations/High similarity of final FFTs: Did not account for individuals’ “differential ability to perceive cues.” Lots of “good-enough” FFTs. Vary components: Directly model “thresholds” and vary training data to represent experience. Measure robustness.

Construct large numbers

• f FFTs

Construct simulation environment Signal-detection analysis 1 2 1 2

p(Hit) p(FA)

1 2

• 4. DESIGN TRAINING AND DECISION SUPPORT

(NATURALISTIC HEURISTICS) Build training and support structures based on the operators’ context and decision making strategies “I thought, therefore, I build.”

Methods: Brief literature review (if any), pontifications about what operators need Result: Unusable systems, failures

“I thought about what I saw, therefore, I build.”

Methods: Literature review of psychology and cognitive

• engineering. Applied methods: cognitive work analysis,

goal-directed interviews, and human-subjects studies. Result: Useable, traceable systems, that increase the performance of operators.