The Presentation of Risk and Uncertainty in the Context of National - - PDF document

SLIDE 1

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/304196307

The Presentation of Risk and Uncertainty in the Context of National Missile Defense Simulations

Article in Human Factors and Ergonomics Society Annual Meeting Proceedings · October 2003

DOI: 10.1177/154193120304700365

CITATIONS

2

READS

14

6 authors, including: Some of the authors of this publication are also working on these related projects: Spatial Cognition View project Usability Science View project Patricia L Mcdermott MITRE

49 PUBLICATIONS 203 CITATIONS

SEE PROFILE

Michael Barnes Army Research Laboratory

113 PUBLICATIONS 1,952 CITATIONS

SEE PROFILE

Douglas Gillan North Carolina State University

147 PUBLICATIONS 1,460 CITATIONS

SEE PROFILE

Ling Rothrock Pennsylvania State University

88 PUBLICATIONS 757 CITATIONS

SEE PROFILE

All content following this page was uploaded by Douglas Gillan on 20 October 2017.

The user has requested enhancement of the downloaded file.

SLIDE 2

THE PRESENTATION OF RISK AND UNCERTAINTY IN THE CONTEXT OF NATIONAL MISSILE DEFENSE SIMULATIONS zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

Patricia McDermott’, Shaun Hutchins*, Michael Barnes3, Corey Koenecke’, Doug Gillan*, zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA & Ling Rothrock4 ‘Micro Analysis & Design Boulder, CO 3Army Research Laboratory

• Ft. Huachuca, zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

AZ

*New Mexico State University Las Cruces, NM Pennsylvania State University College Park, PA zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

4

Risk perception and uncertainty management are important components of military decision making, especially in time-stressed and resource-limited environments. The purpose of this experiment was to understand the interaction of integrality of information, presentation mode, and information frame on situation awareness (SA) and decision-making (missile allocation) in a National Missile Defense (NMD) paradigm. Results of the information frame manipulation (expected gain v. expected loss) support earlier findings that subjects are loss averse. SA Accuracy was higher with graphical displays than alphanumeric displays. The implications for NMD are discussed. INTRODUCTION Risk management is an important component of financial, industrial, and military planning. Decision support displays allow planners to predict risk and allocate limited resources based on an understanding of the uncertainty and possible consequences of their decision options. Military risk management involves asset allocation decisions based on criteria such as own and enemy predicted losses (Schlabach, Hayes, & Goldberg, 1999). However, it would be a mistake to assume that these decisions are made with normative

• methods. Humans, especially in stressful environments,

use “rules-of-thumb” or heuristics that are both efficient and non-normative (Klein, 1999; Wickens & Hollands, 2000). The question that this experiment addresses is how to present risk information to take advantage of the efficiencies

• f these heuristics while minimizing the

effects of human cognitive biases. This experiment draws on three main bodies of research investigating uncertainty perception and decision making: visualization, framing, and perceptual

• format. In uncertainty perception, Johnson-Laird,

Legrenzi, Girotto, Legrenzi & Caverni (1999) proposed a theoretical model of human visualization of probability spaces that implied that improving the humans’ visualization of the problem space should lead to improved decision making as long as the process does not require additional mental calculations. Information frame also influences risk perception. When college students were given risk information as losses they tended to be risk seeking and, inversely, were risk averse if the same information was presented as gains (Shafir & Tversky, 1995). This shows a tendency towards loss aversion in both frames. The third consideration is the perceptual format. In general, risk information is better displayed with a graphical format (Smith & Wickens, 1999); however, some studies have found that textual formats were associated with superior performance. The crucial factor seems to be whether information captures the dynamics

• f the process being represented (Meyer, Shamo &

Gopher, 1999; Wickens & Hollands, 2000). chosen because it has extreme time constraints and decisions have deadly consequences. The physics of the sensors and interceptors enable fairly accurate predictions about value of the intended targets, threat probabilities, and the probability of successful

• interception. Operators deploy a limited supply of

defensive missiles known as Ground Based Interceptors (GBIs) based on their evaluation of risk. A National Missile Defense (

N M D ) paradigm was

THE RESEARCH QUESTION The purpose of our study was to understand the interaction of cognitive and perceptual display factors on SA and missile allocation. The following display parameters were varied: zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

0 zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

Presentation Mode: Information about city population and probability of success was Integrated into a value for expected population saved, or kept Separate; Gains (probability of success and expected

0 Frame: Information was conveyed in terms of

I

I

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 47th ANNUAL MEETING—2003 562

SLIDE 3

population saved) or zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA Losses (probability of failure and expected population lost); and zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

e Format: Graphical or Alphanumeric.

We predicted that appropriate GBI allocation and SA would be improved through the enhancements afforded by an integral risk display. Further, we hypothesized that in comparison to the gain frame, participants in the loss frame would be more likely to take assets out of reserve and more likely to put assets

• n cities with no defensive assets, thus avoiding a sure
• loss. Finally, we hypothesized that graphical displays

would be superior to alphanumeric presentations in supporting SA and decision quality. zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA METHODS The experiment was a 2~2x2 mixed factorial design with Presentation Mode zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

as

a between variable. Format and Frame were repeated measures and Frame was

• blocked. The levels of Format, Presentation Mode, and

Frame were combined to create eight displays. Order was counterbalanced. Forty-eight students participated. Some display aspects were common across the eight

• displays. The reserve GBIs used, the reserve GBIs

remaining, and time elapsed were numerically

• represented. Each target city had its own area that

included (from top to bottom): time till the incoming

missile impacts, probability of successfully intercepting the incoming missile, population of the target city, and GBI missile allocation along with up and down arrow buttons that allowed participants to increase or decrease the number of reserve GBIs allocated. The color green was always used to indicate the Gain frame and the color red was used in the Loss h e . The presentation of population, probability, expected value, and GBIs depended on the display. In the Separate Graphic Gain display (Figure 1) a circle represented time until impact. Slices of the circle disappeared as time elapsed so that the more of the circle that was missing, the less time until impact. A vertical bar represented probability of success between zero and

• ne zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
• the higher the bar, the higher the probability of
• success. A horizontal bar represented population

between zero and 4.5 million - the longer the bar, the higher the population. The number of GBIs was represented by a bar graph. The number of GBIs initially allocated by the computer was shown in light blue and the number of reserve GBIs allocated by the participant was dark blue. represented probability of failure as opposed to success. The critical difference in interpreting this display was In the Separate Graphic Loss display the vertical bar that the smaller the bar, the greater the chance of intercepti

1

;

the incoming missile.

._

____

• zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

AI*ol.%.BA EL M Y

rme b

I& zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

9 7

197

6sl Mumlim

~~ ~

4

2 1

Reserves

m/*

~~

gure 1. Separate graphic gain dis] In the Integral Graphic displays (see Figure 2.) the vertical and horizontal bars were fixed to make two

sides of a rectangle. The height still represented

probability of success/failure and the width still represented population. However, the area of the rectangle represented expected population saved (or expected population loss in the Loss frame). The numerical value of expected population saved/loss was

• displayed. The Integral Graphical display condition,

utilized a square presentation as a pilot study found that subjects were more accurate when this type of graphical presentation was used (Gillan & Hutchins, 2002).

I

I

I

,

1

Figure 2. Integral graphic gain display In the four Alphanumeric displays all values were

• numerical. The Integral Alphanumeric displays also

showed the value of expected population saved/lost. See Figure 3 for an example of an Alphanumeric display, specifically the Integral Alphanumeric Gain display.

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 47th ANNUAL MEETING—2003 563

SLIDE 4

RESULTS zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

• ___l___. zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
• Phoenix. zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

AZ

1 1 1

• Gincinnati. OH

Expected

~ ;Expected zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

Population Saved:

1 1 li

Population

Swed:

Population:

1.627.509

.- -

GBI Allocation 1

1

• - -

__

• Figure 3. Integral alphanumeric gain display

There were four types of scenarios that dictated number of target cities, city size, and initial GBI

• allocation. Scenarios were randomly matched to displays

so that each display had one scenario of each type. Scenarios were two minutes long and contained four SA probes and one reserve missile allocation probe. During SA probes the screen was occluded, the scenario was frozen, and participants had 10 seconds to respond. SA probes asked about relationships such as, “Which city has the highest probability of success?” The reserve allocation probes occurred at the end of the scenario and lasted 15 seconds. As participants allocated missiles, the information on the display (Le., probability of success, population saved) was updated dynamically to show the impact of allocations. Participants could change allocations for the full 15 seconds, which was displayed via a countdown. The session began with a self-paced training presentation that described the participant’s role, the displays, and the scenarios. The order of practice and experimental trials was as follows: four practice scenarios (one of each display type), two practice scenarios in the frame that they would be tested in (i.e., Gain or Loss), eight experimental trials in the first frame, two practice scenarios in the second frame, and eight experimental scenarios in the second frame. In a post questionnaire, participants rated confidence in their performance for each display. Confidence ratings were given on five-point scales with 1 being not very confident and 5 being very confident. The study took approximately 90 minutes. The dependent measures recorded by the simulation software were: reserve GBI allocation, SA probe accuracy, and SA probe response time. Decision making was assessed via GBI allocations. SA was measured by SA probe accuracy and response time. In addition, risk behavior was analyzed in relation to whether GBIs were allocated to small cities. Repeated measures ANOVA were performed on the decision score (Le. reserve GBI allocation), accuracy, response time, small city coverage, and confidence ratings. Table 1 lists significant effects. Source

df MS F statistic zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

v value

• A. Decision Score

Frame x Content zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

1

1.34E+11 1 2 . 4 8 4 ,001

• B. Accuracy

Frame

1

. 1 2 9 1 6 . 6 4 2 . 1 Format

1

. 8 1 9 . 3 7 1 .004

FramexFormat

1

.084 7 . 1 6 3 .010

Frame x Format

1

1.368E-03 4.594 . 3 7

Frame

1

1 . 9 1 3 1 2 4 . 3 5 . 1 Format

1

.280 24.306 . 1

FramexFormat

1

.729 6 . 9 2 6 . 1

• E. Small City Analysis: Appropriately Uncovered Cities

Frame

1

,122

5 . 6 1 8 .022 Format

1

.lo5

7.108 . 1 1

FramexFormat

1

.lo5

5 . 7 3 6 zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

.02

1

Frame

1

2 . 8 5 2 5 . 8 4 .020 Format

1

13.230 1 3 . 4 5 4 . 1

Table 1. Significant Effects for Experiment 2

• C. Response Time
• D. Small city Analysis: Appropriately

Covered Cities

• F. Confidence Ratings

Decision Score A decision score was used to measure deviation from normative GBI allocation. Decision score was equal to (Participant’s Expected Population Loss) minus (Expected Value) where the Participant’s Expected Population Loss was the sum of probability of failure multiplied by the population for every city in the scenario including the possibility of a future attack. Therefore, decision score was measured in lives lost. The Expected Value was calculated using the Real Time Decision Tree software (Rothrock, 2002). There was a significant two-way interaction of frame by type of display as seen in Figure 4

.

In Integral displays, performance was better in the Loss condition (163,151 versus 230,313) but in Separable displays performance was better in the Gain condition (203,497 versus 24 1 , 9 7 1).

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 47th ANNUAL MEETING—2003 564

SLIDE 5

SA Accuracy zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA SA accuracy was an average percent correct. A “no response” was coded as incorrect. There was a two-way interaction of frame by type of display. In the Graphical displays there was almost no difference between Gain and Loss conditions (.859 and .849 respectively) but in the Alphanumeric displays, SA accuracy was higher in the Gain condition than in the Loss condition (359 versus .766). There was a main effect of frame and a main effect of type of display. SA accuracy was better in Gain than Loss (A59 versus .807) and better in Graphical than Alphanumeric (.854 versus .813). zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

1

• separate zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

caltent

Figure 4. Decision Score two-way interaction of frame by content SA Response Time Response time was recorded in seconds. The four instances of “no response” were coded at the maximum time (10 seconds). There was a two-way interaction of frame by type of display. In the Alphanumeric displays, response time on both Gain and Loss displays was roughly equivalent (2.725 and 2.694 respectively). However, the Graphical displays, response time as higher in the Loss frame than the Gain frame (2.790 versus 2.588). Small Cities Analysis In order to see if participants were risk seeking or risk averse in relation to small cities (i.e. did they have a bias to not leave any city uncovered and avoid a sure loss) the coverage of small cities was analyzed. The experiment was designed zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA so that in some of the scenarios the normative allocation involved leaving one small city uncovered (i.e., zero GBI allocated to it). For the cities that should have been covered, there were three effects. There was a two-way interaction of frame by type of display. The difference between Gain and Loss displays was relatively large in the Graphical condition (.269 versus S92) compared to the difference between Gain and Loss displays in the Alphanumeric condition (.4688 versus .545 1). There was a main effect

• f frame such that more small cities were appropriately

covered in the Loss displays than the Gain displays (S69 versus .369). There was also a main effect of format such that more small cities were appropriately covered in the Alphanumeric displays than the Graphical displays (SO7 versus .431). For the cities that should have been left uncovered, similar effects were found. There was a two-way interaction of frame by type of display. In the Graphical condition, there were more cities appropriately left uncovered with the Gain displays than the Loss displays (.205 versus .108) while there was virtually no difference between Gain and Loss displays in the Alphanumeric condition zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA (. 1 1 1 versus zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA . 1 8 ) . There was a main effect of frame with a higher percentage of subjects correctly leaving cities uncovered in the Gain frame than in the Loss frame (. 158 versus .108). There was a main effect of type of display with a higher percentage of subjects correctly leaving cities uncovered in the Graphical condition than in the Alphanumeric condition (.156 versus .109). Confidence Ratings There was a main effect of frame in which Gain was rated higher than Loss (3.744 versus 3.500). There was a main effect of type of display in which Graphic was rated higher than Alphanumeric (3.884 versus 3.359).

I

DISCUSSION The studies were designed to explicate the relationship between visualization (graphical zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

v.

alphanumeric display, integral v. separable display) and the framing decision bias. Humans attempt to avoid sure losses even when the alternative decision would lead to a higher expected value for lives saved. Specifically, loss functions are steeper than gain functions (i.e., the utility

• f saving lives is less that utility of not losing lives

(Shafir & Tversky, 1995). terms of losses the results will be different than if it was framed in terms of possible gains. We expected participants to take more missiles out of reserves to avoid losses compared to the same situation wherein the results were framed in terms of possible lives saved. We predicted this behavior even though the importance of keeping some missile in reserve was emphasized in the

• instructions. Although the framing trend was in the

predicted direction, it was not significant. Participants tended to take out almost all their reserves in both framing conditions resulting in sub-optimal decisions. The average number of missiles left in reserve was less

I

I

This implies that if the same situation is framed in

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 47th ANNUAL MEETING—2003 565

SLIDE 6

than one in both cases. This trend agrees with previous research and is probably an extension of the loss avoidance principle. In another experiment, NMD

• perators reported being more concerned about the

present attack than about possible future attacks to the extent that they tended to underweight the probabilities

• f the future attack (Barnes, Wickens zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

& Smith, zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

2000). Again, this makes sense if they are biased towards reducing losses in the present compared to possible future losses. We also set up conditions where the normative solution was to protect the larger cities and leave a small city unprotected (uncovered). This was an attempt to replicate the risk seeking behavior that Tversky and Kahneman (1 98 1) found in the original study (replicated recently by Mayhorn, Fisk and Whittle (2002). The logic was that test subjects would avoid leaving a small city uncovered (because it is a sure loss) more often in the Loss frame than for Gain frame even in cases where larger cities were not given adequate protection. This prediction was supported by the data. Participants in the loss presentation condition covered more small cities under conditions where it was both appropriate (higher expected value by doing zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA so) and inappropriate (lower expected value by doing so). Thus subjects presented with Loss displays were risk seeking in the sense of Tversky and Kahneman. They would rather risk a possible higher loss for larger cities than accept a sure loss for a smaller city. In general, both where it was appropriate and where it was inappropriate, information in terms of losses made the subjects more sensitive to protecting against the sure losses of smaller cities. We also predicted that improved visualization would lessen the loss avoidance biases. Specifically we predicted that both Graphical and Integral formats would make the implications of expected value solutions more

• bvious thus improving decision making. As predicted,

integrating risk information as an expected value improved decision score performance for the Loss framing condition; however the Gain condition actually showed slightly degraded performance when it was presented as integrated risk information. We believe that this was zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

an

unintentional artifact of the Graphical Gain display in which the population was dificult to interpret if there were zero GBIs allocated to a city. A follow-on experiment is exploring this in more detail. For NMD, operator’s SA is crucial and operators must remain aware of the parameters of the unfolding attack, For SA accuracy, Graphical configural displays were better than the numeric displays, a finding that agreed with most of the literature on the efficacy of graphical representations for uncertainly data (Smith and Wickens, 1999). More surprisingly, Loss information was remembered more accurately than Gain information. This agreed with previous research on NMD performance, wherein the operators had better SA for “missile leakers” (based on probability of loss) than for probability of success information (Barnes et al, 2000). There was evidence of a speed-accuracy trade-off in that for the Graphical displays, response time was about a second slower in the Loss frame than in the Gain frame. All things considered, it appears that SA accuracy is the more important SA measure for this task. ACKNOWLEDGEMENTS Funding for this work was provided by the Army Research Laboratory Advanced Decision Architectures Collaborative Technology Alliance. REFERENCES

Barnes, M.J., Wickens, C.D & Smith, M. (2000). Visualizing uncertainty in an automated National Missile Defense simulation

• environment. Proceedings zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA
• f

the zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

4Ih

Annual FedLab Symposium: Advanced Displays and Interactive Displays. (pp. 107-1 zyxwvutsrqponmlkjihgfedcbaZYXWVUTSRQPONMLKJIHGFEDCBA

1 ])Adelphi,

MD: U.S. Army Research Laboratory. Gillan, D. & Hutchins, S. (2002). Display situation awareness:

A study into information integration and presentation.

Unpublished

• manuscript. Las

Cruces, NM: New Mexico State University. Johnson-Laird, P.N., Legrenzi, P.; Girotti, V., Legrenzi, M.S.,

& Cavemi, J. (1 999). NaTve probability: a mental model of

extensional

• reasoning. Psychological
• Review. 106(

1 ), 62-88. Klein G. (1 999). Sources o

f

Power: How People Make Decisions Cambridge MA: MIT Press. Mayhorn, C.B., Fisk, A.D., & Whittle, J.D. (2002). Decisions, Decisions: Analysis of age, cohort and time of testing of risky decisions options. Human Factors, 44(4), 5 15-521. structure and the relative efficacy of tables and graphs. Human Factors, 41,570-587. Schlabach J.L.,Hayes C.C, & Goldberg D.E. (1999). Fox-GA: A Genetic Algorithm for Generating and Analyzing Battlefield Courses

• f Action. Journal of Evolutionary Computing 7(1), 27-47.

Shafir, E. & Tversky, A (1995). Decision Making. In Eds. Smith, E.E. & Osherson, D. Thinking (pp. 77-100) Cambridge, MA: MIT. Smith, M. & Wickens, C.D. (1999). The effects of highlighting and event history on operator decision making in a National Missile Defense system app lication. (Tech. Rep. No. ARL-99-4) Savoy, IL: University

• f Illinois, Aviation Research Laboratory.

Tversky, A. & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 21

I , 453-458.

Wickens, C.D. & Hollands, J.G. (2000). Engineering Psychology and Human Pe$ormance. Upper Saddle River, NJ: Prentice Hall. Meyer, J., Shamo, M.K., & Gopher, D. (1999). Information PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 47th ANNUAL MEETING—2003 566

View publication stats View publication stats