Analytical Support for Rapid Initial Assessment Charles Twardy, Ed - - PowerPoint PPT Presentation

1


Analytical Support for Rapid Initial Assessment

Charles Twardy, Ed Wright, Kathryn Laskey, Tod Levitt, Kellen Leister, Andy Loerch

George Mason University C4I Center

2


Topics

• Rapid Initiative Assessment (IA) challenges
• Overview of Mason’s IA methodology
• Example

2


3


Challenge: Analysis Support for Initial IA

We focus here: rapid initial assessment.

Models can also be reused here and beyond.

3


4


• Rigorous, rapid, consistent, re-usable analytic

justification for JIEDDO initiative assessments

• Fulfills critical need as warfighter requirements

grow while budgets tighten and scrutiny increases

Solution: Analysis Support for Initial IA

Rapid Initial IA Requirements

• Provide rapid assessments (days to

weeks)

• Model dependence of relevant Measures
• f Effectiveness (MOEs) on system &

environmental variables

• Use available knowledge
• Identify information collection priorities
• Be consistent, repeatable, & extensible

5/19/10


3. Implement as Probabilistic Model 4. Exercise Model & Analyze Results 5. Determine Sensitive Parameters 6. Report Results

Initiative Assessment Process

6


1. Identify MOEs 2. Generate Explanation

Example MOEs: Casualties per Incident Time to Complete Mission Weapons Intelligence Gathered Partial Explanation Example: If there is an IED detonation during robot neutralization, Blue soldiers are not

• exposed. The robot may be damaged or

destroyed. Bayesian network (BN) model for EOD robot

7


IA Approach Benefits

• Consistent framework for

assessing initiatives.

• Clearly communicates to decision

makers, the assessed impact, potential tradeoffs, and the mechanism by which it works.

• Makes the explanation structured,

explicit, executable, and reusable.

• Perform what-if, try scenarios, test

understanding, perform sensitivity analysis.

• Enable development of more

informative test plans.

• Identify relevant MOEs.
• Generate an Explanation of how the

initiative is expected to affect MOEs.

• Implement the explanation as a

probabilistic model.

• Execute & analyze model to assess

performance

• Determine the “sensitive

parameters” (SPs) to help prioritize information collection.

ExplanationProbabilistic Model

• Generate explanation of how initiative affects MOE

– Clutter can interfere with the ability of the sensor to detect IEDs and cause false positives

• Implement explanation as Bayesian network (BN)

– Structured, explicit, executable, and reusable – Models how initiative is likely to perform in

• peration

– Supports what-if and sensitivity analysis

5/19/10


8


CPT for Sensor_Result

9


Available Knowledge

• SMEs (at JIEDDO and elsewhere)
• JUONS and other needs statements
• Initiative documentation
• Current suite of equipment & capabilities
• Additional contractor knowledge
• Blue and Red TTPs
• Previous initiatives
• Previous models
• Previous tests

9


New Class ?

New Initiative Select model From Repository No

Model Analysis

Add Performance Assessment & Sensitive Parameters OFFLINE: Enhance models Yes

Model Sufficient?

Yes Add No (n+1)th

Iteration

Model nth Iteration Model

Model Development Spiral

Recursive Spiral Prototyping

10
 Time Avail ?

No Yes

Develop 1st Iteration Model Modify Model

Model Repository

Analysis

5/19/10
 11

Try Scenarios in the Model, and examine the effect on the MOEs For each MOE, find the most influential variables:

• Intel. Potential

redDetonatesRobot redDetonation probDisableSuccess robotProbEffective robotReadiness

Calculate individual link strengths: Vary some parameters over their range:

12


Specific Technologies

RECCE I Cougar 6x6 Platform Gyrocam (VOSS) Remote Wpn Sys EOD Robot Comms Duke v1 RECCE II Adds LNS Duke (v2)

EOD Robot

In the Remote Deployment System

Remote Wpn Sys VOSS on Mast LNS

Photo from the (S) ATEC C&L Report, July 2008

Example 1 EOD Robot

Assess EOD Robot

• New Class? Yes

Select MOEs Build 1st Iteration Model

13


New Class ?

New Initiative Select model From Repository No

Model Analysis

Add Performance Assessment and Sensitive Parameters OFFLINE: Enhance models Yes

Model Sufficient?

Yes Add No (n+1)th

Iteration

Model nth Iteration Model

Recursive Spiral Prototyping

Time Avail?

No Yes

Develop 1st Iteration Model Modify Model

Model Repository

New Initiative

New Class ?

Yes

Develop 1st Iteration Model

MOEs by Tenet *

14


Tenet Potential MOEs Predict (Intell. Gathered)… Mission Time (Cost) Prevent Number and relative proportion of each type of IED tactic, Number or percentage of interceptions, raids, captures before emplacement, #IEDs/ mission mile Detect-Air P(detect), False Alarm Rate, Sweep Width, Rate of Advance Detect- Ground P(detect), False Alarm Rate, Sweep Width, Rate of Advance, P(spot) Neutralize P(neutralize), Neutralize Time, Intelligence Gathered Mitigate Casualties/Attack, KIA/Attack, WIA/Attack, Damage/Attack

Some MOEs suggested by Perry et al., Minimizing the Threat from Improvised Explosive Devices in Iraq, RAND 2007 Some from the RECCE II Initiative Evaluation Plan (AMSAA, August 2008)

*Tenet: JIEDDO divided initiatives into “tenets” which roughly follow the “left of boom” timeline.

15


Identification of MOEs

Assumptions The EOD robot provides a capability to remotely neutralize (disable or detonate) an IED. If the robot is not available or not successful, a soldier will neutralize the IED.

MOE Assumptions and Considerations Time Robot may take longer than an EOD soldier If the robot is unsuccessful, we still must use a soldier P(neutralize by robot) Distinguish disable from destroy Casualties or Damage per Attack

• Replace with generalized, qualitative P(damage)
• If Red detonates the IED during robot neutralization, soldiers are not
• exposed. The robot may be damaged or lost.
• If the robot is unavailable, or fails, then a soldier will be at risk.
• If the IED is not spotted, robot has no effect on damage / casualties.

P(collecting valuable Intelligence)

• If Blue disables the IED, it can be examined for forensic intelligence.
• If Blue detonates it, there may be some intelligence collected before

the detonation.

• If Red detonates it, there is little intelligence gained.

16


Implement Explanation as BN

Model assumes IED is present and successfully detected.

• If robot is available and working

correctly, it can be used to attempt to disable or detonate an IED.

• If the robot succeeds in disabling the

IED, we can gather forensic intel.

• Little intelligence can be collected if

the robot detonates the IED.

• If there is a Red detonation during

neutralization, Blue soldiers are not

• exposed. The robot may be damaged
• r destroyed.
• If the robot is not available or not

successful, a soldier will be at risk while disabling the IED.

• Using the robot may take longer than

using an EOD soldier.

• If unsuccessful, a soldier must still

disable the IED. 1 2 2 3 3 1

Example 1 EOD Robot (2)

Assess EOD Robot

• New Class? Yes

Build 1st Iteration Model Add 1st Iteration Model to Repository Time Available? No Run the Model, Analysis

17


New Class ?

New Initiative Select model From Repository No

Model Analysis

Add Performance Assessment and Sensitive Parameters OFFLINE: Enhance models Yes

Model Sufficient?

Yes Add No (n+1)th

Iteration

Model nth Iteration Model

Recursive Spiral Prototyping

Time Avail?

No Yes

Develop 1st Iteration Model Modify Model

Model Repository

New Initiative

New Class ?

Yes

Develop 1st Iteration Model

Select model From Repository Add nth Iteration Model

Model Repository

Model Sufficient?

No

Time Avail?

No

Model Analysis

Performance Assessment and Sensitive Parameters

Robot Analysis 1: View Effects

If the robot is not available … a soldier will be at risk while disabling the IED. If a robot is available and it is working correctly, it can be used to attempt to remotely disable or detonate an IED. Lower risk to soldier, more time If the robot succeeds in disabling the IED, it can be examined for forensic intelligence. Less intelligence can be collected if the robot detonates the IED.

18


19


Robot Analysis 2: Sensitive Parameters by MOE

EOD Robot: Top 5 Sensitive Parameters by MOE.

• Assuming robotAvailable, and excluding deterministic functions

ClearTime Intelligence Damage redDetonatesRobot redDetonatesRobot redDetonation redDetonation redDetonation redDetonatesRobot robotReadiness probDisableSuccess robotProbEffective robotProbEffective robotProbEffective robotReadiness probDisableSuccess robotReadiness

• Next Steps (as time allows):
• Investigate sensitive parameters in more detail
• Extend / refine the model: additional variables, situations; extend or refine the

state space of important variables; refine local probability distributions.

• Identify knowledge requirements for the sensitive parameters
• Seek additional information: SMEs, system documents, data collection, …

20


Summary

Challenge or Need IA Methodology Solutions Short timeline Spiral development: start simple, extend later Reuse models; Standard flow starting from MOEs Little quantiative data / need to assess prior to testing Probabilistic models can use available expert and prior knowledge as soft constraints; initiative models make use

• f any existing models onto which they are added

Need analytical support Probabilistic models are explicit representations of how the initiative is thought to work, and can be executed. Prioritize information collection Sensitivity analysis in the model can rank variables by influence, and show the effect of parameter changes Consistent, repeatable, extensible Standardized methodology based on MOEs leads to consistent assessment across initiatives. Model reuse provides repeatability and extensibility Integrate with Portfolio Management MOE statistics from the model feed into PM approach: casualties/damage, time, effectiveness

21


Questions?