A Quantitative Assessment of Flight Training Effectiveness in Mixed - - PDF document

IT2EC 2020 A Quantitative Assessment of Flight Training Effectiveness in Mixed Reality Presentation/Panel

A Quantitative Assessment of Flight Training Effectiveness in Mixed Reality

Peter Bellows1, Amy Dideriksen1, Joe Williams1, Tom Schnell2, Katharine Woodruff2, Colton Thompson2, Mathew Cover2

1Author Title, Collins Aerospace, Cedar Rapids, IA, USA 2 Author Title, University of Iowa Operator Performance Laboratory, Iowa City, IA, USA

Abstract — This paper describes a methodology for quantifying the training value of extended reality (XR)-based platforms versus conventional trainers using cognitive workload and objective performance data. The methodology was validated using low-time evaluation pilots without any prior knowledge of tactical aviation. The study involved simulator training of air-surface (A/S) bomb delivery, and culminated in a live capstone flight for each pilot. The study showed a statistically significantly higher situational awareness (SA) in pilots trained with the XR platform. Additionally, we observed excellent agreement between objective and subjective workload assessment which lays the foundation for workload-adaptive XR based training strategies.

1 Background

Military training designers face a continuous influx of astonishing new technology. While this offers exciting new opportunities for training realism, constant budget restrictions, combined with a lack of hard data on comparative training quality, make it difficult to make informed decisions on which technologies to use. Quantitative tools are needed to measure training value to maximize return on training investment. One area of significant growth is Virtual, Mixed and Augmented Reality (VR / MR / AR - known generally as “XR”) displays. With their expansive 360° 3-D visual environments, these devices offer significantly higher realism for creating the “Digital Twin”. But the increased realism typically comes with an increased cost of modelling and design, particularly for MR, where real- world objects (gear, cockpit instruments, etc.) must be seamlessly blended with the virtual environment. With the wide range of devices available, it is a significant challenge to choose the XR display that best fits the application; training effectiveness is impacted by complex factors (fidelity, resolution, field of view, ergonomics) that may be difficult to predict based solely on technical specifications. Collins Aerospace has partnered with the University of Iowa Operator Performance Laboratory (OPL) to apply quantitative training effectiveness assessment methods to identify the benefits (and limitations) of XR technologies for flight training. The study was a controlled experiment involving 12 test pilots, all being trained to perform an identical mission to drop a simulated bomb from a fighter trainer jet (e.g. T-45) onto a designated target, with specific instructions on delivery parameters in accordance with the Navy T-45 Strike curriculum.

2 Test Method

Our study involved 12 low-time (250-750 hours of total time) private pilots with instrument ratings. The pilots were selected to be similar in experience to military pilots in the early stages of their training. The study proceeded in two steps. For the first step, the pilots were trained via simulation, randomly assigned to two training platforms. Half were trained in a traditional procedure trainer (PT) simulator with a single outside visual display. The other half were trained using Collins’ CoalescenceTM Mixed Reality display system while sitting in the actual fighter- trainer jet, on the ground, with instrumentation to connect the actual flight instruments to the simulator infrastructure (Figure 1). With its immersive 360° out-the-window visuals, and hands-on access to real flight instruments, the MR system can clearly offer better realism. The simulation software (Collins CORESIMTM) and training tasks were identical for both groups, so that only the XR- related user interface elements were contrasted. Our goal was to quantitatively determine whether this greater realism actually translated to behavioural and performance improvements.

• Fig. 1. Traditional vs. aircraft-in-the-loop XR simulation

For the second step, the pilots flew a live capstone mission, performing the same bombing exercise they had learned in simulation. The live flight platform was an Aero Vodochody L-29 Delfin jet trainer with a safety pilot, and with Live, Virtual & Constructive instrumentation to allow for synthetic bombs and terrain (Figure 2) using an F-35 representative HMD (Collins StrikeEyeTM) with a simulated Distributed Aperture System (DAS). The evaluation pilots (EPs) flew their capstone tasks with an

• paque hood in the rear cockpit canopy while the rated

safety pilot (SP) monitored the flight from the front cockpit.

Formatted: Not Superscript/ Subscript Formatted: English (United States) Formatted: Superscript

IT2EC 2020 IT2EC Extended Abstract Template Presentation/Panel

• Fig. 2. L-29 test flight platform, with roll-in bombing maneuver

For both simulation training and live capstone activities, we collected performance data through flight technical measures, and pilot cognitive workload through electrocardiogram (ECG) waveforms. ECG data was measured in real-time via the Cognitive Assessment Tool Set (CATS). Using the combination of task performance and cognitive workload metrics, we assessed the efficacy

• f both training approaches in the transfer of requisite

skills [1, 2]. These results were then compared against each learning approach, to assess the most effective solution. Objective performance criteria were developed based directly on Navy pilot training publications (CNATRA P- 1205 Strike T-45)[3].

3 Test Results

The essential test results shown in Figure 3 indicates that the Coalescence training platform resulted in much higher situation awareness (SA) when compared to the procedure trainer (PT). This SA benefit of Coalescence was statistically highly significant. Figure 4 also indicates that SA increased strongly and statistically highly significantly as the EPs who used the Coalescence platform progressed through the training syllabus from 15 degree delivery patterns to 30 degree patterns and finally to the pop-up delivery pattern. The EPs who used the PT training platform did not experience such a steady increase in SA throughout the curriculum. In fact, a decrease in SA was noticed in the PT stratum for the final pop-up maneuver. Figures 4 and 5 show the objective workload results for the simulation and live flight training sessions. The EPs in the PT platform experienced statistically significantly higher levels of cognitive workload when compared to their Coalescence counterparts. This workload difference is the result of a behavioral adaptation that EPs had to apply for flying the prescribed bombing patterns. Since the PT platform did not offer any off-boresight outside visuals, EPs had to adopt a “nose-to-panel” instrument flight procedure to determine appropriate aircraft location relative to the target location. This instrument flight adaptation required more headwork, which is indicated in the additional objective cognitive workload. Another significant observation is that while cognitive workload stayed about the same for Coalescence pilots between simulation and live flight, for PT-trained pilots the cognitive workload increase significantly in live flight. The assessment methodology was further validated in flight where we found that pilots who were trained with the Coalescence system showed statistically significantly lower levels of cognitive workload than their PT trained EP colleagues.

• Fig. 3. Situation Awareness of Coalescence vs. PT
• Fig. 4. Objective workload in Simulation

Mean of MeanFLTWL Task Method Pop 20deg 15deg PT Coal PT Coal PT Coal 16 14 12 10 8 6 4 2

C oal PT Method

• Fig. 5. Objective workload in Live Flight

4 Conclusions

Our experiment produced several important results. First, by following the training process all the way from simulation to live flight, and demonstrating correlated improvements in cognitive workload, task performance and subjective measures, our research suggests that XR- based training can indeed provide a more effective training experience, given the right application. The fact that cognitive workload for Coalescence-trained EPs was lower in the live capstone flight than that of the PT trained EPs indicates the likelihood that there was better training transfer, such that the pilots felt better prepared and more confident in applying their training to live scenarios. In the training evolution, we found that the Coalescence platform provided its EPs with far higher situation awareness than the PT platform. Pilots who trained in the PT platform could not perform appropriate off-boresight visual cross checks when abeam of the target. This caused artificially higher levels of workload with a resultant reduction in training effectiveness that transferred into the capstone flight as indicated by higher cognitive workload levels in the PT trained pilots during live flight. Additionally, the fact that cognitive workload stayed constant for Coalescence pilots when transitioning to live flight suggests that the immersive XR experience was more successful in training transfer and preparation for the

• perational environment; PT-trained pilots experienced a

significant increase in cognitive workload when transitioning to live flight, by contrast. believe that this approach is applicable to evaluating a broad range of XR technologies and potential training

• applications. The consistent correlation between objective

workload, task performance and subjective measures, for both simulation and live flight, suggests that these

Mean of SART Task Method Pop 30deg 15deg PT Coal PT Coal PT Coal 7 6 5 4 3 2 1

C oal PT Method

IT2EC 2020 IT2EC Extended Abstract Template Presentation/Panel measures may be used predictively; it should not be necessary to take every trade study all the way through live flight exercises to validate the training. There is a great deal of potential future work in this

• domain. While this is a useful data point in evaluating XR-

based training in general, it proves no more and no less than that a specific platform was more effective for a specific application. Additional trade comparative studies will be needed before more general conclusions can be drawn about the value of XR training investments. It may be useful to perform individual studies focused on a single XR function, such as resolution, field of view, hand tracking or other aspects in order to create general training

• guidelines. More general studies such as this would also

be useful to further refine methodologies for cost effective instructional systems design studies.

Acknowledgements

The authors are grateful for the efforts of Ezekiel Gunnink, Joe Williams and Jesse Lane for developing the objective performance scoring methodologies used during this experiment.

References

[1] J. Hoke, C. Reuter, T. Romeas, M. Montariol, T. Schnell and J. Faubert. I/ITSEC 2017. [2] A. Dideriksen, C. Reuter, T. Patry, T. Schnell, J. Hoke and J. Faubert. I/ITSEC 2018. [3] CNATRA P-1205 Strike T-45 Multi-Service Pilot Training System (MPTS). Naval Air Training Command, 2017.

IT2EC 2020 IT2EC Extended Abstract Template Presentation/Panel

Author/Speaker Biographies

Peter Bellows is an Associate Director for Mission Systems Advanced Technology at Collins Aerospace. He is actively involved in research & development of a variety

• f technologies for simulation & training systems: mixed

reality displays; simulation-as-a-service through cloud computing; distributed LVC instrumentation; and training effectiveness assessment. Amy Dideriksen, PMP is a Global Training Research Manager in Mission Systems at Collins Aerospace with

• ver thirty years of training experience and a background

in Instructional Systems Design. She is the principal investigator in Advanced Technologies for research initiatives in Cognitive State Assessment, Training Effectiveness and Adaptive Learning. Joe Williams – need bio

• Dr. Tom “Mach” Schnell is a Professor in Industrial and

Mechanical Engineering with a specialization in Human Factors/Ergonomics at the University of Iowa. He is also the director and chief test pilot of the Operator Performance Laboratory (OPL). He is a Commercial pilot with over 6,400 flight hours, research test pilot, and flight instructor with helicopter, jet, and glider ratings. Colton Thompson is a Flight Test Engineer at the Operator Performance Laboratory (OPL). He received his BS in Aerospace Engineering from Iowa State University. He is currently working toward his Master of Science in the Industrial and Systems Engineering program at the University of Iowa while working full time at OPL. Katharine Woodruff is a master’s student at the University of Iowa in the Industrial and Systems Engineering program. Katharine is now a graduate research assistant at the Operator Performance Laboratory (OPL) where she supports research in pilot training effectiveness, physiological episode detection, and spatial disorientation prevention. Matt Cover – need bio