State-of-the Art of Immersive Training Technology Marco BIAGINI 1 , - - PDF document

IT2EC 2020 IT2EC Extended Abstract Template Presentation/Panel

State-of-the Art of Immersive Training Technology

Marco BIAGINI1, Stefano CAMBI2

1Lt.Col. ITA A, NATO M&S CoE, Rome, Italy 2Lt.Col. ITA AF, NATO M&S CoE, Rome, Italy

Abstract — “The level of complexity in the coalition environment demands a rehearsal framework in pre-deployment training” (US CENTCOM CCJ6-P). The still valid idea introduced by the former Afghan Mission Network concept for a Training, Staging and Mission networks model, is then exploited by the authors proposing an approach to support the process of training and preparing transitioning forces during the pre-deployment phase adopting Immersive Training

• Technology. This approach, implemented within a Modelling and Simulation as a Service (MSaaS) training framework,

is designed to suit a Future Mission Network (FMN) training environment. The paper illustrates the state-of-the-art of immersive devices coupled with virtual immersive simulators, serious games and virtual worlds making them accessible through a MSaaS platform (OCEAN) to support individual and collective training for Commanders, staff and dismounted soldiers. The framework is suitable to be used to conduct remote distributed training and rapid prototyping activities. Furthermore, it is suitable to be implemented in a Joint Enterprise Information environment like the NATO Training and Education Network (NTEN) fitting the NATO Future Mission Network initiative, and more

• ver the foundation of a future NATO Synthetic Environment”.

1 Introduction

“Train as we partner” is one of the pillars defining pre- deployment training requirements for NATO [1]. It requires common access to NATO training resources and the achievement of interoperability between personnel from different countries and organizations. This access includes sharing resources, experiences and information. These education and training processes, along with the use

• f exercises and new technologies, are the foundations of

the NATO Connected Force Initiative [2]. Due to wide- spread financial constraints, budget cuts are affecting training and force preparedness and Virtual immersive simulation is one of the possible solutions to maintain force readiness reducing as much as possible live training

• exercises. Virtual simulation is now used to replace a

significant portion of pilots’ live training sessions. For soldiers, high fidelity and high resolution simulators coupled with high definition (HD) immersive interfaces may soon enable most live training objectives to be achieved through virtual simulation. Nowadays, virtual augmented and mixed reality simulation and built-in serious games applications are enabling development and deployment of innovative training solutions at both the individual and collective level, like in the US Army Synthetic Training Environment (STE) program [3]. In fact there is an emerging opportunity to utilize innovative immersive Human Computer Interfaces (HCI) and the M&S as a Service (M&SaaS) technology and business

• model. Leveraging these technologies to provide M&S

services will help maintain NATO’s forces readiness and combat effectiveness in a more efficient and cost-effective

• way. The key factor is a cloud-based networked

environment enabling expanded training and increasing exercise opportunities.

2 Cloud-Based Simulation Training Challenges

Main challenges and key factors to provide individual and collective training solutions for ground forces using virtual simulation and immersive devices in a cloud and web- based networked environment are:

• Availability of a new generation of affordable

high fidelity immersive devices;

• Mature cloud and web-based virtual simulators

within a M&SaaS business model, enabling at the same time home station training capabilities;

• Ubiquitous

Internet, intranet and private networks availability to provide fast, secure and reliable web access within a proofed training, staging and mission network model. To enable an individual and collective pre-deployment training capability “Train as we partner”, supported by new generation M&S technology, requires the definition

• f suitable training objectives:
• Command Post (CP) individual and collective

personnel training, regarding: CP layout and systems familiarization; collective training for Commander and staff focusing on cross functional/operational planning teams collaboration and team building; CP mission rehearsal and personnel pre-handover/rotation tasks and activities.

• Dismounted soldier and pilot individual and

collective training, regarding Tactical Techniques Tactics and Procedures (TTPs). The framework could be also used for CP rapid prototyping activities and for CD&E activities regarding the study, development, test and experimentation of CP immersive decision support tools to support operations.

IT2EC 2020 IT2EC Extended Abstract Template Presentation/Panel

3 Immersive Device

The only way to immerse trainees in a virtual world/simulation is to try to involve as much senses as possible using haptics interfaces reducing the use of non natural HCI like keyboard, joysticks and so on. Therefore, during training sessions the level of involvement of our senses should be improved in order to boost the learning/experience process. Low cost effective hardware must become the distributed tool in order to boost the experience at a collective level. Latest headset devices and

• ther HCI with haptics technology, virtual gloves, spatial

sensors and so on, are moving fast forward and companies are developing innovative and cheap solutions targeting the consumers market (Oculus Rift and Quest, Valve Index, Leap Motion, Microsoft Azure Kinetic, Hololens, Razer Hydra, Reactive Grip and Treadmills and so on). Quality and innovation of these items make them possible to be utilized broadly for professional training purposes. The immersive HMD, should deliver the capability to work in the virtual world, using real equipment from the real world (interacting under the HMD), based on the “Fly what you see concept”. Spatially, the trainee will refer to what he sees within the virtual environment, always maintaining the awareness of real positioning in space and equipment interaction (reading a map for instance). In

• rder to complete the experience, real complex systems

(weapons, binoculars and so on) should be used with virtual manipulation gloves, maintaining their real weight and ergonomics, saving costs for expensive mock up. The zero need for a prepared infrastructure, delivers the immersive package an unlimited usability potential, being able to be used when, where, and how it is needed.

4 Virtual Worlds, Serious Games and the social dimension

“Second Life”, was one of the most widely used Virtual World Platform, it was used in the past for research activities also by NATO and the US DoD exploiting new technology for training and collaboration purposes. During last years, important initiatives in using the VW platform different from “Second Life”, were using platform developed by Unity 3D like the NATO ACT Virtual Operation Headquarters Platform Prototype. Currently, however, this still good engine has been joined by another

• ne that, for its versatility, as well as, high fidelity visuals,

straight from the box, it is increasingly adopted for serious games and simulation projects for professional training in several sectors. Infact, although, Unity is one of the best platform for developing mobile and 2D games, Unreal engine, in its 4th generation, is best suited for developing highly graphical and photorealistic games, also embedding the social dimension, i.g. “Fortnite”. Nowadays, a new social virtual reality platform, “Sansar”, has been developed by Linden Lab, with a dedicated engine, also for VR device. Regarding the social dimension, one of the most important chance provided by a web based training simulation environment is the opportunity to integrate M&S training and education resources within a social network environment enabling real home base training

• capability. The opportunity to create communities of

practice within the e-NATO training and Education initiative supported by the NTEN (NATO Training and Education Network) and the other initiatives will push the involvement of personnel from allied nations and others NATO partnering countries. It will create the opportunities to plan and conduct training sessions, enabling a real “train as we partner” opportunity, pooling personnel being harmonized under common training resources and requirements over a NATO Synthetic Environment.

5 Artificial Intelligence in Computer Generated Forces (CGF)

To make the simulations more realistic, often the trainee has to interact with virtual agents or, so called, Computer Generated Forces (CGFs) [5]. Most Commercial or Military standard software for simulation provide at least a rudimentary AI interface, through which the scenario developer has to define the behaviour for their CGFs. Currently, a few companies provide custom AI solutions that can be integrated with existing CGF packages and the most, AI introduction consists of adding some degree of decision autonomy to simulated entities. None of the mainstream CGF packages that are currently used

• perationally introduces Machine Learning (ML) and

Deep Learning capabilities. These techniques may provide solutions, through the automatic generation of behaviour

• models. Nowadays, the abundant possibilities of AI, ML

and its subset “Deep Learning”, are not exploited within military simulation and serious games and soon it will become possible to create truly intelligent (autonomous and adaptive) CGFs also with the packages available on the market. Therefore applying ML techniques, instead using “simple” AI models, can be beneficial for modelling CGF behaviours, and guidance needs to be developed for end users. It would be very useful, much more realistic and more effective training, if, the developers of CGF packages for the Commercial or Military standard software for simulation, start incorporating ML

• techniques. In effect, providing the capability to create

CGFs that possess richer behaviours in complex environments, that perform tasks through experience, they will be better tailored to the knowledge and skills of the

• trainee. These combinations could also open new training

challenges for the future implementing AI engines able to evolve TTP’s according to the trainees level of expertise and skills.

6 The Immersive Training Environment

Use of virtual technology to implement a Virtual Command Centre for training, collaboration and operation support opportunities is hardly a new concept. A lot of literature has been written about it since 1995 [4]. Despite several tentative, up to now, this concept has not been seriously adopted due to some factors and issues like security, bandwidth limits, hardware limits, ergonomics issues and others. Above all, the biggest obstacle remains the cultural resilience in using these devices that now are going to be extremely user friendly and widely adopted by

IT2EC 2020 IT2EC Extended Abstract Template Presentation/Panel the gaming industry for the consumer market. Technology is doing steps forward and nowadays there is for sure the

• pportunity to revitalize this concept introducing

innovative technology and devices supporting remotely distributed training capabilities powered by social network technology. Looking at multinational, Joint and combined environments like large Head Quarters (HQ) down to Tactical Command Posts (CP) and Joint Operational Centres, rotation can happen on the entire personnel basis

• r, periodically, involving single or multiple personnel.

When arriving for the first time to a large HQ, personnel is usually a little bit disoriented and it takes time to get used to tasks, routines people and places. Sometimes hand over time last few days and personnel rotate without a complete picture about what their predecessors used to do. In few cases, this could create also an initial disruption in his job activities due also to his level of experience, training and initial lack of knowledge about the environment and the personnel he has to work or interact with. To overcome this issue, the adoption of a web-based immersive training system, combined with a Cloud Service like a MSaaS, will let the users to connect from different locations and interacting with the virtual environment by immersive wireless devices system without caring about the performance of their computers and without installing dedicated software in a really easy way, allowing him to carry out activities such as training, staging, mission support, team building, mission rehearsal and personnel pre-handover/rotation tasks and activities. At the end, this Immersive Training Tool will offer greater training schedule flexibility, lower costs and less travel time with less impact on the environment. Therefore, the web-based environment will overcome the aforementioned common barriers and challenges to create a new approach to remote collaboration and network management backed by a platform framework that has embedded training and assessment capabilities. The main novel features of the platform are:

• No software downloads needed to access core

web-based collaboration, crowdsourcing and training services;

• Integration with immersive wireless HMD,

haptics and other innovative sensors;

• M&S simulation services that are cloud and web-

based using a 3D web plug-in;

• Intuitive Data representation and visualization;
• Embedded interaction with external remote

desktop and virtualization technology. Furthermore to provide real or emulated C2/C4ISTAR systems access within the virtual world environment.

7 The Dismounted Soldier in Mixed Reality Environments

One of the most common Dismounted Soldier Training System limitation affect the soldier locomotion and his manoeuvre capability. Using hand controls to move, is worse than unrealistic: it’s potentially dangerous. There’s such a thing as muscle memory, and a soldier who has repeatedly practiced running for cover by tweaking a mini- joystick may be just a little too slow to use his legs when someone shoots at him for real. Even minor differences between the training environment and real combat can cause big problems. Instead, basing on the Immersive Device technology, using device like HMD, Hololens, Sensor, Haptic Gloves, Reactive Grip and eventually a new generation of Treadmills, according to the activity that the trainer must carry out, the mixed reality dismounted soldier package will fully leverage all the benefits of this systems, enabling mixed reality training in a Live Virtual Constructive cloud-based immersive simulation environment.

8 Conclusions

Virtual, Augmented and Mixed immersive simulations within a Synthetic Training environment will offer possible solutions to maintain force readiness reducing as much as possible live-fire training exercises, especially in a contest were increasingly financial constraints have to be taken in consideration. The use of innovative immersive Human Computer Interfaces (HCI) combined with a M&SaaS technology and sustainable business models will allow the develop of several training and rapid prototyping activities in a cloud web-based networked environment at both individual and collective levels. Nowadays the US Army STE program seems to be one of the mean, while NATO ACT with the ITT Research and Development activities is focusing on immersive.

References

[1] Kopfer M. (2011) “Mission Network as the New Joint, Coalition Norm”, presentation proceeding of LandWarNet 2011. http://www.afcea.org/events/pastevents/documents/LWN 11_Track_1_Session_4.pdf [2] Connected Forces Initiative https://www.nato.int/cps/en/natohq/topics_98527.htm [3] The Synthetic Environment Core (SE Core) program. https://www.peostri.army.mil/synthetic-environment- core-se-core- [4] Morgenthaler M., Steiner G., Mayk I. (1995). “The Virtual Command Post”. http://rubyquest.com/glenn/resume/publications/virtual_c

• mmand_post.htm

[5] A. Toubman, J.J.M. Roessingh, G. Poppinga, M. Hou,

• L. Luotsinen, R.A. Løvlid, C. Meyer, R.J. Rijken and
• M. Turcaník. “Modeling CGF Behavior with Machine

Learning Techniques. Requirements and Future Directions”

Author/Speaker Biographies

Lt.Col. Marco BIAGINI, is the Concept Development and Experimentation (CD&E) Branch Chief at NATO Modelling & Simulation

IT2EC 2020 IT2EC Extended Abstract Template Presentation/Panel Centre of Excellence. He has a Ph.D. in Mathematics, Engineering, and Simulation and master degrees in strategic studies, peace keeping and security studies and, new media and communication. He has more than 15 years

• f experience in the M&S field. He was Battalion

Commander at the Italian Army Unit for Digitization Experimentation (USD) and Section Chief at the Italian Army Simulation and Validation Centre. Lt.Col. Stefano CAMBI, join the Air Force in 1985 and he is serving in NATO Modelling and Simulation Centre of Excellence (M&S COE) as Chief of the Experimentation Section. Former Military Air Traffic Controller, he attended Joint Services Staff Course and served as Commander of the Italian Air Force Support Unit to the NATO E-3A Component di Geilenkirchen (GER).