Heterogeneous Classification System for Underwater Acoustic - - PDF document

UDT 2020 UDT Extended Abstract Template Presentation/Panel

Heterogeneous Classification System for Underwater Acoustic Recognition

• F. CHAILLAN1, S. MEUNIER2

1Research Director, NAVAL GROUP, Ollioules, FRANCE 2Marketing Director, NAVAL GROUP, Paris, FRANCE

Abstract — Automatic underwater acoustic recognition is a capability used on board to provide operator assistance. The principle is to associate a class to each sound recorded by the passive sonar. IA algorithms based on supervised learning are answers to this problem. Despite of their efficiency, these techniques are limited by the lack of contextual information that would make the automatic decision process more robust. As a first answer, this study attempts to introduce the concept of heterogeneous classification, as an extension of the classical automatic classification task. A first simple experimentation on the case of “unclassification task” is presented and discussed.

1 Introduction

In the underwater world, submarines use the data provided by their passive sonars to automatically recognize underwater noises [1]. This classification task provides the operator with an automatic identification proposed to help him to make its own decision. The last decades have gradually shown that AI-based classifiers are replacing classical rule-based classifiers, and have become the most performant and usual approach to designing automatic classification systems [2]. However, designing an AI-based classification tool (AI) that is powerful enough to provide a robust support to the operator is a difficult task. The main issue is linked with the passive sonar context,: as the propagation channel is highly dependent on time and space, the noise emitted by the same physical object can be received by the sensor in a slightly different way, depending on where and when the recording is made. In practice, even well-known noises can become difficult to recognize in the case of unfavourable propagation. Consequently, any AI to be designed must be both performant and robust enough to face this limitation. In the case of a great amount of available data, performance can be enhanced by trying different AI techniques from Machine Learning (ML) [3] or Deep Learning (DL) [4], with fine tuning for each algorithm assessed. Performance is also enhanced by refining data selection, and by optimizing output classes with split & merge

• perations.

Robustness also needs to be improved, but how can we enhance classification robustness when the considered equipment, in this case passive sonar, has already given all the information it can provide? Our insight is to couple a classification provided by the passive sonar with other data available on-board and at shore, yielding to the concept of heterogeneous classification. As a first answer we propose in this work an

• riginal architecture, spatially dedicated to submarines,

which allows to take into account heterogeneous information and to merge it with the original classification in order to make the decision robust and statistically exhaustive. Finally, we try to test our architecture for the heterogeneous classification of underwater acoustic noises by an experiment on the mimicry of the “un- classification” task performed by the operator when he tries to first eliminate safe noises before to focusing on potential menaces.

2 Automatic Recognition of Underwater Acoustic Noises from Passive Sonar Data

To introduce the concept of heterogeneous classification for underwater acoustic recognition, we first briefly describe the classical principle of automatic classification and its limitation in terms of performance induced by the lack of fresh information available. As a consequence, we then try to describe the inventory of all information that can be found and used

• n board, potentially useful to enhance the strength of the

classification task. This allows us to provide a methodology for designing a heterogeneous classification system capable of recognizing underwater acoustic noises by coupling different sources of available data to eventually make its own decision. Thus, the final decision is ideally an exhaustive synthesis of all available

• information. It is finally brought to the operator to

provide him with assistance. 2.1 Automatic Recognition

• f

Underwater Acoustic Noises: the Classical Approach The problem

• f

automatically recognizing underwater acoustic noises from passive sonar data is a difficult one, because the sounds to be recognized may

UDT 2020 UDT Extended Abstract Template Presentation/Panel have similarities whereas they do not belong to the same

• class. These physical similarities are such that it is

possible to confuse the biological or mechanical origin of the noise sources to be recognized. Furthermore, as the passive sonar device does not provide the range, the same noise source caught at different ranges can have different signature; moreover, the propagation channel depends on space and time, so that the same noise source can have a slightly different signature depending on the date and area of the recording. Finally, we recall the sampling frequency of passive sonar is so that all frequencies of the noise source lying above the half sampling frequency are not observed by the sensor. Consequently, this is particularly damaging when trying to recognize biological activity, as some biological noises may have signatures with high frequencies components, at least significantly higher than the half sampling frequency. In this context, the problem is to associate a class with a sound emitted by the acoustic landscape. Each class provides knowledge about the physical origin of the underwater acoustic sources. Consequently, this class serves as a decision-making support for operators. IA algorithms can provide a solution to this complex problem through supervised learning. The purpose is to build the unknown function that perfectly associates each acoustic noise to its class. This function is approximated by the considered learning algorithm, from a set of sound data labelled by a human

• expert. Fig.1 describes the steps to be taken to design

such a system. The first step consists in building a database, containing selected sounds in order to be sufficiently representative of the classes of sounds to be recognized.

• Fig. 1. Principle of automatic recognition of underwater

acoustic noises. The classical approach

Each selected sound, called an example, must be labelled by a human expert. This is a time-consuming task that requires ergonomic annotation tools and the full attention

• f the operator to avoid labelling errors that lead to

reduced performance. Then, the learning step is conducted according to the IA algorithm family

• considered. The two main families are Machine Learning

and Deep Learning. In our context, Machine Learning techniques such as Support Vector Machines [5] have the advantage of being explainable, in the sense that it is possible to rigorously justify why the IA provided an output with a known input. However, these techniques require first accurately extracting the signal of interest among the sea noises, then computing and selecting the features [6]. Features are scalars grouped in a vector which represents the “DNA” of the signal. The IA takes the features as inputs to provide the class associated to the signal. Conversely, the Deep Learning approach is for the IA to take the sound samples directly for input. In practice, the architecture of the network is divided in two main blocks of neural layers: the self-encoding block and the classification block, so that the features are directly computed by the network. Nevertheless, Deep Learning is not explainable. Then, when learning is OK, fine tuning techniques such as cross-validation are applied to the network in

• rder to provide optimal values for the hyper-parameters.

The next step is to assess the performance of the IA. The performance is calculated based on the confusion matrix, the ROC curve and the AUC. Finally, the trained and tuned IA is deployed so that a class can be associated with a sound according to a given performance level. This proposed methodology, synthetized by Fig.1 has been used in [7] to design from scratch the proposed IA algorithm. The proposed IA algorithm for underwater acoustic recognition is used in our study as a reference. As an example, Fig.2 shows an output of this classifier.

• Fig. 2. Output of the IA algorithm for underwater acoustic

noises recognition based on DL approach

The results show that despite high performance, we have no explanation for the output, especially in terms of confidence interval. In particular, we want to make the difference between a cautious decision and a coercive

• ne. Furthermore, the interpretation of the output of the

algorithm must be carefully considered. Indeed, as the

• utput is here a probability density function, the most

likely way of doing is to attribute the class to the mode of the distribution. But it is possible to go further, by considering the pattern of the density instead of the mode. Because a in real situation on board, the operator not only listens to acoustic noises but also reads documents, consults databases, takes into account the location and environmental conditions, we propose to do the same for our IA: in order to make the decision more robust we associate it with all available and useful information.

UDT 2020 UDT Extended Abstract Template Presentation/Panel 2.2 Overview of Available Data Sources on Board Before to introduce the concept of heterogeneous classification for underwater acoustic noises recognition, we try in the following section to list the different types

• f data potentially useful to enhance the classical

classification task that we can find in a submarine. 2.2.1 Watch During operations, the passive sonar surveillance task is performed from a waterfall representation. An image shows the energy distribution of the acoustic landscape illuminated by the sensor along the time and azimuth

• axes. Consequently, the acoustic noises form a signature

with a characteristic pattern. Therefore, the operator has the opportunity to focus on specific patterns, trying to recognize them directly. 2.2.2 Tracks Traditionally, passive sonar surveillance systems provide tracking. Two families of tracks can be distinguished: stationary tracks and transient tracks. Stationary tracks have by definition a permanent signature, whereas transient tracks have short length in relation to the observation interval in time and space. The kinematic analysis of interesting tracks is interesting because it is characteristic of the acoustic source. For instance, the biological activity is such that the trajectory steered by animals presents bends expressed in degrees by seconds that are greater than those produced by ships. Consequently, this behavior is typical of biological activity, so it constitutes useful information for enhancing classification. 2.2.3 Audio When the operator focuses its attention on a specific pattern on the surveillance display, the next step consists

• f listening to the acoustic source under consideration.

The operator then uses signal processing techniques to listen and observe the frequency components of the acoustic noise being studied. 2.2.4 Navigation On-board navigational information is given in real time by the latitude and the longitude of the ship itself. When combined with maps, it may be useful to make assumptions about the types of potential acoustic noises to be recognized. 2.2.5 Ships and bio-acoustic database When the operator has an insight into the identification of a recognized acoustic noise, he can check a classification database containing ship

• information. For instance, he cross-references the
• bserved characteristic frequencies of the acoustic noises

with those of the expected ships, trying to match them to refine its decision. It is also possible for him to consult a bio-acoustic database in order to match the observed acoustic noise with well-known biological signatures. 2.2.6 Notebooks The operator can also consult different kinds of useful documents such as books describing characteristics

• f the ships [8], or maps describing biological activity

around the seas. Thus, coupling the observed acoustic noise with one of these maps, which acts as a spatial prerequisite, should help to reduce uncertainty. 2.2.7 Environment It has previously been shown that the recognition of underwater acoustic noises is difficult because it is generally unsteady. Particularly, behind passive sonar, the acoustic noises signature can change depending on environmental conditions. It induces a variation in the propagation channel that causes the acoustic noise signature to change. Furthermore, depending on the depth, weather condition can affect the signals recorded. This is the case when rain falls, which implies a decrease in the signal-to-noise ratio. Consequently, our IA must be resilient to these changes. One solution is to add to the acoustic noise database several replications of the signals used for the IA to learn, recorded under different weather conditions. 2.2.8 Tactical mission The operation context is strongly linked to the notion of tactical mission. From the beginning of the patrol, the mission is planned and the fleet must stick to the plan. Consequently, a list of potential clients and potential areas to be crossed is available, and must therefore be taken into account for the classification task. It constitutes a preliminary probability of occurrence for various underwater acoustic noises. 2.2.9 Synthesis In summary, Table 1 presents a non-exhaustive synthesis of the available data on board. It represents all the opportunities to bring new information to the classical classification task. This list should be updated during all the study; the challenge is now to use them concretely to enhance audio classification. This inventory actually involves a lot

• f

information, too much to be taken into account to in real time by man. This is precisely why we need to use IA to synthesize all this data. This synthesis is the output of the heterogeneous classification system, materialized by a decision, is presented to the operator in order to help him to take the right decision as to the presence or absence of a threat.

UDT 2020 UDT Extended Abstract Template Presentation/Panel

Table 1. Synthesis of available data sources useful for underwater acoustic noises recognition. Data source Type Processing Enhancement capacity Watch Image Pattern recognition Spatial context Tracks Time series Kinematic analysis Behaviour Audio Sound Spectral analysis Acoustic context Navigation Scalars Geo- localization Spatial context Ships database Text Queries Ships specificities Notebooks Text Parsing Ships specificities Bio-acoustic database Map Queries Spatial context Environment Map Queries Recording context Tactical mission Text Parsing Operation context

2.3 Automatic Recognition

• f

Underwater Acoustic Noises: the Heterogeneous Approach We introduce the concept

• f

heterogeneous classification for the recognition of underwater acoustic noises by extending the classical approach described in section 2.1. Whereas to build its decision from audio data

• nly, our IA now elaborates its decision from all the data

sources enumerated in section 2.2, according to the principle illustrated in Fig.3. Instead of trying directly to merge all data sources, we choose to do the usual audio classification first, and then to enhance the output with the other data. Actually, our insight is to imitate the

• perator when faced with a difficult and ambiguous case

to recognize. To recognize underwater acoustic noises he first tries to use his hears with the audio and his eyes with the screen. Then, if he can’t make his decision directly, he uses databases and other contextual information. In order to take advantage of all data sources listed in Table 1, given the trade-off between Machine Learning and Deep Learning described in section 2.1, we now consider the output of two complementary audio classifiers, the ML one based on SVM Gaussian kernel, specially dedicated to database queries and the DL one described by [7] which offers possibilities of matching with text and images. The architecture we propose is presented in Fig.3, where the output classes have been symbolically reduced to mechanical and biological classes.

• Fig. 3. Principle of heterogeneous classification for automatic

recognition of underwater acoustic noises.

The question now is how to cleverly couple data sources listed in Table 1 with the output of the two audio classifiers ML and DL to obtain an enhanced output. The ML classifier can be used to build automatic queries to challenge bio-acoustic and ships databases which contain characteristic values so that the extracted pattern features can match. This processing can be used to query a sonar database based on centre frequency, bandwidth and duration features when ML has recognized sonar

• emissions. In addition, the ML classifier can be coupled

with text mining algorithm [8] to extract and take into account heuristic rules based for instance on the expected bandwidth, maximum frequency excursion or minimum duration of underwater acoustic noises. The output of the DL classifier output is probability density of the class of underwater acoustic noise to be recognized. Consequently, it can be coupled with map data sources to make this distribution more accurate. Actually adding additional information to the output of the IA reduces uncertainty, so that the entropy of the distribution which becomes unimodal with a predominant mode. To exploit maps, environment and tactical mission data, it is necessary to take into account the navigational information, especially latitude and longitude values. Thus, this makes it possible to define a space box around the ship. This box is applied to the maps in order to restrict the field of possible acoustic sources in the area defined by this dynamical sub-domain. Finally, the enhanced output of the ML and DL audio classifiers must be merged in order to provide the ultimate output of the heterogeneous classification system, which stands for the automatic decision sent to the operator. The methodology we propose should allow showing a decision tree synthetizing the global decision scheme. But in practice, it is possible to make many different trees, corresponding to different couples of audio and

• ther type of data. Among the different possibilities, we

can try to imitate the crew does things. Two different points of view are taken into account: that of the operator and that of the commanding officer.

UDT 2020 UDT Extended Abstract Template Presentation/Panel

3 Distributed Architecture for on-board Heterogeneous Classification

The concept

• f

heterogeneous classification simultaneously involves a great amount of data of different types. Consequently, Data Lake technology is the architecture dedicated to ingest different kinds of data such as sounds, images, time series and text files, as illustrated in Fig.4. Concretely, the Data Lake is nothing more than a suite made of hardware with a set of hard drives, and software with a set of software to manipulate data called ecosystem.

• Fig. 4. On-board Data Lake for heterogeneous classification

We propose in this study a complete architecture based on a Big Data framework of the Hadoop ecosystem coupled with Apache Spark processing [9]. The ecosystem associated with the hardware allows a fast and massive data ingestion. Moreover, this distributed architecture uses mapping and reduction techniques to cleverly store data, whatever their type, in order to query and process them quickly. This feature is particularly dedicated to the learning process which requires a lot of computing and data manipulation time. It is particularly well dedicated to the heterogeneous classification task, based by definition on the association of data with different types such as maps, texts, images, time series and sounds.

4 Experiments

To illustrate the benefit

• f

heterogeneous classification for the recognition of underwater acoustic noises, we propose a simple use case. It deals with the un-classification task, which consists for the operator watching the acoustic landscape to first eliminate biological activity and then to focus on the rest, which potentially contains threats. To do so, we propose to use jointly the ML audio classifier with the bio-acoustic database and the DL audio classifier coupled to the biological activity map and navigation. The first results should show the capacity to enhance the classical classification.

5 Conclusion

This study proposes to design an IA for underwater acoustic recognition based on audio classification, but also on all data sources available on board. This concept is called heterogeneous classification. It consists in merging additional data with the output of the classical audio classifier. After having detailed the classical approach based on audio classification, we have listed available data sources, and thus defined the principle of heterogeneous classification as an extension of the classical approach, completed by the other data sources. We have also proposed a distributed architecture where it is possible to deploy IA and have it learn. With the example of the un-classification principle, we have presented our first experiment, showing the benefit of adding additional information to the original output of the audio classifier. Furthermore, the study

• f

heterogeneous classification does not avoid the study of audio classification, which remains the main decision channel and must therefore have the best performance in generalization. Research is now conducted to completely identify the data sources and to find comprehensive techniques that can integrate all the data sources involved into the

• processing. Our goal is to deploy heterogeneous

classification on board, in order to make our IA more and more efficient over time. The heterogeneous classification system is coupled to an on-board data lake. An on-board IT infrastructure adapted to the constraints of the ship allows reaching the requirements of power consumption, cyber security and more generally of on-board installation. The goal is to make the IA more and more performant by iterating the learning step after each patrol.

References

[1] Waite, A. D., “Sonar for practising engineers”. Chichester: Wiley. 2005. [2] Michael Bittle and Alec Duncan, “A review of current marine mammal detection and classification algorithms for use in automated passive acoustic monitoring.” Proceedings of Acoustics, Victor Harbor, Australia, Nov. 2013. [3] Coelho, L. P., Richert, W., & Brucher, M., “Building machine learning systems with Python: Explore machine learning and deep learning techniques for building intelligent systems using scikit-learn and TensorFlow.”, 2018 [4] Goodfellow, I., Bengio, Y., & Courville, A., “Deep learning”. Cambridge, Mass: The MIT Press. 2017 [5] M. A. Hearst, S. T. Dumais, E. Osuna, J. Platt and

• B. Scholkopf, "Support vector machines," in IEEE

Intelligent Systems and their Applications, vol. 13,

• no. 4, pp. 18-28, July-Aug. 1998.

[6] S. Visalakshi and V. Radha, "A literature review of feature selection techniques and applications:

UDT 2020 UDT Extended Abstract Template Presentation/Panel Review of feature selection in data mining," 2014 IEEE International Conference on Computational Intelligence and Computing Research, Coimbatore, 2014, pp. 1-6. [7] E. Artusi, F. Chaillan, “Automatic recognition of underwater acoustic signature for naval applications”, CMRE MSAW’19, Lerici, 2019. [8] M. Sukanya and S. Biruntha, "Techniques on text mining," IEEE Int. Conf.

• n

Advanced Communication Control and Computing Technologies (ICACCCT), Ramanathapuram, 2012,

• pp. 269-271. 2012.

[9] H. Mehmood et al., "Implementing Big Data Lake for Heterogeneous Data Sources," IEEE 35th International Conference on Data Engineering Workshops (ICDEW), pp. 37-44. 2019, Macao.

Author/Speaker Biographies

Fabien CHAILLAN Signal Processing PhD. (06’), Scientific Computing Engineer (02’), now data Scientist for Naval Group, I’m leading the researches in Signal Processing and IA applied to naval defence, designing algorithms which provide assistance to the operator for the recognition of threats, for command and operation support. My main research domain is underwater acoustic recognition from passive sonar data. Stéphan MEUNIER Former captain of the French navy, specialized in anti-submarine warfare, weapons and combat systems, I’m also former commanding officer of submarine. Since 2018, I’m head of operational marketing at Naval Group and operational advisor for acoustic recognition.