Localization and Detection Paper: 3583 Session: AUD-L3 Acoustic - - PowerPoint PPT Presentation

▶
localization and detection
SMART_READER_LITE
LIVE PREVIEW

Localization and Detection Paper: 3583 Session: AUD-L3 Acoustic - - PowerPoint PPT Presentation

Sep 13, 2023 167 likes •386 views

A Sequence Matching Network for Poly lyphonic Sound Event Localization and Detection Paper: 3583 Session: AUD-L3 Acoustic Event Detection T. N. T. Nguyen*, D. L. Jones , W. S. Gan* *School of Electrical and Electronic Engineering, Nanyang

slide-1
SLIDE 1

A Sequence Matching Network for Poly lyphonic Sound Event Localization and Detection

  • T. N. T. Nguyen*, D. L. Jonesꭝ, W. S. Gan*

*School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore ꭝDepartment of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, USA 6 May 2020 - ICASSP

Paper: 3583 Session: AUD-L3 Acoustic Event Detection

slide-2
SLIDE 2

Sound event localization and detection

2

slide-3
SLIDE 3

Sound event detection (SED) Direction-of-arrival (DOA) estimation

Sound event localization and detection (SELD)

Signal Support Spatial Filtering

3

slide-4
SLIDE 4

SELDnet: joint SED and DOA estimation

The losses of SED and DOA estimation task are jointly optimized.

  • S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen,

“Sound event localization and detection of overlapping sources using convolutional recurrent neural networks,” IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 1, pp. 34–48,March 2019

4

slide-5
SLIDE 5

Two-stage SELD

  • Y. Cao, Q. Q. Kong, T. Iqbal, F. An, W. Wang, and M. D. Plumbley, “Polyphonic sound event detection and localization

using a two-stage strategy,” in Detection and Classification of Acoustic Scenes and Events 2019 Workshop (DCASE2019), 2019.

5

slide-6
SLIDE 6

Observation

Sound event

1. Timestamp (onset, offset) 2. Sound class 3. DOA

SED

Timestamps Sound classes

DOA estimation

Timestamps DOAs Output Sequences Ground-truth Sequences

Event 1 Event 1 Event 2 Event 2 Event 3 Event 3 Event 4 Event 4 Event 5 Event 5 Event 1 Event 1 Event 2 Event 2 Event 3 Event 3 Event 4 Event 4 Event 5 Event 5

False positive

6

slide-7
SLIDE 7

A sequence matching network (SMN) for SELD

CRNN Log-mel GCC-PHAT Input Features n_frames n_classes

SED Network

CNN upsampling Bidirectional GRU FCs softmax FCs softmax FCs softmax FCs softmax Number of events

n_max_event + 1 = 3

SED

n_max_events x (n_classes +1) = 2 x 12

Azimuth

n_max_events x n_azimuths = 2 x 36

Elevation

n_max_events x n_elevations = 2 x 9

concatenate

n_frames x 11 n_frames x 128

Sequence Matching Network

Single-source histogram Complex Spectrogram Input Features n_frames n_angles

DOAE Module

n_classes=11 n_angles=324

7

slide-8
SLIDE 8

Improved SED network

Improvement: data augmentation Use random cut out with the same mask for all logmel and GCC-PHAT channels

Input Features CNN Bidirectional GRU Fully Connected Sigmoid n_classes n_classes n_frames Log-mel GCC-PHAT

8

  • Y. Cao, Q. Q. Kong, T. Iqbal, F. An, W. Wang, and M. D. Plumbley, “Polyphonic sound event detection and localization

using a two-stage strategy,” in Detection and Classification of Acoustic Scenes and Events 2019 Workshop (DCASE2019), 2019.

slide-9
SLIDE 9

DOA estimation

Input features 2D histogram n_frames n_angles Complex Spectrogram Time Frequency Binary Mask elevation azimuth vectorize

9

  • T. N. T. Nguyen, S. K. Zhao, and D. L. Jones, “Robust doa estimation of multiple speech sources,” in 2014 IEEE

International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2014, pp. 2287–2291.

slide-10
SLIDE 10

Output format

Conventional output format Proposed output format

Sound classes: multi-label multi-class classification Azimuths: regression Elevations: regression n_classes n_classes + 1 n_max_events + 1 n_max_events n_elevations n_azimuths Event 1 Event 2 Number of active events Sound classes: multi-class classification Azimuth: multi-class classification Elevation: multi-class classification

10

slide-11
SLIDE 11

Dataset

  • TAU Spatial Sound Events 2019 – Ambisonic (DCASE 2019 – task 3)
  • Data are synthesized using recorded room impulse responses (RIRs) and

clean signals. Maximum 2 overlapping sources in one frame

  • SED: 11 indoor sound classes
  • DOA: 324 angles
  • Azimuth between [0°, 360°), resolution 10°: 36 angles
  • Elevation between [-40°, 40°], resolution 10°: 9 angles

Development: 400 one-minute recordings Evaluation: 100 one- minute recording

11

slide-12
SLIDE 12

Evaluation metrics:

SED

  • Segment-based error rate
  • Segment-based F1 score
  • Segment length: 1 second

DOA estimation

  • Frame-based DOA error
  • Frame-based frame recall
  • Frame length: 0.02 second

12

slide-13
SLIDE 13

New evaluation metrics: to account for correct matching of sound classes and DOAs

  • 1. Matching F1 score (frame-based)
  • 2. Same-class matching accuracy (frame-based)

𝑛𝑏𝑢𝑑ℎ𝑗𝑜𝑕 𝑞𝑠𝑓𝑑𝑗𝑡𝑗𝑝𝑜(𝑛𝑞) = 𝑏 𝑏 + 𝑐 + 𝑑 𝑛𝑏𝑢𝑑ℎ𝑗𝑜𝑕 𝑠𝑓𝑑𝑏𝑚𝑚(𝑛𝑠) = 𝑏 𝑏 + 𝑐 + 𝑒 𝑛𝑏𝑢𝑑ℎ𝑗𝑜𝑕 𝐺1 = 2 ∗ 𝑛𝑞 ∗ 𝑛𝑠 𝑛𝑞 + 𝑛𝑠 𝑛𝑏𝑢𝑑ℎ𝑗𝑜𝑕 𝑏𝑑𝑑𝑣𝑠𝑏𝑑𝑧 𝑁𝐵 = # 𝑝𝑔 𝑑𝑝𝑠𝑠𝑓𝑑𝑢𝑚𝑧 𝑞𝑠𝑓𝑒𝑗𝑑𝑢𝑓𝑒 𝑔𝑠𝑏𝑛𝑓 − 𝑐𝑏𝑡𝑓𝑒 𝑓𝑤𝑓𝑜𝑢𝑡 𝑢ℎ𝑏𝑢 ℎ𝑏𝑤𝑓 𝑡𝑏𝑛𝑓 𝑡𝑝𝑣𝑜𝑒 𝑑𝑚𝑏𝑡𝑡 # 𝑝𝑔 𝑕𝑠𝑝𝑣𝑜𝑒 − 𝑢𝑠𝑣𝑢ℎ 𝑔𝑠𝑏𝑛𝑓 − 𝑐𝑏𝑡𝑓𝑒 𝑓𝑤𝑓𝑜𝑢𝑡 𝑢ℎ𝑏𝑢 ℎ𝑏𝑤𝑓 𝑡𝑏𝑛𝑓 𝑡𝑝𝑣𝑜𝑒 𝑑𝑚𝑏𝑡𝑡

13

slide-14
SLIDE 14

Methods for comparison

Group Methods Descriptions

Baselines

SELDnet

joint SED and DOAE [1], with log-mel and GCC-PHAT input features [2]

Two-stage

two-stage SELD [2]

Improved baseline

Two-stage-aug

Two-stage SELD with additional random cut-out augmentation for input features

Inputs to SMNs

SED-net

the SED network of the Two-stage-aug -> SED sequences for SMNs

DOA-hist

single-source histogram for DOA estimation -> DOA sequences for SMNs [5]

Proposed

SMN

SMN with the conventional SELD output format

SMN-event

SMN with new output format

Top DCASE SELD team ranking

Kapka-en

the consecutive ensemble of CRNN models with heuristics rules; ranked 1 [6]

Two-stage-en

the ensemble based on two-stage training; ranked 2 [7] 14

slide-15
SLIDE 15

SELD evaluation results

Group Methods SED error rate ↓ SED F1 score ↑ DOA error ↓ DOA frame rate ↑ Matching F1 score ↑ Same-class matching accuracy ↑ Baselines SELDnet 0.212 0.880 9.75° 0.851 0.750 0.229 Two-stage 0.143 0.921 8.28° 0.876 0.786 0.270 Improved baseline Two-stage-aug 0.108 0.944 8.42° 0.892 0.797 0.270 Inputs to SMNs SED-net 0.108 0.944 NA NA NA NA DOA-hist NA NA 4.28° 0.825 NA NA Proposed SMN 0.079 0.958 4.97° 0.913 0.869 0.359 SMN-event 0.079 0.957 5.50° 0.924 0.840 0.649 Top DCASE team ranking Kapka-en 0.08 0.947 3.7° 0.968 NA NA Two-stage-en 0.08 0.955 5.5° 0.922 NA NA

↑: the higher, the better ↓: the lower, the better

15

slide-16
SLIDE 16

Conclusions

  • Our proposed sequence matching networks outperformed the state-of-the-

art SELDnet and the two-stage method for sound event localization and detection.

  • The sequence matching network is modular and hierarchical -> improve

the performance while increase the flexibility in designing and optimizing its components.

  • The sequence matching networks increase the correct association between

the sound classes and the corresponding DOAs in multiple-source cases. The new output format can also handle the cases where multiple sound events of the same class have different DOAs.

  • The new evaluation metrics address the problem of matching sound classes

and DOAs which was not achievable using the conventional SELD evaluation metrics.

16

slide-17
SLIDE 17

References

  • 1. S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, “Sound event localization and detection of overlapping

sources using convolutional recurrent neural networks,” IEEE Journal of Selected Topics in Signal Processing,

  • vol. 13, no. 1, pp. 34–48,March 2019
  • 2. Y. Cao, Q. Q. Kong, T. Iqbal, F. An, W. Wang, and M. D. Plumbley, “Polyphonic sound event detection and

localization using a two-stage strategy,” in Detection and Classification of Acoustic Scenes and Events 2019 Workshop (DCASE2019), 2019.

  • 3. S. Adavanne, A. Politis, and T. Virtanen, “A multi-room reverberant dataset for sound event localization and

detection,” in Detection and Classification of Acoustic Scenes and Events 2019 Workshop (DCASE2019), 2019.

  • 4. A. Mesaros, T. Heittola, and T. Virtanen, “Metrics for polyphonic sound event detection,” Applied Sciences,
  • vol. 6, no. 6, pp. 162, 2016.
  • 5. T. N. T. Nguyen, S. K. Zhao, and D. L. Jones, “Robust doa estimation of multiple speech sources,” in 2014 IEEE

International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2014, pp. 2287–2291.

  • 6. S. Kapka and M. Lewandowski, “Sound source detection, localization and classification using consecutive

ensemble of CRNN models,” Tech. Rep., DCASE2019 Challenge, June 2019.

  • 7. Y. Cao, T. Iqbal, Q. Q. Kong, M. Galindo, W. Wang, and M. D Plumbley, “Two-stage sound event localization

and detection using intensity vector and generalized cross-correlation,” Tech. Rep., DCASE2019 Challenge, June 2019.

17

slide-18
SLIDE 18

Acknowledgement

This research was conducted at Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU), which is a collaboration between Singapore Telecommunications Limited (Singtel) and Nanyang Technological University (NTU) that is funded by the Singapore Government through the Industry Alignment Fund ‐ Industry Collaboration Projects Grant.

18

slide-19
SLIDE 19

This Photo by Unknown Author is licensed under CC BY-NC-ND

19

slide-20
SLIDE 20

20