introduction to speaker diarization
play

Introduction to Speaker Diarization Dr. Gerald Friedland - PowerPoint PPT Presentation

Jan 31, 2024 •494 likes •1.92k views

Introduction to Speaker Diarization Dr. Gerald Friedland International Computer Science Institute Berkeley, CA friedland@icsi.berkeley.edu Monday, May 21, 12 Speaker Diarization... tries to answer the question: who spoke when?

  1. Segmentation & Clustering • Originally: Segment first, cluster later Chen, S. S. and Gopalakrishnan, P., “Clustering via the bayesian information criterion with applications in speech recognition,” Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, 2001, Vol. 2, Seattle, USA, pp. 645-648. • More e ffj cient: Top-Down and Bottom-Up Approaches 16 Monday, May 21, 12

  2. Segmentation: Secret Sauce 17 Monday, May 21, 12

  3. Segmentation: Secret Sauce • How do you distinguish speakers? 17 Monday, May 21, 12

  4. Segmentation: Secret Sauce • How do you distinguish speakers? • Combination of MFCC+GMM+BIC seems unbeatable! 17 Monday, May 21, 12

  5. Segmentation: Secret Sauce • How do you distinguish speakers? • Combination of MFCC+GMM+BIC seems unbeatable! • Can be generalized to Audio Percepts 17 Monday, May 21, 12

  6. MFCC: Idea Audio Signal Mel-Scale Pre-emphasis Filterbank Windowing Log-Scale FFT DCT MFCC power cepstrum of signal 18 Monday, May 21, 12

  7. MFCC: Mel Scale 19 Monday, May 21, 12

  8. MFCC: Result 20 Monday, May 21, 12

  9. Gaussian Mixtures 21 Monday, May 21, 12

  10. Training of Mixture Models Goal: Find a i for Expectation: Maximization: 22 Monday, May 21, 12

  11. Bayesian Information Criterion BIC = where X is the sequence of features for a segment, Θ are the parameters of the statistical model for the segment, K is the number of parameters for the model, N is the number of frames in the segment, λ is an optimization parameter. 23 Monday, May 21, 12

  12. Bayesian Information Criterion: Explanation 24 Monday, May 21, 12

  13. Bayesian Information Criterion: Explanation • BIC penalizes the complexity of the model (as of number of parameters in model). 24 Monday, May 21, 12

  14. Bayesian Information Criterion: Explanation • BIC penalizes the complexity of the model (as of number of parameters in model). • BIC measures the e ffj ciency of the parameterized model in terms of predicting the data. 24 Monday, May 21, 12

  15. Bayesian Information Criterion: Explanation • BIC penalizes the complexity of the model (as of number of parameters in model). • BIC measures the e ffj ciency of the parameterized model in terms of predicting the data. • BIC is therfore used to choose the number of clusters according to the intrinsic complexity present in a particular dataset. 24 Monday, May 21, 12

  16. Bayesian Information Criterion: Properties 25 Monday, May 21, 12

  17. Bayesian Information Criterion: Properties • BIC is a minimum description length criterion. 25 Monday, May 21, 12

  18. Bayesian Information Criterion: Properties • BIC is a minimum description length criterion. • BIC is independent of the prior. 25 Monday, May 21, 12

  19. Bayesian Information Criterion: Properties • BIC is a minimum description length criterion. • BIC is independent of the prior. • It is closely related to other penalized likelihood criteria such as RIC and the Akaike information criterion. 25 Monday, May 21, 12

  20. Bottom-Up Algorithm Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster2 Cluster1 Cluster2 Cluster3  Start with too many clusters (initialized randomly)  Purify clusters by comparing and merging similar clusters 26  Resegment and repeat until no more merging needed Monday, May 21, 12

  21. Bottom-Up Algorithm Initialization  Start with too many clusters (initialized randomly)  Purify clusters by comparing and merging similar clusters 26  Resegment and repeat until no more merging needed Monday, May 21, 12

  22. Bottom-Up Algorithm Initialization Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3  Start with too many clusters (initialized randomly)  Purify clusters by comparing and merging similar clusters 26  Resegment and repeat until no more merging needed Monday, May 21, 12

  23. Bottom-Up Algorithm Initialization Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 (Re-)Training  Start with too many clusters (initialized randomly)  Purify clusters by comparing and merging similar clusters 26  Resegment and repeat until no more merging needed Monday, May 21, 12

  24. Bottom-Up Algorithm Initialization Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 (Re-)Training  Start with too many clusters (initialized randomly)  Purify clusters by comparing and merging similar clusters 26  Resegment and repeat until no more merging needed Monday, May 21, 12

  25. Bottom-Up Algorithm Initialization Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 (Re-)Training (Re-)Alignment  Start with too many clusters (initialized randomly)  Purify clusters by comparing and merging similar clusters 26  Resegment and repeat until no more merging needed Monday, May 21, 12

  26. Bottom-Up Algorithm Initialization Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 (Re-)Training (Re-)Alignment Cluster2 Cluster2 Cluster1 Cluster1 Cluster2 Cluster2 Cluster3 Cluster3  Start with too many clusters (initialized randomly)  Purify clusters by comparing and merging similar clusters 26  Resegment and repeat until no more merging needed Monday, May 21, 12

  27. Bottom-Up Algorithm Initialization Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Cluster1 Cluster2 Cluster3 Yes (Re-)Training Merge two Clusters? (Re-)Alignment Cluster2 Cluster2 Cluster1 Cluster1 Cluster2 Cluster2 Cluster3 Cluster3  Start with too many clusters (initialized randomly)  Purify clusters by comparing and merging similar clusters 26  Resegment and repeat until no more merging needed Monday, May 21, 12

  28. Bottom-Up Algorithm Initialization Cluster2 Cluster2 Cluster1 Cluster1 Cluster2 Cluster2 Cluster2 Cluster2 Yes (Re-)Training Merge two Clusters? (Re-)Alignment Cluster2 Cluster2 Cluster1 Cluster1 Cluster2 Cluster2 Cluster3 Cluster3  Start with too many clusters (initialized randomly)  Purify clusters by comparing and merging similar clusters 26  Resegment and repeat until no more merging needed Monday, May 21, 12

  29. Bottom-Up Algorithm Initialization Cluster2 Cluster2 Cluster1 Cluster1 Cluster2 Cluster2 Cluster2 Cluster2 Yes (Re-)Training Merge two Clusters? (Re-)Alignment Cluster2 Cluster2 Cluster1 Cluster1 Cluster2 Cluster2 Cluster3 Cluster3  Start with too many clusters (initialized randomly)  Purify clusters by comparing and merging similar clusters 26  Resegment and repeat until no more merging needed Monday, May 21, 12

  30. Bottom-Up Algorithm Initialization Cluster2 Cluster2 Cluster1 Cluster1 Cluster2 Cluster2 Cluster2 Cluster2 Yes (Re-)Training Merge two Clusters? (Re-)Alignment Cluster1 Cluster1 Cluster2 Cluster2 Cluster1 Cluster1 Cluster2 Cluster2  Start with too many clusters (initialized randomly)  Purify clusters by comparing and merging similar clusters 26  Resegment and repeat until no more merging needed Monday, May 21, 12

  31. Bottom-Up Algorithm Initialization Cluster2 Cluster2 Cluster1 Cluster1 Cluster2 Cluster2 Cluster2 Cluster2 No Yes (Re-)Training Merge two Clusters? End (Re-)Alignment Cluster1 Cluster1 Cluster2 Cluster2 Cluster1 Cluster1 Cluster2 Cluster2  Start with too many clusters (initialized randomly)  Purify clusters by comparing and merging similar clusters 26  Resegment and repeat until no more merging needed Monday, May 21, 12

  32. ICSI’s Speaker Diarization 27 Monday, May 21, 12

  33. ICSI’s Speaker Diarization • Speaker Diarization research @ ICSI since 2001 27 Monday, May 21, 12

  34. ICSI’s Speaker Diarization • Speaker Diarization research @ ICSI since 2001 • Various versions of Diarization Engines developed over the years 27 Monday, May 21, 12

  35. ICSI’s Speaker Diarization • Speaker Diarization research @ ICSI since 2001 • Various versions of Diarization Engines developed over the years • Status: Research code but stable for some applications that are error tolerant 27 Monday, May 21, 12

  36. ICSI’s Speaker Diarization Engine Variants 28 Monday, May 21, 12

  37. ICSI’s Speaker Diarization Engine Variants  Basic (single mic, easy installation) 28 Monday, May 21, 12

  38. ICSI’s Speaker Diarization Engine Variants  Basic (single mic, easy installation)  Fast (single mic, multiple CPU cores) 28 Monday, May 21, 12

  39. ICSI’s Speaker Diarization Engine Variants  Basic (single mic, easy installation)  Fast (single mic, multiple CPU cores)  Super fast (single mic, multiple GPUs) 28 Monday, May 21, 12

  40. ICSI’s Speaker Diarization Engine Variants  Basic (single mic, easy installation)  Fast (single mic, multiple CPU cores)  Super fast (single mic, multiple GPUs)  Accurate but slow (multi mic, additional preprocessing) 28 Monday, May 21, 12

  41. ICSI’s Speaker Diarization Engine Variants  Basic (single mic, easy installation)  Fast (single mic, multiple CPU cores)  Super fast (single mic, multiple GPUs)  Accurate but slow (multi mic, additional preprocessing)  Audio/Visual (single and multi mic, for localization) 28 Monday, May 21, 12

  42. ICSI’s Speaker Diarization Engine Variants  Basic (single mic, easy installation)  Fast (single mic, multiple CPU cores)  Super fast (single mic, multiple GPUs)  Accurate but slow (multi mic, additional preprocessing)  Audio/Visual (single and multi mic, for localization)  Online (single mic, “who is speaking now”) 28 Monday, May 21, 12

  43. Basic Speaker Diarization: Facts 29 Monday, May 21, 12

  44. Basic Speaker Diarization: Facts • Input: 16kHz mono audio 29 Monday, May 21, 12

  45. Basic Speaker Diarization: Facts • Input: 16kHz mono audio • Features: MFCC19, no delta or deltadelta 29 Monday, May 21, 12

  46. Basic Speaker Diarization: Facts • Input: 16kHz mono audio • Features: MFCC19, no delta or deltadelta • Speech/Non-Speech Detector external 29 Monday, May 21, 12

  47. Basic Speaker Diarization: Facts • Input: 16kHz mono audio • Features: MFCC19, no delta or deltadelta • Speech/Non-Speech Detector external • Runtime: ~ realtime (1h audio needs 1h processing on a single CPU, excluding speech/non-speech) 29 Monday, May 21, 12

  48. Multi-CPU Speaker Diarization: Facts 30 Monday, May 21, 12

  49. Multi-CPU Speaker Diarization: Facts • Same as Basic Speaker Diarization 30 Monday, May 21, 12

  50. Multi-CPU Speaker Diarization: Facts • Same as Basic Speaker Diarization • Runtime: Dependent on number of CPUs used. Example: 8 cores runtime = 14.3 x realtime, i.e. 14minutes of audio need 1 minute of processing. 30 Monday, May 21, 12

  51. Multi-CPU Speaker Diarization: Facts • Same as Basic Speaker Diarization • Runtime: Dependent on number of CPUs used. Example: 8 cores runtime = 14.3 x realtime, i.e. 14minutes of audio need 1 minute of processing. • Runtime bottleneck usually: Speech/ Non-Speech Detector 30 Monday, May 21, 12

  52. GPU Speaker Diarization: Facts 31 Monday, May 21, 12

  53. GPU Speaker Diarization: Facts • Same as Basic Speaker Diarization 31 Monday, May 21, 12

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

First Investigations on Self Trained Speaker Diarization el Le Lan 1 , 2 Sylvain Meignier 2 Ga

First Investigations on Self Trained Speaker Diarization el Le Lan 1 , 2 Sylvain Meignier 2 Ga

First Investigations on Self Trained Speaker Diarization el Le Lan 1 , 2 Sylvain Meignier 2 Ga Delphine Charlet 1 Anthony Larcher 2 1 Orange Labs, France first.lastname@orange.com 2 LIUM, Universit e du Maine, France

449 views • 33 slides
speaker change detection using fundamental frequency with application to multi-talker segmentation

speaker change detection using fundamental frequency with application to multi-talker segmentation

speaker change detection using fundamental frequency with application to multi-talker segmentation May 16, 2019 Aidan Hogg, Christine Evers and Patrick Naylor Electrical and Electronic Engineering, Imperial College London, UK diarization

735 views • 29 slides
Segmentation of Broadcast News Brecht Desplanques, Kris Demuynck & Jean-Pierre Martens ELIS

Segmentation of Broadcast News Brecht Desplanques, Kris Demuynck & Jean-Pierre Martens ELIS

Soft VAD in Factor Analysis Based Speaker Segmentation of Broadcast News Brecht Desplanques, Kris Demuynck & Jean-Pierre Martens ELIS Data Science Lab Ghent University/iMinds 21 June 2016 Speaker Diarization for Automatic Subtitle

628 views • 17 slides
PIPELINE Speaker Series September 13, 2018, 8:00 am Speaker Series Agenda Welcome and

PIPELINE Speaker Series September 13, 2018, 8:00 am Speaker Series Agenda Welcome and

PIPELINE Speaker Series September 13, 2018, 8:00 am Speaker Series Agenda Welcome and Introductions Name, organization, and favorite thing about back to school time Introduction of Speaker Series Janel Anderson of Working

812 views • 35 slides
Configuration and Management of Speaker Verification Systems W3C Workshop on Speaker Biometrics

Configuration and Management of Speaker Verification Systems W3C Workshop on Speaker Biometrics

Configuration and Management of Speaker Verification Systems W3C Workshop on Speaker Biometrics and VoiceXML 3.0 Chuck Johnson Architect iBiometrics, Inc. Introduction For peak performance of a Speaker Verification solution, the VoiceXML

464 views • 10 slides
Odyssey 2016, Bilbao, Spain Short- and Long-Term Speech Features for Hybrid HMM-i-Vector based

Odyssey 2016, Bilbao, Spain Short- and Long-Term Speech Features for Hybrid HMM-i-Vector based

Odyssey 2016, Bilbao, Spain Short- and Long-Term Speech Features for Hybrid HMM-i-Vector based Speaker Diarization System Abraham Woubie 1 , Jordi Luque 2 , and Javier Hernando 1 1 TALP Research Center, Dept. of Signal Theory and Communications,

890 views • 23 slides
Applied Bayesian Nonparametrics 3. Infinite Hidden Markov Models Tutorial at CVPR 2012 Erik

Applied Bayesian Nonparametrics 3. Infinite Hidden Markov Models Tutorial at CVPR 2012 Erik

Applied Bayesian Nonparametrics 3. Infinite Hidden Markov Models Tutorial at CVPR 2012 Erik Sudderth Brown University Work by E. Fox, E. Sudderth, M. Jordan, & A. Willsky AOAS 2011: A Sticky HDP-HMM with Application to Speaker Diarization

664 views • 40 slides
Combining Speech and Speaker Recognition - A Joint Modeling Approach Hang Su Supervised by:

Combining Speech and Speaker Recognition - A Joint Modeling Approach Hang Su Supervised by:

Introduction and Motivation Backgrounds on Speech and Speaker Recognition Connecting Speech and Speaker Recognition Joint Modeling of Speech and Speaker Conclusion and Future Work Combining Speech and Speaker Recognition - A Joint Modeling

729 views • 72 slides
FOSDEM - 2 fvrier 2020 Sp Speaker prese sentation Speaker name: Jehan Monnier Function:

FOSDEM - 2 fvrier 2020 Sp Speaker prese sentation Speaker name: Jehan Monnier Function:

FOSDEM - 2 fvrier 2020 Sp Speaker prese sentation Speaker name: Jehan Monnier Function: Software engineer since 1999 Involved in Linphones development since 2010. Agen Ag enda Video intercom uses cases. Introduction to SIP VoIP

635 views • 34 slides
FARM & SEA TO CAMPUS FORUM Menuing for a Regional Food System OUTLINE I. INTRODUCTION

FARM & SEA TO CAMPUS FORUM Menuing for a Regional Food System OUTLINE I. INTRODUCTION

APRIL 9TH, 2018 FARM & SEA TO CAMPUS FORUM Menuing for a Regional Food System OUTLINE I. INTRODUCTION II. SPEAKER #1 - MIKE WEBSTER III. SPEAKER #2 - MARY REILLY IV. SPEAKER #3 - DAVID GOULD V. Q&A VI. CONCLUSION Photo

608 views • 28 slides
Design Flood Hydrology Workshop: Speaker Profiles March 28, 2017 Presentation Outline Speaker

Design Flood Hydrology Workshop: Speaker Profiles March 28, 2017 Presentation Outline Speaker

Design Flood Hydrology Workshop: Speaker Profiles March 28, 2017 Presentation Outline Speaker Bio Brian Chow , MEng., DFEng., Brian has work with the Ministry of Forests, in its Introduction and Welcome PEng., FEC various incarnations for

428 views • 5 slides
Coding Workshop Wednesday 9 th May 2018 In Introduction & Agenda Item Time Speaker

Coding Workshop Wednesday 9 th May 2018 In Introduction & Agenda Item Time Speaker

Coding Workshop Wednesday 9 th May 2018 In Introduction & Agenda Item Time Speaker Arrival & Registration 10:00 - 10:30 Welcome introduction 10:30 10:35 Melanie Hingorani Introduction to the NCO, HRGs, grouping and 10:35 -

1.31k views • 100 slides
Towards 5G HetNets: Speaker Introduction Jeffrey G. Andrews Wireless Networking and

Towards 5G HetNets: Speaker Introduction Jeffrey G. Andrews Wireless Networking and

Towards 5G HetNets: Speaker Introduction Jeffrey G. Andrews Wireless Networking and Communications Group (WNCG) Dept. of Electrical and Computer Engineering The University of Texas at Austin IEEE Comm. Theory Workshop Curacao May 2014 5G and

381 views • 3 slides
- Mr. Speaker, within 8 days of the new financial year all have managed to present to the House the

- Mr. Speaker, within 8 days of the new financial year all have managed to present to the House the

~:; ,~!-: OPENING BUDGET PRESENTATION TO PARLIAMENT HONOURABLEAUDLEYSHAW,MP Minister of Finance and .the Public Servi~e Thursday, April 8, 2010 . . " INTRODUCTION Mr. Speaker. the first quarter of this year has been a very busy period for

947 views • 46 slides
Speech Processing 15-492/18-492 Speaker ID Who is speaking? Speaker ID, Speaker Recognition

Speech Processing 15-492/18-492 Speaker ID Who is speaking? Speaker ID, Speaker Recognition

Speech Processing 15-492/18-492 Speaker ID Who is speaking? Speaker ID, Speaker Recognition Speaker ID, Speaker Recognition When do you use it When do you use it Security, Access Security, Access Speaker specific

924 views • 34 slides
What's New in Apache Syncope 1.2.0 Dr. Colm higeartaigh Speaker Introduction 11/14/14 2

What's New in Apache Syncope 1.2.0 Dr. Colm higeartaigh Speaker Introduction 11/14/14 2

What's New in Apache Syncope 1.2.0 Dr. Colm higeartaigh Speaker Introduction 11/14/14 2 Introduction to Apache Syncope Apache Syncope basics Identity Management solution at Apache Can retrieve and store users/roles/etc from/in

920 views • 25 slides
Do You See What I See? Differential Treatment of Anonymous Users Sheharbano Khattak (University

Do You See What I See? Differential Treatment of Anonymous Users Sheharbano Khattak (University

Do You See What I See? Differential Treatment of Anonymous Users Sheharbano Khattak (University of Cambridge) David Fifield (UC Berkeley) Sadia Afroz (ICSI) Mobin Javed (UC Berkeley) Srikanth Sundaresan (ICSI) Vern Paxson (UC Berkeley,

540 views • 49 slides
Following the Packets: A Walk Through Bros Internal Processing Pipeline Robin Sommer

Following the Packets: A Walk Through Bros Internal Processing Pipeline Robin Sommer

Following the Packets: A Walk Through Bros Internal Processing Pipeline Robin Sommer robin@icir.org Corelight, Inc. International Computer Science Institute Lawrence Berkeley National Laboratory Outline Bros Architecture & Data

896 views • 44 slides
RESPITE: Tandem & multistream research Dan Ellis International Computer Science Institute,

RESPITE: Tandem & multistream research Dan Ellis International Computer Science Institute,

RESPITE: Tandem & multistream research Dan Ellis International Computer Science Institute, Berkeley CA <dpwe@icsi.berkeley.edu> Outline 1 Tandem & LVCSR 2 Mutual information for multistream design 3 Other multistream work at

251 views • 13 slides
The ICSI corpus; Browsing meetings nlssd natural language and speech system design . Steve

The ICSI corpus; Browsing meetings nlssd natural language and speech system design . Steve

The ICSI corpus; Browsing meetings nlssd natural language and speech system design . Steve Renals s.renals@ed.ac.uk NLSSD/Corpus and Browsing p.1/14 Overview The ICSI meetings corpus Browsing meetings Useful reading: Janin et

185 views • 14 slides
Evaluating Network Security using Internet Measurements Oliver Gasser Tuesday 23 rd May, 2017

Evaluating Network Security using Internet Measurements Oliver Gasser Tuesday 23 rd May, 2017

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Evaluating Network Security using Internet Measurements Oliver Gasser Tuesday 23 rd May, 2017 Chair of Network Architectures and Services

900 views • 54 slides
Previous a researcher in the usable security group at ICSI/UCB Now a lead research engineer at Two

Previous a researcher in the usable security group at ICSI/UCB Now a lead research engineer at Two

Previous a researcher in the usable security group at ICSI/UCB Now a lead research engineer at Two Six Labs working on DARPA Brandeis This talk goes over work from UCB 1 Install-time permissions A ccept all or dont use the app at all. No

944 views • 77 slides
File Systems: Allocation Issues, Naming, and Performance CS 111 Operating Systems Peter Reiher

File Systems: Allocation Issues, Naming, and Performance CS 111 Operating Systems Peter Reiher

File Systems: Allocation Issues, Naming, and Performance CS 111 Operating Systems Peter Reiher Lecture 14 CS 111 Page 1 Fall 2015 Outline Allocating and managing file system free space File naming and directories File volumes

806 views • 61 slides
Computer Communication Networks Introduction IECE / ICSI 416 Spring 2020 Prof. Dola Saha 1

Computer Communication Networks Introduction IECE / ICSI 416 Spring 2020 Prof. Dola Saha 1

Computer Communication Networks Introduction IECE / ICSI 416 Spring 2020 Prof. Dola Saha 1 Introductions Instructor Prof. Dola Saha, PhD University of Colorado Boulder http://www.albany.edu/faculty/dsaha/

139 views • 12 slides

More recommend