vector quantized neural networks for acoustic unit
play

Vector Quantized Neural Networks for Acoustic Unit Discovery - PowerPoint PPT Presentation

Sep 27, 2023 •22 likes •622 views

Vector Quantized Neural Networks for Acoustic Unit Discovery Benjamin van Niekerk, Leanne Nortje, Herman Kamper The Generative Factors of Speech HH / Y / UW / M / ER HUMOUR Content: Prosody: Timbre: Discrete phonetic units. Rhythm

  1. Vector Quantized Neural Networks for Acoustic Unit Discovery Benjamin van Niekerk, Leanne Nortje, Herman Kamper

  2. The Generative Factors of Speech HH / Y / UW / M / ER HUMOUR Content: Prosody: Timbre: ● Discrete phonetic units. ● Rhythm ● Quality of a particular voice. ● ≅ 44 phonemes in English. ● Intonation ● Characterized by frequency ● Stresses spectrum.

  3. The Generative Factors of Speech HH / Y / UW / M / ER HUMOUR Content: Prosody: Timbre: ● Discrete phonetic units. ● Rhythm ● Quality of a particular voice. ● ≅ 44 phonemes in English. ● Intonation ● Characterized by frequency ● Stresses spectrum.

  4. The Generative Factors of Speech HH / Y / UW / M / ER HUMOUR Content: Prosody: Timbre: ● Discrete phonetic units. ● Rhythm ● Quality of a particular voice. ● ≅ 44 phonemes in English. ● Intonation ● Characterized by frequency ● Stresses spectrum.

  5. The Generative Factors of Speech HH / Y / UW / M / ER HUMOUR Content: Prosody: Timbre: ● Discrete phonetic units. ● Rhythm ● Quality of a particular voice. ● ≅ 44 phonemes in English. ● Intonation ● Characterized by frequency ● Stresses spectrum.

  6. The Generative Factors of Speech HH / Y / UW / M / ER Content: Prosody: Timbre: ● Discrete phonetic units. ● Rhythm ● Quality of a particular voice. ● ≅ 44 phonemes in English. ● Intonation ● Characterized by frequency ● Stresses spectrum.

  7. The Generative Factors of Speech HH / Y / UW / M / ER Content: Prosody: Timbre: ● Discrete phonetic units. ● Rhythm ● Quality of a particular voice. ● ≅ 44 phonemes in English. ● Intonation ● Characterized by frequency ● Stresses spectrum.

  8. The Generative Factors of Speech HH / Y / UW / M / ER Content: Prosody: Timbre: ● Discrete phonetic units. ● Rhythm ● Quality of a particular voice. ● ≅ 44 phonemes in English. ● Intonation ● Characterized by frequency ● Stresses spectrum.

  9. The Generative Factors of Speech HH / Y / UW / M / ER Content: Prosody: Timbre: ● Discrete phonetic units. ● Rhythm ● Quality of a particular voice. ● ≅ 44 phonemes in English. ● Intonation ● Characterized by frequency ● Stresses spectrum.

  10. The Generative Factors of Speech Content: Prosody: Timbre: ● Discrete phonetic units. ● Rhythm ● Quality of a particular voice. ● ≅ 44 phonemes in English. ● Intonation ● Characterized by frequency ● Stresses spectrum.

  11. What is Acoustic Unit Discovery? The goal is to learn discrete representations of speech that separate phonetic content from the other factors. …all without any labels or annotations!

  12. What is Acoustic Unit Discovery? The goal is to learn discrete representations of speech that separate phonetic content from the other factors. …all without any labels or annotations!

  13. What is Acoustic Unit Discovery? The goal is to learn discrete representations of speech that separate phonetic content from the other factors. …all without any labels or annotations! Encoder

  14. What is Acoustic Unit Discovery? The goal is to learn discrete representations of speech that separate phonetic content from the other factors. …all without any labels or annotations! Encoder

  15. Applications Bootstrap training of low-resource speech systems: Automatic speech recognition Text-to-speech Non-parallel voice conversion

  16. Applications Bootstrap training of low-resource speech systems: Automatic speech recognition Text-to-speech Non-parallel voice conversion

  17. Applications Bootstrap training of low-resource speech systems: Automatic speech recognition Text-to-speech Non-parallel voice conversion

  18. Applications Bootstrap training of low-resource speech systems: Automatic speech recognition Text-to-speech Non-parallel voice conversion

  19. But, how do we learn discrete representations using neural networks?

  20. But, how do we learn discrete representations using neural networks? A. van den Oord, and O. Vinyals. “Neural discrete representation learning.” Advances in Neural Information Processing Systems . 2017.

  21. Vector Quantization Layer Codebook

  22. Vector Quantization Layer Codebook Encoder

  23. Vector Quantization Layer Codebook Encoder

  24. Vector Quantization Layer Codebook Encoder

  25. Vector Quantization Layer Codebook Encoder

  26. Vector Quantization Layer Codebook Encoder

  27. Vector Quantization Layer Codebook Encoder

  28. Vector Quantization Layer Codebook Encoder

  29. Vector Quantization Layer Codebook Encoder

  30. Our contribution: we propose and compare two models for acoustic unit discovery in the ZeroSpeech 2020 Challenge . A Vector-Quantized Variational A combination of Vector-Quantization and 1. 2. Autoencoder (VQ-VAE) Contrastive Predictive Coding (VQ-CPC) VQ layer Encoder Decoder Inspired by: J. Chorowski, et al. “Unsupervised speech representation learning using wavenet autoencoders.” IEEE/ACM transactions on audio, speech, and language processing. 2019.

  31. Our contribution: we propose and compare two models for acoustic unit discovery in the ZeroSpeech 2020 Challenge . A Vector-Quantized Variational A combination of Vector-Quantization and 1. 2. Autoencoder (VQ-VAE) Contrastive Predictive Coding (VQ-CPC) VQ layer Encoder Decoder Inspired by: J. Chorowski, et al. “Unsupervised speech representation learning using wavenet autoencoders.” IEEE/ACM transactions on audio, speech, and language processing. 2019.

  32. Our contribution: we propose and compare two models for acoustic unit discovery in the ZeroSpeech 2020 Challenge . A Vector-Quantized Variational A combination of Vector-Quantization and 1. 2. Autoencoder (VQ-VAE) Contrastive Predictive Coding (VQ-CPC) VQ layer Encoder Decoder Inspired by: J. Chorowski, et al. “Unsupervised speech representation learning using wavenet autoencoders.” Inspired by: A. van den Oord, et al. “Representation Learning with Contrastive Predictive Coding.” 2018. IEEE/ACM transactions on audio, speech, and language processing. 2019.

  33. Vector-Quantized Variational Autoencoder VQ layer Encoder Decoder

  34. Vector-Quantized Variational Autoencoder VQ layer Encoder Decoder minimize reconstruction error

  35. Vector-Quantized Variational Autoencoder Information bottleneck VQ layer Encoder Decoder

  36. Vector-Quantized Variational Autoencoder Information bottleneck VQ layer Encoder Decoder Speaker

  37. Vector-Quantized Variational Autoencoder Information bottleneck VQ layer Encoder Decoder Powerful autoregressive Speaker model

  38. Vector-Quantized Contrastive Predictive Coding Prediction Input

  39. Vector-Quantized Contrastive Predictive Coding Encoder Input

  40. Vector-Quantized Contrastive Predictive Coding VQ layer Encoder Input

  41. Vector-Quantized Contrastive Predictive Coding Context model VQ layer Encoder Input

  42. Vector-Quantized Contrastive Predictive Coding Predictions Context model VQ layer Encoder Input

  43. Vector-Quantized Contrastive Predictive Coding Context vector

  44. Vector-Quantized Contrastive Predictive Coding Positive example Context vector

  45. Vector-Quantized Contrastive Predictive Coding Positive example Context vector Negative examples

  46. Vector-Quantized Contrastive Predictive Coding Positive example Context vector Negative examples

  47. Vector-Quantized Contrastive Predictive Coding Positive example Context vector Negative examples

  48. Evaluation - Voice Conversion Evaluation Metrics: VQ layer Encoder Decoder ● Speaker similarity (1-5 scale). ● Intelligibility (character error rate). ● Mean opinion score (1-5 scale).

  49. Evaluation - Voice Conversion Evaluation Metrics: VQ layer Encoder Decoder ● Speaker similarity (1-5 scale). ● Intelligibility (character error rate). ● Mean opinion score (1-5 scale).

  50. Evaluation - Voice Conversion Source Converted Target Other Conversion

  51. Evaluation - Voice Conversion

  52. Evaluation - Voice Conversion

  53. Evaluation - Voice Conversion

  54. Evaluation - ABX Score Triphone A: bug Encoder

  55. Evaluation - ABX Score Triphone A: Triphone B: bug bag Encoder Encoder

  56. Evaluation - ABX Score Triphone A: Triphone X: Triphone B: bug bag bag Encoder Encoder Encoder

  57. Evaluation - ABX Score Triphone A: Triphone X: Triphone B: bug bag bag Encoder Encoder Encoder

  58. Evaluation - ABX Score

  59. Questions?

  60. Vector Quantized Variational Autoencoder Bottleneck Encoder linear(64) VQ(512) ReLU Decoder batchnorm conv 3 ( 768 ) jitter(0.5) embedding ReLU concat batchnorm upsample conv 3 ( 768 ) biGRU(128) ReLU biGRU(128) batchnorm 50Hz upsample conv 4 stride 2 ( 768 ) ReLU GRU(896) batchnorm linear(256) conv 3 ( 768 ) embedding ReLU ReLU linear(256) batchnorm ReLU conv 3 ( 768 ) 100Hz sample mu-law softmax log-Mel spec speaker

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural

Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural

Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks Interspeech 2019, Graz, Austria Ryan Eloff, Andr e Nortje, Benjamin van Niekerk, Avashna Govender, Leanne Nortje, Arnu Pretorius, Elan

744 views • 38 slides
DCASE 2016 CONVOLUTIONAL NEURAL NETWORKS FOR ACOUSTIC SCENE CLASSIFICATION Michele Valenti 1

DCASE 2016 CONVOLUTIONAL NEURAL NETWORKS FOR ACOUSTIC SCENE CLASSIFICATION Michele Valenti 1

DCASE 2016 CONVOLUTIONAL NEURAL NETWORKS FOR ACOUSTIC SCENE CLASSIFICATION Michele Valenti 1 (valenti.michele.w@gmail.com), Aleksandr Diment 2 , Giambattista Parascandolo 2 , Stefano Squartini 1 , Tuomas Virtanen 2 1 Universit Politecnica delle

602 views • 41 slides
Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker

Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker

Introduction HMM based method Deep Neural Networks (DNNs) Experiments and Results Conclusions Deep Neural Networks and Hidden Markov Models in i-vector-based Text-Dependent Speaker Verification Hossein Zeinali 1 , 2 , Luk s Burget 2 ,

301 views • 11 slides
Convolutional Neural Networks (CNNs) Recurrent Neural Networks (RNNs) L1 Scalar Processor L0

Convolutional Neural Networks (CNNs) Recurrent Neural Networks (RNNs) L1 Scalar Processor L0

Convolutional Neural Networks (CNNs) Recurrent Neural Networks (RNNs) L1 Scalar Processor L0 L0 M*V Processor N N N N N N Project Brainwave Pretrained DNN Model Scalable DNN Hardware Neural Processing Unit Microservice Extract

525 views • 30 slides
Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear

Training Neural Networks CMSC 470 Marine Carpuat Neural Networks so far Powerful non-linear models for classification Predictions are made as a sequence of simple operations matrix-vector operations non-linear activation functions

417 views • 14 slides
General Neural Networks Compositions of linear maps and component-wise non- linearities Neural

General Neural Networks Compositions of linear maps and component-wise non- linearities Neural

Pooling and Invariance in Convolutional Neural Networks General Neural Networks Compositions of linear maps and component-wise non- linearities Neural Networks Common Non-Linearities Rectifier Linear Unit Sigmoid Hyperbolic tangent

844 views • 53 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Acoustic Scene Classification by Ensembling Gradient Boosting Machine and Convolutional Neural

Acoustic Scene Classification by Ensembling Gradient Boosting Machine and Convolutional Neural

Acoustic Scene Classification by Ensembling Gradient Boosting Machine and Convolutional Neural Networks DCASE 2017 Eduardo Fonseca, Rong Gong, Dmitry Bogdanov, Olga Slizovskaia, Emilia Gomez and Xavier Serra Outline Introduction

581 views • 38 slides
CSC421 Intro to Artificial Intelligence UNIT 32: Instance-based Learning and Neural Networks

CSC421 Intro to Artificial Intelligence UNIT 32: Instance-based Learning and Neural Networks

CSC421 Intro to Artificial Intelligence UNIT 32: Instance-based Learning and Neural Networks Outline Nearest-Neighbhor models Kernel Models Neural Networks Machine Learning using Weka Classification using single Gaussian

140 views • 13 slides
Extremely Low-bit Convolution Optimization for Quantized Neural Network on Modern Computer

Extremely Low-bit Convolution Optimization for Quantized Neural Network on Modern Computer

Extremely Low-bit Convolution Optimization for Quantized Neural Network on Modern Computer Architectures Qingchang Han 1,2 , Yongmin Hu 1 , Fengwei Yu 2 , Hailong Yang 1 , Bing Liu 2 , Peng Hu 1,2 , Ruihao Gong 1,2 , Yanfei Wang 2 , Rui Wang 1 ,

375 views • 35 slides
Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks

Neural Networks 0. Logistics Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

852 views • 33 slides
Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks

Neural Networks 1. Introduction Fall 2017 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they have

1.17k views • 91 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2019 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.57k views • 124 slides
Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural

Neural Networks 1. Introduction Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In many problems they

1.63k views • 119 slides
Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography

Relaxation and Hopfield Networks Neural Networks Neural Networks - Hopfield 1 Bibliography Hopfield, J. J., "Neural networks and physical systems with emergent collective computational abilities," Proceedings of the National Academy

367 views • 19 slides
Integrating multi-dimensional information spaces Kostas Saidis, Alex Delis {saiko,ad}@di.uoa.gr

Integrating multi-dimensional information spaces Kostas Saidis, Alex Delis {saiko,ad}@di.uoa.gr

Integrating multi-dimensional information spaces Kostas Saidis, Alex Delis {saiko,ad}@di.uoa.gr University of Athens 2 Oct. 2009, Corfu, Greece 2 nd Workshop on Very Large Digital Libraries (VLDL 2009) In conjunction with ECDL 2009 Size does

460 views • 19 slides
TECHNICAL DISCOVERY Ravindra Singh - @ravindrasingh01 Shashank Merothiya - @shashtra

TECHNICAL DISCOVERY Ravindra Singh - @ravindrasingh01 Shashank Merothiya - @shashtra

TECHNICAL DISCOVERY Ravindra Singh - @ravindrasingh01 Shashank Merothiya - @shashtra www.srijan.net | business@srijan.net Proprietary & Confidential WHY WE NEED A TECHNICAL DISCOVERY? Quality Budget Which leads to... Better Delivery

770 views • 22 slides
Native Content Distribution through Off-Path Content Discovery A Proposal for a Downstream

Native Content Distribution through Off-Path Content Discovery A Proposal for a Downstream

Native Content Distribution through Off-Path Content Discovery A Proposal for a Downstream FIB Opportunistic Off-Path Content Discovery in Information-Centric Networks O. Ascigil, V. Sourlas, I. Psaras, G. Pavlou IEEE LANMAN 2016

431 views • 39 slides
Aural Melissa Chan (chanm3) & Josh Nazarian (jknaz) The problem: Personalized content

Aural Melissa Chan (chanm3) & Josh Nazarian (jknaz) The problem: Personalized content

Aural Melissa Chan (chanm3) & Josh Nazarian (jknaz) The problem: Personalized content discovery and current CDPs are insufficient for expanding to new genres of music. When a person discovers a new track or artist, there is no

416 views • 3 slides
On the Feasibility of a User-Operated Mobile Content Distribution Network Ioannis Psaras , Vasilis

On the Feasibility of a User-Operated Mobile Content Distribution Network Ioannis Psaras , Vasilis

On the Feasibility of a User-Operated Mobile Content Distribution Network Ioannis Psaras , Vasilis Sourlas, Denis Shtefan, Sergi Re and George Pavlou University College London, UK Mayutan Arumaithurai University of Goettingen, Germany Dirk

652 views • 44 slides
Distributed Algorithms for Content Allocation in Interconnected Content Distribution Networks

Distributed Algorithms for Content Allocation in Interconnected Content Distribution Networks

Problem Definition Failure of Distributed Greedy Bilateral Compensation-based Algorithms Conclusion Distributed Algorithms for Content Allocation in Interconnected Content Distribution Networks Valentino Pacifici, Gy orgy D an

657 views • 63 slides
ESP Recreation & Tourism Update Draft Findings and Recommendations November 19, 2020 Update

ESP Recreation & Tourism Update Draft Findings and Recommendations November 19, 2020 Update

ESP Recreation & Tourism Update Draft Findings and Recommendations November 19, 2020 Update Objectives Legislative directive to prepare and update ESP. Assemble best available current data and analyses of visitation, economic

704 views • 17 slides
Optimal content filtering in social networks with limited budget of attention Nidhi Hegde

Optimal content filtering in social networks with limited budget of attention Nidhi Hegde

Optimal content filtering in social networks with limited budget of attention Nidhi Hegde Technicolor, France Bo Jiang (UMass), Laurent Massouli (MSR-INRIA), Laurent Viennot (INRIA) IFCAM Workshop on Social Networks Bangalore, Jan 16 2014

596 views • 38 slides

More recommend