analysis by synthesis of speech prosody from data to
play

Analysis by Synthesis of Speech Prosody: from Data to Models. - PowerPoint PPT Presentation

Jan 05, 2023 •363 likes •1.47k views

Analysis by Synthesis of Speech Prosody: from Data to Models. Daniel Hirst Laboratoire Parole et Langage, CNRS & Universit de Provence, Aix en Provence, France With the past, present and future collaboration of: Caroline Bouzon

  1. Prosodic structure Foot Foot -pec- They ex- -ted his e- -lec- -tion Word Word Word Word Scuola Normale Superiore, Pisa 2009 March 13 06/03/10 ATILF Nancy Daniel Hirst

  2. Prosodic structure ● Narrow rhythm unit (Jassem): sequence of syllables beginning with a stressed syllable and ending at the following word boundary ● Anacrusis (Jassem): sequence of unstressed syllables not included in a narrow rhythm unit. 06/03/10 ATILF Nancy Daniel Hirst

  3. Prosodic structure Foot Foot Ana NRU Ana NRU -dic- They pre- -ted his e- -lec- -tion Word Word Word Word 06/03/10 ATILF Nancy Daniel Hirst

  4. Aix-Marsec database • SEC (Spoken English Corpus) Knowles et al. 1996 • Marsec (Machine Readable SEC) Roach et al. 1993 • Aix-Marsec Auran, Bouzon & Hirst 2004 06/03/10 ATILF Nancy Daniel Hirst

  5. SEC ● 5.5 hours of “authentic” speech ● 53 speakers, c. 55000 words 06/03/10 ATILF Nancy Daniel Hirst

  6. SEC ● 5.5 hours of “authentic” speech ● c. 55000 words, 53 speakers ● Prosodic markup:tonetic stress marks (Knowles & Williams) Scuola Normale Superiore, Pisa 2009 March 13 06/03/10 ATILF Nancy Daniel Hirst

  7. Marsec ● Tonetic stress markup > ASCII (Roach et al.) ● words aligned with signal 06/03/10 ATILF Nancy Daniel Hirst

  8. Aix-Marsec database ● Phonetic transcription ● Phonemes aligned with signal ● Prosodic structure (Praat TextGrids) ● Automatic analysis of intonation (Momel & INTSINT) ● Freely available from the authors 06/03/10 ATILF Nancy Daniel Hirst

  9. TextGrid from Aix-Marsec 06/03/10 ATILF Nancy Daniel Hirst

  10. Hypothesis ● size of whole :: compression of parts If a prosodic constituent is involved in the planning of speech rhythm we should expect the size of the constituent to have a negative effect on the duration of the phonemes which make it up. 06/03/10 ATILF Nancy Daniel Hirst

  11. Method ● Linear correlation and regression – Independent variable: size of constituent (number of phonemes) – Dependent variable: mean lengthening/compression of phonemes (Z score) z i / p = d i / p - m p s p 06/03/10 ATILF Nancy Daniel Hirst

  12. Results - 1 ● Very significant negative correlation of lengthening of phonemes (Z-score) with number of phonemes in – Word – Foot – Narrow Rhythm Unit 06/03/10 ATILF Nancy Daniel Hirst

  13. Results - 2 ● Little or no correlation of lengthening/compression of phonemes (Z-score) with number of phonemes in: – Syllable – Anacrusis 06/03/10 ATILF Nancy Daniel Hirst

  14. Interpretation ● Syllable and anacrusis have little effect on the lengthening of English phonemes ● Word, foot and narrow rhythm unit play significant role (in that order) 06/03/10 ATILF Nancy Daniel Hirst

  15. Prosodic structure Foot Foot Ana NRU Ana NRU -pec- They ex- -ted his e- -lec- -tion Word Word Word Word 06/03/10 ATILF Nancy Daniel Hirst

  16. Results - 3 ● No simple effect of stress !!! 06/03/10 ATILF Nancy Daniel Hirst

  17. Final lengthening 06/03/10 ATILF Nancy Daniel Hirst

  18. Excluding last two phonemes of intonation unit 06/03/10 ATILF Nancy Daniel Hirst

  19. Word-final lengthening? 06/03/10 ATILF Nancy Daniel Hirst

  20. Conclusions ● No compression at level of syllable (cf Jassem et al. 1978) ● Phonemes in stressed syllable have NO specific lengthening (cf Jassem 1952!) ● The solution to Klatt’s unsolved problem is the Narrow Rhythm Unit (for English) (cf Jassem 1952!!!) ● No evidence for specific word-final lengthening 06/03/10 ATILF Nancy Daniel Hirst

  21. Duration of NRU / number of phonemes in NRU 06/03/10 ATILF Nancy Daniel Hirst

  22. mean z-score of phoneme / position in NRU 06/03/10 ATILF Nancy Daniel Hirst

  23. modelling speech melody ● Perception models ● Production models ● Acoustic models 06/03/10 ATILF Nancy Daniel Hirst

  24. Raw f0 06/03/10 ATILF Nancy Daniel Hirst

  25. Raw f0 06/03/10 ATILF Nancy Daniel Hirst

  26. raw f0 06/03/10 ATILF Nancy Daniel Hirst

  27. Raw f0 06/03/10 ATILF Nancy Daniel Hirst

  28. Finnish 06/03/10 ATILF Nancy Daniel Hirst

  29. Kloker 1975 06/03/10 ATILF Nancy Daniel Hirst

  30. Gamma function: y = at b e ct 06/03/10 ATILF Nancy Daniel Hirst

  31. Hirst's law An acoustic model should not depend on which end of the table you are talking about. 06/03/10 ATILF Nancy Daniel Hirst

  32. f0 transition 06/03/10 ATILF Nancy Daniel Hirst

  33. First derivative of raw f0 But who stole Jane's bicycle? (ma'ma'ma...) 06/03/10 ATILF Nancy Daniel Hirst

  34. Quadratic spline function • Spline function ● Sequence of functions of degree n, derivatives of which up to n-1 are everywhere continuous • Quadratic spline ● Sequence of targets linked by two quadratic functions (y = ax 2 + bx +c) 06/03/10 ATILF Nancy Daniel Hirst

  35. Quadratic spline function y =h 1 +(h 2 -h 1 )(x-t 1 ) 2 y =h 2 +(h 1 -h 2 )(x-t 2 ) 2 (t k -t 1 )(t 2 -t 1 ) (t k -t 2 )(t 1 -t 2 ) 06/03/10 ATILF Nancy Daniel Hirst

  36. Quadratic spline function Il faut que je sois à Grenoble, Samedi vers quinze heures 06/03/10 ATILF Nancy Daniel Hirst

  37. Curves vs. straight lines • 't Hart 1991 2 4 2 0 0 2 0 0 5 1 9 5 1 9 5 1 9 0 1 9 0 1 8 5 1 8 5 3 1 8 0 1 8 0 1 7 5 1 7 5 1 7 0 1 7 0 1 1 6 5 1 1 6 5 2 1 6 0 1 6 0 1 5 5 1 5 5 1 5 0 1 5 0 06/03/10 ATILF Nancy Daniel Hirst

  38. Automatic Momel ● Hirst & Espesser 1993 Asymmetric quadratic modal regression • Modal • Quadratic • Asymmetric 06/03/10 ATILF Nancy Daniel Hirst

  39. Mean and Mode mode mean 06/03/10 ATILF Nancy Daniel Hirst

  40. Mean and Mode • Mean value minimising sum of squares of diferences from data • Mode value minimising number of cases more than ∆ from data Generalise to function • Linear regression function minimising sum of squares of diferences from data • Modal regression function minimising number of cases more than ∆ from data 06/03/10 ATILF Nancy Daniel Hirst

  41. Asymmetric regression • no values more than Δ above the function • Minimise number of values more than Δ below it • Here, function is f = at 2 + bt + c 06/03/10 ATILF Nancy Daniel Hirst

  42. Momel ● Hirst & Espesser 1993 06/03/10 ATILF Nancy Daniel Hirst

  43. Evaluation of Momel ● Estelle Campione, 2001 06/03/10 ATILF Nancy Daniel Hirst

  44. Improved algorithm 06/03/10 ATILF Nancy Daniel Hirst

  45. Improved algorithm 06/03/10 ATILF Nancy Daniel Hirst

  46. Momel – theory neutral? ● Theory friendly ● used for – Fujisaki model (Mixdorff) – ToBI (Maghbouleh, Wightman & Cambell, Cho (K-ToBI) – INTSINT 06/03/10 ATILF Nancy Daniel Hirst

  47. INTSINT ● An INternational Transcription System for INTonation ● Based on minimal pitch contrasts in descriptions of intonation patterns ● Used in Hirst & Di Cristo 1998 for 9 different languages – British English, Spanish, European Portuguese, Brazilian Portuguese, French, Romanian, Russian, Moroccan Arabic and Japanese ● Extension for duration and rhythm 06/03/10 ATILF Nancy Daniel Hirst

  48. Basic INTSINT ● Absolute tones T(op) M(id) B(ottom) ● Relative tones H(igher) S(ame) L(ower) ● Iterative relative tones U(pstepped) D(ownstepped) 06/03/10 ATILF Nancy Daniel Hirst

  49. 2 speaker parameters: Hirst 2005 T H S H U k D U S e M y S D L L B range 06/03/10 ATILF Nancy Daniel Hirst

  50. downdrift 2 0 0 1 5 0 1 0 0 5 0 0 M T L H L H L H B 06/03/10 ATILF Nancy Daniel Hirst

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Speech Processing 15-492/18-492 Speech Synthesis Pronunciation Letter to Sound rules Speech

Speech Processing 15-492/18-492 Speech Synthesis Pronunciation Letter to Sound rules Speech

Speech Processing 15-492/18-492 Speech Synthesis Pronunciation Letter to Sound rules Speech Synthesis Linguistic Analysis Linguistic Analysis Pronunciations Pronunciations Prosody Prosody Part of Speech Tagging

382 views • 21 slides
Speech Processing 15- -492/18 492/18- -492 492 Speech Processing 15 Speech Synthesis Prosody

Speech Processing 15- -492/18 492/18- -492 492 Speech Processing 15 Speech Synthesis Prosody

Speech Processing 15- -492/18 492/18- -492 492 Speech Processing 15 Speech Synthesis Prosody Speech Synthesis Speech Synthesis Linguistic Analysis Linguistic Analysis Pronunciations Pronunciations Prosody Prosody

419 views • 24 slides
Speech Processing 11-492/18-492 Speech Synthesis Overview Text processing Speech Synthesis

Speech Processing 11-492/18-492 Speech Synthesis Overview Text processing Speech Synthesis

Speech Processing 11-492/18-492 Speech Synthesis Overview Text processing Speech Synthesis From text to speech Text Analysis Strings of characters to words Linguistic Analysis From words to pronunciations and prosody

489 views • 25 slides
CHiVE Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical

CHiVE Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical

CHiVE Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network Vincent Wan, Chun-an Chan, Tom Kenter, Jakub Vit, Rob Clark Modelling intonation in prosody A conditional variational

422 views • 6 slides
Sub-Project I Prosody, Tones and Text-To-Speech Synthesis Sin-Horng Chen (PI), Chiu-yu Tseng

Sub-Project I Prosody, Tones and Text-To-Speech Synthesis Sin-Horng Chen (PI), Chiu-yu Tseng

Sub-Project I Prosody, Tones and Text-To-Speech Synthesis Sin-Horng Chen (PI), Chiu-yu Tseng (Co-PI), Yih-Ru Wang (Co-PI), Yuan-Fu Liao (Co-PI), Lin-shan Lee, Hsin-min Wang Outline Members Theme of Sub-project I Research Roadmap

276 views • 23 slides
Cyborg speech: Deep multilingual speech synthesis for generating segmental foreign accent with

Cyborg speech: Deep multilingual speech synthesis for generating segmental foreign accent with

Cyborg speech: Deep multilingual speech synthesis for generating segmental foreign accent with natural prosody Gustav Eje Henter 1 , Jaime Lorenzo-Trueba 1 , Xin Wang 1 , Mariko Kondo 2 , Junichi Yamagishi 1,3 gustav@nii.ac.jp, jyamagis@nii.ac.jp

1.22k views • 58 slides
Modeling Prosody Pattern of Chinese Expressive Speech Application in Personalized Speech

Modeling Prosody Pattern of Chinese Expressive Speech Application in Personalized Speech

Modeling Prosody Pattern of Chinese Expressive Speech Application in Personalized Speech Conversion Zhang Zhang, Wu Zhiyong, Jia Jia, Cai Lianhong Outline Background Introduction Modeling prosody pattern of expressive speech

785 views • 21 slides
Speech Processing 15-492/18-492 Speech Synthesis Waveform generation 2 Speech Synthesis Text

Speech Processing 15-492/18-492 Speech Synthesis Waveform generation 2 Speech Synthesis Text

Speech Processing 15-492/18-492 Speech Synthesis Waveform generation 2 Speech Synthesis Text Analysis Text Analysis Chunking, tokenization, token expansion Chunking, tokenization, token expansion Linguistic Analysis

644 views • 29 slides
6-Text To Speech (TTS) Speech Synthesis Speech Synthesis Concept Speech Naturalness Phone

6-Text To Speech (TTS) Speech Synthesis Speech Synthesis Concept Speech Naturalness Phone

6-Text To Speech (TTS) Speech Synthesis Speech Synthesis Concept Speech Naturalness Phone Sequence To Speech Articulatory Approaches Concatenative Approaches HMM-based Approaches Rule-Based Approaches 1 Speech Synthesis Concept

749 views • 57 slides
Generating segment-level foreign-accented synthetic speech with natural speech prosody Gustav Eje

Generating segment-level foreign-accented synthetic speech with natural speech prosody Gustav Eje

Generating segment-level foreign-accented synthetic speech with natural speech prosody Gustav Eje HENTER, Jaime LORENZO-TRUEBA, Xin WANG, Mariko KONDO, Junichi YAMAGISHI gustav@nii.ac.jp Digital Content and Media Sciences Research Division,

751 views • 54 slides
EE E6820: Speech & Audio Processing & Recognition Lecture 6: Music analysis and synthesis

EE E6820: Speech & Audio Processing & Recognition Lecture 6: Music analysis and synthesis

EE E6820: Speech & Audio Processing & Recognition Lecture 6: Music analysis and synthesis 1 Music and nonspeech 2 Music synthesis techniques 3 Sinewave synthesis 4 Music analysis 5 Transcription Dan Ellis

745 views • 38 slides
for Speech Synthesis and Sensor Data Augmentation Deep Generative Neural Network Speech Text

for Speech Synthesis and Sensor Data Augmentation Deep Generative Neural Network Speech Text

Deep Generative Modeling for Speech Synthesis and Sensor Data Augmentation Deep Generative Neural Network Speech Text Praveen Narayanan Ford Motor Company PROJECT DESCRIPTION Use of DNNs increasingly prevalent as a solution for many data

893 views • 53 slides
Lecture 17: LPC speech synthesis and autocorrelation- based pitch tracking ECE 401, Signal and

Lecture 17: LPC speech synthesis and autocorrelation- based pitch tracking ECE 401, Signal and

Lecture 17: LPC speech synthesis and autocorrelation- based pitch tracking ECE 401, Signal and Image Analysis November 5, 2020 Outline The LPC-10 speech synthesis model The LPC-10 excitation model: white noise, pulse train Linear

517 views • 36 slides
Speech Processing 15-492/18-492 Speech Synthesis Overview Text processing Speech Synthesis

Speech Processing 15-492/18-492 Speech Synthesis Overview Text processing Speech Synthesis

Speech Processing 15-492/18-492 Speech Synthesis Overview Text processing Speech Synthesis From text to speech From text to speech Text Analysis Text Analysis Strings of characters to words Strings of characters to words

665 views • 25 slides
Text-to-Speech Synthesis Bernd Mbius Language Science and Technology Saarland University

Text-to-Speech Synthesis Bernd Mbius Language Science and Technology Saarland University

Text-to-Speech Synthesis Bernd Mbius Language Science and Technology Saarland University Lecture 4 June 4, 2020 Diphone Synthesis B Mbius Concatenative synthesis l Concatenative synthesis: general procedure Data-based,

611 views • 27 slides
The Future of Prosody Its about Time Dafydd Gibbon Bielefeld University Jinan University

The Future of Prosody Its about Time Dafydd Gibbon Bielefeld University Jinan University

The Future of Prosody Its about Time Dafydd Gibbon Bielefeld University Jinan University Speech Prosody 9, Pozna, 13 June 2018 TIME The Future of Prosody relevant topics? Ethics of research responsibility for the use of our

1.28k views • 91 slides
Linear Solvers for Singularly Perturbed Problems Numerical Analysis for Singularly Perturbed

Linear Solvers for Singularly Perturbed Problems Numerical Analysis for Singularly Perturbed

Linear Solvers for Singularly Perturbed Problems Numerical Analysis for Singularly Perturbed Problems (dedicated to the 60 th birthday of Martin Stynes) 16 th 18 th Nov, 2011 Niall Madden, NUI Galway Joint work with Scott MacLachlan, Tufts

661 views • 39 slides
LINGUISTIC RHYTHM 1 YU / LAMONT APRIL 05, 2018 2 KEEPING THE BEAT IN LANGUAGE LINGUIST

LINGUISTIC RHYTHM 1 YU / LAMONT APRIL 05, 2018 2 KEEPING THE BEAT IN LANGUAGE LINGUIST

LINGUIST 197M, SPRING 2018. CLASS 10.2 LINGUISTIC RHYTHM 1 YU / LAMONT APRIL 05, 2018 2 KEEPING THE BEAT IN LANGUAGE LINGUIST 197M, SPRING 2018. CLASS 10.2 3 METRICAL THEORY AND METRICS Bruce Hayes Russ Schuh

442 views • 25 slides
Mathematical Problems in Multivariate Public Key Cryptography Timothy Hodges University of

Mathematical Problems in Multivariate Public Key Cryptography Timothy Hodges University of

Mathematical Problems in Multivariate Public Key Cryptography Timothy Hodges University of Cincinnati January 15, 2015 Timothy Hodges (University of Cincinnati) Mathematical Problems in MPKC January 15, 2015 1 / 28 Overview Multivariate

1.05k views • 68 slides
Learning the Stress Patterns of the Worlds Languages Jeffrey Heinz heinz@udel.edu University

Learning the Stress Patterns of the Worlds Languages Jeffrey Heinz heinz@udel.edu University

Introduction Stress Patterns Formal Learning Theory Neighborhood-distinctness Learning Stress Patterns Discussion Learning the Stress Patterns of the Worlds Languages Jeffrey Heinz heinz@udel.edu University of Delaware The 321st

1.45k views • 93 slides
Artificial Intelligence George Konidaris gdk@cs.brown.edu Fall 2019 1410 Team Instructor :

Artificial Intelligence George Konidaris gdk@cs.brown.edu Fall 2019 1410 Team Instructor :

Artificial Intelligence George Konidaris gdk@cs.brown.edu Fall 2019 1410 Team Instructor : George Konidaris Hours : Wed 4-5pm, CIT 447 HTAs: Leon Lei and Aansh Shah TAs : Alex Liu Jesus Contreras Ariel

993 views • 66 slides
I use Blue Waters to simulate an ultracold inferno. Micheline Soley Micheline Soley, Harvard

I use Blue Waters to simulate an ultracold inferno. Micheline Soley Micheline Soley, Harvard

I use Blue Waters to simulate an ultracold inferno. Micheline Soley Micheline Soley, Harvard University Micheline Soley, Harvard University I use Blue Waters to simulate an ultracold inferno. 1 / 15 K K K K Rb Rb Rb Rb 14 K 4000 K

489 views • 15 slides
Random Forests COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Random

Random Forests COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Random

Random Forests COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Random Forests 1 / 10 Outline 1 Motivation 2 Bagging 3 Randomizing Split Dimension 4 Training 5 Inference 6 Out-of-Bag Statistical Risk Estimate COMPSCI 371D

241 views • 10 slides
Lecture 15 : Pairs of Discrete Random Variables 0/ 21 Today we start Chapter 5. The transition we

Lecture 15 : Pairs of Discrete Random Variables 0/ 21 Today we start Chapter 5. The transition we

Lecture 15 : Pairs of Discrete Random Variables 0/ 21 Today we start Chapter 5. The transition we are making is like going from one variable calculus to vector calculus. We should really think of vectors ( X , Y ) of random variables. So suppose

232 views • 22 slides

More recommend