Speech Processing 11-492/18-495 Speech Processing Current Topics - - PowerPoint PPT Presentation

Speech Processing 11-492/18-495

Speech Processing Current Topics and Future challenges Commercial and Research

Current and Future Current and Future

 What are the hot topics in Speech

What are the hot topics in Speech

 What currently works

What currently works

 What could work soon (5-10years)

What could work soon (5-10years)

 What are the industry hot topics

What are the industry hot topics

 What are the research challenges

What are the research challenges

Spoken Dialog: Now Spoken Dialog: Now

 Industry:

Industry:

 Location based querying

Location based querying

 On phone: Apple (Siri)

On phone: Apple (Siri)

 In home: Amazon (Echo)

In home: Amazon (Echo)

 Smartphones, Tablets:

Smartphones, Tablets:

 (Owners have money)

(Owners have money) – IoT deployment IoT deployment

 How do you make money out of this …

How do you make money out of this …

Spoken Dialog: Now Spoken Dialog: Now

 Research

Research

 Error recovery

Error recovery

 Adaptive systems

Adaptive systems

 Rapid deployment

Rapid deployment

 Learning dialog structure from data

Learning dialog structure from data

 Non-task oriented dialog

Non-task oriented dialog

ASR: Now ASR: Now

 Industry

Industry

 Adapting cloud ASR per app.

Adapting cloud ASR per app.

 Broadcast news transcription

Broadcast news transcription

 Robust speech recognition:

Robust speech recognition:

 In car, outside, in noisy office, far field

In car, outside, in noisy office, far field

 LM adaptation from other sources

LM adaptation from other sources

 Using click through and search queries

Using click through and search queries

 Pronunciation variants (“wrong” ones too)

Pronunciation variants (“wrong” ones too)

 Medical transcription

Medical transcription

ASR: Now ASR: Now

 Research:

Research:

 Discriminative training

Discriminative training

 Acoustic parameter projections to discriminate

Acoustic parameter projections to discriminate between the correct answers and competitors between the correct answers and competitors

 Robust recognition

Robust recognition

 Far field microphones

Far field microphones

 Blind source separation

Blind source separation

 Out of vocabulary words

Out of vocabulary words

 Unsupervised training

Unsupervised training

 Deep Learning (Neural Nets)

Deep Learning (Neural Nets)

 Zero-resource ASR

Zero-resource ASR

TTS: Now TTS: Now

 Industry

Industry

 Building custom voices (and your voice)

Building custom voices (and your voice)

 Multilingual on small devices

Multilingual on small devices

 E.g. for GPS Navigation over Europe

E.g. for GPS Navigation over Europe

 Easy methods to build new languages

Easy methods to build new languages

 Conversational Speech

Conversational Speech

TTS: Now TTS: Now

 Research

Research

 Improving neural synthesis

Improving neural synthesis

• Quality/Resources/Runtime computation

Quality/Resources/Runtime computation

 Rapid support in new languages

Rapid support in new languages

 Emotional speech synthesis

Emotional speech synthesis

 Automatic building of voices from data

Automatic building of voices from data

 Without any human intervention

Without any human intervention

 Languages without Orthography

Languages without Orthography

 Synthesis beyond the sentence

Synthesis beyond the sentence

 Synthesis with more text analysis

Synthesis with more text analysis

Speech to Speech Translation Speech to Speech Translation

 Industry

Industry

 One way systems, domain limited systems

One way systems, domain limited systems

 Simple targeted cell phone systems

Simple targeted cell phone systems

 Youtube/Broadcast translation

Youtube/Broadcast translation

 Skype translation

Skype translation

 Research

Research

 Two way systems, large domains

Two way systems, large domains

 One way lecture/broadcast news

One way lecture/broadcast news

VC and SID: Now VC and SID: Now

 Voice conversion

Voice conversion

 Cross Lingual Voice Conversion

Cross Lingual Voice Conversion

 Emotion/style conversion

Emotion/style conversion

 Conversion without training data

Conversion without training data

 Speaker ID

Speaker ID

 Accuracy on large data sets (> 1000 speakers)

Accuracy on large data sets (> 1000 speakers)

 Cross channel/language ID

Cross channel/language ID

 More information in ID (prosody, vocab)

More information in ID (prosody, vocab)

CALL: Now CALL: Now

 Industry

Industry

 Pronunciation training

Pronunciation training

 Scenario practicing

Scenario practicing

 Research

Research

 Game based tools

Game based tools

 Measuring educational contribution

Measuring educational contribution

Speech Processing Future Speech Processing Future

 Hard challenges (PhD topics and beyond)

Hard challenges (PhD topics and beyond)

 All on the research side

All on the research side

 But maybe in Research Labs

But maybe in Research Labs

Speech Reco without Speech Speech Reco without Speech

 Using other modalities

Using other modalities

 Lip movement, muscle movement

Lip movement, muscle movement

 Silent speech

Silent speech

 No generated audio

No generated audio

 Just think about the words

Just think about the words

 Gesture recognition

Gesture recognition

 Brain Computer Interfaces

Brain Computer Interfaces

 ASR without text

ASR without text

 Find “….” in all this audio

Find “….” in all this audio

Beyond the Words Beyond the Words

 Recognition of more than words

Recognition of more than words

 Intent, style, emotion

Intent, style, emotion

 Human-Machine

Human-Machine

 Frustration, confidence, agreement

Frustration, confidence, agreement

 Human-Human

Human-Human

 Rapport, relationships, persuasion

Rapport, relationships, persuasion

 Truth and lies

Truth and lies

 Sentiment

Sentiment

Conversational Systems Conversational Systems

 Participant in a meeting

Participant in a meeting

 True conversational speech

True conversational speech

 Appropriate non-word speech generation

Appropriate non-word speech generation

 Know when to speak, when to laugh, when to listen

Know when to speak, when to laugh, when to listen

 Appropriate timing conversation

Appropriate timing conversation

 Able to interrupt when having something to say

Able to interrupt when having something to say

 Have something to say

Have something to say

Summaries and Discussions Summaries and Discussions

 Describe a paper/movie/event

Describe a paper/movie/event

 Appropriate summary

Appropriate summary

 Allow questions

Allow questions

 Know when to use style/emotion

Know when to use style/emotion

 Not just speech<->text

Not just speech<->text

 Understand more of the text content

Understand more of the text content  Answer complex questions

Answer complex questions

 Engage user and discuss topic

Engage user and discuss topic