Topic: Spectrogram, Cepstrum and Mel-Frequency Analysis Kishore - - PowerPoint PPT Presentation

▶
topic spectrogram cepstrum and mel frequency analysis
SMART_READER_LITE
LIVE PREVIEW

Topic: Spectrogram, Cepstrum and Mel-Frequency Analysis Kishore - - PowerPoint PPT Presentation

Feb 17, 2023 424 likes •943 views

Speech Technology: A Practical Introduction Topic: Spectrogram, Cepstrum and Mel-Frequency Analysis Kishore Prahallad Email: skishore@cs.cmu.edu Carnegie Mellon University & International Institute of Information Technology Hyderabad 1

slide-1
SLIDE 1

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 1

Speech Technology: A Practical Introduction

Topic: Spectrogram, Cepstrum and Mel-Frequency Analysis

Kishore Prahallad Email: skishore@cs.cmu.edu Carnegie Mellon University & International Institute of Information Technology Hyderabad

slide-2
SLIDE 2

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 2

Topics

  • Spectrogram
  • Cepstrum
  • Mel-Frequency Analysis
  • Mel-Frequency Cepstral Coefficients
slide-3
SLIDE 3

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 3

Spectrogram

slide-4
SLIDE 4

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 4

Speech signal represented as a sequence of spectral vectors

FFT FFT FFT

Spectrum

slide-5
SLIDE 5

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 5

Speech signal represented as a sequence of spectral vectors

FFT

Spectrum

FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT

slide-6
SLIDE 6

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 6

Speech signal represented as a sequence of spectral vectors

FFT

Spectrum

FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT

Hz

Amp.

slide-7
SLIDE 7

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 7

Speech signal represented as a sequence of spectral vectors

FFT

Spectrum

FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT

Hz Amplitude Rotate it by 90 degrees

slide-8
SLIDE 8

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 8

Speech signal represented as a sequence of spectral vectors

FFT

Spectrum

FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT

Hz

  • MAP spectral amplitude to a grey level (0-

255) value. 0 represents black and 255 represents white.

  • Higher the amplitude, darker the

corresponding region. Amplitude

slide-9
SLIDE 9

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 9

Speech signal represented as a sequence of spectral vectors

FFT

Spectrum

FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT

Hz Time

slide-10
SLIDE 10

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 10

Speech signal represented as a sequence of spectral vectors

FFT

Spectrum

FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT

Hz Time Time Vs Frequency representation of a speech signal is referred to as spectrogram

slide-11
SLIDE 11

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 11

Some Real Spectrograms

Dark regions indicate peaks (formants) in the spectrum

slide-12
SLIDE 12

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 12

Why we are bothered about spectrograms

Phones and their properties are better observed in spectrogram

slide-13
SLIDE 13

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 13

Why we are bothered about spectrograms

Sounds can be identified much better by the Formants and by their transitions

slide-14
SLIDE 14

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 14

Why we are bothered about spectrograms

Sounds can be identified much better by the Formants and by their transitions Hidden Markov Models implicitly model these spectrograms to perform speech recognition

slide-15
SLIDE 15

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 15

Usefulness of Spectrogram

  • Time-Frequency representation of the speech signal
  • Spectrogram is a tool to study speech sounds (phones)
  • Phones and their properties are visually studied by phoneticians
  • Hidden Markov Models implicitly model spectrograms for speech to

text systems

  • Useful for evaluation of text to speech systems

– A high quality text to speech system should produce synthesized speech whose spectrograms should nearly match with the natural sentences.

slide-16
SLIDE 16

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 16

Cepstral Analysis

slide-17
SLIDE 17

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 17

A Sample Speech Spectrum

Frequency (Hz) dB

  • Peaks denote dominant frequency

components in the speech signal

  • Peaks are referred to as formants
  • Formants carry the identity of the sound
slide-18
SLIDE 18

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 18

What we want to Extract? – Spectral Envelope

  • Formants and a smooth curve connecting them
  • This Smooth curve is referred to as spectral envelope

Frequency (Hz) dB

slide-19
SLIDE 19

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 19

Spectral Envelope

Spectral Envelope Spectrum Spectral details

slide-20
SLIDE 20

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 20

Spectral Envelope

Spectral Envelope Spectrum Spectral details log X[k] log H[k] log E[k]

slide-21
SLIDE 21

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 21

Spectral Envelope

Spectral Envelope Spectrum Spectral details log X[k] log H[k] log E[k] log X[k] = log H[k] + log E[k]

  • 1. Our goal: We want to

separate spectral envelope and spectral details from the spectrum.

  • 2. i.e Given log X[k],
  • btain log H[k] and log

E[k], such that log X[k] = log H[k] + log E[k]

slide-22
SLIDE 22

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 22

How to achieve this separation ?

slide-23
SLIDE 23

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 23

Play a Mathematical Trick

Spectral Envelope Spectral details Spectrum

  • Trick: Take FFT of

the spectrum!!

  • An FFT on spectrum

referred to as Inverse FFT (IFFT).

  • Note: We are dealing

with spectrum in log domain (part of the trick)

  • IFFT of log spectrum

would represent the signal in pseudo- frequency axis

slide-24
SLIDE 24

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 24

Play a Mathematical Trick

Spectral Envelope A pseudo-frequency axis Spectral details Spectrum

slide-25
SLIDE 25

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 25

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis Low Freq. region High Freq. region

slide-26
SLIDE 26

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 26

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis Low Freq. region High Freq. region IFFT

slide-27
SLIDE 27

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 27

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis Low Freq. region High Freq. region IFFT Treat this as a sine wave with 4 cycles per sec.

slide-28
SLIDE 28

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 28

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis Low Freq. region High Freq. region IFFT Treat this as a sine wave with 4 cycles per sec. Gives a peak at 4 Hz in frequency axis

slide-29
SLIDE 29

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 29

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis Low Freq. region High Freq. region IFFT Treat this as a sine wave with 4 cycles per sec. Gives a peak at 4 Hz in frequency axis

slide-30
SLIDE 30

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 30

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis Low Freq. region High Freq. region IFFT

slide-31
SLIDE 31

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 31

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis Low Freq. region High Freq. region IFFT Treat this as a sine wave with 100 cycles per sec. Gives a peak at 100 Hz in frequency axis

slide-32
SLIDE 32

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 32

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis Low Freq. region High Freq. region IFFT IFFT

slide-33
SLIDE 33

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 33

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis

slide-34
SLIDE 34

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 34

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis IFFT log X[k] = log H[k] + log E[k] log H[k] log E[k]

slide-35
SLIDE 35

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 35

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis IFFT log X[k] = log H[k] + log E[k] log H[k] log E[k] x[k] = h[k] + e[k]

slide-36
SLIDE 36

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 36

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis IFFT log X[k] = log H[k] + log E[k] log H[k] log E[k] x[k] = h[k] + e[k] In practice all you have access to only log X[k] and hence you can obtain x[k]

slide-37
SLIDE 37

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 37

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis IFFT log X[k] = log H[k] + log E[k] log H[k] log E[k] x[k] = h[k] + e[k] If you know x[k] Filter the low frequency region to get h[k]

slide-38
SLIDE 38

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 38

Play a Mathematical Trick

Spectral Envelope Spectrum Spectral details A pseudo-frequency axis IFFT log X[k] = log H[k] + log E[k] log H[k] log E[k] x[k] = h[k] + e[k]

  • x[k] is referred to as Cepstrum
  • h[k] is obtained by considering

the low frequency region of x[k].

  • h[k] represents the spectral

envelope and is widely used as feature for speech recognition

slide-39
SLIDE 39

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 39

Cepstral Analysis

] [ ] [ ] [ sides both

  • n

FFT inverse Taking || ] [ || log || ] [ || log || ] [ || log sides both

  • n

Log Take magnitude denotes || . || || ] [ || || ] [ || || ] [ || ] [ ] [ ] [ k e k h k x k E k H k X k E k H k X k E k H k X + = + = − = =

slide-40
SLIDE 40

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 40

Mel-Frequency Analysis

slide-41
SLIDE 41

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 41

Review: What we did

  • We captured spectral envelope (curve

connecting all formants)

  • BUT: Perceptual experiments say human ear

concentrates on certain regions rather than using whole of the spectral envelope….

Frequency (Hz) dB

slide-42
SLIDE 42

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 42

Mel-Frequency Analysis

  • Mel-Frequency analysis of speech is

based on human perception experiments

  • It is observed that human ear acts as filter

– It concentrates on only certain frequency components

  • These filters are non-uniformly spaced on

the frequency axis

– More filters in the low frequency regions – Less no. of filters in high frequency regions

slide-43
SLIDE 43

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 43

Mel-Frequency Filters

slide-44
SLIDE 44

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 44

Mel-Frequency Filters

More no. of filters in low

  • freq. region

Lesser no. of filters in high freq. region

slide-45
SLIDE 45

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 45

Mel-Frequency Cepstral Coefficients (MFCC)

  • Spectrum Mel-Filters Mel-Spectrum
  • Say log X[k] = log (Mel-Spectrum)
  • NOW perform Cepstral analysis on log X[k]

– log X[k] = log H[k] + log E[k] – Taking IFFT – x[k] = h[k] + e[k]

  • Cepstral coefficients h[k] obtained for Mel-

spectrum are referred to as Mel-Frequency Cepstral Coefficients often denoted by *MFCC*

slide-46
SLIDE 46

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 46

Speech signal represented as a sequence of spectral vectors

FFT

Spectrum

FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT

Mel-Filters Cepstral Analy.

slide-47
SLIDE 47

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 47

Speech signal represented as a sequence of CEPSTRAL vectors

FFT

Spectrum

FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT FFT

Cepstral Vectors

slide-48
SLIDE 48

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 48

Why we are going to use MFCC

  • Speech synthesis

– Used for joining two speech segments S1 and S2 – Represent S1 as a sequence of MFCC – Represent S2 as a sequence of MFCC – Join at the point where MFCCs of S1 and S2 have minimal Euclidean distance

  • Used in speech recognition

– MFCC are mostly used features in state-of-art speech recognition system

slide-49
SLIDE 49

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 49

Summary: Process of Feature Extraction

  • Speech is analyzed over short analysis window
  • For each short analysis window a spectrum is obtained

using FFT

  • Spectrum is passed through Mel-Filters to obtain Mel-

Spectrum

  • Cepstral analysis is performed on Mel-Spectrum to
  • btain Mel-Frequency Cepstral Coefficients
  • Thus speech is represented as a sequence of Cepstral

vectors

  • It is these Cepstral vectors which are given to pattern

classifiers for speech recognition purpose

slide-50
SLIDE 50

Speech Technology - Kishore Prahallad (skishore@cs.cmu.edu) 50

Additional Reading

  • Chapter 6

– Pg: 273 – 281 – Pg: 304 – 311 – Pg: 314 - 316