CTP431- Music and Audio Computing Digital Audio Effects Graduate - - PowerPoint PPT Presentation

CTP431- Music and Audio Computing Digital Audio Effects

Graduate School of Culture Technology KAIST Juhan Nam

1

Introduction

§ Amplitude

– Gain, fade in/out, automation curve, compressor

§ Timbre

– Filters, EQ, distortion, modulation, flanger, vocoder

§ Spatial effect

– Delay, Reverberation, panning, binaural (HRTF)

§ Pitch

– Pitch shifting (e.g. auto-tune), Transpose

§ Time stretching

– Timing change, Tempo adjustment

2

Source: http://www.uaudio.com/uad/downloads Source: https://www.izotope.com/en/products/repair-and-edit/rx-post-production-suite.html

Digital System

§ Take the input signal 𝑦 𝑜 as a sequence of numbers and returns the

• utput signal 𝑧 𝑜 as another sequence of numbers

§ We are particularly interested in linear systems that are composed of the following operations

– Multiplication: 𝑧 𝑜 = 𝑐& ' 𝑦 𝑜 – Delaying: 𝑧 𝑜 = 𝑦 𝑜 − 1 – Summation: 𝑧 𝑜 = 𝑦 𝑜 + 𝑦 𝑜 − 1

3

Input Output Digital System 𝑦 𝑜 𝑧 𝑜

Linear Time-Invariant (LTI) System

§ Linearity

– Homogeneity: if 𝑦 𝑜 → 𝑧 𝑜 , then a ' 𝑦 𝑜 → a ' 𝑧 𝑜 – Superposition: if 𝑦- 𝑜 → 𝑧- 𝑜 and 𝑦. 𝑜 → 𝑧. (n), then 𝑦- 𝑜 + 𝑦. 𝑜 → 𝑧- 𝑜 + 𝑧. 𝑜

§ Time-Invariance

– If 𝑦 𝑜 → 𝑧 𝑜 , then 𝑦 𝑜 − 𝑂 → 𝑧 𝑜 − 𝑂 for any 𝑂 – This means that the system does not change its behavior over time

4

Input Output Digital System 𝑦 𝑜 𝑧 𝑜

LTI System

§ LTI systems in frequency domain

– No new sinusoidal components are introduced – Only existing sinusoids components changes in amplitude and phase.

§ Examples of non-LTI systems

– Clipping – Distortion – Aliasing – Modulation

5

LTI Digital Filters

§ A LTI digital filters performs a combination of the three operations

– 𝑧 𝑜 = 𝑐& ' 𝑦 𝑜 + 𝑐- ' 𝑦 𝑜 − 1 + 𝑐. ' 𝑦 𝑜 − 2 + ⋯ + 𝑐2 ' 𝑦 𝑜 − 𝑁

§ This is a general form of Finite Impulse Response (FIR) filter

6

Two Ways of Defining LTI Systems

§ By the relation between input 𝑦 𝑜 and output 𝑧 𝑜

– Difference equation – Signal flow graph

§ By the impulse response of the system

– Measure it by using a unit impulse as input – Convolution operation

7

The Simplest Lowpass Filter

8

𝑧 𝑜 = 𝑦 𝑜 + 𝑦(𝑜 − 1) 𝑨7- 𝑦 𝑜 𝑧 𝑜

+

§ Difference equation § Signal flow graph

“Delay Operator”

The Simplest Lowpass Filter: Sine-Wave Analysis

§ Measure the amplitude and phase changes given a sinusoidal signal input

9

The Simplest Lowpass Filter: Frequency Response

§ Plot the amplitude and phase change over different frequency

– The frequency sweeps from 0 to the Nyquist rate

10

The Simplest Lowpass Filter: Frequency Response

§ Mathematical approach

– Use complex sinusoid as input: 𝑦 𝑜 = 𝑓9:; – Then, the output is: 𝑧 𝑜 = 𝑦 𝑜 + 𝑦 𝑜 − 1 = 𝑓9:; + 𝑓9:(;7-) = 1 + 𝑓79: ' 𝑓9:; = 1 + 𝑓79: ' 𝑦(𝑜) – Frequency response: 𝐼 𝜕 = 1 + 𝑓79: = 𝑓9>

? + 𝑓79> ? 𝑓79> ? = 2cos

(

: .)𝑓79>

?

– Amplitude response: 𝐼(𝜕) = 2 cos

: .

– Phase response: ∠𝐼 𝜕 = −

: .

11

The Simplest Highpass Filter

§ Difference equation: 𝑧 𝑜 = 𝑦 𝑜 − 𝑦(𝑜 − 1) § Frequency response

12

Impulse Response

§ The filter output when the input is a unit impulse

– 𝑦 𝑜 = 𝜀 𝑜 = 1, 0, 0, 0, … → 𝑧 𝑜 = ℎ(𝑜)

§ Characterizes the digital system as a sequence of numbers

– A system is represented just like audio samples!

13

Input Output Filter 𝑦 𝑜 = 𝜀 𝑜 𝑧 𝑜 = ℎ(𝑜) ℎ 𝑜

Examples: Impulse Response

§ The simplest lowpass filter

– h 𝑜 = 1, 1

§ The simplest highpass filter

– h 𝑜 = 1, −1

§ Moving-average filter (order=5)

– h 𝑜 =

• J ,
• J ,
• J ,
• J ,
• J

§ General FIR Filter

– h 𝑜 = 𝑐&, 𝑐-, 𝑐., … , 𝑐2 à A finite length of impulse response

14

§ The output of LTI digital filters is represented by convolution operation between 𝑦 𝑜 and ℎ 𝑜 § Deriving convolution

– The input can be represented as a time-ordered set of weighted impulses

• 𝑦 𝑜 = 𝑦&, 𝑦-, 𝑦., … , 𝑦2 = 𝑦& ' 𝜀 𝑜 + 𝑦- ' 𝜀 𝑜 − 1 + 𝑦. ' 𝜀 𝑜 − 2 + ⋯ + 𝑦2 ' 𝜀 𝑜 − 𝑁

– By the linearity and time-invariance

• 𝑧 𝑜 = 𝑦& ' ℎ 𝑜 + 𝑦- ' ℎ 𝑜 − 1 + 𝑦. ' ℎ 𝑜 − 2 + ⋯ + 𝑦2 ' ℎ 𝑜 − 𝑁 = ∑

𝑦(𝑗) ' ℎ(𝑜 − 𝑗)

2 MN&

Convolution

15

𝑧 𝑜 = 𝑦 𝑜 ∗ ℎ 𝑜 = P 𝑦(𝑗) ' ℎ(𝑜 − 𝑗)

2 MN&

Convolution In Practice

§ The practical expression of convolution

– This represents input 𝑦 𝑜 as a streaming data to the filter ℎ 𝑜

§ The length of convolution output

– If the length of 𝑦 𝑜 is M and the length of ℎ 𝑜 is N, the length of 𝑧 𝑜 is M+N-1

16

𝑧 𝑜 = 𝑦 𝑜 ∗ ℎ 𝑜 = P 𝑦 𝑗 ' ℎ 𝑜 − 𝑗

2 MN&

= P ℎ(𝑗) ' 𝑦(𝑜 − 𝑗)

2 MN&

Demo: Convolution

17

Feedback Filter

§ LTI digital filters allow to use the past outputs as input

– Past outputs: 𝑧 𝑜 − 1 , 𝑧 𝑜 − 2 , … , 𝑧 𝑜 − 𝑂

§ The whole system can be represented as

– 𝑧 𝑜 = 𝑐& ' 𝑦 𝑜 + 𝑏- ' 𝑧 𝑜 − 1 + 𝑏. ' 𝑧 𝑜 − 2 + ⋯ + 𝑏R ' 𝑧 𝑜 − 𝑂 – This is a general form of Infinite Impulse Response (IIR) filter

18

Input Output Filter

+

𝑦 𝑜 𝑧 𝑜 Delay

§ Difference equation § Signal flow graph

– When 𝑏 is slightly less than 1, it is called “Leaky Integrator”

A Simple Feedback Lowpass Filter

19

𝑧 𝑜 = 𝑦 𝑜 + 𝑏 ' 𝑧(𝑜 − 1) 𝑨7- 𝑦 𝑜 𝑧 𝑜

+

𝑏

A Simple Feedback Lowpass Filter: Impulse Response

§ Impulse response

– 𝑧 0 = 𝑦 0 = 1 – 𝑧 1 = 𝑦 1 + 𝑏 ' 𝑧 0 = 𝑏 – 𝑧 2 = 𝑦 2 + 𝑏 ' 𝑧 1 = 𝑏. – … – 𝑧 𝑜 = 𝑦 𝑜 + 𝑏 ' 𝑧 𝑜 − 1 = 𝑏;

§ Stability!

– If 𝑏 < 1, the filter output converges (stable) – If 𝑏 = 1, the filter output oscillates (critical) – If 𝑏 > 1, the filter output diverges (unstable)

20

A Simple Feedback Lowpass Filter: Frequency Response

§ More dramatic change than the simplest lowpass filter (FIR)

– Phase response is not linear

21

𝑧 𝑜 = 𝑦 𝑜 + 0.9 ' 𝑧(𝑜 − 1)

Reson Filter

22

§ Difference equation § Signal flow graph 𝑧 𝑜 = 𝑦 𝑜 + 2𝑠 ' cos𝜄 ' 𝑧 𝑜 − 1 − 𝑠. ' 𝑧 𝑜 − 2 𝑨7- 𝑦 𝑜 𝑧 𝑜

+

2𝑠 ' cos𝜄

+

𝑨7-

−𝑠.

+

Reson Filter: Frequency Response

§ Generate resonance at a particular frequency

– Control the peak height by 𝑠 and the peak frequency by 𝜄

23

For stability: 𝑠 < 1

General Filter Form

§ The general form of digital Filters

– 𝑧 𝑜 = 𝑐& ' 𝑦 𝑜 + 𝑐- ' 𝑦 𝑜 − 1 + 𝑐. ' 𝑦 𝑜 − 2 + … + 𝑐R ' 𝑦 𝑜 − 𝑁 +𝑏- ' 𝑧 𝑜 − 1 + 𝑏. ' 𝑧 𝑜 − 2 + ⋯ + 𝑏R ' 𝑧 𝑜 − 𝑂

24

Z-1 x(n)

+

Z-1 Z-1 . . . b1 b2 bM b0 x(n-1) x(n-2) x(n-M) Z-1 y(n) Z-1 Z-1 . . .

• a1
• a2
• aN

y(n-1) y(n-2) y(n-N)

Frequency Response

§ Sine-wave Analysis

– 𝑦 𝑜 = 𝑓9:; à 𝑦 𝑜 − 𝑛 = 𝑓9:(;7Z) = 𝑓79:Z𝑦 𝑜 for any 𝑛 – Let’s assume that 𝑧 𝑜 = 𝐻 𝜕 𝑓9(:;\] : ) à 𝑧 𝑜 − 𝑛 = 𝑓79:Z𝑧 𝑜 for any 𝑛

§ Putting this into the different equation

25

𝑧 𝑜 = 𝑐& + 𝑐- ' 𝑓79: + 𝑐. ' 𝑓79.: + … + 𝑐2 ' 𝑓79:2 1 + 𝑏- ' 𝑓79: + 𝑏. ' 𝑓79.: + … + 𝑏R ' 𝑓79:R 𝑦(𝑜) 𝑧 𝑜 = 𝑐& ' 𝑦 𝑜 + 𝑐- ' 𝑓79: ' 𝑦 𝑜 + 𝑐. ' 𝑓79.: ' 𝑦 𝑜 + … + 𝑐2 ' 𝑓79:2 ' 𝑦 𝑜 +𝑏- ' 𝑓79: ' 𝑧 𝑜 + 𝑏. ' 𝑓79.: ' 𝑧 𝑜 + ⋯ + 𝑏R ' 𝑓79:R ' 𝑧 𝑜

𝐼(𝜕) = 𝑐& + 𝑐- ' 𝑓79: + 𝑐. ' 𝑓79.: + … + 𝑐2 ' 𝑓79:2 1 + 𝑏- ' 𝑓79: + 𝑏. ' 𝑓79.: + … + 𝑏R ' 𝑓79:R

𝐼(𝜕) : frequency response 𝐻 𝜕 = 𝐼(𝜕) : amplitude response 𝜄 𝜕 = ∠𝐼(𝜕) : phase response

Z-Transform

§ 𝑎-transform

– Define z to be a variable in complex plane: we call it z-plane – When z = ejω (on unit circle), the frequency response is a particular case of the following form – We call this 𝑨-transform or the transfer function of the filter – z-1 corresponds to one sample delay: delay operator or delay element – Filters are often expressed as 𝑨-transform: polynomial of 𝑨7-

26

𝐼 𝑨 = 𝐶(𝑨) 𝐵(𝑨) = 𝑐& + 𝑐- ' 𝑨7- + 𝑐. ' 𝑨7. + … + 𝑐2 ' 𝑨72 1 + 𝑏- ' 𝑨7- + 𝑏. ' 𝑨7. + … + 𝑏R ' 𝑨7R

Practical Filters

§ One-pole one-zero filters

– Leaky integrator – Moving average – DC-removal filters – Bass / treble shelving filter

§ Biquad filters

– Reson filter – Band-pass / notch filters – Equalizer: a set of biquad filters

§ Any high-order filter can be factored into a combination of one-pole one- zero filters or bi-quad filters!

27

H(z) = b0 + b

1z−1

a0 + a1z−1 H(z) = b0 + b

1z−1 + b2z−2

a0 + a1z−1 + a2z−2

Filters

§ Adjust the level of a certain frequency band

– Lowpass – Highpass – Bandpass – Notch – Resonant Filter – Equalizer

§ Parameters

– Cut-off/Center Frequency – Q: sharpness/resonance

28

Low-pass Filter

§ Transfer Function

– fc : cut-off frequency, Q: resonance

29

H(z) = (1−cosΘ 2 ) 1+ 2z−1 +1z−2 (1+α)− 2cosΘz−1 +(1−α)z−2 α = sinΘ 2Q

10

2

10

3

10

4

−30 −20 −10 10 20 30 f=400 f=1000 f=3000 f=8000 Lowpass Filters freqeuncy(log10) Gain(dB) 10

2

10

3

10

4

−30 −20 −10 10 20 30 Q =0.5 Q =1 Q =2 Q =4 Lowpass Filters freqeuncy(log10) Gain(dB)

Θ = 2π fc / fs

High-pass Filter

§ Transfer Function

30

H(z) = (1+cosΘ 2 ) 1− 2z−1 +1z−2 (1+α)− 2cosΘz−1 +(1−α)z−2 α = sinΘ 2Q Θ = 2π fc / fs

10

2

10

3

10

4

−30 −20 −10 10 20 30 f=400 f=1000 f=3000 f=8000 Highpass Filters freqeuncy(log10) Gain(dB) 10

2

10

3

10

4

−30 −20 −10 10 20 30 Q =0.5 Q =1 Q =2 Q =4 Highpass Filters freqeuncy(log10) Gain(dB)

Band-pass filter

§ Transfer Function

31

H(z) = (sinΘ 2 ) 1− z−2 (1+α)− 2cosΘz−1 +(1−α)z−2

10

2

10

3

10

4

−30 −20 −10 10 20 30 f=400 f=1000 f=3000 f=8000 Bandpass Filters freqeuncy(log10) Gain(dB) 10

2

10

3

10

4

−30 −20 −10 10 20 30 Q =0.5 Q =1 Q =2 Q =4 Bandpass Filters freqeuncy(log10) Gain(dB)

α = sinΘ 2Q Θ = 2π fc / fs

Notch filter

§ Transfer Function

32

H(z) = 1− 2cosΘz−1 + z−2 (1+α)− 2cosΘz−1 +(1−α)z−2 α = sinΘ 2Q Θ = 2π fc / fs

10

2

10

3

10

4

−30 −20 −10 10 20 30 f=400 f=1000 f=3000 f=8000 Notch Filters freqeuncy(log10) Gain(dB) 10

2

10

3

10

4

−30 −20 −10 10 20 30 Q =0.5 Q =1 Q =2 Q =4 Notch Filters freqeuncy(log10) Gain(dB)

10

2

10

3

10

4

−30 −20 −10 10 20 30 AdB=−12 AdB=−6 AdB=0 AdB=6 AdB=12 EQ freqeuncy(log10) Gain(dB) 10

2

10

3

10

4

−30 −20 −10 10 20 30 AdB=−12 AdB=−6 AdB=0 AdB=6 AdB=12 EQ freqeuncy(log10) Gain(dB)

Equalizer

§ Transfer Function

33

H(z) = (1+α ⋅ A)− 2cosΘz−1 +(1+α ⋅ A)z−2 (1+α / A)− 2cosΘz−1 +(1−α / A)z−2 α = sinΘ 2Q Θ = 2π fc / fs

Q=1 Q=4

Delay-based Audio Effects

§ Types of delay-based audio effect

– Delay – Chorus – Flanger – Reverberation

34

– Implemented by circular buffer: move read and write pointers instead of shift all samples in the delayline

Comb Filter

35

𝑧 𝑜 = 𝑦 𝑜 + 𝑕 ' 𝑦 𝑜 − 𝑁

𝑨72

𝑦 𝑜 𝑧 𝑜

+

FIR Comb Filter 𝑧 𝑜 = 𝑦 𝑜 + 𝑕 ' 𝑧(𝑜 − 𝑂)

𝑨7R

𝑦 𝑜 𝑧 𝑜

+

𝑕

IIR Comb FIlters

𝑕

Comb Filter: Frequency Response

§ ”Combs” become shaper in the feedback type

36

𝑧 𝑜 = 𝑦 𝑜 + 𝑦(𝑜 − 8) 𝑧 𝑜 = 𝑦 𝑜 + 0.9 ' 𝑧(𝑜 − 8)

Perception of Time Delay

§ The 30 Hz transition

– Given a repeated click sound (e.g. impulse train):

• If the rate is less than 30Hz, they are perceived as discrete events.
• As the rate is above 30 Hz, they are perceive as a tone

– Demo: http://auditoryneuroscience.com/?q=pitch/click_train

§ Feedback comb filter: 𝑧 𝑜 = 𝑦 𝑜 + 𝑏 ' 𝑧(𝑜 − 𝑂)

– If N <

d

e

f& (𝐺 h: sampling rate): models sound propagation and reflection with energy

loss on a string (Karplus-strong model) – If N <

d

e

f& (𝐺 h: sampling rate): generate a looped delay

37

Delay Effect

§ Generate repetitive loop delay

– Parameters

• Feedback gain
• Delay length

– Ping-pong delay: cross feedback between left and right channels in stereo – The delay length is often synchronized with music tempo

38

+

𝑦 𝑜

feedback

𝑧 𝑜

Dry

+

Wet

Delay Line

Chorus Effect

§ Gives the illusion of multiple voices playing in unison

– By summing detuned copies of the input – Low frequency oscillators (LFOs) are used to modulate the position of output tops

• This causes pitch-shift

39

LFOs

x(n) y(n)

Dry

+ +

Wet

Delay Line

Flanger Effect

§ Emulated by summing one static tap and variable tap in the delay line

– “Rocket sound” – Feed-forward comb filter where harmonic notches vary over frequency. – LFO is often synchronized with music tempo

40

x(n)

+

LFOs Static tap Variable tap

y(n)

+

Wet Dry

Delay Line

Reverberation

§ Natural acoustic phenomenon that occurs when sound sources are played in a room

– Thousands of echoes are generated as sound sources are reflected against wall, ceiling and floors – Reflected sounds are delayed, attenuated and low-pass filtered: high-frequency component decay faster – The patterns of myriads of echoes are determined by the volume and geometry of room and materials on the surfaces

41

Sound Source Listener Direct sound Reflected sound

Reverberation

§ Room reverberation is characterized by its impulse response (IR)

– E.g. when a balloon pop is used as a sound source

§ The room IR is composed of three parts

– Direct path – Early reflections – Late-field reverberation: high echo density

§ RT60

– The time that it takes the reverberation to decay by 60 dB from its peak amplitude

42

10 20 30 40 50 60 70 80 90 100

• 0.4
• 0.2

0.2 0.4 0.6 0.8 1 CCRMA Lobby Impulse Response time - milliseconds response amplitude direct path early reflections late-field reverberation

Artificial Reverberation

§ Convolution reverb

– Measure the impulse response of a room – Convolve input with the measured IR

§ Mechanical reverb

– Use metal plate and spring – EMT140 Plate Reverb: https://www.youtube.com/watch?v=HEmJpxCvp9M

§ Delayline-based reverb

– Early reflections: feed-forward delayline – Late-field reverb: allpass/comb filter, feedback delay networks (FDN) – “Programmable” reverberation

43

Delayline-based Reverb

§ Schroeder model § Feedback Delay Networks

44

+

x(n) Z-M1 Z-M2 Z-M3

+

a11 a12 a13 a11 a12 a13 a11 a12 a13 y(n)

• Cascade of allpass-comb filters
• Mutually prime number for delay lengths

Convolution Reverb

§ Measuring impulse responses

– If the input is a unit impulse, SNR is low – Instead, we use specially designed input signals

• Golay code, allpass chirp or sine sweep: their magnitude responses are all flat but

the signals are spread over time – The impulse response is obtained using its inverse signal or inverse discrete Fourier transform

45

s(t)

LTI system

r(t)

test sequence measured response

n(t) h(t)

measurement noise

r(t) = s(t) ∗ h(t) + n(t),

Convolution Reverb

46

500 1000 1500

• 0.5

0.5 sine sweep, s(t) amplitude frequency - kHz sine sweep spectrogram 200 400 600 800 1000 5 10 500 1000 1500 2000

• 1
• 0.5

0.5 1 sine sweep response, r(t) time - milliseconds amplitude time - milliseconds frequency - kHz sine sweep response spectrogram 500 1000 1500 2000 5 10 100 200 300 400 500 600 700 800 900 1000

• 0.04
• 0.02

0.02 0.04 0.06 0.08 measured impulse response time - milliseconds amplitude

s(t) r(t) ˆ h (t)

( J. Abel )

Spatial Hearing

§ A sound source arrives in the ears of a listener with differences in time and level

– The differences are the main cues to identify where the source is. – We call them ITD (Inter-aural Time Difference) and IID (Inter-aural Intensity Difference) – ITD and IID are a function of the arrival angle.

47

R L ITD IID

Head-Related Transfer Function (HRTF)

§ A filter measured as the frequency response that characterizes how a sound source arrives in the outer end of ear canal

– Determined by the refection on head, pinnae or other body parts – Function of azimuth (horizontal angle) and elevation (vertical angle)

48

𝐼i(𝜕, ∅, 𝜄) 𝐼k(𝜕, ∅, 𝜄)

R L

49

Measured Head-Related Impulse Responses

50

Magnitude response

• f the HRIRs

Binaural Synthesis

§ Rendering the spatial effect using the measured HRIRs as FIR filters

– HRIRs are typically several hundreds sample long – Convolution or modeling by IIR filters

51

Input Left output Right output ℎi(𝑢, ∅, 𝜄) ℎk(𝑢, ∅, 𝜄)