User-Guided Variable-Rate Time-Stretching Via Stiffness Control - - PowerPoint PPT Presentation

Nicholas J. Bryan, Jorge Herrera, and Ge Wang Stanford University | CCRMA

User-Guided Variable-Rate Time-Stretching Via Stiffness Control

DAFx 2012

Introduction

• User control over variable-rate time-stretch processing
• Stretch some regions more than others (e.g. stretchability, stiffness)
• Transient preservation, rhythmic warping, emphasis modification, etc.

− −0.5 0.5 1

Rhythmic Warping Demo

Original Stretched (2x) No Stiffness / Stiffness Swung + Stretched (2x)

0.2 0.4 0.6 0.8 1 1.2 1.4 −1 −0.5 0.5 1 1.5 2

Time (s) Amplitude s(t) k(t) (t)

0.2 0.4 0.6 0.8 1 1.2 1.4 −1 −0.5 0.5 1 1.5 2

Amplitude Time (s)

Motivation

• Automatic methods have no mechanism for user input
• Direct manipulation of the stretch rate is hard!

2x ?

Prior Work

− − − −0.5 0.5 1 − − − −

ProTools, Logic Pro, FL Studio, etc.

Nielson and Brandorff, 2002

− −

− −

Proposed Method

• User annotates stiffness + timing constraints
• Solve optimization problem to convert stiffness to stretch factor
• Use pre-existing time-stretch processor to stretch sound

Stiffness Stretch Factor Processor

− −0.5 0.5 1

Step I: Spring Chain

• Model audio as chain of springs
• Solve for equilibrium via Hooke’s Law
• Spring stiffness an intuitive measure (i.e. proportional)

Fi = −kixi

…

Initial Formulation

find x subject to f = 0 xT1 = L fi = ki+1xi+1 − kixi L = final length x = spring displacement f = spring forces k = spring stiffness

• Violates intuition: no initial rest length

Problem

Γ = 2

k1 = 100

k2 = 1 k2 = 1

k1 = 100

Reformulation

find x subject to f = 0 (x + x0)T1 = L x + x0 ≥ 0

find x subject to f = 0 xT1 = L

x0 = Initial Rest Length

Reformulation

• Minimize the force disturbance from equilibrium (smoothly)

minimize

x

||f||2 + µ||x||2 subject to (x + x0)T1 = L x + x0 ≥ 0 µ = Penalty Weight x0 = Initial Rest Length

Step II: Stiffness to Stretch Factor

• Given input and output lengths, compute stretch factor as simple ratio

0.2 0.4 0.6 0.8 1 1.2 1.4 −1 −0.5 0.5 1 1.5 2 2.5 3

Time (s) Amplitude s(t) k(t) (t)

γ = x+x0

x0

Step III: Time-Stretching

• Given optimal variable-rate stretch factor, process sound
• Phase Vocoder (PV)
• Pitch Synchronous overlap add (PSOLA)

Extensions

• Rhythmic warping, smoothing of user input, max stretching limits
• Example: Straight-to-Swing

minimize

x

||f||2 + µ||x||2 subject to (x + x0)T1 = L x + x0 ≥ 0 (x1 + x1

0)T1 = 2 3L/2

(x2 + x2

0)T1 = 1 3L/2

(x3 + x3

0)T1 = 2 3L/2

Results

Varying Stretch Length

0.05 0.1 0.15 0.2 0.25 0.3 0.35 −0.5 0.5 1 1.5 2 2.5 3 3.5 4 4.5

Time (s) (t) = .5 = .75 = 1 = 1.5 = 2.0

• Varying the overall stretch factor gives smooth, intuitive stretch factors

minimize

x

||f||2 + µ||x||2 subject to (x + x0)T1 = L x + x0 ≥ 0

Γ = .5 Γ = .75 Γ = 1.5 Γ = 2.0 Γ = 1.0

Regularization

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.5 1 1.5 2 2.5 3 3.5

Time (s) (t) µ = .001 µ = .01 µ = .1 µ = 1

• Regularization penalty smooths the time-varying stretch factor

minimize

x

||f||2 + µ||x||2 subject to (x + x0)T1 = L x + x0 ≥ 0

µ = .001 µ = .01 µ = .1 µ = 1

Rhythmic Emphasis Modification

0.5 1 1.5 2 2.5 −1 1 2 3 4 5

Time (s) Amplitude

I’m gonna make him an offer he can’t refuse

s(t) k(t) (t)

• riginal

warped warped + stretched (slowed by 1.3x) I’m gonna make him an offer he can’t refuse I’m gonna maaake him an offer he caaaan’t refuse

Conclusions

• Method of user control over variable-rate time-stretching
• Decouples stiffness control + timing constraints to user
• Converts user control into optimal time-dependent stretch rate
• Agnostic to time modification algorithms

Acknowledgments & Thanks

• National Science Foundation Creative IT grant No. IIS-0855758
• School of Humanities and Sciences, Stanford University

Nicholas J. Bryan, Jorge Herrera, and Ge Wang Stanford University | CCRMA

User-Guided Variable-Rate Time-Stretching Via Stiffness Control

DAFx 2012