Music transcription via convex optimization Song Mei ICME, Stanford - - PowerPoint PPT Presentation

Music transcription via convex optimization

Song Mei

ICME, Stanford

June 3, 2015

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 1 / 28

The problem.

Record the music. Signal on the computer. Music scores.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 2 / 28

The problem.

The problem: Given a music signal played by various instruments, we would like to separate the melodies from different instruments and take down the music scores. Why is the problem important? Music scores are easier to read. — To help people to learn to play instruments. Music scores are easier to be compared. — When you hear a song but you don’t know its name, how do you search for it? Music scores are easier to revise. — Musicians and music technicians would like to perform some revision on the scores. Music and Maths have instrinsic connections!

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 3 / 28

Background knowledge of music.

A piece of music is composed of musical notes. The common elements of music are, Pitch, Timbre, Loudness, Rythms, Speed, etc. The pitch, the timbre, and the loudness are the most important elemtents.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 4 / 28

Pitch.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 5 / 28

Pitch.

On a piano keyboard, there are 88 different keys, 55 white, 32 black. They are labeled from A0 to C8. (A0, A+0, B0, C0, ..., G0, G +0, A1, ..., G +1, A2, ..., A8, ..., C8.) One key corresponds to one frequency. On a standard instument, the map between pitches and frequencies is defined as General formula, pitch = 69 + 12 × log2(frequency/440). A4 has frequency 440Hz. A5 has frequency 880Hz, two times larger than A4. Between A4 and A5, there are 11 keys. They are logarithmically equally spaced with 21/12. 27/12 = 1.4983 . . . ≈ 1.5. The frequency of different keys share quasi

• rational ratios.

The rational relation brings harmony to music.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 6 / 28

Timbre.

The waveshapes of real instruments are not sinosoid. Piano. Guitar. The integer multiplications of the fundamental frequency are called the harmonic frequencies. Misconception: people believe the timbre of an instrument is determined by its waveshape, which is not true. If we represent a note as s(t) = K

k=1 Akcos(kω0t + ψk). The timbre

• f an instrument is determined by the relative amplitudes Ak but not

the phases ψk. Human ears are not sensitive to the phases.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 7 / 28

Other statistics.

A piece of music is about 60 secs to 30 mins. During one second, there are usually 1 - 30 notes. The frequency range is from 30 Hz to 4000 Hz, with two different pitches at least 10 Hz apart. For the ’wav’ file of the music, the typical sample rate is 44100 Hz.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 8 / 28

Let us look at our problem again.

Problem: Given a music signal played by various instruments, we would like to separate the melodies from different instruments and take down the music scores. A naive method: We record all the keys for all the instruments. On time domain, they form a finite set of basis. We use matching pursuit to solve this problem. In practice, this will not work! Reason: Though the instruments and keys are finite, there are many other

• factors. The stength to press the key, the duration, and the room

geometry, will all show some effects. We cannot use basis on time domain!

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 9 / 28

Previous methods. Spectrogram.

For all previous methods, the first step is to form a spectrogram. Spectrogram: STFT, WLT, WPT, CQT, Synchrosqueeze, etc. Early researches aim to improve the resolution of the spectrogram. At present, the resolution of the spectrogram is not a big problem. (Synchrosqueeze, Robust spectrotemporal decomp.)

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 10 / 28

Previous methods. NMF

Many heuristic methods are proposed to extract features from spectrogram. Smaragdis and Brown (2003) proposed to perform non-negative matrix factorization on the absolute value of the spectrotogram. [W , H] = nmf (abs(S), k). Here, S ∈ Cn×m is the spectrogram, W ∈ Rn×k, and H ∈ Rk×m. Each column of W is a basis for the relative amplitude of a note. Each row of H is the time envelop of the corresponding note. Good performance on some simple examples. Now, it is often used in the unsupervised learning process to learn the relative amplitudes of different frequencies of a note.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 11 / 28

Previous methods. NMF

abs(S) W (:, 1) H(1, :)

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 12 / 28

Previous methods. Linear model. Lasso.

Given the learned basis, Lee, Yang, and Chen (2011) proposed to use matching pursuit to do the pitch selection and amplitude estimation. After estimation, they used the Maximum A Posterior algorithm to smoothify the result. This state of art result is with 70% accuracy rate. Why fail on 30%? Observation: nearly all algorithms are good when notes are well-separated in the time-frequency domain. Difficulty emerges when notes overlap on time frequency domain. E.g. C4 and C5, C4 and G4, C4 piano and C4 violin. Linear model often fails to discover a pitch. The linear superposition of absolute value of spectrogram assumption is wrong!

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 13 / 28

The assumption of linear model is wrong.

Let s1 and s2 be two notes. ˆ s1(ω) =

K

• k=1

Akeiψkδ(ω − 2πkf1), ˆ s2(ω) =

K

• k=1

Bkeiφkδ(ω − 2πkf2), ˆ s(ω) = a · ˆ s1 + b · ˆ s2, |ˆ s(ω)| = a|ˆ s1| + b|ˆ s2|. When f1/f2 is not rational. |ˆ s(ω)| = a|ˆ s1| + b|ˆ s2|. When f1/f2 is rational.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 14 / 28

Our work. Intuitions.

Music intuitions. When we learn the basis of every note, only the relative amplitude is reliable, and the relative phase information is not reliable. Math intuitions. The superposition is not on the absolute value of the spectrogram. We need to consider the phase! Conclusion. Though the phase information cannot be learned previously, we need to consider the phase in the model!

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 15 / 28

Model formulation. Stationary case.

We observe a stationary ocilliatary signal with noise s(t) =

J

• j=1

αjsj(t) + ε(t) =

J

• j=1

αj

Kj

• k=1

Ajkcos(kωjt + ψjk) + ε(t), where each αj ∈ R+ and ψjk ∈ [0, 2π) are unknown, and Ajk ∈ R+ are known. The problem is to estimate each αj and ψjk. The interesting cases would be that for different j, ωj shares rational

• ratio. To simplify our notations, we assume ωj = ω0, j = 1, 2, . . . , J.

(Or we use their gcd.)

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 16 / 28

Model formulation. Stationary case.

The Fourier transform of (analytical) s(t) gives ˆ s(ω) =

Kj

• k=1

(

J

• j=1

αjAjkeiψjk)δ(ω − kω0) + ˆ ε(ω). We can see ˆ s(kω0) ∼ J

j=1 αjAjkeiψjk.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 17 / 28

Model formulation. Stationary case.

The parameter we need to estimate is αj. We propose to solve the following optimization problem min

αj,ψjk K

• k=1

(|ˆ s(kω0)| − |

J

• j=1

αjAjkeiψjk|)2 + λα1, subject to αj ≥ 0, j = 1, 2, . . . , J. (1) The minimal point of K

k=1(ˆ

s(kω0) − | J

j=1 αjAjkeiψjk|)2 may not be

• unique. So we add on l1 penalty of α to find the most sparse solution.

The relation of this with linear model is that, in linear model, ψjk are constrained to be 0. How to solve this problem?

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 18 / 28

Non-convex problem to Convex problem.

Theorem

The optimization problem 1 is equivalent to the convex optimization below

min

Ck,αj K

• k=1

(|ˆ s(kω0)| − Ck)2 + λα1, subject to αj ≥ 0, j = 1, 2, . . . , J, Ck ≤

J

• j=1

αjAjk, max

l

(αlAjl −

J

• j=l

αjAjk) ≤ Ck. (2)

The minimums of problem 1 and 2 are equal, and the minimal points ˆ αj are the same. This problem is extremely easy to solve using cvx.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 19 / 28

Error estimation.

Theorem

In the noiseless case, the solution of problem 2 may not be unique. In noiseless case, an error bound for the solution ˆ α of problem 2 gives

|ˆ α − α|∞ ≤ 2C 1 − C |α|∞,

where

C = max

j

min

k

• l=k Ajl

Ajk .

In noisy case, an error bound for ˆ α gives

|ˆ α − α|∞ ≤ 2C 1 − C |α|∞ + 1 1 − C max

j

min

k |εk/Ajk|,

This theorem implies that, the estimation of the coefficients α depend on the property of the basis. If the basis are somewhat ’orthogonal’, the estimation is very good.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 20 / 28

Numerical Results.

We recorded 9 keys on piano, C3, E3, G3, C4, E4, G4, C5, E5, G5. These keys have lots of overlapping frequencies. (frequency(C3) = 1/2 × frequency(C4) = 1/3 × frequency(G4) = 1/4 × frequency(C5)). We learn their amplitude basis Ajk. By combining some of the signals with different weights and a small noise using computers, we generate a mixed signal s(t). Here, the training data are real data, but the test data are generated by computers to make it stationary. We perform our model and the linear model on the mixed signal s(t). The l1 penalty of both models are chosen to be a same small number.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 21 / 28

Numerical Results.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 22 / 28

What if a real problem?

In a real problem, the amplitude of the signal is time-decaying. It is easy to generalize our model to the non-stationary case. Thanks to the convex formulation, we can impose further convex constraints. For example, to impose time continuity constraints on the variables.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 23 / 28

Main idea on the time-decaying case

s(t) =

J

• j=1

sj(t) + ε(t) =

J

• j=1

Kj

• k=1

Ajk(t)cos(kωjt + ψjk(t)) + ε(t), In the learning process, we build several basis rather than one basis for one note. We can also formulate the problem into a convex optimization problem. We can add group l1 − l2 penalty to smoothify the estimation. l1 penalty may not work well to make the solution to be sparse. We use l0 penalty instead. Since there are not too many notes that are

• verlapped, we can do greedy search or use the prune technique.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 24 / 28

Numerical Results.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 25 / 28

Numerical Results.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 26 / 28

Conclusions.

Traditional linear model for automatic music transcription tend to miss some notes. Adding the phase into consideration, we build a simple model, which can improve the results. The limitation of the new model is that, there are more degree of freedom so that it tend to overestimate some notes. The music transcription problem is very complicated.

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 27 / 28

Thanks.

Thanks the discussions from Dr. Dennis Sun and Prof. Lexing Ying. Thank you for listening!

Song Mei (ICME, Stanford) Music transcription via convex optimization June 3, 2015 28 / 28