Deep Learning Techniques for Music Generation 3. Generation by - - PowerPoint PPT Presentation

▶
deep learning techniques for music generation 3
SMART_READER_LITE
LIVE PREVIEW

Deep Learning Techniques for Music Generation 3. Generation by - - PowerPoint PPT Presentation

Apr 07, 2023 238 likes •619 views

Deep Learning Techniques for Music Generation 3. Generation by Feedforward Architectures Jean-Pierre Briot Jean-Pierre.Briot@lip6.fr Laboratoire dInformatique de Paris 6 (LIP6) Sorbonne Universit CNRS Programa de Ps-Graduao em

slide-1
SLIDE 1

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Jean-Pierre Briot

Jean-Pierre.Briot@lip6.fr

Laboratoire d’Informatique de Paris 6 (LIP6) Sorbonne Université – CNRS Programa de Pós-Graduação em Informática (PPGI) UNIRIO

Deep Learning Techniques for Music Generation

  • 3. Generation by Feedforward Architectures
slide-2
SLIDE 2

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Direct Use – Feedforward – Ex 1

2

  • Feedforward Architecture
  • Prediction Task
  • Ex1: Predicting a chord associated to a melody segment

– scale/mode -> tonality – Training on a corpus/dataset <melody, chord> – Production (Prediction)

Input: Melody

(Pich of) F

Output: Pitch of a Chord

1st 2nd 3rd 4th

Pitch Pitch

slide-3
SLIDE 3

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Direct Use – Feedforward – Ex 1

3

  • Feedforward Architecture
  • Classification Task
  • Ex1: Predicting a chord associated to a melody segment

– scale/mode -> tonality – Training on a corpus/dataset <melody, chord> – Production (Classification)

F

Input: Melody Output: Chord (Pitch Class)

1st 2nd 3rd 4th A G# A# B … …

Pitch Pitch Class

slide-4
SLIDE 4

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Softmax

Logits Probabilities

4

slide-5
SLIDE 5

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Softmax and Sigmoid

Logits Probabilities

5

  • Softmax is the generalization of Sigmoid
  • From Binary classification to Categorical (Multiclass – Single Label) classification

Probability € [0, 1]

S Probabilities = 1

Sigmoid Softmax

σ(z)i = ezi / Σ ezj σ(z) = 1 ⁄ 1 + e-z

slide-6
SLIDE 6

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Softmax and Sigmoid

Logits Probabilities

6

  • Step function and Argmax are NOT differentiable
  • No gradient -> No possibility of back propagation

Probability € {0, 1} Step function (Perceptron) Softmax

slide-7
SLIDE 7

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Direct Use – Feedforward – Ex: ForwardBach

7

  • Feedforward Architecture
  • Classification Task
  • Ex: Counterpoint (Chorale) generation
  • Training on the set of (389) J.S. Bach Chorales (Choral Gesang)

Input: Melody Output: 3 Melodies

slide-8
SLIDE 8

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

  • Audio

– Waveform – Spectrogram (Fourier Transform) – Other (ex: MFCC)

  • Symbolic

– Note – Rest – Note hold – Duration – Chord – Rhythm – Piano Roll – MIDI – ABC, XML…

Representation

8

slide-9
SLIDE 9

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Representation

C B A# A G# G Score Piano Roll One hot Encoding

1 1

slide-10
SLIDE 10

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Encoding of Features (ex : Note Pitch)

  • Value

– Analogic

  • One-Hot

– Digital

  • Embedding

– Constructed

10

slide-11
SLIDE 11

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

  • Rest

– Zero-hot

» But ambiguity with low probability notes

– One more one-hot element – …

  • Hold

– One more one-hot element

» But only for monophonic melodies

– Replay matrix – …

Encoding

11

slide-12
SLIDE 12

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Representation

1 1 1 1 1 1

A C hold rest

?

12

slide-13
SLIDE 13

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Representation

1 1 1 1 1 1

A C hold rest

13

If time slice = sixteenth

slide-14
SLIDE 14

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Representation Choices

  • More Alternative Possibilities for Note:

– Pitch Class One-hot + Octave number – Binary of MIDI number – …

  • The Most Compact Representation is not always the

Best for Training/Generation

  • Hold as a Note Special Symbol

– Risk of Skewed Class, Because Appears Too Often

14

slide-15
SLIDE 15

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Music / Representation / Network

15

…

Input layer Output layer Hidden layers Soprano Voice Alto Voice Tenor Voice Bass Voice

… …

slide-16
SLIDE 16

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Code

  • Python (3)
  • Keras
  • Theano
  • r TensorFlow
  • Music21

16

slide-17
SLIDE 17

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Direct Use – Feedforward – Ex 2: ForwardBach

17

  • Feedforward Architecture
  • Prediction Task
  • Ex2: Counterpoint (Chorale) generation
  • Training on the set of (389) J.S. Bach Chorales (Choral Gesang)

Input: Melody Output: 3 Melodies

slide-18
SLIDE 18

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

ForwardBach

Original Regenerated Bach BWV 344 Chorale (Training Example)

18

slide-19
SLIDE 19

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

ForwardBach

Original Regenerated Bach BWV 423 Chorale (Test Example)

19

slide-20
SLIDE 20

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Music / Representation / Network Alternative 3 Models Architecture [Cotrim & Briot, 2019]

20

Input layer Output layer Hidden layers Soprano Voice Alto Voice Tenor Voice Bass Voice

slide-21
SLIDE 21

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Forward3Bach [Cotrim & Briot, 2019]

Original Triple Architecture Regenerated Bach BWV 423 Chorale (Test Example)

21

Single Architecture Regenerated

slide-22
SLIDE 22

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Comparison ? What happened ?

22

slide-23
SLIDE 23

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Overfitness Limitations

  • Musical accuracy is not that good (yet)
  • Regeneration of training example is better than Regeneration of

test/validation example

  • Case of Overfitness

23

slide-24
SLIDE 24

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Techniques

  • Limit Accuraccy and Control Overfitness
  • More Examples (Augment the Corpus)

– Keeping a Good Style Representation, Coverture and Consistency – More Consistency and Coverture – Transpose (Align) All Chorales to Only One Key (ex: C)

  • More Synthetic Examples

– More Coverture – Transpose All Chorales in All Keys (12)

  • Regularization

– Weight-based

» L1, L2

– Connexion-based

» Dropout

– Epochs-based

» Early-Stop

– Analysis of Learning Curves

24

slide-25
SLIDE 25

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Softmax

Logits Probabilities

25

slide-26
SLIDE 26

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Softmax + Cross-Entropy

26

  • Cross-Entropy measures dissimilarity between two probability

distributions (prediction and target/true value)

[Ng 2019]

slide-27
SLIDE 27

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Output Activation and Cost/Loss Functions

27

  • Ex. multiclass single label: Classification among a set of possible notes for a

monophonic melody, with only one single possible note choice (single label)

  • Ex. multiclass multilabel: Classification among a set of possible notes for a single-

voice polyphonic melody, therefore with several possible note choices (several labels)

  • Ex. multi multiclass single label: Multiple classification among a set of possible

notes for multivoice monophonic melodies, therefore with only one single possible note choice for each voice; Multiple classification among a set of possible notes for a set of time slices for a monophonic melody, therefore for each time slice with

  • nly one single possible note choice

Application Monophony Polyphony Multivoice

Task Type of the output (ˆ y) Encoding of Output activation Cost (loss) the target (y) function Regression Real IR Identity (Linear) Mean squared error Classification Binary {0, 1} Sigmoid Binary cross-entropy Classification Multiclass single label One-hot Softmax Categorical cross-entropy Classification Multiclass multilabel Many-hot Sigmoid Binary cross-entropy Multiple Multi Multi Sigmoid Binary cross-entropy Classification Multiclass single label One-hot Multi Multi Softmax Categorical cross-entropy

slide-28
SLIDE 28

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Output Activation and Cost/Loss Functions

28

Interpretation

none none argmax or sampling argsort and > threshold & max-notes p argmax or sampling

Other cost functions: Mean absolute error, Kullback-Leibler (KL) divergence…

Task Type of the output (ˆ y) Encoding of Output activation Cost (loss) the target (y) function Regression Real IR Identity (Linear) Mean squared error Classification Binary {0, 1} Sigmoid Binary cross-entropy Classification Multiclass single label One-hot Softmax Categorical cross-entropy Classification Multiclass multilabel Many-hot Sigmoid Binary cross-entropy Multiple Multi Multi Sigmoid Binary cross-entropy Classification Multiclass single label One-hot Multi Multi Softmax Categorical cross-entropy

slide-29
SLIDE 29

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Output Activation and Cost/Loss Functions

29

slide-30
SLIDE 30

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Output Activation and Cost/Loss Functions

30

slide-31
SLIDE 31

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Output Activation and Cost/Loss Functions

31

slide-32
SLIDE 32

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Output Activation and Cost/Loss Functions

32

slide-33
SLIDE 33

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

(Summary of) Principles of Loss Functions

  • Probability theory + Information theory
  • Intuition:

– Information content (Likely Event) : Low – Information content (Unlikely Event) : High

  • Self-information: I(x) = log (1/P(x)) = - log P(x)
  • Ex: I(note=B) = - log P(note=B)
  • I(E1 E2) = I(E1) + I(E2) (E1 and E2 independent events)
  • Entropy of Probability distribution : Si I(note=Notei), weighted by P(note=Notei)
  • H(note) = Si P(note=Notei) I(note=Notei)
  • = - Si P(note=Notei) log P(note=Notei)
  • Expectation-based alternative definition:
  • Expectation: Mean value of f(x) when x~P: Ex~P [f(x)] = Sx P(x) f(x)
  • H(note) = Enote~P I(x) = Enote~P [- log P(note)] = - Enote~P [log P(note)]

Likely event Unlikely event

See also Maximum likelihood principle

33

slide-34
SLIDE 34

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

KL-Divergence and Cross-Entropy

  • Measures of Differences between Distributions (over a same variable: note)

– Assymetric DKB(P||Q) =/= DKB(Q||P) H(P,Q) =/= H(Q,P)

  • Kullback-Leibler Divergence (KL- Divergence):
  • DKB(P||Q) = Enote~P [log P(note)/Q(note)] = Enote~P [log P(note) - log Q(note)]
  • Categorical Cross-Entropy:
  • H(P,Q) = Enote~P [- log Q(note)] = - Enote~P [log Q(note)]
  • Difference with KL-Divergence: log P(note) term, constant with respect to Q
  • DKB(y||y) = Enote~P [log y - log y] = Si yi (log yi - log yi)
  • H(y, y) = - Enote~P [log y] = - Si yi log yi
  • Binary Cross-Entropy:
  • HB(y, y) = - (y0 log y0 + y1 log y1) = - (y log y + (1-y) log (1-y))

34

slide-35
SLIDE 35

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

ForwardBach Brazilian Hymn Counterpoint

35

slide-36
SLIDE 36

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

ForwardBach Brazilian Hymn Counterpoint (2 times slower and removing the intro)

36

slide-37
SLIDE 37

Deep Learning – Music Generation – 2019

Jean-Pierre Briot

Reorchestration of God Save the Queen by DeepBach [Hadjeres, 2018]

37

https://www.youtube.com/watch?time_continue=1&v=x-W0ixD9Cpg