Modelling music Dorien Herremans ISTD, Singapore University of - - PowerPoint PPT Presentation

▶
modelling music
SMART_READER_LITE
LIVE PREVIEW

Modelling music Dorien Herremans ISTD, Singapore University of - - PowerPoint PPT Presentation

Jun 27, 2023 704 likes •1.28k views

Modelling music Dorien Herremans ISTD, Singapore University of Technology and Design IHPC, A*STAR Sound and Music Computing, NUS - 22.03.18 1 Outline Why modelling music? A brief early history of generation systems Approaches to

slide-1
SLIDE 1

Modelling music

Dorien Herremans

ISTD, Singapore University of Technology and Design IHPC, A*STAR 1 Sound and Music Computing, NUS - 22.03.18

slide-2
SLIDE 2

Outline

  • Why modelling music?
  • A brief early history of generation systems
  • Approaches to modelling music?
  • Rules — counterpoint
  • Machine learning:
  • Markov models
  • LSTMs
  • Word2vec
  • Conclusions

2

slide-3
SLIDE 3

A brief early history of generation systems

3

slide-4
SLIDE 4

Automatically generating music?

  • Muzikalisches Wurfelspiel

(Mozart)

  • Actual origin: Der allezeit

fertige Menuetten- und Polonaisencomponist (Johann Philipp Kirnberger, 1757)

4

slide-5
SLIDE 5

Automatically generating music?

  • Other chance inspired

compositions

  • John Cage's “Atlas Eclipticalis”
  • Charles Dodge's 


“The Earth's Magnetic Field”

  • Iannis Xenakis’


“Analogique A” (Markov)

  • Fractal music
  • …

5

slide-6
SLIDE 6

Automatically generating music?

“[The Engine's] operating mechanism might act upon other things besides numbers [. . . ] Supposing, for instance, that the fundamental relations of pitched sounds in the signs of harmony and of musical composition were susceptible of such expressions and adaptations, the engine might compose elaborate and scientific pieces of music of any degree of complexity or extent." 
 
 — Ada Lovelace 1840

6

slide-7
SLIDE 7

Automatically generating music?

  • 1957 Illiac Suite — Lejaren Hiller & Leonard Isaacson
  • Rule based + statistical models

7

slide-8
SLIDE 8

Automatically generating music?

  • David Cope
  • Experiments in Musical

Intelligence

  • 1981 Composer's block
  • Rules to model of his own

style —> different composers

  • e.g. Chopin Nocturne

8

slide-9
SLIDE 9

Automatically generating music?

  • The Continuator 


— Francois Pachet

  • Interactive system: “extend the

technical ability of musicians with stylistically consistent, automatically learnt musical material”

  • Markov models (real time learning)
  • Flow machines

9

http://www.francoispachet.fr/continuator/

slide-10
SLIDE 10

Automatically generating music?

  • Impro-Visor — Bob

Keller (2005)

  • Probabilistic

grammars, deep believe networks,. . .

10

slide-11
SLIDE 11

What’s going on in the field?

Problems

  • Generating melody, rhythm, timbre, harmony (chords),. . .
  • Constraining to a narrative (games, video, emotions,. . . )
  • Interacting with live performer


Approaches

  • Rule-based methods
  • Optimization approaches (genetic algorithms, constraint programming,

local search, etc)

  • Statistical models (Markov models, factor oracles, deep learning, etc)

11

Herremans, Chuan, Chew. 2017. A functional taxonomy of music generation systems. ACM Computing Surveys.

slide-12
SLIDE 12

The rapid rise of digital music

  • In 2015: revenue from digital

music: 6.4 billion USD 





= 45% of the global music industry

  • Asia: 14% of these revenues
  • Number of digital music users

expected to grow 15% a year until 2020

12

slide-13
SLIDE 13

New application opportunities

13

slide-14
SLIDE 14

Approaches to modelling music

14

slide-15
SLIDE 15

Rules and optimisation

  • Music theory
  • Counterpoint - 5th species:

15

—> One of the most formally defined musical styles Rules written by Johan Fux in 1725

slide-16
SLIDE 16

Rules and optimisation

Composing music = combinatorial optimization problem

  • Find the right combination of notes
  • So that the piece sounds ‘good’, i.e. fits a style as well as

possible

16

slide-17
SLIDE 17

Optimization objective

  • 19 melodic and 19 harmonic rules


e.g. Each large leap should be followed by stepwise motion in the opposite direction Penalty points for each rule violation —> Minimized

17

slide-18
SLIDE 18

Finding best combination of notes

  • Exact algorithms
  • Metaheuristics
  • Population Based (genetic algorithms,. . . )
  • Constructive (ant colony, GRASP

, . . . )

  • Search Based (tabu search, variable neighbourhood search,…)

18

A metaheuristic is a high-level problem-independent algorithmic framework that provides a set of guidelines

  • r strategies to develop heuristic optimization algorithms

(Sörensen and Glover, 2014).

slide-19
SLIDE 19

Finding best combination of notes

  • Local search with 3 types of "moves"

19

slide-20
SLIDE 20

Finding best combination of notes

20

slide-21
SLIDE 21

Result

21

  • D. Herremans, K. Sorensen. 2013. Composing Fifth Species Counterpoint Music With A Variable Neighborhood Search
  • Algorithm. Expert Systems with Applications. 40(16):6427{6437.
slide-22
SLIDE 22

From rules to machine learning

  • Limitation: rules need to be defined!

Rule-based objective function Machine learned objective function

22

Why still optimization? Because it allows us to:

  • Easily integrate constraints
  • Enforce structure
  • Correctly evaluate circles
slide-23
SLIDE 23

Markov models

  • Trained on dataset
  • Transition probabilities
  • Issue with machine learning: plagiarism (Papadopoulo et al., 2014)
  • Different viewpoints (Conklin, 1995)



 
 
 
 
 How integrated in optimisation objective function?

23

slide-24
SLIDE 24

High probability sequences?

  • No.
  • High probability can be very repetitive
  • Find the right mix of surprising notes
  • One approach: 


Objective function: Match an expectancy profile

24

  • D. Herremans, S. Weisser, K. Sorensen, and D. Conklin, 2015. Generating structured music using quality metrics based on Markov models. Expert Systems with Applications.
slide-25
SLIDE 25
  • Research started in the 1950s
  • Why don’t we listen to generated music on our phones?



 
 
 
 
 
 
 
 


The future of music generation

25

2 Challenges:

  • How to control emotion in music?

  • Long-term structure
  • D. Herremans, K. Sorensen. 2013. FuX, an Android app that generates counterpoint. Proceedings of IEEE Symposium on Computational

Intelligence for Creativity and Affective Computing (CICAC). Singapore. 48-55.

slide-26
SLIDE 26

MorpheuS music generation

26

2 Challenges:

  • How to control emotion in music?

  • Long-term structure
slide-27
SLIDE 27
  • 1. Modeling emotion: tension
  • Important for music in:
  • Video games
  • Film background
  • YouTube





  • Tension: many aspects: tonal, rhythmic, loudness etc.
  • First comprehensive model by Mary Farbood (2002)

27

slide-28
SLIDE 28
  • 1. Modeling emotion: tension
  • The Spiral Array:


3D geometrical mathematical model of 
 tonality (Chew, 2012)

  • 3D representation of higher level musical entities in tonal space
  • Helix for pitch classes, chords and keys
  • Used in Key detection (Chuan & Chew, 2005), Chord detection,

finding key boundaries (Chew, 2000), etc.
 
 —> 3 aspects of tonal tension

  • Tool available online

28 Herremans D., Chew E.. 2016. Tension ribbons: Quantifying and visualising tonal tension. Second International Conference on Technologies for Music Notation and Representation (TENOR). 2:8-18. Cambridge, UK.

slide-29
SLIDE 29

Cloud diameter

29

C major triad and the C diminished triad

slide-30
SLIDE 30

Cloud momentum

30

C major and C# major chord with their center of effect (ce, in green).

slide-31
SLIDE 31

Tensile strain (distance from the key)

31

C major — C# major — A minor chord together with their distance from the ce of the key (in orange).

slide-32
SLIDE 32

Tonal tension - Tristan chord

  • Wagner's opera ‘Tristan und

Isolde’

  • Bass note and augmented 4th,

6th and 9th

32

slide-33
SLIDE 33
  • We can generate good sounding short fragments of music
  • Guitar solos generated by Integer Programming
  • Short solo fragments
  • 2. Long-term structure

33 Cunha N., Subramanian A., Herremans D. 2017. Generating guitar solos by integer programming. Journal of the Operational Research Society.

slide-34
SLIDE 34

Pattern detection

  • Compression algorithm: COSIATEC (Meredith, 2013)
  • Point set representation
  • Computes a compressed encoding of the piece


—> maximal translatable patterns

34

slide-35
SLIDE 35

Example: Bach prelude 20, bk 2

35

slide-36
SLIDE 36

MorpheuS: optimisation algorithm

  • Problem: find new pitches
  • Objective: match tension profile to template



 
 
 


  • Hard constraint: detected patterns

36

slide-37
SLIDE 37

MorpheuS: generating affective music

37 Herremans D., Chew E. 2017. MorpheuS: generating structured music with constrained patterns and tension. IEEE Transactions on Affective Computing. PP (99).

Random (with patterns)

slide-38
SLIDE 38

MorpheuS: generating affective music

38 Herremans D., Chew E. 2017. MorpheuS: generating structured music with constrained patterns and tension. IEEE Transactions on Affective Computing. PP (99).

slide-39
SLIDE 39

Example: Kabalevsky’s ‘Clowns’

39

slide-40
SLIDE 40

MorpheuS: generating affective music

40

How can we fully model music so we don’t need a template?

slide-41
SLIDE 41

Long-short term networks

With CNN

41

slide-42
SLIDE 42

Modeling music with LSTM

  • Convolutional nets have transformed the field of vision. 


Can we leverage this to music?

  • Challenge: time…
  • RNNs
  • LSTMs
  • …
  • Examples:
  • Boulanger-Lewandowski (2012): piano roll representation with

RNN-RBM

  • Sigtia et al. (2014): Piano roll representation with CNN and RNN

42

slide-43
SLIDE 43

Music as tonnetz images

  • Tonnetz (Euler, 1739)

43

slide-44
SLIDE 44

Extended Tonnetz matrix

44

slide-45
SLIDE 45

Modeling music with LSTM

45 Chuan C.-H., Herremans D.. 2018. Modeling temporal tonal relations in polyphonic music through deep networks with a novel image-based 


  • representation. The Thirty-Second AAAI Conference on Artificial Intelligence. New Orleans, US.

Code: http://dorienherremans/software

slide-46
SLIDE 46

Modeling music with LSTM

  • Filters reflect fifth

relationships

  • Generated music is more

tonally stable, with less tension

  • Higher compression rates


—> more structure

46

slide-47
SLIDE 47

Word2vec & music

47

slide-48
SLIDE 48

Modeling music with word2vec

  • Distributional hypothesis in linguistics:



 Continuous vector space in which semantically similar words are represented geographically close to each other 


  • Model:

Words that are used and

  • ccur in the same contexts

tend to purport similar meanings (Harris, 1954)

slide-49
SLIDE 49

Similarity between language and music

  • Besson and Schön (2001) :
  • Common ancestor of music and language: expression of emotive meaning


e.g. emotional excitement: fast, accelerating, and high-pitched sound patterns

  • Sequential elements that unfold in time:
  • Rhythm
  • Temporal ratios (notes/phonemes, chords/words)
  • Grammar: musical grammar can include contour, cadence for closing, etc (more

flexible)

  • Ability to generate strong expectancies: both unexpected (incongruent) words and

notes generate peaks measurable in brain potentials (N400 & P600)

49

slide-50
SLIDE 50

Modeling music with word2vec

50

A slice of music = word

  • > Captures tonality

Herremans D., Chuan C.-H.. 2017. Modeling Musical Context with Word2vec. First International Workshop On Deep Learning and Music joint with IJCNN. 1:11-18. Anchorage, US

Can we derive semantic similarity through context only?

Code: http://dorienherremans/software

LSTMs, RNNs, etc. typically incorporate musical information (i.e., pitch, pitch class, duration, intervals,…)

six unique slices (unit = 8th note) transpose to C major

slide-51
SLIDE 51

Word2vec: skip-gram

  • Group of models used to create word embeddings [Mikolov et al. [2013]
  • Two-layer neural network architecture based on either:
  • continuous bag-of-words (CBOW): predict word based on context
  • continuous skip-gram model: predict context based on word
  • Low computational complexity:
  • easily handle a corpus with a size ranging in the 


billions of words in a matter of hours

  • CBOW models: faster
  • Skip-gram: performs better on small datasets

51

slide-52
SLIDE 52

Modeling music with word2vec

Cosine distance between the tonic chord C major and its functional chords

slide-53
SLIDE 53

Future of music modeling

  • Generating music with long-term structure and affect:
  • Film, YouTube background, automatic accompaniment apps and video

games

  • Why else model music?
  • Audio transcription
  • Classification problems
  • Emotion recognition
  • Recommender systems
  • …





53 Herremans D., Chuan C.-H., Chew E.. In Press. A Functional Taxonomy of Music Generation Systems. ACM Computing Surveys.

slide-54
SLIDE 54

Conclusions

Machine learning brings a range of new possibilities to digital music/audio, including:

  • There has been an evolution from rule-based systems to machine

learning

  • Challenges remain however!
  • Long term structure
  • Emotions in music

54

slide-55
SLIDE 55

Modelling music

Dorien Herremans

ISTD, Singapore University of Technology and Design IHPC, A*STAR dorienherremans.com 55 Sound and Music Computing, NUS - 22.03.18