Applications of AI in music Smart music through machine learning - - PowerPoint PPT Presentation

▶
applications of ai in music
SMART_READER_LITE
LIVE PREVIEW

Applications of AI in music Smart music through machine learning - - PowerPoint PPT Presentation

Sep 08, 2022 330 likes •588 views

Applications of AI in music Smart music through machine learning Dorien Herremans ISTD, Singapore University of Technology and Design IHPC, A*STAR Music Research Symposium, 2/2/18, IHPC, A*STAR 1 The rapid rise of digital music In

slide-1
SLIDE 1

Applications of AI in music

Smart music through machine learning

Dorien Herremans

ISTD, Singapore University of Technology and Design IHPC, A*STAR 1 Music Research Symposium, 2/2/18, IHPC, A*STAR

slide-2
SLIDE 2

The rapid rise of digital music

  • In 2015: revenue from digital

music: 6.4 billion USD 





= 45% of the global music industry

  • Asia: 14% of these revenues
  • Number of digital music users

expected to grow 15% a year until 2020

2

slide-3
SLIDE 3

New application opportunities

3

slide-4
SLIDE 4

Overview

  • 1. Hit song prediction
  • 2. Modelling music with machine learning

4

slide-5
SLIDE 5

Hit song prediction

5

slide-6
SLIDE 6

Can we predict hits?

  • In 2011 record companies invested 4.5 billion USD in new talent

worldwide (IFPI, 2012)



 
 


“Hit song science is not yet a science” (Pachet and Roy, 2008)


6

slide-7
SLIDE 7

Dance hit song prediction

  • Dataset:
  • Official Charts Company
  • Billboard
  • 1985 until 2013
  • What are hits?


—> Top 10 hits versus Top 30-40

  • 139 Audio features such as 


loudness, tempo, time between beats, 
 timbre, etc.

7

slide-8
SLIDE 8

Dance hit song prediction

  • Classification results: Top 10 vs Top 30-40
  • D. Herremans, D. Martens, K. Sörensen. 2014. Dance Hit Song Prediction. Journal of New Music Research – special issue on music and machine learning.

43(3):291–302. 8

slide-9
SLIDE 9

Hit song prediction through social media

  • “There are people… with some sort of sixth sense for hit songs,

such that they listen to hit songs even before they actually climb to the top of the record charts.” (Smit, 2013)

  • Early adopters (Rogers, 2010)
  • last.fm listening behaviour:
  • ~8000 members
  • 6 months

Herremans D., Bergmans T.. 2017. Hit Song Prediction Based on Early Adopter Data and Audio Features. The 18th International Society for Music Information Retrieval Conference (ISMIR) - Shuzou, China. 9

slide-10
SLIDE 10

Future challenges

  • Social networking data
  • Deeper models
  • More advanced audio features
  • Other genres
  • Sentiment analysis of lyrics



 
 


Related to other classification problems, e.g. composer identification, emotions detection.

10

slide-11
SLIDE 11

Modeling music

11

slide-12
SLIDE 12
  • Research started in the 1950s
  • Why don’t we listen to generated music on our phones?



 
 
 
 
 
 
 
 


For music generation

12

2 Challenges:

  • How to control emotion in music?

  • Long-term structure
slide-13
SLIDE 13
  • 1. Modeling emotion: tension
  • Important for music in:
  • Video games
  • Film background
  • YouTube





  • Tension: many aspects: tonal, rhythmic, loudness etc.

13

slide-14
SLIDE 14
  • 1. Modeling emotion: tension
  • 3D geometrical mathematical model of 


tonality (Chew, 2012)

  • Tool available online
  • First step towards more complex emotional models

14 Herremans D., Chew E.. 2016. Tension ribbons: Quantifying and visualising tonal tension. Second International Conference on Technologies for Music Notation and Representation (TENOR). 2:8-18. Cambridge, UK.

slide-15
SLIDE 15
  • We can generate good sounding short fragments of music
  • Guitar solos generated by Integer Programming
  • Short solo fragments
  • 2. Long-term structure

15 Cunha N., Subramanian A., Herremans D. 2017. Generating guitar solos by integer programming. Journal of the Operational Research Society.

slide-16
SLIDE 16
  • MorpheuS app constrains tension AND

long-term structure:
 
 Pattern detection on template 
 —> constrained during generation

  • 2. Long-term structure

16 Herremans D., Chew E. 2017. MorpheuS: generating structured music with constrained patterns and tension. IEEE Transactions on Affective Computing. PP (99).

slide-17
SLIDE 17

MorpheuS: generating affective music

17 Herremans D., Chew E. 2017. MorpheuS: generating structured music with constrained patterns and tension. IEEE Transactions on Affective Computing. PP (99).

slide-18
SLIDE 18

MorpheuS: generating affective music

18

How can we fully model music so we don’t need a template?

slide-19
SLIDE 19

Modeling music with LSTM

  • Leveraging success from image based systems in deep neural

networks

  • Novel image based representation: tonnetz

19 Chuan C.-H., Herremans D.. 2018. Modeling temporal tonal relations in polyphonic music through deep networks with a novel image-based 


  • representation. The Thirty-Second AAAI Conference on Artificial Intelligence. New Orleans, US.
slide-20
SLIDE 20

Modeling music with LSTM

  • Filters reflect fifth

relationships

  • Generated music is more

tonally stable, with less tension

  • Higher compression rates


—> more structure

20

slide-21
SLIDE 21

Modeling music with word2vec

21

NLP: Words that are used and

  • ccur in the same contexts tend to

purport similar meanings

A slice of music = word

  • > Captures tonality

Herremans D., Chuan C.-H.. 2017. Modeling Musical Context with Word2vec. First International Workshop On Deep Learning and Music joint with IJCNN. 1:11-18. Anchorage, US

We can calculate a ‘semantic’ similarity between slices of music based only on their context

slide-22
SLIDE 22

Future of music modeling

  • Generating music with long-term structure and affect:
  • Film, YouTube background, automatic accompaniment apps and video

games

  • Why else model music?
  • Audio transcription
  • Classification problems
  • Emotion recognition
  • Recommender systems
  • …





22 Herremans D., Chuan C.-H., Chew E.. In Press. A Functional Taxonomy of Music Generation Systems. ACM Computing Surveys.

slide-23
SLIDE 23

Conclusions

Machine learning brings a range of new possibilities to digital music/audio, including:

  • Hit prediction
  • Classification problems, e.g. sentiment analysis of lyrics, emotion classification
  • Accompaniment app for beginner guitarists
  • Chord detection
  • Automatic music production
  • Music generation for game and film
  • Music and health applications
  • Automatic piano fingering
  • …

23

slide-24
SLIDE 24

Applications of AI in music

Smart music through machine learning

Dorien Herremans

ISTD, Singapore University of Technology and Design IHPC, A*STAR dorienherremans.com 24 Music Research Symposium, 2/2/18, IHPC, A*STAR