CTP431- Music and Audio Computing Music Information Retrieval - - PowerPoint PPT Presentation

CTP431- Music and Audio Computing Music Information Retrieval

Graduate School of Culture Technology KAIST Juhan Nam

1

Introduction

2

ü Instrument: ü Genre: ü Composer: ü Key: ü Mood: ü Songs with similar melody ü Can you transcribe the song into a music score ?

• ELO “After all”
• Radiohead “Exit Music”

Chopin Piano E-minor Classical

Melancholy, Sad, …

Information in Music

§ Factual Information

– track, artist, years, composers

§ Musical Information

– Music score: instrument, notes, meter, expressions – Melody, rhythm, chords, structure

§ Semantic Information

– genre, mood, text descriptions

3

Music Understanding by Human

5 http://www.slideshare.net/Daritsetseg/brainstem-auditory-evoked-responses-baer-or-abr-45762118

Music Understanding by Computer

§ Music Information Retrieval (MIR)

– An area of research that aims to infer various types of information from music by computers

6

Applications of MIR

§ Music listening

– Music identification, search and recommendation

§ Music Performance

– Interactive music performance – Musical Instrument learning

§ Music composition

– Automatic composition and arrangement

§ Entertainment

– Singing evaluation, game

§ Sound production

– Sound sample search in sound libraries – Automatic segmentation and digital audio Effects

7

Background

§ Scale and diversity of music contents

– Commercial music tracks

• Spotify: 30M+ songs (2015)
• Bugs music: 10M+ songs (2017)

– User contents

• YouTube: 300h+ video uploaded per min (2015)
• SoundCloud: 12h+ audio uploaded per minute (2014)

– User data

• Profile, play history, rate,
• Spotify: +24M active users (as of Jan, 2014)
• YouTube: +1B unique users’ visit each month (as of Dec, 2014)

§ All the music contents are readily accessible.

– How can we find music of my taste? – Can we have a Google for music?

Music Identification

§ Query by music

– Search a single unique song identified by the query – Audio fingerprinting

10

Shazam

Audio Fingerprinting

(http://labrosa.ee.columbia.edu/matlab/fingerprint/)

Music Identification

§ Query by humming

– Sing with humming and find closest matches – Melody-based match

11

SoundHound Melody Extraction

Music Search and Recommendation

§ Music Recommendation

– Playlist generation: personalized internet radio – Matching songs to users

• Song information: genre, years, artist, audio
• User information: profile, play history, rating, context (places)

– Music service item in industry: Google, Apple, Pandora, Spotify, Melon, Bugs,…

12

Pandora iTunes Music

Current Approaches

§ Manual Curation § Human Expert Analysis § Collaborative Filtering § Content-based Analysis (by computers)

13

Manual Curation

§ Playlist generation by music experts (or users)

– Traditional: AM/FM radio – The majority of current music services are based on this approach

§ Advantages

– Effective for usage-based music services (workout, study, driving or prenatal education) – Good for music discovery – Often with story-telling

§ Limitations

– No personalization – Not scalable

14

[www.soribada.com]

Human Expert Analysis

§ Pandora: music genome project (1999)

– Musicologists analyze a song for about 450 musical attributes in various categories – Big success as a music service

§ Advantages

– High-quality analysis – Good for music discovery

§ Limitations

– Expensive: take 20-30 minutes for a song to be analyzed – Not scalable : only for commercial tracks ?

15

Collaborative Filtering (CF)

§ Basic idea § Formation

– Matrix factorization (or matrix completion) problem

16

Person A: I like songs A, B, C and D. Person B: I like songs A, B, C and E. Person A: Really? You should check out song D. Person B: Wow, you also should check out song E.

16

Juhan Gangnam Style Juhan’s latent vector Gangnam Style’s latent vector

xu ys

pus = xu

Tys

Song Preference

qu1u2 = xu1

T xu2

User Similarity

r

s1s2 = ys1 T ys2

Song Similarity

Collaborative Filtering

§ Advantages

– Capture semantics of music in the aspect of human – Enable personalized recommendation (by nature)

§ Limitations

– The cold start problem: what if a song was never played by anyone? – Popularity bias: likely to recommend (already) well-known songs or songs from the same musician or album

17

Collaborative Filtering

§ Bad examples

18

Can you find songs similar to this musician? [Oord et. al, 2013]

Content-Based Analysis

§ An intelligent approach that makes computers listen to music and predict descriptive words from audio tracks

– Tags: genre, mood, instrument, voice quality, usage – Features: Spectrogram, MFCC, – Algorithms: GMM, SVM, Neural Networks

19

Algorithms Audio Files Audio Features

Text-based Music Retrieval by Auto-tagging

§ Sort the probability of the query tag and choose top-N songs

– Like text-based Google search

§ We also can compute similarity between songs using the estimated tag probabilities

– E.g. cosine distance between two tag probability vectors – Applicable to query by audio

21

Query word: “Female Lead Vocals” Top 5 ranked songs Norah Jones – Don’t know why Dido – Here with me Sheryl Crow – I shall believe No doubt – Simple kind of like Carpenters – Rainy days and Mondays

Demo: Music Galaxy Hitchhiker

(b) Search by Song mode with highlighted search results

Content-based Music Recommendation

§ Blending audio and user data

– Replace the text-based tags with the latent vector of a song

23

Audio Track of “Gangnam Style” Matrix factorization from collaborative filtering

[Oord et. al, 2013]

“user” “song” “Gangnam Style’s latent vector

Music Retrieval Results

24

Collaborative Filtering only Collaborative Filtering + Audio Content

[Oord et. al, 2013]

Content-Based Analysis

§ Advantages

– Free of cold-start and popularity bias – Highly scalable: using high-performance computing – Works for music in other media or user content as well – Can be combined with other approaches

§ Limitations

– Social context is also important: indy, idol, affilation – Do not care of music quality (e.g. level of performance), especially for user contents

25

Automatic Music Transcription (AMT)

§ Predict score information from audio

– Note information: note onset, duration, velocity – Rhythm: tempo, beat, down-beat – Chord – Structure

Zenph’s Re-performance

Zenph’s Re-performance

Entertainment / Education

29

Yousician

Score-Audio Alignment

§ Temporally align audio and score

– Dynamic time warping of AMT results as audio features

§ Applications

– Score Following – Automatic page turning – Auto-accompaniment – Performance analysis

Automatic Page Turner (JKU, Austria)

32

The Piano Music Companion (JKU, Austria)

33

Sonation’s Cadenza

Music Production

https://www.youtube.com/watch?v=RmT6MDOD3uc

Music Production

§ Adaptive Audio Effects: automatic effect control

– Loudness

• Compressor

– Pitch

• Pitch correction (e.g. auto-tune)
• Harmonizer

– Timbre

• Genre-based automatic EQ

Antares Auto-tune

Music Production

§ Singing Expression Transfer

– Given two renditions of the same piece of music – Transfer singing expressions from one voice to another

Note timing, Pitch, Dynamics

Singing Expression Transfer

Feature Extraction

Target Singing Voice Source Singing Voice Time-Scale Modification Pitch Shifting Gain

DTW Smoothing HPSS Envelope Detector Pitch Detector

Modified Singing Voice

stretching ratio smoothed stretching ratio pitch ratio gain ratio harmonic signal

Temporal Alignment Pitch Alignment Dynamics Alignment

𝑡 𝑡" 𝑡"# 𝑡"#$

Singing Expression Transfer: Demo Examples

source target all modified source 벚꽃엔딩 Let it go 취중진담

Music Production

§ Sound Sample search

– Imagine Research’s MediaMind: search sound effect sample for media production (e.g. film, drama) – Izotope’s Breaktweaker: search similar timbre of drum sounds

39

Automatic Music Composition

§ Algorithmic Composition

– An Area of Generative Art

§ Types of Algorithms

– Generative Grammar – Transition Network – Markov Model – Generic Algorithms – Neural Networks

Automatic Music Composition

§ David Cope’s EMI (Experiments in Music Intelligence) (1980s)

– Based on Style Imitation

Augmented Transition Networks

Recent Work: Automatic Music Composition

§ Flow Machine

– Style Imitation based on Markov Model – http://www.flow-machines.com/

§ Magenta

– Python Library based Deep Neural Networks (TensorFlow) – https://magenta.tensorflow.org/welcome-to-magenta

“Daddy’s car”: Sony CSL Lab’s Flow Machines

Automatic Music Composition

§ Background Music Generation: www.jukedeck.com

Automatic Music Arrangement

쿨잼(Cool Jamm) – Hum On

Musical “Process” and “Data”

“Musical” Knowledge Base “Physical” Knowledge Base Performer Composer Instrument Perception Cognition Sound Field Source Sound Temporal Control Symbolic Representation Room Listener Process Data

Music Technology: The Present

“Musical” Knowledge Base “Physical” Knowledge Base Performer Composer Instrument Perception Cognition Sound Field Source Sound Temporal Control Symbolic Representation Room Listener Process Data

Music Technology: The Future

“Musical” Knowledge Base “Physical” Knowledge Base Performer Composer Instrument Perception Cognition Sound Field Source Sound Temporal Control Symbolic Representation Room Listener Process Data