[PPT] - Clova Music: DJ AI (Adrian Kim), M.S. Clova AI PowerPoint Presentation

SLIDE 1

Clova Music: 똑똑한 DJ같은 AI비서

Clova AI Research(CLAIR), Naver Corp. 김정명 (Adrian Kim), M.S.

SLIDE 2

Clova: Cloud-based Virtual Assistant

General Purpose AI platform

SLIDE 3

Clova: Cloud-based Virtual Assistant

https://clova.ai

SLIDE 4

Clova: Cloud-based Virtual Assistant

SLIDE 5

Clova Music

The biggest need from a speaker would be MUSIC

= Music Listening Platform?

SLIDE 6

Clova Music

Intelligent music recommendation service of Clova
Aims to be a human DJ-like curator
Powered with NAVER/LINE music user/content data

SLIDE 7

Introducing the Music Domain

SLIDE 9

Popular Domains...

SLIDE 10

Audio domain data

+

SLIDE 11

Wave

Basic data form is 16 bit integer
You can normalize to [-1, 1]
1D vector of samples
16kHz, 22050Hz, ...
Very information inefficient

For 16kHz, 30 seconds = 480k datapoints! Audio Domain Data

SLIDE 12

Spectrograms

Audio Domain Data Expressive, has more information!

SLIDE 13

Mel-spectrograms

Mel Filter banks frequency bins > 1k Audio Domain Data mel bins = 80, 96, 128

SLIDE 14

Mel-spectrograms

Mel-spectrogram filter distributions give relative focus on lower frequency bins

Image from Choi, et. al. 16 Audio Domain Data

SLIDE 15

Transformation between data types

If complex, inverse stft If only magnitude, Griffin-Lim algorithm stft mel filter bank irreversable WavenetVocoder (Shen et al. 17) Audio Domain Data (128, 2584) (1025, 2584) (1323000,) =2648600 =330752

SLIDE 16

Dirty Labels Low efficiency Must hear to evaluate Convoluted Multiple sources Too large

Audio Data

Issues

Storage problem Memory problem Low quality, weakly labeled (Choi et al. 2017) Takes a lot of time for high quality Not much open data Information per data point is very small

SLIDE 17

Comparing Simple Tasks

MNIST GTZAN Storage 45MB 1.2GB Data pairs 60000 1000 (30 second) Classes 10 digits 10 genres (100 each) Preprocessing Fast Slow Testing Easy Hard Issues

SLIDE 18

Comparing Speech and Music

Bad Boy – Red Velvet News Speech Audio Short, Single source Long, Multiple source Issues

SLIDE 19

Example Baselines

SLIDE 20

What kind of problems can we solve?

Genre/Artist Classification
Automatic Tagging
Music generation
Style transfer
Source separation
Onset detection
Sound embedding
Beat tracking
and more...!

LSTM

e el ion g er n ion

LSTM LSTM LSTM

tional ers

LSTM output Attention-weighted

Attention (softmax)

er

Element-wise multiplication

ected Convolution & pooling Channel summation er

SLIDE 21

Autotagging with Convnets

Input: mel-spectrogram (MSD dataset)
Output: Tags (50 top tags)

Automatic tagging using deep convolutional neural networks, ISMIR 16, Choi et. al

https://github.com/keunwoochoi/music-auto_tagging-keras

2D Convs

SLIDE 22

Note: Filter design in CNNs

2D convs 1D convs Slow training Fast training Local structure in freq Frequencies are discrete nxm filters, 1 channel nx1 filters, m channels

SLIDE 23

Auto Music Transcription with Deep Complex Networks

Input: Spectrogram complex output
Change network components (batchnorm, initialization, activations, convolution)

to match complex domain

Real: real and imaginary values as separate channels complex: as suggested

Deep Complex Networks, Trabelsi et al., To appear at ICLR18

SLIDE 24

WaveNet for TTS

Input: wav format data

Image from https://kakalabblog.wordpress.com/2017/07/18/wavenetnsynth-deep-audio-generative-models/

WaveNet: A Generative Model for Raw Audio, Oord et al., https://arxiv.org/pdf/1609.03499.pdf

SLIDE 25

Industries focusing on Music Research

and more!

SLIDE 26

NSynth: Encoding sounds with Wavenet Autoencoder

https://magenta.tensorflow.org/nsynth

Wavenet based model made by Magenta to produce a neural synthesizer
Latent embeddings(Z) from various sounds made by the model can be used to

produce new sounds

New dataset with instrument, pitch, etc. tags on individual sounds

SLIDE 27

Performance RNN

Trained by Yamaha e-Piano Competition dataset
Midi of 1400+ piano performances
Magenta used LSTMs to predict from 388 events occuring during the timeline

Generated example

https://magenta.tensorflow.org/performance-rnn

SLIDE 28

Discover Weekly

Spotify’s weekly personalized recommendation service
Collaborative Filtering
NLP modeling
Audio modeling

http://benanne.github.io/2014/08/05/spotify-cnns.html#contentbased http://blog.galvanize.com/spotify-discover-weekly-data-science/

SLIDE 29

Any questions?

onto part 2..

SLIDE 30

Clova Music Recommendation System

SLIDE 31

Recommendation in Clova Music

User logs as main data, content data hybrid is possible
Large and sparse online data
Topics:
User log analysis
Music semantic embedding learning
Collaborative filtering with matrix factorization

SLIDE 32

Top queries with Music

노래 틀어줘
자장가 틀어줘
동요 틀어줘
신나는 노래 틀어줘
조용한 노래 틀어줘
핑크퐁 노래 틀어줘
아이유 노래 틀어줘
클래식 틀어줘
분위기 좋은 음악 틀어줘
잔잔한 음악 틀어줘
발라드 틀어줘
Artists > Tracks
Genre, mood, themes > Artists
JUST PLAY > Genres

* Reported at 2017 Oct.

SLIDE 33

Device Usage Patterns

5 10 15 20 25 NAVER_APP NAVER_PC WAVE CLOVA_APP

* Reported at 2017 Oct.

SLIDE 34

가요 기능성음악 팝 동요 OST 클래식 재즈 종교음악 일렉트로… 락 힙합 기타 NAVER_APP WAVE

Device Usage Patterns

* Reported at 2017 Oct.

SLIDE 35

Device Usage Patterns

* Reported at 2017 Oct.

Playing ratio Artist

Artists / Play count ratio
Long tail distribution
Distribution itself is not so

different...

SLIDE 36

Device Usage Patterns

* Reported at 2017 Oct.

Playing ratio Artist

WAVE NAVER MUSIC APP

핑크퐁 EXO 아이유 아이유 동요 젝스키스 동요 방탄소년단 EXO 뉴이스트 뉴이스트(NU`EST) 윤종신 Wanna One 별하나 동요 윤종신 이루마 우원재 오르골뮤직 볼빨간사춘기 볼빨간사춘기 뉴이스트 W 젝스키스 황치열 트니트니 헤이즈 헤이즈 선미 성시경 WINNER 힐링피아노 자장가

SLIDE 37

Implication

Paradigm shift in terms of music consumption on AI speaker devices
New market
Kids, New parents
Lean-out music, lounge music
Classic, Jazz
Music Recommendation takes an important role on AI assistant platforms

SLIDE 38

Recommendation Challenges

Lack of well-defined meta data Personalized Playlists Musical Semantic Embedding Multimodal Semantic Embedding

SLIDE 39

Music Semantic Embedding

Mapping tracks, artists, and words to the same embedding space
Word2Vec
Feature learning
Usages
Item similarities
Used as features

가을 신나는

Lack of well-defined meta data

Semantic Embedding

SLIDE 40

Word2Vec with Tagged playlists

JAMM playlists
User-created playlists in Naver Music
About 72,000 playlists
Keywords from tags
Artists from tracks
Treat trackIds as ”words” within a playlist

Lack of well-defined meta data

Semantic Embedding

SLIDE 41

That song in the charts

벚꽃엔딩/버스커버스커

Semantic Embedding

SLIDE 42

Multimodal Semantic Embedding

We would want to model different playlists for different personalities
Query: 밤편지

< 밤편지_2 > < 밤편지_1 > Personalized Playlists

Semantic Embedding

SLIDE 43

Embedding with session data

User playing sequence as document!
We use multimodal word distributions

formed from Gaussian distributions

Ben Athiwaratkun and Andrew Gordon Wilson, Multimodal Word Distributions, 2017

Personalized Playlists

Semantic Embedding

SLIDE 44

Collaborative Filtering

Most popular method: Matrix Factorization

SLIDE 45

Matrix Factorization for Personalized Recommendation

Basic MF objective
Select tracks and artists that user prefers when generating a playlist
Simple, but hard to apply
Sparsity
Overfitting / Underfitting
Hard to evaluate (need real feedback, not rmse!)
Combining with other models

Collaborative Filtering

SLIDE 46

What can we do?

Learning in 2 phases
Long term: batch learning
Short term: online learning
Negative sampling
When doing negative sampling, consider item distribution
Remove abusing users
Over clicking users
Top 100 only users

Collaborative Filtering

SLIDE 47

Remaining Challenges

Conventional problems
Sparsity
Top 100 songs
Cold-start problems
Explanatory recommendation
Music Recommendation for AI Speakers
Interaction
Lean-in / Lean-back
Personalizing level (Familiar vs New)

SLIDE 48

Music Modeling

SLIDE 49

Music Modeling

Audio data as main data
Topics:
Representation Vector Extraction (Park et al. 17)
Music Emotion Recognition (Jeon et al. 17)
Music Highlight Extraction (Ha et al. 17)
Automatic DJ mix Generation (Kim et al. 17)

SLIDE 50

Extracting representation vectors by Artist Classification

Artists are complete information for music: Very Rare!
Any data can be used

DeepArtistID Using more artists increase the representation quality ArtistNet

Representation Learning Using Artist Labels for Audio Classification Tasks, Park et al., MIREX17 Challenge Representation Learning of Music Using Artist Labels, Parketal.,https://arxiv.org/abs/1710.06648

SLIDE 51

Using extracted features

Transfer learning for many tasks such as genre, mood classification
MIREX 17 Challenge for Mood prediction 1st place

ArtistNet

Representation Learning of Music Using Artist Labels, Parketal.,https://arxiv.org/abs/1710.06648 Representation Learning Using Artist Labels for Audio Classification Tasks, Park et al., MIREX17 Challenge

SLIDE 52

Visualization

ArtistNet

Representation Learning Using Artist Labels for Audio Classification Tasks, Park et al., MIREX17 Challenge Representation Learning of Music Using Artist Labels, Parketal.,https://arxiv.org/abs/1710.06648

SLIDE 53

Querying (artist level)

ArtistNet

Representation Learning Using Artist Labels for Audio Classification Tasks, Park et al., MIREX17 Challenge Representation Learning of Music Using Artist Labels, Parketal.,https://arxiv.org/abs/1710.06648

SLIDE 54

Querying (song level)

bob marley and the wailers– three little birds
dennis brown – tribulation
junior murvin – police and theives
norah jones – dont know why
dionne warwick – walk on by
jewel – enter from the east

ArtistNet

Representation Learning Using Artist Labels for Audio Classification Tasks, Park et al., MIREX17 Challenge Representation Learning of Music Using Artist Labels, Parketal.,https://arxiv.org/abs/1710.06648

SLIDE 55

Music Emotion Recognition

Music Emotion Recognition via End-to-End Multimodal Neural Networks,Jeon etal.,RecSys17

SLIDE 56

4 / 30

Multimodal approach

Task
Predict given track’s polarity (positive/negative)
Data – Naver Music Polarity Emotion Dataset
7484 tracks (Pos: Neg = 1:1)
Polarity Emotion label
Lyrics (27496, 400) word vectors
Mel spectrograms (128 mel bins, 0.06s per frame)

Music Emotion Recognition

Music Emotion Recognition via End-to-End Multimodal Neural Networks,Jeon etal.,RecSys17

SLIDE 57

Results

Using both lyrics AND audio enhance the accuracy a lot.

Data Model Accuracy Audio CNN 0.6479 RNN 0.6303 MCRN 0.6619 Lyrics CNN 0.7815 RNN 0.7716 Both MCRN, CNN 0.8046

Music Emotion Recognition

Music Emotion Recognition via End-to-End Multimodal Neural Networks,Jeon etal.,RecSys17

SLIDE 58

Lyric Analysis

Music Emotion Recognition

Music Emotion Recognition via End-to-End Multimodal Neural Networks,Jeon etal.,RecSys17

SLIDE 59