Course Introduction Juhan Nam Who We Are Instructor: Juhan Nam - - PowerPoint PPT Presentation
GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Course Introduction Juhan Nam Who We Are Instructor: Juhan Nam Associate Professor, Graduate School of Culture Technology (GSCT) Affiliated Professor, Graduate
○ Associate Professor, Graduate School of Culture Technology (GSCT) ○ Affiliated Professor, Graduate School of Artificial Intelligence (GSAI) ○ Music and Audio Computing Lab: https://mac.kaist.ac.kr/
○ Taegyun Kwon, PhD student, GSCT ○ Taejun Kim, PhD student, GSCT ○ Wonil Kim, PhD student, GSCT ○ Seunghyun Lee, MS Student, GSAI
Music in KAIST
Source: http://times.kaist.ac.kr/news/articleView.html?idxno=3835, http://times.kaist.ac.kr/news/articleView.html?idxno=3185
○ Music listening: download music tracks as compressed audio files and uncompress them into waveforms
○ Music performance: musical instrument, karaoke machine
○ Music composition and production: recording, MIDI, processing, mixing
○ Data: audio, MIDI, text (meta data) ○ Processing: audio (un)compression, sound synthesis, recording, digital audio effect, editing, and mixing
○ Spotify: 60M tracks, 40K new tracks per day, and 4B playlists (2020) ○ SoundCloud: 200M tracks (2019) ○ YouTube: 500 min videos upload per min (2020) (music tracks, music videos and performance videos)
○ Meta data is not enough to explain the “content” of music ○ Need more rich descriptions
Source: https://newsroom.spotify.com/company-info/
Naver VIBE
○ Annotate a track with about 450 music attributes
■ Genre, instruments, timbre, vocal quality, …
○ Playlists are generated using the similarity
○ Not biased by the popularity
○ Takes 20-30 mins to annotate a single track by a music expert ○ The dictionary size of music attributes is fixed
Pandora Internet Radio
○ Associate music with not only musical terms (e.g., genre, instrument, timbre) but also listening contexts (e.g. mood, time of day, location and activity)
○ Music score ○ Karaoke ○ Instrument learning game
○ Performance evaluation ○ Score following and page turning ○ Auto-accompaniment
Source: https://promusicianhub.com/yousician-review/, https://musescore.org, https://www.smule.com/
MuseScore Smule Yousician
○ Identify and separate sound sources in adverse acoustic conditions: microphone, reverberation and interfering sources ○ Detect multiple pitches from polyphonic musical instruments: e.g., piano, guitar
Source: http://jameasy.com/ko/company.html, https://magenta.tensorflow.org/onsets-frames
○ Source separation from mixed audio ○ Transcribe polyphonic music into music score or MIDI
○ Illiac Suite (String Quartet No. 4) (1957)
■ The first music score composed by an electronic computer ■ Composed using a Markov model ■ https://www.youtube.com/watch?v=n0njBFLQSk8
○ Experiments in Musical Intelligence (EMI) (1980s)
■ Style imitation using pre-composed patterns (recombinant) ■ https://www.youtube.com/watch?v=t6WeiyvAiYQ&t=52s
○ Numerous approaches in “algorithmic composition”
■ Audio/MIDI programming languages: Music-N, CSound, Max/PD, Common Music, Supercollider, Chuck ■ Rule-based or statistical models
Source: http://www.moz.ac.at/sem/lehre/lib/es/ems/hist/battisti.html
EMI: recombinant music (David Cope) The Experimental Music Studio (Lejaren Hiller and Leonard Isaacson)
○ Learn the sequential order of music data in a highly data-driven way ○ MIDI or audio generation
■ Flow Machine (Sony): https://www.flow-machines.com/ ■ Music Transformer (Google): https://magenta.tensorflow.org/music-transformer ■ Jukebox (OpenAI): https://openai.com/blog/jukebox/
Music Transformer Jukebox Flow Machine (“Hello World”: AI-composed album)
○ The conditions can be semantic, artist, lyrics, score, audio or even preference ○ Possible to create novel pieces (collaborating with human) ?
Audio Score (MIDI) Text
○ Each of them handles a different modality: image, audio, text (i.e., symbol) ○ Due to the nature of data-driven approach (or less use of domain knowledge), the advance of deep learning has been naturally applied to a wide variety of domains that use image, audio and text as a input or output data form ○ Music is one of the domains that have benefited a lot from them
○ Transform a type of data onto a more meaningful vector space (i.e., feature space) ○ The vector spaces from different modalities of data are associated with each other by their correspondence
Audio Score (MIDI) Text Image
○ Pre-Recorded videos
■ Cover the lecture part ■ Uploaded weekly in KLMS (YouTube link) ■ Students must watch videos before the weekly Zoom meeting
○ Weekly Zoom meeting
■ Focus on review, interactive Q&A and hands-on practice ■ Thursday from Week #2: 2:30-3:45 PM (Hopefully, less than 60 minutes)
○ Course introduction
○ Audio data representations
○ Machine learning review: supervised learning
○ Machine learning review: unsupervised learning
○ Chusuk (no class)
○ Convolutional neural network (CNN): music classification and tagging
○ Recurrent neural network (RNN): automatic music transcription
○ Break (no class)
○ Auto-encoder, U-net: source separation
○ Variational auto-encoder (VAE), generative adversarial network (GAN): music generation and sound synthesis
○ Auto-regressive models: music generation and sound synthesis
○ Transformer: music transcription and generation
○ Invited talk or advanced topics (TBD)
○ Invited talk or advanced topics (TBD)
○ TBD
○ Final project presentations