CrossSong puzzle: Generating and unscrambling music mashups with real-time interactivity Jordan B. L. Smith, Graham Percival, Jun Kato, Masataka Goto, and Satoru Fukayama Media Interaction Group National Institute of Advanced Industrial Science and Technology (AIST), Japan 30 July 2015 Oral Session II: Content Processing of Music Audio Signals 1
1. Music, games and puzzles • Music, games and puzzles are all incredibly popular pastimes • Many activities combine two of these pursuits: • Guitar Hero and Rock Band and others combine music and games • Countless games include puzzle elements • Figured bass exercises are music puzzles • But nothing combines all three: music, games and puzzles! 2
Music + Games + Puzzles Goal: make something that combines these! 3
1. Music, games and puzzles: motivation • …but why ? (other than because it hasn’t been done) • Puzzle pleasure: pattern identification and fulfilment • Music pleasure: pattern identification and fulfilment (or denial) • Interesting to play with both at once 4
2. CrossSong puzzle • Design goals: • Puzzle should require musical thinking • Most “music” puzzles are just puzzles that have music • Puzzle should be fun to listen to • An engaging musical experience • Music should be continuous / have a beat 5
2. CrossSong puzzle Goal: Unscramble the tiles 1 2 3 4 13 6 8 7 5 6 7 8 14 4 12 2 9 10 11 12 10 1 3 11 13 14 15 9 15 5 6
2. CrossSong puzzle Construction: Parts of puzzle fit in interlocking rows and columns Constraint at 1 2 3 4 T R A M intersection for crosswords: letter in 5 W O V E Let’s construct each word must be the same this with music 6 I D O L L CrossSong constraint: sounds 7 G E N T must make pleasing mashup! 7
2. CrossSong puzzle User: Input: 8 songs System: Step 1: extract 4-measure clips (All music examples from RWC Popular Database. See M. Goto, H. Hashiguchi, T. Nishimura and R. Oka. 2002. RWC Music Database: Popular, classical, and jazz music databases. In Proceedings of ISMIR. 287–8.) 8
2. CrossSong puzzle User: Input: 8 songs System: “Koino Ver.2.4” by Eves RWC Pop #5 Step 1: extract 4-measure clips 9
2. CrossSong puzzle User: Input: 8 songs System: “Koino Ver.2.4” by Eves Step 1: extract 4-measure clips “Life Is What You Make It To Be” by Donna Burke RWC Pop #89
2. CrossSong puzzle User: Input: 8 songs System: “Koino Ver.2.4” by Eves Step 1: extract 4-measure clips Step 2: match mashed-up with tempo and pitch “Life Is What You Make It To Be” by Donna Burke 11
2. CrossSong puzzle User: a1 a2 a3 a4 Input: 8 songs b1 b2 b3 b4 System: c1 c2 c3 c4 Step 1: extract d1 d2 d3 d4 4-measure clips e1 e2 e3 e4 Step 2: match tempo and pitch f1 f2 f3 f4 g1 g2 g3 g4 Step 3: arrange songs in grid h1 h2 h3 h4 12
2. CrossSong puzzle User: Input: 8 songs b1 b2 b3 b4 a1 a2 a3 a4 System: c1 c2 c3 c4 Step 1: extract d1 d2 d3 d4 4-measure clips e1 e2 e3 e4 Step 2: match tempo and pitch f1 f2 f3 f4 g1 g2 g3 g4 Step 3: arrange songs in grid h1 h2 h3 h4 13
2. CrossSong puzzle User: Input: 8 songs b1 b2 b3 b4 a1 a2 a3 a4 e1 System: c1 c2 c3 c4 Step 1: extract e2 d1 d2 d3 d4 4-measure clips e3 Step 2: match tempo and pitch f1 f2 f3 f4 e4 g1 g2 g3 g4 Step 3: arrange songs in grid h1 h2 h3 h4 14
2. CrossSong puzzle User: Input: 8 songs a1 a2 a3 a4 e1 System: b1 b2 b3 b4 Step 1: extract e2 4-measure clips c1 c2 c3 c4 e3 Step 2: match tempo and pitch f1 f2 f3 f4 d1 d2 d3 d4 e4 g1 g2 g3 g4 Step 3: arrange songs in grid h1 h2 h3 h4 15
2. CrossSong puzzle User: Input: 8 songs Doing That Thing Life Is What You Make It a1 a2 a3 a4 How Deep Is Your Love? e1 f1 g1 h1 Eien No Replica System: Spice Of Life b1 Don’t Lie To Me b2 b3 b4 Step 1: extract e2 f2 g2 h2 4-measure clips c1 Koino Ver.2.4 c2 c3 c4 f3 g3 h3 e3 Step 2: match tempo and pitch d1 Magic In Your Eyes d2 d3 d4 e4 f4 g4 h4 Step 3: arrange songs in grid 16
2. CrossSong puzzle Doing That Thing Don’t Lie To Me a1 a2 a3 a4 e1 f1 g1 h1 Koino Ver.2.4 b1 b2 b3 b4 e2 f2 g2 h2 Magic In Your Eyes c1 c2 c3 c4 Spice Of Life f3 g3 h3 e3 How Deep Is Your Love? d1 d2 d3 d4 e4 f4 g4 h4 Eien No Replica Life Is What You Make It To Be 17
2. CrossSong puzzle Solve puzzle by listening , not looking a1 a2 a3 a4 e1 f1 g1 h1 b1 b2 b3 b4 e2 f2 g2 h2 c1 c2 c3 c4 f3 g3 h3 e3 d1 d2 d3 d4 e4 f4 g4 h4 18
2. CrossSong puzzle Solve puzzle by listening , not looking 19
2. CrossSong puzzle: Demo Solver can listen to isolated tracks before starting Music from RWC Popular Database: #1, 2, 4, 5, 81, 83, 95, 97. See Goto et. al 2002
2. CrossSong puzzle: Demo Basic action Solver can listen to isolated tracks before starting #1: swap tiles Music from RWC Popular Database: #1, 2, 4, 5, 81, 83, 95, 97. See Goto et. al 2002
2. CrossSong puzzle: Demo Basic action #2: choose row or column to play next Music from RWC Popular Database: #1, 2, 4, 5, 81, 83, 95, 97. See Goto et. al 2002
2. CrossSong puzzle: confirming progress • Visual feedback: • Row locks in place after full hearing • Visual feedback confirms progress, but solver must listen to music to make progress • When entire puzzle correct, instant win • Not likely to happen by chance (6 billion possible tile arrangements) 23
2. CrossSong puzzle: confirming progress • Audio feedback: • Balance of mash-up in a single tile depends on tile’s correct neighbours d2 d4 c4 c1 f4 h4 h3 e3 a1 a2 a3 d3 e1 f1 g1 g4 b2 b4 b3 c3 f2 g3 h2 g2 c2 b1 d1 a4 f3 e2 e4 h1 24
2. CrossSong puzzle: confirming progress • Audio feedback: • Balance of mash-up in a single tile depends on tile’s correct neighbours Across song part: a2 d2 d4 c4 c1 f4 h4 h3 e3 a1 a2 a3 d3 e1 f1 g1 g4 b2 b4 b3 c3 f2 g3 h2 g2 c2 b1 d1 a4 f3 e2 e4 h1 25
2. CrossSong puzzle: confirming progress • Audio feedback: • Balance of mash-up in a single tile depends on tile’s correct neighbours Down song part: f1 d2 d4 c4 c1 f4 h4 h3 e3 a1 a2 a3 d3 e1 f1 g1 g4 b2 b4 b3 c3 f2 g3 h2 g2 c2 b1 d1 a4 d1 b1 f3 e4 e2 e4 e2 h1 26
2. CrossSong puzzle: confirming progress • Audio feedback: • Balance of mash-up in a single tile depends on tile’s correct neighbours Normal balance: 50/50 a2/f1 d2 d4 c4 c1 f4 h4 h3 e3 a1 a2 a3 d3 e1 f1 g1 g4 b2 b4 b3 c3 f2 g3 h2 g2 c2 b1 d1 a4 d1 b1 f3 e4 e2 e4 e2 h1 27
2. CrossSong puzzle: confirming progress • Audio feedback: • Balance of mash-up in a single tile depends on tile’s correct neighbours One neighbour in d2 d4 c4 c1 f4 h4 h3 e3 playing direction a1 a2 a3 d3 correct → audio e1 f1 g1 g4 balance favours b2 b4 b3 c3 correct direction f2 g3 h2 g2 ( f1 louder in c2 b1 d1 a4 d1 b1 playback of f3 e4 e2 e4 e2 h1 column) 28
2. CrossSong puzzle: confirming progress • Audio feedback: • Balance of mash-up in a single tile depends on tile’s correct neighbours Both neighbours d2 d4 c4 c1 f4 h4 h3 e3 in playing a1 a2 a3 d3 direction correct e1 f1 g1 g4 → audio isolated b2 b4 b3 c3 (all a2 in f2 g3 h2 g2 playback of row) c2 b1 d1 a4 d1 b1 f3 e4 e2 e4 e2 h1 29
Problem: how do we create puzzles? • Taking random excerpts, the tiles will sound terrible! • Designing a puzzle by hand is too difficult 30
3. Puzzle creation algorithm • “Mashability” estimation algorithm (Davies et al. 2014) • Estimates the quality of the match between two aligned music sequences • Harmonic match: • Rhythmic match: • Spectral balance: 31 • M. E. P . Davies, P . Hamel, K. Yoshii and M. Goto. 2014. AutoMashUpper: Automatic creation of multi-song music mashups. IEEE Transactions on Audio, Speech, and Language Processing 22 (12). 1726–37.
3. Puzzle creation algorithm • “Mashability” estimation algorithm (Davies et al. 2014) • Estimates the quality of the match between two aligned music sequences • Harmonic match: • Rhythmic match: • Spectral balance: 32 • M. E. P . Davies, P . Hamel, K. Yoshii and M. Goto. 2014. AutoMashUpper: Automatic creation of multi-song music mashups. IEEE Transactions on Audio, Speech, and Language Processing 22 (12). 1726–37.
3. Puzzle creation algorithm 1. Detect section breaks • We will only consider 4-measure excerpts at the beginning of a section 2. Compute the mashability of all excerpts with each other • Automatic mashability estimation (Davies et al. 2014) 3. Search space of arrangements to find the one with the greatest estimated mashability 4. Apply pitch shifting / time stretching / loudness matching 33
3. Puzzle creation algorithm 1 2 3 … • Each song: ~ 15 segments to choose from +3 • Each segment: can be … transposed up to 3 semitones –3 • Each set of segments: can be arranged in 8!/2 ways Total number of solutions to check: 15 8 x7 8 x8! / 2 ~10 20 34
Recommend
More recommend
