Measuring & Modeling Musical Expression Douglas Eck University - PowerPoint PPT Presentation

Measuring & Modeling Musical Expression Douglas Eck University of Montreal Department of Computer Science BRAMS Brain Music and Sound International Laboratory for Brain, Music and Sound Research

Overview • Task: human-realistic music performance • Challenges: • expressive timing and dynamics • generating musical variations • choosing appropriate timbres (instruments) • Today: Learning expressive timing and dynamics for the piano • Applications: music generation for film and video games • Work done in collaboration with Stanislas Lauly Douglas Eck douglas.eck@umontreal.ca 2

An interesting task... context-aware music generation VICTORIOUS FRANTIC Faster → SPOOKY RESTFUL more dangerous → Douglas Eck douglas.eck@umontreal.ca 3

VICTORIOUS l e d o m FRANTIC e c n a m r o f r e P Faster → Music Composition (from video game composer) SPOOKY RESTFUL more dangerous → Douglas Eck douglas.eck@umontreal.ca 4

VICTORIOUS l e d o m FRANTIC e c n a m r o f r e P Faster → Music Composition (from video game composer) SPOOKY RESTFUL more dangerous → Douglas Eck douglas.eck@umontreal.ca 5

VICTORIOUS l e d o m FRANTIC e c n a m r o f r e P Faster → ???? Music Composition (from video game composer) SPOOKY RESTFUL more dangerous → Douglas Eck douglas.eck@umontreal.ca 6

Audio similarity + morphing • We can predict words like “sad” and “jazzy” from audio. Resulting wordset useful for music recommendation (Eck et al. NIPS 07) • We can also morph between artists based on word vector similarity This work is a part of Sun Labs “Project Aura” recommendation framework. • Similar technique may allow us to generate a “dangerous” sound based on analysis of songs people think sound dangerous. Douglas Eck douglas.eck@umontreal.ca 7

Audio similarity + morphing • We can predict words like “sad” and “jazzy” from audio. Resulting wordset useful for music recommendation (Eck et al. NIPS 07) • We can also morph between artists based on word vector similarity This work is a part of Sun Labs “Project Aura” recommendation framework. • Similar technique may allow us to generate a “dangerous” sound based on analysis of songs people think sound dangerous. Douglas Eck douglas.eck@umontreal.ca 7

Chopin Etude Opus 10 No 3 • Many challenges: • expressive timing and dynamics • generating musical variations • choosing appropriate timbres (instruments) • Today: Learning expressive timing and dynamics for the piano

Example: Chopin Etude Opus 10 No 3 Deadpan (no expressive timing or dynamics) Human performance (Recorded on Boesendorfer ZEUS) Differences limited to: • timing (onset, length) • velocity (seen as red) • pedaling (blue shading)

What can we measure? • Repp (1989) measured note IOIs in 19 famous recordings of a Beethoven minuet (Sonata op 31 no 3) M I N U E T 1 o o o ? 0 0 6 0 0 5 0 0 . ! I I I [ I I [ I I I I I • I I I E : 3 q 5 6 7 8 9 1 8 t ! 1 • 1 3 , t q t 5 1 6 B R R N O . Grand average timing patterns of performances with repeats plotted separately. TRIO o (From B. Repp “Patterns of expressive timing in performances of a Beethoven 9 c q • - u minuet by nineteen famous pianists”,1990) R 8 8 0 - T Douglas Eck douglas.eck@umontreal.ca 10 ! 7 0 • - 0 N 6 0 0 . ! N I I I I I I I I I I I I I t7 18 19 aO Et EE E3 Eq •5 g6 •7 a8 •9 30 3 1 3 E 3 3 3 q 3 5 3 6 3 7 3 8 B R R N O . CODA 1 0 0 • * 9 0 0 . 8 0 0 - 7 • . 6 0 0 . I I I I I I I 3 9 q a q t q E q 3 q q q 5 BRR ND. F I G . 3 . G r a n d a v e r a g e t i m i n g p a t t e r n • o f f f o 1 r 5 m h u a n m c a w e n s i t , h r e • a t s p l o s t t e • p a r a t T e h l y e . l • t o n • t - o n • t i n t e r v a l i n t h e C o d a ( a r r o w ) i s 1 5 3 8 ms. I . R e p e a t s w h i c h l e d b a c k t o t h e b e g i n n i n g o f t h e M i n u e t , t h e u p b e a t was prolonged, but in bar 8B, which led into the second I t i s e v i d e n t f , i r s t , t h a t r e p e a t s o f t h s e c t i o e s a m n o f t h e m a t e M i n e r i a l h u e t , a n a d a d d i t i o n a l r i t a r d o c c u r r e d o n t h e e x t r e m e l s y i m i l a r t i m i n g p a t t e r n s T . h i p h s c o n r a s e s i s t e n - f i n ( a s l o c y f p r e c o n o - d b ) e a t . S i m i l a r l y a , u n i f o r m r i t a r d w a s f e s s i o n a k l e y b o a r d p l a y e r s w i t h r e s p e c t t o p r o d d e t a u c e i d n i l e d t i m b a r 1 i n g 6 A , w h i c h l e d b a c k t o t h e b e g i n n i n o g f t h e patterns has been noted many times s in e c o n the literature, d M i n u e t s e c begin- t i o n a , n d a n e v e n s t r o n g e r r i t a r d o c c u r r e d ning with Seashore ( 1938, p. 244). The o only n t h e systematic p h r a s e - f i n de- a ( f l i r s t a n d s e c o n d ) n o t e s o f b a r 1 6 B , v i a t i o n s o c c u r r e d i n b a r 1 a n d a t p h w h i r a s e e c h c o n n d i n g s t i t u t s ( b a r s e d t h e 8 , 1 5 - e n d o f t h e M i n u e t , w h e r e a s t h e t h i r d 16, and 23/37-24/38), where the music note constituted was, in fact, the upbeat not to the Trio and was taken i d e n t i c a a l c r o s s r e p e a t s ( s e e F i g . 1 ) : I n b a shorter. r 1 , B e Bar 15 e t h o anticipated v e n these changes, which were more a d d e d a n o r n a m e n t ( a t u r n o n E - f l a t ) i n t h e r e pronounced p e a t ( b a r 1 B in the ) , second playing of the Minuet, following w h i c h w a s s l i g h t l y d r a w n o u t b y m o s the Trio. t p i a n Similarly, i s t s I n . b a r 8 A bar , 37 anticipated the large ritard in bar 6 2 8 J . A c o u s S t . e c . A m . V , o l . 8 8 , N o . 2 , A u g u s 1 t 9 9 0 B r u n H o . R e p p E : x p r e s s i t v i m e i n i g n a B e e t h o v e m n i n u e t 6 2 8