Physical Modeling meets Machine Learning: Teaching Bow Control to a - - PowerPoint PPT Presentation

▶
physical modeling meets machine learning teaching bow
SMART_READER_LITE
LIVE PREVIEW

Physical Modeling meets Machine Learning: Teaching Bow Control to a - - PowerPoint PPT Presentation

Feb 20, 2024 854 likes •1.12k views

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Graham Percival , Nicholas Bailey , George Tzanetakis

slide-1
SLIDE 1

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist

Graham Percival∗, Nicholas Bailey∗, George Tzanetakis†

∗ School of Engineering, University of Glasgow, UK † Department of Computer Science, University of Victoria, Canada

http://percival-music.ca/vivi.html

This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 License.

slide-2
SLIDE 2

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist 1 / 12

Teaching Bow Control to a Virtual Violinist

1 Introduction

Music performance with Vivi, the Virtual Violinist

2 Performing on a virtual instrument

Generating sound: Physical modeling of a violin Pedagogical inspiration for physical parameters

3 Intelligent control loop

Intelligent feedback control of bow force Automatically determining other parameters

4 Making music

Musical performance style Conclusion and future work

slide-3
SLIDE 3

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Introduction 2 / 12

Music performance with Vivi, the Virtual Violinist

Music notation

= 96

  • f

Virtual violinist (Controller) Violin physical model (System)

Note data Physical parameters

Audio analysis and SVM classifiers (Sensor) Audio file Video file

Audio signal (optional) Audio signal Select classifier Sound quality Input Output Control

Automatic processing (intelligent feedback control)

slide-4
SLIDE 4

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Introduction 3 / 12

Music example: “black-box testing”

Input Output

  • mp
  • III
  • = 96
  • p
  • f
  • II

II

  • 4
  • mf
  • p
  • p
  • 6
  • tip
  • mp
  • mb
  • pizz.

f

  • 4

3

= 120

  • II
  • II
  • II
  • II
  • II

10

  • f
  • mp

= 88

  • III
  • p

II

  • arco

lh

  • (pdf produced with GNU LilyPond,

MusicXML input also possible)

Video: black-box.mpeg

slide-5
SLIDE 5

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 4 / 12

Generating sound: Physical modeling of a violin

No recordings of violin performance; we use physics [1]

  • Wave equation for a stiff string with modal dampening

ρL ∂2y(x, t) ∂t2 −T ∂2y(x, t) ∂x2 +EI ∂4y(x, t) ∂x4 +RL(ω)∂y(x, t) ∂t = F(x, t)

[1] M. Demoucron, “On the control of virtual violins: Physical modelling and control of bowed string instruments,” Ph.D. dissertation, IRCAM, Paris, 2008

slide-6
SLIDE 6

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 4 / 12

Generating sound: Physical modeling of a violin

No recordings of violin performance; we use physics [1]

  • Wave equation for a stiff string with modal dampening

ρL ∂2y(x, t) ∂t2 −T ∂2y(x, t) ∂x2 +EI ∂4y(x, t) ∂x4 +RL(ω)∂y(x, t) ∂t = F(x, t)

Implemented as a C++ library, published under GNU GPLv3+ Input parameters

  • Violin string number s
  • Left-hand finger position x1
  • Bow-bridge distance x0, velocity vb, force Fb

[1] M. Demoucron, “On the control of virtual violins: Physical modelling and control of bowed string instruments,” Ph.D. dissertation, IRCAM, Paris, 2008

slide-7
SLIDE 7

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 5 / 12

Music performance with Vivi, the Virtual Violinist

Music notation

= 96

  • f

Virtual violinist (Controller) Violin physical model (System)

Note data Physical parameters

Audio analysis and SVM classifiers (Sensor) Audio file Video file

Audio signal (optional) Audio signal Select classifier Sound quality Input Output Control

slide-8
SLIDE 8

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 6 / 12

Pedagogical inspiration for physical parameters

Pedagogical inspiration

  • Suzuki violin book 1
  • Treat Vivi like a beginning student
slide-9
SLIDE 9

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 6 / 12

Pedagogical inspiration for physical parameters

Pedagogical inspiration

  • Suzuki violin book 1
  • Treat Vivi like a beginning student

Most parameters can come from sheet music and pedagogy

  • String s, finger x1: printed note
  • Bow-bridge distance x0: dynamic “bow lanes” or “Kreisler Highway”
  • Bow velocity vb: teacher saying “use half bow” and giving tempo
slide-10
SLIDE 10

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 6 / 12

Pedagogical inspiration for physical parameters

Pedagogical inspiration

  • Suzuki violin book 1
  • Treat Vivi like a beginning student

Most parameters can come from sheet music and pedagogy

  • String s, finger x1: printed note
  • Bow-bridge distance x0: dynamic “bow lanes” or “Kreisler Highway”
  • Bow velocity vb: teacher saying “use half bow” and giving tempo

Bow force Fb from SVM classifiers

1 not audible: needs a lot more bow force (example) 2 “whispy”: needs a little more bow force (example) 3 acceptable: no change (example) 4 “harsh”: needs less bow force (example) 5 not recognizable: needs much less bow force (example)

slide-11
SLIDE 11

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 7 / 12

Interactive training

Basic training: only 32 files (bad example) After interactive training: 203 files (good example) ≈ 4 hours to be fully trained (including calculations)

slide-12
SLIDE 12

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 8 / 12

Intelligent feedback control of bow force

Central controller Select string and finger position Initial parameter database Violin physical model Audio analysis and SVM classifiers Adjust force

Note data Pitch s x1 Dynamic Select classifier Sound quality Fb x0 vb K Fb audio signal audio signal s, x1, x0, vb, Fb Note setup Control Output

slide-13
SLIDE 13

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 8 / 12

Intelligent feedback control of bow force

Central controller Select string and finger position Initial parameter database Violin physical model Audio analysis and SVM classifiers Adjust force

Note data Pitch s x1 Dynamic Select classifier Sound quality Fb x0 vb K Fb audio signal audio signal s, x1, x0, vb, Fb Note setup Control Output

c ∈ {1, 2, 3, 4, 5} 16 classifiers (4 strings x 4 dynamics)

slide-14
SLIDE 14

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 8 / 12

Intelligent feedback control of bow force

Central controller Select string and finger position Initial parameter database Violin physical model Audio analysis and SVM classifiers Adjust force

Note data Pitch s x1 Dynamic Select classifier Sound quality Fb x0 vb K Fb audio signal audio signal s, x1, x0, vb, Fb Note setup Control Output

c ∈ {1, 2, 3, 4, 5} 16 classifiers (4 strings x 4 dynamics)

slide-15
SLIDE 15

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 8 / 12

Intelligent feedback control of bow force

Central controller Select string and finger position Initial parameter database Violin physical model Audio analysis and SVM classifiers Adjust force

Fb ← FbK (3−c) Note data Pitch s x1 Dynamic Select classifier Sound quality Fb x0 vb K Fb audio signal audio signal s, x1, x0, vb, Fb Note setup Control Output

c ∈ {1, 2, 3, 4, 5} 16 classifiers (4 strings x 4 dynamics)

slide-16
SLIDE 16

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 8 / 12

Intelligent feedback control of bow force

Central controller Select string and finger position Initial parameter database Violin physical model Audio analysis and SVM classifiers Adjust force

Fb ← FbK (3−c) Note data Pitch s x1 Dynamic Select classifier Sound quality Fb x0 vb K Fb audio signal audio signal s, x1, x0, vb, Fb Note setup Control Output

c ∈ {1, 2, 3, 4, 5} 16 classifiers (4 strings x 4 dynamics) pre-calculate

slide-17
SLIDE 17

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 9 / 12

Automatically determining K

Cost of a candidate K

1 Play a simple musical pattern 2 Get list C of judgements c 3 Split C into sublists Ai based

  • n c changing from below to

above 3 (and vice versa)

4 Calculate

cost =

|A|

  • i
  • c∈Ai

(3 − c)2

5 Repeat 12 times and find the

inter-quartile geometric mean

slide-18
SLIDE 18

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 9 / 12

Automatically determining K

Cost of a candidate K

1 Play a simple musical pattern 2 Get list C of judgements c 3 Split C into sublists Ai based

  • n c changing from below to

above 3 (and vice versa)

4 Calculate

cost =

|A|

  • i
  • c∈Ai

(3 − c)2

5 Repeat 12 times and find the

inter-quartile geometric mean Sample force factors of the D string played mf 1 1.05 1.1 1022 1025 1028 Force factor K Cost

Raw data Geometric means

slide-19
SLIDE 19

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 10 / 12

Automatically determining initial bow force Fb

Cost of a candidate initial Fb

1 Play a simple musical pattern 2 Get list C of judgements c 3 Split of C into list A (note

attack): attack is over when 0.5 > 1 N

  • c∈LN

(3 − c)2

(LN = previous N values of C)

4 Calculate

cost =

  • c∈A

(3 − c)2

5 Repeat 4 times and find the

inter-quartile geometric mean

slide-20
SLIDE 20

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 10 / 12

Automatically determining initial bow force Fb

Cost of a candidate initial Fb

1 Play a simple musical pattern 2 Get list C of judgements c 3 Split of C into list A (note

attack): attack is over when 0.5 > 1 N

  • c∈LN

(3 − c)2

(LN = previous N values of C)

4 Calculate

cost =

  • c∈A

(3 − c)2

5 Repeat 4 times and find the

inter-quartile geometric mean Sample initial forces of the D string played mf, open string 2 4 101 102 103 Initial Fb Cost

Raw data Geometric means

slide-21
SLIDE 21

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Making music 11 / 12

Musical performance style

Some notation is unambiguous

  • Slur = don’t change bow direction
slide-22
SLIDE 22

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Making music 11 / 12

Musical performance style

Some notation is unambiguous

  • Slur = don’t change bow direction

Some notation is open to stylistic interpretation

  • Staccato = shorter duration; but by how much?
  • For now, we use hard-coded values.

“Make it sound like a book 1 Suzuki student.”

slide-23
SLIDE 23

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Making music 11 / 12

Musical performance style

Some notation is unambiguous

  • Slur = don’t change bow direction

Some notation is open to stylistic interpretation

  • Staccato = shorter duration; but by how much?
  • For now, we use hard-coded values.

“Make it sound like a book 1 Suzuki student.”

Short-term: let humans specify/adjust stylistic interpretation

  • Most beginning music students (age 4–8 years) simply follow

instructions from their teachers

  • Humans should give high-level judgements (≈ 1–10 Hz); computers

should do low-level processing (control parameters at ≈ 172 Hz)

slide-24
SLIDE 24

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Making music 12 / 12

Conclusion and future work

Vivi performs music with a similar skill level to Suzuki violin students with 1 year of experience

  • Intelligent feedback control
  • SVM training requires ≈ 4 hours of feedback
  • Other parameters are determined from the SVM classifiers
slide-25
SLIDE 25

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Making music 12 / 12

Conclusion and future work

Vivi performs music with a similar skill level to Suzuki violin students with 1 year of experience

  • Intelligent feedback control
  • SVM training requires ≈ 4 hours of feedback
  • Other parameters are determined from the SVM classifiers

Open source, published under the GNU GPLv3+:

  • Artifastring (“artificial fast string”): violin physical modeling library,

C++ with SWIG bindings

  • Vivi, the Virtual Violinist: sheet music → control loop → audio and

video files, python and C++

  • http://percival-music.ca/vivi.html
slide-26
SLIDE 26

Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Making music 12 / 12

Conclusion and future work

Vivi performs music with a similar skill level to Suzuki violin students with 1 year of experience

  • Intelligent feedback control
  • SVM training requires ≈ 4 hours of feedback
  • Other parameters are determined from the SVM classifiers

Open source, published under the GNU GPLv3+:

  • Artifastring (“artificial fast string”): violin physical modeling library,

C++ with SWIG bindings

  • Vivi, the Virtual Violinist: sheet music → control loop → audio and

video files, python and C++

  • http://percival-music.ca/vivi.html

Future work: apply these techniques to “continuous excitation” instruments in STK (Synthesis ToolKit in C++)

  • Clarinet, saxophone, flute, brass