Bowing the violin A case study for auditory-motor pattern modeling - - PowerPoint PPT Presentation

▶
bowing the violin
SMART_READER_LITE
LIVE PREVIEW

Bowing the violin A case study for auditory-motor pattern modeling - - PowerPoint PPT Presentation

Jul 12, 2023 40 likes •744 views

Bowing the violin A case study for auditory-motor pattern modeling in the context of music performance Quim Llimona Torras Advisor: Esteban Maestre Quim Llimona, 2014 Introduction | Experiment | Data | Features | Database | Analysis |

slide-1
SLIDE 1

Quim Llimona, 2014

A case study for auditory-motor pattern modeling in the context of music performance

Quim Llimona Torras

Advisor: Esteban Maestre

Bowing the violin

slide-2
SLIDE 2

Outline

Introduction Experimental design Data acquisition Feature extraction Database construction Preliminary analysis Conclusion

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

2

slide-3
SLIDE 3

Score vs audio

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

3 Quim Llimona, 2014

slide-4
SLIDE 4

Score vs audio

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

3 Quim Llimona, 2014

slide-5
SLIDE 5

Performance encoding

Performer MusicalScore Instrument MusicalSound

IntendedMusicalMessage NoteEventSequence InstrumentalGesture ControlParameters PerceivedMusicalMessage AudioPerceptualFeatures ContinuousNature HighDimensionality DiscreteNature LowDimensionality ContinuousNature LowDimensionality

Many omit the instrumental gesture step

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

4 Quim Llimona, 2014

slide-6
SLIDE 6

MUSMAP Marie Curie IOF action

’s the ¡musician’s ¡bowing ¡ ’s ¡ will ¡ to ¡ expand ¡ his ¡ career ¡

This project is part of the first phase of MUSMAP

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

5 Quim Llimona, 2014

slide-7
SLIDE 7

Objectives

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

6

General

Define methodology and setup Provide software and knowledge

Specific

Design and record experiments Implement bowing acquisition and extraction Process and build a database Upload to repovizz Perform preliminary analysis

slide-8
SLIDE 8

Context: the violin

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

7

slide-9
SLIDE 9

Parts of the violin

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

8

scroll nut bridge fingerboard tailpiece plate

tip hair ribbon frog

slide-10
SLIDE 10

Violin acoustics (i)

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

9 http://www.phys.unsw.edu.au/jw/Bows.html

slide-11
SLIDE 11

Violin acoustics (i)

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

9 http://www.phys.unsw.edu.au/jw/Bows.html

slide-12
SLIDE 12

Violin acoustics (i)

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

10 https://www.youtube.com/watch?v=KPpBvHXYWz4

slide-13
SLIDE 13

Violin acoustics (i)

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

10 https://www.youtube.com/watch?v=KPpBvHXYWz4

slide-14
SLIDE 14

Violin acoustics (ii)

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

11 https://www.youtube.com/watch?v=6JeyiM0YNo4

slide-15
SLIDE 15

Violin acoustics (ii)

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

11 https://www.youtube.com/watch?v=6JeyiM0YNo4

slide-16
SLIDE 16

Violin acoustics (ii)

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

11 https://www.youtube.com/watch?v=6JeyiM0YNo4

Helmholtz regime

slide-17
SLIDE 17

Control parameters (i)

Bow velocity: Controls amplitude Bow force (or pressure): Controls high frequencies Bow-bridge distance: Controls both Others: Position, tilt, skew, inclination

Quim Llimona, 2014 12

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-18
SLIDE 18

Control parameters (ii)

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

13

This is the Schelleng diagram

Sounding point (relative bow-bridge distance) Bow pressure playable region

slide-19
SLIDE 19

Experimental design

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

14 Quim Llimona, 2014

towards joint modeling of auditory and motor spaces

slide-20
SLIDE 20

Player-instrument matrix

Quim Llimona, 2014

Violin 1 Player 1 Violin 2 Player 1 Violin 3 Player 1 Violin 1 Player 2 Violin 2 Player 2 Violin 3 Player 2 Violin 1 Player 3 Violin 2 Player 3 VIolin 3 Player 3

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

15

Three

instruments
 players

slide-21
SLIDE 21

Score

Articulation Duration Tone Dynamics Pitch (string and position) Bow direction Redundancy

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

16

slide-22
SLIDE 22

Sampling dimensions (i)

Legato
 Martelé

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

17

Articulation

slide-23
SLIDE 23

Sampling dimensions (i)

Half
 Quarter

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

18

Duration

legato: 120 bps martele: 132 bps

slide-24
SLIDE 24

Sampling dimensions (ii)

Quim Llimona, 2014

Dynamics

Piano
 Mezzoforte
 Forte

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

19

slide-25
SLIDE 25

Sampling dimensions (iii)

Quim Llimona, 2014

Tone

Sul tasto (1) Ordinary (2) Sul ponticello (3)

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

20

(2) (1) (3)

slide-26
SLIDE 26

Sampling dimensions (iv)

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

2, 5, 7 semitones (0 to 50% length) Strings sampled independently

Quim Llimona, 2014

Pitch

21

slide-27
SLIDE 27

Sampling dimensions (v)

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

Up Down

Quim Llimona, 2014

Bow direction

22

slide-28
SLIDE 28

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

Data acquisition

23 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-29
SLIDE 29

Overview

Quim Llimona, 2014 24

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

audio hi-speed IR cameras (x12)

PERFORMANCE CAPTURE SCENARIO

hi-quality video camera load cell

Audio I/O Sync Generator

MULTIMODAL REPOSITORY

Aligment and formatting

Qualysis Track Manager

Qualysis Acquisition Board

slide-30
SLIDE 30

Audio

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

25

ambience Schoeps Colette close-up DPA 4099-V pick-up Fishman V100

slide-31
SLIDE 31

Video

Quim Llimona, 2014 26

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

reference camera Sony PMW-EX3 (HD)

slide-32
SLIDE 32

Motion capture

Quim Llimona, 2014 27

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

Qualysis

http://www.labbase.net/Supply/SupplyItems-786112.html

It’s an infrared camera based motion capture system with passive markers

slide-33
SLIDE 33

Motion capture | violin (i)

Quim Llimona, 2014 28

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

plate

top_left top_right bottom_left bottom_right

scroll Notice the asymmetry

slide-34
SLIDE 34

Motion capture | violin (ii)

29 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

string_G_bridge string_D_bridge string_A_bridge string_E_bridge string_G_nut string_E_nut fb_center These are virtual markers

slide-35
SLIDE 35

Motion capture | bow

Quim Llimona, 2014 30

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

frog

antenna_left antenna_right stick

  • tip

corner stick tip

The antenna breaks colinearity

slide-36
SLIDE 36

Motion capture | body

Forehead Nape Wrist Elbow Shoulder

Quim Llimona, 2014 31

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-37
SLIDE 37

Load cell

Quim Llimona, 2014 32

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

With motion capture as well For calibration purposes

Virtual string

slide-38
SLIDE 38

Synchronization

SMPTE Word Clock Video frame

Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

33

slide-39
SLIDE 39

Feature extraction

34 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-40
SLIDE 40

Overview (i)

Bow force (estimated from other parameters) Bow velocity Bow position Bow-bridge distance Bow tilt Bow skew Bow inclination Current string Pseudoforce (left and right) Deformation

Quim Llimona, 2014 35

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

Used in force regression

Motion capture Audio

Pitch Energy Aperiodicity

slide-41
SLIDE 41

Overview (ii)

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

36 Quim Llimona, 2014

slide-42
SLIDE 42

High-level features

Bow position Bow velocity Bow force Bow-bridge distance Bow tilt Bow skew Bow inclination Current string

Quim Llimona, 2014 37

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-43
SLIDE 43

Low-level features (i)

38 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

V: Violin vector basis v_b: bridge v_s: string v_n: normal B: Bow vector basis b_0: hair width b_1: hair length (h_r) b_n: normal

alpha, beta, gamma (bow inclination, skew and tilt) extracted from V and B

slide-44
SLIDE 44

Low-level features (ii)

39 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

  • Current string

Shortest P

  • Length measurements

Bow position (x) Bow-bridge distance (d) Bow pseudoforces (f) Plus bow deformation

  • Bow velocity

Derivative of position

slide-45
SLIDE 45

Overview

40 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-46
SLIDE 46

Force regression (i)

Train radial basis SVM with: Bow position Bow pseudoforce (left) Bow pseudoforce (right) Bow deformation Bow tilt

Quim Llimona, 2014 41

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

Evaluation

  • Cross-validation

across all day

  • Schelleng diagram

compliance

An offset is added to pseudoforce for estimation on the violin

slide-47
SLIDE 47

Force regression (ii)

Quim Llimona, 2014 42

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

126 128 130 132 134 136 −0.2 0.2 0.4 0.6 0.8 1 Time (s) Bow force (V) Measured Estimated

slide-48
SLIDE 48

Load cell calibration

Quim Llimona, 2014

50 100 150 1 2 3 4 Time (s) Load cell reading (V)

43

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

Record known weights regularly Perform a polynomial (quadratic) fitting

slide-49
SLIDE 49

Audio features

Extracted with the YIN algorithm: Pitch (autocorrelation-based) Energy Aperiodicity

Quim Llimona, 2014 44

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-50
SLIDE 50

Database

45 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-51
SLIDE 51

80 85 90 95 100 −600 −400 −200 200 400 600 time (sec) bow velocity (mm/s)

Segmentation

Quim Llimona, 2014 46

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-52
SLIDE 52

Metadata

player technique permutation instrument string string_midi take duration duration_beats bpm tone dynamic pitch pitch_st pitch_midi pitch_finger pitch_position string_length bow_direction take_index duration_index dynamic_index tone_index pitch_index

Quim Llimona, 2014 47

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

For each note, a struct is generated with:

slide-53
SLIDE 53

Multimodal online database and visualization tool

login sign up

  • r continue like a guest

See video

repovizz and open data (i)

48 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

http://repovizz.upf.edu

slide-54
SLIDE 54

repovizz and open data (ii)

Quim Llimona, 2014 49

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-55
SLIDE 55

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

Preliminary analysis

50 Quim Llimona, 2014

slide-56
SLIDE 56

Player selection

51 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

The analysis is performed on Player 2

  • Selected as having the force regression with less error

(one player per day)

slide-57
SLIDE 57

Articulation | distribution

Bow−bridge distance (mm) Bow force estimation (V) 20 40 60 80 0.5 1 1.5 2 2.5 Bow velocity (mm/s) Bow force estimation (V) 500 1000 1500 0.5 1 1.5 2 2.5

52 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

Different regions of the space are covered blue:


  • range:

legato
 martele

The plots show predominance!
 This is because we had millions of points

slide-58
SLIDE 58

Articulation | profiles

53 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

Temporal profiles are different, especially velocity

normalized time

blue:


  • range:

legato
 martele try only forte

slide-59
SLIDE 59

Articulation | audio

54 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

normalized time Energy correlates with velocity

slide-60
SLIDE 60

Dynamics | distribution

55 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

Bow−bridge distance (mm) Bow force (V) 20 40 60 80 0.5 1 1.5 2 2.5 Bow velocity (mm/s) Bow force (V) 500 1000 1500 0.5 1 1.5 2 2.5 Bow−bridge distance (mm) Bow force (V) 20 40 60 80 0.5 1 1.5 2 2.5 Bow velocity (mm/s) Bow force (V) 500 1000 1500 0.5 1 1.5 2 2.5

blue:


  • range:

green: f
 mf p legato martele

slide-61
SLIDE 61

Dynamics | profiles

56 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

normalized time

legato martele blue:


  • range:

green: f
 mf p

  • vershoots
slide-62
SLIDE 62

Duration

57 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

blue:


  • range:

half
 quarter

Legato Martele 0.5 1 1.5 Time (s) Bow−bridge distance (mm) Bow force estimation (V) 20 40 60 80 1 2 Bow velocity (mm/s) Bow force estimation (V) 500 1000 1500 1 2

Half notes are slower, closer to the bridge and stronger Tendency to play slower than asked

slide-63
SLIDE 63

Tone | distribution

58 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

blue:


  • range:

green: tasto


  • rdinary

ponticello legato martele

Bow−bridge distance (mm) Bow force (V) 20 40 60 80 0.5 1 1.5 2 2.5 Bow velocity (mm/s) Bow force (V) 500 1000 1500 0.5 1 1.5 2 2.5 Bow−bridge distance (mm) Bow force (V) 20 40 60 80 0.5 1 1.5 2 2.5 Bow velocity (mm/s) Bow force (V) 500 1000 1500 0.5 1 1.5 2 2.5

slide-64
SLIDE 64

Tone | boxplots

59 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

blue:


  • range:

green: tasto


  • rdinary

ponticello Player 1 Player 2

f mf p 15 30 45 60 75 90 Bow−bridge distance (mm) f mf p 15 30 45 60 75 90 Bow−bridge distance (mm) f mf p 15 30 45 60 75 90 Bow−bridge distance (mm)

Player 3 Player 2 only follows on mf

slide-65
SLIDE 65

Conclusion

60 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-66
SLIDE 66

Contributions

  • The MUSMAP I dataset (to be published on repovizz)
  • Software tools (to be published on GitHub)
  • Methodology (incl. experimental setup and features)

Quim Llimona, 2014 61

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-67
SLIDE 67

Future directions

  • Finish repovizz upload (in progress)
  • Software documentation and release (in progress)
  • Curve registration
  • More audio features (MFCC)
  • Mapping between auditory and motor spaces
  • Optimal load cell regression offset and parameters

Quim Llimona, 2014 62

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-68
SLIDE 68

Acknowledegments

Universitat Pompeu Fabra
 Music Technology Group Agència de Gestió d’Ajuts Universitaris i de Recerca (AGAUR) European Research Council
 MUSMAP Marie Curie IOF action McGill University and CIRMMT

Quim Llimona, 2014 63

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion

slide-69
SLIDE 69

Quim Llimona Torras

Advisor: Esteban Maestre

Bowing the violin

A case study for auditory-motor pattern modeling in the context of music performance

Quim Llimona, 2014

slide-70
SLIDE 70

Tone | boxplots

65 Quim Llimona, 2014

blue:


  • range:

green: tasto


  • rdinary

ponticello Player 1 Player 2 Player 3

f mf p 15 30 45 60 75 90 Bow−bridge distance (mm) f mf p 15 30 45 60 75 90 Bow−bridge distance (mm) f mf p 15 30 45 60 75 90 Bow−bridge distance (mm) f mf p 250 500 750 1000 1250 1500 Bow velocity (mm/s) f mf p 250 500 750 1000 1250 1500 Bow velocity (mm/s) f mf p 250 500 750 1000 1250 1500 Bow velocity (mm/s) f mf p 0.25 0.5 0.75 1 1.25 1.5 Bow force (V) f mf p 0.25 0.5 0.75 1 1.25 1.5 Bow force (V) f mf p 0.25 0.5 0.75 1 1.25 1.5 Bow force (V)