Validating a technique for post-hoc estimation of a listener's - - PowerPoint PPT Presentation

▶
validating a technique for post hoc estimation of a
SMART_READER_LITE
LIVE PREVIEW

Validating a technique for post-hoc estimation of a listener's - - PowerPoint PPT Presentation

Aug 27, 2022 138 likes •442 views

Validating a technique for post-hoc estimation of a listener's focus in music structure analysis Jordan B. L. Smith National Institute of Advanced Industrial Science and Technology (AIST), Japan Elaine Chew Queen Mary University of London

slide-1
SLIDE 1

Validating a technique for post-hoc estimation of a listener's focus in music structure analysis

Jordan B. L. Smith

National Institute of Advanced Industrial Science and Technology (AIST), Japan

Elaine Chew

Queen Mary University of London

slide-2
SLIDE 2

What was the listener thinking?


(when they annotated musical structure)

Melody
 Harmony
 Rhythm
 Meter
 Timbre
 etc.

slide-3
SLIDE 3

Isaac Hayes, Run Fay Run

slide-4
SLIDE 4

Isaac Hayes, Run Fay Run

slide-5
SLIDE 5

RELATING GROUPING STRUCTURE TO MUSICAL FEATURES

Audio-derived SSM

A B C

time

slide-6
SLIDE 6

RELATING GROUPING STRUCTURE TO MUSICAL FEATURES

Audio-derived SSM Listener 1: AAB Listener 2: ABB

A B C

slide-7
SLIDE 7

RELATING GROUPING STRUCTURE TO MUSICAL FEATURES

Audio-derived SSM Listener 1: AAB Listener 2: ABB Harmony SSM Timbre SSM

A B C

slide-8
SLIDE 8

RELATING GROUPING STRUCTURE TO MUSICAL FEATURES

Audio-derived SSM Listener 1: AAB Listener 2: ABB Harmony SSM Timbre SSM

A B C Suggestion: We might explain the differences between two annotations by matching them with the musical features.

slide-9
SLIDE 9

RELATING GROUPING STRUCTURE TO MUSICAL FEATURES

Listener 1: AAB Listener 2: ABB Harmony SSM Timbre SSM

slide-10
SLIDE 10

Segment-based masks Feature-derived SSMs Target SSM (original annotation) Reconstruction components

slide-11
SLIDE 11

Segment-based masks Feature-derived SSMs Target SSM (original annotation) Reconstruction components

x11M11 x12M12 x21M21 x31M31 x71M71 x75M75 x15M15 x13M13 ... . . . . . . M11 M12 M21 M31 M71 M75 M15 M13 N

slide-12
SLIDE 12

Segment-based masks Feature-derived SSMs Target SSM (original annotation) Reconstruction components

x11M11 x12M12 x21M21 x31M31 x71M71 x75M75 x15M15 x13M13 ... . . . . . . M11 M12 M21 M31 M71 M75 M15 M13 N

find that minimizes

slide-13
SLIDE 13

WHICH FEATURE BEST EXPLAINS THE ANALYSIS?

slide-14
SLIDE 14

CRITICISM

➤ Model not validated with real-world data: ➤ I.e., analyses paired with attention state of listener

slide-15
SLIDE 15

EYE TRACKING

slide-16
SLIDE 16

EAR TRACKING

slide-17
SLIDE 17

CRITICISM

➤ Model not validated with real-world data: ➤ I.e., analyses paired with attention state of listener ➤ One source of data: ➤ Ran experiment testing whether focus of listener

could affect perception of grouping

slide-18
SLIDE 18

STIMULUS EXAMPLES

  • Organ
  • Melody ABB


Harmony AAB

slide-19
SLIDE 19

STIMULUS EXAMPLES

  • Organ
  • Melody ABB


Harmony AAB

A B B A A B

slide-20
SLIDE 20

STIMULUS EXAMPLES

  • Harpsichord
  • Organ
  • 3
  • Timbre ABB


Harmony AAB

  • Organ
  • Melody ABB


Harmony AAB

A B B A A B

slide-21
SLIDE 21

STIMULUS EXAMPLES

  • Harpsichord
  • Organ
  • 3
  • Timbre ABB


Harmony AAB

  • Organ
  • Melody ABB


Harmony AAB

A B B A A B A B B A A B

slide-22
SLIDE 22

STIMULUS EXAMPLES

  • Harpsichord
  • Organ
  • 3
  • Timbre ABB


Harmony AAB

  • Organ
  • Melody ABB


Harmony AAB

➤ Experiment outcome: giving listeners a distractor task (to fix attention)

affected AAB vs. ABB decisions

➤ “Does this chord progression occur in the stimulus?” → prefer AAB ➤ Note: sometimes, organizing features are convolved

A B B A A B A B B A A B

slide-23
SLIDE 23

CRITICISM

➤ Model not validated with real-world data: ➤ I.e., analyses paired with attention state of listener ➤ One source of data: ➤ Ran experiment testing whether focus of listener

could affect perception of grouping

➤ Is model needlessly complicated? ➤ Try simple correlation instead

slide-24
SLIDE 24

Segment-based masks Feature-derived SSMs Target SSM (original annotation) Reconstruction components

N

QP approach:

  • find sum of reconstruction

components to generate target SSM

  • interpret coefficients as

relevance Correlation approach:

  • compute correlation between

feature SSMs and masks

  • interpret correlation as

relevance

slide-25
SLIDE 25

EXPERIMENT

➤ Use algorithm to estimate relevance of features to analysis of

experiment stimuli

➤ If the most relevant feature is the feature that changed, count

correct

slide-26
SLIDE 26

RESULTS

chance level What feature was found to be most relevant? (i.e., greatest QP weights

  • r correlation)
slide-27
SLIDE 27

SUMMARY

➤ Past work: ➤ 1. Algorithm to estimate feature relevance from analyses

(Smith and Chew, 2013)

➤ 2. Artificial stimuli from psych experiment studying

attention

➤ Current work: ➤ Use stimuli (2) to validate algorithm (1) and new

variations

slide-28
SLIDE 28

SUMMARY

➤ Algorithm performs above baseline… ➤ …but artificial context makes non-perfect performance a

disappointment!

➤ Random effect of small set of stimuli ➤ Mismatch between features and musicological interpretations ➤ Future work: ➤ re-optimize (or train) model with more features, more

artificial stimuli

➤ apply algorithm to investigate trends in large datasets

THANKS!

slide-29
SLIDE 29

ADVERTORIAL SUPPLEMENT

slide-30
SLIDE 30

REFERENCES

➤ Jordan B. L. Smith and Elaine Chew. Using Quadratic

Programming to estimate feature relevance in structural analyses of music. In Proceedings of the ACM International Conference on Multimedia, 113–122, Barcelona, Spain, 2013.

➤ Jordan B. L. Smith. Explaining listener differences in the

perception of musical structure. Ph.D. thesis. Queen Mary University of London. See Chapter 6: “The effect of attention

  • n grouping decisions.”