Music Structure Analysis Meinard Mller International Audio - - PowerPoint PPT Presentation

▶
music structure analysis
SMART_READER_LITE
LIVE PREVIEW

Music Structure Analysis Meinard Mller International Audio - - PowerPoint PPT Presentation

Nov 15, 2022 385 likes •1.75k views

Lecture Music Processing Music Structure Analysis Meinard Mller International Audio Laboratories Erlangen meinard.mueller@audiolabs-erlangen.de Book: Fundamentals of Music Processing Meinard Mller Fundamentals of Music Processing Audio,

slide-1
SLIDE 1

Music Processing Meinard Müller

Lecture

Music Structure Analysis

International Audio Laboratories Erlangen meinard.mueller@audiolabs-erlangen.de

slide-2
SLIDE 2

Book: Fundamentals of Music Processing

Meinard Müller Fundamentals of Music Processing Audio, Analysis, Algorithms, Applications 483 p., 249 illus., hardcover ISBN: 978-3-319-21944-8 Springer, 2015 Accompanying website: www.music-processing.de

slide-3
SLIDE 3

Book: Fundamentals of Music Processing

Meinard Müller Fundamentals of Music Processing Audio, Analysis, Algorithms, Applications 483 p., 249 illus., hardcover ISBN: 978-3-319-21944-8 Springer, 2015 Accompanying website: www.music-processing.de

slide-4
SLIDE 4

Book: Fundamentals of Music Processing

Meinard Müller Fundamentals of Music Processing Audio, Analysis, Algorithms, Applications 483 p., 249 illus., hardcover ISBN: 978-3-319-21944-8 Springer, 2015 Accompanying website: www.music-processing.de

slide-5
SLIDE 5

Chapter 4: Music Structure Analysis

In Chapter 4, we address a central and well-researched area within MIR known as music structure analysis. Given a music recording, the objective is to identify important structural elements and to temporally segment the recording according to these elements. Within this scenario, we discuss fundamental segmentation principles based on repetitions, homogeneity, and novelty— principles that also apply to other types of multimedia beyond music. As an important technical tool, we study in detail the concept of self-similarity matrices and discuss their structural properties. Finally, we briefly touch the topic of evaluation, introducing the notions of precision, recall, and F-measure.

4.1 General Principles 4.2 Self-Similarity Matrices 4.3 Audio Thumbnailing 4.4 Novelty-Based Segmentation 4.5 Evaluation 4.6 Further Notes

slide-6
SLIDE 6

Music Structure Analysis

Example: Zager & Evans “In The Year 2525”

Time (seconds)

slide-7
SLIDE 7

Music Structure Analysis

Time (seconds)

Example: Zager & Evans “In The Year 2525”

slide-8
SLIDE 8

Music Structure Analysis

V1 V2 V3 V4 V5 V6 V7 V8 O B I

Example: Zager & Evans “In The Year 2525”

slide-9
SLIDE 9

Music Structure Analysis

Example: Brahms Hungarian Dance No. 5 (Ormandy)

Time (seconds)

A1 A2 A3 B1 B2 B3 B4 C

slide-10
SLIDE 10

Music Structure Analysis

Time (seconds)

Example: Folk Song Field Recording (Nederlandse Liederenbank)

slide-11
SLIDE 11

Example: Weber, Song (No. 4) from “Der Freischütz”

50 100 150 200

…

...

Kleiber

Time (seconds)

.. .. ..

Music Structure Analysis

50 100 150 200

Introduction Stanzas Dialogues

20 40 60 80 100 120 20 40 60 80 100 120

Ackermann

Time (seconds)

slide-12
SLIDE 12

Music Structure Analysis

  • Stanzas of a folk song
  • Intro, verse, chorus, bridge, outro sections of a pop song
  • Exposition, development, recapitulation, coda of a sonata
  • Musical form ABACADA … of a rondo

General goal: Divide an audio recording into temporal segments corresponding to musical parts and group these segments into musically meaningful categories. Examples:

slide-13
SLIDE 13

Music Structure Analysis

  • Homogeneity:
  • Novelty:
  • Repetition:

General goal: Divide an audio recording into temporal segments corresponding to musical parts and group these segments into musically meaningful categories. Challenge: There are many different principles for creating relationships that form the basis for the musical structure.

Consistency in tempo, instrumentation, key, … Sudden changes, surprising elements … Repeating themes, motives, rhythmic patterns,…

slide-14
SLIDE 14

Music Structure Analysis

Novelty Homogeneity Repetition

slide-15
SLIDE 15

Overview

  • Introduction
  • Feature Representations
  • Self-Similarity Matrices
  • Audio Thumbnailing
  • Novelty-based Segmentation

Thanks:

  • Clausen, Ewert,

Kurth, Grohganz, …

  • Dannenberg, Goto
  • Grosche, Jiang
  • Paulus, Klapuri
  • Peeters, Kaiser, …
  • Serra, Gómez, …
  • Smith, Fujinaga, …
  • Wiering, …
  • Wand, Sunkel,

Jansen

  • …
slide-16
SLIDE 16

Overview

  • Introduction
  • Feature Representations
  • Self-Similarity Matrices
  • Audio Thumbnailing
  • Novelty-based Segmentation

Thanks:

  • Clausen, Ewert,

Kurth, Grohganz, …

  • Dannenberg, Goto
  • Grosche, Jiang
  • Paulus, Klapuri
  • Peeters, Kaiser, …
  • Serra, Gómez, …
  • Smith, Fujinaga, …
  • Wiering, …
  • Wand, Sunkel,

Jansen

  • …
slide-17
SLIDE 17

Feature Representation

General goal: Convert an audio recording into a mid-level representation that captures certain musical properties while supressing other properties.

  • Timbre / Instrumentation
  • Tempo / Rhythm
  • Pitch / Harmony
slide-18
SLIDE 18

Feature Representation

General goal: Convert an audio recording into a mid-level representation that captures certain musical properties while supressing other properties.

  • Timbre / Instrumentation
  • Tempo / Rhythm
  • Pitch / Harmony
slide-19
SLIDE 19

Feature Representation

C1 24 C2 36 C3 48 C4 60 C5 72 C6 84 C7 96 C8 108

Example: Chromatic scale Waveform

Time (seconds) Amplitude

slide-20
SLIDE 20

Feature Representation

Frequency (Hz) Intensity (dB) Intensity (dB) Frequency (Hz) Time (seconds)

C1 24 C2 36 C3 48 C4 60 C5 72 C6 84 C7 96 C8 108

Example: Chromatic scale Spectrogram

slide-21
SLIDE 21

Feature Representation

Frequency (Hz) Intensity (dB) Intensity (dB) Frequency (Hz) Time (seconds)

C1 24 C2 36 C3 48 C4 60 C5 72 C6 84 C7 96 C8 108

Example: Chromatic scale Spectrogram

slide-22
SLIDE 22

Feature Representation

C4: 261 Hz C5: 523 Hz C6: 1046 Hz C7: 2093 Hz C8: 4186 Hz C3: 131 Hz

Intensity (dB) Time (seconds)

C1 24 C2 36 C3 48 C4 60 C5 72 C6 84 C7 96 C8 108

Example: Chromatic scale Spectrogram

slide-23
SLIDE 23

Feature Representation

C4: 261 Hz C5: 523 Hz C6: 1046 Hz C7: 2093 Hz C8: 4186 Hz C3: 131 Hz

Intensity (dB) Time (seconds)

C1 24 C2 36 C3 48 C4 60 C5 72 C6 84 C7 96 C8 108

Example: Chromatic scale Log-frequency spectrogram

slide-24
SLIDE 24

Feature Representation

Pitch (MIDI note number) Intensity (dB) Time (seconds)

C1 24 C2 36 C3 48 C4 60 C5 72 C6 84 C7 96 C8 108

Example: Chromatic scale Log-frequency spectrogram

slide-25
SLIDE 25

Feature Representation

Chroma C

Intensity (dB) Pitch (MIDI note number) Time (seconds)

C1 24 C2 36 C3 48 C4 60 C5 72 C6 84 C7 96 C8 108

Example: Chromatic scale Log-frequency spectrogram

slide-26
SLIDE 26

Feature Representation

Chroma C#

Intensity (dB) Pitch (MIDI note number) Time (seconds)

C1 24 C2 36 C3 48 C4 60 C5 72 C6 84 C7 96 C8 108

Example: Chromatic scale Log-frequency spectrogram

slide-27
SLIDE 27

Feature Representation

C1 24 C2 36 C3 48 C4 60 C5 72 C6 84 C7 96 C8 108

Example: Chromatic scale Chroma representation

Intensity (dB) Time (seconds) Chroma

slide-28
SLIDE 28

Feature Representation

Example: Brahms Hungarian Dance No. 5 (Ormandy)

Time (seconds) A1 A2 A3 B1 B2 B3 B4 C

slide-29
SLIDE 29

Feature Representation

A1 A2 A3 B1 B2 B3 B4 C

Feature extraction Chroma (Harmony) Example: Brahms Hungarian Dance No. 5 (Ormandy)

Time (seconds)

slide-30
SLIDE 30

Feature Representation

A1 A2 A3 B1 B2 B3 B4 C

Feature extraction Chroma (Harmony) Example: Brahms Hungarian Dance No. 5 (Ormandy)

G minor G minor

D G Bb

Time (seconds)

slide-31
SLIDE 31

Feature Representation

A1 A2 A3 B1 B2 B3 B4 C

Feature extraction Chroma (Harmony) Example: Brahms Hungarian Dance No. 5 (Ormandy)

G minor G major G minor

D G Bb D G B

Time (seconds)

slide-32
SLIDE 32

Overview

  • Introduction
  • Feature Representations
  • Self-Similarity Matrices
  • Audio Thumbnailing
  • Novelty-based Segmentation
slide-33
SLIDE 33

Self-Similarity Matrix (SSM)

General idea: Compare each element of the feature sequence with each other element of the feature sequence based on a suitable similarity measure. → Quadratic self-similarity matrix

slide-34
SLIDE 34

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

slide-35
SLIDE 35

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

slide-36
SLIDE 36

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

slide-37
SLIDE 37

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

slide-38
SLIDE 38

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

slide-39
SLIDE 39

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

slide-40
SLIDE 40

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

G major G major

slide-41
SLIDE 41

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

slide-42
SLIDE 42

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

slide-43
SLIDE 43

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

slide-44
SLIDE 44

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

slide-45
SLIDE 45

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

Slower Faster

slide-46
SLIDE 46

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

Faster Slower

slide-47
SLIDE 47

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy) Idealized SSM

slide-48
SLIDE 48

Self-Similarity Matrix (SSM)

Example: Brahms Hungarian Dance No. 5 (Ormandy) Idealized SSM Blocks: Homogeneity Paths: Repetition Corners: Novelty

slide-49
SLIDE 49

SSM Enhancement

  • Feature smoothing
  • Coarsening

Time (samples) Time (samples)

Block Enhancement

slide-50
SLIDE 50

SSM Enhancement

Block Enhancement

  • Feature smoothing
  • Coarsening

Time (samples) Time (samples)

slide-51
SLIDE 51

SSM Enhancement

  • Feature smoothing
  • Coarsening

Time (samples) Time (samples)

Block Enhancement

slide-52
SLIDE 52

SSM Enhancement

Challenge: Presence of musical variations Idea: Enhancement of path structure

  • Fragmented paths and gaps
  • Paths of poor quality
  • Regions of constant (high) similarity
  • Curved paths
slide-53
SLIDE 53

SSM Enhancement

Shostakovich Waltz 2, Jazz Suite No. 2 (Chailly) SSM

slide-54
SLIDE 54

SSM Enhancement

Shostakovich Waltz 2, Jazz Suite No. 2 (Chailly) SSM

slide-55
SLIDE 55

SSM Enhancement

Shostakovich Waltz 2, Jazz Suite No. 2 (Chailly) SSM

slide-56
SLIDE 56

SSM Enhancement

Shostakovich Waltz 2, Jazz Suite No. 2 (Chailly) Enhanced SSM Filtering along main diagonal

slide-57
SLIDE 57

SSM Enhancement

Idea: Usage of contextual information (Foote 1999) smoothing effect

  • Comparison of entire sequences
  • = length of sequences
  • = enhanced SSM
slide-58
SLIDE 58

SSM Enhancement

SSM

slide-59
SLIDE 59

SSM Enhancement

Filtering along main diagonal Enhanced SSM with

slide-60
SLIDE 60

SSM Enhancement

Filtering along 8 different directions and minimizing Enhanced SSM with

slide-61
SLIDE 61

SSM Enhancement

Idea: Smoothing along various directions and minimizing over all directions Tempo changes of -50 to +50 percent

slide-62
SLIDE 62

SSM Enhancement

Time (samples) Time (samples)

Path Enhancement

slide-63
SLIDE 63

SSM Enhancement

Time (samples) Time (samples)

Path Enhancement

  • Diagonal smoothing
slide-64
SLIDE 64

SSM Enhancement

Time (samples) Time (samples)

Path Enhancement

  • Diagonal smoothing
  • Multiple filtering
slide-65
SLIDE 65

SSM Enhancement

Time (samples) Time (samples)

Path Enhancement

  • Diagonal smoothing
  • Multiple filtering
  • Thresholding (relative)
  • Scaling & penalty
slide-66
SLIDE 66

SSM Enhancement

Time (samples) Time (samples)

Further Processing

  • Path extraction
slide-67
SLIDE 67

SSM Enhancement

Time (samples) Time (samples)

Further Processing

  • Path extraction
  • Pairwise relations

100 200 300 400 1

Time (samples)

2 3 4 5 6 7

slide-68
SLIDE 68

SSM Enhancement

Time (samples) Time (samples)

Further Processing

  • Path extraction
  • Pairwise relations
  • Grouping (transitivity)

100 200 300 400 1

Time (samples)

2 3 4 5 6 7

slide-69
SLIDE 69

100 200 300 400

Time (samples)

SSM Enhancement

Time (samples) Time (samples)

Further Processing

  • Path extraction
  • Pairwise relations
  • Grouping (transitivity)

100 200 300 400 1

Time (samples)

2 3 4 5 6 7

slide-70
SLIDE 70

SSM Enhancement

V1 V2 V3 V4 V5 V6 V7 V8 O B I

Example: Zager & Evans “In The Year 2525”

slide-71
SLIDE 71

SSM Enhancement

Example: Zager & Evans “In The Year 2525”

slide-72
SLIDE 72

SSM Enhancement

Example: Zager & Evans “In The Year 2525” Missing relations because of transposed sections

slide-73
SLIDE 73

SSM Enhancement

Example: Zager & Evans “In The Year 2525” Idea: Cyclic shift of one of the chroma sequences

One semitone up

slide-74
SLIDE 74

SSM Enhancement

Example: Zager & Evans “In The Year 2525” Idea: Cyclic shift of one of the chroma sequences

Two semitones up

slide-75
SLIDE 75

SSM Enhancement

Example: Zager & Evans “In The Year 2525” Idea: Overlay Transposition-invariant SSM & Maximize

slide-76
SLIDE 76

SSM Enhancement

Example: Zager & Evans “In The Year 2525” Note: Order of enhancement steps important! Maximization Smoothing & Maximization

slide-77
SLIDE 77

Similarity Matrix Toolbox

Meinard Müller, Nanzhu Jiang, Harald Grohganz SM Toolbox: MATLAB Implementations for Computing and Enhancing Similarity Matrices

http://www.audiolabs-erlangen.de/resources/MIR/SMtoolbox/

slide-78
SLIDE 78

Overview

  • Introduction
  • Feature Representations
  • Self-Similarity Matrices
  • Audio Thumbnailing
  • Novelty-based Segmentation

Thanks:

  • Jiang, Grosche
  • Peeters
  • Cooper, Foote
  • Goto
  • Levy, Sandler
  • Mauch
  • Sapp
slide-79
SLIDE 79

Audio Thumbnailing

A1 A2 A3 B1 B2 B3 B4 C

Example: Brahms Hungarian Dance No. 5 (Ormandy) General goal: Determine the most representative section (“Thumbnail”) of a given music recording.

V1 V2 V3 V4 V5 V6 V7 V8 O B I

Example: Zager & Evans “In The Year 2525” Thumbnail is often assumed to be the most repetitive segment

slide-80
SLIDE 80

Audio Thumbnailing

Two steps

  • Paths of poor quality (fragmented, gaps)
  • Block-like structures
  • Curved paths
  • 1. Path extraction
  • 2. Grouping
  • Noisy relations

(missing, distorted, overlapping)

  • Transitivity computation difficult

Both steps are problematic! Main idea: Do both, path extraction and grouping, jointly

  • One optimization scheme for both steps
  • Stabilizing effect
  • Efficient
slide-81
SLIDE 81

Audio Thumbnailing

Main idea: Do both path extraction and grouping jointly

  • For each audio segment we define a fitness value
  • This fitness value expresses “how well” the segment

explains the entire audio recording

  • The segment with the highest fitness value is

considered to be the thumbnail

  • As main technical concept we introduce the notion of a

path family

slide-82
SLIDE 82

50 100 150 200 20 40 60 80 100 120 140 160 180 200 −2 −1.5 −1 −0.5 0.5 1

Fitness Measure

Enhanced SSM

slide-83
SLIDE 83

Fitness Measure

  • Consider a fixed segment

Path over segment

slide-84
SLIDE 84

Fitness Measure

  • Consider a fixed segment
  • Path over segment
  • Induced segment
  • Score is high

Path over segment

slide-85
SLIDE 85

Fitness Measure

Path over segment

  • Consider a fixed segment
  • Path over segment
  • Induced segment
  • Score is high
  • A second path over segment
  • Induced segment
  • Score is not so high
slide-86
SLIDE 86

Fitness Measure

Path over segment

  • Consider a fixed segment
  • Path over segment
  • Induced segment
  • Score is high
  • A second path over segment
  • Induced segment
  • Score is not so high
  • A third path over segment
  • Induced segment
  • Score is very low
slide-87
SLIDE 87

Fitness Measure

Path family

  • Consider a fixed segment
  • A path family over a segment

is a family of paths such that the induced segments do not overlap.

slide-88
SLIDE 88

Fitness Measure

Path family This is not a path family!

  • Consider a fixed segment
  • A path family over a segment

is a family of paths such that the induced segments do not overlap.

slide-89
SLIDE 89

Fitness Measure

Path family This is a path family!

  • Consider a fixed segment
  • A path family over a segment

is a family of paths such that the induced segments do not overlap. (Even though not a good one)

slide-90
SLIDE 90

Fitness Measure

Optimal path family

  • Consider a fixed segment
slide-91
SLIDE 91

Fitness Measure

Optimal path family

  • Consider a fixed segment
  • Consider over the segment

the optimal path family, i.e., the path family having maximal overall score.

  • Call this value:

Score(segment)

Note: This optimal path family can be computed using dynamic programming.

slide-92
SLIDE 92

Fitness Measure

Optimal path family

  • Consider a fixed segment
  • Consider over the segment

the optimal path family, i.e., the path family having maximal overall score.

  • Call this value:

Score(segment)

  • Furthermore consider the

amount covered by the induced segments.

  • Call this value:

Coverage(segment)

slide-93
SLIDE 93

Fitness Measure

Fitness

  • Consider a fixed segment

P := R := Score(segment) Coverage(segment)

slide-94
SLIDE 94

Fitness Measure

Fitness

  • Consider a fixed segment
  • Self-explanation are trivial!

P := R := Score(segment) Coverage(segment)

slide-95
SLIDE 95

Fitness Measure

Fitness

  • Consider a fixed segment
  • Self-explanation are trivial!
  • Subtract length of segment

P := R := Score(segment) Coverage(segment)

  • length(segment)
  • length(segment)
slide-96
SLIDE 96

Normalize( )

Fitness Measure

Fitness

  • Consider a fixed segment
  • Self-explanation are trivial!
  • Subtract length of segment
  • Normalization

P := R := Score(segment) Coverage(segment)

  • length(segment)
  • length(segment)

] 1 , [  ] 1 , [ 

Normalize( )

slide-97
SLIDE 97

Fitness Measure

Fitness

  • Consider a fixed segment

F := 2 • P • R / (P + R) Fitness(segment)

Normalize( ) Normalize( ) P := R := Score(segment) Coverage(segment)

  • length(segment)
  • length(segment)

] 1 , [  ] 1 , [ 

slide-98
SLIDE 98

Thumbnail

Segment center Segment length

Fitness Scape Plot

Segment length Segment center Fitness

slide-99
SLIDE 99

Thumbnail

Segment center

Fitness Scape Plot

Fitness(segment)

Segment length Segment center Fitness Segment length

slide-100
SLIDE 100

Thumbnail

Segment center

Fitness Scape Plot

Fitness Segment length

slide-101
SLIDE 101

Thumbnail

Segment center

Fitness Scape Plot Note: Self-explanations are ignored → fitness is zero

Fitness Segment length

slide-102
SLIDE 102

Thumbnail

Segment center

Fitness Scape Plot Thumbnail := segment having the highest fitness

Fitness Segment length

slide-103
SLIDE 103

Thumbnail

Fitness Scape Plot Example: Brahms Hungarian Dance No. 5 (Ormandy)

Fitness

A1 A2 A3 B1 B2 B3 B4 C

slide-104
SLIDE 104

Fitness

Thumbnail

Fitness Scape Plot Example: Brahms Hungarian Dance No. 5 (Ormandy)

A1 A2 A3 B1 B2 B3 B4 C

slide-105
SLIDE 105

Fitness

Thumbnail

Fitness Scape Plot Example: Brahms Hungarian Dance No. 5 (Ormandy)

A1 A2 A3 B1 B2 B3 B4 C

slide-106
SLIDE 106

Fitness

Thumbnail

Fitness Scape Plot Example: Brahms Hungarian Dance No. 5 (Ormandy)

A1 A2 A3 B1 B2 B3 B4 C

slide-107
SLIDE 107

Scape Plot

Example: Brahms Hungarian Dance No. 5 (Ormandy)

slide-108
SLIDE 108

Scape Plot

Coloring according to clustering result (grouping) Example: Brahms Hungarian Dance No. 5 (Ormandy)

slide-109
SLIDE 109

Scape Plot

Example: Brahms Hungarian Dance No. 5 (Ormandy) Coloring according to clustering result (grouping)

A1 A2 A3 B1 B2 B3 B4 C

slide-110
SLIDE 110

Thumbnail

Fitness Scape Plot Example: Zager & Evans “In The Year 2525”

Fitness

V1 V2 V3 V4 V5 V6 V7 V8 O B I

slide-111
SLIDE 111

Fitness

Thumbnail

Fitness Scape Plot Example: Zager & Evans “In The Year 2525”

V1 V2 V3 V4 V5 V6 V7 V8 O B I

slide-112
SLIDE 112

Overview

  • Introduction
  • Feature Representations
  • Self-Similarity Matrices
  • Audio Thumbnailing
  • Novelty-based Segmentation

Thanks:

  • Foote
  • Serra, Grosche, Arcos
  • Goto
  • Tzanetakis, Cook
slide-113
SLIDE 113

Novelty-based Segmentation

  • Find instances where musical

changes occur.

  • Find transition between

subsequent musical parts.

General goals: Idea (Foote):

Use checkerboard-like kernel function to detect corner points

  • n main diagonal of SSM.
slide-114
SLIDE 114

Novelty-based Segmentation

Idea (Foote):

Use checkerboard-like kernel function to detect corner points

  • n main diagonal of SSM.
slide-115
SLIDE 115

Novelty-based Segmentation

Idea (Foote):

Use checkerboard-like kernel function to detect corner points

  • n main diagonal of SSM.
slide-116
SLIDE 116

Novelty-based Segmentation

Idea (Foote):

Use checkerboard-like kernel function to detect corner points

  • n main diagonal of SSM.
slide-117
SLIDE 117

Novelty-based Segmentation

Idea (Foote):

Use checkerboard-like kernel function to detect corner points

  • n main diagonal of SSM.
slide-118
SLIDE 118

Novelty-based Segmentation

Idea (Foote):

Use checkerboard-like kernel function to detect corner points

  • n main diagonal of SSM.

Novelty function using

slide-119
SLIDE 119

Novelty-based Segmentation

Idea (Foote):

Use checkerboard-like kernel function to detect corner points

  • n main diagonal of SSM.

Novelty function using Novelty function using

slide-120
SLIDE 120

Novelty-based Segmentation

Idea:

  • Find instances where

structural changes occur.

  • Combine global and local

aspects within a unifying framework

Structure features

slide-121
SLIDE 121

Novelty-based Segmentation

  • Enhanced SSM

Structure features

slide-122
SLIDE 122

Novelty-based Segmentation

  • Enhanced SSM
  • Time-lag SSM

Structure features

slide-123
SLIDE 123

Novelty-based Segmentation

  • Enhanced SSM
  • Time-lag SSM
  • Cyclic time-lag SSM

Structure features

slide-124
SLIDE 124

Novelty-based Segmentation

  • Enhanced SSM
  • Time-lag SSM
  • Cyclic time-lag SSM
  • Columns as features

Structure features

slide-125
SLIDE 125

Novelty-based Segmentation

Example: Chopin Mazurka Op. 24, No. 1

SSM Time-lag SSM

slide-126
SLIDE 126

Novelty-based Segmentation

Example: Chopin Mazurka Op. 24, No. 1

SSM Time-lag SSM

slide-127
SLIDE 127

Novelty-based Segmentation

Example: Chopin Mazurka Op. 24, No. 1

SSM Time-lag SSM

slide-128
SLIDE 128

Novelty-based Segmentation

Structure-based novelty function

Example: Chopin Mazurka Op. 24, No. 1

SSM Time-lag SSM

slide-129
SLIDE 129

Structure Analysis

Conclusions

slide-130
SLIDE 130

Representations

Structure Analysis

Audio MIDI Score

Conclusions

slide-131
SLIDE 131

Representations Musical Aspects

Structure Analysis

Timbre Tempo Harmony Audio MIDI Score

Conclusions

slide-132
SLIDE 132

Representations Segmentation Principles Musical Aspects

Structure Analysis

Homogeneity Novelty Repetition Timbre Tempo Harmony Audio MIDI Score

Conclusions

slide-133
SLIDE 133

Temporal and Hierarchical Context Representations Segmentation Principles Musical Aspects

Structure Analysis

Homogeneity Novelty Repetition Timbre Tempo Harmony Audio MIDI Score

Conclusions

slide-134
SLIDE 134

Conclusions

  • Combined Approaches
  • Hierarchical Approaches
  • Evaluation
  • Explaining Structure
  • MIREX
  • SALAMI-Project
  • Smith, Chew
slide-135
SLIDE 135

Links

  • SM Toolbox (MATLAB)

http://www.audiolabs-erlangen.de/resources/MIR/SMtoolbox/

  • MSAF: Music Structure Analysis Framework (Python)

https://github.com/urinieto/msaf

  • SALAMI Annotation Data

http://ddmal.music.mcgill.ca/research/salami/annotations

  • LibROSA (Python)

https://librosa.github.io/librosa/

  • Evaluation: mir_eval (Python)

https://craffel.github.io/mir_eval/

  • Deep Learning: Boundary Detection

Jan Schlüter (PhD thesis)