[PPT] - Visualization of perceptual qualities in textural sounds PowerPoint Presentation

SLIDE 1

Austrian Research Institute for Artificial Intelligence (OFAI)

International Computer Music Conference (ICMC 2012), Ljubljana/Slovenia

Visualization of perceptual qualities in textural sounds

Thomas Grill, Arthur Flexer

SLIDE 2

SLIDE 3

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Fundamental questions

How can digital sound material be described?
How can sounds be organized?
How can sounds and collections thereof be visualized?
Focus on sampled sound with textural characteristics

3

SLIDE 4

ringing cheeping gasping smashing piercing peeping whooping tinkling raucous chattering crooning bellowing sobbing bumping snarling growling pitch crying thumping burping croaking clattering yapping keening splashing yelping rustling volume squealing howling barking sniveling moaning pealing tone rattling grunting clanging coughing quacking whining gagging fizzing wheezing honking hissing bawling trumpeting swishing sneezing rumbling bubbling ripping cooing chirping shouting shuffling tearing popping roaring thunderous scratching snorting crashing crunching cackling tolling clucking silent tapping soothing crowing tranquil melodious cacophonous singing quiet tune loud tinkling noisy rhythmic mumbling twittering din beat blaring cawing racket chattering murmuring whistling clapping booming whispering mewing snapping snoring yelling mooing crackling sighing

SLIDE 5

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Describing sounds

Predominant scheme: Semantic tagging

(sound origin, recording context, etc.)

Sonic qualities are equally important/interesting,

especially for abstract sounds or use in acousmatic composition

Description ⇨ Organization

5

SLIDE 6

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia Grill, Flexer and Cunningham. Identification of perceptual qualities in textural sounds using the repertory grid method. Proceedings of the 6th Audio Mostly Conference, 2011

What are the most significant qualities of textural sounds?

➡Repertory grid technique used to elicit qualities (personal constructs) "ex nihilo", for a specific selection of subjects (interviewees) and objects under examination (items)

Interviewees (subjects) are asked to name differences

between two randomly chosen sound examples ➡Bipolar qualities spanning range from one sound to the other

Identification of perceptual qualities in textural sounds

6

SLIDE 7

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Repertory Grid for sounds

Elicitation of ~10 bipolar constructs per subject
Subjects rate all 20 sounds (grades 1 to 5)

using own personal constructs

7

motion textural impulse high excentric evolutionary well-defined regular narrative pitched smooth static coherent continuous low contained repetitive diffused irregular static non-pitched porous

A

4 4 4 1 2 4 4 2 4 3 3

B

5 3 5 5 5 1 3 1 5 2 1

C

4 5 2 2 4

5

5 3 5 5 4

D

4 2 5 4 3 4 4 3 4 2 3

E

2 4 1 1 2 4 1 5 5 3 5

F

1 1 2 2 2

3

2 5 5 4 5

G

5 5 5 5 5 2 1 2 5 1 1

H

4 3 3 1 2 5 1 1 5 2 4

I

4 2 2 2 2 5 2 2 4 1 4

J

2 1 5 3 1

2

5 5 3 5 3

K

5 2 4 4 4 4 3 1 5 4 2

L

1 1 1 3 1

2

1 5 5 5 5

M

4 5 5 1 2 2 3 2 5 3 2

N

3 1 4 4 1 4 4 5 5 4 2

O

4 2 4 3 3

3

5 4 3 5 3

P

2 2 3 3 3 4 5 3 5 5 4

Q

5 5 5 3 5

5

1 1 5 1 1

R

3 3 4 2 3 2 2 3 4 2 3

S

2 2 5 2 3 4 4 4 2 3 2

T

1 1 4 4 1 4 3 2 3 5 2

1 … 5

SLIDE 8

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

16 subjects
expert listeners
202 constructs
mostly German

high/low

rdered/chaotic

8

SLIDE 9

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia 9

http://grrrr.org/test/classify

SLIDE 10

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Inter-rater agreement

10

*nine subjects who took part in the elicitation process Construct Agreement α (core group)* Agreement α (all n ≥ 10) high – low 0.588 0.519

rdered – chaotic

0.556 0.447 natural – artificial 0.551 0.492 smooth – coarse 0.527 0.420 tonal – noisy 0.523 0.435 homogeneous – heterogeneous 0.519 0.416 dense – sparse 0.492 0.342 edgy – flowing 0.465 0.376 static – dynamic 0.403 0.383 near – far 0.252 0.249

SLIDE 11

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Pearson correlation between constructs

11

SLIDE 12

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Pearson correlation between constructs

12

SLIDE 13

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Visualizing sounds in a collection

Representation of properties of individual sounds

➡Auditory characteristics

Representation of properties of the sound collection

➡Clusters, similarities, dominating characteristics

Waveforms and sonograms?

13

SLIDE 14

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia Lawrence Marks: On Perceptual Metaphors. Metaphor and Symbolic Activity 11(1), 39–66, 1996

Perceptual metaphors

Strong synesthesia

➡very rare, asymmetric, individual

Weak synesthesia – cross-modal similarity

14

SLIDE 15

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia Wolfgang Köhler, Gestalt psychology,1929

Cross-modal similarity

15

SLIDE 16

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Based on most relevant of the previously elicited

personal constructs

Layout in two dimensions (screen compatibility)

➡Tiled map: Iconic representation of individual sounds, map for structural aspects of the collection

Synesthesia-like mappings from auditory to visual domain

Visualization of perceptual qualities in textural sounds

16

SLIDE 17

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Transferring auditory qualities to the visual domain

high–low ⇨ brightness and color hue

(bright yellow–dark red)

ordered–chaotic ⇨ regularity of elements

(on grid–deviating from grid)

tonal–noisy ⇨ saturation

(colorful–gray)

smooth–coarse ⇨ jaggedness of element outline

(smooth–jagged)

homogeneous–heterogeneous ⇨ variation in color parameters

(no variation–much variation)

17

SLIDE 18

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia 18

high–low

rdered–

chaotic tonal– noisy smooth– coarse homogeneous– heterogeneous

SLIDE 19

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Online survey: Visualization of textural sounds

19

http://grrrr.org/test/texvis

SLIDE 20

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Online survey: Visualization of textural sounds

20

Survey A: One of the five examples represents the

current sound, the generation parameters of the other four examples are taken from a uniform random distribution

Survey B: One of the five examples represents the

current sound, the other four represent other existing sounds of the pool

SLIDE 21

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Evaluation: Survey B – Dependence on expertise

21

group voters / votes mean RMS error (random: 0,243) correctness (random: 20%) non-musicians, ≥ 20 votes 19 / 876 0,178 33,9% classical musical training, ≥ 20 votes 29 / 1570 0,163 40,0% electronic music practice, ≥ 20 votes 48 / 2811 0,137 45,2% electronic music practice, good listening conditions, ≥ 20 votes 36 / 2019 0,133 46,4%

SLIDE 22

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Evaluation: Pearson correlation selection to reference

22

Survey B: electronic music practitioners, good listening conditions, ≥ 10 votes

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

high–low

rdered–chaotic

smooth–coarse tonal–noisy homogeneous– heterogeneous

selected

high–low

rdered–chaotic

smooth–coarse tonal–noisy homogeneous– heterogeneous

reference

0.71

0.24
0.33
0.08
0.22
0.27

0.65 0.43 0.36 0.54

0.34

0.43 0.68 0.54 0.27

0.11

0.37 0.55 0.69 0.20

0.25

0.55 0.26 0.18 0.62

SLIDE 23

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Evaluation: Mean RMS error vs. decision time

23

5 10 15 20 25 30 35 40 45

time per vote(s)

0.05 0.10 0.15 0.20 0.25 0.30 0.35

avg RMS error

users=94, x-y correlation=-0.13 @ significance(p=0.05)=0.20 mean duration=13.83 (6.65)

SLIDE 24

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Evaluation: Mean RMS error vs. perceived difficulty

24

0.0 0.2 0.4 0.6 0.8 1.0

perceived difficulty

0.00 0.05 0.10 0.15 0.20 0.25

avg RMS error

sounds=100, x-y correlation=0.484 @ significance(p=0.05)=0.197

SLIDE 25

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia Thomas Grill: Constructing high-level perceptual audio descriptors for textural sounds. Proceedings of the 9th Sound and Music Computing Conference, 2012

Perceptually informed high-level descriptors for textural audio

Attempt to model the previously elicited personal constructs,

i.e. metaphoric descriptions

Build on extensive experimental data from online survey

covering 100 textural sounds ➡Use a uniform underlying time-frequency representation ➡Small number of adjustable parameters for each descriptor ➡Parameters to be tuned, so that the descriptors correlate well with human perception

25

SLIDE 26

Grill and Flexer: Visualization of perceptual qualities in textural sounds ICMC 2012, Ljubljana/Slovenia

Application: Musical interface – Continuous map of textural sounds

26

http://grrrr.org/test/texvis/map.html