Musical Structure Visualization Program Introduction Background - - PowerPoint PPT Presentation

a report on

Musical Structure Visualization

By Winnie Chan August 2007

2

Program

• Introduction

 Background  Motivations  Challenges

• Definitions and Technology
• Visualization Techniques
• Discussion
• Conclusions

3

Introduction

• We hear music unconsciously
• We listen to music consciously, which

requires efforts to understand

 Musical structure  Composition techniques

• Such understanding is vital for music

appreciation and comprehension

4

Pop Music Vs. Classical Music

• Pop music is meant to be friendly
• Classical music is seemingly

sophisticated

 It requires training to recognize structure

and form of music

 Untrained one may only be able to “feel”

the music and the music is thus boring

5

The Music Being Discussed

• Classical music

 Also known as fine music, classics, …  Within the basic music history context,

from Baroque period to 20th century

 Focuses on instrumental music

(vocal music is put aside)

 Simple definition: Strictly organized

compositions (still very vague though)

6

Background

• Realizing the musical structure is

important to appreciate the music

• Music expertise: Able to single out the

musical elements by

 Reading the score  Listening to the performance

7

Why Learning Music is Harder?

• Fine art (paintings and sculptures) and

literature have a physical, concrete

• bjects that we can “see”
• Music is abstract and dynamic:

 It is difficult for unskilled ears to recognize

complicated musical elements from multi-layered music

 Time-varying nature makes comparisons

and appreciation harder

8

Music is Unique

• The lack of visual equivalence is what

makes music different from non- auditory art forms

• The only art form that is perceived with
• ur hearing ability

9

Musical Scores / Sheet Music (乐谱)

• The most common way to learn music
• Contains objective notations

 Reading scores is demanding  Beginners have to learn the basics of music

theory to know the notations, and still a long way for in-depth understanding

 It is tedious for amateurs to go through the score,

those overwhelming technical details are useful for performance but not listening

• In practice not convenient to get full score

10

Motivations

• Learning curve of classical music is

high; visual display should help

• Structure of music is rarely visualized,

most attempts visualize the sonic features like Windows Media Player

• Multidimensional nature of music is not

well addressed

11

Research Issues

• How to create informative and

insightful visualization which helps people with limited background to

 See the sound visually  Understand the structure of music

semantically

 Identify the musical elements when

listening to the performance

12

Challenges (1) – Defining the data structure

• Seek a general structure that can

capture the essential musical elements in various compositions

• Decide what particular genre (类型)

we work on and what the basic elements are

13

Challenges (2) – Choosing the analysis method

• Automatically analyze MIDI file

 Ensure this digital score is accurate  Derive algorithms to retrieve cognitive elements

from the score (!)

 Help audience to appreciate music qualitatively  Can be quantified artificially, but not sure if

computation is possible

 Be validated to be held universally

• Manually analyze music literature

 Extract useful information  Provide user-interaction for human-defined input

14

Challenges (3) – Designing effective encoding scheme

• Find the best visual mapping

(equivalence) for music elements

• Combine these representations

without much interference

15

Program

• Introduction
• Definitions and Technology

 Sound and Music  Music Visualization

• Visualization Techniques
• Discussion
• Conclusions

16

What is Sound?

• Does not have specific meanings
• Qualities:

 Frequency (pitch)  Volume  Speed  Spatial position of sound source  (other physical qualities)

17

What is Music?

• Is collection of organized sound
• Conveys certain messages / emotions
• Properties:

 Rhythm (节奏)  Chords (和弦)  Tempo (速度)  Dynamics (力度变化)  Timbre (音色)  Texture (组织,结构)  Harmony (和声)  …

18

Music Visualization

• Visualizes the loudness and frequency

spectrum of music, e.g.

 Oscilloscope display on radio  Animated imagery rendered by player

• Should be distinguished from musical

structure visualization

19

Program

• Introduction
• Definitions and Technology
• Visualization Techniques
• Discussion
• Conclusions

20

Arc Diagrams

• Visualizes complex patterns of

repetition by connecting a translucent arc between a pair of matching pair

21

Arc Diagrams (cont.)

• Outcomes are aesthetically pleasing
• Patterns identified may not

correspond to musical repeated units

 Theme, subject, motif (主题,主旋律)  More specific Wagner’s leitmotif and

Berlioz’s idée fix

22

Isochords

• Visualizes chord structure, progression

and voicing of MIDI

• Applies Tonnetz (German word for

tone-network) to place each chord type in a 2D space

• Major contribution:

 Proposes an animated display  Show various chord properties in the

same representation

23

Constructing Isochords

• Music note  dot
• Dots of chord notes  Isochords geometry
• Major / minor  upward / downward pointing

(different chord types have different shapes)

• Consonant notes (协和音)  connected by edge
• Chord progression  adjacent structure
• Modulation (变调)  color
• Which octave (八度音阶)  size of triangle vertices

Demo Warum sollt ich mich denn grämen by J. S. Bach

24

Isochords Discussion

• Chords are important in music analysis

but not for listening purpose

• The structure of music defined here is

low-level about notes and chords

 Constructive and structural materials vs.

abstract and semantic structure

• Most research deal with these low-

level details for music theory training

25

ImproViz

• It shows jazz improvisations (即兴创作)
• Melodic landscape shows contour of

a melodic line

• Harmony palette shows use of chords
• Whole design is solely based on All

Blues by Miles Davis

• Results are crafted in Adobe Illustrator
• Input is some kind of real sheet music

26

ImproViz Example

27

ImproViz Discussion

• It visualizes the improvisations

recorded as a score, not the true impromptu

• It seems to attack a new problem (jazz

improvisations) but the techniques are actually fairly general

• Melodic contour is effective

28

Graphic Scores

• Are made for two electro-acoustic compositions
• Are not generalized
• Do not have any solid visual mapping

29

Suggestions from Graphic Scores

• Traditional musical score is written for performance,

not analysis

• A score is a graphic description of sonic events and

many structural details are often missed

• To make an effective study score for listeners:

 Simple but visually identifiable sonic events (qualitative)  Temporal logic should match spacial logic (time-varying)  Full score should be visible at a glance (overview)  Score reading is not the most important  Study scores are for listening (ears), not for analysis

30

Visual Music

• Done by the author of graphic scores
• Appeared in SIGGRAPH posters and sketches:

 Sonic map  Time slice / Heterophonic map

• Map color to sound, and vice versa

31

Simplified Scores

• Show whole score at once
• Target at users with limited

background

• Map:

 Measure (小节)  vertical line  No. of notes  brightness  Dynamics (力度变化)  width

• Applied fisheye
• Were not formally evaluated

32

Performance Expression Visualization

• It deals with the expressions brought by

different performance

• Input is augmented scores with expressive

attributes that do not appear in the original musical scores

• It visualizes the depth of performance
• Performance is described by cognitive

terms: melody, rhythm and phasing

33

More on Performance Visualization

• Performance visualization displays

qualitative musical characteristics

• MIDI parameters do not have music sense

connected to human perception

• It is not necessary to render the original

scores faithfully – should be abstract for comprehending the underlying music and structure

34

Performance Visualization (1) – Vertical Bar Display Grid

• Time  x-axis
• Relative dynamics

 y-axis

• Local tempo variation

(difference between written and performed)  interval between two grid lines

• Note played  bar

35

Performance Visualization (1) – Vertical Bar Display Mapping

• Dynamics

 height of bar

• Articulation (弹奏技巧)

 width of bar

• Expressiveness of note

 gray scale of bar

• Player’s phasing

 repeated patterns

Connected: legato (连奏) Discrete: staccato (断奏)

36

Performance Visualization (1) – Vertical Bar Display User Studies

• Users were asked to match

performances to graphical displays

• Results not satisfactory as user-

interaction was were limited

 No visual clues for users to trace the

performance when listening to the music

37

Performance Visualization (2) – Chernoff Faces

• Tempo  eyeball position
• Articulation  contour of face
• Dynamics change  shape of nose

38

Performance Visualization (3) – Hierarchical Approaches

• Cone Trees are used to visual hierarchical

musical structure

• It is not certain how the results were

generated

39

Performance Visualization (4) – Music and Emotion

• Mood of music is visualized as a snapshot of

performance with rectangular texture

• Mapping:

 Note  small square  Expression attribute  color

• Squares are arranged based on

importance of the note, not the temporal sequence

40

3D Music Visualization

• Represents music / sound as objects
• Utilizes 3D space to arrange the objects according

to time, pitch, instrument, etc.

• Allows user to manipulate the sound objects to

achieve some performance

41

Commercial Products – TimeSketch

• Facilitates listening and analyzing music
• Be used to create guided listening lessons
• Takes input from experts / educators
• Encodes musical theme as bubble chart

 Theme  half-disks  Hierarchical relationships  larger semi-circle  Related passages  color

42

Commercial Products – Music Animation Machine

• Basic bar-graph:

 Note  bar  Duration as performed  bar length  Pitch  vertical position  Timing  horizontal position  Instruments / voices  bar color

43

Commercial Products – Music Animation Machine (cont.)

• Music Animation Machine MIDI Player has various

modules that visualize chords, intervals, melody and harmony

• These modules are separated and not rendered

synchronically

(demo)

44

Visualization in Computer Music (1)

Self-similarity grid

• Visualizes time structure of

musical waveforms

• Shows similarity with

brightness of pixel Harmonic visualization

• Maps tonality to color
• Time  x-axis
• Analysis window size  y-axis

45

Visualization in Computer Music (2)

Self-organizing map (SOM)

• Visualizes tonal content
• Shows how keys are

related to each other Spiral layout

• Shows tonal evolution
• Represent tonal elements

in 3D space

• Indicates keys with dots

46

Visualization in Computer Music (3)

• Mainly for visual analysis of detailed

musical structure

• Good for studying music theory
• May not be useful for general listeners

47

Program

• Introduction
• Definitions and Technology
• Visualization Techniques
• Discussion
• Conclusions

48

Discussion

• Relevant work on visualizing abstract

musical structure is limited

 Hard to identify qualitative features

automatically from the score

 Visualization usually treated as a tool to

present complicated music details

• Needs of untrained ones are
• verlooked

49

Low-level vs. High-level Structure

Structural elements

• Tones, chords and

keys

• Aids music theory

training

• Assumes some user

background Cognitive structure

• Theme, form,
• rganization
• Visualizes abstract

structure for learning composition

• Does not require any

background

50

Possible Directions

• Accept MIDI files as input
• Apply some music analysis algorithms to generate

initial visualization results

• Allow users to modify the data via user-interaction

 Refine visual mapping  Define more cognitive terms

• Use a sketch-based metaphor, just like highlighting

important features on the score when music students learn a piece of work

51

Lessons Learned

• Most systems:

 Are Implemented in Java  Use XML standard for input data structure

• Observations:

 Use animation  Conduct formal user study

52

Program

• Introduction
• Definitions and Technology
• Visualization Techniques
• Discussion
• Conclusions

53

Conclusion

• Musical structure visualization

 Is a visual reinforcement of the abstract structure of music  Is useful for appreciating and comprehending music  Provides good alternative to typical music literature essays  Allows understanding and learning music perceptually

with minimal expertise

• Visions:

 Aids listening interactively  Be appended to posters and concert programme  Is used for comparing compositions from different periods

and composers

The End – Coda

After silence, that which comes nearest to expressing the inexpressible is music.

• Aldous Huxley, “Music at Night”, 1931

”