20112022: SIMSSA Single Interface for Music Score Searching and - - PowerPoint PPT Presentation

▶
2011 2022 simssa single interface for music score
SMART_READER_LITE
LIVE PREVIEW

20112022: SIMSSA Single Interface for Music Score Searching and - - PowerPoint PPT Presentation

Aug 15, 2023 109 likes •447 views

Update on the SIMSSA Project Single Interface for Music Score Searching and Analysis Ichiro Fujinaga Music Technology Area, Schulich School of Music M c G i l l U n i v e r s i t y 20112022: SIMSSA Single Interface for Music Score

slide-1
SLIDE 1

Update on the SIMSSA Project

Single Interface for Music Score Searching and Analysis

Ichiro Fujinaga

Music Technology Area, Schulich School of Music

M c G i l l U n i v e r s i t y

slide-2
SLIDE 2

Utrecht DM@DH2019 Fujinaga

/32

❖ Create a prototype for single website to search all digitized music scores world wide
 (“Google scores” minus Google) ❖ Use OMR to make them content searchable: concentrate

  • n early music

❖ Includes basic analytical tools on the website ❖ Funded by SSHRC and FRQSC: $3.4M CAD (€2.3M)

2011–2022: SIMSSA
 Single Interface for Music Score Searching and Analysis

2

slide-3
SLIDE 3

Utrecht DM@DH2019 Fujinaga

/32

Optical Music Recognition (OMR)

3

OMR

A process of converting images of music scores into a symbolic computer representation, such as MIDI, MusicXML, or MEI (Music Encoding Initiative).

slide-4
SLIDE 4

Utrecht DM@DH2019 Fujinaga

/32

Steps Involved in OMR

4

Music Symbol Recognition Music Notation Reconstruction Image
 Preprocessing

Digitized
 Score Final Output

Symbol
 Combination Semantic
 Assignment
 (pitch, value) Staves
 Processing Symbol
 Segmentation Musical
 Structure 
 Reconstruction Binarization Structural
 Analysis Noise Removal Image
 Segmentation Symbol
 Classification

slide-5
SLIDE 5

Utrecht DM@DH2019 Fujinaga

/32

SIMSSA Workflow for Neume Notation

5

Layout Analysis 
 Calvo’s Method Layout Correction Pixel.js Symbol Classification Gamera Classification Correction InteractiveClassifier.js Automatic Pitch Detection Pitch and Other Corrections: Neon.js Digitized Manuscript Cantus Database Cantus Ultimus Interface

slide-6
SLIDE 6

Greyscale Binarization Border Removal Lyric Removal Stafg Removal Shape Classification Music Reconstruction Shape/Image Alignment

d d c dc cb c dedcd dfd edc

C clef

punctum

slide-7
SLIDE 7

Layout Analysis 
 Calvo’s Method

slide-8
SLIDE 8

Utrecht DM@DH2019 Fujinaga

/32

Three Difgerent Outputs in One Step!
 Using Convolutional Neural Networks

8

slide-9
SLIDE 9

Utrecht DM@DH2019 Fujinaga

/32

❖ S

9

Separation of Staves, Notes & Texts

Jorge Calvo Zaragoza

slide-10
SLIDE 10

Utrecht DM@DH2019 Fujinaga

/32 10

Selectional Auto Encoders

Jorge Calvo Zaragoza

slide-11
SLIDE 11

Utrecht DM@DH2019 Fujinaga

/32 11

slide-12
SLIDE 12

Utrecht DM@DH2019 Fujinaga

/32

Accuracy & Training Time Comparison

12

Selective Auto Encoders vs Convolutional Neural Nets

Two Medieval Manuscripts: Salzinnes & Einsiedeln

slide-13
SLIDE 13

Utrecht DM@DH2019 Fujinaga

/32

Pixel.js

Zeyad Saleh, Ké Zhang & Eric Liu

13

slide-14
SLIDE 14

Utrecht DM@DH2019 Fujinaga

/32

Original Image & Ground Truth

14

Ground Truth Original Image

slide-15
SLIDE 15

Utrecht DM@DH2019 Fujinaga

/32 15

Classification of an Unseen Page

slide-16
SLIDE 16

Utrecht DM@DH2019 Fujinaga

/32

Classification of an Unseen Page

16

slide-17
SLIDE 17

Utrecht DM@DH2019 Fujinaga

/32 17

Classification of an Unseen Page

slide-18
SLIDE 18

Utrecht DM@DH2019 Fujinaga

/32 18

InteractiveClassifier.js

Minh Anh Nguyen

slide-19
SLIDE 19

Utrecht DM@DH2019 Fujinaga

/32

Neume Mapping Table to MEI

19

slide-20
SLIDE 20

Utrecht DM@DH2019 Fujinaga

/32 20

Neume Mapping Tool

Imane Chafi

slide-21
SLIDE 21

Utrecht DM@DH2019 Fujinaga

/32 21

OCR & Text Alignment

Timothy de Reuse

slide-22
SLIDE 22

Utrecht DM@DH2019 Fujinaga

/32 22

Neon.js: Version 3

Juliette Regimbal

slide-23
SLIDE 23

Utrecht DM@DH2019 Fujinaga

/32

❖ Background images displayed with diva.js

❖ IIIF compliant!

❖ Editing via Verovio

❖ The first of version of Verovio that is editable! ❖ Also text is editable

23

Neon.js: Version 3

Juliette Regimbal

What’s new

slide-24
SLIDE 24

Utrecht DM@DH2019 Fujinaga

/32 24

Neon.js: Text Editing

Jacob Hutnyk

slide-25
SLIDE 25

Utrecht DM@DH2019 Fujinaga

/32 25

Neon.js: Text Editing

Jacob Hutnyk

slide-26
SLIDE 26

Utrecht DM@DH2019 Fujinaga

/32 26

Neon.js: Text Editing

Jacob Hutnyk

slide-27
SLIDE 27

Utrecht DM@DH2019 Fujinaga

/32

SUPERIUS TENOR ALTUS BASSUS

Scoring-up Tool: Song Book

Martha Thomae

27

slide-28
SLIDE 28

Utrecht DM@DH2019 Fujinaga

/32

SUPERIUS TENOR ALTUS BASSUS

28

Scoring-up Tool: Song Book

Martha Thomae

slide-29
SLIDE 29

Utrecht DM@DH2019 Fujinaga

/32

❖ SIMSSA ❖ Document an analysis with Selective Auto Encoders ❖ Pixel.js ❖ Interactive Classifier ❖ Neume Mapping Tool ❖ OCR and text editing ❖ Neon.js

❖ Text editing

❖ Scoring-up Tool

Summary

29

slide-30
SLIDE 30

Utrecht DM@DH2019 Fujinaga

/32

SIMSSA Team @McGill: Summer 2019

30 Rian Adamian Imane Chafi Julie Cumming Alex Daigle Tim de Reuse Glen Ethier Emily Hopkins Jacob Hutnyk Alessandra Ignesti Yaolong Ju Sam Howes Andrew Kam Ian Lorenz Sylvain Margot Cory McKay Zoé McLennan Néstor Nápoles Minh Anh Nguyen Gustavo Pedro Juliette Regimbal Evan Savage Peter Schubert Martha Thomae Andrew Tran Vi-An Tran Gabriel Vigliensoni

slide-31
SLIDE 31

Utrecht DM@DH2019 Fujinaga

/32

Christopher Antila Claire Arthur

William Bain Ryan Bannon Laurier Baribeau Noah Baxter Laura Beauchamp Justin Bell Ruth Berkow Marina Borsodi-Benson

John Ashley Burgoyne

Greg Burlet Jorge Calvo-Zaragoza Remi Chiu

Morgane Ciot Nat Condit-Schultz

Julie Cumming Alex Daigle Marie DeYoung Tim de Reuse

Natasha Dillabough Daniel Donnelly

Neda Eshraghi

Meredith Evans

Wei Gao David Garfinkel Mahtab Ghamsari Arielle Goldman

Ryan Groves

Jamie Klassen Peter Henderson Jason Hockman Emily Hopkinson Andrew Horwitz Yaolong Ju Andrew Kam Anton Khelou Jamie Klassen Reiner Krämer Véronique Lagacé Saining Li Eric Liu Wendy Liu Evan Magoni Zoé McLennan Nicky Mirfallah Lillio Mok Alexander Morgan

Catherine Motuz Maria Murphy Néstor Nápoles

Clare Neil Minh Anh Nguyen Chris Niven Rory O’Connor Laura Osterlund Phyllis Ouyang Jérôme Parent-Lévesque Alexandre Parmentier Gustavo Pedro Sacha Perry-Fagant Alastair Porter Juliette Regimbal Deepanjan Roy Zeyad Saleh Harry Simmonds Brian Stern Tristano Tenaglia Martha Thomae Andrew Tran Vi-An Tran Gabriel Vigliensoni Tim Wilfong Mike Winters Ling-Xiao Yang Ké Zhang

Acknowledgements

31

slide-32
SLIDE 32

Utrecht DM@DH2019 Fujinaga

/32

Webpage: http://simssa.ca Github sources: https://github.com/DDMAL

32

Funded by