MidiLens An Augmented Reality Piano Learning System using HoloLens - PowerPoint PPT Presentation

MidiLens An Augmented Reality Piano Learning System using HoloLens Course: Interaction Engineering (WS_17/18) Ruben Schlagowski 954174 Course Instructor: Prof. Dr. Michael Kipp Martin Stadlmaier 954084 Submission: 02. February 2018

2

[Motivation] ● Simplify musical training ● Make it more effective and fun ● Motivate more people to start playing music 3

[Idea] Using Augmented Reality on a Head Mounted Display (HMD) for music learning on a Piano ! 4

[Related Work] 5

6

[Interaction Techniques] ● Song Selection: Mouse, Keyboard (PC) ● Musical Input: Classical Keyboard (Master-Keyboard) ● HMD Application Control: Speech, Gaze & Gesture (HoloLens) ● Visual Output: Augmented Reality on OST- Displays (HoloLens) ● Auditory Output: Speakers (PC) 7

[Technical Realisation] (Unity/C#) (Windows Forms, C#) 8

Learning Mode A Learning Mode B 9

[User Tests] 10

[Test Sequence] Song Playthroughs Song: 1 Song: 1 Introduction Mode: A Mode: B Song: 2 Song: 2 Interview Mode: A Mode: B 11

[Results] A B Prefered Mode ? 55,6% Yes Potential for 44,4% excessive use? No 22,2% Maybe 33,3% 12

[Live Demo] 13

[Conclusion] ● Generally positive feedback (fun factor!) ● Prefered Learning Mode: A ● Positive effect on skill development questionable ● tracking precision must improve 14

[Future Work] ● Gamification via Score and rising difficulty ● Improve Reliability with Midi over TCP/IP ● More user studies regarding timing/precision ● Add finger positioning 15

Thanks for your attention! 16