MERGING AR AND PIANO EDUCATION: AN APPLICATION DESIGNED AROUND PIANO - - PowerPoint PPT Presentation

MERGING AR AND PIANO EDUCATION: AN APPLICATION DESIGNED AROUND PIANO INSTRUCTORS AND ASSISTED TEACHING

Amare re Birha hanu nu

Stefan Rank Paul Diefenbach & Youngmoo Kim

Presentation 1

Research Question

Research Question

1. I will describe the process of creating an AR application for the HoloLens with the purpose of it being integrated into piano lessons as a tool instructors can use to teach their students

Research Question

1. I will describe the process of creating an AR application for the HoloLens with the purpose of it being integrated into piano lessons as a tool instructors can use to teach their students 2. because I want to uncover design choices, features, and visuals that support student learning based on the opinion and expertise of experienced piano instructors,

Research Question

1. I will describe the process of creating an AR application for the HoloLens with the purpose of it being integrated into piano lessons as a tool instructors can use to teach their students 2. because I want to uncover design choices, features, and visuals that support student learning based on the opinion and expertise of experienced piano instructors, 3. in order to create a basic standard for AR piano training application intended to be integrated into formal piano lessons.

Contribution –

What I Hope to Uncover

Contribution –

What I Hope to Uncover

■ General set of guidelines for AR piano applications based on expert opinions.

Contribution –

What I Hope to Uncover

■ General set of guidelines for AR piano applications based on expert opinions. ■ Teacher preferences, dislikes, and concerns.

Contribution –

What I Hope to Uncover

■ General set of guidelines for AR piano applications based on expert opinions. ■ Teacher preferences, dislikes, and concerns. ■ Minimum requirements for creating an application intended to be used by piano instructors who intend on incorporating AR technology in their lessons.

Contribution –

What I Hope to Uncover

■ General set of guidelines for AR piano applications based on expert opinions. ■ Teacher preferences, dislikes, and concerns. ■ Minimum requirements for creating an application intended to be used by piano instructors who intend on incorporating AR technology in their lessons. ■ Unexplored and experimental ways of providing augmented visual support to the student.

Common Attributes & Features –

as seen in previous examples

Hand Reference Light up Keys Mainly Improvisation Floating Sheet Music Falling Notes Song Selection Hand Reference Falling Notes Song Selection

LATEST RESEARCH

Week 01 – Week 03

The Cognitive Theory of Multimedia Learning

■ Theory centers around the idea that learners build meaningful connections between words and pictures. – Meaningful learning happens when the learner engages in cognitive process such as using relevant words or images to organize a mental model and integrate those representations with prior or new knowledge. ■ To be effective it must be designed to allow for an adequate period of usage. – Actively constructing new knowledge. ■ Design must be learner-centered and not technology centered. – Understand learner limitations.

Sorden, Stephen D. "The Cognitive Theory of Multimedia Learning." (n.d.): n. pag. Web. Apr. 2017.

Transfer Learning

■ Storing knowledge gained while solving one problem then applying that new knowledge to a different yet relatable problem. – Source Task Target Task ■ Has happened if a task has led to improvements in another task. – Target task is new to the user but requires similar skills as the source task. – Improve learning in target task by leveraging knowledge in the source task. ■ Goal is to produce positive transfer between two appropriately related tasks while avoiding negative transfer between tasks that are less related. – Dependent on relationship between tasks. – Strong relationship in commonalities will result in a stronger transfer of knowledge.

Torrey, Lisa, and Jude Shavlik. "Transfer Learning." Handbook of Research on Machine Learning Applications (2009): n. pag. IGI Global. Web.

AR in Education: A Meta-Review & Cross-Media Analysis

■ Compares 26 publications that have compared AR versus non-AR learning methods. – Provides a list of positive and negative impacts of AR experiences on learning. ■ Tend to show significantly faster speed in locating important information. – Show significantly less head movements. – Faster task completions with Less Errors. ■ Learning detriments that could potentially come from AR: – Tunneling: High attentional demand on non-essential parts of the experience. – Usability: May be effective for learning but are commonly difficult to use.

Radu, Iuli. "Augmented reality in education: a meta-review and cross-media analysis" Springer-Verlag London, 2014. Web. Apr. 2017.

AR in Education: A Meta-Review & Cross-Media Analysis

■ Systems can monitor user activity and provide relevant feedback. – Bringing into context progression that has been made. – Reducing the need for the learner to switch their attention between sources. ■ Users that are presented content through AR have stronger memories of their experiences. – Physical immersion and learning from tactile information. ■ Digital content can be used to guide user’s attention. – Good for helping to complete a tasks. – Utilizes visual perception of the spatial relationships of objects in the user’s environment.

Radu, Iuli. "Augmented reality in education: a meta-review and cross-media analysis" Springer-Verlag London, 2014. Web. Apr. 2017.

AR in Education: A Meta-Review & Cross-Media Analysis

■ A heuristic questionnaire constructed from the above study. 1. The application transforms the problem representation such that difficult concepts are easier to understand. 2. The application presents relevant educational information at the appropriate time and place, providing easy access to information and/or reducing extraneous learner tasks. 3. The application directs learner attention to important aspects of the educational experience. 4. The application enables learners to physically enact, or to feel physically immersed in, the educational concepts. 5. The application permits students to interact with spatially challenging phenomena.

Radu, Iuli. "Augmented reality in education: a meta-review and cross-media analysis" Springer-Verlag London, 2014. Web. Apr. 2017.

Affordances and Limitations of Immersive Participatory for AR simulations for Teaching and Learning

■ Documents how teachers and students participating in AR simulations can aid or hinder the teaching or learning process. ■ Students develop patterns of participation that shapes their identities as learners. – The way in which the engage in learning. – What the student believes their ability to learn is. ■ AR has a unique capacity to activate, recruit, and cultivate a sense of projective identity that serves as a mediation between the students’ real world identity and their virtual or game identity. – Virtual identities it can then leverage to influence and shape an ongoing transformation of real-world identities (Gee 2003).

Dunleavy, Matt, Chris Dede, and Rebecca Mitchell. "Affordances and Limitations of Immersive Participatory Augmented Reality Simulations for Teaching and Learning." Journal of Science Education and Technology 18.1 (2008): 7-22. Web.

The Role of Perceived Enjoyment in the Students Acceptance of an AR Teaching Platform: A Structural Equation Modelling Approach

■ Investigate the user’s acceptance to understand the factors that influence a user’s intention to use AR technology during learning experiences. – The TAM model (perceived ease of use, perceived usefulness, and perceived enjoyment) determine one’s behavioral intention to use a technology. ■ AR devices tend to be harder to use than more traditional learning tools but, – Provide a greater intrinsic motivation because of novelty and versatility associated with new technology. ■ Perceived enjoyment (intrinsic motivational factor) and perceived usefulness (extrinsic motivational factor ) are important to understand user intentions.

Balog, Alexandru , and Costin Pribeanu. "The Role of Perceived Enjoyment in the Students’ Acceptance of an Augmented Reality Teaching Platform: a Structural Equation Modelling Approach." Studies in Informatics and Control 19.3 (2010): 319-30. Web. Apr. 2017.

KeyboARd

"KeyboARd." Devpost. N.p., n.d. Web. 24 Apr. 2017. <https://devpost.com/software/keyboard>. https://github.com/camelKaiser/KeyboARd

Options for MIDI/Real-Time Pitch Detection

Tutorial/Source Code

■ Using Microphone Input in Unity3D – Kappine.fi ■ Fundamental Frequencies and Detecting Notes – Kappine.fi ■ Pitch Detection for Music – StackOverflow.com

Unity Asset

■ Koreographer - $36 / Pro $116 – MIDI Converter ■ Pitch Detector - $15 – Real-Time Pitch Detection

PROJECT UPDATE

Spring Break – Week 03

Spring Break – Confirmed 1st Panel Member & Created Form

■ Confirmed first member of my expert panel. – Unofficial First Meeting (Official - Friday April 28th) – Discussed Research and Technology ■ Created Form for Prospect Panel Member to Fill Out – https://docs.google.com/forms/d/1TNQrrIZvRekdI3rMoz62r8NimTmxaKengJx uNA2lrwY/edit ■ Made final revisions to thesis proposal before submitting to advisor and committee. – Submission March, 31st 2017.

Week 01 – HoloAcademy & Thesis Approval

■ Thesis proposal was approved by Stefan and Youngmoo. – Youngmoo said he wants me to add a section on transfer learning. – Stefan told me to start integrating user interaction to application. ■ Completed HoloAcademy 101 and 211 – Holograms 101: Gaze, Gestures, Voice, Spatial Sound, and Spatial Mapping – Holograms 211: Hand Detection, Navigation, Hand Guidance, and Manipulation

Week 02 – Basic Hologram Manipulation

■ Update/Finalize Table of Contents ■ Basic Manipulation of Holograms – Integrated lessons learned from 101 course to make piano stage moveable.

Week 03 – New Script for Generating Notes

■ Reviewed Paul’s Feedback on Thesis Proposal ■ Almost Finished Revising Thesis Intro ■ Started Making a New Script for Managing Virtual Notes are Deployed – OctaveSpawner.cs

TIMELINE

Week 04 – Completion

General Construct

Initial Meeting & Discussion

Confirm Remaining Panel Members

Implement Features and Design Choices Into Prototype 1st Round Iterative Playtest & Feedback 2nd Round Iterative Playtest & Feedback

Update Prototype Update Prototype

3rd Round Iterative Playtest & Feedback 2nd Round Iterative Playtest & Feedback

Update Prototype Thesis Defense

Week 04 – Week 06

April 24th – May 14th

■ Meet with Committee – Feedback on updated system ■ Meet with Dave Rosen – Connection for Piano Instructors ■ Official Initial Meeting – Janet Miller – Prepare Post Meeting Questionnaire (Initial Only) ■ Confirm Final Panel Members ■ Start integrating Sound Recognition/Feedback ■ Post Iterative Playtest Questionnaire ■ Complete Initial Meeting w/ Panel – Integrate Feedback into Project ■ Conduct 1st Round of Iterative Playtest – Integrate Feedback into Project ■ Thesis Paper – Complete Introduction, Literature Review, and Proposed Approach Sections.

Week 07 – Week 10

May 15th – June 5th

■ 2nd Class Presentation ■ Conduct 2nd Iterative Playtest – Integrate Feedback ■ Final Iterative Playtest – Integrate Feedback ■ Thesis Defense ■ Final Class Presentation ■ Thesis Paper – Implementation Section – Expert Review Section – Analysis & Conclusion Section

DEMO

KeynVisionV0.1

THANK YOU!

There now, that wasn’t so bad, was it?