Designing Training and Assessment Games guided by the Evaluation - - PowerPoint PPT Presentation

Ivan Boo Chairperson, Serious Games Association Director, Serious Games Asia

Designing Training and Assessment Games guided by the Evaluation Framework

Association

Our primary mission at the Serious Games Association is to create an innovative

community based on the partnership between the subject-matter-experts

(e.g. healthcare practitioners, educators, etc.) and and the the te

technology-experts (e.g.

game developers, apps developers, data scientist, graphic designers, etc.) This should be a sustainable eco-system with a viable business model that will ultimately be able to contribute to the improvement in the quality of lives for the general population.

• Founded Serious Game

mes Association Singapore in 2012

• Became Serious Game

mes Association in 2016 (200 members)

• Established Serious Game

mes Asia in 2017

Eco System1

Developers Content Experts Instructional Designers

Evaluation Framework1

DSSH Evaluation Framework

(A) Game Description (B) Rationale (C) Functionality (D) Validity (E) Data Protection Star Rating Quality Label Evaluation Criteria 1 Star

• 1. The entry is a serious game
• 2. There is a safe data storage mechanism
• 3. The working mechanism and theoretical

background of the game 2 Star

• 4. Face-validity checked and confirmed by

experts

• 5. The underlying mechanism supported by

evidence 3 Star

• 6. Involvement of medical, educational/

psychological and game development experts in the development process

• 7. Relevant user tests performed

4 Star

• 8. Independent experts have validated the

game in a study 5 Star

• 9. Predictive validity has been confirmed in a

scientific validation study & published in a peer-reviewed journal

DSSH Evaluation Framework

(A) Game Description (B) Rationale (C) Functionality (D) Validity (E) Data Protection Star Rating Quality Label Evaluation Criteria 1 Star

• 1. The entry is a serious game
• 2. There is a safe data storage mechanism
• 3. The working mechanism and theoretical

background of the game 2 Star

• 4. Face-validity checked and confirmed by

experts

• 5. The underlying mechanism supported by

evidence 3 Star

• 6. Involvement of medical, educational/

psychological and game development experts in the development process

• 7. Relevant user tests performed

4 Star

• 8. Independent experts have validated the

game in a study 5 Star

• 9. Predictive validity has been confirmed in a

scientific validation study & published in a peer-reviewed journal

Evaluation Framework2

Game Description

1. Operating System: On what operating system(s) does the game run? 2. Web-link: What is the web-link for the game? 3. Version: Which version of the game is this? (Alfa, Beta, release version) 4. Category: How would you categorize this game? (Commercial project, non-commercial project or

• thers)

5. Access: Please specify access to the game? Why did you choose this type of access? 6. Information about the developer: Please fill out Name, address, webpage, contact person(s), email address, phone and fax number of the manufacturer 7. Information about the owner/distributor: Please fill out Name, address, webpage, contact person(s), email address, phone and fax number of the owner/distributor of the game 8. Information about the funding acquired for the development of the serious game: Please fill out funding sources, name of companies and organisations 9. Is the game free of commercial pop-ups and/or advertisements?

• 10. If not what is advertised? What is the rationale for using advertisements?
• 11. Is the game free for users?
• 12. Is it a pay-per download? Please specify
• 13. Are there sources of income within the game?
• 14. What are the sources of income for the owner/distributor?
• 15. Affiliations of Owner/Distributor
• 16. What affiliations do the publishers have that could influence content or user group?
• 17. What interests do the publishers have that could influence content or user group?
• 18. Are conflicts of interest disclosed ? If yes, do people have to accept the disclosure before they can

play the game? If yes, do people have to accept the disclosure before they can play the game?

Eco System2

Developers Content Experts Instructional Designers

DSSH Evaluation Framework

(A) Game Description (B) Rationale (C) Functionality (D) Validity (E) Data Protection Star Rating Quality Label Evaluation Criteria 1 Star

• 1. The entry is a serious game
• 2. There is a safe data storage mechanism
• 3. The working mechanism and theoretical

background of the game 2 Star

• 4. Face-validity checked and confirmed by

experts

• 5. The underlying mechanism supported by

evidence 3 Star

• 6. Involvement of medical, educational/

psychological and game development experts in the development process

• 7. Relevant user tests performed

4 Star

• 8. Independent experts have validated the

game in a study 5 Star

• 9. Predictive validity has been confirmed in a

scientific validation study & published in a peer-reviewed journal

Evaluation Framework3

Rationale

1. What is (are) the purpose(s) of the game? Please describe the goal of the game in one sentence 2. Is (are) the purpose(s) disclosed to user? 3. Is the serious game link to a device?

• If yes, please describe the device.
• If yes and it is applied in healthcare, does it comply to the necessary requirements (FDA-

approval, CE-mark?). Please upload necessary information. 4. Please state the intended user group. And specify for each user group the applicable disease/ condition of healthcare profession 5. Please specify within the user group the demographic variables important to the game, like gender, age range, or other relevant descriptive items? 6. Is the game used in research/ end-user/ training courses, please specify

Eco System3

Developers Content Experts Instructional Designers Singapore University for Technology & Design (SUTD) National University of Singapore (NUS) Temasek Polytechnic (TP)

SingHealth Academia & Serious Games Association

DSSH Evaluation Framework

(A) Game Description (B) Rationale (C) Functionality (D) Validity (E) Data Protection Star Rating Quality Label Evaluation Criteria 1 Star

• 1. The entry is a serious game
• 2. There is a safe data storage mechanism
• 3. The working mechanism and theoretical

background of the game 2 Star

• 4. Face-validity checked and confirmed by

experts

• 5. The underlying mechanism supported by

evidence 3 Star

• 6. Involvement of medical, educational/

psychological and game development experts in the development process

• 7. Relevant user tests performed

4 Star

• 8. Independent experts have validated the

game in a study 5 Star

• 9. Predictive validity has been confirmed in a

scientific validation study & published in a peer-reviewed journal

Evaluation Framework4

Functionality

1. What will the player learn? Describe the learning or behavioral objectives of the game. 2. How are these objectives translated into gameplay and functionality of the game? 3. Are there any in-game instructions, tutorials? If so, please specify. 4. What parameters are used to measure/follow progress or learning effects? 5. What kind of feedback is provided to players? 6. Every serious game has restrictions and limits. In your opinion what content on the subject is NOT covered by the game? 7. Please state potential undesirable effects. This does not include "gaming the game" (cheating), but effects of negative transfer of learning. 8. Is the undesirable effects dislosed to the user? 9. Please state measures taken to prevent undesirable effects

Validity

1. Were content experts involved in the design process from the start? Please specify how? Also add documentation showing this involvement 2. Were representatives of the user group involved in the design process from the start? Please specify how? Also add documentation showing this involvement 3. Did user testing take place? What were the results, and how were these incorporated in the design? Please add documentation (reports, articles e.g.) 4. Does the game produce the same results on different platforms? 5. Is the content of the game validated with experts? Please add documentation/reports/articles 6. How does learning outcome compare to other methods assessing the same construct. Please add documentation/reports/articles 7. Is this game able to predict improvements of skills? Was this game validated in a Randomized Controlled Trial? Was the study published in a peer-reviewed journal? Please add documentation/ reports/ articles

Eco System4

Developers Content Experts Instructional Designers Game Technologies and its Applications Game Data Analytics and Security Serious Games Design vs. Gamification Research Methodology & Outcomes Learning Pedagogies,

• bjectives and
• utcomes

Innovation Education Research

DSSH Evaluation Framework

(A) Game Description (B) Rationale (C) Functionality (D) Validity (E) Data Protection Star Rating Quality Label Evaluation Criteria 1 Star

• 1. The entry is a serious game
• 2. There is a safe data storage mechanism
• 3. The working mechanism and theoretical

background of the game 2 Star

• 4. Face-validity checked and confirmed by

experts

• 5. The underlying mechanism supported by

evidence 3 Star

• 6. Involvement of medical, educational/

psychological and game development experts in the development process

• 7. Relevant user tests performed

4 Star

• 8. Independent experts have validated the

game in a study 5 Star

• 9. Predictive validity has been confirmed in a

scientific validation study & published in a peer-reviewed journal

Evaluation Framework5

Data Security

1. Please elaborate on (if appropriate) information about data collection mechanisms integrated into the serious game. 2. To which countries is the game distributed? 3. Are patient-specific data stored by the game? 4. If yes, are patient informed consent criteria met according to laws in countries stated above? How is informed consent acquired? 5. Who owns and stores the data resulting from play? During what period are the data stored? 6. Can user delete data temporarily and/or permanently? 7. Is data stored and managed in compliance with the privacy laws of the relevant countries? Please specify 8. Are the abovementioned items on data privacy disclosed to the user?

Eco System5

Developers Content Experts Instructional Designers Game Technologies and its Applications Game Data Analytics and Security Serious Games Design vs. Gamification Research Methodology & Outcomes Learning Pedagogies,

• bjectives and
• utcomes

Innovation Education Research Hospital Management Support Serious Games Accreditation & Research Centre Research University Partnership Industry Partnership

Business Model

Serious Games Accreditation & Research Center Shared Games Repository Smart Devices End-Users Data Analytics Developers Content Experts Instructional Designers Institutional Buyer Completed Games Shared Games Assets

Summary

• 1. Training and Development of Clinicians
• 2. Training and Development of Developers
• 3. Understanding Game Data and Leveraging on its Potential
• 4. Lowering Cost of Game Production
• 5. Rethinking Serious Game Business Model

Can we design Training and Assessment Games using the Evaluation Framework?

Serious Games Association

WWW.SERIOUSGAMESASSOCIATION.ORG

Thank you