Designing Training and Assessment Games guided by the Evaluation - PowerPoint PPT Presentation

Designing Training and Assessment Games guided by the Evaluation Framework Ivan Boo Chairperson, Serious Games Association Director, Serious Games Asia

Our primary mission at the Serious Games Association is to create an innovative Association community based on the partnership between the subject-matter-experts the te technology-experts (e.g. (e.g. healthcare practitioners, educators, etc.) and and the game developers, apps developers, data scientist, graphic designers, etc.) This should be a sustainable eco-system with a viable business model that will ultimately be able to contribute to the improvement in the quality of lives for the general population. • Founded Serious Game mes Association Singapore in 2012 • Became Serious Game mes Association in 2016 (200 members) • Established Serious Game mes Asia in 2017

Eco System 1 Instructional Developers Content Experts Designers

Evaluation Framework 1 DSSH Evaluation Framework Star Rating Quality Label Evaluation Criteria 1 Star 1. The entry is a serious game (A) Game Description 2. There is a safe data storage mechanism (B) Rationale 3. The working mechanism and theoretical background of the game (C) Functionality 2 Star 4. Face-validity checked and confirmed by (D) Validity experts 5. The underlying mechanism supported by (E) Data Protection evidence 3 Star 6. Involvement of medical, educational/ psychological and game development experts in the development process 7. Relevant user tests performed 4 Star 8. Independent experts have validated the game in a study 5 Star 9. Predictive validity has been confirmed in a scientific validation study & published in a peer-reviewed journal

Evaluation Framework 2 DSSH Evaluation Framework Star Rating Quality Label Evaluation Criteria 1 Star 1. The entry is a serious game (A) Game Description 2. There is a safe data storage mechanism (B) Rationale 3. The working mechanism and theoretical background of the game (C) Functionality 2 Star 4. Face-validity checked and confirmed by (D) Validity experts 5. The underlying mechanism supported by (E) Data Protection evidence 3 Star 6. Involvement of medical, educational/ psychological and game development experts in the development process 7. Relevant user tests performed 4 Star 8. Independent experts have validated the game in a study 5 Star 9. Predictive validity has been confirmed in a scientific validation study & published in a peer-reviewed journal

1. Operating System: On what operating system(s) does the game run? 2. Web-link: What is the web-link for the game? Game Description 3. Version: Which version of the game is this? (Alfa, Beta, release version) 4. Category: How would you categorize this game? (Commercial project, non-commercial project or others) 5. Access: Please specify access to the game? Why did you choose this type of access? 6. Information about the developer: Please fill out Name, address, webpage, contact person(s), email address, phone and fax number of the manufacturer 7. Information about the owner/distributor: Please fill out Name, address, webpage, contact person(s), email address, phone and fax number of the owner/distributor of the game 8. Information about the funding acquired for the development of the serious game: Please fill out funding sources, name of companies and organisations 9. Is the game free of commercial pop-ups and/or advertisements? 10. If not what is advertised? What is the rationale for using advertisements? 11. Is the game free for users? 12. Is it a pay-per download? Please specify 13. Are there sources of income within the game? 14. What are the sources of income for the owner/distributor? 15. Affiliations of Owner/Distributor 16. What affiliations do the publishers have that could influence content or user group? 17. What interests do the publishers have that could influence content or user group? 18. Are conflicts of interest disclosed ? If yes, do people have to accept the disclosure before they can play the game? If yes, do people have to accept the disclosure before they can play the game?

Eco System 2 Instructional Developers Content Experts Designers

Evaluation Framework 3 DSSH Evaluation Framework Star Rating Quality Label Evaluation Criteria 1 Star 1. The entry is a serious game (A) Game Description 2. There is a safe data storage mechanism (B) Rationale 3. The working mechanism and theoretical background of the game (C) Functionality 2 Star 4. Face-validity checked and confirmed by (D) Validity experts 5. The underlying mechanism supported by (E) Data Protection evidence 3 Star 6. Involvement of medical, educational/ psychological and game development experts in the development process 7. Relevant user tests performed 4 Star 8. Independent experts have validated the game in a study 5 Star 9. Predictive validity has been confirmed in a scientific validation study & published in a peer-reviewed journal

1. What is (are) the purpose(s) of the game? Please describe the goal of the game in one sentence 2. Is (are) the purpose(s) disclosed to user? 3. Is the serious game link to a device? Rationale • If yes, please describe the device. • If yes and it is applied in healthcare, does it comply to the necessary requirements (FDA- approval, CE-mark?). Please upload necessary information. 4. Please state the intended user group. And specify for each user group the applicable disease/ condition of healthcare profession 5. Please specify within the user group the demographic variables important to the game, like gender, age range, or other relevant descriptive items? 6. Is the game used in research/ end-user/ training courses, please specify

Singapore National Temasek University for University of Polytechnic SingHealth Academia & Technology & Singapore (TP) Serious Games Association Design (SUTD) (NUS) Eco System 3 Instructional Developers Content Experts Designers

Evaluation Framework 4 DSSH Evaluation Framework Star Rating Quality Label Evaluation Criteria 1 Star 1. The entry is a serious game (A) Game Description 2. There is a safe data storage mechanism (B) Rationale 3. The working mechanism and theoretical background of the game (C) Functionality 2 Star 4. Face-validity checked and confirmed by (D) Validity experts 5. The underlying mechanism supported by (E) Data Protection evidence 3 Star 6. Involvement of medical, educational/ psychological and game development experts in the development process 7. Relevant user tests performed 4 Star 8. Independent experts have validated the game in a study 5 Star 9. Predictive validity has been confirmed in a scientific validation study & published in a peer-reviewed journal

1. What will the player learn? Describe the learning or behavioral objectives of the game. Functionality 2. How are these objectives translated into gameplay and functionality of the game? 3. Are there any in-game instructions, tutorials? If so, please specify. 4. What parameters are used to measure/follow progress or learning effects? 5. What kind of feedback is provided to players? 6. Every serious game has restrictions and limits. In your opinion what content on the subject is NOT covered by the game? 7. Please state potential undesirable effects. This does not include "gaming the game" (cheating), but effects of negative transfer of learning. 8. Is the undesirable effects dislosed to the user? 9. Please state measures taken to prevent undesirable effects

1. Were content experts involved in the design process from the start? Please specify how? Also add documentation showing this involvement 2. Were representatives of the user group involved in the design process from the start? Please specify how? Also add documentation showing this involvement 3. Did user testing take place? What were the results, and how were these incorporated in the design? Validity Please add documentation (reports, articles e.g.) 4. Does the game produce the same results on different platforms? 5. Is the content of the game validated with experts? Please add documentation/reports/articles 6. How does learning outcome compare to other methods assessing the same construct. Please add documentation/reports/articles 7. Is this game able to predict improvements of skills? Was this game validated in a Randomized Controlled Trial? Was the study published in a peer-reviewed journal? Please add documentation/ reports/ articles

Serious Game Game Data Research Learning Games Design Technologies Analytics and Methodology & Pedagogies, vs. and its Security Outcomes objectives and Gamification Applications outcomes Eco System 4 Content Instructional Developers Experts Designers Innovation Research Education

Evaluation Framework 5 DSSH Evaluation Framework Star Rating Quality Label Evaluation Criteria 1 Star 1. The entry is a serious game (A) Game Description 2. There is a safe data storage mechanism (B) Rationale 3. The working mechanism and theoretical background of the game (C) Functionality 2 Star 4. Face-validity checked and confirmed by (D) Validity experts 5. The underlying mechanism supported by (E) Data Protection evidence 3 Star 6. Involvement of medical, educational/ psychological and game development experts in the development process 7. Relevant user tests performed 4 Star 8. Independent experts have validated the game in a study 5 Star 9. Predictive validity has been confirmed in a scientific validation study & published in a peer-reviewed journal