Introducing the research method of simulations Comparing the - PowerPoint PPT Presentation

Introducing the research method of simulations Comparing the validity of cut-score methods in higher education tests Iris Yocarini

Studying tests in Higher Education

How accurate is the binding What is the most accurate way study advice decision? to correct for guessing on multiple choice tests? Researching tests in Higher Education How accurate are different cut-score methods that determine the pass/fail test score on a test?

How accurate is the What is the most accurate way to correct binding study advice decision? for guessing on multiple choice tests? Researching tests in Higher Education How accurate are different cut-score methods that determine the pass/fail test score on a test?

How to quantify accuracy ?

Comparing cut-score methods in higher education tests • Different methods • Absolute method • E.g. classical cut-score method • Relative method • Compromise method • E.g. Cohen method

Comparing cut-score methods in higher education tests • Previous studies • Comparing consequences of different cut-score methods Cut-scores From: Cohen-Schotanus, J., & Vleuten, C. P. M. (2010). A standard setting method with the best performing students as point of reference: Practical and affordable. Medical Teacher, 32, 154-160

Comparing cut-score methods in higher education tests • Previous studies • Comparing consequences of different cut-score methods Failure rates From: Cohen-Schotanus, J., & Vleuten, C. P. M. (2010). A standard setting method with the best performing students as point of reference: Practical and affordable. Medical Teacher, 32, 154-160

To assess the accuracy simulations are required

Simulation studies Simulations used for - Predictions - Performance - Training - Entertainment - Education - Proof - Discovery

Simulation studies General steps in simulating data 1. Define a data generating model 2. Simulate different conditions, varying parameters 3. Introduce error or bias 4. Evaluate input to output

Simulation studies Challenges: - Make assumptions about the true data generating model - Complex behavior difficult to model - With many factors you can’t model them all - You obtain many results/ datasets

Simulation studies Benefits • Allows us to study aspects that are difficulty or impossible to investigate otherwise • Simulated data: no involvement/ physical interaction with the system used to investigate • Explicit and transparent research • Many scenarios can be evaluated • Assess which factor has more or less influence

Question? Mail me : yocarini@fsw.eur.nl

Simulation studies Axelrod, R. (1997). Advancing the art of simulation in the social sciences. Obtaining, analyzing, and sharing results of computer models. Complexity , 3, 16-22. Cohen-Schotanus, J., & Vleuten, C. P. M. (2010). A standard setting method with the best performing students as point of reference: Practical and affordable. Medical Teacher, 32, 154-160.