Overview eat - History 1 eat: An R Package for Automation of Data - - PowerPoint PPT Presentation

eat: An R Package for Automation of Data Preparation and IRT Modeling

Karoline Sachse, Martin Hecht, Sebastian Weirich, Nicole Haag, Malte Jansen, Sebastian Wurster, Christiane Penk, Anna Lenski, Thilo Siegle

Institute for Educational Quality Improvement Humboldt-University, Berlin

February 10, 2012

Psychoco, Innsbruck February 10, 2012 1 / 26

Overview

1

eat - History The Institute for Educational Quality Improvement ACER ConQuest The Idea

2

eat - Concept Overview Data

3

eat - Examples Data Preparation Unidimensional 1PL model with automateModels Grouping options in automateModels automateModels

4

Discussion Outlook

Psychoco, Innsbruck February 10, 2012 2 / 26 eat - History The Institute for Educational Quality Improvement

The Institute

Independent research and test institute founded by the 16 federal states in 2004 Nationwide Educational Standards Assessments in German, the first foreign language, Mathematics and Science which allow comparison

• f federal states (N ≈ 30, 000)

Assessment tests in German, Mathematics and the first foreign language in the 8th grade at secondary school (once a year) Assessment tests in German and Mathematics in the 3rd grade at primary school (once a year)

Psychoco, Innsbruck February 10, 2012 3 / 26 eat - History ACER ConQuest

ConQuest

Commercial Software developed by ACER (Wu, Adams & Wilson, 1997) Major scaling tool of the Organisation for Economic Co-operation and Development’s Programme for International Student Assessment (PISA) Fits a large number of different item response models

Rasch, partial credit, rating scale, facets, ... Latent regression Multidimensionality

Estimation

Marginal Maximum Likelihood Gaussian quadrature/ Monte Carlo approximations Person parameter estimation: EAP, MLE, WLE, Plausible values

Psychoco, Innsbruck February 10, 2012 4 / 26

eat - History The Idea

Automation of Data Preparation and Analysis

automate data preparation

1

read in & check SPSS-files

2

merge data frames (booklets)

3

recode & dichotomize data

automate IRT calibration

1

write ConQuest syntax, generate appropriate data input

2

execute ConQuest

3

read in ConQuest output

facilitate reporting

1

write out results (graphics, tables, ...)

⇒ ”eat” (”Educational Assessment Tools”)

Psychoco, Innsbruck February 10, 2012 5 / 26 eat - Concept Overview

Implemented Modules

Psychoco, Innsbruck February 10, 2012 6 / 26 eat - Concept Overview

Wrapping ConQuest

Psychoco, Innsbruck February 10, 2012 7 / 26 eat - Concept Data

Typical Items

Psychoco, Innsbruck February 10, 2012 8 / 26

eat - Concept Data

Typical Items - Scores

Psychoco, Innsbruck February 10, 2012 9 / 26 eat - Concept Data

Data Structure

Psychoco, Innsbruck February 10, 2012 10 / 26 eat - Examples Data Preparation

automateDataPreparation

dataset <- automateDataPreparation ( inputList = inputList, path = path, loadSav = TRUE, checkData = TRUE, mergeData = TRUE, recodeData = TRUE, aggregateData = TRUE, scoreData = TRUE, writeSpss = TRUE )

Psychoco, Innsbruck February 10, 2012 11 / 26 eat - Examples Data Preparation

Data Preparation Input

Psychoco, Innsbruck February 10, 2012 12 / 26

eat - Examples Data Preparation

Data Preparation Logfile

Psychoco, Innsbruck February 10, 2012 13 / 26 eat - Examples Unidimensional 1PL model with automateModels

Simple Unidimensional 1PL Model

results01 <- automateModels( dataset = dataset , folder = folder )

Psychoco, Innsbruck February 10, 2012 14 / 26 eat - Examples Unidimensional 1PL model with automateModels

ConQuest Dataset & Label File Creation

Psychoco, Innsbruck February 10, 2012 15 / 26 eat - Examples Unidimensional 1PL model with automateModels

ConQuest Syntax Creation

Psychoco, Innsbruck February 10, 2012 16 / 26

eat - Examples Unidimensional 1PL model with automateModels

ConQuest Run

ConQuest runs due to automatic creation and execution of batch files

Psychoco, Innsbruck February 10, 2012 17 / 26 eat - Examples Unidimensional 1PL model with automateModels

ConQuest Output Files

Item parameter estimates

.shw, .itn, ...

Person parameter estimates

.wle, .mle, .eap, .pvl, ...

⇒ Many different output files

Psychoco, Innsbruck February 10, 2012 18 / 26 eat - Examples Unidimensional 1PL model with automateModels

ConQuest Item Parameter Output

Psychoco, Innsbruck February 10, 2012 19 / 26 eat - Examples Unidimensional 1PL model with automateModels

ConQuest Person Parameter Output

Psychoco, Innsbruck February 10, 2012 20 / 26

eat - Examples Unidimensional 1PL model with automateModels

eat Reporting

Item parameter estimates Person parameter estimates

Psychoco, Innsbruck February 10, 2012 21 / 26 eat - Examples Unidimensional 1PL model with automateModels

eat Log

all objects (dataset, item.grouping, ...) will be archived into an .RData file an INFO file will be created

Psychoco, Innsbruck February 10, 2012 22 / 26 eat - Examples Grouping options in automateModels

Multidimensional vs. Unidimensional Analysis

results02 <- automateModels( dataset = dataset , id = "id" , folder = folder , item.grouping = item.grouping , select.item.group = c ( "ER" , "EL" ) ) results03 <- automateModels( dataset = dataset , id = "id" , folder = folder , item.grouping = item.grouping , select.item.group = c ( "ER" , "EL" ) , cross="all")

Psychoco, Innsbruck February 10, 2012 23 / 26 eat - Examples Grouping options in automateModels

Person groups & weights

dataset <- cbind ( dataset , "weight1" = as.character(sample(c(0.8, 1, 1.2), nrow(dataset), replace=TRUE)), "weight2" = as.character(sample(c(1), nrow(dataset), replace=TRUE)), stringsAsFactors = FALSE ) results04 <- automateModels( dataset = dataset, folder = folder context.vars = c ( "weight1" , "weight2" ) , item.grouping = item.grouping , select.item.group = "ER" , person.grouping = person.grouping , select.person.group = list ( "gr.9" , "gr.10" ) , weight = list ( "weight1" , "weight2" ) )

Psychoco, Innsbruck February 10, 2012 24 / 26

eat - Examples automateModels

automateModels – Overview

automateModels(dataset, id = NULL, context.vars = NULL, items = NULL, item.grouping = NULL, select.item.group = NULL, person.grouping.vars = NULL, person.grouping.vars.include.all = FALSE, person.grouping = NULL, select.person.group = NULL, additional.item.props = NULL, folder, overwrite.folder = TRUE, analyse.name.prefix = NULL, analyse.name = NULL, analyse.name.elements = NULL, data.name = NULL, m.model = NULL, software = NULL, dif = NULL, weight = NULL, anchor = NULL, regression = NULL, adjust.for.regression = FALSE, q3 = FALSE, missing.rule = NULL, cross = NULL, subfolder.order = NULL, subfolder.mode = NULL, additionalSubFolder = NULL, run.mode = NULL, n.batches = NULL, run.timeout = 1440, run.status.refresh = 0.2, email = NULL, smtpServer = NULL, write.txt.dataset = FALSE, delete.folder.countdown = 5, conquestParameters = NULL )

Psychoco, Innsbruck February 10, 2012 25 / 26 Discussion Outlook

Thank you

Thank you for your attention! http://r-forge.r-project.org/eat eat-commits@lists.r-forge.r-project.org Special thanks to

Alexander Robitzsch (Measurement Statistician, bifie) Martin Mechtel (IT Director, IQB)

Psychoco, Innsbruck February 10, 2012 26 / 26