A Rasch Analysis of Math Tests using M.In.E.R.Va. Platform Grazia - - PowerPoint PPT Presentation

▶

$a rasch analysis of math tests using m in e r va platform$

May 02, 2023 5 likes •510 views

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables A Rasch Analysis of Math Tests using M.In.E.R.Va. Platform Grazia Messineo - Salvatore Vassallo ICETC2014 Messineo - Vassallo Rasch Analysis in Minerva Platform 1/37

SLIDE 1

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables

A Rasch Analysis of Math Tests using M.In.E.R.Va. Platform

Grazia Messineo - Salvatore Vassallo ICETC2014

Messineo - Vassallo Rasch Analysis in Minerva Platform 1/37

SLIDE 2

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

The problem

Students must achieve some automatism in each discipline to succeed Many students have difficulties in Mathematics (see Ocse-Pisa reports) Traditional ways (courses) to help students are difficult to implement, also because of their costs

Messineo - Vassallo Rasch Analysis in Minerva Platform 2/37

SLIDE 3

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

Goals

To help students to recover their deficiencies in a discipline To provide teachers a tool to assign common or personalized tasks to students To provide teachers a tool to automatically verify students’ achievements

Messineo - Vassallo Rasch Analysis in Minerva Platform 3/37

SLIDE 4

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

The structure: contents

Theoretical schemes to strengthen or create knowledge Guided exercises to illustrate the correct way for solving the problems Consolidation and self-assessment exercises Performance assessment exercises

Messineo - Vassallo Rasch Analysis in Minerva Platform 4/37

SLIDE 5

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

The structure: feedback

Assignment of a score Automatic answers correction Hint to get a solution References to theoretical content Solution (complete or in partial form)

Messineo - Vassallo Rasch Analysis in Minerva Platform 5/37

SLIDE 6

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

How it works

The software consists of some L

A

T E X files: The package minerva, the master files, and the files of the exercises Each file of exercises contains one or more variants of a basic exercise The output is a pdf file with interactive content managed by Javascript

Messineo - Vassallo Rasch Analysis in Minerva Platform 6/37

SLIDE 7

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

Advantages

Possibility of being used offline Wide range of exercises:

Multiple choice tests Open questions (with powerful automatic computation) Multiple answers questions (with different marks for any question)

Messineo - Vassallo Rasch Analysis in Minerva Platform 7/37

SLIDE 8

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

Advantages

Use without installing any additional application Excellent display of mathematical text Independence from operating system Ability to use the same tool in different situations and disciplines

Messineo - Vassallo Rasch Analysis in Minerva Platform 8/37

SLIDE 9

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

The generation of a test

A variant is randomly chosen Variants are randomly scrambled Answers are shuffled Values are assigned to random parameters A test is produced (PDF file)

Messineo - Vassallo Rasch Analysis in Minerva Platform 9/37

SLIDE 10

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

The url: http://progettominerva.unicatt.it

Messineo - Vassallo Rasch Analysis in Minerva Platform 10/37

SLIDE 11

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

Students’ interface - 1

Messineo - Vassallo Rasch Analysis in Minerva Platform 11/37

SLIDE 12

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

Students’ interface - 2

Messineo - Vassallo Rasch Analysis in Minerva Platform 12/37

SLIDE 13

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

Teachers’ interface - 1

Messineo - Vassallo Rasch Analysis in Minerva Platform 13/37

SLIDE 14

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables General frame The Web Platform

Results

Messineo - Vassallo Rasch Analysis in Minerva Platform 14/37

SLIDE 15

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Secondary Schools

Trial has been going on since school year 2011-12 Results before and after the use of the tool

Messineo - Vassallo Rasch Analysis in Minerva Platform 15/37

SLIDE 16

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

University

Trial started in February 2014 Students can perform exercises and study theoretical material online A group gives a part of the final exam online: figure shows positive marks.

Messineo - Vassallo Rasch Analysis in Minerva Platform 16/37

SLIDE 17

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Rasch model

Mathematical model to compute the probability pij that the subject i with ability „i answers correctly to question j with difficulty ˛j Rasch measure based on the difference „i ` ˛j Data collected in a raw score matrix: n rows (subjects) and m columns (items) Conversion of values of measures of ability and difficulty into a common scale (logit)

Messineo - Vassallo Rasch Analysis in Minerva Platform 17/37

SLIDE 18

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

The analysis performed in 2014

Data collected from a test of Mathematics Two sets of students: 20 of Catholic University and 16 of a Secondary School Questionnaire with 16 multiple choice questions, 8 on derivatives and 8 on domains

Messineo - Vassallo Rasch Analysis in Minerva Platform 18/37

SLIDE 19

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Summary Statistics

Messineo - Vassallo Rasch Analysis in Minerva Platform 19/37

SLIDE 20

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Summary Statistics interpretation

For students: separation index lower than 2 and reliability index lower than 0:8

Messineo - Vassallo Rasch Analysis in Minerva Platform 20/37

SLIDE 21

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Summary Statistics interpretation

For students: separation index lower than 2 and reliability index lower than 0:8! more items may be needed for a better separation of ability levels (actually we can divide the sample into 3 levels)

Messineo - Vassallo Rasch Analysis in Minerva Platform 20/37

SLIDE 22

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Summary Statistics interpretation

For students: separation index lower than 2 and reliability index lower than 0:8! more items may be needed for a better separation of ability levels (actually we can divide the sample into 3 levels) For items: separation index lower 3 and reliability index lower than 0:9

Messineo - Vassallo Rasch Analysis in Minerva Platform 20/37

SLIDE 23

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Summary Statistics interpretation

For students: separation index lower than 2 and reliability index lower than 0:8! more items may be needed for a better separation of ability levels (actually we can divide the sample into 3 levels) For items: separation index lower 3 and reliability index lower than 0:9 ! the students’ sample is not large enough to confirm the item difficulty scale.

Messineo - Vassallo Rasch Analysis in Minerva Platform 20/37

SLIDE 24

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Person-Item Distribution Map

Messineo - Vassallo Rasch Analysis in Minerva Platform 21/37

SLIDE 25

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Person-Item Distribution Map

Messineo - Vassallo Rasch Analysis in Minerva Platform 21/37

SLIDE 26

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Person-Item Distribution Map

Messineo - Vassallo Rasch Analysis in Minerva Platform 21/37

SLIDE 27

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Person-Item Distribution Map

Messineo - Vassallo Rasch Analysis in Minerva Platform 21/37

SLIDE 28

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Person-Item Distribution Map interpretation

students’ ability is on the left, from highest (top) to lowest (bottom); items are on the right, ordered by difficulty, from most difficult to easiest; both are measured on the same scale; average of difficulty of items is set equal to 0; students’ mean value is 0.12 ! students’ performance is a little above the expected performance.

Messineo - Vassallo Rasch Analysis in Minerva Platform 22/37

SLIDE 29

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Guttman Scalogram

Messineo - Vassallo Rasch Analysis in Minerva Platform 23/37

SLIDE 30

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Guttman Scalogram

Messineo - Vassallo Rasch Analysis in Minerva Platform 23/37

SLIDE 31

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Guttman Scalogram

Messineo - Vassallo Rasch Analysis in Minerva Platform 23/37

SLIDE 32

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Guttman Scalogram

Messineo - Vassallo Rasch Analysis in Minerva Platform 23/37

SLIDE 33

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Guttman Scalogram interpretation

A ! unexpected correct answer @ ! unexpected wrong answer helps identify “lucky guesses” and other patterns

f response

Messineo - Vassallo Rasch Analysis in Minerva Platform 24/37

SLIDE 34

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Item Category/Option/Distractor Frequencies

Messineo - Vassallo Rasch Analysis in Minerva Platform 25/37

SLIDE 35

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Scatterplot

Messineo - Vassallo Rasch Analysis in Minerva Platform 26/37

SLIDE 36

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

Scatterplot interpretation

Comparison of performance in the two groups

utliers are the items outside the two bands that

represent the limits of the confidence interval (95%) DOM20 is significatively more difficult for University students DER15 is significatively more difficult for Secondary School students

Messineo - Vassallo Rasch Analysis in Minerva Platform 27/37

SLIDE 37

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis

THANK YOU

Messineo - Vassallo Rasch Analysis in Minerva Platform 28/37

SLIDE 38

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis Messineo - Vassallo Rasch Analysis in Minerva Platform 29/37

SLIDE 39

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis Messineo - Vassallo Rasch Analysis in Minerva Platform 30/37

SLIDE 40

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis Messineo - Vassallo Rasch Analysis in Minerva Platform 30/37

SLIDE 41

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis Messineo - Vassallo Rasch Analysis in Minerva Platform 30/37

SLIDE 42

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis Messineo - Vassallo Rasch Analysis in Minerva Platform 30/37

SLIDE 43

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis Messineo - Vassallo Rasch Analysis in Minerva Platform 31/37

SLIDE 44

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis Messineo - Vassallo Rasch Analysis in Minerva Platform 32/37

SLIDE 45

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis Messineo - Vassallo Rasch Analysis in Minerva Platform 32/37

SLIDE 46

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables Initial data Individual Analysis Differential Analysis Messineo - Vassallo Rasch Analysis in Minerva Platform 33/37

SLIDE 47

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables

DIF Plot for students

Messineo - Vassallo Rasch Analysis in Minerva Platform 34/37

SLIDE 48

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables

DIF Plot - interpretation

Plot shows the average difference for each group between the observed and expected response DER15, DER03, DOM20 show significative differences in the two groups

Messineo - Vassallo Rasch Analysis in Minerva Platform 35/37

SLIDE 49

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables

Scatterplot Female-Male

Messineo - Vassallo Rasch Analysis in Minerva Platform 36/37

SLIDE 50

The M.In.E.R.Va Project Statistical Analysis and Rasch Model Other Tables

DIF Plot for items

Messineo - Vassallo Rasch Analysis in Minerva Platform 37/37