Through Feature Analysis Eva L. Baker, Ayesha Madni, Jenny Kao, - - PowerPoint PPT Presentation

Career Readiness Inferences from High School Assessments Through Feature Analysis

Eva L. Baker, Ayesha Madni, Jenny Kao, Noelle Griffin, Kilchan Choi, & Li Cai NCSA Jun 28, 2018

2

Overview

• Can existing examinations be re-purposed as a measure of

workforce readiness?

• Strategy - using Feature Analysis as a technique to identify

relevant item elements and confirm them by analyzing student test data

• Summary of process and descriptive results for items

drawn from Smarter Balanced Assessment Consortium (SB), ELA & Math tests

• International (South Korea) benchmark analysis

3

In the EAG Project

• Goal was to identify features related to career readiness in

the 8th and 11th grade SB English Language Arts (ELA) and Math tests

• Career readiness features were drawn from the

Department of Labor O*NET list as well as from prior CRESST work

• O*NET includes skills, abilities, and work activities for

various occupations

• https://www.onetonline.org

4

Feature Analysis

• Feature Analysis (FA): the systematic rating of test items

using attributes compared with test results

• Intended to identify attributes that contribute to

performance or predict learning

• Prior research applied FA to state assessments, Smarter

Balanced Assessment Consortium tests, Navy examinations, and PBS Kids games.

• Earlier efforts used categories of cognition, content,

linguistics, and task features.

5

6

7

8

Feature Rating

• Qualitative rating of items against a set of attributes

followed by a subsequent quantitative analysis to determine item performance.

• A subset of SB items were selected for ELA and math from

each of the three cut-scores considering claims, targets, and item types across grades 8 and 11 and years 2015 and 2016 (320 ELA items and 280 math items)

• Career ready features informed by ontologies, extensive

resource analysis for EMT and web developer domains, and expert review and analysis

• Career readiness features categorized into skills, abilities,

and work activities as defined by O*NET

9

Some Research Questions

• 1. What particular attributes/features do the items contain?
• 2. Which features appear more frequently across items?
• 3. Are there differences in feature representation across

domain areas?

• 4. What particular attributes/features of items explain

variations in item characteristics such as increased or reduced difficulty across items and why?

• 5. What features are most important to elicit and assess key

readiness outcomes?

10

• Almost all of the ELA items have
• nly one “skill” feature present:

reading comprehension or active listening

• No item was rated as requiring

complex problem solving, critical thinking, or judgment and decision making

• With the exception of a few

features, overall low feature frequency

• About half of the items were

rated as requiring 3 or 4 ability features

ELA Feature Analysis

ELA Results:

• 126 ELA items for 8th grade

rated

• 138 ELA items for 11th grade

rated

• 152 of these items

administered in 2015

• 158 items administered in

2016

11

Career-Readiness Features for ELA

Skills (7 features) Abilities (12 features) Work Activities & Context (11 features)

Active learning (0%) Deductive reasoning (79.2%) Analyzing data & information (7.6%) Active listening (21.6%) Flexibility of closure (12.1%) Documenting/recording info (8.7%) Complex problem solving (0%) Fluency of ideas (8.3%) Getting information (5.3%) Critical thinking (0.4%) Inductive reasoning (43.2%) Identifying objects, actions, & events (9.8%) Judgment & decision making (0%) Information ordering (8.7%) Importance of being exact or accurate (100%) Monitoring (0%) Memorization (0%) Interacting with computers (0.4%) Reading comprehension (72.0%) Oral comprehension (21.6%) Judging the qualities of things, services, or people (1.5%) Problem sensitivity (0%) Making decisions and solving problems (2.3%) Time sharing (84.5%) Organizing, planning, and prioritizing work (0%) Written comprehension (100%) Processing information (0%) Written expression (8.3%) Thinking creatively (1.1%) Visualization (0%)

12

ELA Feature Correlations

• As expected, active learning negatively correlated

with reading comprehension

• Fluency of ideas strong positive correlation with

documenting/recording information (0.98)

• Information ordering, analyzing data and

information, and fluency of ideas highly correlated

• Smaller positive correlation between inductive and

deductive reasoning

13

• Reading comprehension

predominant skill

• Limited representation of

additional skills - 12% of items contained critical thinking and 25% required complex problem solving

• Deductive reasoning – “ability”

feature required by 97% of the items

• Inductive reasoning rated for

35% of the items

• Math reasoning was required in

58% of the items and number facility was required in 73% of the items

Math Feature Analysis

Math Results:

• 65 Math items for 8th grade

rated

• 121 Math items for 11th grade

rated

• 145 of these items

administered in 2015

• 41 items administered in 2016

14

Career-Readiness Features for Math

Skills (4 features) Abilities (10 features) Work Activities & Context (8 features)

Active learning (4.8%) Deductive reasoning (97.3%) Analyzing data and information (77.4%) Complex problem solving (25.3%) Flexibility of closure (3.2%) Estimating the quantifiable characteristics of products, events, or information (21.0%) Critical thinking (11.8%) Inductive reasoning (35.5%) Getting information (25.8%) Reading comprehension (76.3%) Information ordering (2.7%) Identifying objects, actions, and events (48.4%) Mathematical reasoning (58.1%) Importance of being exact or accurate (71.5%) Memorization (0%) Making decisions and solving problems (22.0%) Number facility (72.6%) Organizing, planning, and prioritizing work (34.4%) Time sharing (5.9%) Processing information (67.7%) Written expression (1.1%) Visualization (12.4%)

15

Math Feature Correlations

• Medium positive correlation between complex

problem solving and inductive reasoning

• Small to medium positive correlation between

critical thinking and organizing, planning, and prioritizing work

• Medium positive correlation between reading

comprehension and analyzing data and information

16

Take-Away from Descriptive Information

• Smarter Balanced:
• “Assessment is relevant to college and career.”
• “Measure critical thinking and problem solving skills students need

for success.”

• Most predominant – reading comprehension
• Limited presence - critical thinking, problem solving
• Deductive and inductive reasoning are present across both

math and ELA

• Lower frequency for features that may apply across

different careers

17

More Statistical Modeling

To examine the relationship between the career readiness feature ratings with item level psychometric characteristics – in the present case item difficulty parameter.

18

Complexity with adaptive test Weighted Least Squares (WLS) estimator:

• Var (𝑍

𝑗) = 𝜏𝑗 2, where 𝜏1 2, 𝜏2 2, … , 𝜏𝑜 2 are known error

variance from calibration

• 𝜏𝑗

2 = variance of item difficulty parameter for item i

• Weight (w) = 1/𝜏𝑗

2, thus, items with larger error

variance put less weights in regression fitting

• Variance of item difficulty = square of standard errors
• f item difficulty / N of items in the item pool

WLS with Known Weights

19

8th Grade ELA (𝑆2 = 0.56)

Feature Estimate SE t p Intercept 1.028 0.259 3.97 0.000 Fluency of ideas* 1.233 0.335 3.68 0.000 Deductive reasoning 0.215 0.220 0.98 0.330 Flexibility of closure 0.116 0.162 0.71 0.477 Analyzing data or information 0.010 0.691 0.01 0.989 Identifying objects, actions, and events

• 0.002

0.692 0.00 0.998 Getting information

• 0.030

0.498

• 0.06

0.952 Timesharing

• 0.051

0.145

• 0.35

0.727 Inductive reasoning

• 0.070

0.187

• 0.38

0.708 Reading comprehension*

• 0.568

0.198

• 2.87

0.005 Active listening*

• 1.180

0.249

• 4.74

<.0001

20

Feature Estimate SE t p Intercept 0.558 0.296 1.88 0.062 Fluency of ideas 1.266 0.680 1.86 0.065 Identifying objects, actions, and events* 1.219 0.435 2.8 0.006 Making decisions and solving problems 0.848 0.777 1.09 0.277 Deductive reasoning* 0.659 0.141 4.68 <.0001 Timesharing 0.079 0.170 0.47 0.641 Written expression 0.075 0.434 0.17 0.863 Getting information 0.056 0.364 0.15 0.878 Inductive reasoning

• 0.048 0.139
• 0.34

0.732 Active listening

• 0.321 0.344
• 0.93

0.353 Reading comprehension

• 0.331 0.328
• 1.01

0.314 Flexibility of closure*

• 0.961 0.125
• 7.67

<.0001 Information ordering

• 1.265 0.911
• 1.39

0.167

11th Grade ELA (𝑆2 = 0.54)

21 21

Feature Estimate SE t p Intercept 0.806 0.790 1.02 0.313 Reading comprehension* 1.829 0.715 2.56 0.014 Getting information* 1.550 0.706 2.19 0.033 Processing information* 0.883 0.410 2.15 0.037 Making decisions & solving problems* 0.791 0.197 4.02 0.000 Visualization 0.379 0.578 0.66 0.516 Estimating 0.327 0.296 1.11 0.275 Number Facility 0.218 0.504 0.43 0.667 (more n.s. features) Complex problem solving

• 0.302

0.286

• 1.06

0.297 Critical thinking

• 0.766

0.698

• 1.1

0.278 Timesharing

• 0.816

1.157

• 0.71

0.484 Information ordering*

• 1.775

0.731

• 2.43

0.019 Analyzing data or information*

• 1.906

0.646

• 2.95

0.005

8th Grade Math (𝑆2 = 0.53)

11th Grade Math (𝑆2 = 0.45)

Feature Estimate SE t p Intercept 1.424 0.317 4.49 <.0001 Written expression 1.322 1.017 1.3 0.197 Timesharing* 0.718 0.346 2.07 0.041 Math reasoning* 0.624 0.192 3.25 0.002 Information ordering 0.452 0.961 0.47 0.639 Organizing, planning, prioritizing work 0.364 0.210 1.74 0.086 Reading comprehension 0.323 0.250 1.29 0.200 Complex problem solving 0.300 0.218 1.38 0.172 Number Facility 0.285 0.213 1.34 0.183 (more n.s. features) Analyzing data or information

• 0.328

0.264

• 1.24

0.217 Estimating

• 0.330

0.295

• 1.12

0.266 Getting information

• 0.384

0.241

• 1.6

0.113 Processing information

• 0.390

0.238

• 1.64

0.105 Making decisions & solving problems*

• 0.681

0.264

• 2.58

0.011

23

Korean Math Items

• 20 high school math items from a provincial assessment at

12th grade

• Translated and back-translated
• Same features as in SB items
• Same quantitative approach

24

Career-Readiness Features for Korean Items

Skills (4 features) Abilities (10 features) Work Activities & Context (8 features)

Active learning (10%) Deductive reasoning (100%) Analyzing data and information (100%) Complex problem solving (0%) Flexibility of closure (0%) Estimating the quantifiable characteristics of products, events, or information (5%) Critical thinking (50%) Inductive reasoning (0%) Getting information (30%) Reading comprehension (60%) Information ordering (0%) Identifying objects, actions, and events (10%) Mathematical reasoning (55%) Importance of being exact or accurate (90%) Memorization (0%) Making decisions and solving problems (0%) Number facility (90%) Organizing, planning, and prioritizing work (60%) Time sharing (5%) Processing information (100%) Written expression (0%) Visualization (30%)

Korean Math (𝑆2 = 0.56)

Feature Estimate SE t p Intercept

• 0.034

0.195

• 0.17

0.866 Critical thinking 0.525 0.220 2.38 0.031 Reading comprehension 0.205 0.195 1.05 0.310 Visualization 0.497 0.233 2.14 0.050 Math reasoning

• 0.294

0.198

• 1.49

0.158

@UCLA_CRESST WWW.FACEBOOK.COM/CRESSTUCLA TINYURL.COM/JX8YASB UCLACRESST WWW.CRESST.ORG

THANK YOU

Supported by US ED Grant # S368A150011

CONTACT US