USING COMPUTERIZED ASSESSMENT TO IDENTIFY PROFILES OF READING & - - PowerPoint PPT Presentation

SLIDE 1

Barbara Foorman and Yaacov Petscher

• Florida Center for Reading Research at

Florida State University Liz Brooke and Alison Mitchell

• Lexia Learning

USING COMPUTERIZED ASSESSMENT TO IDENTIFY PROFILES OF READING & LANGUAGE SKILLS IN ELEMENTARY AND SECONDARY STUDENTS

SLIDE 2

¡ The Structure of Reading ¡ Identifying Profiles ¡ Connections to Instruction

AGENDA

SLIDE 3

How does reading relate to language? How do we reconcile the structure of reading with profiles of strengths and weaknesses?

THE STRUCTURE OF READING

SLIDE 4

FL FLORID ORIDA CENT A CENTER F ER FOR READING OR READING RES RESEAR EARCH (F (FCR CRR) ) READING G ASSESSMENT ASSESSMENT ( (FRA) FRA)

As part of federal grants, FCRR developed the FRA

• 2010-2014: Developed computer-adaptive K-12

component skills battery (item tryouts, IRT analyses, & linking studies. Called the FCRR Reading Assessment (FRA)

• 2010-2015: Conducted cross-sectional and longitudinal

study of reading & language development.

SLIDE 5

STR STRUCTU CTURE O OF R READ ADING K NG K -

• 3

3

Foorman et al. (2015) Reading & Writing Decoding fluency Syntax Phonological awareness Vocabulary Listening Comp Oral language Reading Comprehension

SLIDE 6

STR STRUCTU CTURE O OF R READ ADING 4 - NG 4 - 1 10

Foorman, et al. (2015) Journal of Educational Psychology Decoding fluency Syntax Vocabulary Oral language Reading Comprehension 72% - 99% variance

SLIDE 7

BA BACK CKGR GROUND O ON PR PROFILES S

• Targeting instruction to profiles of students’ strengths &

weaknesses is central to teaching

• Profiles often based on descriptive data of reading errors, text

reading levels, or learning profiles.

• Regression-based techniques used to quantify profiles of good &

poor readers and profiles within poor readers.

• Regression-based approaches use arbitrary achievement cut points

(e.g., below 40th percentile on standardized test).

• A latent class approach (LCA) utilizes multiple measures to reduce

measurement error and improve reliability and stability of classification.

• When the latent variable is continuous, the approach is often called

latent profile analysis (LPA). LPA used here.

SLIDE 8

¡ What are the latent profiles of reading and language skills in a large, representative sample of Florida students in grades K-10? ¡ What are the relations among the latent profiles and a norm-referenced reading test in K-2 and a latent variable of reading comprehension in grades 3-10?

RES RESEAR EARCH QUESTIONS QUESTIONS

SLIDE 9

MET METHOD HOD

• Participants: 7,752 students in K-10; 2295 in K-2 and 5,457 in 3-10.

Representative of Florida demographics.

• Procedure:
• K-2 FRA individually administered in two 45-min sessions in

mid-year; 3-10 FRA administered in computer lab in two 45- min sessions in mid-year.

• SESAT Word Reading administered in small groups in K;

teachers administered SAT-10 and FCAT as usual.

• Design and analysis: FRA raw scores converted to Z scores. Latent

profile and general linear modeling were conducted at each grade (with linear step-up correction to correct against false discovery rate).

SLIDE 10

Con Constr struct ct Task/Ab /Abbreviation Gr Grade e

Pho honological A Awareness Pho honological a awareness ( (PA) A) K K Al Alpha habetics L LS Letter S Sounds ( (AP AP1/2 /2; L LS) K K D Decoding Word R Reading ( (WR) G1 + + G G2 E Encoding Spelling ( (Spell) G2 G2 Oral L Language V Vocabulary Vocabulary P Pairs ( (VOC) K- K-2 S Syntax Sentence C Comprehe hension ( (SC) K K L Listening C Comprehe hension Following D Directions ( (FD) K- K-2

CONSTRUCTS/FRA SCREENING TASKS IN GRADES K-2

SLIDE 11

Con Constr struct ct Tasks/Ab /Abbreviation

Word recognition Word Recognition (WRT) Academic Language Vocabulary (morphological awareness) Vocabulary Knowledge (VKT) Discourse (verb tense, anaphora, connectives) Syntactic Knowledge (SKT) Reading Comprehension Reading Comprehension (RCT)

CONSTRUCTS/FRA SCREENING TASKS IN GRADES 3-10

SLIDE 12

Gr Grade(s) e(s) Test/S /Subtest

Kindergarten Stanford Early Scholastic Achievement Test (SESAT) Word Reading 1-10 Stanford Achievement Test (10th ed; SAT-10) Reading comprehension 3-10 Florida Comprehensive Assessment Test (FCAT 2.0) Reading

STANDARDIZED READING OUTCOMES IN GRADES K-10

• Note. A latent factor score for Reading Comprehension was created from the

developmental scale scores from the FRA’s RCT, SAT-10, and FCAT 2.0

SLIDE 13

¡ Print-related measures were moderately correlated in K (PA with LS, .48, and with SESAT WR, .58; and SESAT WR with LS, .51). OL measures were moderately correlated with each other. ¡ In G1-G2, print-related measures were more strongly correlated: SAT-10 with WR (.75 in first and .58 in G2); WR & Spell in G2 (.77). Oral language measures were moderately correlated with each other in all three grades and VOC was moderately correlated with SAT-10 in G1 (.58) & G2 (.62). ¡ The three RC measures were strongly correlated, with the RCT bivariate correlations ranging from .67 in G8 to .81 in G5. FCAT and SAT-10 correlations ranged from a low of .71 in G8 to a high

• f .81 in G3 & G5. VKT and SKT were moderately correlated in

these grades (.31 to .46) as were the bivariate correlations of WRT (.29 to .51).

CORRELATIONS

SLIDE 14

What are the challenges associated with identifying profiles of student strengths and weaknesses? What are ways to do so that are meaningful, but remain reliable and valid and can be assessed efficiently?

IDENTIFYING PROFILES

SLIDE 15

LATENT PROFILE ANALYSIS – WHAT?

Hybrid Mode Models ls Con Conti tinuou

• us

s Me Measure asures s Ca Categor egorica cal Me Measure asures s Continuous Latent Factor Analysis Item Response Theory Categorical Latent Latent Profile Analysis Latent Class Analysis

SLIDE 16

LATENT PROFILE ANALYSIS – WHAT?

Vocabulary PPVT EVT SYN Class PPVT EVT SYN

SLIDE 17

LATENT PROFILE ANALYSIS – WHY?

SLIDE 18

WHY ADAPTIVE MEASURES?

3-6 hours!!!

SLIDE 19

• Note. LL=log likelihood, AIC=Akaike Information Criteria, aBIC=sample adjusted

Bayes Information Criteria, -2LL=log likelihood ratio test. Values in bold represent Final selected class. *p<.001.

19

Latent profile model fit for kindergarten through grade 5 and grade 8 Grade Profiles Parameters LL AIC aBIC

• 2LL

K 2 19

• 3255.01 6548.01 6624.87

3 22

• 2681.62 5407.25 5496.24 1146.77*

4 28

• 2653.41 5362.82 5476.08

56.42* 5 34

• 2629.75 5357.51 5465.04

47.32* 6 40

• 2618.44 5316.88 5458.68

22.63* 1 2 10

• 2818.26 5656.52 5705.49

3 14

• 2785.14 5598.28 5666.84

66.24* 4 18

• 2768.46 5572.92 5661.01

33.36* 5 22

• 2752.99 5549.98 5657.71

30.94* 6 26

• 2743.43 5546.86 5674.17

19.12* 2 2 13

• 3768.29 7562.59 7624.79

3 18

• 3697.13 7430.26 7516.38

142.33* 4 23

• 3669.02 7384.03 7494.08

56.22* 5 28

• 3655.54 7367.07 7501.04

26.96* 6 33

• 3642.95 7355.89 7513.78

25.17* 3 2 10

• 2202.90 4425.81 4438.14

3 14

• 2173.78 4375.56 4392.83

58.24* 4 18

• 2154.42 4344.83 4367.04

38.73* 5 22

• 2129.87 4303.74 4330.88

49.10* 6 26

• 2104.72 4261.43 4293.51

50.30* 4 2 10

• 2166.75 4353.49 4365.49

3 14

• 2140.35 4308.69 4325.50

52.80* 4 18

• 2112.79 4261.58 4283.18

55.12* 5 22

• 2097.77 4239.54 4265.95

30.04* 6 26

• 2087.22 4226.44 4257.65

21.10* 5 2 10

• 2451.39 4922.79 4935.95

3 14

• 2405.92 4839.83 4858.25

90.96* 4 18

• 2383.61 4803.22 4826.91

44.61* 5 22

• 2363.13 4770.25 4799.20

40.97*

SLIDE 20

• Note. VOC=Vocabulary Pairs; FD=Following Directions; PA=Phonological Awareness;

LS=Letter Sounds; SC=Sentence Comprehension

20

Latent Profile Analysis of FRA Measures in Kindergarten (N=422)

• 3
• 2
• 1

1 2 VOC FD PA LS SC Kindergarten Z Z-Score c1 c2 c3 c4 c5 c6 c6; 19% c3; 42% c4; 23% c5; 7% c1; 7% c2; 2%

SLIDE 21

350 400 450 500 550

SESAT

1 2 3 4 5 6

group

<.0001 Prob > F 37.11 F

Distribution of SESAT

350 400 450 500 550

SESAT

1 2 3 4 5 6

group

<.0001 Prob > F 37.11 F

Distribution of SESAT

• Note. The average absolute value of the standardized difference in SESAT WR

performance across all classes was Hedge’s g = 1.10, indicating the magnitude of differences in FRA skill profile performance on standardized outcome

K S K SES ESAT W WR B R BY LA LATENT ENT CLAS CLASSES ES

SLIDE 22

22

Latent Profile Analysis of FRA Measures in Grade 1 (N=989)

• Note. VOC=Vocabulary Pairs; FD=Following Directions;

WR=Word Reading

• 4
• 3
• 2
• 1

1 2 VOC FD WR Grade 1 1 Z Z-Score c1 c2 c3 c4 c5 c2; 35% c5; 43% c1; 1% c2; 35% c3; 3%

SLIDE 23

• Note. The average absolute value of the standardized difference in SAT-10 RC

performance across all classes was Hedge’s g = 1.43, indicating the magnitude of differences in FRA skill profile performance on standardized outcome

G1 S G1 SAT-1

• 10 R

0 RC B C BY L Y LATENT CL TENT CLASSES ASSES

SLIDE 24

24

Latent Profile Analysis of FRA Measures in Grade 2 (N=884)

• Note. VOC=Vocabulary Pairs; FD=Following Directions;

Spell=Spelling; WR=Word Reading

• 2
• 1

1 2 VOC FD Spell WR Grade 2 2 Z Z-Scores c1 c2 c3 c4 c5 c6 c5; 32% c2; 10% c4; 32% c3; 15% c6; 5% c1; 52%

SLIDE 25

• Note. The average absolute value of the standardized difference in SAT-10 RC

performance across all classes was Hedge’s g = 1.48, indicating the magnitude of differences in FRA skill profile performance on standardized outcome

G2 S G2 SAT-1

• 10 R

0 RC B C BY L Y LATE TENT CL NT CLASSE ASSES

SLIDE 26

Latent Profile Analysis of FRA Measures in Grade 5 (N=659)

• Note. VKT=Vocabulary Knowledge Task; WRT=Word Recognition

Task; SKT=Syntactic Knowledge Task

• 4
• 3
• 2
• 1

1 2 VKT WRT SKT Grade 5 5 Z Z-Score c1 c2 c3 c4 c5 c5; 7 % c4; 34 % c3; 53 % c2; 4 % c1; 1 %

SLIDE 27

• 3
• 2
• 1

1 2 3

RC factor itself

1 2 3 4 5

c

<.0001 Prob > F 132.73 F

Distribution of frc

• 3
• 2
• 1

1 2 3

RC factor itself

1 2 3 4 5

c

<.0001 Prob > F 132.73 F

Distribution of frc

• Note. The average absolute value of the standardized difference in latent RC

performance across all classes was Hedge’s g =2.53, indicating the magnitude of differences in FRA skill profile performance on standardized outcome

G5 R G5 RC F C FACT CTOR B BY L Y LATE TENT CL NT CLASSE ASSES

SLIDE 28

Latent Profile Analysis of FRA Measures in Grade 8 (N=629)

• Note. VKT=Vocabulary Knowledge Task; WRT=Word Recognition

Task; SKT=Syntactic Knowledge Task

• 3
• 2
• 1

1 2 VKT WRT SKT Grade 8 8 Z Z-Score c1 c2 c3 c2; 25% c1; 72% c3; 3%

SLIDE 29

• 2

2

RC factor itself

1 2 3

c

<.0001 Prob > F 196.02 F

Distribution of frc

• 2

2

RC factor itself

1 2 3

c

<.0001 Prob > F 196.02 F

Distribution of frc

• Note. The average absolute value of the standardized difference in latent RC

performance across all classes was Hedge’s g =2.19, indicating the magnitude of differences in FRA skill profile performance on standardized outcome

G8 R G8 RC F C FACT CTOR B BY L Y LATE TENT CL NT CLASSE ASSES

SLIDE 30

¡ LPA identified 5-6 classes in K-5 and only 3 in secondary grades. ¡ Latent profiles significantly related to standardized reading

• utcomes, accounting for 24% (in G3) to 61% (G9) of the

variance, with the mode being 42%. ¡ Range of Hedges g (for average absolute values of the standardized difference in reading outcome across all latent classes) was 1.10 (in K) to 2.53 (in G5). ¡ Profiles above G5 fell into a pattern of low, medium, and high. ¡ The 5-6 reading and language profiles found in the elementary grades reflect heterogeneity of skills.

SU SUMMAR ARY O OF F RES RESUL ULTS

SLIDE 31

¡ Fact that latent profiles accounted for substantial differences in RC in a large diverse sample of students spanning 11 grades contributes to a field dominated by (a) unreliable or unstable classifications, and (b) LCA of clinical samples (e.g., Catts et al., 2012; Justice et al., 2015) or low-performing students (Logan & Petscher, 2010; Brasseur-Hock et al., 2011). ¡ Heterogeneity of skill profiles in the elementary grades in contrast to the low, medium, & high profiles in the secondary grades suggests focusing on differentiating instruction in K-5. ¡ Importance of FRA academic language (vocab and syntax) to RC, more than word recognition, apparent in G3 and above.

CO CONCL NCLUSI USIONS NS

SLIDE 32

¡ Cross-sectional rather than longitudinal. [Replications of findings across grades helps.] ¡ Profiles and their relations to reading outcomes are limited to the measures used. [At least a latent variable of reading comprehension was used, which related strongly to the single measures in the FRA reading and language measures.] ¡ Next step is to test results of the heterogeneous profiles from this exploratory LPA with confirmatory latent class analysis.

LIMIT LIMITATIONS IONS/NEX NEXT STE STEPS PS

SLIDE 33

Im Immediat diate and and Ac Actionable Dat Data

CONNECTION TO INSTRUCTION

How do we make these types of results meaningful for educators?

SLIDE 34

HOW DO EDUCATORS MAKE SENSE OF ALL OF THIS DATA??

SLIDE 35

ONLY 16% OF EDUCATORS FEEL PREPARED

National Center for Literacy Education (2014)

16%

SLIDE 36

QUESTIONS TO DRIVE INSTRUCTION & DECISIONS

SLIDE 37

¡ As a whole school or district, what percent of my students are at-risk for reading failure? What percent are on track for reading success? ¡ Have we seen growth in my students across the year? If so, how much growth? ¡ What subgroups of students are on track for reading success and which subgroups are at-risk for reading failure? ¡ Do I see any patterns of strengths and weaknesses across grades in my district/school?

§ This will help address PD and Curriculum needs

QUESTIONS ADMINISTRATORS NEED ANSWERED

SLIDE 38

¡ How many of my students are

§ Above grade level, § On grade level or § Below grade level?

¡ If they are above grade level, how far above? If they are below, how far below? ¡ Most importantly, how do I help those who are below, get to where they need to be?

QUESTIONS TEACHERS NEED ANSWERED

SLIDE 39

¡ What percent of my students are on track for reading success? ¡ What patterns of strengths and weaknesses do I see in my class? How can I adjust my instruction to meet the needs? ¡ What do these patterns mean for my small group instruction? Are there certain students who need the same type of instruction? If so, who are they and what is the instruction they need?

QUESTIONS TEACHERS NEED ANSWERED

SLIDE 40

“The goal (of assessment) is to gain enough “The goal (of assessment) is to gain enough information about student progress to make information about student progress to make efffective decisions while minimizing the time spent ective decisions while minimizing the time spent administering assessments.” administering assessments.”

• Torgesen, 2006, p.3

REMEMBER…

SLIDE 41

¡ Include oral language measures in screening, diagnostic, and progress monitoring of reading to accurately predict reading comprehension. ¡ Describe profiles of strengths and weaknesses in a valid, reliable, and efficient manner. ¡ Use profiles to differentiate instruction.

CONCLUDING THOUGHTS

SLIDE 42

bfoorman@fcrr.org ypetscher@fcrr.org lbrooke@lexialearning.com amitchell@lexialearning.com

THANK YOU!