Contributions to Generalizability Theory: Dick Jaegers Indirect but - - PowerPoint PPT Presentation

Contributions to Generalizability Theory: Dick Jaeger’s Indirect but Strong Mentoring Effects

Richard J. Shavelson Stanford University

Richard M. Jaeger Memorial Symposium

American Educational Research Association Meeting in Seattle, Washington April 13, 2001

Jaeger’s Old-Time Fortran Program

Jaeger’s Dissertation

• Formulated useful models for complementary

testing programs—programs that provided both student level and institution level information

• Demonstrated the feasibility of using sampling

procedures in school testing programs

• Determined the relative efficiencies of various

sampling methods for schools

• Developed a set of computer programs for

applying sampling framework

• Provided a guide to finite population sampling.

Figure 2 MDS

‘s Old Fashioned Output

Comparison of the Quality of Jaeger’s and Shavelson’s Dissertations

Author Page Length How Thick? Weight?

Title Have Colon? Words in Title

Jaeger 416 3” 3 lbs

Yes 12 Shavelson

159 1.2” 1.4 lbs

No 14

Sampling Framework

Standard

Science as Inquiry: “Design and Conduct a Scientific Investigation”

Construct

Declara- tive Procedu- ral Strategic Extent Structure ?

Observed Behavior

• n the

Task/Response Sampled Domain Force & Motion

Task/ Response Task/ Response Task/ Response Task/ Response Task/ Response Task/ Response Friction Task/ Response

Define Define Define Sample Generalizable to Other Tasks on the Domain?

Sampling Framework for Reliability and Validity

Is a score assigned generalizable, for example, across:

• Tasks?
• Occasions?
• Judges?
• Methods?
• Expertise?

Validity Reliability

Award Winning Blank Table

Simplifying G Theory

Fig with Task Samp Variance

Sampling Variability Task? Occasion? Or Both?

• Estimated

Percent Source of Variance Total Variability n Component Variability

• Person (p)

26 0.07 4 Rater (r) 2 0.00a Task (t) 2 0.00a Occasion (o) 2 0.01 1 pr 0.01 1 pt 0.63 32 po 0.00a rt 0.00 ro 0.00 to 0.00a prt 0.00a pro 0.01 pto 1.16 59 rto 0.00a prto,e 0.08 4

• aA small, negative variance component was set to zero.

Both!

Wiley’s 4 Bootstrap Commandments

• Thou shalt select a different bootstrap resampling strategy

for each variance component to be estimated.

• Thou shalt select a bootstrap strategy that involves

resampling only across the dimensions(s) represented by the variance component of interest

• Thou shalt select a bootstrap strategy that involves

resampling of effects represented by the variance component

• f interest
• Thou shalt adjust point estimates generated by bootstrap

strategies based on the properties of the bootstrap as well as the behavior of the statistic to be estimated

Dick Will Be Sitting On My Shoulder

Unfortunately, Dick left us before he could hear… and evaluate the good Bootstrap news. But he will always be sitting on my shoulder, my conscious, asking if I got it right.