Contributions to Generalizability Theory: Dick Jaegers Indirect but - PowerPoint PPT Presentation

Contributions to Generalizability Theory: Dick Jaeger’s Indirect but Strong Mentoring Effects Richard J. Shavelson Stanford University Richard M. Jaeger Memorial Symposium American Educational Research Association Meeting in Seattle, Washington April 13, 2001

Jaeger’s Old-Time Fortran Program

Jaeger’s Dissertation • Formulated useful models for complementary testing programs—programs that provided both student level and institution level information • Demonstrated the feasibility of using sampling procedures in school testing programs • Determined the relative efficiencies of various sampling methods for schools • Developed a set of computer programs for applying sampling framework • Provided a guide to finite population sampling.

Figure 2 MDS

‘s Old Fashioned Output

Comparison of the Quality of Jaeger’s and Shavelson’s Dissertations Title Words Have in Author Page How Weight? Colon? Title Length Thick? Jaeger 416 3” 3 lbs Yes 12 Shavelson 159 1.2” 1.4 lbs No 14

Sampling Framework Standard Science as Inquiry: “Design and Conduct a Scientific Investigation” Define Define Construct ? Declara- tive Domain Extent Force & Motion Structure Procedu- Define ral Task/ Task/ Response Response Task/ Strategic Response Friction Task/ Task/ Response Response Task/ Task/ Response Response Observed Sample Behavior on the Task/Response Sampled Generalizable to Other Tasks on the Domain?

Sampling Framework for Reliability and Validity Is a score assigned generalizable, for example, across: • Tasks? • Occasions? Reliability • Judges? • Methods? Validity • Expertise?

Award Winning Blank Table

Simplifying G Theory

Fig with Task Samp Variance

Sampling Variability Task? Occasion? Or Both? ---------------------------------------------------------------------------------- Estimated Percent Source of Variance Total Variability n Component Variability ---------------------------------------------------------------------------------- Person (p) 26 0.07 4 Rater (r) 2 0.00 a 0 Task (t) 2 0.00 a 0 Occasion (o) 2 0.01 1 pr 0.01 1 pt 0.63 32 po 0.00 a 0 rt 0.00 0 ro 0.00 0 Both! to 0.00 a 0 prt 0.00 a 0 pro 0.01 0 pto 1.16 59 rto 0.00 a 0 prto,e 0.08 4 ---------------------------------------------------------------------------------- a A small, negative variance component was set to zero.

Wiley’s 4 Bootstrap Commandments • Thou shalt select a different bootstrap resampling strategy for each variance component to be estimated. • Thou shalt select a bootstrap strategy that involves resampling only across the dimensions(s) represented by the variance component of interest • Thou shalt select a bootstrap strategy that involves resampling of effects represented by the variance component of interest • Thou shalt adjust point estimates generated by bootstrap strategies based on the properties of the bootstrap as well as the behavior of the statistic to be estimated

Dick Will Be Sitting On My Shoulder Unfortunately, Dick left us before he could hear… and evaluate the good Bootstrap news. But he will always be sitting on my shoulder, my conscious, asking if I got it right.