Working with What Works Clearinghouse Standards to Evaluate Designs - - PowerPoint PPT Presentation
Working with What Works Clearinghouse Standards to Evaluate Designs for Broadening Participation Kate Winter (Kate Winter Evaluation, LLC) & Eva Fernndez, Sabrina Avila, Patrick Johnson, & Jennifer Valad (CUNY Queens College) 2018
2018 Transforming STEM Higher Education Association of American Colleges & Universities ~ Network for Academic Renewal November 10, 2018
(1) The data elements and the data collection procedures are clearly described and appropriate to measure the attainment of activity objectives and to measure the success of the project in achieving the goals of the comprehensive development plan; (up to 5 points) (2) The data analysis procedures are clearly described and are likely to produce formative and summative results on attaining activity objectives and measuring the success of the project on achieving the goals of the comprehensive development plan; (up to 5 points) and (3) The evaluation will provide guidance about effective strategies suitable for replication or testing in
http://hsistem.qc.cuny.edu
TLAs WWC: What Works Clearinghouse RCT: Randomized Control Trial RDD: Regression Discontinuity Design QED: Quasi-Experimental Design ITT: Intention To Treat
*Liberal thresholds are used in postsecondary education reviews
The following case studies are real. At the request of the survivors, the names have been changed. Out of respect for the dead, The rest has been told exactly as it occurred. (Adapted from Fargo, 1996)
Department Normal offers 20 sections of NORM-101 to the project, with 10 students each. The treatment—a redesigned full-term set of lab experiences with standardized materials, activities, and assessments—was implemented the same way by every lab instructor that used it. In all, 10 treatment and 10 control sections were randomly assigned. Following assignment, 6 treatment-assigned instructors indicated that they were not involved in the redesign efforts and would not be risking their students’ learning to untested
significant value in the new materials during the workshops when they were being built and eagerly implemented them with their sections. Halfway through the term, 1 of the treatment-assigned faculty who
improvements in students engaged in the treatment. The treatment appears so successful that the chair wants to realign the final exam in the lecture course based on the amazing new materials being covered in the treatment labs, to more accurately reflect student learning. The score on this exam serves as the primary outcome measure for the lab treatment. Inexplicably, the external evaluator is pushing back against this idea. Department Enthusiastic had 4 of 10 instructors who were so excited about the new grant that they decided to implement their redesigned sections without waiting for instruction to do so, impacting 9 of 20
that they could compare their results at the end of the term to determine which approaches were most effective. Everyone is confused by the external evaluator’s sigh and sad smile when they inform her. Department Small only offers 2 lectures and 4 labs of SMAL-102 each term, divided equally among 2 instructors. Fortunately, both instructors are willing to offer treatment sections, but it is unclear if they are each willing to offer BOTH treatment and control sections in the same term. The external evaluator keeps telling them they cannot
analysis being reported to the funding agency, so the department is
continues to push the project coordinators to closely monitor this department to ensure that the instructors each offer 1 treatment and 1 control section of the lab. At the end of the term, Lead Institution is ready to provide institutional data about students in study-assigned sections to the external evaluator for analysis. Per WWC guidelines, since there are no baseline measures for course GPA, retention, or graduation, one measure of prior academic achievement (a composite of HS or transfer GPA and SAT/ACT) and one of SES (Pell eligibility) are used as covariates in the analysis model. However, prior academic achievement scores are frequently missing for transfer students. Similarly, many students do not complete the FAFSA, so SES data are not available for them. Despite there being more than 3,000 students in the courses where the redesigns are taking place, the restriction to
headcount significantly. Of 1,000 students in the study sections, 25% are missing one baseline and 25% are missing the other, with 25% of students missing something missing both. This leaves 625 students in the analytic sample if baseline scores are not imputed. Meanwhile, the on-campus IRB is pushing to require Informed Consent from all students in the study and the evaluator has your data lead on speed-dial.
Lead Institution When was the last time you worked with Institutional Research (or equivalent) to access data? How complete was that data? Do you remember what proportion of your students complete the FAFSA? What strategies might help maximize the completeness of the data? Department Small Given the 2 faculty and 2 lectures with 4 labs each structure of this course, what
confound where only one instructor offers the treatment or control condition? Implementation fidelity – having easily implemen- ted standardized materials and strategies facili- tates standard implementation. Compliance – although one treatment sec- tion joins late and only threa- tens implemen- tation fidelity. Department Enthusiastic
sections’ data be used in the RCT analysis? Why or why not?
were variations on a theme aimed at the same outcome, could the treatment sections be analyzed together against the remaining “business as usual” sections, assuming baseline equivalency (i.e., a QED)? Why or why not?
treatment, so that he or she was the only source of that treatment, would it be possible to analyze the impacts of those treatments through an experimental design? Why or why not?
any of the treatment conditions had an impact? Implementation fidelity. Overalignment Department Normal
treatment and control sections were
Assuming all sections are full and all students have the needed data (i.e, there is no attrition):
assigned treatment sections? Control sections?
treatment (at any level)? How many to control conditions?
treatment and 73 in the control in order to detect an effect size of a 0.4 (alpha = .05 and power of .8). Will we have enough students following an ITT protocol (as assigned)? How about following a ToT protocol (as received)?
Case 1: Fall Term
Implementation fidelity, though if this is consi- dered as testing multiple treat- ments through a QED, then it hits the N of 1 confound: only 1 individual
different intervention. No random assignment. Avoids N of 1 con- found by having the same instruc- tor offer both a treatment and a control condition. As long as EITHER instructor offers BOTH, the data will count in the RCT. N of 1 confound if each instructor offers 1 of the 2 lectures. Issues accessing data
Attrition
administered the post-test (i.e., 0% cluster-level attrition within the treatment sections), and the control cluster-level attrition remained the same, would the cluster-level attrition be high or low per WWC guidelines?
attrition not counting against a study twice, would the student-level attrition be high or low?
are going to “lose” clusters and/or individuals from a study:
samples?
treatment attrition with reasonable to high control attrition?
and control groups (assuming you still have enough data points to have the statistical significance to detect treatment effects), is that a problem for the study based on attrition, per WWC guidelines? Individual-level attrition Cluster-level attrition Increasing compliance, drastically shrinking sample sizes Department Unsupervised
implementation fidelity and compliance data hinder ToT (as received) analysis?
implementation fidelity and compliance data NOT hinder ITT (as assigned) analysis?
approaches to maximize monitoring and tracking with small project staffs? Compliance monitoring/ implementation fidelity monitoring
Department Unsupervised offers 20 sections of UNSU-110 to the
assigned sections were monitored, and not as thoroughly as desired. Following assignment, 1 treatment-assigned instructor implemented the treatment as designed for the full term, 2 offered it for all but the first three weeks, 2 more offered half the treatment activities for the full term, 2 decided they were not ready and did not implement anything new, and no data are available for the remaining 3 treatment-assigned sections or any of the control-assigned sections. While this does not impact the ITT protocol, your evaluator is claiming that this hinders her efforts to provide you with formative assessment
There are 2 campuses participating in this grant-funded project. At campus 1 in the Fall term, 20 sections were included in the RCT allowing for 10 sections each of treatment and control (Department Normal). However, many faculty misunderstood how RCT works and were found to be in non-compliance, which undermined the data. At campus 2, therefore, only 4 classes were included in RCT because the department did not want to risk non-compliance by having a single instructor assigned to both a treatment and a control condition, so
sections to be randomly assigned. These 4 instructors/sections with their 15 students each were then randomly assigned to the treatment (2 sections, 30 students) and control (2 sections, 30 students)
larger sample, the evaluator claims that campus 2 had more meaningful data and found a treatment effect size of .7 SD, while campus 1 detected no treatment effect. While the primary project outcomes of interest are course GPA, retention, and graduation, which do not lend themselves easily to attrition from the study (even a student who leaves has a data point and therefore remains in the study), another component of the analysis entails psycho-social surveys of students’ sense of belonging in STEM and self-confidence/self-efficacy engaging in STEM coursework and pursuing STEM careers. These constructs are measured using validated survey instruments administered to treatment and control students before Fall term starts and again at the end of the academic year through the study sections. Of the 50 sections (25 treatment and 25 control) that administered the pre-test and were supposed to administer the post-test survey, 12 did not (5 txt, 7 cntrl; 20% and 28% attrition, respectively, with an overall attrition
pre-test, but only 52% of the students in the remaining treatment sections took it and 48% of the students in remaining control sections (50% overall attrition, 4% differential attrition). This means that from the original pre-test pool, we only have post-test data for 42% of treatment section students and 35% of control section students (62%
understands why the evaluator is not freaking out about such low response rates and such high attrition.
Implementation fidelity Compliance. Sample Size v. Compliance
control) had closed due to lack of enrollment, would it have impacted the validity of the data (i.e., introduced confounding factor)? Why or why not?
campus to increase the likelihood that treatment-assigned faculty will offer the treatment and control-assigned faculty will not? Compliance monitoring/ Implementation fidelity monitoring
Case 2: Spring Term
Yay! Compliance! WWC-approved
You do not need a lot of statistical power to detect such a large effect, but you do need Clean Data. Smaller samples are very useful if they are in compliance. WWC-approved measure Distraction – this does NOT repre- sent actual indivi- dual level attrition because cluster- level attrition would then count against us twice
Winter, K., Fernández, E., Avila, S., Johnson, P., & Valad, J. (2018). Working with What Works Clearinghouse Standards to evaluate designs for broadening participation. 2018 Transforming STEM Higher Education, AAC&U Network for Academic Renewal. https://goo.gl/WXXQpJ
WWC Evidence Standards Glossary of Terms
1
Term Definition Why It’s Problematic Info Analytic Sample Size The number of cases with data points in the analysis. Depending on attrition and the number of variables needed in the analysis (blocking variables, required covariates, etc.), small analytic samples may not have the statistical power necessary to detect the small effect sizes common in education research. Attrition The percentage of a sample randomly assigned to a study that is not part of the final analysis sample, calculated overall and for each study
(in cluster RCTs) and the individual level (in any RCT). High attrition precludes RCTs and RDDs from meeting the highest WWC evidence standards and requires demonstration of baseline equivalence.
2
Cross-Over See Non-Compliance Effect Size A standardized measure of the magnitude of an
(measured in standard deviations) in an average student’s outcome that can be expected if that student is given the intervention. Because effect sizes are standardized, they can be compared across outcomes and studies. Many interventions in education have small effect sizes (less than .3 SD), which necessitate having enough statistical power AND high enough compliance to detect them.
3
Experimental Design A method of research in the social sciences in which a controlled experimental factor is subjected to special treatment for purposes of comparison with a factor kept constant. For WWC
4
purposes, includes RCTs, RDDS, and QEDs. Requires specific processes and structures, can increase costs and resources needed to conduct the study. FERPA The Family Educational Rights and Privacy Act (FERPA) (20 U.S.C. § 1232g; 34 CFR Part 99) is a Federal law that protects the privacy of student education records. The law applies to all schools that receive funds under an applicable program of the U.S. Department of Education. Depending on how an institution interprets the rules, this can severely limit access to necessary student data, shrinking analytic samples and reducing statistical power. Secondary analysis of existing data, when used anonymously to understand student outcomes in the aggregate and disaggregated by sub-groups of interest, is typically not protected by FERPA.
5
Implementation Fidelity When an intervention is implemented as proposed, with regards to content, quality, timing, and duration If an intervention requires certain elements to be effective and these are not implemented as intended, it can reduce or eliminate the effectiveness of the intervention. Variation in implementation can therefore impact the treatment effect.
6
Informed Consent Informed consent is an ethical and legal requirement for research involving human
is informed about all aspects of the trial, which are important for the participant to make a decision and after studying all aspects of the trial the participant voluntarily confirms his or her willingness to participate. For studies of routine educational outcomes, such as GPA and graduation, it is rarely necessary to inform individuals that data already collected (i.e., secondary analysis) is being used anonymously and in an aggregated form to understand institutional or intervention effectiveness. Requiring informed consent to access routinely collected and anonymous data can significantly reduce sample sizes and statistical power, while potentially creating additional burden for individuals. For studies collecting new data directly from students (interviews, surveys), informed consent is typically required, but new survey technologies permit anonymous data collection and these methods can be exempted from IRB review.
7
1
Many harvested verbatim from the WWC Glossary of Terms (https://ies.ed.gov/ncee/wwc/Glossary/Group%20Design%20Standards)
2
https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_brief_attrition_080715.pdf
3 https://effectsizefaq.com/2010/05/30/what-are-some-conventions-for-interpreting-different-effect-sizes/ 4 https://www.merriam-webster.com/dictionary/experimental%20design 5 https://www2.ed.gov/policy/gen/guid/fpco/ferpa/index.html 6
https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-2-40
7
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3777303/
Term (Cont.) Definition Why It’s Problematic Info ITT (Intention to Treat) Protocol All cases within all study conditions are analyzed as if they received the treatment condition to which they were randomly assigned, regardless of whether they did. This is the required approach to meet WWC evidence standards. If some treatment-assigned cases do not receive exposure to treatment, and they are analyzed as if they had, it can dilute the treatment average and hide a treatment effect. Conversely, if control-assigned cases receive exposure to treatment, and they are analyzed as if they had not, it can enhance the control average and hide a treatment effect.
8
N of 1 Confound A component of a study that is completely aligned with one of the study conditions, aka a confounding factor. For example, a study may have one intervention school and a different comparison school. In this case, it is impossible to separate how much of the observed effect was due to the intervention and how much was due to the particular school in which the intervention was used. In small departments, it can be difficult to create randomly assigned scenarios where there are more than one instructor offering the treatment and more than one offering the control. In these instances, it is necessary to have the same instructor offer both conditions (instructor is no longer completely aligned with either condition), which many instructors do not wish to do.
9
Non-Compliance When a case does not receive the study condition to which it was randomly assigned. See ITT Protocol Overalignment A study closely tailors an outcome measure to a condition, or the measure repeats some aspect of a condition (usually the treatment). If a measure is overaligned, the study findings may not accurately indicate the effect of the intervention.
10
Per Protocol See Treatment on the Treated (ToT) QED (Quasi- experimental Design) These designs have a treatment and control group but do not randomly assign cases to them. Rather, baseline equivalence must be established
analyzing outcome differences between the groups. Requires specific processes and structures, can increase costs and resources needed to conduct the study.
11
RCT (Randomized Controlled Trial) A study design that randomly assigns participants into an experimental group or a control group. As the study is conducted, the only expected difference between the control and experimental groups in a randomized controlled trial (RCT) is the outcome variable being studied.
12
Requires specific processes and structures, can increase costs and resources needed to conduct the study. Can be difficult to get good data if the design is not implemented well (i.e., random assignment or compliance issues, implementation fidelity issues, attrition issues, etc.)
13
RDD (Regression Discontinuity Designs) These designs use a pretest-posttest program-comparison group strategy by assigning participants to conditions solely on the basis of a cutoff score on a pre-program measure. Requires specific processes and structures (mathematical magic), can increase costs and resources needed to conduct the study.
14
Statistical Power Statistical power is the likelihood that a study will detect an effect when there is an effect there to be detected. If statistical power is high, the probability of making a Type II error, or concluding there is no effect when, in fact, there is one, goes down. Statistical power is affected chiefly by the size of the effect and the size of the sample used to detect
effects, while large samples offer greater test sensitivity than small samples.
15
Treatment on the Treated (ToT) Also called “per protocol” and analyzes the treatment effect on all who received treatment versus all who did not, regardless of random assignment. This approach does not meet WWC evidence standards and requires close monitoring and tracking of treatment implementation.
8
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3159210/
9
https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_brief_confounds_101117.pdf
10
https://ies.ed.gov/ncee/wwc/Docs/OnlineTraining/wwc_training_m5.pdf
11 http://www.socialresearchmethods.net/kb/quasiexp.php 12 https://himmelfarb.gwu.edu/tutorials/studydesign101/rcts.html 13 Ibid. 14
http://www.socialresearchmethods.net/kb/quasird.php
15 https://effectsizefaq.com/2010/05/31/what-is-statistical-power/
Winter, K., Fernández, E., Avila, S., Johnson, P ., & Valad, J. (2018). Working with What Works Clearinghouse Standards to evaluate designs for broadening participation. 2018 Transforming STEM Higher Education , AAC&U Network for Academic
https://goo.gl/99NXkX ; Slides: https://goo.gl/WXXQpJ