Conducting Social Policy Research in the Private Sector A Day in the Life of a Member of the Abt Associates Team November 1, 2019 Michael Frye, Ph.D.
Agenda § Introductions § Overview of Abt Associates § Description of my day-to-day work at Abt, specifically on the What Works Clearinghouse § Open discussion about opportunities to work at Abt
How did I get here? § BS in statistics from Virginia Tech in 2006 § Discovered social policy research in the private sector after graduation § Worked for RMC Research Corporation from 2006-2012 § Started working at Abt Associates in 2012 § Completed MS (2008) and Ph.D. (2015) programs while working
Who We Are 4
Core Capabilities § Communications & Behavior Change § Data Capture & Surveys § Digital Transformations § Research, Monitoring, and Evaluation § Technical Assistance & Implementation 5
Key Markets & Global Expertise § Education § Environment & Energy § Food Security & Agriculture § Governance & Justice § Health § Behavioral Health § Chronic/Non-Communicable Diseases § Health Systems § Infectious Diseases § Private Sector Health § Reproductive, Maternal, Newborn, and Child Health § Housing, Communities, & Asset Building § Workforce & Economic Mobility 6
Our clients § U.S. federal, state, and local governments § Health and Human Services (HHS) § Department of Labor (DOL) § Department of Education (Ed) § Food and Nutrition Service, USDA § Environmental Protection Agency (EPA) § Centers for Disease Control (CDC) § United States Agency for International Development (USAID) § International governments § Department for International Development (DFID) § Department of Foreign Affairs and Trade (DFAT) § Universities and foundations § Bill and Melinda Gates Foundation § CDC Foundation § Medtronic 7
SEP projects Abt’s Social and Economic Policy (SEP) division conducts social § policy research in the following market areas: § Education § Housing, Communities, & Asset Building § Workforce & Economic Mobility § SEP research projects tend to be contracts with federal agencies like Institute of Education Sciences, the Department of Education’s research office. We compete with other similar organizations for these projects. § Sometimes, we will partner with other organizations, universities, or § individuals to form the strongest team for a complex research project. Project scope and duration can vary. § 8
SEP’s education practice Abt currently has 50 different education projects with 19 different § clients. Topics include: § § Child welfare and youth development § Teacher preparation § Literacy § College transitions and success § English language learners § Adult education § Early childhood § STEM education and training § Expanded learning time § Private school vouchers § Charter schools § Methods include: randomized controlled trials (RCTs), evidence reviews, evaluation technical assistance, quasi-experimental designs (QEDs), descriptive studies. 9
Education projects § What Works Clearinghouse Postsecondary Education and Postsecondary Preparation Evidence Reporting (WWC-PEPPER) contract. § DC Opportunity Scholarships Program: a QED study that examines a voucher program that allows students in DC to attend private schools. § College Access: RCTs that test enhancements to two programs: (1) the use of text messaging within Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) and (2) a combination of student materials, advisor development, and messaging (Find the Fit) within the Upward Bound program. 10
What do I do at Abt? § Project Director on the WWC-PEPPER contract, and also advisor on another WWC project to develop Practice Guides § Lead an evidence review for a project to examine how districts nationwide spend Title IV-A money § Manage the Federal Student Aid Experimental Sites Initiatives: a series of experiments that test variations in the disbursement of student aid and the delivery of financial aid services to students § Support an implementation study that examines variation in instructional quality and education experience of students receiving DC-OSP vouchers § Supervise multiple staff an run an initiative to mentor junior staff
What Is the WWC? § The What Works Clearinghouse (WWC) reviews the existing research on different programs, products, practices, and policies in education. § Its goal is to provide educators with the information they need to make evidence-based decisions using the results from high-quality research . § It disseminates information through products such as Intervention Reports and Practice Guides . 12
WWC Contractors Department of Education (ED) Institute of Education Sciences (IES) National Center for National Center for National Center for National Center for Education Research Education Evaluation Special Education Education Statistics & Regional Assistance Research WWC Postsecondary WWC PK-12 WWC Grants WWC Coordination WWC PGs Mathematica Policy Development American Institutes for IRG (with MPR) Abt Associates Research (with Abt) Services Group Research 2M (with Abt) 13
What Is WWC-PEPPER? § Covers WWC reviews of research on programs, products, practices, and policies designed to support success at the postsecondary level. § Began in 2012. Abt’s WWC-PEPPER contract began in October 2016 and ends October 2021. § Continues the WWC’s reputation as a “central and trusted” resource for evidence by delivering actionable information to education practitioners, namely in the form of WWC Intervention Reports and Practice Guides. 14
How does the WWC work? § The WWC has developed a set of Evidence Standards that can be applied to group design studies to determine if the findings are credible. § Guided by these standards, the WWC conducts systematic reviews in the following steps: 1. Relevant studies are identified via a literature search. 2. Studies are screened to confirm eligibility. 3. Eligible studies are reviewed by WWC certified staff 4. Findings from studies that meet standards are synthesized and published in WWC products, which are posted on the website. 15
The WWC website 16
WWC products § Intervention Reports that summarize research findings on the effectiveness of a particular curriculum, program, practice, or policy. Practice Guides that provide education practitioners with a set of § recommendations to improve practice in their classrooms and schools around a broad education topic (e.g. using technology, teaching students to write) § These recommendations capture the intersection of subject matter expertise and supporting evidence from studies that meet the WWC standards. § Information on individual studies that were reviewed either for the above mentioned products or other purposes (e.g. grant competitions). § Data on all of the studies reviewed by the WWC. § Handbooks and protocols that define the standards and procedures that guide our reviews. Webinars that inform the public of the WWC practices and products that are § available. 17
WWC Intervention Reports 18
WWC Practice Guides 19
WWC Review Process § All reviews are guided by a protocol that specifies: § Topic statement § Interventions of interest § Eligible populations § Outcomes included § Technical issues § Literature search strategy § Studies are double-coded by WWC certified reviewers and then reconciled. § If a study receives a rating of Does Not Meet WWC Standards from the first reviewer, it will be single coded and reconciled. 20
WWC Evidence Standards § WWC standards cover four research designs that introduce a strong counterfactual (i.e., establishing what would have happened in the absence of the intervention): § Randomized controlled trials (RCTs): Randomly assigning subjects (students, classrooms, schools, districts) to a treatment group or a control group. § Quasi-experimental designs (QEDs): Developing a comparison group in a non-random fashion (usually through matching). § Regression discontinuity design (RDDs): Using a “forcing variable” to assign treatment; impacts are assessed at the cut point. § Single-case designs (SCDs): Studying a single participant (or group of participants) using outcomes that are measured repeatedly across different conditions. The subject effectively serves as its own control. 21
Key Concepts § Attrition: RCTs must have (a) low overall attrition, (b) low differential attrition, and (c) low attrition at both the cluster and subcluster levels in cluster designs. § Baseline equivalence: QEDs must establish baseline equivalence on the analytic sample for each contrast; baseline measure must be correlated with outcome measure. § Confounding factors: An aspect of the intervention was perfectly aligned (“confounded”) with either the intervention or the comparison group. In these cases, it is not possible to tell whether the intervention or the confounding factor is responsible for the difference in outcomes. – N=1 problem – A characteristic of the intervention is systematically associated with the outcome (e.g., teachers in treatment group all have more experience than controls) – Bundled interventions – Time confounds (treatment and control groups formed in different years) § Valid outcomes: Outcomes must meet standards for face validity, reliability, consistency, lack of overalignment. 22
Recommend
More recommend
