The he Right Right Tool ool for the or the Job: ob: A Bayesian - - PowerPoint PPT Presentation

β–Ά
the he right right tool ool for the or the job ob a
SMART_READER_LITE
LIVE PREVIEW

The he Right Right Tool ool for the or the Job: ob: A Bayesian - - PowerPoint PPT Presentation

Apr 23, 2023 158 likes β€’422 views

The he Right Right Tool ool for the or the Job: ob: A Bayesian Meta-Regression of Employment and Training Studies Presentation at the OPRE Methods Inquiries Meeting October 20, 2017 Lauren Vollmer Emily Sama - Miller Alyssa

slide-1
SLIDE 1

The he Right Right Tool

  • ol for the
  • r the Job:
  • b:

Presentation at the OPRE Methods Inquiries Meeting

A Bayesian Meta-Regression of Employment and Training Studies

Lauren Vollmer β€’ Emily Sama-Miller β€’ Alyssa Maccarone

October 20, 2017

slide-2
SLIDE 2

2 2

Motivation

slide-3
SLIDE 3

3 3

What is Meta-Regression?

𝑍

𝑗 = 𝛽 + π›Ύπ‘Œπ‘— + πœπ‘—

  • 𝑍

𝑗: earnings for person 𝑗

  • π‘Œπ‘—: background information about person 𝑗
slide-4
SLIDE 4

4 4

What is Meta-Regression?

𝑍

𝑗 = 𝛽 + π›Ύπ‘Œπ‘— + πœπ‘—

  • 𝑍

𝑗: estimate in study 𝑗

  • π‘Œπ‘—: background information about study 𝑗
slide-5
SLIDE 5

5 5

Why Meta-Regression?

slide-6
SLIDE 6

6 6

Why Meta-Regression?

slide-7
SLIDE 7

7 7

Why Meta-Regression?

  • Synthesize information rigorously across related

studies

– Overall effect across studies – Average effect across outcomes within a study

  • Quantify relationships between study features and
  • utcomes
  • Weight observations according to their precision
slide-8
SLIDE 8

8 8

Why Bayesian Meta-Regression?

slide-9
SLIDE 9

9 9

  • Incorporate prior information

Why Bayesian Meta-Regression?

slide-10
SLIDE 10

10 10

  • Incorporate prior information

– β€œBorrow strength” from related studies – Examine variation in effects without sacrificing precision – Enhance the plausibility of the estimates

Why Bayesian Meta-Regression?

slide-11
SLIDE 11

11 11

Why Bayesian Meta-Regression?

  • Incorporate prior information

– β€œBorrow strength” from related studies – Examine variation in effects without sacrificing precision – Enhance the plausibility of the estimates

  • Describe conclusions probabilistically
slide-12
SLIDE 12

12 12

Why Bayesian Meta-Regression?

  • Incorporate prior information

– β€œBorrow strength” from related studies – Examine variation in effects without sacrificing precision – Enhance the plausibility of the estimates

  • Describe conclusions probabilistically

– β€œThere is a 15 percent chance that intervention X improves

  • utcome Y by 5 percent or more.”

– Use plain, intuitive language – Focus on practically meaningful thresholds – Avoid binary or β€œbright line” thinking

slide-13
SLIDE 13

13 13

Example: Employment Strategies Evidence Review (ESER)

slide-14
SLIDE 14

14 14

Employment Strategies Evidence Review

  • Project for the Office of Planning, Research, and Evaluation

at the Administration for Children and Families

  • Systematic review of literature on employment and training

programs and policies for low-income workers

– Published between 1990 and 2014 – Conducted in the US, UK, or Canada

  • Reviewers rated the quality of each study’s causal evidence

as high, moderate, or low

slide-15
SLIDE 15

15 15

An ESER Study

Target Population Outcome 1 Outcome 2 Outcome 𝑙 Strategy 1 Strategy 2 Strategy 3 Strategy 𝑑

… …

Outcome 3 Intervention

slide-16
SLIDE 16

16 16

ESER Meta-Regression Research Questions

  • 1. What works?

– Past interventions – Specific employment strategies

  • 2. What works in which domains?
  • 3. What works for which populations?
  • 4. What works for which populations in which domains?
slide-17
SLIDE 17

17 17

Meta-Regression Implementation

  • Standardize impact estimates using effect sizes

– ESER studies did not provide adequate data to calculate effect sizes for continuous outcomes (e.g. earnings) – Restricted attention to binary outcomes:

  • Employment
  • Public assistance receipt
  • Educational attainment

– Use the odds ratio effect size metric

  • Align the sign of favorable/unfavorable impacts

across outcomes

– A positive estimate should denote a favorable impact – Public assistance receipt β†’ independence from public assistance

slide-18
SLIDE 18

18 18

Meta-Regression Model: Main Effects

π‘§π‘—π‘˜ = 𝛽 + π‘π‘˜ + 𝑐𝑒 𝑗 + 𝑑𝑑𝐽𝑑 π‘˜

𝑇 𝑑=1

+ π‘•π‘žπ½π‘ž π‘˜

𝑄 π‘ž=1

+ 𝑔

𝑑𝑒𝐽𝑑 π‘˜ 𝐽𝑒 𝑗 𝐸 𝑒=1 𝑇 𝑑=1

+ β„Žπ‘žπ‘’π½π‘ž π‘˜ 𝐽𝑒 𝑗

𝐸 𝑒=1 𝑄 π‘ž=1

+ π‘šπ‘‘π‘žπ½π‘‘ π‘˜ π½π‘ž π‘˜

𝑄 π‘ž=1 𝑇 𝑑=1

+ π‘›π‘’π‘‘π‘žπ½π‘’ 𝑗 𝐽𝑑 π‘˜ π½π‘ž π‘˜

𝑄 π‘ž=1 𝑇 𝑑=1 𝐸 𝑒=1

+ πœπ‘—π‘˜ πœπ‘—π‘˜ ∼ 𝑂(0, 𝜐2 + π‘‘π‘—π‘˜

2

  • utcome domain
  • employment strategy
  • population characteristic
slide-19
SLIDE 19

19 19

Meta-Regression Model: Interaction Terms

π‘§π‘—π‘˜ = 𝛽 + π‘π‘˜ + 𝑐𝑒 𝑗 + 𝑑𝑑𝐽𝑑 π‘˜

𝑇 𝑑=1

+ π‘•π‘žπ½π‘ž π‘˜

𝑄 π‘ž=1

+ 𝑔

𝑑𝑒𝐽𝑑 π‘˜ 𝐽𝑒 𝑗 𝐸 𝑒=1 𝑇 𝑑=1

+ β„Žπ‘žπ‘’π½π‘ž π‘˜ 𝐽𝑒 𝑗

𝐸 𝑒=1 𝑄 π‘ž=1

+ π‘šπ‘‘π‘žπ½π‘‘ π‘˜ π½π‘ž π‘˜

𝑄 π‘ž=1 𝑇 𝑑=1

+ π‘›π‘’π‘‘π‘žπ½π‘’ 𝑗 𝐽𝑑 π‘˜ π½π‘ž π‘˜

𝑄 π‘ž=1 𝑇 𝑑=1 𝐸 𝑒=1

+ πœπ‘—π‘˜ πœπ‘—π‘˜ ∼ 𝑂(0, 𝜐2 + π‘‘π‘—π‘˜

2

  • strategy by domain
  • target population by domain
  • strategy by target population
  • strategy by target population by domain
slide-20
SLIDE 20

20 20

Results

slide-21
SLIDE 21

21 21

Intervention Impacts

slide-22
SLIDE 22

22 22

Strategy Impacts

Strategy Any improvement (%) Improvement

  • f 5% or

more (%) Improvement

  • f 10% or

more (%) Financial incentives and sanctions 93.02 1.40 0.01 Education 92.77 0.69 0.00 Work experience 92.59 1.20 0.00 Training 92.19 0.73 0.00 Work readiness activities 89.63 0.25 0.00 Job development 88.73 0.41 0.00 Case management 88.33 0.33 0.00 Health services 88.13 0.64 0.00 Employment and retention services 81.59 0.18 0.00 Supportive services 81.05 0.05 0.00

slide-23
SLIDE 23

23 23

Strategy-by-Domain Impacts

slide-24
SLIDE 24

24 24

Questions?

slide-25
SLIDE 25

25 25

For More Information

  • Lauren Vollmer

– lvollmer@mathematica-mpr.com

  • Emily Sama-Miller

– esamamiller@mathematica-mpr.com

  • Alyssa Maccarone

– amaccarone@mathematica-mpr.com https://employmentstrategies.acf.hhs.gov/

slide-26
SLIDE 26

26 26

Appendix: Meta-Regression Priors

Main Effects Interaction Terms Hyperpriors and Variance Components 𝛽 ∼ 𝑂 0, 10 𝑔

𝑑𝑒 ∼ 𝑂 0, 𝜏 𝑔 2

πœˆπ‘‘ ∼ 𝑂 0, 1 π‘π‘˜ ∼ 𝑂 0, πœπ‘

2

β„Žπ‘žπ‘’ ∼ 𝑂 0, πœβ„Ž

2

πœˆπ‘• ∼ 𝑂(0, 1) 𝑐𝑒 𝑗 ∼ 𝑂 0, πœπ‘

2

π‘šπ‘‘π‘ž ∼ 𝑂 πœˆπ‘š, πœπ‘š

2

πœˆπ‘š ∼ 𝑂(0, 1) 𝑑𝑑 ∼ 𝑂 πœˆπ‘‘, πœπ‘‘

2

π‘›π‘’π‘‘π‘ž ∼ 𝑂 0, πœπ‘›

2

𝜐 ∼ halfβˆ’π‘‚(0, 2.5) π‘•π‘ž ∼ 𝑂 πœˆπ‘•, πœπ‘•

2

πœπ‘¦ ∼ halfβˆ’π‘‚(0, 𝜚2) 𝜚 ∼ Unif(0, 5)