EVALUATION How we do it in Texas! Lou Ann Grossberg University of - - PowerPoint PPT Presentation

How we do it in Texas!

Lou Ann Grossberg University of Texas at Austin September 14, 2017

EVALUATION

www.uttobacco.org

Learning Objectives:

• Discuss elements of a comprehensive tobacco prevention

and control program

• Describe the elements of the Strategic Prevention

Framework

• Differentiate outcome versus process evaluation

approaches

• Describe the sources, caveats and limitations of outcome

evaluation data

• Describe the purpose and components of process

evaluations

• Discuss the importance of analyzing the practical

implications of program evaluation

“Evaluation is a systematic process to determine merit, worth, value or significance.”

Definition:

Mass-Reach Health Communication Interventions Infrastructure, Administration, and Management Cessation Interventions State and Community Interventions Surveillance and Evaluation Comprehensive Tobacco Control Programs

Texas Department of State Health Services

Tobacco Prevention & Control Coalition Program

Texas Department of State Health Services

Tobacco Prevention & Control Coalition Program

2008-2013 TPCCs Current TPCCs Funding Average $312,394 Range $126,000 - $752,434 Population Average 198,318 Range 12,455 - 842,304

TPCC EVALUATION Local Evaluators Texas DSHS Tobacco Program Epidemiological Workgroup UT Tobacco Research & Evaluation Team Local Evaluators UT Tobacco Research & Evaluation Team TPCC EVALUATION Local Evaluators UT Tobacco Research & Evaluation Team TPCC EVALUATION Local Evaluators UT Tobacco Research & Evaluation Team Epidemiological Workgroup TPCC EVALUATION UT Tobacco Research & Evaluation Team Epidemiological Workgroup

TPCC EVALUATION

UT Tobacco Research & Evaluation Team

Tobacco Prevention & Control Coalition Program University of Texas at Austin

Lead the cross-community evaluation

• Quasi-experimental design that employs both qualitative and

quantitative methods

• Compare findings from the nine sites to those of the rest of

Texas

• Assure that common measures are collected across the nine

sites (convene Statewide Epidemiology Group)

• Document and assess activities, accomplishments,

barriers/challenges, and changes at the state and local level

Tobacco Prevention & Control Coalition Program

Evaluation Questions based on Statewide Goals

• Prevent tobacco use among young people
• Ensure compliance with state and local tobacco laws with

adequate enforcement

• Increase cessation among young people and adults
• Eliminate exposure to secondhand smoke
• Reduce tobacco use among populations with the highest

burden of tobacco related health disparities

• Maintain state and community capacity for

comprehensive tobacco prevention and control

Tobacco Prevention & Control Coalition Program

Tobacco Prevention & Control Coalition Program

Cross-community Evaluation

• Outcome/effectiveness evaluation:

Measures program effects in the target population by assessing the progress in the outcomes or

• utcome objectives that the program is to achieve.
• Process/implementation evaluation:

Determines whether program activities have been implemented as intended, and promotes use of a continuing quality improvement process to achieve

• utcomes.

Outcome Evaluation

• Texas Youth Tobacco Survey
• Tobacco Enforcement Data
• Texas Behavioral Risk Factor Surveillance System
• Texas Quitline
• Municipal Smoke-Free Ordinances
• End-User Survey Data

877.YES.QUIT YESQUIT.ORG

Outcome Evaluation

Texas Youth Tobacco Survey (YTS) Description:

• The YTS is conducted every year in Independent

School Districts (ISDs) in participating TPCC counties, and every other year (even-numbered years) in the rest of Texas to assess tobacco use patterns among middle and high school students Limitations:

• The YTS is only conducted every other year for the

state

• Not all school districts and not all TPCC counties

participate in the YTS

Outcome Evaluation

Tobacco Enforcement Data Description:

• The number of local enforcement grantees, the number of

controlled buys/stings conducted, and the violation rate in the TPCC areas combined and in non-TPCC areas using data from the Comptroller’s Enforcement Grantee Program (FY08-FY12) and the DSHS Tobacco Enforcement Program (FY12 and FY13). Limitations:

• Funding for enforcement/minor access was terminated

for this grant cohort

Description:

• The BRFSS is administered through the DSHS Center for Health

Statistics

• The BRFSS is a federally funded telephone survey of randomly selected

adult Texans to collect data on lifestyle risk factors, including tobacco use, contributing to the leading causes of death and chronic diseases

• DSHS Tobacco Prevention and Control Branch funded an oversample
• f TPCC counties

Limitations:

• The BRFSS is self-reported, only administered in English and Spanish,

and is a telephone survey

• The BRFSS timeline does not coincide with our timeline (Jan.-Dec.)
• The BRFSS data are really meant to be analyzed at the state level, not at

the local community level

Outcome Evaluation

Texas Behavioral Risk Factor Surveillance System (BRFSS)

Outcome Evaluation

Texas Quitline Description:

• The Texas Quitline is funded by the DSHS. Callers

receive free and confidential counseling services, support and information from trained professionals Limitations:

• The Quitline data only represents registered

participants, no follow-up data is available (i.e. quit rates)

877.YES.QUIT YESQUIT.ORG

Outcome Evaluation

Municipal Smoke-Free Ordinances

Description:

• An interactive website that identifies cities in Texas that have
• rdinances restricting second-hand smoke exposure, analyzes

and rates (1-5) the coverage provided by the second hand smoke ordinances, tracks changes over time, and provides easily accessible information for users Limitations:

• There is a lag between when an ordinance is passed and when

it is entered in the database

• Only includes incorporated municipalities with populations

greater than 5,000

Outcome Evaluation

End User Survey (EUS)

Description:

• Survey administered locally that provides a snapshot of

tobacco use and cessation measures

• Convenience sample (time location sampling)
• Local priority populations (low SES, veterans, etc.)
• Federally Qualified Health Centers, Community Health

Centers & local clinics

Limitations:

• The EUS is based on a convenience

sample of a local priority population and should not be used to make generalizations

• Results of the EUS should be

triangulated with other sources of current and future data

End User Survey (EUS)

Purpose

• To fill the gap in knowledge about tobacco-

related needs in local priority populations

• To guide and inform future

assessment/evaluation efforts in priority populations

• To identify and examine areas of need for future

coalition funding to carry out tobacco interventions with priority populations

End User Survey (EUS)

A convenience sample of priority populations:

• Tobacco use
• Awareness of cessation resources
• Intention to quit and prior quit attempts
• Acceptability of recommended cessation resources
• Cessation assistance from health professionals in past

year

• Exposure to and impact of mass media campaigns
• Secondhand smoke exposure

Process/Implementation Evaluation

The cross-community process evaluation serves three purposes:

• Documents implementation of comprehensive

strategies across the TPCC sites

• Demonstrates accountability on the use of public

funds

• Promotes use of a continuing quality improvement

process to achieve outcomes

Cross-Community Process Evaluation

• 1. To what extent did sites follow

the planning and implementation process in the SPF?

• 2. To what extent did site use

local data and how was it used?

• 3. To what extent did sites plan

for and implement strategies to sustain the work of the coalition?

Cross-Community Process Evaluation

• 1. To what extent did sites follow the planning and

implementation process in the SPF?

• Document completion of the SPF assessment,

planning and implementation steps  action plans by goals  annual needs assessments  local evaluation reports

• Document barriers, successes, and outcomes not

measured in the outcome evaluation or DSHS performance monitoring system  success stories  monthly conference calls  meeting minutes

Cross-Community Process Evaluation

• 2. To what extent did sites use local data and how

was it used?

• Ensure implementation of local evaluation study

 complete 200 End User Surveys per county  local evaluation study proposal  Present local evaluation results with coalition

• Document presentation of EUS results to coalition

 share EUS findings to coalition

• Document sources of local data that the coalitions

use to make decisions  participation in monthly conference calls

Cross-Community Process Evaluation

• 3. To what extent did sites plan for and implement

strategies to sustain the work of the coalition?

• Promote the work of the coalition beyond the initial

funding period  Coalition Sustainability Checklist

• Guide actions to engage priority populations –

inclusivity, mission and group process  Coalition Member Survey and Summary Report

Youth Tobacco Use

Past 30-day Use of Cigarettes (MS and HS combined)

11.7 N/A 8.1 7 10 8.8

2 4 6 8 10 12 14

2014 2015 2016 Past Month Use (%) Texas TPCC-Combined

Compliance with state and local tobacco laws with adequate enforcement

Current Smoker Adults Ages 18 years and Over

Adult Tobacco Use

17.7 14.5 15.2 14.3 18.2 14.2 15 14.8

2 4 6 8 10 12 14 16 18 20

2011-2013 2014 2015 2016 Current Smoker (%) State of Texas TPCC - All

Increase Cessation Among Young People and Adults

Texas Quitline Calls Rates per 100,000 population

63.52 60.08 62.44 53.49 51.64 67.57

10 20 30 40 50 60 70 80

2014 2015 2016 Rate per 100,000 population Rest of Texas TPCCs Combined

Eliminate exposure to secondhand smoke

3% of Texas cities covered by local comprehensive smoke- free laws

King and colleagues, 2015

43% 49% of Texas municipal population covered by comprehensive smoke- free policies

Winnike and colleagues, 2016

Lessons Learned

• Set expectations early in the grant term for cross-

community evaluation standards and for drilling down in the local data to fill the data gaps and to support coalition strategies

• Communication and coordination between the coalition

and the data collectors is imperative. Reliable tobacco use prevalence and patterns at the local level identify priority populations and inform the selection of most impactful program activities

• Clearly define the role of the local evaluator. An

independent outside evaluator can hold the coalition accountable and provide support for strong data-driven decision making

• Remain flexible!

Questions?

Lou Ann Grossberg University of Texas at Austin 512.232.4166 louann.grossberg@austin.utexas.edu www.uttobacco.org