NATIONAL OUTCOMES EVALUATION : FINDINGS FROM THE SMSS AND MIS, AND - - PowerPoint PPT Presentation

The Garrett Lee Smith (GLS) Suicide Prevention National Outcomes Evaluation is supported through contract no. HHSS283201200007I/HHSS28342002T (reference no. 283-12-0702) awarded to ICF International by the Center for Mental Health Services (CMHS), Substance Abuse and Mental Health Services Administration (SAMHSA), US Department of Health and Human Services (HHS).

NATIONAL OUTCOMES EVALUATION : FINDINGS FROM THE SMSS AND MIS, AND PREPARING FOR THE STUDENT BEHAVIORAL HEALTH FORM (SBHF)

Megan Brooks, MA

May 2, 2016

2

• What we have learned from the National Outcomes

Evaluation and prior campus evaluation efforts

• SMSS background and results
• Management Information System (MIS) results
• Moving forward with the Student Behavioral Health

Form (SBHF)

• Youth Exploratory Services Interview (YESI)

ON TODAY’S AGENDA

SHORT MESSAGE SERVICE SURVEY

ANALYSES CONDUCTED BY HOPE SOMMERFELDT, MA

4

WHAT IS THE SHORT MESSAGE SERVICE SURVEY (SMSS)?

• Text message (SMS) survey to students at GLS funded

campuses

• Gather data about exposure to on-campus suicide

prevention interventions, suicidal ideation, and past suicide attempts

5

BENEFITS OF SMS FOR DATA COLLECTION

• 98% of Americans aged 18 – 29 have a cell phone1
• 2014 Harris Poll shows that 83% of college students

regularly use a Smart Phone.2

• 97% of persons ages 18 – 29 send or receive text

messages3

• Students can respond any time and place
• Can start at one time and finish later
• Private (if they secure their phone)

1. Pew Research Center’s Internet Survey Project., January 9 – 12, 2014. From: http://www.pewinternet.org/files/2014/02/PIP_25th-anniversary-of-the- Web_0227141.pdf 2. Harris Poll Pearson Student Mobile Device Survey, February 13 – March 12, 2014. From http://www.pearsoned.com/wp-content/uploads/Pearson-HE-Student- Mobile-Device-Survey-PUBLIC-Report-051614.pdf 3. Pew Research Center’s Internet & American Life Project Spring Tracking Survey, April 17 – May 19, 2013. From http://www.pewinternet.org/2013/09/19/additional-demographic-analysis/

6

SMSS RECRUITMENT METHODS

Intercept Sample of currently enrolled students were stopped, by university or evaluation staff in- person to text in to the survey (University Staff

• r Evaluation Team)

School Listserv Currently enrolled students were invited to provide a phone number through a school sponsored listserv posting (University Staff) Student Email addresses An email was sent to students from currently enrolled on campus asking them to provide a phone number (Evaluation Team) Student Cell Phone Number Random sample of phone numbers for currently enrolled students received a text message invitation to participate (Evaluation Team)

7

SMSS DATA COLLECTION

• During the past 12 months, did

you attend class at [CAMPUS]?

• During the past 12 months did

you see, read, or hear of any suicide prevention effort at [CAMPUS]?

• During the past 12 months, did

you ever seriously consider attempting suicide?

• During the past 12 months, how

many times did you actually attempt suicide?

8

SMSS DATA COLLECTION – INCREASING RESPONSE RATE

• Previous data collection through a ‘cold’ email had very

low response

• $5 gift code incentive to complete the survey
• Incentive delivered right to their phones via text at the

end of the survey

• A reminder after 60 minutes of

idle time once the participant begins responding to questions

9

CRISIS PROTOCOL

ICF monitors Responses that indicate risk Student requests HELP via text ICF alerts local crisis center ICF communicates with center about the student of concern Follow up Trained professionals follow up with the student

• Warning text prior to suicidal ideation questions
• Option to skip suicidal ideation questions is clear
• The National Suicide Prevention Lifeline number given
• 24 hour monitoring for crisis situations

10

EXPOSURE, IDEATION, AND ATTEMPTS (WAVE 1)

11

SMSS DATA COLLECTION (WAVE 2)

• Data collection began in February 2016 and is
• ngoing (goal of 14 Cohort 7 Campuses that

completed wave 1)

• To date, responses from 454 students at 6 GLS

funded campuses

• Between 18 and 112 responses per campus

(mean = 81, median =87.5)

12

SNEAK PEAK AT FUTURE ANALYSIS WAVE 1 AND WAVE 2 *

65.7% 10.2% 3.3% 81.7% 12.8% 2.4% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Exposure Ideation Attempt Wave 1 Wave 2

* Includes only grantees that have completed both waves. Wave 1 N = 413; Wave 2 N = 454

13

POSSIBLE FUTURE SMSS ANALYSIS

• Tests of significance between Wave 1 and Wave 2
• Inclusions of prevention strategies to see if different

strategies had an impact on exposure, ideation, and attempt

USE OF STUDENT MENTAL HEALTH SERVICES ON GLS CAMPUSES

ANALYSES CONDUCTED BY YE XU, MA, MS

15

RESEARCH QUESTIONS

• Is there an increase in mental health service use after receipt
• f GLS award?
• Is the rate of mental health service use associated with the

number of students, faculty, and/or staff trained on suicide prevention within the campus community?

• Is the rate of mental health service use associated with the

strategies implemented by campus grantees?

16

DATA SOURCES AND MEASURES

• Management Information System (MIS)

– Number of students using mental health services, per year

• Training Activity Summary Page (TASP)

– Number of individuals trained by role (student, staff, faculty), per year

• Prevention Strategies Inventory (PSI)

– Implementation of prevention strategies, in particular Screenings and Public Awareness Campaigns

• Secondary Data Sources

– Integrated Postsecondary Education Data System (IPEDS) for student enrollment, number

• f faculty and staff

– Carnegie Classification of Institutions of Higher Education (campus size and setting categories)

17

DESCRIPTION OF MODEL

Students, Faculty, Staff Trained Implementation of Screening or Public Awareness Campaign (PAC)

Mental health service use

18

SAMPLE

• Panel data on 84 GLS campuses from cohorts 5 - 8

– Cohort 5 campuses initially funded in year 2011 – Cohort 6 campuses initially funded in year 2012 – Cohort 7 campuses initially funded in year 2013 – Cohort 8 campuses initially funded in year 2014

• On average we have data for 3 years preceding the

initial funding year and 2 years following that initial funding year

19

METHOD

• Outcome of interest

– Student mental health service use, reported annually by GLS grantees using MIS abstraction

• Predictors

– Year of GLS award – Pre and post GLS award – Types of gatekeepers trained on suicide prevention, and supplemented by data from the Integrated Postsecondary Education Data System (IPEDS) – Implementation of Screening and Public Awareness Campaign on campus

20

DESCRIPTIVE

N Mean SD

Total number of students registered on campus (MIS)

466 17,264 12,425

Average number of faculty on campus (IPEDS)

466 2,115 2,458

Average number of staff on campus (IPEDS)

466 2,385 3,041

Average number of students trained (TASP/TES)

459 106 323

Average number of faculty trained (TASP/TES)

452 7 22

Average number of staff trained (TASP/TES)

457 22 54

Average number of students receiving mental health service on campus (MIS)

466 1,081 1,107

21

DESCRIPTIVE

Pre-GLS award Post-GLS award Average Training and Mental health rate on campus Four years prior to funding year Three years prior to funding year Two years prior to funding year One year prior to funding year Funding year Funding year1 Funding year2 Average percent of students trained on campus 0.7% 4.3% 3.4% Average percent of faculty trained on campus 2.2% 6.3% 1.8% Average percent of staff trained on campus 4.4% 7.6% 5.2% Average percent of students receiving mental health services

• n campus

5.9% 6.0% 6.2% 6.4% 6.9% 7.5% 7.9%

22

DESCRIPTIVE

Pre-GLS award Post-GLS award Average percent of students trained on campus

• 2.5% (N=202)

Average percent of faculty trained on campus

• 3.4% (N=202)

Average percent of staff trained

• n campus
• 5.6% (N=202)

Average percent of students receiving mental health services

• n campus

6.1% (N=264) 7.3% (n=202)

23

DESCRIPTIVE

Number of campuses % of Campuses with PAC % of Campuses with Screening Cohort 5 (n=19) 73.7% 78.9% Cohort 6 (n=32) 65.6% 62.5% Cohort 7 (n=19) 68.4% 57.9% Cohort 8 (n=14) 35.7% 71.4% Total (n=84) 63.1% 66.7%

24

DESCRIPTIVE

Average percent of students receiving mental health services on campus Campuses without PAC Campuses with PAC 5.5% 7.3%

25

DESCRIPTIVE

Average percent of students receiving mental health services on campus Campuses without Screening Campuses with Screening 6.5% 6.7%

26

DESCRIPTIVE

Number of students Small campuses Medium campuses Large campuses Pre- GLS award Post-GLS award Pre-GLS award Post- GLS award Pre-GLS award Post- GLS award (n=24) (n=25) (n=85) (n=73) (n=155) (n=104) Average percent of students receiving mental health services on campus 8.9% 12.2% 5.8% 6.6% 5.9% 6.7%

27

DESCRIPTIVE

Number of trainees Small campuses (n=25) Medium campuses (n=73) Large campuses (n=104) Average percent of students trained on campus 5.9% 3.2% 1.3% Average percent of faculty trained on campus 16.9% 2.2% 1.4% Average percent of staff trained on campus 14.2% 5.5% 3.8%

28

ANALYSIS

Mixed model

• Fixed effect

 Year of GLS award  Pre and post GLS award  Implementing PAC and Screening activities on campus (time- invariant)  Percent of student, faculty and staff trained on campus (time- varying)

• Random effect

 Intercept  Year of GLS award

29

RESULTS

• Significant increase in student mental health

service use rates over time (p=0.027)

• Campuses that have more staff trained in

GLS show greater increases in student mental health service use rate (p=0.013)

• Campuses that have more students trained in

GLS show greater increases in student mental health service use rate (p=0.066)

30

RESULTS

Effect Estimate Standard Error DF tValue Pr > |t| Intercept 0.07481 0.00621 81 12.05 <.0001 Years of GLS award 0.00173 0.00077 83 2.26 0.0267 Pre and Post GLS award indicator 0.00465 0.00279 277

• 1.66

0.0973 Implementing PAC 0.02173 0.01008 277

• 2.15

0.0321 Implementing Screening 0.00054 0.01031 277

• 0.05

0.9584 GLS * Percent of student trained on campus 0.03389 0.01838 277 1.84 0.0662 GLS * Percent of faculty trained on campus

• 0.02363

0.01236 277

• 1.91

0.057 GLS * Percent of staff trained

• n campus

0.04553 0.01812 277 2.51 0.0126 GLS * Implementing PAC 0.004652 0.0038 277 1.22 0.2219 GLS * Implementing Screening 0.003756 0.00382 277 0.98 0.3268

31

RESULTS

7.48% 7.94%

0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20%

Pre-GLS award Post-GLS award

Percent of Students Using Mental Health Services

• n Campus

Out of the total enrollment on GLS-funded campuses included in this analysis (N=1,382,556), an additional 0.46% of students received mental health services following GLS award (p=0.097)

32

RESULTS

7.48% 7.82% 7.94%

7.2% 7.3% 7.4% 7.5% 7.6% 7.7% 7.8% 7.9% 8.0%

Pre-GLS 10% students trained 10% staff trained

Percent of Student Using Mental Health Services on Campus

33

AD-HOC: MODEL RESULT BY SIZE OF CAMPUS

Size of Campus Small Campuses Medium Campuses Large Campuses Effect

Estimate p Estimate p Estimate p

GLS * Percent of students trained on campus 0.070 0.001

• 0.014

0.823

• 0.099

0.051 GLS * Percent of faculty trained on campus

• 0.056

0.002

• 0.010

0.863 0.020 0.498 GLS * Percent of staff trained

• n campus

0.102 0.001 0.042 0.333 0.026 0.265

34

DISCUSSION

• There is evidence supporting the increase in mental

health service use over years, however, no strong increase after the receipt of GLS award.

• The increase in mental health service use appears

to be related to the number of staff trained on suicide prevention on campus.

• Similarly, the number of students trained was

marginally associated with the increase in mental health service use.

35

DISCUSSION

• Staff are potentially more knowledgeable about the referral

process and how to navigate university policies and procedures regarding mental health services for students.

• Staff may be more likely come into contact with a broader

spectrum of students and therefore are more likely to encounter youth who are in need of mental health services than student gatekeepers.

• Staff may be more likely to serve in traditional gatekeeper

roles (e.g., student counseling or advisor), thus leading to the significant effects for staff rather than faculty

36

DISCUSSION

• It seems that the relationship between training and

student mental health use is likely driven by small campuses

• Small campuses have higher training rates of

students, faculty and staff, and therefore are potentially more likely to reach “saturation” than medium and large campus

37

DISCUSSION

• Future exploration will investigate “saturation”

threshold training rates across campuses

MOVING FORWARD WITH THE SBHF

39

PURPOSE OF SBHF

• Includes collection of information on the implementation
• f Suicide Safer Environment care practices from

campus health care providers as well as the annual number of suicide attempts and deaths as a source of long-term outcome data

40

SBHF IMPLEMENTATION AND LOGISTICS

• All campus grantees will participate in the SBHF in early summer
• The SBHF is a web-based form on the SPDC
• The SBHF administrator will receive a password to access the form
• Data in the SBHF will be reported for the current academic year (approx.

September 2015- May 2016) and the FOUR academic years prior to the grant

• At the end of the grant, you will have data for a total of 7 academic years

41

SBHF CONTENT

• The instrument includes a mix of quantitative questions (e.g., how

many suicide attempts occurred on campus?) and policy related questions (e.g., after a suicide attempt, what happens?).

42

SBHF DATA ELEMENTS

• Student use of behavioral health services (including counts
• f specific services)
• Campus use of standardized screenings (and screening tool)
• Linkages to community providers
• Follow-up support
• Student suicide attempts
• Student deaths by suicide

43

SBHF TIMELINE

Identify a SBHF Administrator to compile and enter data Determine sources of information for data elements– this may require coordination with multiple on and

• ff-campus sources

SBHF Administrator will receive an email with a password for the web- based survey The Administrator will have several weeks to complete the data entry for the current year and the four years prior to the grant

Now Spring 2017- Spring 2018

TAL will confirm SBHF Administrator Administrator will complete data for academic year An email will be sent to the project staff to identify the SBHF administrator; please respond!

Next few weeks Approximately June

44

QUESTIONS ON SBHF?

YOUTH EXPLORATORY SERVICES INTERVIEW (YESI)—CAMPUS VERSION

46

PURPOSE OF YESI

• The YESI—Campus Version includes information on the

Continuity of Care experiences of students identified as at risk for suicide by GLS trained campus gatekeepers. The protocol is used to understand information about the (1) identification (gatekeeper) and referral process, (2) perceptions of the identification, (3) follow-up care receipt or barriers to receipt (4) current emotional and

• verall health, and (5) participant demographics.

47

YESI IMPLEMENTATION AND LOGISTICS

• Campus grantees will be solicited by the Campus TAL and the YESI Study Lead to

determine interest in participation

• ICF will work with selected grantees (4-5 campuses) to secure local level

approvals and train local campus counselor center staff on YESI protocols (May 2016 – September 2016)

• Nine YESI interviews will be conducted from October 2016 – May 2017 by ICF

(via trained crisis counselors)

• YESI data will be analyzed, synthesized, and reported to SAMHSA in aggregate

48

YESI CONTENT

• The instrument includes a mix of quantitative (e.g., how many times

have you been seen by campus counseling center services?) and qualitative related questions (e.g., thinking back on that identification by the campus gatekeeper, how, if at all, would you have changed that experience, process or discussion?).

49

YESI NEXT STEPS

• We’ll be focusing first on selecting grantees to participate
• Need to have a campus counseling center and conduct gatekeeper

trainings

• Grantee and broader school needs to be interested/willing to

participate

• To begin this process, your TAL, Jessica Wolff will send a broad

communication about the YESI to all campuses

• The YESI lead may also reach out to specific campuses based on our

reviews of grantee programs

• If you are interested, or would like to hear more information, talk to me
• r Christine Walrath here at the meeting, or reach out to Jessica Wolff

directly