Using Open-Ended Needs Assessment Data for Improvement Charlie - - PowerPoint PPT Presentation

Using Open-Ended Needs Assessment Data for Improvement

Charlie Goss, Analytics Specialist Office of Institutional Effectiveness: Liberty University csgoss@liberty.edu

1

Participants will be able to…

• Develop an open-ended needs assessment for the

purpose of gathering actionable feedback.

• Analyze the responses using an established

method.

• Package the analysis results in a way that facilitates

end-user review as well as development of improvements.

2

Session Outline

• Development of Needs Assessment
• Survey Launch
• Content Analysis
• Dissemination and Improvements
• Activity

3

What is a needs assessment?

• Sample definition:

– “a systematic, rational process for collecting and analyzing information to describe the needs of a population”1

• Often done in education, health care, and counseling
• Examining the gap between “is” and “ought to be”
• Eliciting current and future needs
• Quantitative or open-ended
• A robust indirect measure; no rubrics involved

4

1-New York State OASAS, “OASAS Prevention Program – Needs Assessment” (October 2015), https://www.oasas.ny.gov/prevention/needs/needsassessment.cfm

Liberty’s Needs Assessment

• Goal: obtain list of needs to be addressed and facilitate

improvement plans

• Ask students directly what improvements they want
• No “positive” information sought; only opportunities
• Specific feedback not confined to preformatted options
• Collection of stated needs (known and new)
• Built to be acted upon, not just to show adequacy and

satisfaction

5

Open-Ended Surveys

Pros:

• Rich, nuanced data
• Respondent can be more

specific than selecting standard response options like “Strongly Agree”

• May get unexpected

answers and gain new insights

• Reading the review vs. just

looking at the number of stars Cons:

• Is a labor-intensive process,

not fully automated

• Takes time to analyze
• Can be difficult to organize

into themes (meaningful chunks)

• Must separate the wheat

from the chaff (lots of useless responses)

6

Building the Needs Assessment

• 13 questions developed; each student received 1.

– List up to three services you would like to see added to the university. – List up to three problems that you most frequently encounter in your courses.

• Question types: Problems (4), Facilities (2), Services (3),

Student Development (2), Future Needs (1), Skills (1)

• Branded as a one-question survey to increase response

rate

• Sent to all Fall 2014 residential and online students

(about 1,000-5,000 per question)

7

Backstage Survey Settings

• Number of characters limited to 50

– Good: “not knowing what kind of notes to take” (38) – Too Long: “I think students should be allowed to have one skip per week. Also, there should be fewer limits on dining

• ptions on campus. I am always starving in between classes.”

(165)

• Three separate response fields; only one required

– Mary Smith response 1: “Internet is slow” – Mary Smith response 2: “parking is limited” – Mary Smith response 3: “I want more dining options”

• Easier analysis, one theme per response

8

Timeline and Response Rate

Timeline

• Launched 9/15/14 (Monday)
• Closed 9/23/14 (Tuesday)
• Elapsed time: 9 days

Response Rate Resident students 12.6% (1,721 / 13,611) Online students 10.2% (3,893 / 38,350) TOTAL 10.8% (5,614 / 51,961)

9

Response Metadata

• Total Responses: 13,811
• Responses per Question: between 300 and 1,600
• Average Response Length: 32 characters

– E.g., “getting in touch with an advisor”

• Average Responses per Student: 2.7 out of 3

10

Overview of Analysis Process

• 1. All 13 response sets split by delivery mode into 23 files
• 2. Two reviewers for increased reliability and efficiency
• 3. Incorporating themes across all files
• 4. Stacking of all 23 files with responses and themes
• 5. Sub-Themes developed
• 6. Results used for residential and online action plans

Total Time: 3 weeks at 8 man-hours per day

11

Software Toolkit

12

SPSS

*Readying the raw data to be analyzed *Stacking all themes together at the end

Excel

*Used for detailed reading of all responses *Themes manually assigned *Final spreadsheet

Text Analytics

*First-pass automated word search

Transposing Raw Data

• Stud. ID Student’s First Response

Student’s Second Response Student’s Third Response 05 Finding work study

• pportunities.

Post office accessibility Nothing else really. 06 Not being allowed to drive my car on campus The bus system making me late for classes Strict curfew hours

• Stud. ID Resp. Num. Responses

05 1 Finding work study opportunities. 05 2 Post office accessibility 05 3 Nothing else really. 06 1 Not being allowed to drive my car on campus 06 2 The bus system making me late for classes 06 3 Strict curfew hours One row per response allowed each response to receive its own theme later.

13

Content Analysis

• Perusing all responses, searching for common themes
• Themes=repeating needs and ideas grouped together

into an organized whole (e.g., Advising or Financial Aid)

• Fake examples below:

14

Response Theme More spaces in parking garage Parking Docs Diner has a lot of people Dining Dorm internet speed is lacking Internet

Text Analytics: Themes and Keywords

Advising - Text Match: adviser advise advising advisor Blackboard - Text Match: blackboard black board bb Books - Text Match: text text book mbs book Course Assignments - Text Match: discussion board reading quiz Course Resources - Text Match: course schedule course chart syllabus study guide course preparation Any responses with a text match were assigned to the corresponding theme. False hits were okay at this stage.

15

Individual Responses First Theme Final Theme ASIST-everything about it IT streamline the portal login too many list/links IT ipad compatibility with liberty and blackboard Blackboard Phones / Devices Online outreach Communication I am an online student so cannot think of anything Irrelevant Automated calling. Too many calls for same thing Fewer Outreaches to Student Constant FA emails with same information Fin Aid Matters Finance Opening a program for Architecture. More Courses / Programs Tutoring for online students its there but where? Tutoring Tutoring everything else is great! Irrelevant Online advising. Advising Advising Mobile blackboard Blackboard Phones / Devices Online interaction and timeliness More Community Connecting with Others Text selection Books Bookstore

First-pass word search (automatic) Detailed look and theme selection (manual)

Analysis Sample

16

Two-Reviewer Theme Consolidation

Example Tasks Reviewer 1 Theme (Residential) Reviewer 2 Theme (Online) Final Theme in Both Response Sets Convert similar themes to the same theme Quicker Service Student Services Quicker Service Assign new themes

• ne reviewer may

have overseen Answers to Tests Answers to Tests Note specific themes for each delivery mode Parking Parking (residential

• nly)

17

Forward-Theme Assignment

• Applied themes forward that had been assigned in

earlier analyses to ensure they were not forgotten later

• Ensured consistency of themes across all 23 analyses

18

Q1 Res Financial Aid Parking Dormitories Faculty Academics Q7 Res Financial Aid Parking Dormitories Faculty Q7 Res Financial Aid Parking Dormitories Faculty Academics

Reverse-Theme Assignment

• Assigned themes found in later analyses to earlier

response sets

• Ensured consistency of themes across all 23 analyses

19

Q1 Res Finance Parking Books Faculty Q7 Res Financial Aid Parking Bookstore Books Faculty Q1 Res Financial Aid Parking Bookstore Books Faculty

Development of Sub-Themes

20

Task Before After Themes with many responses broken down into sub- themes Skills - Personal (299) Skills - Personal Ethics (7) Goal-Setting (6) Organizing Skills (20) Themes with few responses subsumed as sub- themes within new themes Student Housing (41) Student Affairs Student Housing (41) Student Care (18)

Stacking of Sub-Themes

• From 23 themed files to 1 large file

21

Q1 Res Bus Issues Parking Internet Facilities Q2 Res Dormitories Parking Exercise Athletics Q13 Res Tests

• Disc. Board

Website Bookstore

…

Q13 Onl Internet Textbooks

• Disc. Board
• Writ. Ctr.

Stacked File Athletics Bookstore Bus Issues

• Disc. Board

Dormitories Exercise Facilities Internet Parking Tests Textbooks Website

• Writ. Ctr.

Descriptive Statistics

• Minor cleanup of themes and sub-themes

– E.g., ‘Registar’ changed to ‘Registrar’ to match others

• Frequencies and crosstabs on themed data using SPSS
• Responses and themes overall, by delivery mode
• Some sub-themes not included in final spreadsheet

– Irrelevant (did not answer question sensibly – “n/a,” etc.) – Sensitive (mentioned specific professor) – Duplicate (submitted same answer two or three times)

22

Snapshot of Findings

• Themes: 32
• Sub-Themes: 232
• Pertinent Responses: 12,172
• Example of Theme and Sub-Themes:

23

Response Theme Sub-Theme Bus routes remaining the same Transportation Bus Issues Traffic being slow. Transportation Traffic Delays Stoplights at main intersections Transportation Transportation - Other

Developing a Final Spreadsheet

• New, anonymous Response ID developed for each

response

Q13id91rsp2 [Question Number + student case number + response number]

• From 13 questions, 6 question types created
• Glossary of themes developed
• Spreadsheet created, features added using Visual Basic

24

Final Spreadsheet

25

Searching the Results

• Search buttons and filters for easy searching

(created using macros and VBA within Excel)

• Instructions created for ease of use
• Demonstration of Spreadsheet

26

Final Package

• Fall 2014 Needs Assessment Spreadsheet
• Instructions for Using the Spreadsheet
• Needs Assessment Action Plan Template
• Needs Assessment Fall 2014 Executive Summary

27

Required Improvements

• Academic and co-curricular departments required to

make an improvement based on the data

• Template created, tying plans to the university

assessment model

• Departments could re-assess an area of concern during

spring Assessment Day

• Some areas sought more demographic information such

as age and commuter status

28

29

Example Improvement Plans

• How-To Videos

– One school is partnering with the library to provide how-to videos for students, including information on what resources are currently available and how to use those resources.

• Video Conferencing

– One school is adding video conferencing to its online courses in response to students expressing a desire for more real-time chat, correspondence, and connection with classmates and instructors.

• Improving Lighting

– The lighting on campus is being improved.

30

Looking Ahead

• Second iteration launched in Fall 2015
• Began analyzing responses while survey was still active
• Process allows searching for similar themes
• Must be careful when doing longitudinal reporting –

cannot say something has gotten better if a theme has fewer responses than last time

• Beginning a three-year cycle for needs assessment

31

Activity

Person A - Residential responses Person B - Online responses Steps: 1: Individual Theme Assignment 2: Final Theme Assigned by Pair How did you come to an agreement on your themes?

32

Facebook

• https://www.facebook.com/LibertyIE
• “Like” the Liberty University Institutional Effectiveness

Facebook page!

– Announcements – Tips and ideas – Photos – Web Resources

33