SUPPORT DATA COLLECTION AND ANALYSIS 1 WE ARE GLAD YOU ARE HERE! - - PowerPoint PPT Presentation

PRACTICAL STRATEGIES TO SUPPORT DATA COLLECTION AND ANALYSIS

1

WE ARE GLAD YOU ARE HERE!

ITACC: 2017 AIDD Technical Assistance Institute

2

LEARNING OBJECTIVES

• Participants will recognize basic concepts related to

data collection;

• Participants will understand data needed to report on

new performance measures;

• Participants will explore data collection activities and

strategies.

ITACC: 2017 AIDD Technical Assistance Institute

3

4 THINGS TO KNOW

• Know why you want to evaluate a project/activity.
• Process of evaluating progress towards goals
• Collect and Use Data
• Accountability for results
• Clarity of outcomes

ITACC: 2017 AIDD Technical Assistance Institute

4

Notes for Slide 4: Before collecting data – Know why you want to evaluate a project/activity. Logic model identifies desired outcomes, resources, and activities necessary to accomplish the

• utcomes.
• Process of evaluating progress towards goals:
• Align structures, processes, and routines to enable a methodical and routine evaluation

process between work and desired outcomes.

• Collect and Use Data
• Establish and implement routines and processes for collecting, analyzing, and

monitoring data

• Accountability for results
• Making decisions to continue, improve, or end practices based on data; communicating

results with internal and external stakeholders

• Clarity of outcomes
• Communicate priorities, set ambitious, clear, and measurable goals and outcomes with

aligned strategies and activities.

5

CONNECTIONS WITH STRUCTURES AND PROCESSES

Systematic approach to ensure quality and progress towards goals Aligning structures, processes, and routines Logic Model Evaluation Plan annual work plan – data collection

ITACC: 2017 AIDD Technical Assistance Institute

6

COLLECT AND USE DATA

The collection of data can inform continuous improvement, provide feedback, and help guide decisions.

ITACC: 2017 AIDD Technical Assistance Institute

7

Notes for Slide 6: Collect and Use Data Establish and implement routines and processes for collecting, analyzing, and monitoring data. There is a resource on the itacchelp.org website - http://itacchelp.org/wp-content/uploads/2015/12/2015-TAI-Evaluation-FAQ-3.pdf

8

ACCOUNTABILITY FOR RESULTS

• Decisions based on results
• Improve
• Adjust
• Terminate
• Incentives
• Additional or continuation funding
• Communicating results
• Internal and external stakeholders

ITACC: 2017 AIDD Technical Assistance Institute

9

Notes for Slide 7: Making decisions to continue, improve or end practices based on data; implementing incentives tied to performance; and engaging and communicating results with internal and external stakeholders.

10

CLARITY OF OUTCOMES

• Establish Priorities
• Communicate priorities
• Set ambitious, clear, measurable goals and outcomes

with aligned strategies and activities.

ITACC: 2017 AIDD Technical Assistance Institute

11

Notes for Slide 8: To ensure the Council is getting what they want from projects and activities, it is important to establish priorities for the activity and to communicate the expectations. Of course, setting clear, measurable goals and outcomes that align well with the strategies and activities will help

12

HAVE YOU EXPERIENCED THIS WITH GRANTEES?

ITACC: 2017 AIDD Technical Assistance Institute

13

Notes for Slide 9: When councils engage other entities to help accomplish goals and objectives – and changes are made to the initial plans, don’t forget to adjust the evaluation component as well – ITACC staff frequently hears about data being provided to Councils that does not speak to the overall

• utcomes of the funded project/activity – having a process in place that helps grantees understand

the data that is needed to measure impact is critical!

14

MEASURING IMPACTS AND RESULTS

ITACC: 2017 AIDD Technical Assistance Institute

15

Notes for Slide 10: Let’s take a look at the new performance measures – the new measures call for connection between investments, outputs, and the outcomes of the activity – this translates into results and short-term impact, intermediate impact, and ultimate impacts of Council activities

16

PM INFO DATA = IT’S NOT JUST NUMBERS

• Data = facts and statistics collected together for reference
• r analysis.
• Do you know what you need to know?
• Think Impact, Impact, Impact!!
• Lessons learned:
• Training for sub grantees is very important
• Councils need to take the lead
• Request more than you think you’ll need
• Continuous monitoring and follow up for clarification
• Self reporting and incomplete survey’s
• Use examples and tools from states who’ve been doing it

ITACC: 2017 AIDD Technical Assistance Institute

17

Notes from Slide 11: Data: Good data isn’t just a bunch of raw numbers – it’s a compilation of facts based on a clear understanding of WHAT you are tying to find out. The results should tell a story. Do you know what you need to know? These PM’s aren’t new. Although they keep changing slightly, the basic concept of what you need to collect isn’t. Make sure you really read and think through these PM’s so you know what you need to collect. Lessons Learned: Initial training is vital. Talk to the grantees about the new way we need to collect and report on data. Put the work in up front - look ahead at what you need to report in the PPR, then track back to figure how you will get that information from the grantees in a way that’s clear and concise. For us it was creating our own reporting templates which will get to in a min. Take the lead in telling sub-grantee’s what you need to know and how you want it reported. I tell the grantees if they question whether info is relevant – put it in and I will determine that because I’d rather have more than not enough. Continuous monitoring = allows you to make sure that you’re getting the data, and that it’s relevant to what you need to report in the PPR. Follow up with sub-grantees is very important. Make sure you get clarification. Looking at surveys = not everyone completes every question on a survey! You have to calculate percentages based on the number of people who answered a question – not based on the number

• f people who completed the survey!

18

CAN'T HAVE ONE WITHOUT THE OTHER

• Logic Model
• Work plan

State Plan

• Excel spreadsheets

Data Collection

• DD Suite
• Surveys
• Sub-grantee reports
• Impact Statements
• Personal Stories
• Sign in sheets
• Focus Groups
• Conversations with sub-grantees
• Narrative
• Performance Measures

PPR/IMPACT

How We Show Our Worth

ITACC: 2017 AIDD Technical Assistance Institute

19

THE OREO EFFECT

STATE PLAN DATA PPR OUR IMPACT

ITACC: 2017 AIDD Technical Assistance Institute

20

Notes from Slide 13: The Oreo effect. The filling is the data collection, it’s what holds the top and bottom wafers together. Without it, you’d just have chocolate wafers, which are ok, but it’s just not an Oreo.

21

LET’S LOOK AT SOME TOOLS

ITACC: 2017 AIDD Technical Assistance Institute

22

Notes from Slide 14: With change – Council’s have the opportunity to revisit the evaluation tools that will best meet the Councils needs…

23

SOUTH CAROLINA

• Quarterly Progress Reports
• Surveying Process
• Monitoring Visits
• Grantees can enhance minimum reporting

requirements to best-fit project/activity

• Ultimate question from Council staff is: “how are you

measuring the impact of your programming?”

ITACC: 2017 AIDD Technical Assistance Institute

24

Notes for Slide 15: Adapted AIDD performance measures into quarterly progress reporting and surveying processes with follow-up provided by monitoring visits. Program monitoring site visits complement the sub-grantee reporting process in addition to ensuring compliance and integrity to Terms and Conditions, relevant state and federal rules, and original grant Request for Proposal. While SC DD Council provides the minimum requirements for survey data collected, we encourage sub- grantees to modify our tools to best meet their program needs (surveys and other impact-gathering tools may be modified only—quarterly progress report is standard). Most common question from DD Council program staff to sub-grantees is “how are you measuring the impact of your programming?”

• Sub-grantees opting to modify our survey tools, must consult with programming staff to ensure

data being collected is relevant to all involved.

• Program monitoring is best when program staff can see project activities in action. Get to meet

people being impacted by activities and raise awareness of SC DD Council (we are often confused with our state’s DD service provider, DDSN)

• Most systems change projects include a committee or workgroup that DD Council is usually invited

to be a part of (not a requirement, though)

• Impact has a variety of uses and meanings in SC. Can mean short term outcomes (e.g., participants

developed a peer network of support while learning about new employment benefits) but also long- term (e.g., participants organized themselves into a local “advocacy” group in Charleston and speak regularly at City Council meetings. This contact resulted in a member being asked to join the Mayor’s Advisory Committee a year after the individual participated in the original grant activities). Projects working directly with individuals and families are asked to assess how their grant activities are affecting participants lives outside of grant activities. EG, a new skill or knowledge area might be introduced to participants, but how are the participants using these new skills and how have their lives in general been affected by being a part of sub-grantees’ activities?

25

Tools We Use in South Carolina

Quarterly Progress Report (QPR) Handout

• Quarterly Progress Report (QPR) (Handout)

ITACC: 2017 AIDD Technical Assistance Institute

26

Notes from Slide 16: Word document created with “developer tools tab,” “design mode,” “controls panel.” including “click text” boxes to help guide sub-grantees through the QPR. The document is “locked” so sub- grantees cannot accidentally modify it and can only enter info in the “click text” boxes. There are instances in which a sub-grantee needs to add rows to the objectives section of the QPR, in which case, the sub-grantee must come to program staff. This is good because some sub-grantees include a lot of objectives to their programming and this affords an opportunity to discuss realistic expectations for a 12-month program—more than six objectives to a grant project is a “rose flag” (not a red flag) to program staff to take a closer look at the sub-grantees programming expectations so they don’t freak out (and Council program staff doesn’t freak out, too).

27

QPR – THREE PRIMARY FOCUS AREAS

• Number of people served
• Numbers – quantitative
• Grant Activities
• Narrative – qualitative
• Partners and System Change
• Numbers and narrative

ITACC: 2017 AIDD Technical Assistance Institute

28

Notes from Slide 17:

• 1. Number of people served (numbers, quantitative)
• Individuals with intellectual and developmental disabilities
• Sub-grantees are trained in the DD Act definition of Intellectual and developmental disability
• Family members
• Other: sub-grantee is asked to describe who participated in activities (direct service professionals, teachers, law enforcement staff, doctors,

lawyers, etc.)

• 1: People served: Only those participating in organized grant activities (conference exhibiting, publicity, awareness raising, and similar events

should never be counted in number of project participants). Sub grantees are asked to show how they are tracking participants and gathering information on disability.

• 2. Grant Activities (narrative, qualitative)
• Evidence base: best and/or promising practices
• Activities narrative
• Sub-grantee reflects on activities only, specifically, what they expected to happen versus what actually happened.
• Helps SC DD Council staff discern whether or not the sub-grantees’ activities are contributing to improvements in existing or creation of

new best and/or promising practices (AIDD PM).

• Quarterly narrative
• Collect success stories, challenges experienced, lessons learned, collaborations, and unexpected outcomes.
• This section helps drive our focus for DD Council newsletters and for annual Program Performance Review.
• Particularly important to capture nuances in data not reflected in numbers
• Also important in helping sub-grantees to avoid duplication in reporting
• 2: Grant activities: focus on the activities, not the impact. In addition to program staff evaluating whether or not the sub-grantees

activities are leading to improvements in best/promising practices, this section helps program staff evaluate whether or not the sub- grantees’ activities have clear links to the overall program outcomes. Quarterly narrative: focus on impact. Sub grantees are encouraged to share success stories and lessons learned. Program staff gathers questions for monitoring visits (from the whole QPR, but particularly this these narrative sections). Duplication—some grantees partner on multiple projects including conferences. Program staff has to ensure participants are being counted only once.

• 3. Partners and Systems Change
• Draw sub-grantees’ attention to changes in policy and procedure both internal and among project partners and stakeholders
• Some sub-grantees’ projects are working to directly influence law and regulations
• Numbers are requested as part of AIDD PM, and SC DD Council also asks for descriptions of policy and systems change work
• This section is important in helping program staff work

Sub-grantees often influence policy and procedure and do not realize it especially processes internal to their own organization.

• -Descriptions are requested to follow up on during monitoring visits.
• - The prepared survey tools provided by DD Council are not set-up to gather systems change data, they are geared towards projects working directly

with people. Systems change work is more nuanced and, therefore, we work with individual systems change projects to create assessments for impact.

• -Review contexts, use QPR instructions.

29

TRAINING FOR GRANTEES

Sub-grantees receive yearly training on QPR and surveying. Important Contexts for Sub-Grantee QPR Training

• 1. Best practice verses promising practice
• 2. Organized grant activities
• 3. Advocacy in our SC DD Council (and DD Network)

context

• 4. DD Act IPSII Outcomes

Questions before we review survey tools?

ITACC: 2017 AIDD Technical Assistance Institute

30

SC SURVEY TOOLS

• SC DD Council prepared two survey tools to be used by sub-

grantees.

• Sub-grantees are trained that these include the minimum

requirements for surveying and they are encouraged to modify surveys based on their program needs and focus.

• Sub-grantee projects not working directly with individuals

and families or who are working on systems change projects must work with DD Council program staff to create a combined satisfaction and outcomes survey specific to their program.

http://www.scddc.state.sc.us/Reportandreimb.html

ITACC: 2017 AIDD Technical Assistance Institute

31

Notes for Slide 19:

• -SC DD Council’s State Plan Evaluation Plan states we will measure four key areas of satisfaction:

behavioral, affective, emotional, and cognitive satisfaction. Therefore, program staff TA ensures that sub-grantees creating their own satisfaction survey is relevant to these measures.

• -The outcomes survey collects basic demographic data and measures the level of advocacy

(keeping in mind a broad definition of advocacy/self advocacy) increases as a result of the project's

• activities. Outcomes survey mirrors AIDD PMs.
• -These templates are word documents that are easily edited. Many sub-grantees modify them into

Survey Monkey or other online platforms.

32

SC SURVEY (HANDOUT)

ITACC: 2017 AIDD Technical Assistance Institute

33

ITACC: 2017 AIDD Technical Assistance Institute

SC SATISFACTION SURVEY

• SC DD Council’s

satisfaction survey collects quantitative and qualitative data

• Key Focus on

behavioral, affective, emotional, and cognitive satisfaction

Affective (like/dislike) and behavioral satisfaction:

• May measure how something

specific affected one’s behavior Behavioral satisfaction:

• May measure intent or similar

behavior Affective (like/dislike) and Emotional satisfaction:

• Measures satisfaction overall or on

specific topics Cognitive Satisfaction:

• Measures judgment

34

Notes for Slide 21: There are overlaps in affective/behavioral; affective/cognitive satisfaction because affective (liking/ disliking something specific) measures indicate how a specific item affects one’s behavior (behavioral) or judgment (cognitive) Affective (like/dislike) and behavioral satisfaction: May measure how something specific affected one’s behavior

• Would you recommend this event to a friend?

Behavioral satisfaction: May measure intent or similar behavior

• Would you participate in another event provided by this organization?

Affective (like/dislike) and Emotional satisfaction: Measures satisfaction overall or on specific topics

• How satisfied are you with the way the material was presented?
• How satisfied are you with the subject matter covered?

Cognitive Satisfaction: Measures judgment

• What was your favorite part of the event?
• What was the most useful thing you learned?
• How would you change this event?

35

Outcomes Survey

• Adapts AIDD PMs not included in the QPR to collect

data directly from individuals and families.

• Sub-grantees working on systems change projects must

work with DD Council program staff to create a targeted data collection tool. Data Measures that must be included (AIDD PM’s)

Advocacy, self-advocacy, and self-determined behavior Individual or family member’s ability to speak up for themselves Board, committee, coalition involvement

http://www.scddc.state.sc.us/documents/DDC%20Outcomes%20Survey.docx

ITACC: 2017 AIDD Technical Assistance Institute

36

Notes for Slide 22:

• These outcomes surveys are intended for individuals with intellectual and developmental disabilities

and their family members.

• Some sub-grantees feel this survey is not appropriate for people with severe disabilities in which case

we partner with them to work on data collection tools appropriate for their audience.

• Strategies include one-on-one interviews or group interviews. Have included sub-grantee

project staff and DD Council program staff or, my favorite, peer lead group interviews.

• -Advocacy: Sub-grantees are trained that DD Council takes a broad view on advocacy, self

advocacy, and self determined behavior. Definition provided on the survey and sub-grantees are encouraged to emphasize the “speaking up for someone else or yourself” aspect of

• advocacy. Attempt to measure a before and after, but we do not encourage pre and post-

testing because you don’t know what you don’t know.

• Advocacy, self advocacy, and self determined behavior
• Both learning how to speak up and increasing speaking up whether in a formal way or in a

general way

• Advocacy, self advocacy, self determined behavior including IEP meetings, providing legislative

testimony, Advocacy Day for Access and Independence, Community Engagement Day

• An individual or family member’s ability to describe what they need or want either in general or in

formal services (depends on the project)

• Board participation
• Maybe an individual or family member gained membership. If not, we ask what kind of board

they would like to be a part of and make connections when appropriate

37

Systems Change Outcomes and Impacts Data Collection

• Sub-grantees and Council staff work together to create a survey

that measures:

• Satisfaction
• Project specific goals
• Existing policies
• Other measures
• Surveys are anonymous and administered online (in general)
• Purpose:
• Gather information on how systems operate currently
• What level of appetite exists to move systems in a defined

direction

• What policies and laws can be targeted for change
• Future areas of focus for the DD Council (in general)
• Data collected may be used to amend or influence updates to

State plan and annual work plans

ITACC: 2017 AIDD Technical Assistance Institute

38

Notes for Slide 23: Use SDM survey as an example.

• -What steps should stakeholders take to further SDM in SC?
• -What specific activities can your organization do to further SDM in SC?
• -What policies exist in your organization relating to guardianship and SDM?

39

NEVADA

• Collection
• Make sub-grantees do the work for you
• Individualized reporting templates and surveys
• Provide in depth training when grant is awarded
• Sub-grantee checklist
• Calculation
• PM Spreadsheet
• Impact
• Council PM’s from 5 year plan
• Impact of Activity

(sub grantee report template)

• Satisfaction Survey’s
• Personal stories

ITACC: 2017 AIDD Technical Assistance Institute

40

Notes for Slide 24: COLLECTION

• We represent the smaller states who may not have 10+ sub-grantees. We also do a lot of our

projects in-house.

• Sub grantee reporting templates and surveys: I create them based on their application – how

they said they would meet the specific requirements of the Goal/Objective in our 5 year state plan (Review sections of template) add what I need to know for the PPR.

• PM’s - we found that we get information we can use that we might not have been able to

collect if we limited the PM’s we had them report on).

• Review sections of surveys (Everything we do is based on our 5 year plan objectives. We get
• ur I/DD and family satisfaction data from the surveys. Each grantee is required to complete

surveys for each activity and we also do surveys for Council or “in-house” activities. Each survey has the correlating objective printed on it. That info ends up in our Sat Fact. Survey results tab on our spreadsheet.)

• It’s more work for me up front but ends up less on the back end.

CALCULATION

• Review PM spreadsheet (how do I calculate my numbers?)

IMPACT

• We’ve come full circle back to the 5 year state plan! (cartoon – if you’re doing it right, it should

feel like your 5 yr state plan is following you around! And although we may all take different directions, we all end up back at the same place.

• Review Council PM’s in PPR

41

TEXAS

• Average 30-35 grants each year
• Multiple start and end dates

depending on project- impacts PPR reporting year

• Rely on grantee to identify
• utcomes of individual projects

and annual workplan Current Data Collection:

• Numbers related to specific

performance measures

• Additional data from project

narrative in DD Suite

ITACC: 2017 AIDD Technical Assistance Institute

42

TEXAS CONTINUED

Shifts in Data Collection:

• “Assigned” performance measures based on state

plan goal (can add others as project allows)

• Sample tools for satisfaction and advocacy

activities

• One-on-one TA for data collection methodology
• Collection of personal stories
• Continue to rely on project narratives

ITACC: 2017 AIDD Technical Assistance Institute

43

QUESTIONS

ITACC: 2017 AIDD Technical Assistance Institute

44

YOUR TURN! – NETWORKING ACTIVITY

• Let’s use our remaining minutes to share data

collection ideas and strategies with each other

• What did you hear about today that you may want to

use or modify for use?

• What are you doing that others may find helpful?

ITACC: 2017 AIDD Technical Assistance Institute

45