THE OREGON COMMUNITY FOUNDATION Latinos in Oregon: Studio to School Initiative Trends and Opportunities Evaluation Jam Session # 3 in a Changing State April 12 th , 2017
Logistics • Thank you for patience! • We are recording • If you can’t hear us, check your speaker or switch to the phone. Stay on your computer to see the slides!
Logistics • Everyone is muted, we’ll unmute for Q&A if feasible • Please mute your microphones as well • Use chat box for questions, etc.
Upcoming Evaluation Deadlines When Topic May 22 Assessing Arts Learning: Measuring student 10am to 11am learning in and through the arts. Remember that slides and recordings for Jam Sessions 1, 2 & 3 are on the Studio to School website! • Logic models due April 14 th • Email to Madeline: mbaars@oregoncf.org • Next eJournal post due May 1 • Spring interviews with project teams
From anecdotes to useful information “To the professionally trained and attuned ear, an anecdote is scientific data — a note in a symphony of human experience. Of course, you have to know how to listen.” - Michael Quinn Patton
Quantitative versus Qualitative Quantitative data : things you count, quantify, and measure numerically. Examples: • Attendance • Test scores • # of performances • # of instruments
Quantitative versus Qualitative Qualitative data : descriptions of reasons, opinions, motivations, context.
Qualitative data can help you: • Understand program’s effects; • Explore successes and challenges; • Communicate what the program is doing.
Basics of qualitative data If you want to change attitudes toward or increase the likelihood of certain behaviors, you need qualitative data about attitudes and feelings toward these behaviors.
Use qualitative methods to communicate on an emotional level.
Examples of qualitative data • Answers to open-ended survey questions • Information from an interview / focus group • Observations from a site visit • Journal entries
Student interviews: RACC Hillsboro https://www.youtube.com/watch?v=FDw3O8HXGms
Questions?
Remember your logic model? • Use your logic model to guide the questions you ask and data you gather • Look at the output and outcomes columns to formulate evaluation questions • Output column = raw data
Anecdotes versus data A collection of anecdotes can become data when systematically, intentionally, and carefully recorded and thoughtfully analyzed.
Qualitative data collection • Collect multiple anecdotes • Look for patterns and common themes that emerge across anecdotes • Check the accuracy with other sources Example: gather anecdotes to capture the experience of a theater performance from the perspective of the audience.
What kind of data do you need? • Impact data: program’s effect on participants • Process data : how the program works
Determining the right method Depends on many factors, including: • Staff capacity • Program and community culture • Data sharing plans • Purpose for collecting data
Qualitative data collection Method Characteristics Observation Program activities/operations, context. Interviews Personal stories and context. & focus groups Lots of detail. Open-ended survey Provide stories and context. questions Generally less detail. Case studies Produce detailed data of context, challenges, successes and outcomes. Often a combination of qualitative and quantitative sources.
Case study • Detailed examination of a program, initiative, project or individual. • Highlight context, challenges, successes and outcomes. • Combine quantitative and qualitative data, producing detailed information • Include data from many sources: interviews, observations, surveys, document reviews.
Case studies: RACC Hillsboro
Questions?
Storytelling with data Combines quantitative and qualitative data. Participatory form of evaluation.
Storytelling with data Giving a human face to evaluation strengthens program messages. Incorporate perspectives that speak to the concerns of your target audience. Highlight program progress .
Best practices in story gathering • Ask permission to record • Explain how stories will be used • Seek approval before sharing • Change names & distinguishing characteristics
Story circles A simple, focus group-style method. At the end of a meeting, ask everyone for a story. Example prompts: • “What have you learned through being a part of [program]?” • “How has participating in [program] changed your life?” Ask the group to summarize the most important lessons from their stories. Record this! Repeat every 3-6 months to assess progress.
Vignettes Short stories that illustrate and contextualize facts/numbers. For example, a written report might include: “We held four concerts, with fifty attendees at each.” Add an personal account from a teacher: “We have never had performances like this at our school before. So many members of the community came – even those who don’t have kids! I was amazed. We’re planning to make the performances even bigger next year.” That’s qualitative info supporting quantitative data– personal statements giving data life!
Visual storytelling methods • Photovoic e: https://photovoice.org/ • Scrapbooking : • Collect artifacts of program activities, outcomes • Document challenges, accomplishments, growth • Story-quilting: • Each participant makes a square of a personal reflection on the program, process, or outcomes. • Create a quilt of individual/collective experiences. • Story Theater : • Participants to role-play program process, challenges, accomplishments, and outcomes.
Qualitative data analysis Remember that collecting and sharing stories is not the same thing as analysis. You have data from interviews, observations, written documents or journals, open-ended survey questions…
Qualitative data analysis Some accounts stand alone, providing information about how the program is working. In most cases, you must analyze qualitative data in systematic ways.
Qualitative data analysis methods • Check circumstances, background, source(s) • Categorization and coding • Identify themes, patterns • Participatory analysis • Subjects interpret their interview transcripts • Apply rubrics • Help with rubric creation: http://www.studiotoschool.org/evaluation_handbook_arts-2/ http://www.studiotoschool.org/evaluation_guide_arts_based/
Considerations with qualitative data Start with the data, let it tell you the story • Don’t only include success stories!
Considerations with qualitative data Be aware of stories becoming “stale”
Considerations with qualitative data Complement stories with other data. Using many forms of data, and multiple perspectives, enhances evaluation quality and impact. This turns anecdotes into accurate, useful information.
Questions?
Reminders • Slides and recording available on the website • http://www.studiotoschool.org/info-for-grantees/project- evaluation-worksheets/ • Upcoming jam session: May 22, 10am • Logic models due April 14 th • Email to Madeline: mbaars@oregoncf.org • eJournal post due May 1 st • Spring interviews coming • Rendezvous Aug 1 – 3 at Oregon Gardens in Silverton
Recommend
More recommend
