Latinos in Oregon: Evaluation Jam Session Trends and Opportunities - - PowerPoint PPT Presentation

Latinos in Oregon:

Trends and Opportunities in a Changing State

THE OREGON COMMUNITY FOUNDATION

Studio to School Initiative Evaluation Jam Session Qualitative Analysis Part Two

April 16, 2018

Logistics

• Thank you for patience!
• We are recording
• If you can’t hear us,

check your speaker or switch to the phone. Stay on your computer to see the slides!

Logistics

• Everyone is muted, we’ll

unmute for Q&A if feasible

• Please mute your

microphones as well

• Use chat box for

questions, etc.

Reminders

• Today’s jam session slides and recording

available on the website:

• http://www.studiotoschool.org/info-for-

grantees/project-evaluation-worksheets/

Agenda

• Today
• Thematic analysis vs

rubrics

• Creating rubrics
• Using rubrics

Other Rubric Resources

• Authentic assessment

webinar

• Rubric videos
• Website Resources
• OCF team!

What is a rubric?

• Type of rating

scale, like a customer satisfaction survey

• Key difference: the

levels of a rubric are well defined

Rubrics

Novice Proficient Advanced Criteria Description Description Description Criteria Description Description Description Criteria Description Description Description Criteria Description Description Description

Rubrics vs Thematic Analysis

Thematic Analysis Rubric Analysis More time spent gathering and interpreting data More time spent creating interpretation tools Inductive and deductive reasoning Primarily deductive reasoning No value attached to analysis Value attached (So what?)

Process

Process

Inductive vs Deductive

Top Down Buckets

Bottom up Buckets

Use both!

Detailed description

Rubrics are more top down

Data

The “So what?” step

vs

How do you know which approach to choose?

• Why choose a

rubric?

• Why choose

thematic analysis?

Deciding between rubrics and thematic analysis

Rubric Thematic analysis

When top-down makes sense Bottom up or a combination is appropriate Levels or stages make sense Not sensible to develop levels Change is anticipated/sought in particular ways Stand-alone effort

Questions

Process of using a rubric

Define what “art skills” means for your program Create rubrics to look at student work Create and collect examples of student work Use the rubric you created to assess student work

Focus on arts skills

• Today the focus is on

using rubrics for arts skills ratings

• Can also use them for

rating other parts of program performance

Define “art skills”

• What do you want students

to learn?

• What would it look like for

students to use those skills?

• More than just creating art!
• Thinking about art
• Reflecting on art

Start with your logic model!

“Students will demonstrate an increased knowledge of storytelling”

“Student’s will improve their arts vocabulary”

“Student’s build skills, knowledge, language and engagement in the arts, music education, and performance.”

“Increase thematic discussion and reflection through an artistic lens and discuss what art means personally.”

“Student performance in adjudicated competition improves over time.”

“Increase in student learning- skill, knowledge, ability and appreciation in music.”

“Students develop their media arts ability during and after school.”

“Acquisition or improvement of arts skills demonstrated through student understanding and application of the SHOM”

Student Arts Skills

“Core subject teachers gain knowledge, ability, confidence and commitment to integrate the arts into core subjects and classrooms”

“Teachers gain skill and comfort in art integration”

“Increased teacher proficiency/ capacity to integrate the arts”

“Teachers better integrate the arts into core curriculum”

“Teachers have enhanced their ability to implement arts integration into their classroom.” “Teachers can confidently incorporate arts learning in the classroom” “Children’s original thinking elicited in arts experiences will provoke new understanding on the parts of staff.”

Teacher Ability to Integrate Arts

From Portfolio to outcome

Outcome Criteria Criteria Criteria

What art skills should students learn?

Outcome: “Student have musical skills and knowledge to successfully participate in high school orchestra.” 1. Ability to refine musical work over time. 2. Understanding context and structure of musical work. 3. Demonstrate technical skill with their instrument. 4. Present/perform musical work.

What art skills should students learn?

Outcome: “Student have musical skills and knowledge to successfully participate in high school orchestra.” 1. Ability to refine musical work over time. 2. Understanding context and structure of musical work. 3. Demonstrate technical skill with their instrument. 4. Present/perform musical work.

Novice Proficient Advanced Criteria Description Description Description Criteria Description Description Description Criteria Description Description Description Criteria Description Description Description

Develop Descriptions

For each rubric criteria, consider:

• What would a novice be

able to do?

• What about a student

who has mastered that skill?

• How would students’

different levels of mastery look different in their work?

Examples

Four examples

• Generic rubric for program performance
• Generic rubric for working with open ended

responses

• Rubric looking at the big picture of arts skills
• Specific example of one arts skills

Generic Rubric

Rating Description Excellent Clear example of exemplary performance or best practice in this domain; no weaknesses Very good Very good or excellent performance on virtually all aspects; strong overall but not exemplary; no weaknesses of any real consequence Good Reasonably good performance overall; might have a few slight weaknesses but nothing serious Barely Adequate Fair performance; some serious (but nonfatal) weaknesses on a few aspects Poor Clear evidence of unsatisfactory functioning; serious weaknesses across the board or on crucial aspects

From Evaluation Methodology Basics by E. J. Davidson

Open ended response rubric

Rating Description Excellent

Evidence of a strong positive impact: Very positive comments, with a substantial number that indicated a very strong impact few if any neutral

• r negative comments

Very good

Evidence of noticeable positive impact: a good number of positive comments (few neutral or negative), clearly showing that the program had made a noticeable positive effect on students

Good

Evidence of some positive impact: a mix of positive and negative comments, skewed somewhat towards the positive; evidence pointing in the right direction but not to a very noticeable impact.

Barely Adequate

Limited or no impact either way: a real mix of comments, no clear skew in either the positive or negative direction

Poor

Evidence of some negative impact: a mix of positive and negative comments, skewed somewhat towards the negative; not enough evidence to call this a really noticeable negative impact. From Evaluation Methodology Basics by E. J. Davidson

Generic Arts Rubrics

Rating Description

Advanced

Students at the Advanced level independently identify challenging arts problems based on their interests or for specific purposes, and bring creativity and insight to finding artistic

• solutions. They are able to use at least one art form as an effective avenue for personal

communication, demonstrating a higher level of technical and expressive proficiency.

Accomplished

Students at the Accomplished level are -- with minimal assistance -- able to identify or solve arts problems based on their interests or for a particular purpose; conduct research to inform artistic decisions; and create and refine arts products, performances, or presentations that demonstrate technical proficiency, personal communication and expression.

Proficient

Students at the Proficient level have developed the foundational technical and expressive skills and understandings in an art form necessary to solve assigned problems or prepare assigned repertoire for presentation.

Intermediate

Students at the Intermediate level are continuing study in a chosen specialized art form. Their development continues in artistic understanding and technical and expressive skills enabling the student to begin to independently and collaboratively create, perform and respond at their given skill level. Their presentation and performance opportunities in ensembles at school and in the community increase and students actively participate in rehearsals.

Novice

Students at the Novice level have started specialization in an art form of their choice. They are beginning to develop the basic artistic understanding and technique necessary to advance their skill level. Their expressive skills may be identified and exploratory work begins. They may participate in presentation and performance opportunities as they are able.

Adapted from National Core Arts Standards

Specific Skill Arts Rubric

Rating Description

Advanced

Engage in constructive critique with peers, then reflect on, reengage , revise or refine work of art and design in response to personal artistic vision.

Accomplished

Is able to reflect on and explain important information about personal artwork in an artists statement or another format. Applies relevant criterial to examine reflect on and plan revision for work of art or design in progress.

Proficient

Able to create an artists statement using art vocabulary to describe personal choices in art making. Reflects on whether personal artwork conveys the intended meaning and revise accordingly.

Intermediate

Is able to revise artwork in progress on the basis on insights gained through peer discussion.

Novice

Is able to discuss and reflect with peers about choices made in creating artwork.

Refine and complete artistic work

Adapted from National Core Arts Standards

Questions

Rubric Version One

Refining Rubrics

Rate Portfolio Pieces Rubric Version Two

Revision

Rate Portfolio Pieces

Refining Rubrics

From Portfolio to outcome

Outcome Criteria Criteria Criteria Portfolio Piece Portfolio Piece Portfolio Piece

The “Process-folio”

Three categories of things you can include in a portfolio:

• Process items, which

show a student’s creative process.

• Performances/Products,

which are the art that the student has created.

• Reflection items, which

show how a student thinks about the work that they do.

Benchmarking

• Benchmarks are examples of

students work that everyone agrees are at a certain level

• Bridge between written

description and student work

• Can define just the middle or

multiple levels

Benchmarking the middle

Benchmarking the middle

Testing your rubric

• Get your rubric buddies back

together to look at some student work

• Use the same type of work that

you will be using when you rate for the evaluation

• Have people rate independently

and then discuss the ratings

Testing your rubric

Time to revise

Example: Rating portfolios

• Rate student work using the

rubrics

• Generally this will look like

scores for each student on each criteria

• Take notes about thing you see

while rating

Example: Principles

Dormant Emerging Deepening Embedding Thriving Definition

• f Stages

There are no or very few arts education

• pportunities available to

students at school. No or very little attention is being paid to the need for arts education; no one is organizing efforts to improve or expand arts education opportunities at this time. There are some arts education offerings available through the school, but they are largely

• pportunistic, rather than

being integrated or connected to larger school vision or curriculum. Gaps are being identified; there is a growing interest in improvement, but a lack

• f leadership support for

arts education. Steps are being taken to make arts education more accessible and available to more students. Efforts are underway to improve the quality of arts education

• fferings and deepen arts

integration within the school. Arts education programming is

• coordinated. Its purpose in

supporting student learning is clear. Arts educators engage

• thers, such as school

administrators, in planning and implementation of programming. Proactive measures are underway to ensure that all students experience high-quality arts education programming, and the arts are being creatively integrated into the broader school curriculum. The school culture is supportive of arts learning and recognizes its value. As a result, arts education is being embedded into school structures. Systems are in place for an equitable, high-quality arts education to flourish and sustain. Schools and communities truly “get” and promote the value of arts education. Arts educators find creative ways to provide innovative, engaging, and relevant and culturally responsive arts education programming for all students.

Sharing what you found

• Rating plus description
• Big picture on how students

are doing using the rubric scores

• Narrative about what you

see and what that means.

Example

4 6 18 9 3 Novice Intermediate Proficient Accomplished Advanced

Ability to refine and complete artistic work

The majority of our students are able to reflect on whether their art is conveying the intended meaning and refine their art based on things they learned in peer

• discussion. A typical example of this is a

student who said “__________”. A number

• f students are showing greater ability to

critique their own work and provide more constructive criticism to their peers. For example, one student _______. The students currently at novice and intermediate are primarily struggling with using art vocabulary and articulating what message they are trying to give to their audience.

Goal

• Remember that the goal is

useful discussions about what is happening in your program

Questions?