youth pqa external methods? What might create an obstacle to - - PDF document

1

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

youth pqa external assessor reliability training

Training Tips Slides

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Before we Assess

Check your bias!

• What triggers your judgment?
• What styles of instruction do you

tend to like or not like? What about methods?

• What might create an obstacle to

your being able to be objective? Remember, the goal is to collect

• bjective data! Trust the tool to do

the assessment.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

The Role of an External Assessor

• You are here to be a reliable rater.
• This means your scores provide a touchstone or

point of reference for self-assessment scores.

• Being reliable means you have a more complete

understanding of the language and intent of each item in the tool.

• Set aside your biases. Recognize what you are and

are not scoring—see next slide.

• Score the tool as defined and written.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Overall Program Quality

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Completing and Scoring the PQA

• Use the Observation Guide when you observe; it

lists every item and will help trigger your memory, as well as remind you to ask follow up questions when needed.

• Always try to see multiple items in every

interaction and cross-reference constantly.

• Look to a preponderance of evidence but favor

higher scores; give programs the benefit of the

• doubt. Remember: NO 2’s or 4’s.
• Fill in the front pages! All the site data is

necessary: YOU are part of our growing sample!

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Quality Continuum for an Individual Item

2

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

More Tips on Scoring the PQA

• Only ask follow-up questions for items with

question(s) in the evidence column. Ask the questions as they are printed.

• Anecdotal evidence needs to be entered for every

row in order to score.

• Score an X if specified items or scales are not

to be scored. (See notes for which items/scales this is allowed—i.e., conflict scale if no conflict present.)

• Read each level of the item carefully. Use

handbook and Frequently Confusing Items sheet!

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for These Scales

• Nourishment: Water fountain in hallway OK.
• Emergency Preparedness: Only score additional safety

equipment if safety equipment is necessary for activity.

• Emergency Preparedness: Only score outdoor program

space if outdoor space is present and utilized.

• Emergency Preparedness: If session is completely
• utdoors, mark X for supervised indoor entrances.
• Emergency Preparedness: Note, if the program is outside,

the emergency procedures and fire extinguisher may be inside—score 3 if it is accessible.

• Emergency Preparedness: Doors locked from outside do

not need to be monitored in the same ways as a main door.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Emotional Safety

• This scale is one of the most potentially subjective.
• Collect as much evidence as possible and assess

the overall climate.

• Observe both youth and staff in order to collect

evidence of:

 Interactions between youth and staff.  Interactions among youth.  Individual youth behavior.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Warm Welcome

• “Warm and Respectful” and “Positive Body

Language” are potentially subjective items.

 Look for evidence of disrespect, sarcasm, insults, dismissive hand gestures, etc.  A wide variety of styles can still be assessed as a “5.”

• Item 1: Opening “hurrah” does not count as a
• greeting. With a greeting, youth presence is

acknowledged, as individuals or a group.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Session Flow

• Item 2: Materials simply need to be in the room and

easily accessible for them to be “ready.”

 They are NOT ready if staff or youth have to leave the room to get them.

• Item 4: To assess clear explanations and

appropriate time, observe how youth react to staff:

 What do they do after instructions are given?  What do they say or do to indicate whether they feel rushed, bored, or the pace is just right?

• Item 5: Use evidence from activities observed

rather than activities that are planned.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Active Engagement

• “Active Learning” is not necessarily “hands-on” learning.
• Item 2: “Talking about what they are doing” must be a

structured opportunity, not something that just happens.

• Item 3: Abstract and concrete should be complementary.

 Pay attention to the purpose of the workshop or activity you are observing.

• Item 4: Tangible products are things you can see, feel or
• hear. Simply learning a skill is not a tangible product. A

performance is presenting or demonstrating a skill or product to an audience.

3

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Skill Building

• Most of this scale relies on a specific skill or set of

skills being addressed in the session.

• Item 1: What youth are going to “do” is not

necessarily the same as what they will “learn.”

 Note how staff frames learning objective.

• Items 2 & 5: Observing youth behavior will give you

clues about whether or not they are being challenged to improve and/or struggling.

 Note staff responses to wrong answers and errors.  Observe staff interactions with youth to assess the scale.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Encouragement

• Item 1: Evaluative language is not “bad” and can be (in fact

it is often) present in a session that is assessed with a “5.”

 Listen for staff to follow-up an evaluative statement with specific, non-evaluative language.

• Item 2: Take notes on all of the questions you hear.

Determine whether or not they were open-ended when you go to score the tool.

• Item 3: Look for staff to be an active part of what happens in

the session.

 This can take on a lot of different forms (e.g., giving instructions, working alongside, circulating, etc.).

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Reframing Conflict

• When a conflict occurs, pay close attention to how

staff reacts.

• Observe as much of the interaction between staff

and youth as possible.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Belonging

• Item 1: Look for structured opportunities.
• Item 2: Carefully observe youth interactions with

each other.

 Observe complete interactions whenever possible.

• Item 3 can be a difficult item for programs to

achieve a high assessment score.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Collaboration

• These are 3 ways to promote collaboration.

 Often they work together, encouraging deeper forms of true collaboration.  A program may have any combination of these elements.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Collaboration

• Item 1: Look for opportunities for youth to work with
• ther youth.
• Item 2: “Interdependent Roles” simply means that

the activity wouldn’t work individually.

 Doesn’t always have to be explicitly assigned.

• Item 3: “Shared goals” are mutual or joint goals.

4

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Leadership

• Item 1: Opportunity for “group process skill” can be relatively

simple to achieve.

 Focus on whether or not ALL youth got the opportunity.

• Item 2: Mentoring isn’t just working together – one youth

needs to help another.

 Observe whether or not the opportunity is planned and structured or spontaneous.  Must be one-on-one.

• Item 3: It is not common to see evidence for a 5 with “All

youth lead group.” That’s ok!

 Leader has to be named, appointed or given control.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Adult Partners

• Item 1: Pay attention to the types of questions the

staff asks, the choices he/she provides, and what follow-through is like.

• Item 2: Pay attention to the behavioral indications

and directions staff give. Is there a stated reason why?

 Does not apply to instructions for task or activity.  Explanations must be given at least once for each behavioral expectation/direction staff gives.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Planning

• Planning happens before an activity.
• Planning is not simply a choice.
• Planning is figuring out steps, sequence,

roles, timeframe, etc.

• Can be short or long-term.
• Item 2: Programs infrequently score a 5.
• Verbal brainstorming is most common.

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Choice

• Content = what

 Do youth get to choose what they do?  Discrete options = 3, Open-ended = 5

• Process = how

 Do youth get to choose how they do something?  Discrete options = 3, Open-ended = 5

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Tips for Reflection

• Reflection typically happens after an activity.

 Explores feelings that came up during activity  Explores learning connections  Synonyms: debrief, process

• Item 3: Feedback is youth offering advice about the

activities to the staff.

• Item 4: Extends to the program offering. (Refer to

page ii in the PQA). If you don’t have evidence, ask the follow-up question.

 Youth present/share the results of their efforts

David P. Weikart Center for Youth Program Quality : Youth PQA External Assessor Reliability Training

Thank you!

• Feel free to contact the Weikart Center with any

questions you may have.

• Barb Hillaker – Research Consultant

 barbara@cypq.org  734-714-2526