[PDF] - school-age pqa external methods? What might create an obstacle to PDF Document

SLIDE 1

1

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

school-age pqa external assessor reliability training

Training Tips Slides

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Before we Assess

Check your bias!

What triggers your judgment?
What styles of instruction do you

tend to like or not like? What about methods?

What might create an obstacle to

your being able to be objective? Remember, the goal is to collect

bjective data! Trust the tool to do

the assessment.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

The Role of an External Assessor

You are here to be a reliable rater.
This means your scores provide a touchstone or

point of reference for self-assessment scores.

Being reliable means you have a more complete

understanding of the language and intent of each item in the tool.

Set aside your biases. Recognize what you are and

are not scoring—see next slide.

Score the tool as defined and written.

Overall Program Quality

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Completing and Scoring the PQA

Use the Observation Guide when you observe; it

lists every item and will help trigger your memory and remind you to ask follow up questions when necessary.

Always try to see multiple items in every

interaction and cross-reference constantly.

Look to a preponderance of evidence but favor

higher scores; give programs the benefit of the

doubt. Remember: NO 2’s or 4’s.
Fill in the front pages! All the site data is

necessary: YOU are part of our growing sample!

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Quality Continuum for an Individual Item

SLIDE 2

2

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

More Tips on Scoring the PQA

Only ask follow-up questions for items with

question(s) in the evidence column. Ask the questions as they are printed.

Anecdotal evidence needs to be entered for every

row in order to score.

Score an X if specified items or scales are not

to be scored. (See notes for which items/scales this is allowed—i.e. managing feelings scale if no disruptive feelings or conflict are present.)

Read each level of the item carefully. Use

handbook and Frequently Confusing Items sheet!

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for These Physical Safety Scales

Nourishment: Water fountain in hallway OK.
Emergency Preparedness: Only score additional safety

equipment if safety equipment is necessary for activity.

Emergency Preparedness: Only score outdoor program

space if outdoor space is present and utilized.

Emergency Preparedness: If session is completely
utdoors, mark X for supervised indoor entrances.
Emergency Preparedness: Note, if the program is outside,

the emergency procedures and fire extinguisher may be inside—score 3 if it is accessible.

Emergency Preparedness: Doors locked from outside do

not need to be monitored in the same ways as a main door.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Emotional Safety

This scale is one of the most potentially subjective.
Collect as much evidence as possible and assess

the overall climate.

Observe both children and staff in order to collect

evidence:

 Interactions between children and staff.  Interactions among children.  Individual children behavior

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Warm Welcome

“Warm and Respectful” and “Positive Body

Language” are potentially subjective items.

 Look for evidence of disrespect, sarcasm, insults, dismissive hand gestures, etc.  A wide variety of styles can still be assessed as a “5.”

Item 1: Opening “hurrah” does not count as a
greeting. With a greeting, youth presence is

acknowledged, as individuals or a group.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Session Flow

Item 2: Materials simply need to be in the room and

easily accessible for them to be “ready.”

 They are NOT ready if staff or youth have to leave the room to get them.

Item 4: To assess clear explanations and

appropriate time, observe how youth react to staff:

 What do they do after instructions are given?  What do they say or do to indicate whether they feel rushed, bored, or the pace is just right?

Item 5: Use evidence from activities observed

rather than activities that are planned.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Active Engagement

“Active Learning” is not necessarily “hands-on”
learning. Includes children actively engaging with

ideas.

Item 2: “Talking about what they are doing” must

be a structured opportunity, not something that just happens.

SLIDE 3

3

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Skill Building

Most of this scale relies on a specific skill or set of

skills being addressed in the session.

Item 1: What youth are going to “do” is not

necessarily the same as what they will “learn.”

 Note how staff frames learning objective.

Items 2 & 5: Observing youth behavior will give

you clues about whether or not they are being challenged to improve and/or struggling.

 Note staff responses to wrong answers and errors.  Observe staff interactions with youth to assess the scale.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Encouragement

Item 1: Non-evaluative language:

 Praise is not “bad.” It is usually present in a session that is assessed with a “5.”  Listen for staff to follow-up an evaluative statement with specific, non-evaluative language.  Look for at least one instance: a specific praise, an

bjective description of what the child did, staff asking

child to describe his or her work

Item 2: Take notes on all of the questions you hear.

Determine whether or not they were open-ended when you go to score the tool.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Child-Centered Space

When to use:

This scale envisions a space that is arranged for a

particular type of program that uses interest centers where children select and direct their own work.

Mostly for younger children (K-2).
Interest centers: Supplies set in particular corner,

space, set apart. Has variety of supplies around a particular theme, type of activity.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Responsibility

2nd item should be scored based on evidence from

1st item.

Focus here is on routine activities – distinct from

group process skills or projects.

 Tasks that support communal responsibility.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Managing Feelings

More focused on feelings than Youth PQA

Reframing Conflict. (That’s why it’s in the Interaction domain.)

Determine whether or not to score the scale. If no

strong feelings or serious negative behavior that disrupts learning, do not score. Put Xs.

If scoring, fit the real scenario to the scoring criteria

the best you can.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Belonging

Item 1: Look for structured opportunities to get to

know each other better.

Item 2: Carefully observe how children interact with

each other.

 Observe complete interactions whenever possible.

Item 3: Can be a difficult item for programs to

achieve a high assessment score.

 “Identify strongly” = identify positively

SLIDE 4

4

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Leadership

Item 1: Practice group process skills.

 If staff sets up small groups, that is automatically an opportunity for group process skills.  Group process skills may also be practiced in large group setting.

Item 2: Opportunities to help another child.

 One on one.  Demonstrating or explaining something in which one child has more experience, knowledge, or skill.

Item 3: Structured opportunity to lead group.

 “group” = 2 or more others (including whole group).  Structured = staff sets up leadership role, may ask children to choose a leader.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Interaction with Adults

Item 1: Staff at eye level.

 Sitting on the floor or at a table, bending, leaning down, squatting.  If children are almost as tall as the staff person, give benefit of doubt and score a 5 anyway.

Item 2: Staff works side-by-side.

 This means doing or participating in the same activity as the children.

Item 3: Staff circulates.

 “Interacts” may be a brief acknowledgment of their work, comment or a question. Circulates—wanders around room to where the children are.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Planning

Planning happens before an activity.
Can be short or long-term.
Item 2: Difficult to achieve an assessment of

5.

Verbal brainstorming is most common

strategy.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for School-Age Choice

Item 2: Type of choice

 Discrete = from a list of options or choices  Open-ended = not a limited or specified number of choices

Item 3: Authentic choice—any non-trivial choice

 All, some, or no children have choice.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Tips for Reflection

Reflection happens after an activity.
Item 3: Feedback is children offering

advice/information about the activities to the staff.

David P. Weikart Center for children Program Quality : School-Age PQA External Assessor Reliability Training

Thank you!

Feel free to contact the Weikart Center with any

questions you may have.

Barb Hillaker – Research Consultant