WELCOME to Examiner Training 2014! Day 1 1 Before We Begin. - - PowerPoint PPT Presentation

WELCOME

to Examiner Training 2014!

Day 1

Before We Begin….

Logistics

• Safety: Note location of exits
• Phones, Restrooms
• Please Silence Phones/Pagers/Alerts
• Parking Lot

Parking Lot

Quality Texas Meeting Behaviors

• Be respectful
• Share openly
• Have only one conversation

at a time

• Mute phones
• Return from breaks on time
• Take care of personal needs
• Clean up after yourself
• Stay on point
• Work toward consensus
• Understand that silence means

affirmation

• Employ “ELMO” as needed
• Have fun

Ground und Rul ules

Agenda for Day 1

• Introductions
• About Quality Texas
• Code of Conduct
• The Assessment Process
• Introduction to the Criteria
• Core Values
• Key Factors
• Practice 1 – Using the Six-Step Process with your

Pre-Work Assignment

Learning Outcomes: You Will. . .

• Participate actively in Examiner Training
• Understand the roles and responsibilities of

Examiners

• Gain knowledge of the key elements in the Criteria

for Performance Excellence

• Gain knowledge of the process to evaluate an

application at any one of four levels i.e., Award, Progress, Commitment, Engagement.

• Write feedback comments that add value for the

applicant

Introductions – At Your Table

• Who you are!

– Name / City

• What you do!

– Organization – Job title

• Experience with QTF/Baldrige Criteria
• Your expectations of this Training

About Quality Texas

Quality Texas Foundation

Lynn Tomaszewski, Chief Executive Officer

ltomaszewski@qualitytexas.org

Kristin Krenzer, Director, Performance Excellence Program

kkrenzer@qualitytexas.org

5201 N. O'Connor Blvd. Suite 100 - Level B1 Irving, Texas 75039 www.texas-quality.org 469-828-1157

Key Facts About Quality Texas Foundation

• Founded in 1994 as a 501(c)3 non-profit corporation
• Endorsed by the Texas Governor’s office
• Full-time staff of two people
• Over 300 state-wide volunteers do most of the work
• Funded through memberships, customized training

workshops, Performance Excellence Program participant fees, annual Texas Quest Conference

• Recognized as one of the most effective state programs,

with more Baldrige recipients than any other state

QTF’s Vision and Mission

Vision (What We're Striving To Do): By helping our customers develop successful

• rganizations, we improve our communities, our state,
• ur country, and our world.

Mission (Why We Exist): We help create excellent businesses, hospitals and schools by sharing knowledge about the Baldrige process and other methods that drive organizational efficiency, effectiveness and sustainability.

Services We Provide

• Assessment and Feedback
• Award and Recognition
• Training and Coaching Solutions

– Customized training and coaching based on customer needs within their organization (arranged through Quality Texas CEO) – Public training workshops

• Examiner Training (You are here!)
• Applicant Training
• Membership Levels w/various benefits per level

– Corporate and Individual options

Why Organizations Apply

• Reflects organizational goals, objectives, and values
• Helps build a common, holistic view of the organization (a

“system” perspective)

• Provides a common language and method to examine

processes and performance

• Receives objective feedback
• Tracks progress

– Get to the “next level” of performance

• Fosters benchmarking within and across industries

Code of Conduct - Examiner Ethics

Code of Conduct: Four Key Principles

• Protect the integrity of the

assessment and feedback process

• Exhibit professional conduct at

all times

• Protect the promise of

confidentiality

• Protect the program’s

intellectual property

Ethics Excursion #1

I’d like to have a trusted employee in my own company help me with all of the applicant information that goes back and forth between the Quality Texas office and me as an Examiner. My secretary has worked with me for the past eight years and has attended many classes on how to handle confidential information through our company training department. He handles information far more sensitive and confidential than the information I receive from the Quality Texas office. I would like to dictate my scorebook and let him type it up for me. He types much faster than I can, so it wouldn’t take as long to complete my work. I’d trust him with my life.

Ethics Excursion # 2

My Stage 1 application has some information in there that I find hard to believe. I’m from the same industry and their Return On Assets is more than twice my company’s – and we think we’re doing pretty well. I’m thinking of calling the applicant’s Official Contact Point to get an explanation before I finish writing my scorebook.

QTF Organizational Integrity

• Notify the Quality Texas CEO of violations to

the Code of Ethical Conduct

• Ethics Committee consists of:

– Ethics Officer – Quality Texas Chief Executive Officer – Quality Texas Director Performance Excellence Program

Quality Texas

The Assessment Process

A six-step process, using: – Three phases in the assessment process

• Independent Review
• Consensus
• Site Visit

– One of Four levels of applications

• Engagement
• Commitment
• Progress
• Award

– One of Three Criteria versions

• Business and Non-Profit
• Healthcare
• Education

Three Criteria Versions

• Business, Manufacturing, Service,

Government, and Non-profit

• Healthcare
• Education

ASSESSMENT PROCESS

RESO ESOURCES ES NEX EXT ST STEP EPS

• FEEDBACK REPORT FOR

• RECOMMENDATION TO

• JUDGES REVIEW
• BOARD OF DIRECTORS DECISIONS
• FEEDBACK TO APPLICANT
• AWARDS CEREMONY
• Criteria Books
• Application
• QTF Website
• QTF Staff

COR ORE V VALUES

• Visionary Leadership
• Customer-driven excellence
• Organization & Personal Learning
• Valuing Workforce Members / Partners
• Score
• ID Key Factors

Strengths & OFIs

• Determination of

Site Visit Issues

• Consensus Scorebook
• Category Champion

leads Consensus discussion and scoring

• Feedback Report
• Verify / Clarify
• Training
• Agility
• Focus on the Future
• Managing for Innovation
• Management by Fact
• Societal Responsibility
• Focus on Results and Creating Value
• Systems Perspective
• Review Criteria
• Read Application
• Attend Training
• Training
• Scoring Guidelines
• Scorebook Navigator

MBNQA W b i

APPLICATION LEVELS

RESOURCES

SCOR

ORING F FACTOR ORS

Process

Resu sults

COMMITMENT

PROGRESS

AWARD

• Org. Profile
• 10 Page App.
• No Site Visit
• Feedback

• Criteria
• Training
• Scorebook

Navigator

• Application

SCORES

PERFORMANCE EXCELLENCE

STRENGTHS OPPORTUNITIES FOR IMPROVEMENT

• Org. Profile
• 15 Page App.
• No Site Visit
• Feedback

• Org. Profile
• 25 Page App.
• Site Visit (if

• Feedback

• Org. Profile
• 50 Page App.
• Site Visit
• Feedback

AP APPLI LICAT ATIONS RECEIVED Oct ctober- March ch ( (5 cy cycle cles) OC OCTOB OBER

Examiners’ Roles and Responsibilities

• Provide analysis and insight to lead the
• rganization to the next level of maturity
• Assessment Process Team Members

– Examiners – Senior Examiners – Scorebook Editor – Back-up Team Leader – Team Leader – Process Coach

• Each team has a Subject Matter

Expert (Can be any role)

Benefits of Being an Examiner

• Learn the Quality Texas/Baldrige Performance Excellence Criteria

through training and experience

• Learn validated best practices from leading organizations in all

industry sectors

• Expand professional network
• Develop assessment, analysis, writing, teamwork and leadership

skills

• Earn a valuable professional credential
• Give back to the community by helping organizations be

successful through useful feedback

Conditions of Involvement Absolute Essentials to the Credibility, Success and Prestige of the

Assessment and Feedback Process

• Commit to the entire process (until the feedback

is presented to the applicant)

• Maintain Confidentiality
• Complete the Training and Case Study
• Honor Time Commitments
• Represent Quality Texas (not your organization)
• Follow Code of Conduct
• Avoid Conflicts of Interest

Value for the Examiner and the Sponsor

• Work with a diverse team to reach consensus
• Network with other quality and business experts to build

professional friendships

• Understand and apply the Criteria for Performance

Excellence to a variety of organizations

• Develop analytical and consensus-building skills for

personal use.

• Attend annual Texas Award Banquet and Conference (at a

special discounted rate) – June, 2015 (dates TBA)

• Special recognition at conference

Introduction to the Criteria

1 - Leadership 2 - Strategic Planning 3 - Customer Focus 6 – Operations Focus 7 - Results 4 - Measurement, Analysis and Knowledge Management 5 - Workforce Focus

The Criteria Framework:

A Systems Perspective

Organizational Profile: Environment, Relationships, and Challenges

Key Characteristics of the Criteria

• Focus on results
• Are non-prescriptive and adaptable
• Support a systems perspective
• Support goal-based diagnosis

Criteria Structure

Categories 1-7 Items (17 Items 1.1 – 7.5) Areas to Address (a, b, etc.)

More Specificity

Core Values and Concepts

Multiple Requirements (Specific Questions within areas to address)

Criteria Terminology

Core Values and Concepts

• Visionary Leadership
• Customer-Driven Excellence
• Organizational/Personal Learning
• Valuing Employees and Partners
• Agility
• Focus on the Future
• Managing for Innovation
• Management by Fact
• Societal Responsibility
• Focus on Results and

Creating Value

• Systems Perspective

Bre Break

Activity Time (min)

1. Select a scribe, timekeeper and reporter 1 2. The facilitator will assign your table a core value; individually, review your assigned core value from the Criteria booklet

Table 1: Visionary Leadership Table 7: Systems Perspective Table 2: Customer-focused Excellence Table 8: Management by Fact Table 3: Agility Table 9: Valuing Workforce Table 4: Managing for Innovation Table 10: Organizational Learning Table 5: Focus on the Future Table 11: Societal Responsibility Table 6: Focus on Results and Creating Value

2 3. As a group, discuss the significance of the core value in the Criteria 5 4. Capture the “gist” of your assigned core value in 5 to 7 “Bullet Points” on chart paper 5 5. Summary (by Facilitator with table input) 10

Exercise: Core Values and Concepts

Some Key Concepts…

• What vs. How?
• Systematic Process
• Linkages

“How” vs. “What” Questions

“How” questions ask the applicant to describe key process information.

A process can be described like this: Or like this:

 Step A  Step B  …Step N

Process Descriptions

Systematic Process Defined

Defined Measured Repeatable Improved Shared

Approaches that are

• well-ordered
• repeatable
• use data and

information Systematic processes build in the opportunity for evaluation, improvement, and sharing.

“What” Questions Ask the Applicant

to Give Specific Information On Plans, Objectives, Goals or Results

Independent Review – How Do I Start?

• 1. Review the appropriate Criteria booklet
• Core Values and Concepts
• Criteria Requirements
• Criteria Response Guidelines
• Scoring System and Scoring Guidelines
• 2. Read the entire application
• 3. Highlight key points or approaches
• 4. Develop your Key Factors list
• 5. Use the six-step process to evaluate the

applicant’s response to each Item

• Organizational Environment

– Product offerings – Vision and mission

• Core competencies

– Workforce profile – Assets – Regulatory requirements

• Organizational Relationships

– Organizational structure – Customers and stakeholders – Suppliers and partners

• Competitive Environment

– Competitive position – Competitiveness changes – Comparative data

• Strategic Context
• Strategic challenges and

advantages

• Performance Improvement

System

P.1 Organizational Description P.2 Organizational Situation

The Organizational Profile (OP)

Key Factors

• Provide a concise summary of the most important aspects of

the applicant’s organizational environment

• Describe significant facts or aspects about the organization
• Help focus the criteria in terms of its

relevance to the organization

• Key factors can come from the Organizational Profile, eligibility

forms, application form, or anywhere in the application document.

How to Use the Organizational Profile (OP)

• Develop a short, bullet point list of “Key Factors”
• Choose 4-6 Key Factors previously developed that

pertain to the Criteria questions for each Item

• Develop observations, comments, and scoring based
• n the application’s response to the questions, using

the Key Factors as a filter to ensure significance and relevance of their comments and accuracy of their scoring

Exercise: Key Factors

Activity Time (min) 1. Select a scribe, timekeeper and reporter 1 2. Review as a group assigned section of Organizational Profile

Table 1 P1.a (1,2 ,& 3) Table 2 P1.a (4 & 5) Table 3 P1.b (1 & 2) Table 4 P1.b (3) Table 5 P2.a (1, 2, & 3) Table 6 P2.b & c

5 3. Discuss at your table the Key Factors you have identified in the Case Study for your assigned Item 10 4. Record the Key Factors on chart paper 3 5. Report Out (1 minute per table) 10

How Do I Evaluate Process Items?

Process items are evaluated using four factors:

• Approach
• Deployment
• Learning
• Integration

Approach (A)

“Approach” refers to

• Appropriateness of methods
• Effectiveness of the methods
• The degree to which the

approach is repeatable and based on reliable data and information (i.e., systematic)

Deployment (D)

Deployment refers to the extent to which the approach is applied

• In addressing Item requirements relevant and important to

the organization

• The approach is applied

consistently

• The approach is used by

all appropriate work units

Learning (L)

• Refinement in approach
• Cycles of evaluation and improvement
• Breakthrough change through innovation
• Sharing of refinements and

innovation with others in the organization

Integration (I)

• Alignment with organizational needs
• Alignment with stakeholder needs
• Measures, information, and improvement systems

are complementary across processes and work units

• Plans, processes, measures,

and results are “harmonized” across the organization

What We’ve Learned

Step 1: Read the Criteria

Page 7 in Your Guide

Exercise: Step 1 Read the Criteria

Activity Time (min) 1. Select a scribe, timekeeper and reporter (1) 2. Review individually your assigned Criteria Item 3 3. Discuss at your table as a group your thoughts on the key requirements for this Item 4. Record the requirements on chart paper 7 5. Report Out (1 minute per table) 10

20 Minutes Total

Step 2: Determine Most Relevant Key Factors

Page 8 in Your Guide

Exercise: Step 2 Determine the Most Relevant Key Factors

Activity Time (min) 1. Select a scribe, timekeeper and reporter (1) 2. Review individually your assigned pre-work Criteria Item 3 3. Discuss at your table as a group and come to agreement on the relevant 4 to 6 key factors for your assigned item. 4. Write your selected key factors on a flip chart for use in subsequent exercises (capture key words and phrases) 12 5. Report Out (1 minute per table) 15

30 Minutes Total

Step 3: Read and Analyze the Application

Page 14 in Your Guide

Exercise: Step 3 Read and Analyze the Application

Activity Time (min) 1. Select a scribe, timekeeper and reporter (1) 2. Review individually your assigned Criteria Item 10 3. Discuss at your table as a group your observations

• n the applicant’s response against the Criteria

questions. 10 4. Report Out (1 minute per table) 5

25 Minutes Total

Bre Break

Step 4: Identify Strengths and OFIs

Page 15 in Your Guide

Exercise: Step 4 Identify Strengths and OFIs

Activity Time (min) 1. Select a scribe, timekeeper and reporter (1) 2. Use a round robin approach to discuss all potential strengths and OFIs identified by your table mates 10 3. From these, discuss at your table and select as a group, a total of around 6 strengths and OFIs that you feel are most important for the applicant 4. Record these on chart paper 15 5. Report Out (1 minute per table) 5

30 Minutes Total

Comment Writing

Elements of a well written comment

The Feedback Report: The Final Product

The report should be:

• Useful to the applicant

– Comments should be clear and actionable – Focused on the applicant’s most important key factors

• Encouraging

– Not adversarial (it’s not an audit) – Help applicant reach the next level – Make applicant an advocate for the program

• Respectful

– Polite in tone – Not judgmental or prescriptive

How to Give Useful Feedback

Do:

• Base your comments on the Criteria
• Reference the evaluation factors
• Include an opening “nugget” to give

the comment significance

• Keep each comment to a single issue
• Make Key Factor references
• Be accurate; check your statements
• Recognize the page limitations
• Give benefit of the doubt
• Be polite in tone
• Include a few examples
• Check spelling and grammar

Don’t:

• Be judgmental or prescriptive
• Forget linkages
• Forget the Key Factors
• Stray from the Criteria
• Make conflicting strength and OFI

statements

• Forget to:

– Check your score against the balance and content of comments – Check your facts – Check grammar and spelling

• Forget you’re providing a service

Well-Written Comments: “NERD”

• N – Nugget – Up front comment based on the Item

Criteria or a Key Factor (i.e., explains why this is important to the applicant)

• E – Evidence or Example - State what the applicant

has (Strength) or is missing or not addressed (OFI)

• R – Relevance – Why is this comment important? If

an OFI, include actionable (but not prescriptive) information that will help the applicant to improve.

• D – Done – Read to see if comment makes sense and

helps the organization understand its current status

Sample Process Strength

2.2a(4) To mitigate the strategic challenge of competitors wishing to hire its engaged workforce, the applicant ensures that workforce plans support any needed changes. For example, through data and budget analysis and surveys, the applicant takes a proactive approach to workforce capacity and allocates instructional staff to areas of greatest need through “vertical teamwork.”

Nugget Examples Relevance

Sample Process OFI

4.2a(1) It is not clear how the applicant systematically transfers knowledge specific to the needs of parents and volunteers in support of the PhilP that all are accountable for student

• performance. For example, parents and volunteers do not

appear to be included in teachers’ grade-level discussions, and parents do not appear to have access to teachers’ online forums, blogs, and classroom support server (Figure 4.2-1).

Example Nugget Relevance

See pages 16 – 18 for comment writing guidelines

Step 5: Write Feedback Ready Comments

Page 19 in Your Guide

Exercise: Step 5 Write Feedback Ready Comments

Activity Time (min) 1. Select a scribe, timekeeper and reporter (1) 2. Draft feedback ready strength and OFI 3. Give feedback to table mates 4. Redraft the comment, using their input 5. Record the final feedback ready comment on chart paper 15 5 10 6. Report Out (1 minute per table) 10

40 Minutes Total

Introduction to Scoring

• Process Scoring Guidelines and Results Scoring

Guidelines

• Process Scores Address: Approach, Deployment,

Learning, Integration (ADLI)

• Results Scores Address:

Levels, Trends, Comparisons, Integration (LeTCI)

8.5 6.5 8.0

Process Item Scoring Guidelines

• f the Item, is evident.

• f the Item, is evident.

• rientation is not evident;

• rientation are evident.

• rganizational learning,

• rganizational-level

• rganizational learning

• rganization-wide tools;

• rganization.

• rganizational needs

• rganizational needs

• ther Process Items.

• rganizational needs

Step 6: Determine the Scoring Range and Score

Page 21 in Your Guide

Exercise: Step 6 Scoring

Activity Time (min) 1. Select a scribe, timekeeper and reporter (1) 2. Using your comments from Step 4 and the Scoring Guidelines, discuss as a group the most appropriate scoring range for your assigned Item 3. Next, discuss the appropriate score within that range 4. Record the scoring range and score on chart paper 10 5. Report Out (1 minute per table) 5

15 Minutes Total

Day 1 Reflections – Closing Activity

• Work as a group at your table to

develop the following:

– One or Two “Burning Questions” you still have about your role as an Examiner for Quality Texas – One or Two Key Points you learned today

Parking Lot

Day 2 Preview

• Evaluate more process Items
• Evaluate your pre-work results Item

Complete Training Evaluation Forms AND Plus/Delta