WELCOME 2019 EXAMINERS! Examiner Training Day 2 1 Welcome Back! - - PowerPoint PPT Presentation

WELCOME 2019 EXAMINERS!

Examiner Training – Day 2

1

Welcome Back!

• Review Ground Rules
• Quench any burning issues
• Debrief Quiz
• Icebreaker

PREPARING RESULTS ITEMS FOR CONSENSUS

“However beautiful the strategy, you should occasionally look at the results.”

Winston Churchill

Review for Conflict Log on to the system Review the Framework Read the Application Create Key Factor List

Click “Done”! Prepare Consensus items Participate in Consensus meeting Prepare Site Visit issues Complete Site Visit Prepare Final Scorebook

12 process items

• Criteria
• Key Factors
• Analyze: PROCESS

(ADLI)

• 6-10 Observations
• Score

5 results items

• Criteria
• Key Factors
• Analyze: RESULTS

(LeTCI)

• 6-10 Observations
• Score

IR Review

4

How to Analyze Results Items

• Group a set of results, rather than

evaluating each graph separately

• Identify gaps (results not provided) from:

‒ Criteria ‒ Key factors ‒ Key processes you noted in categories 1-6

5

Analyze Results that Are Provided:

• Levels

– Validity or meaningful scale

• Trends *

– Rate of improvement or sustainability

• Comparisons

– Appropriateness; benchmarks

• Integration

– Importance, harmonization, segmentation

* Note: Unfavorable trends may be explained in the narrative

6

Alignment of Item Questions

P.1a(3) – Workforce Profile

What is your WORKFORCE profile . . . employee groups and SEGMENTS? . . . What are the KEY drivers that engage them. . .? What are your organization’s special health and safety requirements?

5.1b(1) – Workforce Climate

HOW do you ensure workplace health, security, and accessibility for the

WORKFORCE? What are your PERFORMANCE MEASURES and improvement GOALS

for your workplace environmental factors? . . .

7.3a(2) – Workforce Results (Climate)

What are your RESULTS for KEY MEASURES or INDICATORS of your WORKFORCE climate, including those for WORKFORCE health, security, accessibility, and services and benefits, as appropriate? . . .

7

LifeBridge Workforce Profile

8

5.1b(1) LOTS systematically assesses workplace environmental factors to ensure and improve workforce health by using a comprehensive wellness

• program. Components of this comprehensive wellness program include

exceptional health insurance packages, $20/month reimbursement towards wellness-related expenses, no-cost wellness screenings (Figure 7.3-5) and flu shots, 24/7 access to an employee assistance program, and

• ther health opportunities. Workplace environmental differences are not

pertinent for these benefits; these programs encourage employees to become more proactive in improving their overall health. … Security … Accessibility A sample listing of performance measures for workforce health, security, and workplace accessibility can be found in Workplace Environment (Figure 5.1-5) and Workforce Preparedness (Figure 7.3-6). These performance measures are designed to provide a work environment conducive to supporting the LOTS workforce to accomplish the mission.

9

LifeBridge Results

11

LifeBridge Results

12

Outlining Results Observations

• Based on your analysis, identify 6-10 strength

and OFI observations that are most valuable to the applicant

• Think vertically (Criteria items, B/O/M) and

horizontally (evaluation factors)

• Note relevant figure number(s)
• Note whether results are favorable or

unfavorable

• Note item reference, relevant key factors

13

Results Items

7.1 Product and Process Results 7.2 Customer Results 7.3 Workforce Results 7.4 Leadership and Governance Results 7.5 Financial, Market, and Strategy Results

14

Process/Results Linkages

Category Linkages Results 1 – Leadership 7.4 2 – Strategy 7.5 3 – Customers 7.2 4 – Measurement, Analysis, and Knowledge Management 7.1-7.5 5 – Workforce 7.3 6 – Operations 7.1 ** Sprinkled throughout categories 7.5

15

Preparing an Item for Consensus Meeting

16

• 1. Review the Criteria questions
• 2. Assign 4-6 key factors
• 3. Read applicant’s response
• 4. Review team members’ Independent Reviews

SYNTHESIZE

Synthesize team members’ IR into 6-10 comment topics Develop rationale: B/O/M, evaluation factors, and initials

WRITE

Draft 6-10 feedback comments based on your chosen topics Use TEST

SCORE

Score the item based

• n your consensus

review

REVISE

Revise comments based on feedback from your backup Provide feedback to your partner

PREPARE

Prepare to present your items during the team’s consensus meeting

What Should I Synthesize?

Strength and OFI observations for:

– Performance Levels (Le) – Trends (T) – Use of Comparisons, performance to benchmarks (C) – Alignment or Integration with organizational needs and process categories (I)

Gaps

– When a key Criteria question is not addressed – When something the applicant has identified as important is not addressed (KF) – When an expected result is not presented

17

Remember…

• It is your job to select the MOST insightful

6–10 pieces of information . . .

• Not all Criteria questions are equally

important to every applicant!

18

Results Comment Tips

• Group the graphs (in some meaningful way) and

summarize your analysis

• Relate results to the applicant’s:

– Key factors – Strategies – Processes/results described elsewhere

• BUT . . . comment on the results, not the process of
• btaining the results

19

Results Comment Tips

• Comparisons – comment on:

– Performance relative to comparisons – Appropriateness of comparisons – The fact that comparisons are not provided

• Use comparative terms that are clear:

– Trends are favorable/unfavorable, beneficial/adverse – Results have improved/declined – Performance is better/worse

• Comment on segmentation

– Performance of specific segments – The fact segmented data are not provided in key areas

20

Sample Consensus Results Strength

7.3a(3) Comment: LOTS demonstrates consistent levels, trends, and comparisons for its employee engagement results. For employee connection to mission (Figure 7.3-11), which measures staff members' understanding of how their individual positions contribute to the mission, results are at 100%, which exceed the Excel Employee Engagement benchmark. In addition, overall workforce engagement levels are consistent with the benchmark results from 2016 to 2018. Results in this area reinforce the organization's core competency of a mission-driven workforce that is highly motivated and engaged in saving lives. Rationale: [AY] They address the multiple level questions but do not meet all

• f the multiple level questions related to Diversity of the Workforce,

segments or groups. See OFI in this item related to what is missing. Le, T, I identified in a favorable manner

21

Sample Consensus Results OFI

7.3a(3) Comment: The workforce-focused performance results given are not segmented by the groups noted in Figure P.1-4. For example, workforce engagement results are not segmented by department, gender, tenure, or job type. Segmentation of results such as overall job satisfaction (Figure 7.3-14) and retention (Figure 7.3-13) may enable LOTS to monitor differences among these groups and make improvements to support its strategic advantage of a supportive culture and address the challenge of retention. Rationale: Supported by ST, CJ, and AY. This is directly related to the I

• f the Basic level question.

22

Goals vs. Projections

GOALS

“Hoped for” performance In general, we do NOT evaluate performance against goals:

• We don’t know how the

applicant sets goals

• Do they set low goals or

stretch goals?

PROJECTIONS

“Expected” performance Examine relevance:

• Do projections make sense?
• Are they related to the

strategic plan?

23

Scoring Results Items

• 1. Consider your 6-10 comments paying special

attention to Le-T-C-I and B/O/M

• 2. Determine the scoring range that “best fits”

the item

• 3. Score the item

24

25

Scoring Example – Item 7.3

Overall scoring range: 30-45% Item score: 45% Rationale: Two strengths and five OFIs, no double strengths or OFIs. One strength is very specific and the other strength addresses the multiple level questions without meeting all of them. OFIs are across the board from basic to multiple level questions. Levels: 50-65% with good organizational performance levels reported responsive to the overall questions. Trends: 50- 65% with beneficial trends evident in areas important to the organization’s mission Comparisons: 50-65% Some current performance levels have been evaluated against relevant comparisons which show good relative

• performance. Integration: 10-25% Results are reported for a few areas of

importance to the accomplishment of the organizations mission. Little

• segmentation. The missing measures, lack of segmentation and OFIs related

to 7.3-1 are a barrier to the 50-65% range.

26

Assignment

27

Table Assignment 1 7.1 2 7.2 3 7.1 4 7.2 5 7.1 6 7.2 7 7.1 8 7.2

Preparing an Item for Consensus Meeting

28

• 1. Review the Criteria questions
• 2. Assign 4-6 key factors
• 3. Read applicant’s response
• 4. Review team members’ Independent Reviews

SYNTHESIZE

Synthesize team members’ IR into 6-10 comment topics Develop rationale: B/O/M, evaluation factors, and initials

WRITE

Draft 6-10 feedback comments based on your chosen topics Use TEST

SCORE

Score the item based

• n your consensus

review

REVISE

Revise comments based on feedback from your backup Provide feedback to your partner

PREPARE

Prepare to present your items during the team’s consensus meeting

Individual Exercise

29

Set your timetable: (pgs. 93-97)

• Review Criteria questions

2 min.

• Identify 4-6 relevant key factors

5 min.

• Review your homework and the team’s IR to find

commonalities and conflicts for your assigned item

• Select 6 topics (most important key messages) to

become strength and OFI comments

• Note your rationale for each comment

30 min.

• Write 2 comments (1 strength + 1 OFI)

15 min.

• Score the item

5 min.

Preparing an Item for Consensus Meeting

• 1. Review the Criteria questions
• 2. Assign 4-6 key factors
• 3. Read applicant’s response
• 4. Review team members’ Independent Reviews

SYNTHESIZE

Synthesize team members’ IR into 6-10 comment topics Develop rationale: B/O/M, evaluation factors, and initials

WRITE

Draft 6-10 feedback comments based on your chosen topics Use TEST

SCORE

Score the item based

• n your consensus

review

REVISE

Revise comments based on feedback from your backup Provide feedback to your partner

PREPARE

Prepare to present your items during the team’s consensus meeting

30

Exercise

• Pass your comments to the person to your right
• In the right-hand column, give feedback to your teammate:

– Is the comment Criteria based? – Can you understand it? What would improve it? – What is the topic sentence? – Are there examples from the application? – So what?

• As a table, agree on the item score
• Agree on one key learning to share

32

33

Overall and Multiple Questions

In Which Scoring Range…

(Le) Must the applicant’s results be responsive to the multiple questions? (T) Are “some” adverse trends acceptable? (C) Is the applicant in the “early stages” of

• btaining comparative data?

(I) Are projections required?

35

CONSENSUS MEETING

“If everyone is moving forward together,

then success takes care of itself.”

Henry Ford

The Importance of Teamwork

37

What is Consensus?

• Everyone can support the team’s decision.
• When you leave the meeting will you say

“we decided” or “they decided”?

38

Preparing an Item for Consensus Meeting

• 1. Review the Criteria questions
• 2. Assign 4-6 key factors
• 3. Read applicant’s response
• 4. Review team members’ Independent Reviews

SYNTHESIZE

Synthesize team members’ IR into 6-10 comment topics Develop rationale: B/O/M, evaluation factors, and initials

WRITE

Draft 6-10 feedback comments based on your chosen topics Use TEST

SCORE

Score the item based

• n your consensus

review

REVISE

Revise comments based on feedback from your backup Provide feedback to your partner

PREPARE

Prepare to present your items during the team’s consensus meeting

39

Purpose of Consensus

• Reach agreement on:

– Key factors – Strengths and OFIs – Scores

• Agree on site visit issues
• Begin to develop key themes

40

Sample Roles

Examiner Item Lead Back-up Other 1 1.1, 1.2, 7.4 6.1, 6.2, 7.1 Team leader 2 2.1, 2.2 5.1, 5.2, 7.3 Scorebook editor 3 3.1, 3.2, 7.2 4.1, 4.2, 7.5 Back-up team leader 4 4.1, 4.2, 7.5 3.1, 3.2, 7.2 Criteria cop 5 5.1, 5.2, 7.3 2.1, 2.2 Arrange logistics 6 6.1, 6.2, 7.1 1.1, 1.2, 7.4 Time keeper

41

Topics for Consensus Meeting

• Changes to the language of a comment that affect:

– The meaning of the comment – The A-D-L-I or Le-T-C-I assessment factors that pertain – Basic/Overall/Multiple level of a comment – Distribution of strengths and OFIs at B/O/M – The score – Key themes

• Comments that are not “key” or Criteria-based
• If a “key” area to address is missed
• Conflicts between strengths & OFIs, or comments in different

categories

• Potential site visit issues

42

NOT Topics for Consensus Meeting

• Anything that does not alter the meaning of a comment or the

score

• Grammatical errors

– Typos – Misspellings – Word format changes – Extra words or words left out – Sentences or clauses that are redundant – Substituting acronyms for entity names, and the reverse – Separating the clauses of a lengthy sentence to make shorter sentences.

43

Exercise

44

Tips for Consensus Meeting

• Be prepared
• Don’t be defensive – check your ego at the door!
• Begin with an ice breaker
• Set ground rules

– Does silence mean agreement?

• Use a round robin approach to be sure everyone’s

voice is heard

• Only discuss areas of disagreement

45

KEY THEMES

“If you just focus on the smallest details, you never get the big picture right.” Leroy Hood

Key Theme Overview

• Executive summary of feedback report for applicant’s senior

leaders

• Strategic in nature
• Summarize the most vital issues for the organization as a

whole – e.g., what must the organization do to become/remain competitive and ensure long term sustainability?

• Typically prepared by team leader or scorebook editor with

support from team members

• Used by examiners and judges to get overview of applicant

maturity

47

What Is a Key Theme?

• A synthesis of multiple comments

– Don’t just cut and paste

• A perception or observation that:

– Recurs and is expanded upon throughout the scorebook – Is common (cross-cutting) to more than one item/category – Is of high value or significance to the applicant – May address a core value of the Criteria – Is traceable to individual item comments

48

49

Key Theme Valuing Employees 1.1 – PEP and annual goal plans, Communication processes 5.2 – Leadership System reinforces

• pen culture and

engagement 2.1 – deployment of strategy to workforce

+ + +

Key Themes Reflect

• a. The most important strengths or outstanding practices (of

potential value to other organizations) found in processes

• b. The most significant opportunities, concerns, or

vulnerabilities found in processes c. The most significant strengths (related to data, comparisons, and integration) found in results

• d. The most significant opportunities, vulnerabilities, and/ or

gaps (related to data, comparisons, and integration) found in results

50

Sample Key Theme (a)

LOTS has implemented multiple approaches that demonstrate that it values its employees, strengthening the organization’s core competency of a mission-driven workforce. Beginning at the top, the leadership team models ethical behavior and uses multiple touchpoints to communicate with the workforce. Workforce members’ engagement in the success of the organization is supported through the cascading of strategic goals and by addressing their needs via various benefit-plan options and promoting a safe operating environment. Additionally, LOTS has designed the Learning and Development System to improve workforce learning and then uses multidisciplinary meetings and Rounding for Outcomes to capture and disseminate valuable workforce knowledge. All of these workforce-focused approaches highlight the philosophy “We take care of ‘Our People’ so they can take care of others.”

51

Steps in Developing Key Themes

• 1. Capture initial impressions on your first read-

through of the application (IR)

• 2. Use the “key” icon to flag potential key themes

during consensus review

• 3. Scorebook editor or other team member

develops draft topics for key themes for team review prior to consensus meeting and discussion during consensus meeting

• 4. TL and scorebook editor finalize key themes after

item comments are finalized (after site visit)

52

Core Values as Key Themes

• Core values may become the topic of a key

theme strength:

– Use core values to provide value – as the basis of a strength key theme comment – Do not use core values to assess the applicant – NOT as the basis of an OFI key theme comment

53

Other Sources of Key Themes

• Process items

– A-D-L-I evaluation factors – ++ and – comments

• Results items

– Le-T-C-I evaluation factors – ++ and -- comments

• Scoring guidelines
• Scoring band descriptors

54

Class Discussion

Does LifeBridge demonstrate any of the Baldrige core values? What other potential key themes might be relevant for LifeBridge?

55

Table Exercise Instructions

Using the Consensus Scorebook, assist your table anchor in drafting a key theme on the Key Themes Worksheet (p. 115-117)

56

Table Key Theme Where to Look Suggested Topics 1 & 5 A Process Strengths (Cat 1-6)

• Systems Perspective
• Data and Information processes, Integration

2 & 6 B Process OFIs (Cat 1-6)

• Missing cycles of learning for key processes
• Lack of processes to promote innovation and

intelligent risk-taking 3 & 7 C Results Strengths (Cat 7)

• Favorable comparisons for many key results
• Near perfect performance levels for most

important measures 4 & 8 D Results OFIs (Cat 7)

• Missing results
• Adverse trends

SITE VISIT PREPARATION

“What people say, what people do, and what they say they do are entirely different things.” Margaret Mead

Examiner Evaluation Cycle

Complete Pre-work & Training

Independent

Review Consensus Review Site Visit Review Finalize Feedback Report

58

Review for Conflict Log on to the system Review the Framework Read the Application Create Key Factor List

Click “Done”! Prepare Consensus items Participate in Consensus meeting Prepare Site Visit issues Complete Site Visit Prepare Final Scorebook

12 process items

• Criteria
• Key Factors
• Analyze: PROCESS

(ADLI)

• 6-10 Observations
• Score

5 results items

• Criteria
• Key Factors
• Analyze: RESULTS

(LeTCI)

• 6-10 Observations
• Score

Site Visit Review

59

Purpose of a Site Visit

• To learn more about the applicant

– Not just to get your questions answered!

• To clarify OFIs
• To verify role model strengths
• To allow the applicant to tell its story
• To meet the applicant’s expectations

60

Site Visit Phases

Planning On Site Post Site Visit

61

Site Visit Phases

Planning On Site Post Site Visit

62

Prior to Planning Phase…

• Begin logistics as soon as teams are formed

– Dates of availability for team – Dates of availability for applicant – Travel arrangements

• Potential site visit dates:

– September 15 – October 4

63

Stair Steps to Excellence

64

X X

Planning Phase

Reread Org Profile/ Key Factors, and Key Themes to get grounded Discuss Site Visit agenda and approach during consensus meeting Watch site visit video Develop site visit issues and complete Scorebook Navigator form, SVI worksheets Send site visit issues to applicant 10-14 days prior to site visit

65

Site Visit Video

66

Selecting Site Visit Issues

• Use site visit issues to

– Verify role model strengths and – Clarify OFIs

• How many site visit issues do I need?

– As a general rule, prepare a site visit issue for each OFI and double strength (++) – Your team leader will assist you in determining the correct number of site visit issues based on

• the length of the site visit
• the size of the organization

67

Sample Site Visit Issues

“Verify LOTS’ use of the Workforce Planning Process (Figure 5.1-1) to address workforce change management.” “Clarify how LOTS ensures its workforce represents the diverse ideas and cultures of its hiring communities.”

68

69

70

Complete the Form in Scorebook Navigator

• The comment and rationale will auto-populate
• Enter any documents you would like to review
• Enter whom you will interview
• Write ONLY the SVI question (verify/clarify) in

the “potential questions” box

• You will complete the “Findings” box later
• Only complete this form for the comments

you have selected as site visit issues!

71

72

73

74

Next Steps

• Once team members have completed the site visit

forms in Scorebook Navigator, the team leader will pull a report including:

– Requested Documents – Potential Interviewees – SVI Questions (Verify/Clarify)

• Team leader edits and formats this list 10-14 days prior

to the site visit

• Team members then go back to prepare questions on

the SVI Worksheet (Word)

75

The Site Visit Issue Worksheet

• Complete a separate Site Visit Issue (SVI) Worksheet for each

site visit issue identified

– Download template from TNCPE website – Sample SVI for Item 5.1a(2) on pg. 133

• Complete the entire front page of the worksheet prior to site

visit:

– Site visit issue (verify or clarify statement) – Comment(s) affected (cross-walk to Scorebook Navigator) – Strategy (A-D-L-I or Le-T-C-I questions)

• You will use the worksheet to conduct your interviews and

take notes on site

76

What Questions Should I Ask?

• Use your process evaluation factors (A-D-L-I) to

“move the applicant to the right”

• If your site visit issue clarifies the applicant’s

approach, be prepared with follow-up questions regarding deployment, learning, and integration.

• This approach works for results too (Le-T-C-I)

most of the time.

77

Let’s Review…

• Complete SVI form in Scorebook Navigator first

– Documents, interviewees – Site Visit Issue only (verify/clarify questions)

• Then complete a SVI worksheet in Word for each issue
• The applicant will receive your site visit issues

(verify/clarify statements) and requested documents but not your prepared A-D-L-I questions or strategy

• Prioritize your SVI worksheets in case you run out of

time

78

Assignment

Table Item 1 3.2 2 4.2 3 3.2 4 4.2 5 3.2 6 4.2 7 3.2 8 4.2

79

Table Exercise

• Review Criteria questions, key factors and comments for

your item.

• Using the Consensus Scorebook, make note of key

processes LifeBridge uses to address Criteria questions.

• Identify one OFI site visit issue and complete:

– The SVI form in Scorebook Navigator (poster)

• Documents
• Interviewees
• SVI question (verify/clarify)

– The SVI Worksheet in Word (poster)

• A-D-L-I questions

80

Remember…

Trust the process!

81

What about Results Items?

• Process works the same way
• Identify SVI Issues
• Complete SVI form in Scorebook Navigator

– Documents – Interviewees – Verify/Clarify Questions

• Complete SVI Worksheet (Word)

– Still “move to the right” (Le-T-C-I) WHERE APPROPRIATE – You may discover lots of your “missing results”

82

THE SITE VISIT

Without continual growth and progress, such words as improvement, achievement, and success have no meaning. Benjamin Franklin

Site Visit Phases

Planning On Site Post Site Visit

84

On-Site Phase

Opening Meeting Category interviews Review of documents Team caucus Closing Meeting Remember, after site visit, “it is not clear” means you didn’t do your job!

85

Sample Level 3 Site Visit Schedule

Day 1 Day 2 8:00 am Opening Meeting Follow-up Category 1 Follow-up Category 6 9:30 am Category 1, Results Category 6, Results Follow-up Category 2 Follow-up Category 3 11:30 am Lunch/Caucus Lunch/Caucus Noon Category 2, Results Category 3, Results Follow-up Category 4 Follow-up Category 5 2:00 pm Category 4, Results Category 5, Results Visit other sites; review documentation Final Caucus 4:00 pm Examiner Caucus Closing Meeting

86

Opening Meeting

• Introductions are tricky
• Team Leader presents TNCPE Opening

Meeting slides

• Applicant may have a presentation
• Come ready to work!

87

Site Visit Interview Tips

• If the applicant gives examples or anecdotes, ask a follow up

question

• At the end of an interview ask “Is there anything else you’d like to

tell us?”

• Follow process questions with corresponding results questions
• Use evaluation factors (A-D-L-I and Le-T-C-I) to guide your questions
• Interview in pairs: one person conducts the interview and the other

takes notes

• Be committed to be an examiner for the entire

site visit

88

Ask Open-Ended Questions

• Tell me how it works
• How often do you do it? Who participates?
• Walk me through . . .
• Show me how . . .
• How has it changed? Have you always done it this way?
• How do you know? (vs. Do you know?)

(Be prepared to ask your questions more than one way)

89

Assignments

Table Item 1 1.1 2 2.2 3 3.1 4 6.1 5 1.1 6 2.2 7 3.1 8 6.1

90

Exercise – Site Visit Part 1

• Individually, select an OFI from the Consensus Scorebook

for your assigned item

– Try not to double up! Make sure all OFIs are covered around the table.

• Complete the Scorebook Navigator form (pg. 142)

– Documents to review – Who to interview – Site Visit Issue (Verify/Clarify question)

• Complete a Site Visit Issue Worksheet with questions you

will ask the applicant (pg. 141)

– A-D-L-I

91

Exercise – Site Visit Part 2

• Take turns interviewing your table anchor using your

SVI worksheet

• Choose a back-up to take notes
• Listen to other site visit interviews for information

that may impact your SVI

• Table anchors—give feedback to the interviewer.

Consider question phrasing and interview style.

• Share best practices
• Save for the next exercise!

92

AFTER THE SITE VISIT

“Beginning in itself has no value, it is an end which makes beginning meaningful, we must end what we begun.” Amit Kalantri

Site Visit Phases

Planning On Site Post Site Visit

94

Post-Site Visit Phase

Enter site visit findings in Scorebook Navigator Revise comments and key themes Complete score summary form Complete summary of sites visited form Complete Award Recommendation form

95

Site Visit Findings

• Use your SVI Worksheet (Word) to take notes

during site visit

• Team leader will collect these when complete

(and save in case there are questions)

• Document your evidence/findings in the

Scorebook Navigator form with a conclusion:

– OFI remains – Delete OFI – OFI becomes a strength

96

97

98

Possible Findings

• Keep OFI, no change
• Edit OFI – change from approach to deployment. They

demonstrated an approach but workgroups xyz are not using it yet.

• Change OFI to strength – process is in place, fully

deployed, with cycles of learning.

• Delete OFI – evidence on site presented to show this

has been addressed.

– Question: Can you identify a new OFI using ADLI, LeTCI, or B/O/M?

• Don’t forget to save!

99

100

Update Comments to Match Findings

• Some comments will stay the same
• If you said you were going to edit a comment

(for example, to change from an approach OFI to deployment OFI), make that change

• Clean up writing/grammar
• There should be no more “it is not clear”

Deleting Comments

• To delete a comment, update the strength or

OFI in the system to say “deleted per site visit”

• Do NOT delete the entire row
• The Site Visit Issue findings should explain why

the comment was deleted

• You must save the item worksheet again

102

103

104

To Add a New Comment

• Enter the strength or OFI in a new row in your

item worksheet

• For the rationale, say “added per site visit”
• If the strength replaces an OFI (or vice versa),

reference the other comment

• If not, explain in the rationale how you found

this new comment

• You must save your item worksheet again

105

106

Individual Exercise

• Use the same site visit issue as before:

– Complete the SVI form in Scorebook Navigator

• What would you write in the findings box? (pg. 142)

– Re-write the comment to match your findings on scratch paper

107

Table Exercise

• Report out: What did you find on site? How

did your comment change?

• Agree as a table how these findings would

have changed the item score

108

Evaluation of Day 2

• +’s and Δ’s
• 2 key learnings to share
• Tomorrow is T-shirt Day!