W6 September 29, 2004 3:00 PM C USTOMER F OCUSED B USINESS M ETRICS - - PDF document

w6
SMART_READER_LITE
LIVE PREVIEW

W6 September 29, 2004 3:00 PM C USTOMER F OCUSED B USINESS M ETRICS - - PDF document

BIO PRESENTATION W6 September 29, 2004 3:00 PM C USTOMER F OCUSED B USINESS M ETRICS THROUGHOUT THE SDLC Steve Wrenn Liberty Mutual Insurance Information Systems Better Software Conference & EXPO September 27-30, 2004 San Jose, CA USA


slide-1
SLIDE 1

BIO PRESENTATION Better Software Conference & EXPO September 27-30, 2004 San Jose, CA USA

W6

September 29, 2004 3:00 PM

CUSTOMER FOCUSED BUSINESS METRICS THROUGHOUT THE SDLC

Steve Wrenn Liberty Mutual Insurance Information Systems

slide-2
SLIDE 2

STEPHEN WRENN

Stephen Wrenn is currently the Sr. Director of IT Service & Quality Assurance at Liberty Mutual Insurance Group. There he is responsible for bringing people, process and technology together to solve insurance business problems on a global basis. Prior to Liberty Mutual, Stephen was the VP of Global Service & Support for Enterasys Networks, where he led a team of service professionals in delivering industry leading network quality initiatives, and global expansion of Enterasys professional services. Stephen has over 20 years experience in IT company’s, holding leadership positions at Hewlett-Packard, Cabletron, and Cisco Systems. He has built technical support

  • rganizations in both the US and abroad, most recently encompassing offices in London,

Singapore and Hong Kong. Stephen holds a B.S. in Electrical Engineering from the University of Massachusetts and an MBA from the Whittemore School of Business at the University of New Hampshire.

slide-3
SLIDE 3

Customer Focused Business Metrics Throughout the SDLC

Better Software Conference & Expo ’04 Measuring and Reporting Track

Steve Wrenn, Director IT Service & Quality Assurance Liberty Mutual Insurance Group, Information Systems

slide-4
SLIDE 4

Introduction

A quick overview of Liberty Mutual IS and it’s

culture.

What we set out to do. Our process or journey

Constraints SDLC vs. PLC Quality Framework Development Focus

Outcomes What we learned

slide-5
SLIDE 5

Where We Were at Liberty

First of all you have to understand a few things

about Liberty Mutual

Insurance Business is inherently Risk Adverse Private Company with, until recently, mostly internal

customer’s for IT

Currently profitable in a very stable and long-life

business

After that very Much Like a Lot of IT

Development Shops

Not enough time to do everything Lack of controls…after all, Coding is an art right?

slide-6
SLIDE 6

What We Set Out To do

Run IT Like a Business….

Increase Level of Service to our customers by

delivering high quality products and services

  • n time and on budget.

Get it right the first time… One and Done!

slide-7
SLIDE 7

Our Process to “Run IT Like A business”

First decide where we want to go.

Strategic objectives

Business objectives Competitive advantage Application Development Focus

Getting the Voice of the Customer (VOC).

Who’s the customer? What do customers want/need

Operational Focus

TQM, Six-Sigma, CMMI Level ?

slide-8
SLIDE 8

Customer Focus

Faster Better Cheaper

CUSTOMER DELIGHT! Creativity, Flexibility and Responsiveness are the Keys!

slide-9
SLIDE 9

Our Process…….continued

Then evaluate where we currently stand. (Nothing new

here, a simple GAP analysis should suffice nicely…yeah right!)

Baseline metrics….what metrics?

Our work is so unique you can’t measure it

Evaluation of current processes

Things are working fine, we’re making money, why do we want

to change things

Processes were all different, so any metrics collected could

not be compared.

All hard to do without knowing current output (systems

thinking).

We finally settled on quality control data gathering for 6

months.

slide-10
SLIDE 10

Our Constraints

The old project management triple

constraint of scope, schedule and budget, and also a mindset that said get it done on time, at all costs.

“What do you think was the main cause of this?”

slide-11
SLIDE 11

It All Starts With Requirements

Or does it?

slide-12
SLIDE 12

SDLC vs. PLC Mindset

The software development life cycle of most

companies are very well thought out and focused on getting a project done on time, on schedule and on scope.

How many of them are actually focused on

delivering the customer what they want and need to be successful?

How do they know if they are?

slide-13
SLIDE 13

The PLC Feedback-Loop

Think customer input, which in most companies

comes from the users of your current products and services. (VOC)

How many of you proactively use this

information to feedback to your development teams the fruits of their labors, or to design your next product?

“The past is a pretty good predictor of the future”

slide-14
SLIDE 14

LMIG IT Quality Framework / Methodology Product Life Cycle (PLC) Plan Build Run Retire

Software Development Life Cycle (SDLC) Production

Customer (End-User) Voice of the Customer (Requests/Concerns) Change Management (For New & Production) Problem Management (Restore & Eliminate) Tools to enable Effectiveness & Efficiency Metrics for Success and Adjustments (Targets) B.U.

Release/Acceptance Pre-Production

PDCA PDCA PDCA

  • Requirements
  • SQA/Test Plan
  • QA Deliverables
  • Stress Testing
  • Final SLA
  • System Burn-in
  • SLA Monitoring
  • Support Training
  • EOL Strategy

Quality Assurance

slide-15
SLIDE 15

Our Development Focus

Plan and Build

Define, Design, Develop & Test Before you jump into this, there is a little

education that is extremely valuable.

“How do you keep the team focused on getting the customer’s what they need to be successful?”

slide-16
SLIDE 16

A Focused Development Team

Help them start to think like business managers Take the time to explain to them what the fruits

  • f their labors are.

They are not developing code, they are building

pieces to a product that fit into a system or systems that somehow make money for the company!

Show them how their applications contribute, they

may surprise you with their ideas here.

Show them how quality, both high and low, effects the

  • utcome
slide-17
SLIDE 17

From the Quality/Test Perspective

Worked with the development team to put

metrics in place that constantly let them know how they and their processes are doing.

Plan, Do, Check, Act….make it self-managing by

building a metrics program or dashboard

Closed-loop, systems thinking and training There can be a lot of cultural shock here.

slide-18
SLIDE 18

Liberty IT Continuous Process Improvement

People Process Technology

Optimum Productivity!

Knowledge Sharing, Customer Focus and Leadership are Key!

slide-19
SLIDE 19

Our Outcomes

We developed a quality process and

measurement program that collected few, but very specific measurements at each stage of our development cycle.

We improved our first release quality by 40% in

the first 6 months of our initiative How did we do it?

slide-20
SLIDE 20

LMGIT - SDLC Key Quality Deliverables - Worksheet

Requirements Definition Architecture &

Scope Definition Requirements Definition Analysis Architecture & Design Construction & Test Implementation

PMO Required

  • Request for Project Funding

Report

  • Create Project Charter

*

  • Create Initial IT Project Plan

(006) with Key Milestones*

  • PMWB Forecast
  • Risk /Issues Log
  • Change Management

Process

QA Best Practices

  • BISAT/ Opportunity

Evaluation (100) * /**

  • Create Initial SQA Plan*
  • Review with Technology Mgt

Board (TMB) *

  • Review with Enterprise

Portfolio Review Board (EPRB) *

  • Obtain Key Stakeholder

Decision for “Go / No Go” for proceeding with Bus. Requirements *

  • Statement of Work /

Contract (if applicable)

  • Sarbanes Oxley

PMO Required

  • First cut SLA Performance

*

  • Create Initial Test Strategy

Document

QA Best Practices

  • Business Alternatives (125)

*

  • Bus Objectives (130)

*

  • Proposed Workflow (132)

*

  • System Context Diagram -

“To Be” (140) *

  • Requirements Management

Plan

  • Glossary of Terms

QA Best Practices Updated

  • Updated SQA Plan
  • Updated IT Project Plan

PMO Required

  • Freeze Detailed Business

Requirements (135s) - OO Projects = Use Case Scenarios, Class Diagrams, Activity Diagrams, etc. * (all changes now go through the Change Management Process) ***

  • Technical Architecture

Preliminary Review Report *

  • Logical Data Model

*

  • Completed Disaster

Recovery Requirements *

QA Best Practices

  • Security Requirements

*

  • System Interface

Requirements *

  • Create Test Plan
  • Create Test Cases -

Functional, System, Integration, & UAT

  • Baseline Schedule/ Budget
  • Business Rules
  • Data Conversion Rules
  • Archive Purge Requirements
  • Updated Glossary of Terms

QA Best Practices Updated

  • Updated / Test Strategy -

Finalized *

  • Baselined IT Project Plan

PMO Required

  • Functional Specs (490)

*

  • Infrastructure Recovery

Requirements (IRR) - 1

st Cut

QA Best Practices

  • Technical Alternatives (260)
  • Detailed Technical &
  • Architec. Design Review

*

  • User Interface Specifications

System Storyboard/ Flow (250/255) HTML / Word Page Mock- Ups (495)

  • Non Functional /

Supplementary Specs - Performance Reqmnts, Archive /Purge, Security, Disaster Recovery, etc.

  • OO / Baselined Object Model
  • Finalizd Funct’l Data Model

*

  • Desktop QA Plan

*

  • Program Specs (570)
  • End User Training Strategy
  • Implementation Strategy
  • QA Best Practices

Updated

  • Finalized Test Cases -

Functional, Systems Integration, and UAT

  • Finalized Test Plan *
  • Updated Project Plan

PMO Required

  • User Acceptance Testing

QA Best Practices

  • Source Code Mgt Plan/

Configuration Mgt

  • Service Center Change

Ticket

  • Define/Create Testing

Environments

  • Software Code Created
  • Conduct Peer Code Reviews
  • Unit Testing
  • Functional Testing
  • Systems Integrated Testing
  • Performance Testing
  • Volume Testing
  • Stress Testing
  • Desktop QA Certification

Notification

  • UAT Testing Sign-Off

* / ****

  • Risk Assessment to Prod

*

  • Software Code Frozen

* (all coding changes now go thru Change Mgt Process )

  • Baselined Phys Data Model
  • Test Communication Plan for

Teams/ Management

  • QA Best Practices

Updated

  • Finalized IRR Document

*

  • Finalized SLA *
  • Updated IT Project Plan

PMO Required

  • Implementation Plan

Including Implementation Checklist (back out plan and Signoff (800)

  • Lessons Learned and Action

Plan Document (060)

QA Best Practices

  • Resolved Requirements

Change Management Issue

  • Production Support Plans

*

  • On-line Help System: User

Guide (700); Help Desk (701); Training (710); Installation Instructions (720)

  • Release Notes - Business

End User and IS Release Notes

  • Perform Post Implementation

Verification Testing

  • Obtain Client Sign Off for

“Go/ No Go” to deploy to Production *

  • IT - QA Sign Off per SQA

Plan

  • Monitor System Performance

for 90 Days

  • Published Re-Usable Assets

in Component Re-Use System

  • Maintenance Support Plans

David H. Maurer / June 15, 2004

* Key Stakeholder

Sign-Offs

** (NNN) Refers to the

LMSDP Process Artifacts IT Project Plan PMWB (Project Managers Workbench ) Risk/ Issues Log Change Management Process/ Log (Mantis) -> ServiceCenter SQA Plan

Project Management

LMGIT - SDLC - Key Quality Deliverables Worksheet - V1.03

Project Name: ______________________ Project Code: ______________ IT Project Mgr : _______________________ Bus Project Mgr : ______________________ *** Date Business Requirements Frozen: _____/_____/____

Proj Mgr: __________________ QA Mgr ___________________

**** Date User Acceptance Testing Completed : _____/____ _/_____

Proj Mgr ___________________ QA Mgr _______________________

slide-21
SLIDE 21

SDLC Quality Gates / Metrics

Initiation / Scope Definition

Instituted a rigorous project qualifying review

ROI or competitive advantage Strategic fit…or else!

Requirements Definition

Do we know what this requirement is for? Has it been verified for testability and signed of by the

Test or QA manager?

Have the Requirements been signed off by all

parties?

slide-22
SLIDE 22

Requirements Definition Customer Needs / Wants Analysis

slide-23
SLIDE 23

Requirements Definition Quality Requirements

5 Major Issues Preventing High Quality

Requirements (and thus setting the project up for failure from the start)

Requirements not complete Requirements not testable (to gray) Requirements developed without end-user input Requirements thought to be non-negotiable Requirements not signed of by all parties.

slide-24
SLIDE 24

SDLC Quality Gates / Metrics

Analysis

Test plans completed Requirements freeze in place (change mgt)

Architecture and Design

Test cases completed (100%) Requirements Freeze in place

slide-25
SLIDE 25

SDLC Quality Metrics Gates

Construction and Test

Changes in this stage require higher level of

sign-off

SLA developed

Implementation

SLA signed-off by all parties 30, 60, 90 day monitoring of quality Project estimation chart completed

slide-26
SLIDE 26

What We Learned

Culture change is more difficult than any

technical challenge you will encounter

The more time spent in planning the less time

spent in other phases or “An ounce of planning is worth a pound of development”.

Signatures are very powerful in setting and

keeping expectations

We very often take too much for granted when it

comes to business / technical understanding.

Metrics need to be SMART

slide-27
SLIDE 27

Liberty IT Monthly or Quarterly Report Card

FY’04 Q1 Deliverables Key Monthly Accomplishments

Strategy #1: Engage proactively and regularly with customers, partners and sales organizations, and technology development stakeholders, to ensure delivery of consistent, high quality, and relevant service and support worldwide. –>/= 4.6 out of 5.0 on WW customer satisfaction survey –>15% Increase in Productivity –<99% of budgeted expenses Strategy #2: Grow current service offerings in order to enable the business groups to deliver better service to their customers, and thus further differentiate the total Enterasys value proposition from the competition. –>110% WW Bookings targets (15% growth) –>65% Contribution Margin (cost down 10%, rev up 15%) Strategy #3: Develop and deliver new services that add value to our products and solutions, and differentiate them in the marketplace. –>50% of services developed in last 3 years

Status

  • Measurable accomplishments towards your business plans
  • Productivity, costs, customer satisfaction, etc….
  • Examples…….
  • Completed Design Stage on project XX
  • % of projects in development that are on time
  • Change management measures
  • Quality before release measures
  • Quality in production measures

CONCERNED COMPLETE ON-TARGET

1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 F Y ' 1 Q 1 Q 2 Q 3 Q 4 F Y ' 2 Q 1 Q 2 F c s t Q 3 F c s t Q 4 F c s t US Canada LA EMEA AP

Customer Satisfaction Rating Customer Satisfaction Rating

$0 $5,000 $10,000 $15,000 $20,000 $25,000 $30,000 $35,000 $40,000 $45,000 $50,000 FY'01 Q1 Q2 Q3 Q4 FY'02 Q1 Q2 Q3 Q4

Bkgs Actual F'cst

slide-28
SLIDE 28

SDLC Quality Gates / Metrics Data vs. Information

Data is IMPORTANT!…Without Data

…………..…..Opinion Prevails

Data + Analysis = Intelligent

Information

Data by itself is the library without a

librarian.

slide-29
SLIDE 29

The Importance of Proper Measurement

Positive or Negative?

slide-30
SLIDE 30

The Importance of Proper Measurement

Positive

  • r

Negative ?

slide-31
SLIDE 31

The Importance of Proper Measurement

Positive or Negative?

slide-32
SLIDE 32

LMIG IT Quality Framework / Methodology Product Life Cycle (PLC) Plan Build Run Retire

Software Development Life Cycle (SDLC) Production

Customer (End-User) Voice of the Customer (Requests/Concerns) Change Management (For New & Production Apps) Problem Management (Restore & Eliminate) Tools to enable Effectiveness & Efficiency Metrics for Success and Adjustments (Targets) B.U.

Release/Acceptance Pre-Production

Thoughts …

“Don’t Tell Me … Prove It!” “Close-The-Loop … Systems Thinking”

PDCA PDCA PDCA

  • Requirements
  • SQA/Test Plan
  • QA Deliverables
  • Stress Testing
  • Final SLA
  • System Burn-in
  • SLA Monitoring
  • Support Training
  • EOL Strategy

Quality Assurance