Key Business Indicator Trends During the Journey from SW-CMM Level - - PowerPoint PPT Presentation

Integrated Systems and Solutions

Management & Data Systems

Key Business Indicator Trends During the Journey from SW-CMM Level 2 to CMMI Level 5 at Lockheed Martin Management & Data Systems Peter McLoone Peter.j.mcloone@lmco.com 610 354 6986

Integrated Systems & Solutions

Management & Data Systems

2 2

Management & Data Systems Management & Data Systems

Headquartered in Bethesda, MD, Lockheed Martin Headquartered in Bethesda, MD, Lockheed Martin employs about 125,000 people worldwide and is employs about 125,000 people worldwide and is principally engaged in the research, design, principally engaged in the research, design, development, manufacture and integration of advanced development, manufacture and integration of advanced technology systems, products and services. technology systems, products and services. Headquartered in King of Prussia, PA, the 9000 Headquartered in King of Prussia, PA, the 9000 employees of Lockheed Martin M&DS have provided employees of Lockheed Martin M&DS have provided leadership in systems integration, systems engineering, leadership in systems integration, systems engineering, software development and program management in software development and program management in support of vital national systems for more than 30 years. support of vital national systems for more than 30 years.

Integrated Systems & Solutions

Management & Data Systems

3 3

M& DS Domain Strengths M& DS Domain Strengths

Collection Tasking Collection Collection Tasking Tasking Collection Operations Collection Collection Operations Operations Exploitation, Analysis and Reporting Exploitation, Exploitation, Analysis and Analysis and Reporting Reporting Dissemination Dissemination Dissemination Processing Processing Processing Statement of Need Statement of Statement of Need Need System Engineering & Integration System Management Infrastructure Support Operations & Maintenance System Engineering & System Engineering & Integration Integration System Management System Management Infrastructure Support Infrastructure Support Operations & Maintenance Operations & Maintenance

Satellite Command & Control Integrated Ground Stations Mission Planning, Resource

Allocation

• Space/Ground Station Development

Space/Ground Station Development

• Operations Management

Operations Management

• Digital Signal

Digital Signal Processing Processing

• Advanced

Advanced Algorithm Algorithm Development Development

• Sensor Data &

Sensor Data & System System Processing Processing

• Exploitation

Exploitation Systems Systems

• Interactive Target

Interactive Target Recognition Recognition

• Image

Image Understanding Understanding

• Report Generation

Report Generation

• Analytic

Analytic Collaboration Tools Collaboration Tools

• Imagery Retrieval,

Imagery Retrieval, Archiving & Distribution Archiving & Distribution

Integrated Integrated End End-

• to

to-

• End

End Capabilities Capabilities in Intelligence, in Intelligence, Surveillance Surveillance and and Reconnaissance Reconnaissance Systems Systems

Integrated Systems & Solutions

Management & Data Systems

4 4

M& DS Process Improvement History M& DS Process Improvement History

• Long history of process orientation although most programs were

Long history of process orientation although most programs were classified and classified and stovepiped stovepiped

• 1993

1993-

• 1995: Software Capability Maturity Model Level 2 (Internal

1995: Software Capability Maturity Model Level 2 (Internal Assessment) Assessment)

• 1996

1996-

• 1997: SW CMM Level 3; formally assessed in 1996

1997: SW CMM Level 3; formally assessed in 1996

• 1998

1998-

• 1999: SW CMM Level 4; formally assessed in 1998

1999: SW CMM Level 4; formally assessed in 1998

• 2000

2000-

• 2001: SW CMM Level 5, SE CMM Level 5 for 12

2001: SW CMM Level 5, SE CMM Level 5 for 12 PAs PAs, Level 4 for , Level 4 for 1 PA; both formally assessed in 2000 1 PA; both formally assessed in 2000

• 2002

2002-

• present: Capability Maturity Model, Integrated SE/SW Version

present: Capability Maturity Model, Integrated SE/SW Version 1.1 Staged Model Maturity Level 5; formally assessed in 2002 1.1 Staged Model Maturity Level 5; formally assessed in 2002

Integrated Systems & Solutions

Management & Data Systems

5 5

M& DS Business Objectives M& DS Business Objectives

• Annual Goal Setting Process conducted by senior management and d

Annual Goal Setting Process conducted by senior management and driven by our riven by our Executive Office Executive Office

• Uses M&DS Guiding Principles as a framework: Customer Satisfacti

Uses M&DS Guiding Principles as a framework: Customer Satisfaction, Employee

• n, Employee

Fulfillment, Corporate Commitment, etc Fulfillment, Corporate Commitment, etc

• Result is a Balanced Scorecard like set of 40

Result is a Balanced Scorecard like set of 40-

• 50 Goals most having an underlying

50 Goals most having an underlying quantitative orientation quantitative orientation

• Engineering related goals include productivity or product qualit

Engineering related goals include productivity or product quality objectives such y objectives such as: as:

− − Reduce defect density during testing by at least

Reduce defect density during testing by at least xx xx per cent per cent

− − Improve systems engineering productivity by at least

Improve systems engineering productivity by at least xx xx per cent per cent

• Each goal is owned by a member of senior management who select a

Each goal is owned by a member of senior management who select a senior staff senior staff member to manage achieving each goal member to manage achieving each goal

• Each goal has a plan

Each goal has a plan statused statused monthly at an executive review meeting with an monthly at an executive review meeting with an emphasis on quantitative information emphasis on quantitative information

• Engineering goal plans usually include process changes to be imp

Engineering goal plans usually include process changes to be implemented by a lemented by a set of target programs. The process changes have usually been p set of target programs. The process changes have usually been piloted. iloted.

Integrated Systems & Solutions

Management & Data Systems

6 6

Product Quality Goal Product Quality Goal Flow dow n Flow dow n

• Reduce defect density during testing by

Reduce defect density during testing by xx xx per cent per cent

• Strategy: Implement In Process Quality

Strategy: Implement In Process Quality

− − Select and track visible product quality indicators

Select and track visible product quality indicators

− − Implement Fagan type inspections from Systems Requirements Analy

Implement Fagan type inspections from Systems Requirements Analysis forward sis forward

− − Perform Defect Causal Analysis on a regular basis

Perform Defect Causal Analysis on a regular basis

• Used the Defect Detection Profile as a fundamental indicator and

Used the Defect Detection Profile as a fundamental indicator and allocated defect allocated defect detection targets to each phase of development. Targets were est detection targets to each phase of development. Targets were established based ablished based

• n the modeling of historical data
• n the modeling of historical data
• Inspections were a critical

Inspections were a critical subprocess subprocess in all development phases through Code in all development phases through Code and Unit Test so it was an obvious candidate for statistical man and Unit Test so it was an obvious candidate for statistical management. Control

• agement. Control

Charts of Preparation Rate, Meeting Pace Rate, and Defect Densit Charts of Preparation Rate, Meeting Pace Rate, and Defect Density were utilized. y were utilized.

• Target programs were identified, process changes infused, and re

Target programs were identified, process changes infused, and results monitored sults monitored

Integrated Systems & Solutions

Management & Data Systems

7 7

Defect Detection Profile/ Defect Detection Profile/ Inspection Control Charts Inspection Control Charts

Example Defect Detection Profile

ABD/R L0 ABD/R L1 ABD L2 SWD CUT PIV SIV Depl

Defect Detection Point

Defects / (New LOC + Added LOC + Modified LOC) Normal Range Total Incr 1 Incr 2 Incr 3 Incr 4 Incr 5 Target

Pace Rate

T C 2 2 1 T C 2 2 4 T C 2 2 5 T C 2 3 6 T C 2 3 4 T C 2 3 7 T C 2 4 2 T C 2 3 9 T C 2 4 T C 2 4 4 T C 2 4 1 T C 2 3 8 T C 2 4 6 T C 2 4 9 T C 2 5 4 T C 2 6 T C 2 6 4 T C 2 6 5 T C 2 6 7 T C 2 6 8 LOC/Hour Pace Rate Center Line Upper Natural Process Limit Lower Natural Process Limit

Defect Density Control Chart

T C 2 2 1 T C 2 2 4 T C 2 2 5 T C 2 3 6 T C 2 3 4 T C 2 3 7 T C 2 4 2 T C 2 3 9 T C 2 4 T C 2 4 4 T C 2 4 1 T C 2 3 8 T C 2 4 6 T C 2 4 9 T C 2 5 4 T C 2 6 T C 2 Defects/KLOC Defect Density: Major, Minor Center Line (Individuals) Upper Natural P Lower Natural Process Limit One Sigma+ One Sigma - Two Sigma Target Model Baseline Model

Prep Rate Control Chart

TC20021 TC20024 TC20025 TC20036 TC20034 TC20037 TC20042 TC20039 TC20040 TC20044 TC20041 TC20038 TC20046 TC20049 TC20054 TC20064 TC20065 TC20067 TC20068 LOC/Hour LOC/Ave Prep HR Center Line Upper Natural Process Limit Lower Natural Process Limit

Integrated Systems & Solutions

Management & Data Systems

8 8

Quantitative Project Management Quantitative Project Management

• Plan

Plan

• Perform

Perform

• Evaluate

Evaluate

• Improve

Improve

Integrated Systems & Solutions

Management & Data Systems

9 9

Planning Planning

• Identify the project

Identify the project’ ’s information needs s information needs

• Prioritize the needs

Prioritize the needs

• Select/define the most cost effective indicators that address th

Select/define the most cost effective indicators that address the e highest priority needs highest priority needs

• Determine decision criteria (thresholds and targets) for each

Determine decision criteria (thresholds and targets) for each indicator indicator

• Determine reporting mechanisms

Determine reporting mechanisms

• Determine data storage and control mechanisms

Determine data storage and control mechanisms Involve all Stakeholders and Write Dow n the Results

Integrated Systems & Solutions

Management & Data Systems

10 10

Performing Performing

• Collect Data

Collect Data

• Produce Indicators

Produce Indicators

• Use Decision Criteria

Use Decision Criteria

• Evaluate Anomalies and Determine Corrective Action

Evaluate Anomalies and Determine Corrective Action

− − Root Cause

Root Cause

− − Additional Data Analysis

Additional Data Analysis

• Report Results

Report Results

• Implement Corrective Actions

Implement Corrective Actions Integrate into the Program’s Business Rhythm

Integrated Systems & Solutions

Management & Data Systems

11 11

Evaluating & Improving Evaluating & Improving

• Execution conforms to process definition

Execution conforms to process definition

• The QM process is efficient and timely

The QM process is efficient and timely

• The indicators are accurate

The indicators are accurate

• The indicators are complete

The indicators are complete

• The indicators are used proactively in decision making

The indicators are used proactively in decision making Evaluating effectiveness must be a planned event Evaluating effectiveness must be a planned event

Integrated Systems & Solutions

Management & Data Systems

12 12

How Did We Do How Did We Do

• In the beginning we were not smart enough to figure out ahead of

In the beginning we were not smart enough to figure out ahead of time how to time how to design a clever set of measures to track over the last ten years design a clever set of measures to track over the last ten years

• We do have a fairly consistent data collection discipline for fi

We do have a fairly consistent data collection discipline for financials although nancials although accounting methods do change as well over such a long period accounting methods do change as well over such a long period

• As we

As we’ ’ve moved up the maturity ladder, we ve moved up the maturity ladder, we’ ’ve certainly evolved our engineering ve certainly evolved our engineering data collection a lot and have paid more attention to data quali data collection a lot and have paid more attention to data quality problems ty problems

• But we still are able to evaluate and relate to maturity model l

But we still are able to evaluate and relate to maturity model levels: evels:

− − Award Fee

Award Fee

− − Overhead Rate

Overhead Rate

− − Software Productivity

Software Productivity

− − Software Unit Cost

Software Unit Cost

− − Defect Find & Fix Cost

Defect Find & Fix Cost

Integrated Systems & Solutions

Management & Data Systems

13 13

Aw ard Fee Aw ard Fee

Customer Satisfaction Continues to Improve Customer Satisfaction Continues to Improve

Award Fee vs CMM Level

SW CMM L2 SW CMM L3 SW CMM L4 SW CMM L5 CMMI L5 Per Cent SW CMM L2: 1993-1995 SW CMM L3: 1996-1997 SW CMM L4: 1998-1999 SW CMM L5: 2000-2001 CMMI L5: 2002 Potential Additional Award Fee Additional Award Fee Achieved = 55% of the Potential Additional Award Fee

Integrated Systems & Solutions

Management & Data Systems

14 14

Overhead Rates Overhead Rates

CMMI Does Not Come w ith Overhead Baggage CMMI Does Not Come w ith Overhead Baggage

Overhead Rate

93.0% 94.0% 95.0% 96.0% 97.0% 98.0% 99.0% 100.0% 101.0% SW CMM L3 SW CMM L4 SW CMM L5 CMMI L5 Overhead Rate Overhead Rate as a Percentage of SW CMM L3 Near the end of the SW CMM L2 period, the overhead pools were

• changed. A SW CMM L2 Overhead

is therefore not included.

Integrated Systems & Solutions

Management & Data Systems

15 15

Softw are Productivity Softw are Productivity

Steady and Consistent Improvement Steady and Consistent Improvement

Software Productivity

80% 90% 100% 110% 120% 130% 140% Pre-NW/IPQ Baseline Program A I2-I3 Program A I4 Program B I2, I3 Program B I4 Per Cent of Baseline NW/IPQ: NWODB / IPQ New Ways of Doing Business (NWODB) are an integrated set of engineering process and methodology techniques piloted during the SW CMM Level 4 period and fully infused during the SW CMM Level 5 period. In-Process Quality (IPQ):

• Select and track visible product quality indicators
• Implement Fagan type inspections from Systems

Requirements Analysis forward

• Perform Defect Causal Analysis on a regular

basis Ix: Increment x

Integrated Systems & Solutions

Management & Data Systems

16 16

Softw are Unit Cost Softw are Unit Cost

Average Engineering Hourly Rate Average Engineering Hourly Rate x Overhead Rate x Overhead Rate x Hours per LOC x Hours per LOC x Constant Dollar Adjustment Factor x Constant Dollar Adjustment Factor

Software Unit Cost

0% 20% 40% 60% 80% 100% 120% Pre NW/IPQ Post NW/IPQ

Real Real Bottom Line Bottom Line Savings Savings

Integrated Systems & Solutions

Management & Data Systems

17 17

Defect Find & Fix Defect Find & Fix

Dollars per Kloc

75% 80% 85% 90% 95% 100% 105% SW CMM Level 3 Program CMMI Level 5 Program

Hours/KLOC

Architecture Design Software Design Code & Unit Test Product Integration & Verification System Integration & Verification Deployment SW CMM Level 3 Program CMMI Level 5 Program

Conservative Assumptions: Conservative Assumptions:

• Deployment Find &Fix Time Same as Systems I & V

Deployment Find &Fix Time Same as Systems I & V

• L3 Program had same Find & Fix as the L5 Program

L3 Program had same Find & Fix as the L5 Program

• L3 Program Defect Densities for AD, SD, and CUT could be

L3 Program Defect Densities for AD, SD, and CUT could be adequately modeled using SWEEP and assuming total L3 adequately modeled using SWEEP and assuming total L3 Program Defect Density was the same as the L5 Program Program Defect Density was the same as the L5 Program

Improved Improved Product Quality Product Quality With Real With Real Cost Savings Cost Savings

Integrated Systems & Solutions

Management & Data Systems

18 18

Summary Summary

• A study of several senior management visible measures

A study of several senior management visible measures consistency show value in pursuing CMM driven process consistency show value in pursuing CMM driven process improvement improvement

• Our senior management believes this emphasis has had a positive

Our senior management believes this emphasis has had a positive effect on other functions within the company effect on other functions within the company

• It does make a difference to our customers

It does make a difference to our customers

Senior management commitment has been a key to success Senior management commitment has been a key to success