Naval Center for Cost Analysis (NCCA ) Validation and Improvement of - - PowerPoint PPT Presentation

▶
naval center for cost analysis ncca
SMART_READER_LITE
LIVE PREVIEW

Naval Center for Cost Analysis (NCCA ) Validation and Improvement of - - PowerPoint PPT Presentation

May 15, 2023 770 likes •1.16k views

UNCLASSIFIED (FOR OFFICIAL USE ONLY) Naval Center for Cost Analysis (NCCA ) Validation and Improvement of the Rayleigh Curve Method 10-13 JUN 2014 POC: Mr. Jake Mender Ship & Ship Weapons Branch Naval Center for Cost Analysis

slide-1
SLIDE 1

Validation and Improvement of the Rayleigh Curve Method

10-13 JUN 2014

Naval Center for Cost Analysis (NCCA)

UNCLASSIFIED (FOR OFFICIAL USE ONLY)

POC: Mr. Jake Mender Ship & Ship Weapons Branch Naval Center for Cost Analysis Michael.Mender@navy.mil

UNCLASSIFIED (FOR OFFICIAL USE ONLY)

slide-2
SLIDE 2

2

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Overview

  • 1. Background and Motivation
  • 2. Parameter Estimation
  • 3. Evaluating Rayleigh Method
  • 4. Conclusions
slide-3
SLIDE 3

3

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Background

slide-4
SLIDE 4

4

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Rayleigh Hypothesis

  • Theory that effort on a project follows a

standard pattern

– Pattern is approximated by Rayleigh Function – Developed from Manpower Utilization model developed by P.V. Norden1 in the 1960s

  • If true, allows for total effort and duration to be

estimated from the trend of early data

– For our purposes this is conveyed by ACWP as reported in EVM CPRs

1. Norden, P.V., “Useful Tools for Project Management,” Operations Research in Research and Development, B.V. Dean, Editor, John Wiley and Sons, 1963

slide-5
SLIDE 5

5

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Rayleigh Function Basics

slide-6
SLIDE 6

6

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Motivation

  • 1. Rayleigh is popular, but vintage support

– NCCA does not have access to original data set

  • 2. Desire to validate and verify theory

– Does EVM follow Rayleigh “path” ? – Does theory still hold for current contracts? – How accurate is it? – Are there any pitfalls analysts should be aware of?

slide-7
SLIDE 7

7

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Example EVM Data

Report Date Days From Start ACWP Cumulative ACWP Current Estimated Completion Date (ECD) Estimate at Completion (EAC) 6/25/2020 30 3,737,226 $ 3,737,226 $ 12/27/2023 44,862,882 7/26/2020 61 5,682,668 $ 1,945,442 $ 12/27/2023 44,862,882 8/26/2020 92 7,683,822 $ 2,001,154 $ 12/27/2023 47,291,764 9/25/2020 122 9,687,672 $ 2,003,850 $ 12/27/2023 62,556,969 10/26/2020 153 12,435,553 $ 2,747,881 $ 12/27/2023 71,045,926 11/25/2020 183 15,144,794 $ 2,709,241 $ 12/27/2023 71,045,926 12/26/2020 214 17,548,516 $ 2,403,722 $ 12/27/2023 71,142,074 1/26/2021 245 20,261,352 $ 2,712,836 $ 12/27/2023 72,469,288 2/23/2021 273 22,780,991 $ 2,519,639 $ 12/27/2023 73,054,269 3/20/2021 298 25,757,113 $ 2,976,122 $ 12/27/2023 79,993,162 4/20/2021 329 29,099,859 $ 3,342,746 $ 12/27/2023 109,207,141 5/20/2021 359 31,647,355 $ 2,547,496 $ 12/27/2023 109,207,141 6/19/2021 389 34,117,572 $ 2,470,217 $ 12/27/2023 111,012,404 7/24/2021 424 38,510,766 $ 4,393,195 $ 12/27/2023 111,012,404 7/27/2021 427 41,920,008 $ 3,409,241 $ 12/27/2023 113,618,308 8/27/2021 458 46,542,342 $ 4,622,334 $ 12/27/2023 113,618,308 9/26/2021 488 51,164,676 $ 4,622,334 $ 12/27/2023 114,752,325

slide-8
SLIDE 8

8

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Plot of ACWP vs. Time

$- $5,000,000 $10,000,000 $15,000,000 $20,000,000 $25,000,000 $30,000,000 $35,000,000 $40,000,000 $45,000,000 200 400 600 800 1000 1200 1400 1600 ACWP Cumulative Days from Work Start $- $500,000 $1,000,000 $1,500,000 $2,000,000 $2,500,000 200 400 600 800 1000 1200 1400 1600 ACWP Current Days from Work Start

Plotting EVM data with respect to time reveals trends

Cumulative ACWP Current ACWP

slide-9
SLIDE 9

9

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Data vs. Function

Estimating Rayleigh Parameters allows estimation of final cost and duration – assuming effort is Rayleigh distributed

  • 5,000,000
10,000,000 15,000,000 20,000,000 25,000,000 30,000,000 35,000,000 40,000,000 45,000,000 200 400 600 800 1000 1200 1400 1600 ACWP Current Days from Work Start Rayleigh ACWP Actual ACWP
  • 500,000
1,000,000 1,500,000 2,000,000 2,500,000 200 400 600 800 1000 1200 1400 1600 ACWP Cumulative Days from Work Start Rayleigh ACWP Actual ACWP

α = .29 (1451 days) K = 41,558,08 α = .29 (1451 days) K = 41,558,08

slide-10
SLIDE 10

10

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Parameter Estimation

slide-11
SLIDE 11

11

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Multiple Options

  • 1. Optimization
  • 2. Linear Transform and Regression
  • 3. Method of Moments
  • 4. Maximum Likelihood
  • 5. Bayesian Methods

Today’s focus is on Linear Transform and Regression, prior work analyzed Optimization

slide-12
SLIDE 12

12

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Non-Linear Optimization

slide-13
SLIDE 13

13

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Optimization Constraints

  • 100

200 300 400 500 600 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 ACWP ($) Elapsed Time Constrained Unconstrained Actual

Unconstrained Constrained Cost 6,151 15,629 Duration 2.82 4.96

Constraint Selection is really important!

slide-14
SLIDE 14

14

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Linear Transform & Regression

  • 1. Given issues with optimization, desirable to find

method that has analytical closed form solution

– Not easy, as Rayleigh function is not obviously transformed

  • 2. Abernathy (1984)1 developed a linear transform

– Uses Rayleigh PDF as starting point

  • 3. Provides straightforward means to generate

parameter estimates via linear regression

– Requires way to estimate empirical derivative

1. Abernathy, T., “An Application of the Rayleigh Distribution to Contract Cost Data,” Master’s Thesis, Naval Postgraduate School, Monterey, California,1984.

slide-15
SLIDE 15

15

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

  • 5
  • 4
  • 3
  • 2
  • 1

1 500000 1000000 1500000 2000000 2500000

ln(PDF/t) time squared (t2)

Step 4 - Replace x-axis with t2

Rayleigh Linear Transform

  • 100

200 300 400 500 600 700 200 400 600 800 1000 1200 1400 1600 PDF time (t)

Step 1 - PDF

  • 1

1 2 2 3 200 400 600 800 1000 1200 1400 1600 PDF/t time (t)

Step 2 - PDF/t

  • 5
  • 4
  • 3
  • 2
  • 1

1 200 400 600 800 1000 1200 1400 1600 ln(PDF/t) time (t)

Step 3 - ln(PDF/t)

slide-16
SLIDE 16

16

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Methods for Derivative Estimation

slide-17
SLIDE 17

17

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Tangent Line Accuracy

  • 0.1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 50000 100000 150000 200000 250000 300000 350000 400000

ln(f(t)/t) t^2

Transformed PDF vs. Tangent Line

Divided Differences "True" PDF

Errors mean we need to use a different method – Polynomial Interpolation

  • 1. Evaluate accuracy of

derivative estimator by comparing to Rayleigh PDF

  • 2. Tangent line method

struggles with non- constant derivatives

  • 3. Drives need for

alternate derivative estimator

slide-18
SLIDE 18

18

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Polynomial Interpolation - Visual

f(0) f(1) f(2) f(3) f(4) f(5) f(6) y = -7.75x6 + 112.75x5 - 522.92x4 + 596.25x3 + 1330.7x2 - 2129x + 920 R² = 1 500 1000 1500 2000 2500 3000 3500 1 2 3 4 5 6 7 f(x) x

  • 1. Can always create a polynomial of order n-1 (6) that passes through all data
  • 2. This polynomial is easily differentiated
slide-19
SLIDE 19

19

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Lagrange Polynomial Construction

slide-20
SLIDE 20

20

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

2-Point Example (order 1)

f1 f2 100 200 300 400 500 600 700 800 900 1000 10 20 30 40 50 60 70 f(x) x

1) Each dashed line is the Lagrange Polynomial for that f(i) 2) Polynomial is order 1 with n=2, results in straight line

slide-21
SLIDE 21

21

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

2-Point Example (order 1)

f1 f2 100 200 300 400 500 600 700 800 900 1000 10 20 30 40 50 60 70 f(x) x

1) Polynomials for each f(i) are summed, giving interpolating function of order 1 2) Replicates tangent line method discussed previously

slide-22
SLIDE 22

22

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

3-point example (order 2)

f1 f2 f3

  • 400
  • 200

200 400 600 800 1000 1200 1400 1600 20 40 60 80 100 f(x) x

1) Each dashed line is the Lagrange Polynomial for that f(i) 2) Polynomial is order 2 with n=2, results in parabola

slide-23
SLIDE 23

23

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

3-point example (order 2)

f1 f2 f3 200 400 600 800 1000 1200 1400 1600 20 40 60 80 100 f(x) x

1) Polynomials for each f(i) are summed, giving interpolating function of order 2 2) Results in different derivative estimate than tangent function

slide-24
SLIDE 24

24

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Polynomial Parameter Estimation

  • 1. Use polynomials to

estimate derivative

  • 2. Transform derivative

estimate

  • 3. Use linear regression

to estimate line parameters

  • 4. Transform line

parameters back to Rayleigh form

  • 5
  • 4
  • 3
  • 2
  • 1

1 500000 1000000 1500000 2000000 2500000 ln(PDF/t) time squared (t^2)

Step 4 - ln(PDF/t)

slide-25
SLIDE 25

25

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Evaluating Rayleigh Method

slide-26
SLIDE 26

26

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Basic Method

  • 1. For each contract in data set, start with first 3

data points

– Generate Rayleigh fit using only first 3 points – Evaluate performance of that Rayleigh curve

  • 2. Repeat step 1 using first 4 data points,

continue until the last observation for the contract is reached

  • 3. Repeat steps 1 and 2 using different fitting

methods for comparison purposes

slide-27
SLIDE 27

27

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Data

  • 20 Programs from EVM data base
  • Criteria:

– Must be completed – At least 90% complete – Less than 25% complete at first report – RDT&E Funded, SDD contracts only

  • 320 curves to evaluate (per method)

Minimum Maximum Start Year 1994 2004 Duration (Years) 2.00 7.97 Value($K) 6,227 2,139,304

slide-28
SLIDE 28

28

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Important Notes

  • 1. Extremely poor fits are excluded

– Would add support to my conclusion

  • 2. Some parameter fits were not calculable

– i.e. LN(-1) – Excluded from analysis

  • 3. Focus of analysis was on first 18 points

– Evaluating use as an “early warning system”

slide-29
SLIDE 29

29

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

R2 of Polynomial Estimates

(average across all contracts reporting at observation (i))

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 10 20 30 40 50 60 70 80 90 100 R2 (Estimated Rayleigh vs. Data) Number of Observations Used to Estimate Parameters Order 2 Polynomial Order 3 Polynomial

Would expect R2 very close to 1.0 if data follows Rayleigh Curve Larger is better

slide-30
SLIDE 30

30

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Error of Cost Parameter Estimates

(average across 20 contracts)

0.00% 50.00% 100.00% 150.00% 200.00% 250.00% 300.00% 350.00% 400.00% 2 4 6 8 10 12 14 16 18 20 Absolute % Difference from Final Cost Number of Observations Used to Estimate Parameters CUMULATIVE CPI METHOD COST/SCHEDULE METHOD COST/SCHEDULE WEIGHTED FACTOR METHOD Traditional Average 2 Poly 3 Poly Optimization

Optimization and Derivative methods perform worse than traditional methods Smaller is better

slide-31
SLIDE 31

31

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Error of Time Parameter Estimates

(average across 20 contracts)

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 2 4 6 8 10 12 14 16 18 20 Average % Difference from Final Duration Number of Observations Used to Estimate Parameters 2 Poly 3 Poly Optimization

Optimization and Derivative methods don’t perform well Smaller is better

slide-32
SLIDE 32

32

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Percent Error Over All Points

(Average across 20 contracts)

0% 10% 20% 30% 40% 50% 60% 70% 80% 2 4 6 8 10 12 14 16 18 20 Average % Error Over Whole Contract Number of Observations Used to Estimate Parameters 3 Poly 2 Poly Optimization

Similar to R2 , expect minimal error over all points if EVM follows Rayleigh curve Smaller is better

slide-33
SLIDE 33

33

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

0.1 0.2 0.3 0.4 0.5 0.6 CPI Method Cost/Schedule Method CPI/SPI Weighted Method Optimization 2 Poly 3 Poly Estimated Percent Error /w 18 Observations Estimating Method

Lower Bound 5% Mean Upper Bound 95%

ANOVA (18 Observations)

Regression “Gold Card” Methods Optimization Rayleigh methods appear to perform worse than traditional methods

Can’t say with certainty that Optimization is worse!

Smaller is better

slide-34
SLIDE 34

34

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Conclusions

slide-35
SLIDE 35

35

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Conclusions

  • For this data set, Rayleigh estimates do not

improve on Gold Card methods

  • The Rayleigh model does not fit the contracts in

this data set

  • Analysts need to be careful if using Rayleigh

– Results need to be supported with actual data – Rayleigh is just a mathematical function, not magic

  • Consistent with results found by Abernathy

– Rayleigh parameters can be estimated, but no success as predictor

slide-36
SLIDE 36

36

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Future Work

  • Expand analysis to include more contracts
  • Continue to Refine parameter estimation tool
  • Use analysis to generate standard risk

distributions for R&D estimates

  • Develop guidance for optimization constraints
  • Evaluate other mathematical functions to

replace Rayleigh

slide-37
SLIDE 37

37

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Acknowledgements

  • 1. Mr. Mike Popp
  • 2. Mr. Marc Greenberg
  • 3. Mr. Ben Breaux
  • 4. Ms. Heather Brown
  • 5. Dr. Casey Trail
slide-38
SLIDE 38

38

UNCLASSIFIED (FOR OFFICIAL USE ONLY) UNCLASSIFIED (FOR OFFICIAL USE ONLY)

Questions?