Data-Driven Guidelines for Correlation of Cost and Schedule Growth ICEAA Training Workshop, San Diego Summer 2015 Sidi Huang Dr. Mark Pedigo Cris Shaw Ken Odom This document is confidential and is intended solely for the use and information of the client to whom it is addressed.
Correlation Effects in Monte Carlo Simulations One of the most difficult things to account for in a Monte Carlo simulation is correlation between the independent variables – If correlation is not accounted for, the Coefficient of Variation of the top level distribution will be artificially shrunk When independent variables are correlated, they will tend to grow and shrink in tandem. Ignoring correlation will result in a poor analysis; generally reporting overly optimistic results. Correlation is just an observed relationship, there does not have to be an explanation for why it happens, although often we want to know if there is one 2
Default Guidance for Correlation Values Although there have been empirically driven studies for cost correlation, there have not been considerable empirical studies on task duration correlation “Using reasonable nonzero values, such as 0.2 or 0.3, generally leads to a more realistic representation of total cost uncertainty” “Estimating System Cost” (Stephen A. Book, Crosslink Winter 2000/2001 ) Perform data-driven research to find a default correlation guidance value for schedule uncertainty Data files from historical NASA missions – NASA Cost Analysis Data Requirements (CADRe) – Mission Milestone Reviews – Mission Quarterly or Monthly Status Reviews 3
Correlation of Schedule Activity Durations Approach #1: Methodology Find consistent milestones for two Missions (e.g., PDR, CDR, and Launch); missions were held constant Focus on finding a correlation by Activity (e.g., spacecraft development) between two missions Take the many correlation values for each Activity for many missions, pool the correlation results for that Activity (e.g. spacecraft development) and compare to correlation results for another Activity (e.g. instrument development) to find “magic correlation value” Spacecraft development growth from PDR to CDR 4
Correlation of Schedule Activity Durations Approach #1: Outcome Three data points were too few to capture correlation Spacecraft Schedule Growth 20% 19% 18% 17% Percent Growth 16% 15% Program 1 14% Program 2 13% 12% 11% 10% 1 2 3 Milestones 5
Correlation of Schedule Activity Durations Approach #1: Observations Not every Mission had data for the same milestone reviews (over 15 different milestone reviews identified) With few data points, the correlation was very volatile; required the two missions to react in similar ways for each milestone Lesson learned: Find a way to increase the number of data points captured 6
Correlation of Schedule Activity Durations Approach #2: Methodology To solve the problem of too few data points, find more data for one mission (either from multiple milestones or from multiple monthly status reviews) that captured two activities; activities were held constant Find a correlation between two activities (e.g., spacecraft and instrument development) between one mission After identifying correlation values for many missions, pool correlation results for that activity (e.g. spacecraft development) and compare to correlation results for another activity (e.g. instrument development) to find “magic correlation value” 7
Correlation of Schedule Activity Durations Approach #2: Outcome Plausible but extraordinarily time intensive Some missions might have many years of no growth, then one period of growth (e.g. due to schedule replan) Mission Schedule Growth (Two instruments) 60% 45% Percent Growth 30% Instrument 1 Instrument 2 15% 0% 0 2 4 6 8 10 12 14 16 Period 8
Correlation of Schedule Activity Durations Approach #2: Observations Extraordinarily time intensive Required every monthly or quarterly status review to show the same level of detail When an Activity was completed, no more changes were made and correlation data ended Lessons learned: Focus on “early” and “late” data 9
Correlation of Schedule Activity Durations Approach #3: Methodology Find correlation of uncertainty distributions in schedule duration – Implying schedule estimates have uncertainty distributions – Two estimates for two activities have correlated uncertainty distributions Collect an “early” estimate and a “late” estimate – The “early” estimate will be an estimate that has a lot of uncertainty – The “late” estimate will not have very much uncertainty and will primarily be actuals – The difference in estimation is the uncertainty that occurred (i.e. the uncertainty distribution closed) Compare the early-estimation to late-estimation differences – This comparison will capture the early estimate uncertainty – If the estimation differences are correlated then the uncertainty distributions are correlated 10
Correlation of Schedule Activity Durations Approach #3: Outcome Instruments Instrument Instrument Mission When missions are Comparison X growth Y growth loaded with multiple Mission 1 1:2 11.4% 16.4% instruments, there is 1:3 11.4% 11.0% an opportunity for 1:4 11.4% 3.4% greater 1:5 11.4% 21.2% instrument:instrument 2:3 16.4% 11.0% comparisons 2:4 16.4% 3.4% 2:5 16.4% 21.2% A random sample of 3:4 11.0% 3.4% up to 3 data points per 3:5 11.0% 21.2% mission 4:5 3.4% 21.2% (instrument:instrument Mission 2 1:2 54.4% 16.5% comparison) was 1:3 54.4% -0.7% taken 2:3 16.5% -0.7% Mission 3 1:2 24.2% 21.3% 11
Correlation of Schedule Activity Durations Approach #3: Empirical Results Spacecraft : Instrument schedule growth correlation – Pearson’s r = 0.679 (13 missions; n = 13) 100.0% Spacecraft : Instrument Schedule Growth 80.0% Instrument Development 60.0% 40.0% Spcrft : Instrmt 20.0% RESIDUALS 0.0% 1.5 1 Residuals -20.0% 0.5 0 -50.0% 0.0% 50.0% 100.0% -40.0% -0.5 -50.0% 0.0% 50.0% 100.0% 150.0% 200.0% 250.0% -1 X Variable 1 Spacecraft Development 12
Correlation of Schedule Activity Durations Approach #3: Empirical Results Instrument : Instrument schedule growth correlation – Pearson’s r = 0.605 (34 instruments across 9 missions; n = 25) 140.0% Instrument : Instrument Schedule Growth 120.0% 100.0% 80.0% Instrument Y 60.0% Instrmt : Instrmt 40.0% RESIDUALS 20.0% 1 Residuals 0.0% 0.5 0 -20.0% -50.0% 0.0% 50.0% 100.0% 150.0% -20.0% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% 120.0% 140.0% -0.5 Instrument X X Variable 1 13
Correlation of Costs Empirical Results PM/SE : Flight System cost growth correlation – Pearson’s r = 0.117 (26 missions; n = 26) 160% PM/SE : Flight System Cost Growth 140% Flight System / Spacecraft (growth%) 120% 100% PM/SE : Flight 80% system RESIDUALS 60% 3 40% 2 Residuals 1 20% 0 0% 0% 50% 100% 150% -1 -50% 0% 50% 100% 150% 200% 250% 300% 350% 400% -20% -2 Program Management / Systems Engineering (growth%) X Variable 1 14
Correlation of Costs Empirical Results PM/SE : Payloads cost growth correlation – Pearson’s r = 0.394 (26 missions; n = 26) 250% PM/SE : Payloads Cost Growth 200% Payloads (growth%) 150% PM/SE : Payloads 100% RESIDUALS 3 50% 2 Residuals 1 0% 0 -100% -50% 0% 50% 100% 150% 200% 250% 300% 350% 400% -100% 0% 100% 200% 300% -1 -50% -2 Program Management / Systems Engineering (growth%) X Variable 1 15
Correlation of Costs Empirical Results Flight System : Payloads cost growth correlation – Pearson’s r = 0.303 (29 missions; n = 29) 250% Flight System : Payloads Cost Growth 200% Payloads (growth%) 150% Flight Sytm: Pyld 100% RESIDUALS 1 50% 0.5 Residuals 0 0% -100% 0% 100% 200% 300% -20% 0% 20% 40% 60% 80% 100% 120% 140% 160% -0.5 -50% -1 Flight System / Spacecraft X Variable 1 16
Conclusions and Future Research Ignoring correlation in running Monte Carlo simulations generally report overly optimistic results For schedule uncertainty, a default correlation value closer to 0.6 is shown to be effective Further categorization of missions by certain metrics could find unique correlation values for different types of missions – Example metrics: mission duration, cost, launch year, mass, power – Advanced metrics: schedule topology, missions where costs are skewed toward flight system versus costs skewed toward payload Categorization of cost elements to Time-Independent or Time- Dependent costs, and reconciling differences between WBS elements and CES elements 17
Recommend
More recommend
