A Comprehensive Approach for Lifelong Learning Dr. Darcy Lilley, - - PowerPoint PPT Presentation

A Comprehensive Approach for Lifelong Learning

• Dr. Darcy Lilley, EdD., DAFC

Air Mobility Command Kevin Cincotta, CCEA, PMP ICF International Lauren Nolte, CCEA ICF International

1

2014 ICEAA A Annual Professional essional Developmen ment t Worksh shop

• p

June 10-13, 2014 Denver, CO

Abstract: Key Points

• The Air Force Air Mobility Command (AMC) Enterprise Learning Office

(ELO) mission is to transform AMC into a premier Air Force learning

• rganization, achieve learning through optimum approaches and develop

Mobility Airmen into life-long learners and well -cultivated critical thinkers who demonstrate institutional Air Force competencies with a positive approach to managing their own learning. In this context, learning has three main components: training, education, and experience. The re-engineering

• f learning to develop and deploy optimum approaches focuses on all
• components. AMC ELO is initially focusing on training.
• Training is generally represented as only one line within a cost estimate.
• This paper presents a training CES, conveys its value in the broader

context of transforming learning, and outlines an approach for using the CES in the context of a BCA. Finally, preliminary results of the BCA are presented and interpreted.

2

Agenda

• Background
• Implications for the BCA
• Cost Element Structure (CES), Benefits

Estimating Structure (BES), and Data Collection Instrument (DCI)

• Business Case Analysis (BCA)

Implementation

• Preliminary Results
• Conclusion

3

Background: AMC ELO

• The Air Force Air Mobility Command (AMC) provides global air

mobility to our Armed Forces

• The command also plays a crucial role in providing humanitarian

support at home and around the world. AMC Airmen—active duty, Air National Guard, Air Force Reserve and Civil Reserve Air Fleet - provide airlift and aerial refueling for all of the United States' armed

• forces. Many special duty and operational support aircraft and

stateside aeromedical evacuation missions are also assigned to AMC

• The mission of the AMC Enterprise Learning Office (ELO) is , in part, to

transform AMC into a premier Air Force learning organization. Key components include:

– Achieve learning through optimum approaches – Develop critical thinking skills – Develop Mobility Airmen into lifelong learners – Promulgate and socialize positive approach to Mobility Airmen managing their own learning

4

Background: Selected Key Terms

• Agile Learning Design: An approach to content development that focuses on speed,

flexibility and collaboration. The term evolved from the software development industry, in which electronic content development (e.g., e-learning) has similar characteristics to software development.

• Andragogy: The methods , techniques , or teaching strategies used (specifically) for

adult learners.

• Blended Learning: A formal education program in which a student learns, at least in

part, through online delivery of content and instruction with some element of student control over time, place, path or pace.

• “Flip the Classroom:” A form of blended learning in which students learn new

content online by watching video lectures, usually at home, and what used to be homework (assigned problems) is now done in class with teachers offering more personalized guidance and interaction with students, instead of lecturing. This is also known as backwards classroom, flipped classroom, reverse teaching

• Virtual Learning Environment (VLE): A web-based education system based that

models conventional in-person education by providing equivalent virtual access to classes, class content, tests, homework, grades, assessments, and other external resources such as academic or museum website links. It is also a social space where students and teacher can interact through threaded discussions or chat.

5

Background: Business Case Analysis (BCA)

• One of the key tasks associated with transformation of learning at

AMC is a Business Case Analysis (BCA)

• A BCA is a comparative analysis among competing alternatives

– Not to be confused with a budgetary estimate – Defines a Status Quo (SQ), As-Is, or Baseline Alternative (Alternative 1) – Defines one to three non-SQ To-Be Alternative(s) – Wherever possible, monetizes costs and benefits associated with each alternative, including implicit costs, imputed costs, and (some) externalities – Sunk costs excluded – Wash costs may be included or excluded at analyst discretion – Each alternative subject to identical overarching ground rules and assumptions and period of analysis

• Each alternative is estimated using a common Cost Element Structure

(CES) and Benefits Estimating Structure (BES)

6

• Baseline Alternative: Reflects the state of the world in which AMC

training proceeds as it would have without any of the contemplated elements of transformation

• To-Be Alternative: Reflects the state of the world in which AMC

training is transformed using a variety of optimum approaches to

• learning. For the purpose of the BCA, the transformation includes

several key elements:

1. Implementation of Agile Instructional System Design (ISD), an approach to training content development that focuses on speed, flexibility, and collaboration 2. Modification of the virtual vs. classroom mix of courses using blended learning environment 3. Implement best practices related to andragogy, including Flip the Classroom 4. Make maximum feasible use of VLEs

7

Background: AMC BCA

The BCA for AMC ELO considers two alternatives:

Implications for the BCA

• We need a CES to organize costs
• We need a BES to organize benefits
• We need to quantify the costs and benefits of the

Baseline and To-Be Alternatives, and monetize that value wherever possible

• Approaches must be in line with industry best

practices, well-documented, traceable, and repeatable

• Ideally, we would like to set the standard for how

training cost estimates and comparative analyses are to be done

8

Cost Element Structure (CES)

Follows many best practices of, but not to be confused with, a Work Breakdown Structure (WBS)

9

Attribute CES WBS 1

Elements are… Cost Elements Work Packages Follows general outlining principles, Follows 100% rule   Mutually Exclusive   No Orphans   No Single Children   May Contain Activities   Highest Level Is… At least two phases One Deliverable

• 1. Based on WBS best practices found in the Project Management Body of Knowledge (PMBoK) Guide, 5th edition.

WBS vs. CES: Which One for Cost Estimating?

• It is a best practice to use a CES to breakdown elements of cost, just as

it is a best practice to use a WBS to breakdown elements of work

• In the Project Management Body of Knowledge (PMBoK) Guide, all

costs are estimated at the Activity Level (rolled up to the work packages they support later)

• PMBoK, The Cost Estimating Body of Knowledge (CEBoK), and the GAO

Cost Estimating and Assessment Guide state that WBS’s should be product-oriented

• But our end item is not a product. It is a service
• We propose that cost estimates should be oriented toward the

ultimate aim of the program, regardless of whether it is a product. An activity-oriented CES is an appropriate breakdown for the training

• service. It also facilitates activity-based costing (ABC).
• The items are beyond the scope of MIL-STD-881C, which contains no

appendix for Training, no provisions for the acquisition of defense non- materiel items, and no guidance beyond the acquisition phase

10

WBS/CES Research

• Our literature review yielded very little in the way of

standardized, published WBS’s and CES’s that are specific to training

• Usually just two cost elements (Initial Training, Recurring

Training), to train users on a thing. Here, training is the thing!

• Some suggestions we received:

– Heavily customize AIS/ERP Appendix of MIL-STD-881C – Heavily customize OSD PA&E AIS EA Guide – Heavily customize DHS IT LCC WBS – Take a long walk off of a short plank

• We eventually found a specific training Cost Breakdown

Structure (CBS) developed by the North Atlantic Treaty Organization (NATO) 1

11 1. Smit, Marcel C. A Cost Analysis of Military Training. North Atlantic Treaty Organization (undated).

NATO Training CBS

• Provided a good starting point for our training CES
• At the highest level, organizes cost elements into three main buckets:

1.0 Investment, 2.0 Operating & Support, & 3.0 Decommissioning

• Investment & Decommissioning elements contain a useful way to decompose

the cost of implementing a new, “to-be” training alternative

• Operating & Support cost elements provides good ideas on capturing the

costs of operating a training program that is already “up and running” (e.g. cost of instructors, training devices, updating courseware, software licenses)

• While we used the Decommissioning portion without modification, we

found some limitations with the Investment and Operating & Support structures:

– Little visibility into hardware vs. software costs – Full cost of student time not captured – No labor costs for managing training investment and operations – No breakout of instructor and student time by type (e.g. Military Active, Civilian, Contractor, etc.) – Distinction between training content delivery and training content maintenance unclear

12

NATO Training CBS

13

Investment Operating & Support Decommissioning

AMC Training CES

Our CES borrows elements from the NATO training CBS but is expanded and reorganized to facilitate comparison of alternatives and the ability to answer questions posed by leadership

– Added additional elements needed to account for the full cost of training

• Student labor
• Program management labor

– Added additional, lower-level elements needed to distinguish among costs requiring different data & cost estimating techniques

• Active Military vs. Reserve Military vs. Civilian vs. Contractor Personnel
• Hardware vs. Software

– Re-organized portions of the structure to better reflect the activities that compose training, creating distinctions between:

• Content delivery vs. content maintenance
• Aircrew vs. non-aircrew training – an important distinction for AMC as these

two types of training use different systems, processes, and buckets of money

14

AMC Training CES

15

The result is a comprehensive structure for representing training costs that goes well beyond the normal 2-3 lines for training

Training costs as represented in the MIL-STD-881C WBS for Automated Information Systems

Our new Training CES (140 elements)!

Training CES – 1.0 Investment

16

Adds cost to manage Investment Breaks out the hardware and software components of training device acquisition Breaks out content development into lower-level activities Breaks out labor costs by type Includes student labor

Training CES – 2.0 Operating & Support

17

Adds cost to manage training Creates separate categories for Content Delivery vs. Content Maintenance/Updates (further broken out into Aircrew vs. Non-Aircrew) Breaks out labor costs by type Includes student labor Creates clear

• rganization of

maintenance

• vs. refresh costs

for both hardware and software

Training BES Research

• Similar to the CES research, we found little in the way
• f standardized, published BES’s that are specific to

training

• We reviewed the BES provided in the OSD PA&E AIS

EA Guide 1, but this structure lacks an intuitive way to

• rganize and categorize benefits

– Top level elements are organized around “types of money”/costs - RDT&E, O&M, Procurement, Construction, and Military Personnel 2 – While this could be useful if all benefits could be expressed in terms of cost savings, not all benefits can be translated to a cost savings within the timeframe of the analysis

18

• 1. Wilson, Ronald C. OSD PA&E AIS EA Guide (1995).
• 2. Structure is derived directly from the major cost categories defined in OMB Circular A-11, Preparation and Submission of Budget Estimates

Training BES Research

• We decided to base our structure largely on the guidance provided in the

Cost Estimating Body of Knowledge (CEBoK) 1

• Structure organized around three major types of benefits:

– 1.0 Quantifiable Monetary – 2.0 Quantifiable Non-Monetary – 3.0 Non-Quantifiable

• 1.0 Quantifiable Monetary was further decomposed into 1.1 Cost Savings,

1.2 Cost Avoidance, and 1.3 Improved Productivity benefit elements (also based on CEBoK guidance)

• Finally, we based the remaining lower-level elements on the initiatives being

pursued in the To-Be Alternative

– In the case of our BCA, all of our Quantifiable Monetary lower-level elements fell into 1.1 Cost Savings or 1.3 Improved Productivity – Developed specific 1.1 and 1.3 lower-level elements based on an analysis of how the two key initiatives of the To-Be Alternative drive both cost savings and productivity improvements (see mapping on next slide)

19

• 1. Cost Estimating Body of Knowledge (CEBoK) Module 13, “Economic Analysis”, International Cost Estimating and Analysis Association

Training BES Creation

20

Notional Implementation of “To-Be Alternative” New Agile ISD Process Increase % of “Virtual” Courses Facilities Rental Travel/TDY Refreshments Contractor Labor Core Material Updates & Maintenance More Instructor Time More Student Time

1.1 Cost Savings 1.3 Productivity Improvements

Less time (labor cost) needed to update content Costs reduced when there are less classroom courses More “self-paced” courses reduces need to contract instructors Virtual Self-Paced courses – reduced time to complete courses, instructors do not have to be present All Virtual courses– reduced or no travel time

1.1.2 1.1.1.1.1-1.1.1.1.3 1.1.1.2.1-1.1.1.2.3 1.1.1.1.4, 1.1.1.2.4 1.3.1.-1.3.2

Blue element Red element

= Linked to “Increase %

• f Virtual Courses”

= Linked to “New Agile ISD Process”

Key

Complete Training BES

21

Majority of the BES is made up of “1.0 Quantifiable Monetary” benefit elements – either 1.1 Cost Savings or 1.3 Improved Productivity

1.1 Cost Savings 1.3 Improved Productivity

NOTE: BES was designed to measure incremental benefits, or benefits relative to the Baseline Alternative

Avoiding Double-Counting in BES

22

• Important not to double-count. For example, Cost Savings (1.1 Benefit

Elements) can easily be represented as a reduction in certain cost elements in the CES

• We followed CEBoK guidance and chose to address Cost Savings in the

cost estimate and Improved Productivity in the benefits estimate

– Cost Savings are still represented in the BES, but the dollar amount is shown as $0 – Cost Savings BES elements are closely tied to the CES elements where the savings are shown

Example: Content Delivery (Aircrew Training) BES CES Each of the Cost Savings BES elements map to a corresponding CES element where the cost savings are captured in the cost estimate for the To-Be Alternative

Data Collection Instrument (DCI)

• Developed DCI to facilitate collection of the Baseline Alternative

costs

• Only 2.0 Operating & Support elements apply to the Baseline
• Initial conversations with stakeholders indicated they would

prefer to report costs at a higher level and “allocate” those costs to lower-level elements using percentages

– Time and effort to collect cost data at the lowest levels too great

• Excel-based instrument includes:

– Interactive tree structure containing all 2.0 cost elements – Input areas for costs and percent allocations – Detailed definitions for all cost elements – Sample cost drivers (variables) for each cost element, that the stakeholder can provide in the absence of actual cost data (e.g., for Travel/TDY costs, stakeholder can provide “number of trips (and people traveling) per year, with starting and ending locations, and duration”)

23

Data Collection Instrument (DCI)

24

Tree structure containing the cost elements expands/collapses Enter Baseline costs for “level 2” cost elements Provide allocations percentages for remaining, lower-level cost elements If stakeholder cannot provide costs, instrument provides guidance on what variables to provide for each cost element (to help us estimate)

Using the DCI - Issues

• Further conversations with more stakeholders revealed that each

group has different data collection limitations:

– Proprietary information (e.g. contract data) – Level of detail at which cost data exists varies by organization

• Many questions came up about the scope of the analysis – what to

include or not include

– What if part of the training is funded by another organization outside AMC? – What about training that does not have a defined curriculum/syllabus?

• Found that we needed some additional context about the data we

received in order to avoid issues with double-counting and/or not including everything

– Do the costs being provided represent everything for training? If not, what percentage is represented? – Are any other organizations within (or outside) AMC funding part of the costs?

25

Using the DCI – Addressing the Issues

• Developed customized DCIs for each stakeholder group that can be mapped

back to the same CES

• Created Scope Document that provides a clear definition of the “universe” of

costs to collect and guidance for handling different types of scenarios (e.g. how to handle AMC-funded training costs for non-AMC students)

• Developed list of “Context Questions” that we asked stakeholders to answer

along with their data submission (e.g. “What is the current expectation for how each of the costs will change over the next 10 years?”)

26

Enter costs at the lowest level; DCI automatically sums the parent-level elements (no percent allocations) Example: DCI customized to collect cost data at the lowest possible cost element

Collecting Data for To-Be Alternative

• Approach for estimating costs

– Investment & Decommissioning costs (unique to the To-Be Alternative) estimated based on market research of similar transformation initiatives – Operating & Support costs were estimated as deltas from the Baseline costs, based on:

• Market research on expected cost savings or increases where applicable
• Algorithms using parameters about the new environment created by the To-

Be Alternative (e.g. percentage of training that will be classroom-based vs. virtual)

• Approach for quantifying benefits

– All benefits except Productivity Improvements captured on the cost side – Market research of the productivity changes the To-Be Alternative will create (e.g. what the reduced instructor time per course will be) – Algorithms using parameters about the new environment created by the To-Be Alternative (e.g. percentage of training that will be classroom-based

• vs. virtual)

27

BCA Implementation

• Purpose of the Business Case Analysis (BCA) is to recommend a

preferred alternative, based on a comparative analysis of the costs and benefits of the Baseline and To-Be Alternatives

– We chose to limit the scope of our BCA to only monetized costs and benefits (All costs in the CES and the 1.0 elements in the BES) – Used a 10 year period of analysis

• Analysis requires a Cost Benefit Model with the following

components:

– Ability to alter parameters and assumptions around the alternatives to “test” different scenarios – Costs and benefits of each alternative phased by FY, adjusted for inflation, and discounted to represent the data in present value terms – “Then-Year” costs of each alternative phased by FY to be used for budgeting – Automatic calculation of “measures of merit” used to evaluate the To-Be Alternative

28

Summary: BCA Cost & Benefits by Alternative

29

Baseline Alternative To-Be Alternative

Costs 1.0 Investment None



2.0 Operating & Support

 

Note: Defaults to Baseline cost unless there is an identified cost delta 3.0 Decommissioning None



Benefits (Quantifiable Monetary) 1.1 Cost Savings None Note: All benefits are measured relative to the Baseline



Note: Shown as $0 in the BES, as they are captured as cost savings in the 2.0 portion of the CES 1.3 Productivity Improvements



BCA - Parameters

30

Cost Benefit Model has four main parameters tabs containing data that the user may enter/modify:

– General Parameters – contains global parameters such as the OMB discount rates used to calculate discounted, Present Value (PV) dollars – Baseline Alternative 2.0 Costs – contains all 2.0 Operating & Support costs for the Baseline, collected via the DCI, in FY 2015 $ – To-Be Alternative 1.0 and 3.0 Costs – contains estimated 1.0 Investment and 3.0 Decommissioning costs for the To-Be Alternative, in FY 2015 $ – To-Be Alternative Benefits & 2.0 Cost Savings – contains the estimated monetary value of both Productivity Improvements and 2.0 Operating & Support Cost Savings, calculated based on user-entered parameters (e.g.

• ld vs. new mix of virtual vs. classroom training)
• Also contains ability to modify the schedule for phasing in cost savings

and productivity improvements (e.g. 0% of savings realized in first year, 50% realized in second year, etc.)

BCA – Parameters (cont.) NOTIONAL DATA ONLY

31

Modify Parameters See how it impacts the value of Productivity Improvements & Cost Savings

NOTE: All data are notional

To-Be Alternative Benefits & 2.0 Cost Savings Parameters

Note: For illustration purposes, only a subset

• f parameters shown here

BCA – Costs & Benefits NOTIONAL DATA ONLY

• Calculated both Cost and Benefit BY and PV dollars by element

and FY (2015-2024)

– BY $ remove the effects of inflation and put all dollars in terms of one base year (2015) – PV $ are needed to account for the time value of money, in order to equitably compare the two alternatives

• Calculated only Cost TY dollars by cost element and FY (2015-

2024) – to be used with the chosen alternative for budgeting

32

Each cost element mapped to an appropriation type in order to calculate TY dollars using the Air Force inflation rates provided by appropriation

NOTE: All data are notional

BCA – Measures of Merit

33

Measure Purpose Interpretation Calculation

Net Present Value (NPV) Represents the discounted value of expected net benefits over the period of analysis (10 years) Higher = Better, >$0 is favorable NPV 1 = PV(Benefits) – PV(Incremental Cost) Savings/ Investment Ratio (SIR) Highlights the relationship between financial benefits and the investment needed to achieve those benefits Higher = Better, >1.00 is favorable SIR = [PV(Benefits) – PV(Incremental Cost)]/PV(Investment)

Model automatically calculates four Measures of Merit used to evaluate the To-Be Alternative:

• 1. Note: While CEBoK calculates this measure using all costs and benefits of each alternative (and then compares), we chose to
• nly look at incremental costs and benefits of the To-Be Alternative. This convention makes the NPV of the Baseline $0, by

definition.

BCA – Measures of Merit (cont.)

34

Model automatically calculates four Measures of Merit used to evaluate the To-Be Alternative:

Measure Purpose Interpretation Calculation

Discounted Payback Period Determines the length of time needed for an alternative to realize enough savings to

• ffset the

investment costs Lower = Better Calculated by finding the time needed (in years) for PV(Cumulative Savings) to exceed PV(Cumulative Investment) (we automated in Excel) Real Internal Rate of Return (IRR) Capture the merit of the alternative as a single percentage value – the real discount rate which makes NPV = 0 Should be positive in

• rder to be

considered a worthy alternative Higher = Better Excel function calculates the % based

• n the Net Value (BY) of the To-Be

Alternative in each year of the period

• f analysis

Preliminary Results NOTIONAL DATA ONLY

35

NOTE: All data are notional

Measure Value Interpretation

Net Present Value (NPV) $85,497.7K Over 10 years, the expected present value of net benefits provided by the To-Be Alternative is over $85M Savings/ Investment Ratio (SIR) 10.21 There is a greater than 10:1 ratio between the financial benefits of the To-Be Alternative and the investment needed to achieve those benefits over the 10 year period Discounted Payback Period 2.3 years It will take a little over 2 years for the investment in the To-Be Alternative to pay for itself, in discounted dollars Internal Rate of Return (IRR) 95% A 95% real discount rate would be needed to make the NPV=0

To-Be Alternative Measures of Merit

Conclusion

• Transforming AMC into a premier Air Force learning
• rganization through optimum approaches to training requires

an analytically sound BCA in order to:

– Obtain a clear measurement of the costs and benefits of the transformation relative to the “current state of the world” – Justify the initial investment needed to implement the transformation

• We have established a new standard for estimating training

costs and conducting a comparative analysis of training investment alternatives

– Created comprehensive CES and BES exclusively for training – Developed BCA (and supporting Cost Benefit Model) that is well- documented, traceable, and repeatable—in a framework uniquely suited to evaluation of alternative training investments and frameworks

36