Agenda 1. Problem Definition 2. Scope 3. Requirements 4. Approach - - PowerPoint PPT Presentation

Master’s Project Final Presentation for IF-BAD

(Instrument For Budget Analysis and Decision-making)

Fall 2018 Scott Kuhn Sasha Snisarevska Gang Xiang SYST 699

Agenda

• 1. Problem Definition
• 2. Scope
• 3. Requirements
• 4. Approach
• 5. Verification and Tool Use Examples

2

Agenda

• 1. Problem Definition

1.1 Sponsorship 1.2 Background 1.3 Problem Statement 1.4 Project Objectives 1.5 Stakeholders

• 2. Scope
• 3. Requirements
• 4. Approach
• 5. Verification and Tool Use Examples

3

Sponsorship

• SPA (Systems Planning and Analysis, Inc.)

○ Helps decision makers identify problems, assess risks, optimize resources, develop strategies, and test and refine ideas ○ Support a large variety of DoD and other government agencies missions ○ Created simulation and decision analytics tools and models

• SPA’s customer: SIAC (Strategic Intelligence Analysis Cell)

○ Analytic foundation for Department of Defense investment decisions to restore technical

• vermatch of the U.S. Armed Forces
• SIAC’s goals

○ Understand the enemy ○ Analyze U.S. capabilities ○ Track technology trends ○ Assess emerging threats

4

Background

• Research, Development, Test and Evaluation (RDT&E)

○ Seven levels of budget activity ■ 01 – Basic Research ■ 02 – Applied Research ■ 03 – Advanced Technology Development ■ 04 – Advanced Component Development and Prototypes ■ 05 – System Development and Demonstration ■ 06 – RDT&E Management Support ■ 07 – Operational System Development ○ The data is publicly available for previous years and five years in advance ○ For example, the 2019 President’s Budget was released in February 2018 and contains figures for 2017 and 2018, as well as projected amounts for 2019-2023 ○ Federal Budget FY 2019 for defense is around $700 billion, and has increased for the last 4 years

https://gallery.yopriceville.com/Free-Clipart-Pictures/Money-PNG/Money_Bag_PNG_Clipart_Picture#.W52j9uhKhPY

5

Background cnt’d

Two sources of data:

• Budget Database

○ President’s Budget data since 2011

• President’s Budget

○ publicly available XML data files Budget Database

6

Background cnt’d

• Acquisition Process Milestones (Hagan)

○ Milestone A - Start tech/risk research and analysis ○ Milestone B - Start engineering/manufacturing work ○ Milestone C - Start production, deployment and operations support

7

Data Structure

Project Milestone Accomplishment Program Element Service Budget Activity

Light Tactical Wheeled Vehicles

Service: Army PE#: 1 LTV Prototype Suspension Improvement Budget Activity: 5 PE#: 3 P#: 1 P#: 3 MS: B A#: 1 A#: 3

8

Problem Statement

Agencies conducting DoD research have milestones set years in advance for various capabilities they are developing, however their timelines may not be accurate, because they tend to slip or change over time. Publicly available budget sources hold timeline and milestone information, but there exists no tool that can analyze milestone data.

9

With what we started...

Thousands pages of pdf/xml documentation...

10

Project Objectives

• 1. Develop a system that tracks predicted milestone dates across accomplishments,

projects, program elements, and services.

• 2. Present milestone data across multiple years in an easy-to-read graphical element.

11

Agenda

• 1. Problem Definition
• 2. Scope
• 3. Requirements
• 4. Approach
• 5. Verification and Tool Use Examples

12

Scope

• RDT&E Budget Activity Numbers

○ 4 through 7 (Advanced Concept Development & Prototypes through Operational System Development)

• Procurement funding is outside the scope (done after RDT&E phase)
• 10/2/18 - Predicting accomplishment/milestone completion is outside scope

13

Stakeholders

• Department of Defense

○ Army, Navy, Air Force, DARPA

• DoD Laboratories
• Federally Funded Research & Development Centers and University-Affiliated

Research Centers

• Industry

14

15

Agenda

• 1. Problem Definition
• 2. Scope
• 3. Requirements

3.1 Mission Requirements 3.2 System Requirements 3.3 Functional Requirements

• 4. Approach
• 5. Verification and Tool Use Examples

16

Mission Requirements

MR1: System shall visualize RDT&E milestone data, over a predetermined period

System Requirements

SR1: System shall interface with existing milestone database sources SR2: System shall plot milestones for specified accomplishments over time

17

Functional Requirements

FR1: System shall support a capability of milestone analysis based on historical data. FR2: System shall provide a user-friendly Graphical User Interface (GUI). FR3: System shall support searches and filters for accomplishments by keywords, project descriptions, program element, project, and program element number. FR4: System shall display accomplishment descriptions FR5: System shall display project descriptions

18

Agenda

• 1. Problem Definition
• 2. Scope
• 3. Requirements
• 4. Approach

4.1 Technical Approach

• 5. Verification and Tool Use Examples

19

Technical Approach

20

IF-BAD’s Capabilities

• Database connection
• User’s Settings
• Filtering
• Search
• Sorting
• Selection/Deselection
• Plotting Milestones

21

Filter Capability

User must follow a hierarchical

• rder when input the data such as:

1. Service Title 2. Budget Activity Number 3. Program Element Title 4. Project Title Pros: High accuracy Cons: User must know details about the searched item.

22

Search Engine

The text search is carried out by Simple Search and Advanced Search. Simple Search: Single or multiple search for items

• f one type (e.g., titles, project

numbers) Advanced Search: “AND” or “combined” search through items of different types

• Ignores Uppercase/Lowercase, spaces

23

24

Demo

• Database connection
• Filtering
• Sorting
• Selection/Deselection

https://drive.google.com/open?id=1xzOx0ceqsBTemPWyvfKPvbrkP6kIOwfO

25

Get Milestones (1)

26

Levenshtein Distance

• Measure of similarity between two strings
• Number of deletions/insertions/substitutions required to transform one into the other

○ “GUMBO” and “GUMBALL” have a distance of 3 ○ Percent = 1 - Distance/LargestStringLength

Ex: Why Levenshtein? User wants “JWARN 2” milestones:

(Gilleland, 2006)

2014 JWARN 2 2015 3) JWARN 2 2018 JWARN Incr. 2 Same accomplishment, different titles! With String Matching With Levenshtein

27

Get Milestones (2)

• Inputs

○

• Accomp. ID

○

• Accomp. Title

○ Accomp.’s project’s Project ID

• Outputs

○ Formatted Output Data

28

Get Milestones(3) - Parse Milestones Out of Events

• Ignores special characters
• Avoids common non-milestone titles
• Looks through subproject, accomplishment, event titles
• Handles acronyms

29

Assumptions

• All milestone events have “ms” or “milestone” inside the title
• Accomplishment acronyms/titles can change over time, but not dramatically
• Accomplishments have milestones only within their project/program element
• Users will select accomplishments that are meaningful

○ In use: when milestone title is generic, it belongs to the selected accomplishment if the accomplishment is in the list of accomplishments

30

Limitations & Solutions

Limitations Solutions

Levenshtein Criteria Sensitivity. Too high, can’t grab any milestones. Too low, grabs unrelated milestones. Configurable in user settings, no need to touch code source No strict format for PB publications. President’s Budget content changes over time. User manual details how to edit code source. System has many features (Search, Filters, Code Source, Analysis). Is it usable? User manual. Help tooltips. Considers similar accomplishments as one (Ex: JWARN 2 vs JWARN Incr. 2)

31

Agenda

• 1. Problem Definition
• 2. Scope
• 3. Requirements
• 4. Approach
• 5. Verification and Tool Use Examples

32

Verification Test 1/Test 2

Test 1 (Search, Filter, Select, De-select)

1. System is open on the Tool/Main Page 2. Search/Filter for this accomplishment by (Filter/Simple Search/Adv Search) 3. Select accomplishment 4. Double click to read the accomplishment’s description 5. Only once, deselect one accomplish. Re-add it 6. Repeat 2-4 for all test accomplishments

Test 2 (Analysis, Hide, Unhide)

1. Do Test 1 2. Click “Start Analysis” 3. Hide all accomplishments, then unhide them 4. Check President’s Budget PDF files to confirm milestone analysis accuracy

33

13 chosen test accomplishments of various formats (After Slides)

Verification

System has met 6/6 requirements

• SR1 - Connect with DB
• Test 1
• SR2 - Plot Milestones
• Test 2
• F1
• Milestone Analysis
• Test 2
• F2 - GUI
• Inspection/Test 2
• F3 - Searches/Filters
• Test 1
• F4 - Accomplishment Descriptions
• Test 1
• F5 - Project Descriptions
• Test 1

34

Verification - Derived & Extra

Met 22/24 Derived or Extra Requirements Missing Extra Requirements:

• Text Analysis of Descriptions to Find

Source of Slippage

• Allow Users to Manually Link

Accomplishments

35

Tool Example 1 - One Accomplishment Under One Project

Example 1: Tracking GMV (Ground Mobility Vehicle) Testing progress over time Why am I tracking it? Understand progress of GMV (Ground Mobility Vehicle) Testing to ensure the U.S. Armed Forces has a plan to outpace our enemies What information can I get from this tool? See if planned milestones dates for GMV (Ground Mobility Vehicle) Testing have stayed in place or have been delayed, then compare those dates with enemies

36

37

38

Tool Example 2 - Keyword Search

Example 2: Searching armor piercing related technologies throughout the database Why am I tracking it? Have a broad view of all research which deals with armor piercing related technologies What information can I get from this tool? See the list of accomplishments which deal with armor piercing and can check on the progress of the selected armor related technology

39

40

“Funds in the 0604802A EP5 ADVAP program in FY 2017 are a realignment of funds from program 0603639A EC2, Advanced Armor-Piercing (ADVAP) for Small Cal Ammunition, for more efficient and effective program management. “

41

Green missing points due to dramatic title change, Levenshtein recognized the accomplishment title similarity and connected the titles from 2015 and 2019.

Future Recommendations

• Integrate with sponsor’s currently used budget analysis tools
• Research/improve current milestone analysis functions
• Add text analysis on descriptions (find source of slippage)
• Improve accomplishment linking over time (save to database, let user links manually)

2014 JWARN 2 2015 3) JWARN 2 2018 JWARN Incr. 2 With String Matching With Levenshtein 2019

Joint Warning & Reporting Network 2

With User Linking Manually

42

With what we started...

Thousands pages of pdf/xml documentation...

43

With what we ended up

44

Thank you! Questions?

https://www.filmsforaction.org/articles/how-to-fund-a-universal-basic-income-fairly/

• snisare@gmu.edu

peter.xi.xiang@gmail.com scott.l.kuhn@gmail.com

45

References

• Office of the Under Secretary of Defense. (2018). Defense Budget Overview.
• U.S. Army. (2017). RDT&E Budget Justification. Retrieved September 15,2018, from

http://www.dtic.mil/descriptivesum/Y2018/Army/U_0603801A_4_PB_2018.xml

• Hagan, G. (2011, April). Overview of the DOD Systems Acquisition Process. In DARPA SBIR Phase I Training Workshop.

Retrieved December 16, 2018, from https://www.darpa.mil/attachments/(6T4)%20Global%20Nav%20-%20Work%20With%20Us%20-%20For%20Small%20Busin ess%20-%20Resource%20(Approved).pdf

• Jayson C., Singh J., Sutton P. (2017) Department of Defense Research, Development, Test and Evaluation Budget Exhibit

Analysis Tool Enhancements. Retrieved September 15, 2018, from http://seor.vse.gmu.edu/~klaskey/Capstone/BLAST%20Website/docs/BLAST%20Proposal.pdf

• United States Department of Defense Fiscal Year 2017 Budget Request (PDF). Office of the Under Secretary of Defense. 2016.
• Gilleland, M. Levenshtein Distance, in Three Flavors. Retrieved November 28, 2018, from

https://people.cs.pitt.edu/~kirk/cs1501/Pruhs/Spring2006/assignments/editdistance/Levenshtein%20Distance.htm

46

Deliverables

• Final Report Paper (12/4)
• Software Tool (12/4)
• Website (12/4)

○ We will ensure all information on website is acceptable to release publicly

• Final Presentation (12/7)
• User Manual (12/4)
• Sponsor Handoff (~ 12/12)

47

Use Case Diagram

48

Activity Diagram

49

Activity Diagram 2

50

51

52

Filter/Sort Options

• Filter

○ Service ○ Budget Activity (BA) ○ PE ○ Project

• Sort

○ Alphabetical ○ Budget Delta (Change in budget compared to last fiscal year)

• Search

○ Text Search (Titles, descriptions, PE Numbers) ○ PBYear ○ BA

53

Get Milestone (4) - Testing a String for our Selected Accomp. Title

• Ignores special characters
• Avoids common non-milestone titles
• Looks through subproject, accomplishment, event titles
• Handles acronyms

54

Milestone Levenshtein Methods

Situations Only One Accomplishment? Has Subproject Title? Detailed Milestone Title? Generic Milestone Title? Examples Project: F-22 Accomp List: 1. Jet Fuel Event { Event Title: “Milestone B”, Event Year: 2016, Subproject: “F 22 Jet Fuels” } Event { Event Title: “F-22 Jet Fuel MS B”, Event Year: 2016, Subproject: “” } Event { Event Title: “Milestone B”, Event Year: 2016, } Accomp List: 1. F-22 Fuels 2. Documentation 3. Tech Analysis Solutions

Compare “F-22 Jet Fuel” to “Jet Fuel” Compare “F-22 Jet Fuel” to “F 22 Jet Fuels” Compare “F-22 Jet Fuel” to “F-22 Jet Fuel MS B” Compare “F-22 Jet Fuel” to All Accomplishments

Example: I want milestones for Selected Accomplishment Title “F-22 Jet Fuel”

55

Test Input Set

12 Valid Accomplishments, with a variety of different formats 1 Invalid Accomplishment

56