Modeling & Simulation, Testing & Validation GETTING AHEAD - - PowerPoint PPT Presentation

Modeling & Simulation, Testing & Validation

7/31/2018

GETTING AHEAD OF THE GAME: QUANTIFYING THE VALUE OF LABORATORY TESTING PRIOR TO FULL-UP SYSTEM LEVEL

DISTRIBUTION STATEMENT A. Approved for public release; distribution unlimited. (OPSEC1189)

Joel Luna Sharon Snider Frontier Technology, Inc. Beavercreek, OH Mark Brudnak, PhD Bryan LaRose Melissa Morgan Dan Kosinski Mike Scott US Army TARDEC Warren, MI

Modeling & Simulation, Testing & Validation

7/31/2018

Background

• Automotive Reliability T&E Working Group, c. 2014
• Goal: Improve effectiveness and efficiency of government automotive

reliability T&E through development of guidance, policy, training.

• Initial Findings:

– Insight from industry and lessons learned can be leveraged to deliver a more reliable and sustainable product to the soldier – Government full-up system level testing is not excessive, rather inefficient for reliability growth

– Government reliability testing should be integrated with the systems engineering process to include developmental testing of components and subsystems with defined entrance criteria prior to FUSL testing (crawl – walk – run)

– Tailor testing to align with full range of operational usage expectations rather than average usage (OMS/MP).

Modeling & Simulation, Testing & Validation

7/31/2018

Need for Development Testing

Reliability Test & Evaluation is laid over the system engineering & development process Currently full-up system level T&E is utilized to address continued system reliability shortfalls

• Planning assumes relatively mature system on entry.
• Practice shows many systems do not meet entry goals.

Less testing for FUSL evaluation – More testing for development!

Evaluation

(full up system testing)

Development

(capability integration)

B C

Modeling & Simulation, Testing & Validation

7/31/2018

Need Optimal Balance of Lab and Field Testing

Lab Testing Field Testing

Suitable mix

Pros: Early FM discovery Time compression Repeatability/Control Lower cost Cons: Partial FM coverage No environment Subsystem  system reliability translation Pros: Natural environment Established method Full system test Cons: Late FM discovery Cost Less repeatability/ control Minimize:

Time Cost Risk

Modeling & Simulation, Testing & Validation

7/31/2018

Lifecycle Test Optimization SBIR A16-085 Effort

Modeling & Simulation, Testing & Validation

7/31/2018

Army Lifecycle Test Optimization (ALTO) Tool Dashboard Input Test Monitoring Calculator Tools

Modeling & Simulation, Testing & Validation

7/31/2018

Sample ALTO Workflow Concept

Modeling & Simulation, Testing & Validation

7/31/2018

Steps for Creating a Study in ALTO

1. Create study file specific to the Unit Under Test (UUT) 2. Define initial planning factors and constraints (reliability, schedule, budget, risk) 3. Import/define data (e.g., FMECA data, subsystem/failure mode assignment, MS and FEF values) 4. (Optional) Explore characteristics of data 5. Obtain optimal test allocations between subsystems/system and phases using ALTO Optimizer 6. Review dashboard, determine areas of concern 7. Refine constraints and planning factors as needed, including revising schedule 8. Update data based on current test progress 9. Repeat 5-8 as needed

Modeling & Simulation, Testing & Validation

7/31/2018

Importing, Viewing, and Editing Data

Failure Mode, Component, or Part Failure Rates Subsystem Level Failure Rates Add New Failure Rates

Modeling & Simulation, Testing & Validation

7/31/2018

ALTO Dashboard for Quick Review and Re-Planning

Status for Subsystem, System DT and OT Make adjustments to Inspect and Resolve Issues Schedule Summary System Reliability Estimate

Modeling & Simulation, Testing & Validation

7/31/2018

Views to Quickly Make Adjustments to Test Plan

Adjust Optimization Parameters Adjust Test Planning Inputs Subsystem System View Calculated Results

Modeling & Simulation, Testing & Validation

7/31/2018

Test Monitoring

Test Miles by Test, Unit Under Test Test Details for each Test, Unit Under Test

• The user can create any number of tests and units under

test that contribute ultimately to the system reliability

Modeling & Simulation, Testing & Validation

7/31/2018

View Estimated and Test – System Reliability

Compare latest Test estimate with Planned estimate

Modeling & Simulation, Testing & Validation

7/31/2018

Next Steps

• Incorporate uncertainty and risk reduction in

development testing (i.e., non-growth reliability testing)

– Incorporate in multi-objective optimization

• Develop consistent framework for merging test and

design reliability estimate data

– Designating test incidents that are considered failures – Computing and comparing uncertainty

• Incorporate simulation to characterize uncertainty of final

development MMBF

• Conduct sample case demonstrating tool

Modeling & Simulation, Testing & Validation

7/31/2018

Questions? Thank You!

Modeling & Simulation, Testing & Validation

7/31/2018

Army Lifecycle Test Optimization (ALTO) Tool Features

• Provide an innovative capability to conduct lifecycle test optimization

– Defendable and repeatable results – Balance cost, schedule, and risk for test planners – Provide recommended solutions for optimizing test given time and money constraints – React to changes to time, cost, and risk in any phase of acquisition

• Concept software functions for lifecycle test optimization

– Import US Army data sources (FMECA data, Test Incident Data) – Edit/Enter reliability data, factors and constraints – Compute measures from input data using mathematical models – Provide immediate output for quick look analysis using calculators – Perform optimization with algorithms built into the software – Integrate with reliability calculators – Display metrics (tables, charts, etc.) – Produce an optimal balance of system and subsystem DT level testing

• Implement a design that is flexible and adaptable

– Implements multiple modeling approaches and can an extend to multiple data sources – Exports/imports data to interface with other applications