An industrial case study of TACO Benjamin Lesage , Stephen Law, Iain - - PowerPoint PPT Presentation

An industrial case study of TACO

Icons courtesy of https://icons8.com/

Benjamin Lesage, Stephen Law, Iain Bate

Context

TACO – RTNS 2018

• Rolls-Royce VISIUMCORE platform



Integrated instruction tracing and timing



Instructions’ execution is time-invariant



Limited time-relevant state

• WCET Timing analysis
• Builds WCET from low-level block timings
• Requires full coverage of executed code
• Full Authority Digital Engine Controller
• Designed to DO-178C DAL A guidelines
• Coverage through low-level tests
• Automatic test case generation technique [1]
• Provide better coverage and timing data

2 [1] Stephen Law and Iain Bate, Achieving Appropriate Test Coverage for Reliable Measurement-Based Timing Analysis, ECRTS, 2016

Context

TACO – RTNS 2018

• Rolls-Royce VISIUMCORE platform



Integrated instruction tracing and timing



Instructions’ execution is time-invariant



Limited time-relevant state

• WCET Timing analysis
• Builds WCET from low-level block timings
• Requires full coverage of executed code
• Full Authority Digital Engine Controller
• Designed to DO-178C DAL A guidelines
• Coverage through low-level tests
• Automatic test case generation technique [1]
• Provide better coverage and timing data

3 [1] Stephen Law and Iain Bate, Achieving Appropriate Test Coverage for Reliable Measurement-Based Timing Analysis, ECRTS, 2016

Formal testing late in life cycle On-Target testing is expensive

Coverage Technique [1] - Example

TACO – RTNS 2018 4

function F (B: Boolean, C: Byte)

A > 0.75 B

• [1] Stephen Law and Iain Bate, Achieving Appropriate Test Coverage for Reliable Measurement-Based Timing Analysis, ECRTS, 2016

Coverage Technique [1] - Example

TACO – RTNS 2018 5

function F (B: Boolean, C: Byte)

A > 0.75 B

{A: 0.5, B: True, C: 7} Generate a test vector

[1] Stephen Law and Iain Bate, Achieving Appropriate Test Coverage for Reliable Measurement-Based Timing Analysis, ECRTS, 2016

Coverage Technique [1] - Example

TACO – RTNS 2018 6

function F (B: Boolean, C: Byte)

A > 0.75 B

{A: 0.5, B: True, C: 7} Execute the function

[1] Stephen Law and Iain Bate, Achieving Appropriate Test Coverage for Reliable Measurement-Based Timing Analysis, ECRTS, 2016

Coverage Technique [1] - Example

TACO – RTNS 2018 7

function F (B: Boolean, C: Byte)

A > 0.75 B

{A: 0.5, B: True, C: 7} Collect coverage metrics

[1] Stephen Law and Iain Bate, Achieving Appropriate Test Coverage for Reliable Measurement-Based Timing Analysis, ECRTS, 2016

Coverage Technique [1] - Example

TACO – RTNS 2018 8

function F (B: Boolean, C: Byte)

A > 0.75 B

{A: 0.5, B: True, C: 7} {A: 0.95, B: True, C: 7} Mutate the test vector

[1] Stephen Law and Iain Bate, Achieving Appropriate Test Coverage for Reliable Measurement-Based Timing Analysis, ECRTS, 2016

Coverage Technique [1] - Example

TACO – RTNS 2018 9

function F (B: Boolean, C: Byte)

A > 0.75 B

{A: 0.5, B: True, C: 7} {A: 0.95, B: True, C: 7}

[1] Stephen Law and Iain Bate, Achieving Appropriate Test Coverage for Reliable Measurement-Based Timing Analysis, ECRTS, 2016

Coverage Technique [1] - Example

TACO – RTNS 2018 10

function F (B: Boolean, C: Byte)

A > 0.75 B

{A: 0.5, B: True, C: 7} {A: 0.95, B: True, C: 7} {A: 0.95, B: True, C: 6} Reject poor solutions

[1] Stephen Law and Iain Bate, Achieving Appropriate Test Coverage for Reliable Measurement-Based Timing Analysis, ECRTS, 2016

Coverage Technique [1] - Example

TACO – RTNS 2018 11

function F (B: Boolean, C: Byte)

A > 0.75 B

{A: 0.5, B: True, C: 7} {A: 0.95, B: True, C: 7} {A: 0.95, B: True, C: 6} {A: 0.95, B: True, C: 7}

[1] Stephen Law and Iain Bate, Achieving Appropriate Test Coverage for Reliable Measurement-Based Timing Analysis, ECRTS, 2016

Coverage Technique [1] - Example

TACO – RTNS 2018 12

function F (B: Boolean, C: Byte)

A > 0.75 B

{A: 0.5, B: True, C: 7} {A: 0.95, B: True, C: 7} {A: 0.95, B: True, C: 6} {A: 0.95, B: True, C: 7} {A: 0.95, B: False, C: 7}

[1] Stephen Law and Iain Bate, Achieving Appropriate Test Coverage for Reliable Measurement-Based Timing Analysis, ECRTS, 2016

Coverage Technique [1] - Example

TACO – RTNS 2018 13

function F (B: Boolean, C: Byte)

A > 0.75 B

 Search-based timing analysis tool

 Support measurement-based WCET analysis  Drive the execution of a tested function  Generate a sequence of test vectors

 Solutions evaluated on coverage metrics

 Executed blocks of code, loops branches  Different heuristics target different objectives

Requires knowledge about functions’ inputs, types and value ranges

TACO - Overview

TACO – RTNS 2018 14

Driver Executable Testport Instrumented code Source code System Model Profit The system model captures all software interfaces and requirements The system model captures all software interfaces and requirements

TACO - Overview

TACO – RTNS 2018 15

Driver Executable Testport Instrumented code Source code System Model Profit The source code is generated from the model The source code is generated from the model

TACO - Overview

TACO – RTNS 2018 16

Driver Executable Testport Instrumented code Source code System Model Profit Instrumentation inserts primitives to capture timing and coverage Instrumentation inserts primitives to capture timing and coverage

TACO - Overview

TACO – RTNS 2018 17

Driver Executable Testport Instrumented code Source code System Model Profit The Driver [1] produces test vectors to be executed The Driver [1] produces test vectors to be executed

TACO - Overview

TACO – RTNS 2018 18

Driver Executable Testport Instrumented code Source code System Model Profit A testport is also generated from the system model A testport is also generated from the system model

TACO – Testport

TACO – RTNS 2018 19

Testport Driver Source code

• The testport is the interface between the test function and driver
• Initialises inputs, runs and measures the function
• Provides feedback to the driver
• A common interface means item and driver can be swapped

TACO – Testport

TACO – RTNS 2018 20

Testport Driver Source code

• The testport is the interface between the test function and driver
• Initialises inputs, runs and measures the function
• Provides feedback to the driver
• A common interface means item and driver can be swapped

TACO - Overview

TACO – RTNS 2018 21

Driver Executable Testport Instrumented code Source code System Model Profit A testport is also generated from the system model A testport is also generated from the system model

TACO - Overview

TACO – RTNS 2018 22

Driver Executable Testport Instrumented code Source code System Model Profit Testport, driver and tested function are combined Testport, driver and tested function are combined

TACO - Overview

TACO – RTNS 2018 23

Driver Executable Testport Instrumented code Source code System Model Profit Generated traces can be processed for analysis Generated traces can be processed for analysis

TACO - Overview

TACO – RTNS 2018 24

Driver Executable Testport Instrumented code Source code System Model Profit

Reducing on-target testing

TACO – RTNS 2018 25

Driver Executable Testport Instrumented code Source code System Model

Reducing on-target testing

TACO – RTNS 2018 26

• The coverage technique [1] relies only on coverage information
• Coverage is measured at the source level
• Coverage is platform-independent
• Host-based testing can reduce the requirements on target
• Collect coverage and inputs on host
• Replay selection of tests on target

Host

Reducing on-target testing

TACO – RTNS 2018 27

• The coverage technique [1] relies only on coverage information
• Coverage is measured at the source level
• Coverage is platform-independent
• Host-based testing can reduce the requirements on target
• Collect coverage and inputs on host
• Replay selection of tests on target

Host

Reducing on-target testing

TACO – RTNS 2018 28

• The coverage technique [1] relies only on coverage information
• Coverage is measured at the source level
• Coverage is platform-independent
• Host-based testing can reduce the requirements on target
• Collect coverage and inputs on host
• Replay selection of tests on target

VISIUMCORE

Evaluation - Objectives

TACO – RTNS 2018

• TACO scalability
• Can TACO be automatically applied to a full control system?
• Driver portability:
• Can [1] be applied on different platforms?
• Driver scalability:
• Can [1] be applied to a full control system?
• What coverage can be achieved?
• Reducing on-target testing:
• Can TACO reduce requirements on target?

29

Evaluation - Configuration

TACO – RTNS 2018

• Unmodified control system
• Designed according to DO-178C as DAL-A
• 1800 items for analysis, including 250+ scheduled tasks
• Two evaluation platforms:
• Target: VISIUMCORE
• Host: i686
• Two heuristics for driver:
• Ran: Random search through input space
• BCHLr: Search focused on unexplored branches and loops
• 100 runs of TACO per item, heuristic and platform

30

Evaluation - Scalability of TACO

TACO – RTNS 2018 31

• Count analysed items if:
• TACO generates a testport
• Compiles with drivers
• Runs and generate traces
• Higher is better
• Ore items processed y framework
• Higher likelihood for coverage

Evaluation - Scalability of TACO

TACO – RTNS 2018 32

• More than 90% items analysed
• Contextual information missing from model
• Less items analysed on VISIUMCORE
• Driver is platform independent
• but lower resources on target
• Less items analysed using BCHLr
• Higher-entry memory requirements

Evaluation - Portability of Driver

TACO – RTNS 2018 33

• Compare coverage
• Across platforms
• For a same item
• For a same heuristic
• Closer to y=x is better
• Same performance across platforms
• Coverage collectable on Host

Evaluation - Portability of Driver

TACO – RTNS 2018 34

• Compare absolute coverage difference
• Across platforms
• For a same item
• For a same heuristic
• Lower is better
• Same performance across platforms
• Coverage collectable on Host
• Small differences across platforms
• Less than 15 functions > 1% difference
• Variations due to PRNG

Evaluation - Coverage

TACO – RTNS 2018 35

Evaluation - Coverage

TACO – RTNS 2018 36

Achieved coverage level Platform and driver

Evaluation - Coverage

TACO – RTNS 2018 37

better

Evaluation - System-wide coverage

TACO – RTNS 2018 38

root() funA() funB() funC() funD() funE() funB() root() funA() funB() funC() funD() funE() funB()

Evaluation - System-wide coverage

TACO – RTNS 2018 39

root() funA() funB() funC() funD() funE() funB()

Evaluation - System-wide coverage

TACO – RTNS 2018 40

• More functions supported on i686-BCHLr
• Less items compiled with VISIUMCORE-BCHLr

Evaluation - Reducing on-target testing

TACO – RTNS 2018 41

• Same coverage achieved on Host
• 22 Millions test vectors, 48 Hours
• 6131 test vectors, 658 items, 4 Hours on target

Conclusion

TACO – RTNS 2018

• Test Automation for COverage
• Exercise all inputs of a test item
• Drive the code through different paths
• Drive search for specific targets
• Scales to full system analysis
• Achieves reliable test coverage
• Reduces on-target testing
• Offers a collection of interacting tools
• Easy to parse, reuse, and extend

42

Future work

TACO – RTNS 2018 43

• Provide timing estimates early in the application life cycle
• Rely on low-cost platforms
• Correlate timings on Target and Host
• Quantify confidence and certainty in achieved coverage
• Define stopping criteria for search algorithm
• Understand returns of additional computational effort
• Refine test vector selection from host results
• Target longest execution paths
• Assess changes to the software