Test-Case Generation for Runtime Analysis and Vice-Versa: - - PowerPoint PPT Presentation

▶
test case generation for runtime analysis and vice versa
SMART_READER_LITE
LIVE PREVIEW

Test-Case Generation for Runtime Analysis and Vice-Versa: - - PowerPoint PPT Presentation

Aug 31, 2022 152 likes •615 views

Test-Case Generation for Runtime Analysis and Vice-Versa: Verification of Aircraft Separation Assurance Dimitra Giannakopoulou Marko Dimjaevi University of Utah NASA Ames Research Center ISSTA 2015, Baltimore, Maryland July 17, 2015 1 /

slide-1
SLIDE 1

Test-Case Generation for Runtime Analysis and Vice-Versa: Verification of Aircraft Separation Assurance

Marko Dimjašević University of Utah Dimitra Giannakopoulou NASA Ames Research Center ISSTA 2015, Baltimore, Maryland July 17, 2015

1 / 21

slide-2
SLIDE 2

Goals

◮ Propose verification properties for aircraft separation

assurance software

◮ Verify properties at runtime

2 / 21

slide-3
SLIDE 3

AutoResolver

◮ Part of US federal government’s NextGen project ◮ Developed at NASA Ames Research Center ◮ Software system for aircraft separation assurance ◮ 65K lines of Java code ◮ Its environment’s core: 450K lines of code

3 / 21

slide-4
SLIDE 4

Conflict, Loss of Separation, Separation Assurance

4 / 21

slide-5
SLIDE 5

Monitored Requirements

Verification Properties

5 / 21

slide-6
SLIDE 6

Monitored Requirements

Verification Properties

P1 There should be a resolution for every conflict.

5 / 21

slide-7
SLIDE 7

Monitored Requirements

Verification Properties

P1 There should be a resolution for every conflict. P2 Initial conflicts are resolved in the non-decreasing order of their first time to loss of separation.

5 / 21

slide-8
SLIDE 8

Monitored Requirements

Verification Properties

P1 There should be a resolution for every conflict. P2 Initial conflicts are resolved in the non-decreasing order of their first time to loss of separation. P3 New conflicts arising as a result of conflict resolution should be inserted into the list of conflicts according to their first loss of separation time.

5 / 21

slide-9
SLIDE 9

Monitored Requirements

Verification Properties

P1 There should be a resolution for every conflict. P2 Initial conflicts are resolved in the non-decreasing order of their first time to loss of separation. P3 New conflicts arising as a result of conflict resolution should be inserted into the list of conflicts according to their first loss of separation time. P4 No picked resolution is allowed to cause a more imminent secondary conflict.

5 / 21

slide-10
SLIDE 10

Monitored Requirements

Verification Properties

P1 There should be a resolution for every conflict. P2 Initial conflicts are resolved in the non-decreasing order of their first time to loss of separation. P3 New conflicts arising as a result of conflict resolution should be inserted into the list of conflicts according to their first loss of separation time. P4 No picked resolution is allowed to cause a more imminent secondary conflict.

Resolution Monitor

M1 For each conflict, report its resolution type and how it changes over time.

5 / 21

slide-11
SLIDE 11

Wrapper

Motivation

6 / 21

slide-12
SLIDE 12

Wrapper

Motivation

◮ Environment stubbing ◮ Light-weight testing with different kinds of input than

trajectories

◮ E.g. airspeed, initial heading, climb rate, heading

change, trajectory time, destination coordinates

6 / 21

slide-13
SLIDE 13

Wrapper

Motivation

◮ Environment stubbing ◮ Light-weight testing with different kinds of input than

trajectories

◮ E.g. airspeed, initial heading, climb rate, heading

change, trajectory time, destination coordinates

◮ Test-case generation: control conflict creation process

6 / 21

slide-14
SLIDE 14

Wrapper

Motivation

◮ Environment stubbing ◮ Light-weight testing with different kinds of input than

trajectories

◮ E.g. airspeed, initial heading, climb rate, heading

change, trajectory time, destination coordinates

◮ Test-case generation: control conflict creation process

Purpose

◮ Test-case generation ◮ Property verification at runtime

6 / 21

slide-15
SLIDE 15

Wrapper — Aspect-Oriented Programming

◮ Avoid usual way: instrumentation for verification ◮ Leave AutoResolver’s source code intact

7 / 21

slide-16
SLIDE 16

Wrapper — Aspect-Oriented Programming

◮ Avoid usual way: instrumentation for verification ◮ Leave AutoResolver’s source code intact

AspectJ

◮ Java language extension ◮ Bytecode weaving (instrumentation)

7 / 21

slide-17
SLIDE 17

Wrapper — Aspect-Oriented Programming

◮ Avoid usual way: instrumentation for verification ◮ Leave AutoResolver’s source code intact

AspectJ

◮ Java language extension ◮ Bytecode weaving (instrumentation)

In-house verification

◮ No external verification tool used (SMT solvers, MOP

tools)

7 / 21

slide-18
SLIDE 18

Wrapper — Properties

Properties as AspectJ aspects

◮ 1 property = 1 aspect ◮ 1 aspect = multiple pointcuts and advices

8 / 21

slide-19
SLIDE 19

Wrapper — Properties

Properties as AspectJ aspects

◮ 1 property = 1 aspect ◮ 1 aspect = multiple pointcuts and advices

Pointcuts

◮ Where are interesting points of execution in

AutoResolver?

8 / 21

slide-20
SLIDE 20

Wrapper — Properties

Properties as AspectJ aspects

◮ 1 property = 1 aspect ◮ 1 aspect = multiple pointcuts and advices

Pointcuts

◮ Where are interesting points of execution in

AutoResolver?

◮ Points in wrapper itself

8 / 21

slide-21
SLIDE 21

Wrapper — Properties

Properties as AspectJ aspects

◮ 1 property = 1 aspect ◮ 1 aspect = multiple pointcuts and advices

Pointcuts

◮ Where are interesting points of execution in

AutoResolver?

◮ Points in wrapper itself

Advices

◮ Actions to be taken at pointcuts

8 / 21

slide-22
SLIDE 22

AspectJ Example

pointcut callAR(AacTestWrapper wrapper): call(public ArrayList conflictDetectResolve()) && target(wrapper) && !cflow(myAspect()) && !cflow(callFlyForMethod(*, *)) && if(isEnabled); after(AacTestWrapper wrapper): callAR(wrapper) { for (t = 60.0; t <= 480.0; t += 60.0) { AacTestWrapper w = wrapper.flyFor(t); w.conflictDetectResolve(); } }

9 / 21

slide-23
SLIDE 23

Runtime Verification

◮ Verification at

runtime

10 / 21

slide-24
SLIDE 24

Runtime Verification

◮ Verification at

runtime

◮ Need for good

runtime drivers

◮ Test cases 10 / 21

slide-25
SLIDE 25

Runtime Verification

◮ Verification at

runtime

◮ Need for good

runtime drivers

◮ Test cases

“Testing shows the presence, not the absence of bugs.” — Dijkstra

10 / 21

slide-26
SLIDE 26

Test-Case Generation

◮ Arbitrary many conflicts

11 / 21

slide-27
SLIDE 27

Test-Case Generation

◮ Arbitrary many conflicts ◮ Secondary conflicts — challenging to create

11 / 21

slide-28
SLIDE 28

Test-Case Generation

◮ Arbitrary many conflicts ◮ Secondary conflicts — challenging to create ◮ Time dimension added at runtime

11 / 21

slide-29
SLIDE 29

Generating Secondary Conflicts

◮ Secondary conflicts: created along a resolution trajectory

12 / 21

slide-30
SLIDE 30

Generating Secondary Conflicts

◮ Secondary conflicts: created along a resolution trajectory

Extend black-box test cases through reflection and with runtime verification

12 / 21

slide-31
SLIDE 31

Generating Secondary Conflicts

◮ Secondary conflicts: created along a resolution trajectory

Extend black-box test cases through reflection and with runtime verification

T2 T1 T ′

1

T2 T2 T ′

1

T3

12 / 21

slide-32
SLIDE 32

Test Case — Example

public void test0() throws Throwable { AacTestWrapper wrapper = new AacTestWrapper(); wrapper.setUpCR(CR_parameters1); wrapper.setUpCL(CL_parameters2); wrapper.setUpCR(CR_parameters3); wrapper.conflictDetectResolve(); }

13 / 21

slide-33
SLIDE 33

Evaluation

◮ Test suite of 3.5 million test cases

◮ Each test case with about 5 conflicts

◮ Every test case executed at 9 different time points

◮ Fly all aircraft for some time and then call AutoResolver ◮ Effectively: 3.5 million · 9 = 31.5 million test cases

◮ Check if every requirement is exercised

◮ Second-level monitors 14 / 21

slide-34
SLIDE 34

Results — Property P1

There should be a resolution for every conflict.

15 / 21

slide-35
SLIDE 35

Results — Property P1

There should be a resolution for every conflict.

◮ It does not hold, but this is not a bug ◮ AutoResolver does not resolve conflicts that:

◮ involve aircraft already in violation ◮ happen earlier than a predetermined time limit (1 minute) ◮ happen later than a predetermined time limit (8 minutes) ◮ “Neither plane able to maneuver/neither plane able to be

unfrozen” (current resolution round)

15 / 21

slide-36
SLIDE 36

Results — Property P2

Initial conflicts are resolved in the non-decreasing

  • rder of their first time to loss of separation.

16 / 21

slide-37
SLIDE 37

Results — Property P2

Initial conflicts are resolved in the non-decreasing

  • rder of their first time to loss of separation.

◮ No violation found

16 / 21

slide-38
SLIDE 38

Results — Property P3

New conflicts arising as a result of conflict resolution should be inserted into the list of conflicts according to their first loss of separation time.

17 / 21

slide-39
SLIDE 39

Results — Property P3

New conflicts arising as a result of conflict resolution should be inserted into the list of conflicts according to their first loss of separation time.

◮ No violation found ◮ No test case that exercises respective parts of code

◮ Second-level monitor

◮ Need support for weather conflict type

17 / 21

slide-40
SLIDE 40

Results — Property P4

No picked resolution is allowed to cause a more imminent secondary conflict.

18 / 21

slide-41
SLIDE 41

Results — Property P4

No picked resolution is allowed to cause a more imminent secondary conflict.

◮ No violation found

◮ Several test cases used to indicate violation (bug found

in wrapper)

18 / 21

slide-42
SLIDE 42

Results — Resolution Monitor M1

For each conflict, report its resolution type and how it changes over time.

19 / 21

slide-43
SLIDE 43

Results — Resolution Monitor M1

For each conflict, report its resolution type and how it changes over time.

ttlos [s] Delay time [s] Res type 430.0 0.0 26 370.0 60.0 26 310.0 120.0 26 250.0 180.0 26 190.0 240.0 26 130.0 300.0 13 70.0 360.0 13 10.0 420.0 not resolved 0.0 480.0 not resolved

19 / 21

slide-44
SLIDE 44

Results — Resolution Monitor M1 — Continued

No-conflict window?

ttlos [s] Delay time [s] Res type 445.0 0.0 3 — 60.0 — — 120.0 — 265.0 180.0 3 205.0 240.0 3 145.0 300.0 3 85.0 360.0 3 25.0 420.0 not resolved 0.0 480.0 not resolved

20 / 21

slide-45
SLIDE 45

Summary

◮ Light-weight verification of aircraft separation assurance

software

◮ Runtime verification for test-case generation ◮ Test-case generation for runtime verification

21 / 21

slide-46
SLIDE 46

Summary

◮ Light-weight verification of aircraft separation assurance

software

◮ Runtime verification for test-case generation ◮ Test-case generation for runtime verification

Credits

◮ Crossroads — Umberto Nicoletti ◮ This presentation — Marko Dimjašević, CC-BY-SA 4.0

21 / 21