RoboStar Technology Systematic Software Testing for Robotics Robert - - PowerPoint PPT Presentation

▶
robostar technology systematic software testing for
SMART_READER_LITE
LIVE PREVIEW

RoboStar Technology Systematic Software Testing for Robotics Robert - - PowerPoint PPT Presentation

Jan 22, 2024 94 likes •462 views

RoboStar Technology Systematic Software Testing for Robotics Robert M. Hierons 1 and Raluca Lefticaru 2 University of Sheffield, University of Bradford, UK 14th November 2019 Thanks : Ana Cavalcanti, James Baxter, Manuel N u nez, Mercedes

slide-1
SLIDE 1

RoboStar Technology Systematic Software Testing for Robotics

Robert M. Hierons1 and Raluca Lefticaru2

University of Sheffield, University of Bradford, UK

14th November 2019

Thanks: Ana Cavalcanti, James Baxter, Manuel N´ u˜ nez, Mercedes Merayo

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar1 Technology – Software Engineering for Robotics 1 / 25

slide-2
SLIDE 2

Overview

◮ Motivation ◮ The Approach ◮ Mutation with Wodel ◮ Counterexample generation ◮ Test Generation

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar2 Technology – Software Engineering for Robotics 2 / 25

slide-3
SLIDE 3

Motivation

Testing ◮ Aim: generating tests cases from RoboChart models ◮ Why test?

◮ Reality gap ◮ Validates the behaviour of the model ◮ Explore cases developer may not have considered

◮ Mutation testing — finds problems arising from variations ◮ RoboChart testing can be extended to RoboSim testing

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar3 Technology – Software Engineering for Robotics 3 / 25

slide-4
SLIDE 4

Example RoboChart Model

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar4 Technology – Software Engineering for Robotics 4 / 25

slide-5
SLIDE 5

Motivation

Formal Semantics

◮ Models: state-based, cyclic nature, discrete time ◮ Semantics: provided by mapping models to a discrete variant of timed CSP – tock CSP

◮ CSP models have states ◮ there are transitions between states ◮ transitions have labels (inputs, outputs, tock - progression of time)

◮ Aim: automate testing on the basis of this formal semantics.

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar5 Technology – Software Engineering for Robotics 5 / 25

slide-6
SLIDE 6

Motivation

Implementation relations

◮ Standard minimal test hypothesis: the system under test (SUT) behaves like an unknown model that can be described using the same formalism. ◮ Define correctness of a system under test (SUT) via implementation relations between models.

◮ Timed trace inclusion ◮ Timed refusal trace inclusion

◮ This enables us to have automated testing that is sound.

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar6 Technology – Software Engineering for Robotics 6 / 25

slide-7
SLIDE 7

Overview of Approach

Mutation Testing

◮ Use a mutation testing approach (fault seeding). ◮ Overall structure:

◮ Seed faults into the specification. ◮ Each seeded fault defines a mutant. ◮ Given a mutant M′ of specification M, find a test case that detects (kills) M′.

◮ We run the resultant test cases on the system under test

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar7 Technology – Software Engineering for Robotics 7 / 25

slide-8
SLIDE 8

Overview of Approach

A mutant of the RoboChart Model

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar8 Technology – Software Engineering for Robotics 8 / 25

slide-9
SLIDE 9

Overview of Approach

Mutation Testing

◮ Our mutants are first order mutants: generated by making one change ◮ If the SUT passes tests that kill the mutants then the SUT cannot be one

  • f these mutants

◮ We mutate the model, not the CSP generated from the model ◮ Should more appropriately capture likely faults

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar9 Technology – Software Engineering for Robotics 9 / 25

slide-10
SLIDE 10

Overview of Approach

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar10 Technology – Software Engineering for Robotics 10 / 25

slide-11
SLIDE 11

Overview of Approach

10https://www.cs.york.ac.uk/robostar/notations-tools/ Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar10 Technology – Software Engineering for Robotics 10 / 25

slide-12
SLIDE 12

Overview of Approach

10https://www.cs.york.ac.uk/robostar/notations-tools/ Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar10 Technology – Software Engineering for Robotics 10 / 25

slide-13
SLIDE 13

Overview of Approach

10https://www.cs.york.ac.uk/robostar/notations-tools/ Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar10 Technology – Software Engineering for Robotics 10 / 25

slide-14
SLIDE 14

Overview of Approach

10https://www.cs.york.ac.uk/robostar/notations-tools/ Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar10 Technology – Software Engineering for Robotics 10 / 25

slide-15
SLIDE 15

Overview of Approach

10https://www.cs.york.ac.uk/robostar/notations-tools/ Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar10 Technology – Software Engineering for Robotics 10 / 25

slide-16
SLIDE 16

Overview of Approach

10https://www.cs.york.ac.uk/robostar/notations-tools/ Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar10 Technology – Software Engineering for Robotics 10 / 25

slide-17
SLIDE 17

Overview of Approach

10https://www.cs.york.ac.uk/robostar/notations-tools/ Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar10 Technology – Software Engineering for Robotics 10 / 25

slide-18
SLIDE 18

Overview of Approach

10https://www.cs.york.ac.uk/robostar/notations-tools/ Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar10 Technology – Software Engineering for Robotics 10 / 25

slide-19
SLIDE 19

Overview of Approach

10https://www.cs.york.ac.uk/robostar/notations-tools/ Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar10 Technology – Software Engineering for Robotics 10 / 25

slide-20
SLIDE 20

Overview of Approach

10https://www.cs.york.ac.uk/robostar/notations-tools/ Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar11 Technology – Software Engineering for Robotics 10 / 25

slide-21
SLIDE 21

Wodel

◮ Wodel12 is a tool for generating mutants from provided models using a set

  • f mutation operators.

◮ Wodel is provided as an Eclipse plugin that interacts with the Eclipse Modelling Framework (EMF). ◮ Models are instances of a metamodel provided to Wodel. ◮ The mutation operators are described by Wodel’s DSL.

12http://gomezabajo.github.io/Wodel/ Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar13 Technology – Software Engineering for Robotics 11 / 25

slide-22
SLIDE 22

Example RoboChart Model

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar14 Technology – Software Engineering for Robotics 12 / 25

slide-23
SLIDE 23

RoboChart XMI Models

◮ Wodel acts on models in XMI format ◮ We thus export RoboChart models from RoboTool to XMI ◮ Handle multi-file models by merging into one file

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar15 Technology – Software Engineering for Robotics 13 / 25

slide-24
SLIDE 24

Mutation Operators

◮ Used to create the mutants. ◮ Currently, they:

◮ Modify transitions between states; or ◮ Delete states

◮ Wodel applies the mutation operators at random according to their specification.

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar16 Technology – Software Engineering for Robotics 14 / 25

slide-25
SLIDE 25

Mutation Operators – Examples

mStActEnDu { retype

  • ne

EntryAction as DuringAction // m o d i f i e s a s t a t e by changing an e n t r y a c t i o n i n t o a during a c t i o n } mTransSource { t r = s e l e c t

  • ne

T r a n s i t i o n where { ˆsource < > one I n i t i a l } modify t a r g e t ˆsource from t r to

  • ther

State // changes the s t a r t s t a t e

  • f

a t r a n s i t i o n , except the

  • ne

from the i n i t i a l j u n c t i o n }

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar17 Technology – Software Engineering for Robotics 15 / 25

slide-26
SLIDE 26

Mutant Example – mStActEnDu operator

Delivering state has its entry action changed into a during action

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar18 Technology – Software Engineering for Robotics 16 / 25

slide-27
SLIDE 27

Mutant Example – mTransSource operator

The transition source is modified from Delivering to Looking state

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar19 Technology – Software Engineering for Robotics 17 / 25

slide-28
SLIDE 28

Mutant Example – rSeqStatement operator

Two actions are removed from the sequence in the transition from the Delivering state

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar20 Technology – Software Engineering for Robotics 18 / 25

slide-29
SLIDE 29

Generating CSP for Mutants

◮ Wodel outputs mutants in XMI format. ◮ They are imported into RoboTool. ◮ RoboTool generates CSP for imported mutants.

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar21 Technology – Software Engineering for Robotics 19 / 25

slide-30
SLIDE 30

Comparing Mutants with FDR

◮ Check if mutants are correct implementations of original model ◮ Obtain counterexample traces from those that don’t ◮ Eclipse plugin to handle mutant comparison

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar22 Technology – Software Engineering for Robotics 20 / 25

slide-31
SLIDE 31

Counterexample Traces

Rescue switchOn . i n − > R e s c u e t a k e o f f . out − > tock − > moveCall − > moveRet − > Rescue found . i n − > Rescue land . out − > R e s c u e o r i g i n . i n Rescue switchOn . i n − > R e s c u e t a k e o f f . out − > tock − > tock Rescue switchOn . i n − > R e s c u e t a k e o f f . out − > tock − > moveCall − > moveRet − > Rescue found . i n − > Rescue land . out − > moveCall Rescue switchOn . i n − > R e s c u e t a k e o f f . out − > tock − > moveCall − > moveRet − > Rescue found . i n − > Rescue land . out − > R e s c u e t a k e o f f . out Rescue switchOn . i n − > R e s c u e t a k e o f f . out − > tock − > moveCall − > moveRet − > R e s c u e o r i g i n . i n

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar23 Technology – Software Engineering for Robotics 21 / 25

slide-32
SLIDE 32

Considerations about the Environment

◮ Some generated traces are infeasible — they don’t represent cases that could occur. ◮ e.g. finding the target immediately after starting movement ◮ Such traces aren’t useful, since they don’t relate to the robot’s environment. ◮ To eliminate these traces, we need to model the environment. Software Model Mapping Robot Model Locations Model Environment

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar24 Technology – Software Engineering for Robotics 22 / 25

slide-33
SLIDE 33

Test Generation

◮ From each counterexample trace, we generate a test T. ◮ T performs events of the system-under-test, SUT, and reports verdict using events pass, fail and inc (inconclusive). ◮ The test corresponding to a trace a1, . . . , an is: T = inc → a1 → · · · → inc → an−1 → pass → an → fail → Stop ◮ Test execution is represented by a parallel composition: (SUT |[ αSUT ]| T) \ αSUT ◮ The last event observed determines the result of the test.

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar25 Technology – Software Engineering for Robotics 23 / 25

slide-34
SLIDE 34

Future Work

◮ So far we have just considered traces - we can also test for refusals at the end of a trace (or within a trace). ◮ Our tests are sequences - we could combine these to form trees ◮ Probabilities and continuous variables

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar26 Technology – Software Engineering for Robotics 24 / 25

slide-35
SLIDE 35

Conclusion

◮ Generate tests from RoboChart by mutation using Wodel ◮ Comparison of mutants and original using FDR ◮ Counterexample traces generate sound tests ◮ Future work to allow testing for refusals

Robert M. Hierons and Raluca Lefticaru University of Sheffield, University of Bradford, UK RoboStar27 Technology – Software Engineering for Robotics 25 / 25