[PPT] - T * est Tanja E.J. Vos So#ware Tes+ng and Quality Group (StaQ) PowerPoint Presentation

SLIDE 1 So#ware Tes+ng and Quality Group (StaQ) Research center for So#ware Produc+on Methods (PROS) Universidad Politecnica de Valencia Spain

est

T

*

Vakgroep Informa+ca Open Universiteit Netherlands

Tanja E.J. Vos

SLIDE 2

¿What is ? est

T

*

SLIDE 3

Why is usefull?

Automated tes+ng at the user interface level
UI is where everything comes together
From a user’s perspec+ve
Almost all so<ware has a user interface

est

T

*

SLIDE 4

¿What is not ?

It is not capture/replay
It is not visual tes+ng
It does not generate scripts

est

T

*

SLIDE 5

ABC

It inferres a model
Model = widget tree
Generate sequences of test

type: TButton ... rect: [15, 25, 65, 55] hasFocus: true enabled: false title: "Button" Window Button Text Slider Menu MI MI MI MI type: TMenuItem ... title: "File"

What is ?

est

T

*

SLIDE 6

How does work?

est

T

*

START SUT SCAN GUI + OBTAIN WIDGET TREE more actions? Domain Experts DERIVE SET OF USER ACTIONS EXECUTE ACTION No Yes Action Definitions Oracle Definition STOP SUT

ptional

instrumentation Replayable Erroneous Sequences ORACLE FAULT? Yes No more sequences? SELECT ACTION

SLIDE 7 !"#$%&' ()*%% +,&--*. +/0$/#"1* +//%,&. 2//%,&. 2//%31*0 2//%31*0 2//%,&. 2//%,&. 2//%31*0 2//%,&. 2//%,&. +//%31*0 +//%31*0 +/0$/#"1* +//%,&. +/0$/#"1* 2//%,&. 2//%31*0 2//%31*0 +//%31*0 (1&14#5"-* +5&6*% 2//%,&. 2//%31*0 +/0$/#"1* 7./8.*##3-9":&1/. 7./8.*##;*8"/-<= +&->&# +/0$/#"1* +/0$/#"1* +/0$/#"1* 2//%,&. 2//%31*0 2//%31*0 2//%31*0 2//%31*0 +/0$/#"1* +/0$/#"1* 7&8*,//? 2.** 2.**+/%40- 2.**+/%40- +/0$/#"1* +/0$/#"1* 7&8*,//? +/0$/#"1* 5&6*% +/0$/#"1* +2&6@/%9*. 2//%,&. 2//%31*0 A"*B@/.0 +2&6@/%9*. A"*B@/.0 +2&631*0 +2&6@/%9*. 2//%,&. 2//%31*0 A"*B@/.0 +2&631*0 (&#) (&#) 2."0+/00/-C3D&-9%* +//%,&. +/0$/#"1* +//%31*0 E*-4 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 @/.0 E&"-FE*-4 G9"#&6%*9H (*&.:)

Active Widget Tree

SLIDE 8 !"#$%&' ()*%% +,&--*. +/0$/#"1* +//%,&. 2//%,&. 2//%31*0 2//%31*0 2//%,&. 2//%,&. 2//%31*0 2//%,&. 2//%,&. +//%31*0 +//%31*0 +/0$/#"1* +//%,&. +/0$/#"1* 2//%,&. 2//%31*0 2//%31*0 +//%31*0 (1&14#5"-* +5&6*% 2//%,&. 2//%31*0 +/0$/#"1* 7./8.*##3-9":&1/. 7./8.*##;*8"/-<= +&->&# +/0$/#"1* +/0$/#"1* +/0$/#"1* 2//%,&. 2//%31*0 2//%31*0 2//%31*0 2//%31*0 +/0$/#"1* +/0$/#"1* 7&8*,//? 2.** 2.**+/%40- 2.**+/%40- +/0$/#"1* +/0$/#"1* 7&8*,//? +/0$/#"1* 5&6*% +/0$/#"1* +2&6@/%9*. 2//%,&. 2//%31*0 A"*B@/.0 +2&6@/%9*. A"*B@/.0 +2&631*0 +2&6@/%9*. 2//%,&. 2//%31*0 A"*B@/.0 +2&631*0 (&#) (&#) 2."0+/00/-C3D&-9%* +//%,&. +/0$/#"1* +//%31*0 E*-4 E*-431*0 E*-4 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 !./$!/B- E*-4

Active Widget Tree

SLIDE 9

How does work?

est

T

*

START SUT SCAN GUI + OBTAIN WIDGET TREE more actions? Domain Experts DERIVE SET OF USER ACTIONS EXECUTE ACTION No Yes Action Definitions Oracle Definition STOP SUT

ptional

instrumentation Replayable Erroneous Sequences ORACLE FAULT? Yes No more sequences? SELECT ACTION

SLIDE 10

works for all kinds of so#ware!

GUI state, Faults Runtime data Test Suites ABSTRACTION LAYER Test Execution Test Evaluation Test Execution Test Evaluation Test Execution Test Evaluation Test Execution Test Evaluation What do you need?? Test Suite Generator Abstract Representation (WIDGET TREE) Abstract Oracles Cheap Oracles Free Oracles

est

T

*

SLIDE 11

READY

est

T

*

PATH to SUT

SLIDE 12

SET

est

T

*

undesired ac+ons undesired processes

SLIDE 13 13

GO!

SLIDE 14

Oracles give verdicts about each state of the GUI

SLIDE 15

100% Automated oracles

Crashes
Hangs

Verdict oracle_Crash (State state){ if(!state.get(IsRunning,false)) return new Verdict("System crashed!"); } Verdict oracle_Responsiveness (State state){ if(state.get(NotResponding, true)) return new Verdict("System not responding!"); }

SLIDE 16 16

Oracles for suspicious +tles and outputs
Specify them with a regular expressión

SLIDE 17

.*NullPointerExcep+on

.*|[Ee]rror|[Pp]roblem

Oracle – Suspicious +tles

SLIDE 18

Oracle – Suspicious +tles (under the hood)

Verdicts oracle_SuspiciousTitles(State state){ verdicts = new Verdicts(): String regEx = settings().get(SuspiciousTitles); // search all widgets for suspicious titles for(Widget w : state){ String title = w.get(Title, ""); if(title.matches(regEx)){ verdicts.add(new Verdict(“suspicious title..”); } return verdicts; }

SLIDE 19

Add any oracle by addi+onal programming

est

T

*

Verdicts oracle_ImagesWAI(State state) { verdicts = new Verdicts(): for(Widget w : state){ Role role = w.get(Tags.Role); if (role.equals("UIAImage") && title.isEmpty()) verdicts.add(new Verdict("Not all images have an alternate textual description"); } return verdicts; }

SLIDE 20

How does change tes+ng?

est

T

*

Requirements

Test Suite

Execution

Test Suite Requirements

Execution

SLIDE 21

How was used?

Microso< office
Test the test tool TESTONA (eclise based)
ERP desktop applica+on
Over 10 web applica+ons of Spanish companies
Oracle Apex… (SOGETI)
InnovATOS…..

est

T

*

SLIDE 22 22

Wat weten we over kwaliteit van

het systeem na testen met TESTAR?

Nieuwe coverage criteria nodig!
Wat voor type fouten?

Classifica+e!

Her-injecteren van fouten?
Muta+e van Uis om tests te

evalueren

Volgende stappen: Kwaliteits criteria

SLIDE 23 23

Beyond random
Selecteer ac+es op zo een manier dat

we een gekozen doel op+maliseren

Maximaliseer aantal verschillende

ac+es

Maximaliseer aantal verschillende

event handlers dat word aangeroepen

MCS (desktop) Web analogy?
Ant colony
Q-learning
Novelty search

Volgende stappen: Action selection

SLIDE 24 24

Abstracte defini+es van test doelen
Bijv: generate test sequences that:
fill in all test fields before doing OK
exercises every widget minimum 1

+me

verifies whether all hyperlinks

have a valid target site

raises an error if the UI consumes

more than T MB of memory

etc…...

Volgende stappen: More abstraction

SLIDE 25 25

Oracles
Vorige versies
Common palerns?
DSL ipv java?
…
Visualisa+e
+jdens het testen
na het testen
Toepassen bij bedrijven!
Applica+on areas
Web
IoT

Volgende stappen: …................

SLIDE 26

Tot slot, nog wat andere dingen:

ICTopen
innovATOS

– hlps://events.nl.atos.net/#/LocalEvents/1

European So<ware Tes+ng Innova+on Alliance

– samen met VERSEN? – hlp://blog.uiin.org/2016/02/european-so<ware- tes+ng-innova+on-alliance-euros+a-kick-off- mee+ng-planned-in-amsterdam/

h2020 proposal (deadline 12 april)

est

T

*