T * est Tanja E.J. Vos So#ware Tes+ng and Quality Group (StaQ) - PowerPoint PPT Presentation

T * est Tanja E.J. Vos So#ware Tes+ng and Quality Group (StaQ) Vakgroep Informa+ca Research center for So#ware Produc+on Methods (PROS) Open Universiteit Universidad Politecnica de Valencia Netherlands Spain

T * est ¿What is ?

T * Why is usefull? est • Automated tes+ng at the user interface level • UI is where everything comes together • From a user’s perspec+ve • Almost all so<ware has a user interface

T * est ¿What is not ? • It is not capture/replay • It is not visual tes+ng • It does not generate scripts

T * est What is ? Window ABC Text Button Menu Slider MI MI MI MI type: TButton type: TMenuItem title: " Button " • It inferres a model title: " File " enabled: false ... hasFocus: true • Model = widget tree rect: [15, 25, 65, 55] ... • Generate sequences of test

T * Domain Experts est How does work? Action Oracle Definitions Definition SCAN GUI + DERIVE SET START OBTAIN OF USER SUT WIDGET TREE ACTIONS optional instrumentation SELECT ACTION more sequences? Yes No more EXECUTE STOP actions? ACTION SUT No Yes ORACLE FAULT? Replayable Erroneous Sequences

Active Widget Tree 2//%31*0 G9"#&6%*9H 2//%,&. 2//%31*0 2//%,&. (*&.:) 2//%,&. 2//%31*0 2//%,&. +//%,&. 2//%,&. +/0$/#"1* +//%31*0 +//%31*0 2//%31*0 +/0$/#"1* 2//%,&. +,&--*. +/0$/#"1* +//%,&. 2//%31*0 +//%31*0 +5&6*% 2//%31*0 2//%,&. 2//%31*0 2//%31*0 (1&14#5"-* +/0$/#"1* 7./8.*##3-9":&1/. 2//%31*0 2//%,&. 2//%31*0 2.**+/%40- +/0$/#"1* +/0$/#"1* 7&8*,//? 2.** 2.**+/%40- @/.0 +&->&# +/0$/#"1* +/0$/#"1* 7&8*,//? +/0$/#"1* 5&6*% 7./8.*##;*8"/-<= +/0$/#"1* !"#$%&' ()*%% +/0$/#"1* +2&6@/%9*. 2//%,&. 2//%31*0 +/0$/#"1* +/0$/#"1* +2&6@/%9*. A"*B@/.0 A"*B@/.0 +2&6@/%9*. +2&631*0 2."0+/00/-C3D&-9%* (&#) 2//%,&. 2//%31*0 (&#) A"*B@/.0 +//%,&. +/0$/#"1* +2&631*0 +//%31*0 E*-431*0 E*-4 E*-431*0 E&"-FE*-4 E*-431*0 E*-431*0 E*-431*0

Active Widget Tree 2//%31*0 2//%,&. 2//%31*0 2//%,&. 2//%,&. 2//%31*0 2//%,&. +//%,&. 2//%,&. +/0$/#"1* +//%31*0 +//%31*0 2//%31*0 +/0$/#"1* 2//%,&. +,&--*. +/0$/#"1* +//%,&. 2//%31*0 +//%31*0 +5&6*% 2//%31*0 2//%,&. 2//%31*0 2//%31*0 (1&14#5"-* +/0$/#"1* 7./8.*##3-9":&1/. 2//%31*0 2//%,&. 2//%31*0 2.**+/%40- +/0$/#"1* +/0$/#"1* 7&8*,//? 2.** 2.**+/%40- +&->&# +/0$/#"1* +/0$/#"1* 7&8*,//? +/0$/#"1* 5&6*% 7./8.*##;*8"/-<= +/0$/#"1* !"#$%&' ()*%% +/0$/#"1* +2&6@/%9*. 2//%,&. 2//%31*0 +/0$/#"1* +/0$/#"1* +2&6@/%9*. A"*B@/.0 A"*B@/.0 +2&6@/%9*. +2&631*0 2."0+/00/-C3D&-9%* (&#) 2//%,&. 2//%31*0 (&#) A"*B@/.0 +//%,&. +/0$/#"1* +2&631*0 +//%31*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-4 E*-431*0 E*-431*0 E*-431*0 E*-431*0 !./$!/B- E*-431*0 E*-4 E*-431*0 E*-431*0 E*-4 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0 E*-431*0

T * Domain Experts est How does work? Action Oracle Definitions Definition SCAN GUI + DERIVE SET START OBTAIN OF USER SUT WIDGET TREE ACTIONS optional instrumentation SELECT ACTION more sequences? Yes No more EXECUTE STOP actions? ACTION SUT No Yes ORACLE FAULT? Replayable Erroneous Sequences

ABSTRACTION LAYER T * est works for all kinds of Free Abstract Representation Oracles (WIDGET TREE) so#ware! Cheap Oracles Test Suite Generator Abstract Oracles GUI state, Faults Test Suites Runtime data Evaluation Execution Test Test Execution Evaluation Evaluation Evaluation Execution Execution Test Test Test Test Test Test What do you need??

T * READY est PATH to SUT

T * SET est undesired ac+ons undesired processes

GO! 13

Oracles give verdicts about each state of the GUI

100% Automated oracles Verdict oracle_Crash (State state){ if (!state.get(IsRunning,false)) return new Verdict("System crashed!"); } Verdict oracle_Responsiveness (State state){ if (state.get(NotResponding, true)) return new Verdict("System not responding!"); } • Crashes • Hangs

• Oracles for suspicious +tles and outputs • Specify them with a regular expressión 16

Oracle – Suspicious +tles . *NullPointerExcep+on .*|[Ee]rror|[Pp]roblem

Verdicts oracle_SuspiciousTitles(State state){ verdicts = new Verdicts(): String regEx = settings().get(SuspiciousTitles); // search all widgets for suspicious titles for(Widget w : state){ String title = w.get(Title, ""); if(title.matches(regEx)){ verdicts.add(new Verdict(“suspicious title..”); } return verdicts; } Oracle – Suspicious +tles (under the hood)

Verdicts oracle_ImagesWAI(State state) { verdicts = new Verdicts(): for(Widget w : state){ Role role = w.get(Tags.Role); if (role.equals("UIAImage") && title.isEmpty()) verdicts.add(new Verdict("Not all images have an alternate textual description"); } return verdicts; } Add any oracle by addi+onal programming T * est

T * How does change tes+ng? est Execution Test Requirements Suite Test Requirements Execution Suite

T * est How was used? • Microso< office • Test the test tool TESTONA (eclise based) • ERP desktop applica+on • Over 10 web applica+ons of Spanish companies • Oracle Apex… (SOGETI) • InnovATOS…..

Volgende stappen: Kwaliteits criteria • Wat weten we over kwaliteit van het systeem na testen met TESTAR? • Nieuwe coverage criteria nodig! • Wat voor type fouten? Classifica+e! Her-injecteren van fouten? • • Muta+e van Uis om tests te evalueren 22

Volgende stappen: Action selection • Beyond random • Selecteer ac+es op zo een manier dat we een gekozen doel op+maliseren • Maximaliseer aantal verschillende ac+es • Maximaliseer aantal verschillende event handlers dat word aangeroepen • MCS (desktop) Web analogy? • Ant colony • Q-learning • Novelty search 23

Volgende stappen: More abstraction • Abstracte defini+es van test doelen • Bijv: generate test sequences that: • fill in all test fields before doing OK • exercises every widget minimum 1 +me • verifies whether all hyperlinks have a valid target site • raises an error if the UI consumes more than T MB of memory • etc…... 24

Volgende stappen: … ................ • Oracles • Vorige versies • Common palerns? • DSL ipv java? • … • Visualisa+e • +jdens het testen • na het testen • Toepassen bij bedrijven! • Applica+on areas • Web • IoT 25

T * est Tot slot, nog wat andere dingen: • ICTopen • innovATOS – hlps://events.nl.atos.net/#/LocalEvents/1 • European So<ware Tes+ng Innova+on Alliance – samen met VERSEN? – hlp://blog.uiin.org/2016/02/european-so<ware- tes+ng-innova+on-alliance-euros+a-kick-off- mee+ng-planned-in-amsterdam/ • h2020 proposal (deadline 12 april)