Considering Execution Environment Resilience: A White-Box Approach - PowerPoint PPT Presentation

Considering Execution Environment Resilience: A White-Box Approach Stefan Klikovits 1 , 2 , David PY Lawrence 1 , 3 , Manuel Gonzalez-Berges 2 , Didier Buchs 1 1 Université de Genève, Carouge, Switzerland 2 CERN, Geneva, Switzerland 3 Honeywell International Sarl., Rolle, Switzerland Tuesday 11 th August, 2015

What is this all about? Considering Execution Environment Resilience: A White-Box Approach S.Klikovits, D. Lawrence, M. Gonzalez- Berges, D. Buchs How to ◮ generate test cases w. little user interaction ◮ on a large scale ◮ unit/component level 2 / 21

Welcome to CERN Considering Execution Environment Resilience: A White-Box Approach S.Klikovits, D. Lawrence, M. Gonzalez- Berges, D. Buchs Credit: CERN (www.cern.ch) 3 / 21

Welcome to CERN Considering Execution Environment Resilience: A White-Box Approach S.Klikovits, D. Lawrence, ◮ LHC, experiments, infrastructure (e.g. power grid) M. Gonzalez- Berges, ◮ large-scale, widespread, complex systems D. Buchs ◮ many types of hard- and software ◮ > 100 subsystems, 10 , 000s of devices, 100 , 000s of measurement points ◮ thousands of physicists/engineers/workers high reliability and resilience expectations 3 / 21

How do we supervise it? Considering Execution Environment Resilience: A White-Box Approach S.Klikovits, D. Lawrence, M. Gonzalez- Berges, D. Buchs ◮ two frameworks (UNICOS, JCOP) built on top ◮ Control (CTRL): proprietary scripting language 4 / 21

So where is the problem? Considering Execution Environment Resilience: A White-Box Approach S.Klikovits, D. Lawrence, M. Gonzalez- Berges, D. Buchs ◮ until recently no automated unit test support ◮ frequent changes in execution environment ◮ (mostly) manual verification ◮ big expenses (time) on QA side 5 / 21

Testing Considering Execution Environment Resilience: A White-Box Approach S.Klikovits, D. Lawrence, M. Gonzalez- Berges, D. Buchs 1 f ( x ){ 1 t e s t _ f ( ) { 2 i f GLOBAL_VAR: 2 dbSet ( " t e s t " ,5) // p r e p a r e 3 r e t u r n dbGet ( x ) 3 GLOBAL_VAR = True 4 e l s e : 4 x = f ( " t e s t " ) // a c t 5 r e t u r n − 1 5 a s s e r t ( x == 5) // a s s e r t 6 } 6 } Test case for f(x) f(x) 6 / 21

What do we want? Considering Execution Environment Resilience: A White-Box Approach S.Klikovits, D. Lawrence, M. Gonzalez- Berges, D. Buchs test cases code Iterative TEst Case System ◮ regression testing ◮ consider dependencies ◮ automatic test case generation (ATCG) ◮ build on existing research & tools ◮ generate unit & component tests 7 / 21

Automated Test Case Generation Considering Execution Environment Resilience: A White-Box Approach S.Klikovits, D. Lawrence, M. Gonzalez- Berges, D. Buchs ◮ source code based ◮ black-box (function signature) vs. white-box (function body) 8 / 21

Semi-purification Considering Execution Environment Resilience: A White-Box Approach ◮ replace dependencies with parameters S.Klikovits, D. Lawrence, M. Gonzalez- Berges, D. Buchs 1 f ( x ){ 1 f_sp ( x , a ,b){ 2 i f GLOBAL_VAR : 2 i f a : 3 r e t u r n dpGet(x) 3 r e t u r n b 4 e l s e : 4 e l s e : 5 − 1 5 − 1 r e t u r n r e t u r n 6 } 6 } A non-pure function Semi-purified f(x) a a a a 9 / 21

Semi-purification Considering Execution Environment Resilience: A White-Box Approach ◮ replace dependencies with parameters S.Klikovits, D. Lawrence, M. Gonzalez- Berges, D. Buchs 1 f ( x ){ 1 f_sp ( x , a ,b){ 2 i f GLOBAL_VAR : 2 i f a : 3 r e t u r n dpGet(x) 3 r e t u r n b 4 e l s e : 4 e l s e : 5 r e t u r n − 1 5 r e t u r n − 1 6 } 6 } A non-pure function Semi-purified f(x) 1 test_f_sp ( ) { 2 x = f ( " t e s t " , True , 5 ) // a c t 3 a s s e r t ( x == 5) // a s s e r t 4 } Test case 9 / 21

Semi-purification (cont.) Considering Execution Environment Resilience: A White-Box Approach S.Klikovits, D. Lawrence, ◮ replace dependencies with parameters M. Gonzalez- Berges, D. Buchs 1 f u n c t i o n A ( x ){ 1 functionA_sp ( x , y){ 2 a = f u n c t i o n B ( x ) 2 a = f u n c t i o n B ( x , y) 3 r e t u r n a 3 r e t u r n a 4 } 4 } 5 5 6 f u n c t i o n B ( x ){ 6 functionB_sp ( x , y){ 7 b = GLOBAL_VAR 7 b = y 8 b++ 8 b++ 9 r e t u r n b 9 r e t u r n b 10 } 10 } Function with SRC Semi-purified w. SRC 10 / 21

Semi-purification (cont.) Considering Execution Environment Resilience: A White-Box Approach S.Klikovits, D. Lawrence, ◮ replace dependencies with parameters M. Gonzalez- Berges, D. Buchs 1 f u n c t i o n A ( x ){ 1 functionA_sp ( x , y){ 2 a = f u n c t i o n B ( x ) 2 a = f u n c t i o n B ( x , y) 3 r e t u r n a 3 r e t u r n a 4 } 4 } 5 5 6 f u n c t i o n B ( x ){ 6 functionB_sp ( x , y){ 7 b = GLOBAL_VAR 7 b = y 8 b++ 8 b++ 9 r e t u r n b 9 r e t u r n b 10 } 10 } Function with SRC Semi-purified w. SRC 10 / 21

Semi-purification: Concept Considering Execution Environment Resilience: A White-Box Approach S.Klikovits, D. Lawrence, ◮ code contains dependencies M. Gonzalez- Berges, ◮ global variables, data base values, subroutine calls, D. Buchs other resources ◮ manual way: test doubles (mocks, stubs, fakes, . . . ) [ME06] ◮ remove dependencies ◮ based on localization [SW03, SK13] ◮ input parameters instead of dependencies ◮ use any ATCG (black- and white-box) 11 / 21

“Shortcut” Considering Execution Environment Resilience: A White-Box Approach SP S.Klikovits, f ( P ) { D } f ( P ∪ P D ) {} D. Lawrence, M. Gonzalez- Berges, D. Buchs ATCG SP − 1 � TI , TS � � TI ∪ TI D � Figure: Test case generation schema ◮ ◮ P . . . Parameters TI . . . Test Input ◮ ◮ D . . . Dependencies TS . . . Test Setup Routine 12 / 21

Identified Bottlenecks Considering Execution Environment Resilience: A White-Box Approach S.Klikovits, D. Lawrence, M. Gonzalez- Berges, D. Buchs ◮ Loops ◮ Shared Subroutines ◮ Concurrency 13 / 21

SP: Bottlenecks (cont.) Considering Execution Environment Resilience: ◮ Loops A White-Box Approach S.Klikovits, 1 s l e e p U n t i l R e a d y ( ){ // a = bool D. Lawrence, 2 M. Gonzalez- 3 w h i l e dpGet(notReadyDP) : Berges, 4 s l e e p ( 5 ) // s l e e p f o r 5 seconds D. Buchs 5 6 } A semi-purified loop 14 / 21

SP: Bottlenecks (cont.) Considering Execution Environment ◮ Loops Resilience: A White-Box Approach 1 s l e e p U n t i l R e a d y (a){ // a = bool S.Klikovits, 2 D. Lawrence, 3 w h i l e a : // replaces dpGet(notReadyDP) M. Gonzalez- 4 s l e e p ( 5 ) // s l e e p f o r 5 seconds Berges, 5 D. Buchs 6 } A semi-purified loop Test Cases: ◮ a: False ⇒ loop not executed ◮ a: True ⇒ endless loop 14 / 21

SP: Bottlenecks (cont.) Considering Execution Environment Resilience: ◮ Loops A White-Box Approach S.Klikovits, 1 s l e e p U n t i l R e a d y (a){ // a = [bool] D. Lawrence, 2 i = 0 M. Gonzalez- 3 w h i l e a[i] : // replaces dpGet(notReadyDP) Berges, 4 s l e e p ( 5 ) // s l e e p f o r 5 seconds D. Buchs 5 i++ 6 } A semi-purified loop Test Cases: ◮ a: [False] ⇒ loop not executed ◮ a: [True, True, . . . , False] ⇒ loop execution Questions: ◮ how long should the list be? ◮ how to modify correctly? ◮ Test modified code or w. threads? 14 / 21

SP: Bottlenecks (cont.) Considering Execution Environment Resilience: A White-Box Approach ◮ Shared subroutine dependencies S.Klikovits, D. Lawrence, M. Gonzalez- 1 var SPEED_VAR = 1 Berges, 2 a d j u s t S p e e d ( ){ D. Buchs 3 x = getTheSpeed ( ) 4 i f x < 10 : 5 doubleTheSpeed ( ) 6 } CUT 1 getTheSpeed ( ){ 1 doubleTheSpeed ( ){ 2 r e t u r n SPEED_VAR 2 speed = SPEED_VAR 3 } 3 SPEED_VAR Subroutine 1 Subroutine 2 15 / 21

SP: Bottlenecks (cont.) Considering Execution Environment Resilience: A White-Box Approach ◮ Shared subroutine dependencies S.Klikovits, D. Lawrence, M. Gonzalez- 1 var SPEED_VAR = 1 Berges, 2 a d j u s t S p e e d ( ){ D. Buchs 3 x = getTheSpeed ( ) 4 i f x < 10 : 5 doubleTheSpeed ( ) 6 } CUT 1 getTheSpeed ( ){ 1 doubleTheSpeed ( ){ 2 r e t u r n SPEED_VAR 2 speed = SPEED_VAR 3 } 3 SPEED_VAR Subroutine 1 Subroutine 2 15 / 21

SP: Bottlenecks (cont.) Considering Execution Environment Resilience: A White-Box Approach ◮ Shared subroutine dependencies S.Klikovits, D. Lawrence, M. Gonzalez- 1 Berges, 2 a d j u s t S p e e d (a){ D. Buchs 3 x = getTheSpeed (a) 4 i f x < 10 : 5 doubleTheSpeed ( ) 6 } CUT 1 getTheSpeed (a){ 1 doubleTheSpeed ( ){ 2 r e t u r n a // SPEED_VAR 2 speed = SPEED_VAR 3 } 3 SPEED_VAR Subroutine 1 Subroutine 2 15 / 21

SP: Bottlenecks (cont.) Considering Execution Environment Resilience: A White-Box Approach ◮ Shared subroutine dependencies S.Klikovits, D. Lawrence, M. Gonzalez- 1 Berges, 2 a d j u s t S p e e d (a, b){ D. Buchs 3 x = getTheSpeed (a) 4 i f x < 10 : 5 doubleTheSpeed (b) 6 } CUT 1 getTheSpeed (a){ 1 doubleTheSpeed (b){ 2 r e t u r n a // SPEED_VAR 2 speed = b // SPEED_VAR 3 } 3 b = speed*2 Subroutine 1 Subroutine 2 15 / 21