I W C T 2 0 1 7 C O M B I N AT O R I A L T E S T I N G ( C T ) - - PowerPoint PPT Presentation

▶

Dec 31, 2023 8 likes •189 views

U S I N G T I M E D B A S E - C H O I C E C O V E R A G E C R I T E R I O N F O R T E S T I N G I N D U S T R I A L C O N T R O L S O F T WA R E H E N N I N G B E R G S T R M A N D E D U A R D E N O I U I W C T 2 0 1 7 C O M B I N

SLIDE 1

U S I N G T I M E D B A S E - C H O I C E C O V E R A G E C R I T E R I O N F O R T E S T I N G I N D U S T R I A L C O N T R O L S O F T WA R E

H E N N I N G B E R G S T R Ö M A N D E D U A R D E N O I U

I W C T 2 0 1 7

SLIDE 2

C O M B I N AT O R I A L T E S T I N G ( C T )

CT tries to select test input values:
the test goal is a combination strategy - test criteria.
Test Level:
System: create inputs on user-level interaction
Unit: create inputs for method param. and variables

SLIDE 3

C O M B I N AT O R I A L T E S T I N G ( C T )

CT tries to select test input values:
the test goal is a combination strategy - test criteria.
Test Level:
System: create inputs on user-level interaction
Unit: create inputs for method param. and variables

T E S T C R I T E R I A A P P L I E D T O U N I T S

SLIDE 4

P R O G R A M M A B L E L O G I C C O N T R O L L E R S ( P L C )

Are real-time systems
Found in trains, nuclear power plants, automation
Run on domain-specific operating systems.

SLIDE 5

I N D U S T R I A L C O N T R O L S O F T WA R E ( U N I T ) P L C W R I T T E N I N I E C 6 1 1 3 1 - 3

T O N L E G E A N D O R I N 1 I N 3 5 s O U T I N 2 I N 4

5 1 2 5

SLIDE 6

T E S T I N G P L C S O F T WA R E

E N 5 0 1 2 8 - R A I LWAY I E C 6 2 3 0 4 - M E D I C A L E N 6 2 1 3 8 - N U C L E A R

D O - 1 7 8 B - A E R O S PA C E

SLIDE 7

T E S T I N G P L C S O F T WA R E

E N 5 0 1 2 8 - R A I LWAY

Unit (component) level testing
A design is used as expected output (test oracle)
The use of functional testing is mandated
Some level of code coverage is recommended

time component Design Creation Test Suite Creation t0 t1 Test Suite Execution Test Result Checking Test Suite Reporting t2 t3 t4 t5

SLIDE 8

B A S E C H O I C E C R I T E R I O N

AND 5s TON GT SR

IN2 IN3 IN4 OUT1 OUT2 IN1

# Tests IN1 IN2 IN3 IN4 1 1 1 5 4 2 1 5 4 3 1 5 4 4 1 1 5 3 5 1 1 5 5 6 1 1 4 4 7 1 1 3 4

program cycle P=500ms

SLIDE 9

T I M E D B A S E - C H O I C E C R I T E R I O N

1. Create the basic input model
2. Identify the timing constraint
3. Identify a base and time choice test.
4. Create a test suite.
time choice T = 6s
(1,1,5,4) is fixed for 6s.
Tests 1 to 7 are fixed

for 6s each  

AND 5s TON GT SR

IN2 IN3 IN4 OUT1 OUT2 IN1

# Tests IN1 IN2 IN3 IN4 1 1 1 5 4 2 1 5 4 3 1 5 4 4 1 1 5 3 5 1 1 5 5 6 1 1 4 4 7 1 1 3 4

SLIDE 10

C A S E S T U D Y

Compare timed base-choice with base choice in terms
f code coverage and fault detection.

I N D U S T R I A L C A S E S T U D Y

SLIDE 11

C A S E S T U D Y

M E T H O D

B C

T E S T S B R A N C H C O V E R A G E FA U LT D E T E C T I O N

T B C

T E S T S

R A N D

1 1 P R O G R A M S T E S T S

SLIDE 12

M U TAT I O N A N A LY S I S

C O M PA R I S O N VA L U E N E G AT I O N L O G I C A L A R I T H M E T I C T I M E R

MUTANT OPERATORS

(Yoo et al 2007) (Shin et al. 2012)

MUTANTS MUTANTS MUTANTS IEC 61131-3 PROGRAM

SLIDE 13

M U TAT I O N A N A LY S I S

O R I G I N A L P R O G R A M

M U TA N T P R O G R A M M U TA N T P R O G R A M

M U TA N T P R O G R A M O R I G I N A L P R O G R A M

M U TA N T P R O G R A M M U TA N T P R O G R A M

M U TA N T P R O G R A M

F I N D T E S T S W H I C H W H E N E X E C U T E D O N B O T H O R I G I N A L & M U TA N T P R O G R A M

O U T P U T / S TAT E I S D I F F E R E N T

SLIDE 14

R E S U LT S

T B C A C H I E V E S B E T T E R FA U LT D E T E C T I O N S C O R E S T H A N B C O R R A N D .

Rand

BC TBC 20 40 60 80 100 Mutation Score (%)

SLIDE 15

R E S U LT S

T B C A C H I E V E S B E T T E R D E C I S I O N C O V E R A G E S C O R E S T H A N B C O R R A N D .

Rand BC TBC 50 60 70 80 90 100 Decision Coverage (%)

SLIDE 16

R E S U LT S

Rand

BC TBC 10 20 30 40 Number of tests

SLIDE 17

T H E F U T U R E

multiple base and time choices
evaluate the use of stronger criteria
Use naturally-occurring faults
Use other systems from other domains

SLIDE 18

U S I N G T I M E D B A S E - C H O I C E C O V E R A G E C R I T E R I O N F O R T E S T I N G I N D U S T R I A L C O N T R O L S O F T WA R E

T B C

B C

FA U LT D E T E C T I O N

T B C B C

C O D E C O V E R A G E

IEC 61131-3 Programs Base-Choice (BC) Random (Rand) Faulty Programs Tests Execution Framework Timed Base-Choice (TBC) Fault Detection Coverage

Rand

BC TBC 20 40 60 80 100 Mutation Score (%)