Complementary Criteria for Testing Temporal Logic Properties - PowerPoint PPT Presentation

Complementary Criteria for Testing Temporal Logic Properties Gordon Fraser and Franz Wotawa Graz University of Technology, Austria

Model-Based Development Process Informal ideas Specification Design Code Realization

Model-Based Development Process Informal ideas Informal world World of models Specification Design Code Real world Realization

Model-Based Development Process Informal ideas Informal world World of models Specification Validation Design Code Real world Realization

Model-Based Development Process Informal ideas Informal world World of models Specification Validation Design Formal verification Code Real world Realization

Model-Based Development Process Informal ideas Informal world World of models Specification Model−based testing Validation Design Formal verification Code Real world Realization

Specification Based Testing Informal ideas Informal world World of models Specification Specification based testing Validation Design Formal verification Code Real world Realization

Specification Based Testing Informal ideas Informal world World of models Requirement Executable Specification Properties Formal verification Design Code Real world Realization

Specification Based Testing Informal ideas Informal world World of models Specification Requirement Executable Specification Properties based testing Formal verification Design Code Real world Realization

✎ ✎ Test cases derived from executable spec ✎ When is a property covered? ✎ How is it covered? ✎ How to generate tests for properties? ✎ Which tests to generate?

✎ Test cases derived from executable spec ✎ When is a property covered? ✎ How is it covered? ✎ How to generate tests for properties? ✎ Which tests to generate? ✎ Contribution: 2 new criteria to measure and generate tests

Test cases derived from executable spec ✎ When is a property covered? ✎ How is it covered? ✎ How to generate tests for properties? ✎ Which tests to generate? ✎ Contribution: 2 new criteria to measure and generate tests ✎ Assumption: Properties specified in temporal logic

Model Checking Model Property ? satisfies G (x −> F y)

Model Checking Model Property satisfies G (x −> F y)

Model Checking Model Counterexample Property violates G (x −> F y)

Specification Based Testing Informal ideas Informal world World of models Specification Executable Requirement Properties Specification based testing Formal verification Design Code Real world Realization

Specification Based Testing

Specification Based Testing Model Checker LTL Properties Counterexamples

Test Case Generation with Model Checkers Test suite Test + coverage info Test Test Suite predicates generator Coverage Counter Criteria Trap example property never(tp) Cex = SPECIFICATION sequence of states that cover Model tp checker = test

Example Specification: NuSMV ▼❖❉❯▲❊ ♠❛✐♥ ❱❆❘ ❛❝❝❡❧❡r❛t❡✿ ❜♦♦❧❡❛♥❀ ❜r❛❦❡✿ ❜♦♦❧❡❛♥❀ ✈❡❧♦❝✐t②✿ ④ st♦♣✱ s❧♦✇✱ ❢❛st ⑥❀ ❆❙❙■●◆ ✐♥✐t✭✈❡❧♦❝✐t②✮ ✿❂ st♦♣❀ ♥❡①t✭✈❡❧♦❝✐t②✮ ✿❂ ❝❛s❡ ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀ ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ s❧♦✇ ✿ ❢❛st❀ ✦❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ ❢❛st ✿ s❧♦✇❀ ✦❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ s❧♦✇ ✿ st♦♣❀ ❜r❛❦❡✿ st♦♣❀ ❚❘❯❊ ✿ ✈❡❧♦❝✐t②❀ ❡s❛❝❀

Example Specification: NuSMV

Temporal Logics

Temporal Logics Example Property G ✿ ✭ velocity ❂ fast ✮

Temporal Logics Example Property X velocity ❂ slow

Temporal Logics Example Property F ✿ accelerate

Temporal Logics Example Property accelerate U brake

Example: NuSMV ▼❖❉❯▲❊ ♠❛✐♥ ❱❆❘ ❛❝❝❡❧❡r❛t❡✿ ❜♦♦❧❡❛♥❀ ❜r❛❦❡✿ ❜♦♦❧❡❛♥❀ ✈❡❧♦❝✐t②✿ ④ st♦♣✱ s❧♦✇✱ ❢❛st ⑥❀ ❆❙❙■●◆ ✐♥✐t✭✈❡❧♦❝✐t②✮ ✿❂ st♦♣❀ ♥❡①t✭✈❡❧♦❝✐t②✮ ✿❂ ❝❛s❡ ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀ ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ s❧♦✇ ✿ ❢❛st❀ ✦❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ ❢❛st ✿ s❧♦✇❀ ✦❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ s❧♦✇ ✿ st♦♣❀ ❜r❛❦❡✿ st♦♣❀ ❚❘❯❊ ✿ ✈❡❧♦❝✐t②❀ ❡s❛❝❀

Example: NuSMV ▼❖❉❯▲❊ ♠❛✐♥ ❱❆❘ ❛❝❝❡❧❡r❛t❡✿ ❜♦♦❧❡❛♥❀ ❜r❛❦❡✿ ❜♦♦❧❡❛♥❀ ✈❡❧♦❝✐t②✿ ④ st♦♣✱ s❧♦✇✱ ❢❛st ⑥❀ ❆❙❙■●◆ ✐♥✐t✭✈❡❧♦❝✐t②✮ ✿❂ st♦♣❀ ♥❡①t✭✈❡❧♦❝✐t②✮ ✿❂ ❝❛s❡ ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀ ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ s❧♦✇ ✿ ❢❛st❀ ✦❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ ❢❛st ✿ s❧♦✇❀ ✦❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ s❧♦✇ ✿ st♦♣❀ ❜r❛❦❡✿ st♦♣❀ ❚❘❯❊ ✿ ✈❡❧♦❝✐t②❀ ❡s❛❝❀

✎ ✭ ❫ ✿ ❫ ❂ ✦ ✿ ✭ ❂ ✮✮ ✎ ✭ ✿ ✭ ❫ ✿ ❫ ❂ ✮ ✦ ✭ ❂ ✮✮ Coverage Criteria ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀ Predicate Coverage

✎ ✭ ✿ ✭ ❫ ✿ ❫ ❂ ✮ ✦ ✭ ❂ ✮✮ Coverage Criteria ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀ Predicate Coverage ✎ G ✭ accelerate ❫ ✿ brake ❫ velocity ❂ stop ✦ X ✿ ✭ velocity ❂ slow ✮✮

Coverage Criteria ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀ Predicate Coverage ✎ G ✭ accelerate ❫ ✿ brake ❫ velocity ❂ stop ✦ X ✿ ✭ velocity ❂ slow ✮✮ ✎ G ✭ ✿ ✭ accelerate ❫ ✿ brake ❫ velocity ❂ stop ✮ ✦ X ✭ velocity ❂ slow ✮✮

Coverage Criteria ❛❝❝❡❧❡r❛t❡ ✫ ✦❜r❛❦❡ ✫ ✈❡❧♦❝✐t② ❂ st♦♣ ✿ s❧♦✇❀ Predicate Coverage ✎ G ✭ accelerate ❫ ✿ brake ❫ velocity ❂ stop ✦ X ✿ ✭ velocity ❂ slow ✮✮ ✎ G ✭ ✿ ✭ accelerate ❫ ✿ brake ❫ velocity ❂ stop ✮ ✦ X ✭ velocity ❂ slow ✮✮ ✎ 15 specification based test criteria ✎ 2 property based criteria 1 Based on vacuity 2 Based on MCDC

✎ Unique First Cause Coverage ✎ MCDC: Test cases where clauses affect predicates

Unique First Cause Coverage ✎ MCDC: Test cases where clauses affect predicates ✎ UFC [Whalen et al., 2006]: Clause c is the unique first cause of a formula A , if in the first state along a path where A is satisfied, it is satisfied because of c

✭ ❴ ✮ Unique First Cause Coverage

Unique First Cause Coverage F ✭ a ❴ b ✮

Unique First Cause Coverage ✎ Defined as set of rules ✎ Apply rules to property ✦ set of test predicates ✎ Rules applied to all properties ✎ Test predicates: Measure coverage and generate tests

✎ ✎ ✎ ❫ ✎ ✎ ✿ ❫ ✿ ✎ ✎ ✦ Unique First Cause Coverage is not enough G ✭✭ c 1 ❫ c 2 ✮ ✩ X d ✮

✎ ✿ ❫ ✿ ✎ ✎ ✦ Unique First Cause Coverage is not enough G ✭✭ c 1 ❫ c 2 ✮ ✩ X d ✮ ✎ UFC covers: ✎ Where c 1 causes d to be false. ✎ Where c 2 causes d to be false. ✎ Where c 1 ❫ c 2 causes d to be true.

✿ ❫ ✿ ✎ ✎ ✦ Unique First Cause Coverage is not enough G ✭✭ c 1 ❫ c 2 ✮ ✩ X d ✮ ✎ UFC covers: ✎ Where c 1 causes d to be false. ✎ Where c 2 causes d to be false. ✎ Where c 1 ❫ c 2 causes d to be true. ✎ What if d represents a safety critical value?

✎ ✦ Unique First Cause Coverage is not enough G ✭✭ c 1 ❫ c 2 ✮ ✩ X d ✮ ✎ UFC covers: ✎ Where c 1 causes d to be false. ✎ Where c 2 causes d to be false. ✎ Where c 1 ❫ c 2 causes d to be true. ✎ What if d represents a safety critical value? ✎ Is ✿ c 1 ❫ ✿ c 2 covered?

Unique First Cause Coverage is not enough G ✭✭ c 1 ❫ c 2 ✮ ✩ X d ✮ ✎ UFC covers: ✎ Where c 1 causes d to be false. ✎ Where c 2 causes d to be false. ✎ Where c 1 ❫ c 2 causes d to be true. ✎ What if d represents a safety critical value? ✎ Is ✿ c 1 ❫ ✿ c 2 covered? ✎ ✦ Property Inactive Clause Coverage