Combining Model Checking and Testing in a Continuous HW/SW - - PowerPoint PPT Presentation

▶
combining model checking and testing in a continuous hw
SMART_READER_LITE
LIVE PREVIEW

Combining Model Checking and Testing in a Continuous HW/SW - - PowerPoint PPT Presentation

Feb 08, 2023 502 likes •753 views

Combining Model Checking and Testing in a Continuous HW/SW Co-Verification Process Paula Herber, Florian Friedemann, and Sabine Glesner Berlin Institute of Technology Software Engineering for Embedded Systems Group TAP - Tests and Proofs

slide-1
SLIDE 1

Combining Model Checking and Testing in a Continuous HW/SW Co-Verification Process

Paula Herber, Florian Friedemann, and Sabine Glesner

Berlin Institute of Technology Software Engineering for Embedded Systems Group

TAP - Tests and Proofs Zurich, July 2009

slide-2
SLIDE 2

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Motivation

HW/SW Co-Design

  • modeling and simulation with system level design languages
  • stepwise refinement from abstract design to implementation
  • SystemC
  • designs are executable on different abstraction levels
  • validation and verification by co-simulation

Problems

  • impossible to cover all possible input scenarios (incomplete)
  • consistency between abstraction levels hard to ensure
  • limited degree of automatization (manual evaluation)

TAP 2009 - Paula Herber 2/18

slide-3
SLIDE 3

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Motivation

HW/SW Co-Design

  • modeling and simulation with system level design languages
  • stepwise refinement from abstract design to implementation
  • SystemC
  • designs are executable on different abstraction levels
  • validation and verification by co-simulation

Problems

  • impossible to cover all possible input scenarios (incomplete)
  • consistency between abstraction levels hard to ensure
  • limited degree of automatization (manual evaluation)

How can we assure quality in a more systematic way?

TAP 2009 - Paula Herber 2/18

slide-4
SLIDE 4

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Continuous HW/SW Co-Verification Approach

1 verify requirements on abstract design via model checking 2 generate conformance tests for each refined design not satisfied satisfied

Design

yes no

Refined Design Abstract

relation conformance

Conformance Testing Checking Model

conformance evaluation S

Test

I

Specification TS Requirements Specification R

TAP 2009 - Paula Herber 3/18

slide-5
SLIDE 5

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Continuous HW/SW Co-Verification Approach

1 verify requirements on abstract design via model checking 2 generate conformance tests for each refined design

not satisfied satisfied

Design

yes no

Refined Design Abstract

relation conformance

Conformance Testing Checking Model

conformance evaluation S

Test

I

Specification TS Requirements Specification R

  • but: semantics of SystemC is only informally defined

➠ map SystemC to Uppaal timed automata [CODES+ISSS 2008] ➠ use the ❯♣♣❛❛❧ model to generate conformance tests!

TAP 2009 - Paula Herber 3/18

slide-6
SLIDE 6

SystemC and Uppaal Model Checking Test Generation Results Conclusions

1 SystemC and Uppaal 2 Model Checking of SystemC Designs 3 Conformance Test Generation 4 Experimental Results 5 Conclusions

TAP 2009 - Paula Herber 4/18

slide-7
SLIDE 7

SystemC and Uppaal Model Checking Test Generation Results Conclusions

SystemC

  • introduced by the Open SystemC Initiative (OSCI) 1999
  • semantics is (informally) defined in IEEE Std. 1666-2005

SystemC as system level design language

  • description of both hardware and software on different levels
  • f abstraction
  • extends C++ by concurrency, time, hardware data types,

reactivity, hierarchy, and abstract communication SystemC as framework for HW/SW co-simulation

  • light-weight simulation kernel executes SystemC designs in a

discrete-event simulation

TAP 2009 - Paula Herber 5/18

slide-8
SLIDE 8

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Uppaal

  • modeling, simulation, and verification of timed automata
  • jointly developed by the Universities of Uppsala and Aalborg

Timed automata

  • finite-state machines extended by clock variables
  • clock constraints model time-dependent behavior
  • Networks of timed automata model concurrent processes

Extensions in Uppaal

  • parameterized timed automata templates
  • data variables with bounded domains
  • binary and broadcast channels
  • urgent and committed locations

x <= maxtime ack! value = f(t) x >= mintime request? x = 0 TAP 2009 - Paula Herber 6/18

slide-9
SLIDE 9

SystemC and Uppaal Model Checking Test Generation Results Conclusions

1 SystemC and Uppaal 2 Model Checking of SystemC Designs 3 Conformance Test Generation 4 Experimental Results 5 Conclusions

TAP 2009 - Paula Herber 7/18

slide-10
SLIDE 10

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Model Checking of SystemC Designs

[Herber, Fellmuth, Glesner, CODES+ISSS 2008]

1 transform SystemC designs into Uppaal timed automata 2 use the Uppaal model checker to prove safety, liveness, and

timing properties

Uppaal Model Abstract SystemC Design (STATE) Tool Transformation Checker Model Uppaal Requirements Specification (temporal properties) not satisfied satisfied TAP 2009 - Paula Herber 8/18

slide-11
SLIDE 11

SystemC and Uppaal Model Checking Test Generation Results Conclusions

HW/SW Co-Verification Framework

Uppaal Model Test Benches SystemC Abstract SystemC Design Refined SystemC Design

  • r Implementation

(STATE) Tool Transformation Executor Test Bench Checker Model Uppaal Requirements Specification (temporal properties) Coverage Criteria not satisfied satisfied manual refinement yes/no/inconclusive Conformance Test Generation TAP 2009 - Paula Herber 9/18

slide-12
SLIDE 12

SystemC and Uppaal Model Checking Test Generation Results Conclusions

1 SystemC and Uppaal 2 Model Checking of SystemC Designs 3 Conformance Test Generation 4 Experimental Results 5 Conclusions

TAP 2009 - Paula Herber 10/18

slide-13
SLIDE 13

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Test Model

  • embedded systems interact closely with their environment
  • simulation requires generation of inputs and consumption of
  • utputs, this is provided by
  • a test bench in SystemC
  • an explicit environment or test model in Uppaal

SystemC test bench

1 input generator:

provides an input trace

2 output monitor:

accepts all possible outputs translation to Uppaal → test automaton → generic tester    test model

TAP 2009 - Paula Herber 11/18

slide-14
SLIDE 14

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Conformance Test Generation

Goal:

  • compute all possible outputs of a given Uppaal model

Approach:

  • execute test model

together with abstract system model

  • record traces in the

generic tester

  • construct a checker

automaton from that

system abstract model

  • utputs

inputs

  • utputs

inputs test automaton generic tester test automaton checker automaton test model test bench model refined

  • r impl.

test generation conformance conformance relation refinement manual

TAP 2009 - Paula Herber 12/18

slide-15
SLIDE 15

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Conformance Test Generation

Goal:

  • compute all possible outputs of a given Uppaal model

Approach:

  • execute test model

together with abstract system model

  • record traces in the

generic tester

  • construct a checker

automaton from that

system abstract model

  • utputs

inputs

  • utputs

inputs test automaton generic tester test automaton checker automaton test model test bench model refined

  • r impl.

test generation conformance conformance relation refinement manual

➠ requires symbolic execution of the Uppaal model

TAP 2009 - Paula Herber 12/18

slide-16
SLIDE 16

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Symbolic Execution

Symbolic Semantics of Uppaal:

  • uses clock zones to abstract from time points
  • symbolic state: location vector, clock zone, variable valuations

Symbolic Execution:

1 start with the initial symbolic state 2 compute all possible symbolic successor states

Challenge:

  • compute outputs offline for non-deterministic specifications

➠ restrict to finite input traces ➠ identify and aggregate semantically equivalent symbolic states ➠ limit the number of internal computation steps

TAP 2009 - Paula Herber 13/18

slide-17
SLIDE 17

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Checker Automaton

  • result of symbolic execution: all possible output traces

Construction of a checker automaton

1 merge end nodes into a pass node 2 mark nodes with inconclusive if computation step limit exceeded 3 each unexpected trace leads to the test verdict fail

inconclusive init pass fail

TAP 2009 - Paula Herber 14/18

slide-18
SLIDE 18

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Checker Automaton

  • result of symbolic execution: all possible output traces

Construction of a checker automaton

1 merge end nodes into a pass node 2 mark nodes with inconclusive if computation step limit exceeded 3 each unexpected trace leads to the test verdict fail

inconclusive init pass fail

➠ from checker automata, SystemC test benches for automatic conformance evaluation can be generated automatically (tbd)

TAP 2009 - Paula Herber 14/18

slide-19
SLIDE 19

SystemC and Uppaal Model Checking Test Generation Results Conclusions

1 SystemC and Uppaal 2 Model Checking of SystemC Designs 3 Conformance Test Generation 4 Experimental Results 5 Conclusions

TAP 2009 - Paula Herber 15/18

slide-20
SLIDE 20

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Case Study: Packet Switch Example

1 Transformation from SystemC to Uppaal and model

checking of safety and timing properties

Computation time (in seconds) 1m1s 2m1s 1m2s 2m2s transformation 1.46 1.59 1.52 1.70 no deadlock 20.82 54.90 42.21 209.56 forward within time limit 127.70 45.04 296.89 543.18 2 Translation into an executable representation and

conformance test generation

Computation time (in seconds) 1m1s 2m1s 1m2s 2m2s translation 7.82 9.91 9.38 10.93 compilation 8.54 10.04 9.05 9.7 test generation 8.75 14.32 23.95 34.14

TAP 2009 - Paula Herber 16/18

slide-21
SLIDE 21

SystemC and Uppaal Model Checking Test Generation Results Conclusions

1 SystemC and Uppaal 2 Model Checking of SystemC Designs 3 Conformance Test Generation 4 Experimental Results 5 Conclusions

TAP 2009 - Paula Herber 17/18

slide-22
SLIDE 22

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Conclusion

Continuous HW/SW co-verification process

combines the power of model checking and of testing yields an automated co-verification flow for SystemC designs

Conformance test generation

  • ffline for non-deterministic specifications

allows for automated testing whether a refined SystemC

design conforms to a verified abstract SystemC design

TAP 2009 - Paula Herber 18/18

slide-23
SLIDE 23

SystemC and Uppaal Model Checking Test Generation Results Conclusions

Conclusion

Continuous HW/SW co-verification process

combines the power of model checking and of testing yields an automated co-verification flow for SystemC designs

Conformance test generation

  • ffline for non-deterministic specifications

allows for automated testing whether a refined SystemC

design conforms to a verified abstract SystemC design Future Work:

  • generate SystemC test benches from checker automata
  • evaluate error detecting capability by a larger case study
  • coverage-driven input selection

TAP 2009 - Paula Herber 18/18