SLIDE 1
Searching for Strategies that Verify MDE Toolchains Simon Poulding, - - PowerPoint PPT Presentation
Searching for Strategies that Verify MDE Toolchains Simon Poulding, - - PowerPoint PPT Presentation
Searching for Strategies that Verify MDE Toolchains Simon Poulding, University of York & DAASE collaboration with Louis Rose, University of York Context Approach Implementation Case Study Metamodels In Model-Driven Engineering (MDE),
SLIDE 2
SLIDE 3
Metamodels
A
id : String
B
cost : Int
C
size : Int
1..* 0..*
A
id : F45
B
cost : 7
C
size : 25
C
size : 30
A
id : J03
B
cost : 3
B
cost : 18
Metamodel Instance 1 Instance 2
In Model-Driven Engineering (MDE), model instances must comply to a metamodel that specifies attributes and associations
SLIDE 4
Model Transformations
A1 B1 B2 C1 X1 Y1 Y2 Y3
model transformation input model
- utput
model A common operation in MDE toolchains is the transformation of a model to another that conforms to a different metamodel
SLIDE 5
Testing Model Transformations
A2 B3 B4 A1 B1 C1 C2 C3 A7 B8 B9 C8 C9
. . .
X1 Y1 Y2 Y3 X2 Y4 Y5 X6 Y7 Y8 Y9
. . . . . .
test case 1 test case 2 test case N To test a transformations requires a set of (possibly random) test cases in which the input data is a model instance
SLIDE 6
Motivation
Our case study is motivated by a project in which model of behaviour is transformed into a form than can be embodied on a Lego Mindstorms robot
http://lego.wikia.com/wiki/File:LEGO_Mindstorms_NXT.jpg.jpeg
SLIDE 7
Problem Statement
How can we randomly generate models for testing transformations so that a small test set of the models satisfies our testing objective?
SLIDE 8
Context Approach Implementation Case Study
SLIDE 9
Context-Free Grammars
3 4 + 2 * 5 3 * 2 - 5 / 0 / 3 - S → Expr Expr → Num | Expr Op Expr Op → ‘+’ | ‘-’ | ‘*’ | ‘/’ Num → ‘0’ | ‘1’ | ‘2’ | ‘3’ | ‘4’ | ‘5’
A Context-Free Grammar can be used to specify how to construct ‘well-formed’ test data
SLIDE 10
S → Expr Expr → Num | Expr Op Expr Op → ‘+’ | ‘-’ | ‘*’ | ‘/’ Num → ‘0’ | ‘1’ | ‘2’ | ‘3’ | ‘4’ | ‘5’
Stochastic Grammars
1 3 3 3 3 3
By annotating productions rule with weights, a probability distribution is defined
- ver the language defined by the grammar
SLIDE 11
S → Expr Expr → Num | Expr Op Expr Op → ‘+’ | ‘-’ | ‘*’ | ‘/’ Num → ‘0’ | ‘1’ | ‘2’ | ‘3’ | ‘4’ | ‘5’
Innovation: Conditional Weights
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 3 3 3 3 3
‘+’ ‘-’ ‘*’ ‘/’
Making the weights conditional on the values of other variables introduces a limited form of context-sensitivity
SLIDE 12
Innovation: Binned Scalars
S → Expr Expr → Num | Expr Op Expr Op → ‘+’ | ‘-’ | ‘*’ | ‘/’ Num → ‘0’ | ‘1’ | ‘2’ | ‘3’ | ‘4’ | ‘5’ S → Expr Expr → Num | Expr Op Expr Op → ‘+’ | ‘-’ | ‘*’ | ‘/’ Num → ‘[0,228]’ | ‘[229,433]’ | ‘[434,511]’
Adaptively ‘binning’ scalar variables enables a compact representation of distributions over large intervals
SLIDE 13
Metaheuristic Search
S → Expr Expr → Num | Expr Op Expr Op → ‘+’ | ‘-’ | ‘*’ | ‘/’ Num → ‘[0,228]’ | ‘[229,433]’ | ‘[434,511]’
1 3 2
To optimise a distribution, search is applied to the weights, the conditionality between variables, and the partitioning of scalar ranges
SLIDE 14
Context Approach Implementation Case Study
SLIDE 15
HUTN
A
id : F45
B
cost : 7
C
size : 25
C
size : 30
A { id: “F45” b: B { cost: 7} c: C { size: 25}, C {size: 30} }
HUTN is a textual notation for model instances
Model Instance Model Instance
SLIDE 16
Grammar To Emit HUTN
A
id : String
B
cost : Int
C
size : Int
1..* 0..*
Metamodel
S → A A → ‘A’ ‘{’ ‘id:’ String ‘b:’ B1..* ‘c:’ C0..* ‘}’ B1..* → B | B ’,’ B1..* B → ‘{’ ‘cost:’ Cost ‘}’ Cost → ‘[0,100]’ · · · → · · ·
HUTN Grammar
We use a stochastic context-free grammar that emits HUTN which complies with the chosen metamodel
SLIDE 17
Optimisation Process
metamodel HUTN grammar HUTN instance model instance HUTN instance model instance HUTN instance model instance HUTN instance model instance HUTN instance model instance
(2) sample instances from stochastic grammar (1) transform metamodel to HUTN grammar (3) transform HUTN instances to model instances (4) measure fitness
- f model instances
(5) use fitness to optimise grammar The HUTN grammar is optimised by evaluating set of models sampled from the candidate grammar
SLIDE 18
Physical Implementation
metamodel HUTN grammar HUTN instance model instance HUTN instance model instance HUTN instance model instance HUTN instance model instance HUTN instance model instance
Search Executable Instrumented Transformation
Search executable optimises and samples from the grammar; in a servlet, instrumented MDE transformation converts HUTN to instances and assesses; components communicate over HTTP HTTP
SLIDE 19
Context Approach Implementation Case Study
SLIDE 20
Source Metamodel
Specifies the models that are the inputs to the transformation under test
http://lego.wikia.com/wiki/File:LEGO_Mindstorms_NXT.jpg.jpeg
SLIDE 21
Objective: Statistical Testing
frequency coverage element Want to maximise frequency of covering every rule, guard, and condition as frequently as possible in order to minimise test set size
SLIDE 22
Experiments Optimised using hill-climbing (800 evaluations) Optimised using random search (800 evaluations) Unoptimised (‘uniform’ distribution) v v
Compare efficiency of optimised and unoptimised grammar; random search as measure of ‘difficulty’
SLIDE 23
Results
- ptimised (hill-climb)
unoptimised
- ptimised (random search)
100 200 300 number of test cases Number of test cases to cover all elements (with a 90% likelihood) - smaller is better
SLIDE 24
Other Outcomes
Process highlighted ambiguities and missing information in the original metamodel
SLIDE 25
Future Work
Automate metamodel to HUTN grammar conversion; speed up evaluation
metamodel HUTN grammar HUTN instance model instance HUTN instance model instance HUTN instance model instance HUTN instance model instance HUTN instance model instance
Search Executable Instrumented Transformation
HTTP
SLIDE 26