SLIDE 1
ICT
ICT
Example Product Line: The Eclipse IDEs
2
An Algorithm for Generating t-wise Covering Arrays from Large - - PowerPoint PPT Presentation
An Algorithm for Generating t-wise Covering Arrays from Large - - PowerPoint PPT Presentation
An Algorithm for Generating t-wise Covering Arrays from Large Feature Models Martin Fagereng Johansen ystein Haugen Franck Fleurey SPLC 2012 Salvador, Brazil ICT Example Product Line: The Eclipse IDEs ICT 2 Constraints Between
SLIDE 2
SLIDE 3
ICT
Constraints Between Features
3
356,352 possible products
SLIDE 4
ICT
Product Line Verification
How do we gain confidence that any valid product works?
4
SLIDE 5
ICT
Faulty Features
5
Unit tests may find faults inside a single feature.
n test suites required for a product line with n features.
What about faulty cooperation between features?
What if they interact incorrectly?
SLIDE 6
ICT
Interaction Faults
6
2-wise interaction fault
reproducible by including 2 specific features the others do not matter
SLIDE 7
ICT
Interaction Faults
7
3-wise interaction fault
reproducible by including 3 specific features the others do not matter
SLIDE 8
ICT
Kuhn et al. 2004:
Most bugs can be attributed to the interaction of a few features.
Empirics Show:
8
SLIDE 9
ICT
Mathematical property:
Only a few products needed to cover all simple interactions
Other examples (pair-wise testing):
For the "e-shop product line" with 287 features: 21 products For the Linux kernel with almost 7,000 features: 480 products
Covering Arrays
9
SLIDE 10
ICT
- 1. Generate a covering array
Can be reused until the feature model is changed
- 2. Build each product
- 3. Apply a single system testing technique to each product
Note: CIT was originally intended for single system testing
Covering arrays over input instead
- f interactions.
Combinatorial Interaction Testing (CIT)
scalability issue
10
SLIDE 11
ICT
Background
Our MODELS 2011 paper concludes:
Covering array generation is tractable in practice.
Difficult to satisfy FMs imply no products to sell, which is absurd.
An efficient algorithm was not provided.
2-wise testing limit: about 500 features 3-wise testing limit: about 200 features
An efficient algorithm is contributed in this paper.
2-wise testing
Now works for the Linux Kernel feature model (6888 features)
3-wise testing
Now works for the eCos feature model (1244 features) (An optimized C/C++ implementation + some good hardware should
work even for the Linux Kernel feature model.)
11
SLIDE 12
ICT
Overview of the Algorithm
Implementation Supports
Simple XML Feature
Models (SXFM)
GUI DSL DIMACS CVL (Proposed OMG
standard) CSV-file
ICPL
12
t
SLIDE 13
ICT
Data Structures
a = (feature, included) – an assignment e = {a1, a2, ..., at} – a t-set – a set of t assignments Tt – the set of all t-sets It – the set of all invalid t-sets Ut – the set of all valid t-sets (the "universe") C – configuration – a set of assignments, one for each feature CAt – {C1, C2, …, Cx} – a Covering Array of strength t
Equations
𝑈
𝑢 = 2𝑢 𝑔 𝑢 , i.e. 95 million pair-wise interactions for the Linux kernel
Empirically, 𝐽𝑢 ≪ 𝑉𝑢
𝐷𝐵𝑢−1 ⊆ 𝐷𝐵𝑢, thus, generating 𝐷𝐵𝑢−1 before 𝐷𝐵𝑢 is an option.
13
Groundwork
SLIDE 14
ICT
Recursive generation of covering arrays
- f lesser
strength Greedy Loop
Fit as many as
possible
Remove those
covered
… and repeat
until all inter- actions are covered
Not parallel
14
SLIDE 15
ICT
Pick an interaction
Skip if covered
Does it fit the product?
yes no
Make sure all features are assigned Not parallel
15
Quick!
SLIDE 16
ICT
Algorithm 3
Is the assignment valid?
Algorithm 7
For all uncovered
interactions
Is the interaction
covered?
Algorithm 8
Pick an interaction Check if it is invalid
These are data-parallel sub-algorithms
16
SLIDE 17
ICT
ICPL – our new algorithm CASA – Simulated annealing algorithm by Garvin et al. MoSo-Polite – algorithm by Oster et al. IPOG – algorithm by Lei et al. Experiment Machine
Could execute 6 threads in parallel 32 GiB RAM
Compared to Other Tools
17
SLIDE 18
ICT
Time Taken to Generate
18
1 2 3 1-wise 1 2 3 4 2-wise 1 2 3 4 5 3-wise ICPL CASA IPOG* MoSo- Polite
Statistic estimates
The 𝑑 in 𝑃(𝑔𝑑) where 𝑔 is the number of features
SLIDE 19
ICT
Size of Covering Arrays
19
10 20 30 40 50 60 70 1-wise 50 100 150 200 250 300 350 2-wise 200 400 600 800 3-wise ICPL CASA IPOG* MoSo- Polite
SLIDE 20
ICT
Large Feature Models
…
20
SLIDE 21
ICT
Summary
Our contribution
A scalable algorithm for t-wise (1-3) covering array generation An empirical evaluation and comparison
Implementation available
The implementation is available as open source (EPL) Experiments are reproducible
All the data is available
All 28,500 measurements available for the paper's resource
website + charts and summaries
21