SLIDE 1
Gilles Perrouin gilles.perrouin@unamur.be
@GPerrouin
gilles.perrouin@unamur.be 1
gilles.perrouin@unamur.be 2
Uniform Sampling of SAT Solutions for Configurable Systems: Are We - - PowerPoint PPT Presentation
Uniform Sampling of SAT Solutions for Configurable Systems: Are We - - PowerPoint PPT Presentation
Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet? Gilles Perrouin gilles.perrouin@unamur.be @GPerrouin Journe GLE / LOUISE / RIMEL - GDR GPL CNRS - Talence 12 Avril 2019 gilles.perrouin@unamur.be 1 Uniform
SLIDE 2
SLIDE 3
3
Configurable Systems
SLIDE 4
4
48 33 320 10,000+ # Variants (independent Boolean options)
233 2320
A Universe of Options
SLIDE 5
gilles.perrouin@unamur.be 5
Modelling the Universe: Feature Models
Multifunction printer Print Scan Fax Connection USB Ethernet Laser Inkjet
OR AND XOR Optional feature
- K. Kang, S. Cohen, J. Hess, W. Novak, and S. Peterson.
Feature-Oriented Domain Analysis (FODA) Feasibility Study. Technical Report CMU/SEI-90-TR-21, 1990.
SLIDE 6
gilles.perrouin@unamur.be 6
Modelling the Universe: Feature Models
Multifunction printer Print Scan Fax Connection USB Ethernet Laser Inkjet
- M. Mendonca, A. Wasowski, and K. Czarnecki,
Sat-based analysis of feature models is easy, SPLC ’09
Conjunctive Normal Form (CNF)
SLIDE 7
gilles.perrouin@unamur.be 7
Configurable Systems Sampling
Combinatorial Interaction Testing Dissimilarity Testing Dedicated Heuristics
- M. F. Johansen, Ø. Haugen, and F. Fleurey, An algorithm for generating t-
wise covering arrays from large feature models, SPLC 2012
- B. J. Garvin, M. B. Cohen, and M. B. Dwyer, Evaluating improvements to a
meta-heuristic search for constrained interaction testing, EMSE 2011.
- C. Henard, M. Papadakis, G. Perrouin, J. Klein, P. Heymans, and Y. L.
Traon, Bypassing the combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines, TSE, 2014.
- F. Medeiros, C. Kästner, M. Ribeiro, R. Gheyi, and S. Apel, A comparison of 10
sampling algorithms for configurable systems, ICSE 2016.
SLIDE 8
gilles.perrouin@unamur.be 8
Configurable Systems Sampling
Combinatorial Interaction Testing Dissimilarity Testing Dedicated Heuristics
SLIDE 9
gilles.perrouin@unamur.be 9
Randomness influence in Sampling
Let’s take the (first) configurations returned by the solver…
- A. Halin, A. Nuttinck, M. Acher, X. Devroey, G. Perrouin,
and B. Baudry, Test them all, is it worth it? assessing configuration sampling on the jhipster web development stack, EMSE, 2018.
Most-enabled-disabled (sample of 2 variants) 0 fault covered (but covers 33% of 6 faults on average) Hampers dissimilarity sampling in a search-based exploration: “local” focus due solver’s internal order
- C. Henard, M. Papadakis, G. Perrouin, J. Klein, P. Heymans, and Y. L.
Traon, Bypassing the combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines, TSE, 2014.
SLIDE 10
gilles.perrouin@unamur.be 10
Uniform Random Sampling
Uniform Random Sampling may outperform t-wise sampling Uniform Random Sampling helps with population initialisation in Evolutionary Algorithms
- H. Maaranen, K. Miettinen, and M. M. Mäkelä, Quasi-random initial
population for genetic algorithms, Comput. Math. Appl, 2004.
- A. de Perthuis de Laillevault, B. Doerr, and C. Doerr, Money for nothing:
Speeding up evolutionary algorithms through better initialization, GECCO 2015.
- A. Arcuri and L. Briand, Formal analysis of the probability of
interaction fault detection using random testing, TSE, 2012.
SLIDE 11
gilles.perrouin@unamur.be 11
Uniform Random SAT Sampling
QuickSampler Unigen
- S. Chakraborty, K. S. Meel, and M. Y. Vardi, A scalable and nearly uniform
generator of sat witnesses, CAV 2013.
- S. Chakraborty, D. J. Fremont, K. S. Meel, S. A. Seshia, and M. Y. Vardi, On
parallel scalable uniform SAT witness generation, TACAS 2015
- R. Dutra, K. Laeufer, J. Bachrach, and K. Sen, Efficient
sampling of SAT solutions for testing, ICSE 2018
SLIDE 12
gilles.perrouin@unamur.be 12
Motivation
Since feature models are also SAT formulas we can use UniGen/ QuickSampler to generate uniform samples for configurable systems. Are we there yet ?
SLIDE 13
gilles.perrouin@unamur.be 13
Research Questions
RQ1 (scalability and execution time): Are UniGen and QuickSampler able to generate samples out of feature models? RQ2 (uniformity): Do UniGen and QuickSampler generate uniform configuraQons out of feature models? RQ3 (relevance for tesQng): How does QuickSampler’s sacrifices on uniformity impact its bug-finding ability in JHipster?
SLIDE 14
gilles.perrouin@unamur.be 14
Empirical Evaluation
128 Feature Models 10,000+ 7.7*10417 SAT SoluQons previous benchmarks: 1048 SAT SoluQons
SLIDE 15
gilles.perrouin@unamur.be 15
Scalability (RQ1)
Does not produce any (valid by construcQon) sample within 2 hours for all 128 feature models but Does produce 1 million samples (1 sample/ms) within 2 hours for all 128 feature models but 4. It produces 75 % of valid samples (invalid ones can be removed by SAT check a^erwards)
SLIDE 16
gilles.perrouin@unamur.be 16
Uniformity (RQ2)
Counting the number of times each possible solution is generated does not scale as it requires 4 times as many samples as the feature model has solutions (1050). Subsampling biases towards uniformity Rather, we focus on the frequency of appearance of individual features (fth) and measure deviations (dev): fth(v) = #SAT(ϕ ∧ v) #SAT(ϕ) dev(v) = 100 * | fth(v) − fobs(v)| fth(v)
SLIDE 17
gilles.perrouin@unamur.be 17
QuickSampler Deviation: AIM711
50 % 10 %
Individual feature deviations shown in ascending order
SLIDE 18
gilles.perrouin@unamur.be 18
QuickSampler Deviation: toybox
Individual feature deviations shown in ascending order
50 % 10 %
SLIDE 19
gilles.perrouin@unamur.be 19
QuickSampler Deviation: ucLinux
50 % 10 %
Individual feature deviations shown in ascending order
SLIDE 20
gilles.perrouin@unamur.be 20
Uniformity (RQ2)
QuickSampler is not close to uniformity for feature models with deviations up to 800% (in contrast with non-feature models benchmarks previously used) Negligible observed deviations on non- feature models and
SLIDE 21
gilles.perrouin@unamur.be 21
Relevance for testing (RQ3)
QuickSampler slightly over-represent Uaa involved in 3 of the 6 interactions bugs identified in JHipster. Under-representation of some features does not impact bug finding in JHipster. dev(MongoDB) = 116% dev(Cassandra) = 107%, … dev(Uaa) = 9% OSS full stack configurator, 26,000+ configurations exhaustively assessed in previous work (Halin et al) => Ground truth
SLIDE 22
gilles.perrouin@unamur.be 22
Future Work
Fully assess Minimal Independent Support role in the scalability of Unigen (no improvement seen yet) Experiment with new uniform sampler (developed while the study was made): Investigate if uniform sampling is a cost- effective bug finding strategy
Sharma, S., Gupta, R., Roy, S., & Meel, K. S. (2018). Knowledge Compilation meets Uniform Sampling. EPiC Series in Computing, 57, 620-636.
SLIDE 23
gilles.perrouin@unamur.be 23
128 Feature Models
Uniform but not scalable
Multifunction printer Print Scan Fax Connection USB Ethernet Laser Inkjet
(Mostly) Scalable but not uniform
SLIDE 24
gilles.perrouin@unamur.be 24