Finding Software Bugs Using Active Automata Learning Frits - PowerPoint PPT Presentation

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Finding Software Bugs Using Active Automata Learning Frits Vaandrager Radboud University Nijmegen RV 2018, Limassol, November 2018 Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Outline Introduction to Model Learning 1 Applications 2 Theory & Tools for Model Learning 3 Conclusions and Future Work 4 Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Research Question Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Research Question We assume SUT behaves deterministically and can be reset. Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Machine Learning in General Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Learning Regular Languages Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Regular Languages and Congruences Definition The equivalence relation ∼ L on Σ ∗ induced by a language L ⊆ Σ ∗ : ∀ w ∈ Σ ∗ : u · w ∈ L ⇔ v · w ∈ L u ∼ L v iff This relation is a right-congruence with respect to concatenation: ∀ u , v , w ∈ Σ ∗ : u ∼ L v ⇒ u · w ∼ L v · w Theorem (Myhill-Nerode, 1958) Language L is regular iff ∼ L has finitely equivalence classes. Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Visualisation Consider the regular language a ( a | b ) ∗ b Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Visualisation Consider the regular language a ( a | b ) ∗ b Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Hankel Matrix Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Hankel Matrix Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Hankel Matrix Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Hankel Matrix Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Hankel Matrix Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Hankel Matrix Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Hankel Matrix u ∼ L v iff rows of u and v in Hankel matrix for L have the same color Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Hankel Matrix u ∼ L v iff rows of u and v in Hankel matrix for L have the same color Language L is regular iff its Hankel matrix contains a finite number of distinct rows, i.e., colors Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Hankel Matrix u ∼ L v iff rows of u and v in Hankel matrix for L have the same color Language L is regular iff its Hankel matrix contains a finite number of distinct rows, i.e., colors The number of states in the smallest DFA for L equals the number of colors in the Hankel matrix Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work What is the FSM for this Hankel Matrix? Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work What is the FSM for this Hankel Matrix? Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Solution Colors of rows in Hankel matrix give us the states. Access strings and one-letter extensions allow us to determine transitions. Column for empty suffix gives us the accepting states. Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work What if Hankel Matrix is Incomplete? Problem to color such a matrix is NP-hard! Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Minimally Adequate Teacher (Angluin) string MQ +/- Learner Teacher hypothesis EQ y/n, counterexample Learner asks membership queries and equivalence queries Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Angluin’s L ∗ Algorithm Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Black Box Checking (Peled, Vardi & Yannakakis) MQ SUL TQs CT EQ Learner Teacher Learner: Formulate hypotheses Conformance Tester (CT): Test correctness hypotheses Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Black Box Checking (Peled, Vardi & Yannakakis) MQ SUL TQs CT EQ Learner Teacher Learner: Formulate hypotheses Conformance Tester (CT): Test correctness hypotheses Model learning and conformance testing two sides of same coin! Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Implements MAT framework for DFAs and Mealy machines Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work A Theory of Mappers (FMSD, 2015) Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work A Theory of Mappers (cnt) Formally, a mapper can be viewed as a transducer (deterministic Mealy machine). A mapper A induces an abstraction operation α A and a concretization operator γ A . Theorem For a mapper A and nondeterministic Mealy machines M and H , α A ( M ) ≤ H iff M ≤ γ A ( H ) . (modulo a minor technical condition) Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Our Research Method Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work EMV Protocol (Aarts et al, 2013) EMV = Europay/Mastercard/Visa Compatibility between smartcards and terminals SEPA requires EMV compliance EMV standard has > 700 pages Learning took at most 1500 membership queries, less than 30 minutes Useful for fingerprinting cards Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work E.dentifier2 (WOOT’14) Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work State Machines for Old and New E.dentifier2 Frits Vaandrager Finding Bugs Using Automata Learning

Introduction to Model Learning Applications Theory & Tools for Model Learning Conclusions and Future Work Bugs in Protocol Implementations Standard violations found in implementations of major protocols: TLS (Usenix Security’15) TCP (CAV’16) SSH (Spin’17) Frits Vaandrager Finding Bugs Using Automata Learning