Empirical Software Metrics for Benchmarking of Verification Tools - PowerPoint PPT Presentation

Empirical Software Metrics for Benchmarking of Verification Tools Yulia Demyanova, Thomas Pani , Helmut Veith, Florian Zuleger TU Wien FMCAD’17 Student Forum · October 3, 2017

Fu Fully-au autom omated Soft oftwar are Verific ification ion Input Output .c 𝝌 Property

Fu Fully-au autom omated Soft oftwar are Verific ification ion Input Output .c SV Tool 𝝌 Property

Fully-au Fu autom omated Soft oftwar are Verific ification ion Input Output “ 𝝌 holds.” .c SV Tool “ 𝝌 does not hold.” 𝝌 Property “I don’t know.”

Fully-au Fu autom omated Soft oftwar are Verific ification ion Input Output “ 𝝌 holds.” .c SV Tool “ 𝝌 does not hold.” 𝝌 Property “I don’t know.”

Fully-au Fu autom omated Soft oftwar are Verific ification ion Input Output “ 𝝌 holds.” .c SV Tool “ 𝝌 does not hold.” 𝝌 Property “I don’t know.”

Pick cking a verification tool Input Output BLAST “ 𝝌 holds.” CBMC .c ? “ 𝝌 does not hold.” 𝝌 Property CPAchecker “I don’t know.” SMACK …

Pick cking a verification tool Input Output BLAST “ 𝝌 holds.” CBMC .c ? “ 𝝌 does not hold.” 𝝌 Property CPAchecker “I don’t know.” SMACK largely complementary: • specific application areas ( device drivers, … ) … restricted program models ( integer programs, … ) • unsound, incomplete, buggy… •

Pick cking a verification tool Input Output “best” tool tool 1 : BLAST .c tool 2 : CBMC ? tool 3 : CPAchecker 𝝌 Property … tool n : SMACK

Pick cking a verification tool Input Output “best” tool tool 1 : BLAST .c tool 2 : CBMC VeriFolio (ML model) tool 3 : CPAchecker 𝝌 Property … tool n : SMACK

Co Contr tributi tions Input Output “best” tool tool 1 : BLAST .c tool 2 : CBMC VeriFolio (ML model) tool 3 : CPAchecker 𝝌 Property … tool n : SMACK

Ex Expe peri riments – SV SV-CO COMP’16 Score (Overall) Medals

Ex Expe peri riments – SV SV-CO COMP’16 Score (Overall) Medals VeriFolio 3269 1965 UltimateAutomizer 1907 CPA-Seq sim. for SV-COMP’14 & ‘15

Than Th anks! [DPVZ13] On the concept of variable roles and its use in software analysis. FMCAD , 2013 [PVZ15] Loop patterns in C programs. ECEASST 72 , 2015 [DPVZ15] Empirical Software Metrics for Benchmarking of Verification Tools. CAV , 2015 [DPVZ17] Empirical Software Metrics for Benchmarking of Verification Tools. FMSD 50(2-3) , 2017 [DRZ17] Systematic predicate abstraction using variable roles . NFM , 2017 Implementation http://forsyte.at/software/verifolio/