Beyond Code Coverage an Approach for Test Suite Assessment and - PowerPoint PPT Presentation

UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED D epartment of Software Engineering Beyond Code Coverage – an Approach for Test Suite Assessment and Improvement Dávid Tengeri* , Árpád Beszédes*, Tamás Gergely*, László Vidács � , Dávid Havas* and Tibor Gyimóthy* *Department of Software Engineering, University of Szeged, Hungary � MTA-SZTE Research Group on Artificial Intelligence, University of Szeged, Hungary IEEE TAICPART 2015, Graz, Austria

UNIVERSITY OF SZEGED Industrial Motivation D epartment of Software Engineering UNIVERSITAS SCIENTIARUM SZEGEDIENSIS WebKit Source code Test suite 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 2

UNIVERSITY OF SZEGED Industrial Motivation D epartment of Software Engineering WebKit UNIVERSITAS SCIENTIARUM SZEGEDIENSIS WebKit • Open source web browser engine • ~ 2.2 million lines of code (mostly C++) • More than 27 000 test cases Source code Test suite 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 3

UNIVERSITY OF SZEGED Industrial Motivation D epartment of Software Engineering UNIVERSITAS SCIENTIARUM SZEGEDIENSIS WebKit Test suite • Challenge with the test suite: • Understand and maintain the test suite Source code Test suite • What is its “quality”? 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 4

UNIVERSITY OF SZEGED Motivation and Goals D epartment of Software Engineering � “Quality” of a (regression) test suite: ■ What is the likelihood of defect detection? UNIVERSITAS SCIENTIARUM SZEGEDIENSIS ■ How efficient, modular, etc. it is? � How can the quality be improved? � Quality assessment of source code – problem solved (?) � Quality assessment of test suites ? Test suite Assessment and Improvement Method (TAIME) 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 5

UNIVERSITY OF SZEGED Determine functional units D epartment of Software Engineering features UNIVERSITAS SCIENTIARUM SZEGEDIENSIS statements, functions subset of test suite 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 6

UNIVERSITY OF SZEGED Assessment of WebKit D epartment of Software Engineering UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Function level coverage of groups in WebKit 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 7

17th April, 2015 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED D epartment of Software Engineering The TAIME approach IEEE TAICPART 2015, Graz, Austria 8

UNIVERSITY OF SZEGED The TAIME approach D epartment of Software Engineering Measure � How good the tests are overall in … UNIVERSITAS SCIENTIARUM SZEGEDIENSIS ■ COV: … executing all parts of the software ■ PART: … being able to localize defects ■ TpP: … being relatively few of them yet effective � How good the test groups are in … ■ SPEC: … specializing to their code group compared to other test groups ■ UNIQ: … uniquely covering their code group compared to other code 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 9

Use cases UNIVERSITY OF SZEGED D epartment of Software Engineering Assessment One-shot refactoring ■ Detect any issues that ■ Delete, reorganize, rewrite, UNIVERSITAS SCIENTIARUM SZEGEDIENSIS require further investigation create test cases ■ Helps find the initial goal in ■ To improve test suite quality the improvement phases Change-oriented test White-box test design suite evolution ■ Monitor and maintain test suite quality during design ■ Create new test cases or possibly remove test cases ■ Use specific measurements as white-box criteria ■ To preserve or improve test suite quality 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 10

UNIVERSITY OF SZEGED Improvement of SoDA D epartment of Software Engineering � Software Development Analysis Framework UNIVERSITAS SCIENTIARUM SZEGEDIENSIS ■ http://soda.sed.hu ■ Platform independent ■ Plugin based ■ TAIME support (with GUI) � SoDA Repository ■ Benchmark programs (SIR, WebKit, GCC) ■ Set of measurement results 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 11

UNIVERSITY OF SZEGED Improvement of SoDA D epartment of Software Engineering � White-box test design UNIVERSITAS SCIENTIARUM SZEGEDIENSIS 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 12

17th April, 2015 UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED D epartment of Software Engineering Improvement of SoDA IEEE TAICPART 2015, Graz, Austria 13

UNIVERSITY OF SZEGED Improvement of the cluster unit D epartment of Software Engineering Improving the coverage of UNIVERSITAS SCIENTIARUM SZEGEDIENSIS cluster unit 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 14

UNIVERSITY OF SZEGED Improvement of the cluster unit D epartment of Software Engineering Improving the coverage of other units UNIVERSITAS SCIENTIARUM SZEGEDIENSIS 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 15

UNIVERSITY OF SZEGED Improvement of the cluster unit D epartment of Software Engineering Removing the usage of cluster unit in UNIVERSITAS SCIENTIARUM SZEGEDIENSIS other units 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 16

UNIVERSITY OF SZEGED TAIME D epartment of Software Engineering One-shot UNIVERSITAS SCIENTIARUM SZEGEDIENSIS refactoring White-box test design Change- oriented test suite evolution http://soda.sed.hu Assessment 17th April, 2015 IEEE TAICPART 2015, Graz, Austria 17