STARTS: STARTS: STARTS: STARTS: STAtic STAtic Regression Test - PowerPoint PPT Presentation

STARTS: STARTS: STARTS: STARTS: STAtic STAtic Regression Test Selection Regression Test Selection STAtic STAtic Regression Test Selection Regression Test Selection Owolabi Legunsen , August Shi, Darko Marinov ASE 2017 Urbana-Champaign, Illinois November 1, 2017 CCF-1409423 CCF-1421503 1

Regression Testing A • Rerun tests after every change to check that T1 T1 B existing functionality is not broken C C T2 T2 • RetestAll: run all tests after each change D T3 T3 Problem: RetestAll can be slow especially E T4 T4 when there are many tests! F 2

Regression Test Selection (RTS) A T1 T1 • RTS speeds up regression testing by rerunning B only tests that are affected by the code changes T2 T2 • End-to-end time for RTS steps must be less than C C time to rerun all tests T3 D Run All Tests (RetestAll) T4 E Time Savings Find Dependencies Analyze Run Affected Tests F End-to-End Time for RTS 3

STARTS • STARTS is an RTS tool that finds dependencies statically • Dynamic may be too slow, or infeasible in some settings • We implemented STARTS as a Maven plugin • STARTS source code is publicly-available on GitHub: • https://github.com/TestingResearchIllinois/starts 4

STARTS Architecture mvn starts:diff Type-Dependency Graph jdeps yasgl Changed Types Old Checksums Checksum Old Dependencies Comparator Type-level Dependencies Checksum Dependency New Checksums New classfiles Finder Updater Impacted Test Finder Test Runner List of all test classes Impacted Tests New Checksums New Dependencies mvn starts:select mvn starts:starts 5

STARTS: Finding Changes mvn starts:diff A Find changes since last time STARTS was run T1 B Old Checksums Checksum Changed Types C Old Dependencies Comparator T2 D T3 Checksum E New Classfiles New Checksums Finder T4 F 6

STARTS: Selecting Impacted Tests mvn starts:select A Select impacted tests without running them T1 T1 B T2 T2 Changed Types C C T3 D Old Checksums Old Dependencies Impacted T4 Test Impacted Tests E Finder List of all Test Classes F 7

STARTS: Running Impacted Tests mvn starts:starts A Select and run impacted tests T1 T1 B T2 T2 Changed Types C C T3 D Old Checksums Old Dependencies Impacted Test T4 Test Impacted Tests Runner E Finder List of all Test Classes F 8

STARTS: Updating for the Next Run Type-Dependency jdeps New Classfiles yasgl Graph Type-Level Dependencies Dependency New Checksums New Checksums Updater New Dependencies jdeps is part of the standard JDK and quickly finds direct class dependencies yasgl is our custom graph library for computing the transitive closure 9

Results on 840 versions of 32 GitHub projects Tests Selected Selected RetestAll STARTS (#) Tests (#) Tests (%) Time (s) Time (%) Avg (SHORT) 58.0 16.4 32.4 17.6 87.8 Avg (LONG) 155.9 54.1 40.5 236.8 68.2 Avg (OVERALL) 91.7 29.4 35.2 93.0 81.0 STARTS is more effective for longer-running projects STARTS Breakdown (%) a nalysis e xecution g raph constr. Compilation Avg (SHORT) 1.0 25.9 8.7 64.4 STARTS analysis and graph construction time is relatively efficient Avg (LONG) 0.8 70.8 2.3 26.2 Avg (OVERALL) 0.9 41.3 6.5 51.3 10

Conclusions • STARTS is a publicly-available, purely static, class-level regression test selection tool • We are investigating ways to make STARTS safer • Maybe also more precise? • STARTS is available on GitHub • https://github.com/TestingResearchIllinois/starts legunse2@illinois.edu 11