STARTS: STARTS: STARTS: STARTS: STAtic STAtic Regression Test - - PowerPoint PPT Presentation

starts starts starts starts static static regression test
SMART_READER_LITE
LIVE PREVIEW

STARTS: STARTS: STARTS: STARTS: STAtic STAtic Regression Test - - PowerPoint PPT Presentation

STARTS: STARTS: STARTS: STARTS: STAtic STAtic Regression Test Selection Regression Test Selection STAtic STAtic Regression Test Selection Regression Test Selection Owolabi Legunsen , August Shi, Darko Marinov ASE 2017 Urbana-Champaign,


slide-1
SLIDE 1

STARTS: STARTS: STARTS: STARTS: STAtic STAtic STAtic STAtic Regression Test Selection Regression Test Selection Regression Test Selection Regression Test Selection

Owolabi Legunsen, August Shi, Darko Marinov

ASE 2017 Urbana-Champaign, Illinois November 1, 2017

CCF-1409423 CCF-1421503

1

slide-2
SLIDE 2

Regression Testing

T1 T2 T3 T4 A B C D E F T1 T2 T3 T4 C

Problem: RetestAll can be slow especially when there are many tests!

  • Rerun tests after every change to check that

existing functionality is not broken

  • RetestAll: run all tests after each change

2

slide-3
SLIDE 3

Regression Test Selection (RTS)

  • RTS speeds up regression testing by rerunning
  • nly tests that are affected by the code changes

T1 T2 T3 T4 A B C D E F C T1 T2

Run All Tests (RetestAll) Find Dependencies Analyze Run Affected Tests

Time Savings End-to-End Time for RTS

  • End-to-end time for RTS steps must be less than

time to rerun all tests

3

slide-4
SLIDE 4

STARTS

4

  • STARTS is an RTS tool that finds dependencies statically
  • Dynamic may be too slow, or infeasible in some settings
  • We implemented STARTS as a Maven plugin
  • STARTS source code is publicly-available on GitHub:
  • https://github.com/TestingResearchIllinois/starts
slide-5
SLIDE 5

STARTS Architecture

Old Checksums Old Dependencies New classfiles List of all test classes Checksum Comparator Checksum Finder New Checksums Impacted Test Finder Test Runner Impacted Tests Dependency Updater Type-Dependency Graph Type-level Dependencies Changed Types yasgl mvn starts:select mvn starts:starts mvn starts:diff New Checksums New Dependencies jdeps

5

slide-6
SLIDE 6

STARTS: Finding Changes

mvn starts:diff Find changes since last time STARTS was run

A B C D E F T1 T2 T3 T4

6

Old Checksums Old Dependencies Changed Types

Checksum Comparator

New Checksums

Checksum Finder

New Classfiles

slide-7
SLIDE 7

STARTS: Selecting Impacted Tests

T1 T2 T3 T4 A B C D E F C T1 T2

mvn starts:select Select impacted tests without running them

7

Old Checksums Old Dependencies List of all Test Classes Impacted Test Finder Changed Types Impacted Tests

slide-8
SLIDE 8

STARTS: Running Impacted Tests

T1 T2 T3 T4 A B C D E F C T1 T2

mvn starts:starts Select and run impacted tests

8

Old Checksums Old Dependencies List of all Test Classes Impacted Test Finder Changed Types Impacted Tests Test Runner

slide-9
SLIDE 9

STARTS: Updating for the Next Run

9

New Classfiles jdeps yasgl Type-Dependency Graph Type-Level Dependencies New Checksums Dependency Updater New Checksums New Dependencies

yasgl is our custom graph library for computing the transitive closure jdeps is part of the standard JDK and quickly finds direct class dependencies

slide-10
SLIDE 10

Results on 840 versions of 32 GitHub projects

Tests (#) Selected Tests (#) Selected Tests (%) RetestAll Time (s) STARTS Time (%) Avg (SHORT) 58.0 16.4 32.4 17.6 87.8 Avg (LONG) 155.9 54.1 40.5 236.8 68.2 Avg (OVERALL) 91.7 29.4 35.2 93.0 81.0

STARTS Breakdown (%)

analysis execution graph constr. Compilation Avg (SHORT) 1.0 25.9 8.7 64.4 Avg (LONG) 0.8 70.8 2.3 26.2 Avg (OVERALL) 0.9 41.3 6.5 51.3

STARTS is more effective for longer-running projects STARTS analysis and graph construction time is relatively efficient

10

slide-11
SLIDE 11

Conclusions

  • STARTS is a publicly-available, purely static, class-level

regression test selection tool

  • We are investigating ways to make STARTS safer
  • Maybe also more precise?
  • STARTS is available on GitHub
  • https://github.com/TestingResearchIllinois/starts

legunse2@illinois.edu

11