Testing in Resource Constrained Execution Environments Gregory M. - - PowerPoint PPT Presentation

Testing in Resource Constrained Execution Environments

Gregory M. Kapfhammer Department of Computer Science Allegheny College Mary Lou Soffa Department of Computer Science University of Virginia Daniel Mossé Department of Computer Science University of Pittsburgh

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 1/19

Contributions

Use of native code unloading during test suite execution in a resource constrained environment Identification of the testing techniques that yield the greatest reduction in execution time and native code size Characterization of how software applications and test suites restrict and/or support resource constrained testing Cost-benefit analysis for the use of sample-based and exhaustive profiles of program testing behavior

Executes test suites faster when memory resources are limited!

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 2/19

Test Suite Execution with a JVM

Program Stack Fast? Interpreter? Machine Virtual JIT? Adaptive? Native Code Cache Heap

methodA testOne

Input Output Byte Code

T P

During testing the JVM must manage limited resources

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 3/19

Resource Constrained Testing

exec time: 1% size: 100 KB inv ct: 8 inv ct: 1200 exec time: 15% size: 64 KB inv ct: 15 size: 50 KB exec time: 2% inv ct: 50 size: 75 KB exec time: 22%

TE v

inv ct: 1 exec time: 1% inv ct: 2 exec time: 2%

ms m t

...

Test Executor

TE u

...

Memory Resident Native Code Bodies

All Tests size: 128 KB

T1 Tn

...

Test Suite T

P

Program

JIT compiler produces native code that exhausts limited memory resources Frequent invocation of GC increases testing time

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 4/19

Test Suite Execution Strategies

Omit tests? - Could reduce overall confi dence in the

correctness of P

Use non-constrained environment? - Defects related to P’s

interaction with environment might not be isolated

Execute tests individually? - Might increase overall testing

time and violate test order dependencies

Unload with offline profile? - Not useful if P and T change

frequently during regression testing

Our Approach: Use online behavior profi les to guide the

unloading of native code

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 5/19

Experiment Goals and Design

Research Question: Can an adaptive code unloading JVM

reduce time and space overheads associated with memory constrained testing?

Experiment Metrics: percent reduction in time, T %

R (P, T) and

space, S%

R (P, T)

Jikes RVM 2.2.1, JUnit 3.8.1, GNU/Linux 2.4.18 Sample-based (S) and exhaustive (X) program profi les Timer (TM), garbage collection (GC), and code cache size (CS) triggers the unloading technique

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 6/19

Case Study Applications

Name Min Size (MB) # Tests NCSS

UniqueBoundedStack (UBS) 8 24 362 Library (L) 8 53 551 ShoppingCart (SC) 8 20 229 Stack (S) 8 58 624 JDepend (JD) 10 53 2124 IDTable (ID) 11 24 315 Empirically determined the MIN Jikes RVM heap size

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 7/19

Testing Time Overhead: Full RVM

UBS L SC S JD I Application, Full RVM 1 2 3 4 5 Execution Time sec UBS L SC S JD I 0.47 0.52 0.468 0.48 3.67 0.688

When memory is not constrained, testing time is acceptable

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 8/19

Testing Time Overhead: Min RVM

UBS L SC S JD I Application, Min JVM 5 10 15 20 25 Execution Time sec UBS L SC S JD I 4.276 10.644 3.686 5.196 21.108 3.376

Testing time increases signifi cantly when memory is Min

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 9/19

Summary of Reductions for Library

Name

T %

R (P, T)

S%

R (P, T)

S-GC 32.7 78.8 X-GC 32.1 65.0 S-TM 32.0 72.8 X-TM 31.5 62.3 S-CS 34.3 61.4 X-CS 33.4 59.8

Signifi cant reductions in time and space required for testing

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 10/19

Testing Time Overhead: Library

SGC XGC SCS XCS STM XTM Library 2 4 6 8 10 12 Execution Time sec SGC XGC SCS XCS STM XTM 7.168 7.222 6.998 7.09 7.238 7.292

S vs. X: Similar reductions for all code unloading techniques

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 11/19

Testing Space Overhead: Library

SGC XGC SCS XCS STM XTM Library 20000 40000 60000 80000 Code Size KB SGC XGC SCS XCS STM XTM 16052.7 26463.7 29212.1 30444.1 20612.2 28545.1

Code size reduction does not always yield best testing time

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 12/19

Code Size Fluctuation: Library

1 2 3 4 5 6 7 Time sec 10000 20000 30000 40000 Code Size KB SCS

SGC

S-GC causes code size fluctuation that increases testing time

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 13/19

Summary of Reductions for ID

Name

T %

R (P, T)

S%

R (P, T)

S-GC

• 1.1

42.5 X-GC

• 1.1

26.7 S-TM

• 1.2

44.5 X-TM

• .29

28.8 S-CS

• .77

51.4 X-CS

• 1.4

61.4

Unloading can decrease code size while still creating an

• verall increase in testing time

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 14/19

Number of Code Unloads

2.5 5 7.5 10 12.5 15 17.5 20 # Code Unloading Times ID JD S SC L UBS Application ID JD S SC L UBS 2.0 16.0 12.0 4.0 11.0 11.4 4.0 16.6 6.0 4.0 9.0 8.0 4.0 14.0 2.0 1.4 2.0 2.0 SGC STM SCS

All techniques cause ID to perform few unloading sessions

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 15/19

Number of Unloaded Code Bodies

500 1000 1500 2000 2500 # Unloaded Native Code Bodies ID JD S SC L UBS Application ID JD S SC L UBS 164.0 1970.2 589.8 163.0 542.8 469.4 635.8 2228.2 437.4 303.4 533.0 398.4 664.4 1953.8 326.6 194.0 272.4 238.6 SGC STM SCS

ID’s large working set forces unloading of many code bodies

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 16/19

Summary of Reductions

Name

T %

R (P, T)

S%

R (P, T)

S-GC 16.1 68.4 X-GC 16.4 52.8 S-TM 17.1 62.6 X-TM 16.4 45.9 S-CS 17.6 58.8 X-CS 15.3 54.8

Across all applications, adaptive code unloading techniques reduce both testing time and space overhead

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 17/19

Conclusions and Future Work

Dynamic compilation in JVMs can increase testing time if memory is constrained Adaptive unloading of native code often reduces memory overhead, avoids GC invocation, and reduces testing time Impact of unloading varies with respect to size of application’s working set and program testing behavior Regression test suite prioritization and reduction techniques that consider structural coverage and time and space overheads

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 18/19

Additional Resources

Kapfhammer et al. Testing in Resource Constrained Execution Environments. In IEEE/ACM Automated Software

• Engineering. November 7 - 11, 2005.

http : //cs.allegheny.edu/˜gkapfham/research/juggernaut/

Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 19/19