Empirical Evaluation of an Approach to Resource Constrained Test - - PowerPoint PPT Presentation

Empirical Evaluation of an Approach to Resource Constrained Test Suite Execution

Gregory M. Kapfhammer Department of Computer Science Allegheny College (in conjunction with Mary Lou Soffa and Daniel Mossé)

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 1/30

Looking Ahead

Use of native code unloading during test suite execution in a resource constrained environment Identification of the testing techniques that yield the greatest reduction in execution time and native code size Characterization of how software applications and test suites restrict and/or support resource constrained testing Cost-benefit analysis for the use of sample-based and exhaustive profiles of program testing behavior

Executes test suites faster when memory resources are limited!

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 2/30

Traditional Testing Techniques

Specification Testing Testing Random

MTTF(P) = k

Reliability Software

Input Freq

Testing Structural

Input Space

Different approaches to establishing a confi dence in the correctness of and fi nding faults within software

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 3/30

Test Suite Execution

setUp tearDown

T1 m1

setUp tearDown

T mn n

• racle
• racle

A test executes a method and uses an oracle to determine if the method’s output is correct Test suite execution frameworks exist for many different programming languages (e.g., JUnit for Java)

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 4/30

Test Suite Execution with a JVM

Program Stack Fast? Interpreter? Machine Virtual JIT? Adaptive? Native Code Cache Heap

methodA testOne

Input Output Byte Code

T P

During testing the JVM must manage limited resources

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 5/30

Resource Constrained Testing

exec time: 1% size: 100 KB inv ct: 8 inv ct: 1200 exec time: 15% size: 64 KB inv ct: 15 size: 50 KB exec time: 2% inv ct: 50 size: 75 KB exec time: 22%

TE v

inv ct: 1 exec time: 1% inv ct: 2 exec time: 2%

ms m t

...

Test Executor

TE u

...

Memory Resident Native Code Bodies

All Tests size: 128 KB

T1 Tn

...

Test Suite T

P

Program

JIT compiler produces native code that exhausts limited memory resources Frequent invocation of GC increases testing time

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 6/30

Test Suite Execution Strategies

Omit tests? - Could reduce overall confi dence in the

correctness of P

Use non-constrained environment? - Defects related to P’s

interaction with environment might not be isolated

Execute tests individually? - Might increase overall testing

time and violate test order dependencies

Unload with offline profile? - Not useful if P and T change

frequently during regression testing

Our Approach: Use online behavior profi les to guide the

unloading of native code

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 7/30

Testing with Native Code Unloading

exec time: 1% size: 100 KB inv ct: 8 inv ct: 1200 exec time: 15% size: 64 KB inv ct: 15 size: 50 KB exec time: 2% inv ct: 50 size: 75 KB exec time: 22%

TE v

inv ct: 1 exec time: 1% inv ct: 2 exec time: 2%

ms m t

...

Test Executor

TE u

...

All Tests size: 128 KB

T1 Tn

...

Test Suite T

P

Program

What to unload? When to unload? Garbage Collector Code Cache Size

k

Timer

{

When - three separate techniques and What - behavior profile stored in code body (Zhang and Krintz)

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 8/30

Unloading Configurations: GC and TM

{S, X} − {GC, TM} = C, UC, U, H

Parameter Meaning

C init period (GC cycles, secs) UC init unload freq (GC cycles, secs) U non-init unload freq (GC cycles, secs) H heap residency threshold (%)

Use GC execution or timer expiration to trigger code unloading

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 9/30

Unloading Configuration: CS

{S, X} − CS = Zinit, Zincr, UCS

Parameter Meaning

Zinit init code cache size (bytes) Zincr code cache increment size (bytes) UCS unload session resize trigger (count)

Unload when native code cache is full

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 10/30

Experiment Goals and Design

Research Question: Can an adaptive code unloading JVM

reduce time and space overheads associated with memory constrained testing?

Experiment Metrics: percent reduction in time, T %

R (P, T) and

space, S%

R (P, T)

Jikes RVM 2.2.1, JUnit 3.8.1, GNU/Linux 2.4.18 Sample-based (S) and exhaustive (X) program profi les Timer (TM), garbage collection (GC), and code cache size (CS) triggers the unloading technique

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 11/30

Case Study Applications

Name Min Size (MB) # Tests NCSS

UniqueBoundedStack (UBS) 8 24 362 Library (L) 8 53 551 ShoppingCart (SC) 8 20 229 Stack (S) 8 58 624 JDepend (JD) 10 53 2124 IDTable (ID) 11 24 315 Empirically determined the MIN Jikes RVM heap size

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 12/30

Native Code Unloading Configurations

Name GC CS TM

UBS 4, 1, 1, 0.0 49370, 512, 5 3, .5, 1, 0.0 L 5, 1, 3, 0.0 49370, 512, 5 3, .5, 1, 0.0 SC 3, 1, 1, 0.0 49370, 512, 5 2, .5, 1, 0.0 S 4, 1, 1, 0.0 49370, 512, 5 3, .5, 1, 0.0 JD 8, 1, 4, 0.0 49370, 512, 5 3, .5, 1, 0.0 ID 1, 1, 3, 0.0 65536, 8192, 5 2, .5, 1, 0.0

Ten minutes (or less) of configuration time for each technique S and X use same configuration to a ensure fair comparison

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 13/30

Testing Time Overhead: Full RVM

UBS L SC S JD I Application, Full RVM 1 2 3 4 5 Execution Time sec UBS L SC S JD I 0.47 0.52 0.468 0.48 3.67 0.688

When memory is not constrained, testing time is acceptable

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 14/30

Testing Time Overhead: Min RVM

UBS L SC S JD I Application, Min JVM 5 10 15 20 25 30 Execution Time sec UBS L SC S JD I 4.276 10.644 3.686 5.196 21.108 3.376

Testing time increases signifi cantly when memory is Min

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 15/30

Summary of Reductions for Library

Name

T %

R (P, T)

S%

R (P, T)

S-GC 32.7 78.8 X-GC 32.1 65.0 S-TM 32.0 72.8 X-TM 31.5 62.3 S-CS 34.3 61.4 X-CS 33.4 59.8

Signifi cant reductions in time and space required for testing

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 16/30

Testing Time Overhead: Library

SGC XGC SCS XCS STM XTM Library 2 4 6 8 10 12 Execution Time sec SGC XGC SCS XCS STM XTM 7.168 7.222 6.998 7.09 7.238 7.292

S vs. X: Similar reductions for all code unloading techniques

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 17/30

Testing Space Overhead: Library

SGC XGC SCS XCS STM XTM Library 20000 40000 60000 80000 Code Size KB SGC XGC SCS XCS STM XTM 16052.7 26463.7 29212.1 30444.1 20612.2 28545.1

Code size reduction does not always yield best testing time

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 18/30

Code Size Fluctuation: Library

1 2 3 4 5 6 7 Time sec 10000 20000 30000 40000 Code Size KB SCS

SGC

S-GC causes code size fluctuation that increases testing time

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 19/30

Summary of Reductions for Stack

Name

T %

R (P, T)

S%

R (P, T)

S-GC 24.3 79.0 X-GC 25.4 63.4 S-TM 25.0 64.9 X-TM 24.6 47.8 S-CS 24.7 61.6 X-CS 20.9 46.9

Across all applications: S-TM or S-CS normally produce the largest reduction in testing time overhead

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 20/30

Testing Time Overhead: Stack

SGC XGC SCS XCS STM XTM Stack 1 2 3 4 5 6 Execution Time sec SGC XGC SCS XCS STM XTM 3.932 3.872 3.914 4.11 3.896 3.918

S vs. X: Similar time reductions for another application

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 21/30

Testing Space Overhead: Stack

SGC XGC SCS XCS STM XTM Stack 10000 20000 30000 40000 50000 60000 70000 Code Size KB SGC XGC SCS XCS STM XTM 13646.5 23797.1 24999.4 34524.5 22794.2 33977.3

S-GC also produces best code size reduction without creating greatest reduction in testing time

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 22/30

Code Size Fluctuation: Stack

0.5 1 1.5 2 2.5 3 3.5 Time sec 10000 20000 30000 40000 Code Size KB SCS SGC

S-GC also causes code size fluctuations in other applications

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 23/30

Summary of Reductions for ID

Name

T %

R (P, T)

S%

R (P, T)

S-GC

• 1.1

42.5 X-GC

• 1.1

26.7 S-TM

• 1.2

44.5 X-TM

• .29

28.8 S-CS

• .77

51.4 X-CS

• 1.4

61.4

Unloading can decrease native code size while still creating an overall increase in testing time

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 24/30

Number of Code Unloads

2.5 5 7.5 10 12.5 15 17.5 20 # Code Unloading Times ID JD S SC L UBS Application ID JD S SC L UBS 2.0 16.0 12.0 4.0 11.0 11.4 4.0 16.6 6.0 4.0 9.0 8.0 4.0 14.0 2.0 1.4 2.0 2.0 SGC STM SCS

All techniques cause ID to perform few unloading sessions

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 25/30

Number of Unloaded Code Bodies

500 1000 1500 2000 2500 # Unloaded Native Code Bodies ID JD S SC L UBS Application ID JD S SC L UBS 164.0 1970.2 589.8 163.0 542.8 469.4 635.8 2228.2 437.4 303.4 533.0 398.4 664.4 1953.8 326.6 194.0 272.4 238.6 SGC STM SCS

ID’s large working set forces unloading of many code bodies

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 26/30

Summary of Reductions

Name

T %

R (P, T)

S%

R (P, T)

S-GC 16.1 68.4 X-GC 16.4 52.8 S-TM 17.1 62.6 X-TM 16.4 45.9 S-CS 17.6 58.8 X-CS 15.3 54.8

Across all applications, adaptive code unloading techniques reduce both testing time and space overhead

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 27/30

Conclusions

Dynamic compilation in JVMs can increase testing time if memory is constrained Adaptive unloading of native code often reduces memory overhead, avoids GC invocation, and reduces testing time Impact of unloading varies with respect to size of application’s working set and program testing behavior Code unloading JVM can be rapidly confi gured to produce useful reductions in the time and space

• verheads of testing

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 28/30

Future Work

Include new case study applications and test suites Experiments to measure the impact of garbage collection and heap compression algorithms (e.g., Jikes RVM MMTk) Regression test suite prioritization and reduction techniques that consider structural coverage and time

• r space overheads

Real testing framework for emerging operating systems that support Java in ad hoc networks (e.g., MagnetOS)

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 29/30

Additional Resources

Kapfhammer et al. Testing in Resource Constrained Execution Environments. In IEEE/ACM Automated Software

• Engineering. November 7 - 11, 2005.

http : //cs.allegheny.edu/˜gkapfham/research/juggernaut/

Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 30/30