Evaluation of validation tools of Java Agata Gruza and Ramya - - PowerPoint PPT Presentation

Agata Gruza and Ramya Krishna Koricherla Department of Computer Science

Evaluation of validation tools of Java

1.

Introduction

2.

Related Work

3.

Hypothesis

4.

Case Study



Validation tools



Data collection

5.

Results

6.

Threats to validity

7.

Conclusion

8.

Questions

What is a validation tool?

 Validation tool – used to detect bugs in code  Bugs -- coding standards, multithreaded, security and

performance issues.

• 1. Introduction

Why validation tool is important?

 Mars Polar Lander deployed its landing legs at low altitude

and crashed.

 DART 1 collided into the satellite and turned off itself.(new GPS system

was used)

 Ariane 5 had to self destruct because of exceptional handling was

disabled. http://www.cs.sjsu.edu/~pearce/summer09/javaVV .pdf

Why Java?

 Used in safety critical applications.  Used where software quality and reliability matters.

Validation tools:

1.

FindBugs

2.

PMD

3.

Checkstyle

4.

UCDetector

• 2. Related work

1.

In the paper “A Comparison of Bug Finding Tools for Java” Rutar at al. compared FindBugs, PMD, Jlint, Bandera, and ESC/Java.

2.

In the paper “Finding Bugs is Easy” Hovemeyer et al. described bug pattern detectors of FindBugs.

3.

In the paper “Duplicated Code with Linked Editing ”Toomin et al. were talking about pattern duplication.

4.

In the paper “Analysis Tool Evaluation : PMD” , authors described the rule sets of PMD validation tool.

• 3. Hypothesis

 Null Hypotheses:

There is no difference in the number of bugs between validation tools.

 Alternative hypotheses:

There is a significant difference between the validation tools.

• 4. Case study

 19 Java programs are checked against four tools on

Eclipse 4.2.0

 Findbugs:

Version used: Findbugs 2.0; Uses Bug Pattern Detectors

 Class structure and inheritance hierarchy  Linear code scan  Control sensitive  Data flow

• 4. Case study

PMD = Project Mess Detector

Version used PMD 5.0.2; Uses Abstract Syntax Tree concept Major areas:

 Coding standards  Coding anti-patterns  Duplication

Rule sets – naming conventions, braces, strict exceptions, import statements, Java Bean, clone, etc.,

• 4. Case study

Checkstyle

Version used Checkstyle 5.6 Used to improve code quality, readability, re-usability Supports writing custom Checkstyle rule

UCDetector

Version used UCDetector1.10.1

 Dead code  Visibility of code  Final declarations

• 4. Case study

Data collection

 19 programs between 50 to 100 lines of code each.  Generated report of each program for all tools

• 4. Case study

 Important bugs - bugs generated by analyzing code in details.

Examples: Cyclomatic complexity, switch, Boolean, data flow

 Unimportant bugs- bugs related to style of program.

Examples: comments and naming conventions.

• 5. Results

ANOVA test –

Normal distribution

• 5. Results

 SSA = 18398.17  SSB = 2300  SSE= = 1658.833  SST =SSA+SSB+SSE= 22357  %of variance by alternatives= * 100=82.29  %of variance by blocking variable= * 100=10.287  %of variance by errors= * 100= 7.41  4) Statistical significance  MSA = = 6132.72, where a = number of alternatives  MSE= 153.33  MSB= = 331.766 where b= number of blocks= 6  Fcomputed= 39.99  For α=0.01 F0= 5.417. Fcomputed > F0, we reject null hypotheses.  That is there is significant difference in the number of errors detected by each validation

tool.

% of unimportant and important bugs Name of the Tool % of important bugs detected % unimportant bugs detected Findbugs 100% 0% PMD 50% 50% Checkstyle 13% 87%

• 6. Threats to validity

 1) CONCLUSION VALIDITY

Specimen size- In our project we considered 19 modest size Java programs (between 50 to 100 lines of code each).

 2) INTERNAL VALIDITY

Knowledge of Java- Performance monitoring (Human errors) Analysis Method(Important and Unimportant bugs) Time

3) CONSTRUCT VALIDITY

Many measure the same occurrence 4) EXTERNAL VALIDITY Cannot generalize

 CONCLUSION:

Findbugs is efficient tool as it recognizes 100% of important Errors.

Useful to check multithreaded correctness, malicious code vulnerability, security

issues, performance issues.

PMD detects 50% of important bugs.

Useful to check for clone implementation, controversial (Boolean), J2EE, JavaBeans, Exceptions and also good programming practice

Checkstyle detects only 13% of important bugs. Most are related to

programming style ( coding standards).

 UCDetector:

 Useful to check for dead code, to check for appropriate access specifiers.

FUTURE WORK: Can evaluate more tools and compare with already compared tools.

THANK YOU ! QUESTIONS?