Evaluation of validation tools of Java Agata Gruza and Ramya - PowerPoint PPT Presentation

Evaluation of validation tools of Java Agata Gruza and Ramya Krishna Koricherla Department of Computer Science

Introduction 1. Related Work 2. Hypothesis 3. Case Study 4. Validation tools  Data collection  Results 5. Threats to validity 6. Conclusion 7. Questions 8.

What is a validation tool?  Validation tool – used to detect bugs in code  Bugs -- coding standards, multithreaded, security and performance issues.

1. Introduction Why validation tool is important?  Mars Polar Lander deployed its landing legs at low altitude and crashed.  DART 1 collided into the satellite and turned off itself.(new GPS system was used)  Ariane 5 had to self destruct because of exceptional handling was disabled. http://www.cs.sjsu.edu/~pearce/summer09/javaVV .pdf

Why Java?  Used in safety critical applications.  Used where software quality and reliability matters. Validation tools: FindBugs 1. PMD 2. Checkstyle 3. UCDetector 4.

2. Related work In the paper “A Comparison of Bug Finding Tools for Java” 1. Rutar at al. compared FindBugs, PMD, Jlint, Bandera, and ESC/Java. In the paper “Finding Bugs is Easy” Hovemeyer et al. described 2. bug pattern detectors of FindBugs. In the paper “Duplicated Code with Linked Editing ”Toomin 3. et al. were talking about pattern duplication. In the paper “Analysis Tool Evaluation : PMD” , authors 4. described the rule sets of PMD validation tool.

3. Hypothesis  Null Hypotheses: There is no difference in the number of bugs between validation tools.  Alternative hypotheses: There is a significant difference between the validation tools.

4. Case study  19 Java programs are checked against four tools on Eclipse 4.2.0  Findbugs: Version used: Findbugs 2.0; Uses Bug Pattern Detectors  Class structure and inheritance hierarchy  Linear code scan  Control sensitive  Data flow

4. Case study PMD = Project Mess Detector Version used PMD 5.0.2; Uses Abstract Syntax Tree concept Major areas:  Coding standards  Coding anti-patterns  Duplication Rule sets – naming conventions, braces, strict exceptions, import statements, Java Bean, clone, etc.,

4. Case study Checkstyle Version used Checkstyle 5.6 Used to improve code quality, readability, re-usability Supports writing custom Checkstyle rule UCDetector Version used UCDetector1.10.1  Dead code  Visibility of code  Final declarations

4. Case study Data collection  19 programs between 50 to 100 lines of code each.  Generated report of each program for all tools

4. Case study  Important bugs - bugs generated by analyzing code in details. Examples: Cyclomatic complexity, switch, Boolean, data flow  Unimportant bugs- bugs related to style of program. Examples: comments and naming conventions.

5. Results ANOVA test – Normal distribution

5. Results  SSA = 18398.17  SSB = 2300  SSE= = 1658.833  SST =SSA+SSB+SSE= 22357  %of variance by alternatives= * 100=82.29  %of variance by blocking variable= * 100=10.287  %of variance by errors= * 100= 7.41  4) Statistical significance  MSA = = 6132.72, where a = number of alternatives  MSE= 153.33  MSB= = 331.766 where b= number of blocks= 6  F computed = 39.99  For α =0.01 F 0 = 5.417. F computed > F 0 , we reject null hypotheses.  That is there is significant difference in the number of errors detected by each validation tool.

% of unimportant and important bugs Name of the Tool % of important bugs detected % unimportant bugs detected Findbugs 100% 0% PMD 50% 50% Checkstyle 13% 87%

6. Threats to validity  1) CONCLUSION VALIDITY Specimen size - In our project we considered 19 modest size Java programs (between 50 to 100 lines of code each).  2) INTERNAL VALIDITY Knowledge of Java- Performance monitoring (Human errors) Analysis Method(Important and Unimportant bugs) Time

3) CONSTRUCT VALIDITY Many measure the same occurrence 4) EXTERNAL VALIDITY Cannot generalize

 CONCLUSION: Findbugs is efficient tool as it recognizes 100% of important Errors. Useful to check multithreaded correctness, malicious code vulnerability, security issues, performance issues. PMD detects 50% of important bugs. Useful to check for clone implementation, controversial (Boolean), J2EE, JavaBeans, Exceptions and also good programming practice Checkstyle detects only 13% of important bugs . Most are related to programming style ( coding standards).

 UCDetector:  Useful to check for dead code, to check for appropriate access specifiers. FUTURE WORK: Can evaluate more tools and compare with already compared tools.

THANK YOU ! QUESTIONS?