Improving quality modelling Stphane Vaucher PhD Defence November - PowerPoint PPT Presentation

Improving quality modelling Stéphane Vaucher PhD Defence November 15, 2010

Presentation structure ● Context ● Problems addressed ● Subjectivity ● Support for evolution ● Composition ● Conclusion DIRO - Thesis defence - Stéphane Vaucher 2

Context DIRO - Thesis defence - Stéphane Vaucher 3

Context ● Software systems are – Large – Complex ● Typical banking/retail company – >50 MLOCs of code – Mission critical ● Quality is important – Bugs – Ability to modify existing code DIRO - Thesis defence - Stéphane Vaucher 4

Testers Project Stakeholders Managers QA team C-level Managers Developers Users DIRO - Thesis defence - Stéphane Vaucher 5

Testers Stakeholders Project Managers Conforms to spec? Easy to estimate? QA team C-level Managers Passes quality audits? Help gain market share? Developer s Users Easy to use? Easy to modify? DIRO - Thesis defence - Stéphane Vaucher 6

Software Evolution ● Systems grow over time ● 8-12% annual employee turnaround DIRO - Thesis defence - Stéphane Vaucher 7 ● Changes can improve/degrade quality

Quality Evaluation Problem ● How to ensure software quality? ● Large: – Automation is essential ● State of the art: – Use of ML or statistics to build predictive models using historical data DIRO - Thesis defence - Stéphane Vaucher 8

Metrics Estimation Quality model DIRO - Thesis defence - Stéphane Vaucher 9

Metrics Estimation Quality model ➔ Product ➔ Bugs ➔ Process ➔ Effort ➔ Resources ➔ Operation performance ➔ Cost of ownership DIRO - Thesis defence - Stéphane Vaucher 10

Context of my Research ● Contacted by a large company – Quality problem: too many bugs ● Situation – ~50% of systems are outsourced – No control over development process used – Want control over quality of systems ● Tried to build models using commercial/OS tools based on “best practices” – ISO9126 – Standard on product quality DIRO - Thesis defence - Stéphane Vaucher 11

State of the Practice ● Most quality models are rule-based ● Do not leverage historical data ● Based on static analysis of the code – (ex: size metrics) ● Example: – Partner's quality portal – CAST software – Sonar – McCabe DIRO - Thesis defence - Stéphane Vaucher 12

Presentation structure ● Context ● Problems addressed ● Subjectivity ● Support for evolution ● Composition ● Conclusion DIRO - Thesis defence - Stéphane Vaucher 13

Problem Many problems found in industry are not addressed in the literature and require research DIRO - Thesis defence - Stéphane Vaucher 14

Problem 1 DIRO - Thesis defence - Stéphane Vaucher 15

Estimates are used by stakeholders Code Audit work Estimate coding task Metrics Estimation Quality model Locate module to test DIRO - Thesis defence - Stéphane Vaucher 16

Estimates must correspond to expectations Metrics Estimation Quality model DIRO - Thesis defence - Stéphane Vaucher 17

The notion of quality depends on the stakeholder Quality is subjective DIRO - Thesis defence - Stéphane Vaucher 18

Problem 2 DIRO - Thesis defence - Stéphane Vaucher 19

Tracking Quality Change Version 1 Version 2 DIRO - Thesis defence - Stéphane Vaucher 20

Tracking Quality Change Changes Identify Changes Version 1 Version 2 Assess quality Assess quality Quality DIRO - Thesis defence - Stéphane Vaucher 21

Quality changes with the system Quality models should help identify good and bad changes DIRO - Thesis defence - Stéphane Vaucher 22

Abstraction level of systems System ... Subsystem 1 Subsystem 2 Subsystem n ... Package 1 Package 2 Package m ... ... Class 1 Class 2 Class i ... Method 1 Method 2 Method j DIRO - Thesis defence - Stéphane Vaucher 23

Abstraction level of systems Manager System Score DIRO - Thesis defence - Stéphane Vaucher 24

Abstraction level of systems Tester Method Score DIRO - Thesis defence - Stéphane Vaucher 25

Composition of Quality Manager System Score ... Subsystem 1 Subsystem n ... Package 1 Package m ... ... Tester Class 1 Class i ... Method 1 Method j Score DIRO - Thesis defence - Stéphane Vaucher 26

The quality of a system depends on the quality of its parts DIRO - Thesis defence - Stéphane Vaucher 27

Thesis ● Three dimensions for improvement of quality models – Support of subjectivity – Support for software evolution – Explicit support of software composition DIRO - Thesis defence - Stéphane Vaucher 28

Presentation structure ● Context ● Problems addressed ● Subjectivity ● Support for evolution ● Composition ● Conclusion DIRO - Thesis defence - Stéphane Vaucher 29

Supporting Subjectivity DIRO - Thesis defence - Stéphane Vaucher 30

Subjectivity ● Quality audits were a source of conflict for our partner No Yes Coding Audit team team ● Rule-based quality models – (e.g. If LOC(method) > 10, bad) – Disagreements on rules/thresholds DIRO - Thesis defence - Stéphane Vaucher 31

Subjectivity: Approaches ● Fuzzy logic: how close is the code to the rules ● Bayes' theory: how probable is the code good given its structure ● Focus on what most agree upon DIRO - Thesis defence - Stéphane Vaucher 32

Subjectivity: Proposal ● Encode subjectivity as the probability that a stakeholder would agree with the model ● P(quality=good|symptoms) Yes Yes No 2/3 DIRO - Thesis defence - Stéphane Vaucher 33

Supporting Subjectivity Detecting Design Defects DIRO - Thesis defence - Stéphane Vaucher 34

Design defects ● Recurring bad design decisions ● From discussions with our partner ● Common defects defined by (Brown, 1996) ● Use of metrics to detect defects (Marinescu, 2004) ● State of the art defect detection (Moha, 2009) – Domain-specific language to specify detection rules DIRO - Thesis defence - Stéphane Vaucher 35

Defects ● Blob: class that centralises functionality ● Spaghetti code: code without structure ● Functional decomposition: classes that are function calls DIRO - Thesis defence - Stéphane Vaucher 36

Data Set ● Original data set from (Moha, 2009) ● Gantt and Xerces: 2 medium open-source packages ● Vote of 5 students (3 originally + 2 additional) ● Unanimous in ~30% of cases DIRO - Thesis defence - Stéphane Vaucher 37

Symptoms ● Reused symptoms from Moha and Brown ● Analysis: symptoms are system-specific ● 2 options – General symptoms: detection on a new system – Specific symptoms: detection on a new version of a known system DIRO - Thesis defence - Stéphane Vaucher 38

Objectives ● Outperform state of the art detection techniques ● Rank results in order of importance: how likely someone will agree with an estimate ● 2 scenarios: – Cross-project: learn on a project, apply to the other – Intra-project: learn and apply on the same project (3-fold cross validation) DIRO - Thesis defence - Stéphane Vaucher 39

Inspection Process ● Inspection in order of probability : P(Defect|Sympt.) ● Comparison with DECOR: a set of candidate classes Class P(Spag|Symptoms) Inspection order XPathException CoreDocumentImpl 99.89% AttrImpl 99.88% ... AttrImpl XPathException 0.11% CoreDocumentImpl ● Recall: % of all defects returned DIRO - Thesis defence - Stéphane Vaucher 40 ● Precision: % true positives returned

Sample output 1 0.9 0.8 0.7 0.6 Bayes(P) Bayes(R) 0.5 Bayes(F1) DECOR(P) 0.4 DECOR(R) DECOR(F1) 0.3 0.2 0.1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 DIRO - Thesis defence - Stéphane Vaucher 41 Classes inspected

Cross-project validation 1 1 1 0.9 0.9 0.9 0.8 0.8 0.8 0.7 0.7 Bayes(P) 0.7 Bayes(P) 0.6 0.6 Bayes(R) 0.6 Bayes(R) 0.5 Bayes(F1) 0.5 0.5 Bayes(F1) DECOR(P) 0.4 0.4 0.4 DECOR(P) DECOR(R) 0.3 0.3 DECOR(R) 0.3 DECOR(F1) DECOR(F1) 0.2 0.2 0.2 0.1 0.1 0.1 0 0 0 1 2 3 4 5 6 7 8 9 10 11 1 2 3 4 5 6 7 8 9 10 11 12 13 14 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 1 1 0.9 1 0.8 0.9 0.9 0.7 0.8 0.8 Bayes(P) 0.6 0.7 Bayes(P) Bayes(R) 0.7 0.5 0.6 Bayes(R) Bayes(F1) 0.6 0.4 Bayes(F1) 0.5 DECOR(P) 0.5 0.3 DECOR(P) 0.4 DECOR(R) 0.2 0.4 0.3 DECOR(R) DECOR(F1) 0.1 0.3 DECOR(F1) 0.2 0 0.2 0.1 1234567891 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 0 0.1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 0 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 3 0 3 1 3 2 3 3 3 4 3 5 3 6 3 7 3 8 3 9 4 0 4 1 4 2 4 3 4 4 ● Same result set, but requires 1/2 inspections DIRO - Thesis defence - Stéphane Vaucher 42