SLIDE 1
The Impact of Code Review Coverage and Code Review Participation on Software Quality
Shane McIntosh Ahmed E. Hassan Bram Adams Yasutaka Kamei
2
Shane Bram Yasu
Code reviews: An opportunity for constructive criticism of code changes
The Impact of Code Review Coverage and Code Review Participation on - - PowerPoint PPT Presentation
The Impact of Code Review Coverage and Code Review Participation on - - PowerPoint PPT Presentation
The Impact of Code Review Coverage and Code Review Participation on Software Quality Shane Yasutaka Bram Ahmed E. McIntosh Kamei Adams Hassan Code reviews : An opportunity for constructive criticism of code changes Yasu Shane Bram
SLIDE 2
SLIDE 3
Patch
+ +
- 2
Shane Bram Yasu
Code reviews: An opportunity for constructive criticism of code changes
SLIDE 4
Patch
+ +
- 2
Bram and Yasu, I made a code change. Could you have a look?
Shane Bram Yasu
Code reviews: An opportunity for constructive criticism of code changes
SLIDE 5
Patch
+ +
- 2
Bram and Yasu, I made a code change. Could you have a look? No way, Shane!
Shane Bram Yasu
Code reviews: An opportunity for constructive criticism of code changes
SLIDE 6
Patch
+ +
- 2
Bram and Yasu, I made a code change. Could you have a look? No way, Shane! Your code sucks!
Shane Bram Yasu
Code reviews: An opportunity for constructive criticism of code changes
SLIDE 7
3
Shane Bram Yasu
Code reviews: An opportunity for constructive criticism of code changes
SLIDE 8
Patch
+ +
- 3
Shane Bram Yasu
Code reviews: An opportunity for constructive criticism of code changes
SLIDE 9
Patch
+ +
- 3
Bram and Yasu, I made a code change. Could you have a look?
Shane Bram Yasu
Code reviews: An opportunity for constructive criticism of code changes
SLIDE 10
Patch
+ +
- 3
Bram and Yasu, I made a code change. Could you have a look? Watch about NULL values at line 3!
Shane Bram Yasu
Code reviews: An opportunity for constructive criticism of code changes
SLIDE 11
Patch
+ +
- 3
Bram and Yasu, I made a code change. Could you have a look? Watch about NULL values at line 3! Your code still sucks!
Shane Bram Yasu
Code reviews: An opportunity for constructive criticism of code changes
SLIDE 12
SLIDE 13
Traditional Code Inspections
Mandated reviewer checklists and in-person meetings
SLIDE 14
Traditional Code Inspections
Mandated reviewer checklists and in-person meetings
Measurably improves software quality
SLIDE 15
Traditional Code Inspections
Mandated reviewer checklists and in-person meetings
Modern Code Review
Lightweight, tool-supported
Measurably improves software quality
SLIDE 16
Traditional Code Inspections
Mandated reviewer checklists and in-person meetings
Modern Code Review
Lightweight, tool-supported
Measurably improves software quality ???
SLIDE 17
5
Are properties of the modern code review process linked to software quality?
SLIDE 18
Coverage
5
Are properties of the modern code review process linked to software quality?
SLIDE 19
Participation Coverage
5
Are properties of the modern code review process linked to software quality?
SLIDE 20
6
Are properties of the modern code review process linked to software quality?
SLIDE 21
Patch
+ +
- Patch
+ +
- 7
Shane Bram Yasu
Gerrit Code Review Process
Gerrit
SLIDE 22
Patch
+ +
- Patch
+ +
- 7
Shane Bram Yasu
Gerrit Code Review Process
Gerrit
SLIDE 23
Patch
+ +
- Patch
+ +
- 7
Shane Bram Yasu
Gerrit Code Review Process
Gerrit Request review
SLIDE 24
Patch
+ +
- Patch
+ +
- 7
Shane Bram Yasu
Gerrit Code Review Process
Gerrit Watch about NULL values at line 3!
SLIDE 25
Patch
+ +
- Patch
+ +
- 7
Shane Bram Yasu
Gerrit Code Review Process
Gerrit Watch about NULL values at line 3! Your code sucks!
SLIDE 26
Patch
+ +
- Patch
+ +
- 7
Shane Bram Yasu
Gerrit Code Review Process
Gerrit Watch about NULL values at line 3! Your code sucks!
SLIDE 27
8
Shane Bram Yasu
Gerrit Code Review Process
Gerrit
SLIDE 28
8
Shane Bram Yasu
Gerrit Code Review Process
Gerrit
SLIDE 29
8
Shane Bram Yasu
Gerrit Code Review Process
Gerrit Upstream
SLIDE 30
8
Shane Bram Yasu
Gerrit Code Review Process
Gerrit Upstream
Changes are automatically integrated after review criteria are satisfied
SLIDE 31
9
Reviews can be linked to commits in source code repositories
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <alan.alpert@nokia.com> Date: Thu Mar 1 20:09:52 2012 +1000- Add a hidden XandYAxis enum value
Gerrit Upstream
SLIDE 32
9
Reviews can be linked to commits in source code repositories
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <alan.alpert@nokia.com> Date: Thu Mar 1 20:09:52 2012 +1000- Add a hidden XandYAxis enum value
Gerrit Upstream
SLIDE 33
9
Reviews can be linked to commits in source code repositories
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <alan.alpert@nokia.com> Date: Thu Mar 1 20:09:52 2012 +1000- Add a hidden XandYAxis enum value
Gerrit Upstream
SLIDE 34
9
Reviews can be linked to commits in source code repositories
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <alan.alpert@nokia.com> Date: Thu Mar 1 20:09:52 2012 +1000- Add a hidden XandYAxis enum value
Gerrit Upstream
SLIDE 35
10
Are properties of the modern code review process linked to software quality?
SLIDE 36
11
Software Quality as Post-Release Defect Counts
Upstream Development Release Feature development Defect repairing Merge Commit types
SLIDE 37
11
Software Quality as Post-Release Defect Counts
Upstream Development Release Feature development Defect repairing Merge Commit types
SLIDE 38
11
Software Quality as Post-Release Defect Counts
Upstream Development Release Feature development Defect repairing Merge Commit types
SLIDE 39
11
Software Quality as Post-Release Defect Counts
Upstream Development Release Feature development Defect repairing Merge Commit types
SLIDE 40
11
Software Quality as Post-Release Defect Counts
Upstream v1.0 Development Release Feature development Defect repairing Merge Commit types
SLIDE 41
11
Software Quality as Post-Release Defect Counts
Upstream v1.0 Development Release Feature development Defect repairing Merge Commit types
SLIDE 42
11
Software Quality as Post-Release Defect Counts
Upstream v1.0 Development Release Feature development Defect repairing Merge Commit types
SLIDE 43
12
Defect Models
- Comp. A
- Comp. B
- Comp. C
How many defects?
SLIDE 44
12
Defect Models
- Comp. A
- Comp. B
- Comp. C
How many defects?
5 1
SLIDE 45
v1.0
13
Development Release
Explanatory Defect Models
SLIDE 46
13
Development Release Defect model v1.0
Explanatory Defect Models
SLIDE 47
14
Explanatory Defect Models
Defect model v1.0
SLIDE 48
14
Explanatory Defect Models
Defect model v1.0
Analyze relationship between predictors and defects
SLIDE 49
15
The Baseline
SLIDE 50
15
Product Metrics
The Baseline
SLIDE 51
15
Product Metrics
The Baseline
e.g., SLOC, complexity
SLIDE 52
15
Product Metrics
The Baseline
Process Metrics e.g., SLOC, complexity
SLIDE 53
15
Product Metrics
The Baseline
Process Metrics e.g., SLOC, complexity e.g., Prior defects, churn
SLIDE 54
15
Product Metrics
The Baseline
Process Metrics Human Factors Metrics e.g., SLOC, complexity e.g., Prior defects, churn
SLIDE 55
15
Product Metrics
The Baseline
Process Metrics Human Factors Metrics e.g., SLOC, complexity e.g., Prior defects, churn e.g., # Minor contributors
SLIDE 56
The Context
16
SLIDE 57
The Context
16
1,339 components 254 defect-prone v5.0.0 1,337 components 187 defect-prone v5.1.0
SLIDE 58
The Context
16
1,339 components 254 defect-prone v5.0.0 1,337 components 187 defect-prone v5.1.0 170 components 15 defect-prone v5.10.0
SLIDE 59
The Context
16
1,339 components 254 defect-prone v5.0.0 1,337 components 187 defect-prone v5.1.0 170 components 15 defect-prone v5.10.0 218 components 24 defect-prone v4.3.0
SLIDE 60
Are properties of the code reviewing process linked to software quality?
Coverage Participation
SLIDE 61
18
Are properties of the code reviewing process linked to software quality?
Coverage Participation
SLIDE 62
18
Are properties of the code reviewing process linked to software quality?
Metrics
- Reviewed changes (%)
Reviewed churn (%)
Coverage Participation
SLIDE 63
19
Concrete Predicted Defect Count
How many defects?
Create artificial component setting explanatory variables to median values
Artificial Comp. 500 SLOC Explanatory Variable Value Reviewed changes 100% Churn 150
SLIDE 64
20
Concrete Predicted Defect Count
How many defects?
Calculate number of predicted defects
Artificial Comp. 500 SLOC Explanatory Variable Value Reviewed changes 100% Churn 150
SLIDE 65
20
Concrete Predicted Defect Count
How many defects?
Calculate number of predicted defects
Artificial Comp. 500 SLOC Explanatory Variable Value Reviewed changes 100% Churn 150
SLIDE 66
21
Concrete Predicted Defect Count
How many defects?
Artificial Comp. 500 SLOC Explanatory Variable Value Reviewed changes 90% Churn 150
Modify variable under test while keeping other explanatory variables constant
SLIDE 67
21
Concrete Predicted Defect Count
How many defects?
Artificial Comp. 500 SLOC Explanatory Variable Value Reviewed changes 90% Churn 150
Modify variable under test while keeping other explanatory variables constant
SLIDE 68
21
Concrete Predicted Defect Count
How many defects?
Artificial Comp. 500 SLOC Explanatory Variable Value Reviewed changes 90% Churn 150
1
Modify variable under test while keeping other explanatory variables constant
SLIDE 69
22
Proportion of Reviewed Changes
- 1
2 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Proportion of Reviewed Changes Concrete Predicted Defect Count
- Qt 5.0.0
VTK 5.10.0
SLIDE 70
23
Are properties of the code reviewing process linked to software quality?
Coverage Participation
Metrics
- Reviewed changes (%)
Reviewed churn (%)
SLIDE 71
23
Are properties of the code reviewing process linked to software quality?
Coverage Participation
Results
- Coverage shares a link, but it does
not tell the whole story
Metrics
- Reviewed changes (%)
Reviewed churn (%)
SLIDE 72
24
⇧ ⇧ ⇧ ⇧ Reviewed changes
- 0.25
9∗∗∗ ⇧
- 0.30
15∗∗∗ ⇧ Reviewed churn † † † † † Discarded during correlation analysis (|ρ| > 0.7) ‡ Discarded during VIF analysis (VIF coefficient > 5) Statistical significance of explanatory power according to Drop One analysis: ⇧ p 0.05; ∗ p < 0.05; ∗∗ p < 0.01; ∗∗∗ p < 0.001
Additional factors are likely at play
Qt VTK ITK 5.0.0 5.1.0 5.10.0 4.3.0
2
Coef. ∆AIC Coef. ∆AIC Coef. ∆AIC Coef. ∆AIC Size ⇧ 0.46 6 0.19 223.4 ⇧
SLIDE 73
24
⇧ ⇧ ⇧ ⇧ Reviewed changes
- 0.25
9∗∗∗ ⇧
- 0.30
15∗∗∗ ⇧ Reviewed churn † † † † † Discarded during correlation analysis (|ρ| > 0.7) ‡ Discarded during VIF analysis (VIF coefficient > 5) Statistical significance of explanatory power according to Drop One analysis: ⇧ p 0.05; ∗ p < 0.05; ∗∗ p < 0.01; ∗∗∗ p < 0.001
Additional factors are likely at play
Qt VTK ITK 5.0.0 5.1.0 5.10.0 4.3.0
2
Reviewed changes is not a significant contributor of explanatory power
Coef. ∆AIC Coef. ∆AIC Coef. ∆AIC Coef. ∆AIC Size ⇧ 0.46 6 0.19 223.4 ⇧
SLIDE 74
Are properties of the code reviewing process linked to software quality?
Metrics
- Reviewed changes (%)
Reviewed churn (%)
Coverage Participation
Results
- Coverage shares a link, but it does
not tell the whole story
SLIDE 75
Are properties of the code reviewing process linked to software quality?
Metrics
- Self-approved changes (%)
Hastily reviewed changes (%) Changes without discussion (%)
Metrics
- Reviewed changes (%)
Reviewed churn (%)
Coverage Participation
Results
- Coverage shares a link, but it does
not tell the whole story
SLIDE 76
Hold on, we need to control for coverage!
SLIDE 77
27
Code Review Participation
- 1
2 3 4 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Metric Value Concrete Predicted Defect Count
- Qt 5.0.0
Qt 5.1.0 ITK 4.3.0
- Hastily reviewed
No discussion Self−Approved
SLIDE 78
27
Code Review Participation
- 1
2 3 4 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Metric Value Concrete Predicted Defect Count
- Qt 5.0.0
Qt 5.1.0 ITK 4.3.0
- Hastily reviewed
No discussion Self−Approved
5 additional post-release defects
SLIDE 79
27
Code Review Participation
- 1
2 3 4 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Metric Value Concrete Predicted Defect Count
- Qt 5.0.0
Qt 5.1.0 ITK 4.3.0
- Hastily reviewed
No discussion Self−Approved
5 additional post-release defects 3 additional post-release defects
SLIDE 80
27
Code Review Participation
- 1
2 3 4 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Metric Value Concrete Predicted Defect Count
- Qt 5.0.0
Qt 5.1.0 ITK 4.3.0
- Hastily reviewed
No discussion Self−Approved
5 additional post-release defects 3 additional post-release defects Small increase leads to defects
SLIDE 81
28
Are properties of the code reviewing process linked to software quality?
Metrics
- Self-approved changes (%)
Hastily reviewed changes (%) Changes without discussion (%)
Metrics
- Reviewed changes (%)
Reviewed churn (%)
Coverage Participation
Results
- Coverage shares a link, but it does
not tell the whole story
SLIDE 82
28
Are properties of the code reviewing process linked to software quality?
Metrics
- Self-approved changes (%)
Hastily reviewed changes (%) Changes without discussion (%)
Metrics
- Reviewed changes (%)
Reviewed churn (%)
Coverage Participation
Results
- Components w/ lax review
participation are estimated to have five additional post-release defects
Results
- Coverage shares a link, but it does
not tell the whole story
SLIDE 83
SLIDE 84
SLIDE 85
SLIDE 86
SLIDE 87
SLIDE 88
SLIDE 89
@shane_mcintosh shanemcintosh@acm.org