The Impact of Human Discussions on Just-In-Time Quality Assurance - - PowerPoint PPT Presentation

▶
the impact of human discussions on just in time quality
SMART_READER_LITE
LIVE PREVIEW

The Impact of Human Discussions on Just-In-Time Quality Assurance - - PowerPoint PPT Presentation

Nov 10, 2023 356 likes •836 views

The Impact of Human Discussions on Just-In-Time Quality Assurance Parastou Tourani, Bram Adams MCIS, Polytechnique Montral Thursday, March 3, 16 Life Cycle of a Bug Bug Report Fix Change Bug Introducing Change Thursday, March 3, 16

slide-1
SLIDE 1

The Impact of Human Discussions

  • n Just-In-Time Quality Assurance

Parastou Tourani, Bram Adams MCIS, Polytechnique Montréal

Thursday, March 3, 16

slide-2
SLIDE 2

Life Cycle of a Bug

Bug Introducing Change Bug Report Fix Change

Thursday, March 3, 16

slide-3
SLIDE 3

Automatic Identification

  • f Bug Introducing Changes

Thursday, March 3, 16

slide-4
SLIDE 4

Granularity File Level, Package Level

Thursday, March 3, 16

slide-5
SLIDE 5

Granularity File Level, Package Level

File-Level

Thursday, March 3, 16

slide-6
SLIDE 6

Granularity File Level, Package Level

File-Level Package-Level

Thursday, March 3, 16

slide-7
SLIDE 7

Granularity File Level, Package Level

File-Level Package-Level

Thursday, March 3, 16

slide-8
SLIDE 8

Granularity File Level, Package Level

File-Level Package-Level

Thursday, March 3, 16

slide-9
SLIDE 9

Granularity: Change or Commit Level (JIT defect model)

Thursday, March 3, 16

slide-10
SLIDE 10

Granularity: Change or Commit Level (JIT defect model)

Clear and Early

Thursday, March 3, 16

slide-11
SLIDE 11

JIT Defect Model Based on Change Metrics

Process Metrics, like:

  • Number of modified subsystems

Product Metrics, like:

  • Lines of code in a file before the

change

Human Factor Metrics, like:

  • Developer Experience

JIT Prediction Model

Thursday, March 3, 16

slide-12
SLIDE 12

Software Development Collaborative Activity Quality of Discussions Human Factors

Thursday, March 3, 16

slide-13
SLIDE 13

Issue Discussions Review Discussion

Two Major Discussions

Thursday, March 3, 16

slide-14
SLIDE 14

Rigid, Limited adoption and, Limited efficiency

Traditional Meeting

Thursday, March 3, 16

slide-15
SLIDE 15

Regular and quick, With recorded discussions, Traceable

Thursday, March 3, 16

slide-16
SLIDE 16

Different Aspects of Discussions

Focus Sentimen Length

Time

✴ReviewTime ✴IssueTime ✴ReplyLag

✴Issue_Reporter

_Experience

✴Number_Of_

Patchsets

✴Reviewer_Experience

Thursday, March 3, 16

slide-17
SLIDE 17

Sentiment

Sentistrength

Human Human Language Texts withTheir Sentiment Scores Comments Cleaning Comments Extracting Sentiments

Extracting Sentiments

Thursday, March 3, 16

slide-18
SLIDE 18

Different Dimensions

  • f Metrics for JIT prediction

Review

Issue

Change

Thursday, March 3, 16

slide-19
SLIDE 19

Different Dimensions

  • f Metrics for JIT prediction

Review

Issue

Change

Thursday, March 3, 16

slide-20
SLIDE 20

Linking Reviews and Issues

Launchpad

Gerrit

Thursday, March 3, 16

slide-21
SLIDE 21

Linking Reviews and Issues

Launchpad

Gerrit

Thursday, March 3, 16

slide-22
SLIDE 22

Studied Projects

(with top most number of linkage between issues and reviews)

Cinder, Devstack, Glance, Heat, Keystone, Neutron, Nova, Swift, Tempest, Openstack-Manuals Cdt, Jgit, Egit, Linuxtools, Scout.rt

Thursday, March 3, 16

slide-23
SLIDE 23

Research Questions

RQ1: How well can issue discussion metrics explain defect-introducing changes? RQ2: How well can review discussion metrics explain defect-introducing changes? RQ3: How well can issue and review discussion metrics can explain defect- introducing changes?

Thursday, March 3, 16

slide-24
SLIDE 24

General Approach

General Logistics Model

ROC Curve and Area Under Curve(AUC)

Thursday, March 3, 16

slide-25
SLIDE 25

Research Questions

RQ1: How well can issue discussion metrics explain defect-introducing changes? RQ2: How well can review discussion metrics explain defect-introducing changes? RQ3: How well can issue and review discussion metrics can explain defect- introducing changes?

Thursday, March 3, 16

slide-26
SLIDE 26

RQ1:Five of the projects see an

improvement of 3% up to 10%

57 59.5 62 64.5 67

Glance Tempest

Enhanced Baseline

17.5 35 52.5 70

Heat Cinder Jgit

Enhanced Baseline

Precision Recall

52 54.25 56.5 58.75 61

Cinder Glance Heat Tempest

Enhanced Baseline

AUC

Thursday, March 3, 16

slide-27
SLIDE 27

RQ1: Top Most Important Factors (have negative effect)

Change Related Metrics: Type, LA, Number of Developers One Focus Metric: Issue Commenter Experience

Thursday, March 3, 16

slide-28
SLIDE 28

Research Questions

RQ1: How well can issue discussion metrics explain defect-introducing changes? RQ2: How well can review discussion metrics explain defect-introducing changes? RQ3: How well can issue and review discussion metrics can explain defect- introducing changes?

Thursday, March 3, 16

slide-29
SLIDE 29

RQ2: 9 projects improve their precision and/or recall by 3% to 8%.

20 40 60 80

neutron ...-manuals jgit egit

Enhanced Baseline

20 40 60 80

d e v s t a c k h e a t k e y s t

  • n

e n e u t r

  • n

. . .

  • m

a n u a l s t e m p e s t c d t e g i t j g i t

Enhanced Baseline

20 40 60 80

Camel neutron...-manuals jgit egit

Enhanced Baseline

Precision Recall AUC

Thursday, March 3, 16

slide-30
SLIDE 30

RQ2: Top Most Important Factors

From Change Related Metrics: Type, LA/LD, AGE From Focus Metric: Review Commenter Experience From Time Metric: Review Time

Thursday, March 3, 16

slide-31
SLIDE 31

Research Questions

RQ1: How well can issue discussion metrics explain defect-introducing changes? RQ2: How well can review discussion metrics explain defect-introducing changes? RQ3: How well can issue and review discussion metrics can explain defect-introducing changes?

Thursday, March 3, 16

slide-32
SLIDE 32

RQ3: 8 of the projects improve both precision and recall by 3% up to 17%.

cinder glance heat neutron ...-manuals tempest egit jgit scout.rt

Enhanced Baseline

20 40 60 80

cinder glance heat neutron nova ...-manuals tempest egit scout.rt

Enhanced Baseline

AUC Recall

22.5 45 67.5 90

cinder glance heat neutron nova ...-manuals swift tempest egit scout.rt

Enhanced Baseline

Precision

Thursday, March 3, 16

slide-33
SLIDE 33

RQ3: Top Most Important Factors

From Time Metric:

  • Review Time
  • Issue Discussion Lag

From Change Related Metrics:

  • Type, AGE
  • LA

Thursday, March 3, 16

slide-34
SLIDE 34

Thursday, March 3, 16

slide-35
SLIDE 35

Conclusion

  • An enhanced defect prediction model was

built considering review discussion metrics.

  • The proposed model was validated on five
  • pen source projects.
  • There is a connection between defect-

prone commits and review discussion metrics.

  • Review discussion metrics complement

change-based metrics.

Thursday, March 3, 16

slide-36
SLIDE 36

Enhanced Defect Prediction Model Metrics

Process Metrics, like:

  • Number of modified subsystems (NS)
  • Change is a defect fix? (FIX)

Product Metrics, like:

  • Lines of code in a file before the change

(LT)

Human Factor Metrics, like:

  • Developer Experience

(EXP)

Review Discussion Metrics, like:

Sentiment of Comments

Thursday, March 3, 16

slide-37
SLIDE 37

Thursday, March 3, 16

slide-38
SLIDE 38

Thursday, March 3, 16

slide-39
SLIDE 39

Thursday, March 3, 16

slide-40
SLIDE 40

Review

Thursday, March 3, 16

slide-41
SLIDE 41

Review

Thursday, March 3, 16

slide-42
SLIDE 42

Review

Issue

Thursday, March 3, 16

slide-43
SLIDE 43

Review

Issue

Change

Thursday, March 3, 16

slide-44
SLIDE 44

Review

Issue

Change

Thursday, March 3, 16

slide-45
SLIDE 45

Pattern 3 Pattern 2 Pattern 1

Issue Time Review Time

Thursday, March 3, 16

slide-46
SLIDE 46

Thursday, March 3, 16