When testing meets Code Review: Why and How Developers Review Tests - - PowerPoint PPT Presentation

▶
when testing meets code review why and how developers
SMART_READER_LITE
LIVE PREVIEW

When testing meets Code Review: Why and How Developers Review Tests - - PowerPoint PPT Presentation

Dec 13, 2022 538 likes •970 views

When testing meets Code Review: Why and How Developers Review Tests Davide Spadini, Mauricio Aniche, Margaret-Anne Storey, Magiel Bruntink, Alberto Bacchelli When testing meets Code Review: Why and How Developers Review Tests Davide Spadini ,

slide-1
SLIDE 1

When testing meets Code Review: Why and How Developers Review Tests

Davide Spadini, Mauricio Aniche, Margaret-Anne Storey, Magiel Bruntink, Alberto Bacchelli

slide-2
SLIDE 2

When testing meets Code Review: Why and How Developers Review Tests

Davide Spadini, Mauricio Aniche,

Margaret-Anne Storey, Magiel Bruntink, Alberto Bacchelli

@DavideSpadini ishepard

slide-3
SLIDE 3
slide-4
SLIDE 4

Code review

slide-5
SLIDE 5

Research Questions

RQ1: How rigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests?

We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack

12 interviews

slide-6
SLIDE 6

Research Questions

RQ1: How rigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests?

We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack

12 interviews

slide-7
SLIDE 7

RQ1: How rigorously is test code reviewed? — Method

# of

  • prod. files

# of test files # of code reviews # of reviewers # of comments Eclipse

215k 19k 60k 1k 95k

Openstack

75k 48k 199k 9k 894k

Qt

158k 8k 114k 1k 19k

Total

450k 77k 374k 12k 1,010k

slide-8
SLIDE 8

RQ1: How rigorously is test code reviewed? — Results

Together

A.java ATest.java

slide-9
SLIDE 9

RQ1: How rigorously is test code reviewed? — Results

Together

A.java ATest.java

slide-10
SLIDE 10

RQ1: How rigorously is test code reviewed? — Results

Together

A.java ATest.java

Production alone

A.java ATest.java

slide-11
SLIDE 11

RQ1: How rigorously is test code reviewed? — Results

Together

A.java ATest.java

Production alone

A.java

slide-12
SLIDE 12

RQ1: How rigorously is test code reviewed? — Results

Together

A.java ATest.java

Test alone

A.java ATest.java

Production alone

A.java

slide-13
SLIDE 13

RQ1: How rigorously is test code reviewed? — Results

Together

A.java ATest.java

Production alone

A.java

Test alone

ATest.java

slide-14
SLIDE 14

RQ1: How rigorously is test code reviewed? — Results

Together

A.java ATest.java

Production alone

A.java

Test alone

ATest.java

15 30 45 60 75

% of files with review comments

slide-15
SLIDE 15

RQ1: How rigorously is test code reviewed? — Results

Together

A.java ATest.java

Production alone

A.java

Test alone

ATest.java

15 30 45 60 75

% of files with review comments

slide-16
SLIDE 16

RQ1: How rigorously is test code reviewed? — Results

Together

A.java ATest.java

Production alone

A.java

Test alone

ATest.java

15 30 45 60 75

% of files with review comments

slide-17
SLIDE 17

RQ1: How rigorously is test code reviewed? — Results

Together

A.java ATest.java

Production alone

A.java

Test alone

ATest.java

15 30 45 60 75

% of files with review comments

When together, production code is 1.9 more likely do be discussed

slide-18
SLIDE 18

RQ1: How rigorously is test code reviewed? — Results

Together

A.java ATest.java

Production alone

A.java

Test alone

ATest.java

15 30 45 60 75

% of files with review comments

When together, production code is 1.9 more likely do be discussed When taken separately, test code is 1.16 more likely to be discussed

slide-19
SLIDE 19

Avg # of comments per file Avg # of reviewers Avg length of comments Together Production Test

3.00 1.27 5.49 19.09 15.32

Production alone

1.64 3.95 18.13

Test alone

2.30 5.15 17.01

RQ1: How rigorously is test code reviewed? — Results

slide-20
SLIDE 20

# of comments, avg length, avg # reviewers

  • Prod. files are

2 times more likely to be discussed than test files Test files are discussed more when alone

RQ1: How rigorously is test code reviewed? — Summary

slide-21
SLIDE 21

Research Questions

RQ1: How rigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests?

We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack

12 interviews

slide-22
SLIDE 22

RQ2: What do reviewers discuss in test code reviews? — Method

> 1,000,000 comments 600 comments 1st round: 6 categories* 2nd round: for each category, more fine-grained

*Bacchelli & Bird. Expectations, Outcomes, and Challenges of Modern Code Review. ICSE 2013

slide-23
SLIDE 23

RQ2: What do reviewers discuss in test code reviews? — Results

0% 10% 20% 30% 40%

production reviews (Bacchelli & Bird, ICSE 2013) test reviews (this study)

Code improvements Understanding Social communication Defects Knowledge transfer

slide-24
SLIDE 24

10 20 30 40 Testing practices Code styling Un/Tested paths Better naming Unnecessary code Assertion handling

Code improvements

RQ2: What do reviewers discuss in test code reviews? — Results

slide-25
SLIDE 25

RQ2: What do reviewers discuss in test code reviews? — Results

0% 10% 20% 30% 40%

production reviews (Bacchelli & Bird, ICSE 2013) test reviews (this study)

Understanding Social communication Defects Knowledge transfer Code improvements

slide-26
SLIDE 26

10 20 30 40 50 Severe issues Less severe issues Wrong assertions

Defects

RQ2: What do reviewers discuss in test code reviews? — Results

“You need to instantiate LttngKernelTrace here

  • therwise you will get a

class cast exception” — Severe issue “This isnt being used, hence pep8 error.” — Less severe issue

slide-27
SLIDE 27

RQ2: What do reviewers discuss in test code reviews? — Results

0% 10% 20% 30% 40%

production reviews (Bacchelli & Bird, ICSE 2013) test reviews (this study)

Code improvements Understanding Social communication Knowledge transfer Defects

slide-28
SLIDE 28

15 30 45 60 Link to external documentation How-to

Knowledge transfer

RQ2: What do reviewers discuss in test code reviews? — Results

slide-29
SLIDE 29

RQ2: What do reviewers discuss in test code reviews? — Summary

Testing practices, tested paths and assertions Defects are not among the most discussed topics Developers often comment with how-to or ext. documentation

slide-30
SLIDE 30

Research Questions

RQ1: How rigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests?

We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack

12 interviews

slide-31
SLIDE 31

RQ3: Which practices do reviewers follow for test files? — Method

Interviews!

slide-32
SLIDE 32

1 Openstack 1 Qt 1 Eclipse 1 Microsoft 1 Ericsson 2 Alura 5 OSS

12 interviews

RQ3: Which practices do reviewers follow for test files? — Method

slide-33
SLIDE 33

RQ3: Which practices do reviewers follow for test files? — Summary

Checking out the change and run the tests Understanding if all paths are covered

Test Driven Review

slide-34
SLIDE 34

Research questions

RQ1: How rigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests?

We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack

12 interviews

slide-35
SLIDE 35

Management policies prioritize strongly production code Novice developers do not know the effect of poor testing

Lack of navigation support and in- depth code coverage info

Reviewing tests requires context on both tests and production

RQ4: What problems and challenges do developers face when reviewing tests?

slide-36
SLIDE 36

Management policies prioritize strongly production code Novice developers do not know the effect of poor testing

Lack of navigation support and in- depth code coverage info

Reviewing tests requires context on both tests and production

RQ4: What problems and challenges do developers face when reviewing tests?

Provide the context for reviewing test Plan enough time for test review Educate on test reviewing and writing Detailed code coverage information

Implications and recommendations

slide-37
SLIDE 37
slide-38
SLIDE 38

Code Review

  • Developers have limited time to do

code review

  • Should test files be reviewed?
  • If production files are more prone

to contain bugs…reviewer should focus on production!

slide-39
SLIDE 39

Should test files be reviewed?

  • We conducted a preliminary investigation to see whether test and production

code files are equally associated with defects

  • We built a statistical model and used as an explanatory variable “being a

test”

slide-40
SLIDE 40

Should test files be reviewed?

  • 3 OSS projects: Qt, Eclipse and Openstack
  • Dependent variable: post-release defects
  • Explanatory variables: metrics well related to defect proneness: product metrics,

process metrics and human factors.

  • Plus, our variable: isTest
slide-41
SLIDE 41

Metrics in order of importance

Attribute Average merit Average Rank

Churn 0.753 1 Author ownership 0.599 2.2 ± 0.4 Cumulative churn 0.588 2.8 ± 0.4 Total authors 0.521 4 Major authors 0.506 5 Size 0.411 6 Prior defects 0.293 7 Minor authors 0.149 8 Is test 0.085 9

Should test files be reviewed?

  • Yes!
  • The decision to review a file

should not be based on whether the file contains production or test code, as this has no association with defects.