Moritz Beller Delft University of Technology @Inventitech Andy - - PowerPoint PPT Presentation

▶
moritz beller
SMART_READER_LITE
LIVE PREVIEW

Moritz Beller Delft University of Technology @Inventitech Andy - - PowerPoint PPT Presentation

Sep 09, 2023 198 likes •1.51k views

Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix? Moritz Beller Delft University of Technology @Inventitech Andy Zaidman, Georgios Gousios, Alberto Bacchelli, Elmar Jrgens, Radjino Bohlanath, Shane McIntosh Modern Code

slide-1
SLIDE 1

Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?

Moritz Beller

Delft University of Technology @Inventitech Andy Zaidman, Georgios Gousios, Alberto Bacchelli, Elmar Jürgens, Radjino Bohlanath, Shane McIntosh

slide-2
SLIDE 2

Modern Code Review Workflow

slide-3
SLIDE 3

Modern Code Review Workflow Old

slide-4
SLIDE 4

Modern Code Review Workflow

slide-5
SLIDE 5

Modern Code Review Workflow

slide-6
SLIDE 6

Modern Code Review Workflow

slide-7
SLIDE 7

Modern Code Review Workflow

slide-8
SLIDE 8

Modern Code Review Workflow

slide-9
SLIDE 9

Modern Code Review Workflow

slide-10
SLIDE 10

Modern Code Review Workflow

slide-11
SLIDE 11

Modern Code Review Workflow

slide-12
SLIDE 12

Modern Code Review Workflow

slide-13
SLIDE 13

Modern Code Review Workflow

slide-14
SLIDE 14

Modern Code Review Workflow +

slide-15
SLIDE 15
slide-16
SLIDE 16

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread();

slide-17
SLIDE 17

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread();

slide-18
SLIDE 18

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread();

slide-19
SLIDE 19

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ public Runnable watchDogUiThread = new StartupUIThread();

slide-20
SLIDE 20

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ public Runnable watchDogUiThread = new StartupUIThread();

slide-21
SLIDE 21

Review-triggered Change

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ public Runnable watchDogUiThread = new StartupUIThread();

slide-22
SLIDE 22

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread();

slide-23
SLIDE 23

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread();

slide-24
SLIDE 24

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ private Runnable watchDogUiThread = new StartupUIThread();

slide-25
SLIDE 25

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ private Runnable watchDogUiThread = new StartupUIThread();

slide-26
SLIDE 26

Self-motivated + Review-triggered Change

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ private Runnable watchDogUiThread = new StartupUIThread();

slide-27
SLIDE 27

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread();

slide-28
SLIDE 28

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread();

slide-29
SLIDE 29

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** * The ui thread for WatchDog registration. From * this thread, the WatchDog program is started. */ public Runnable watchDog = new StartupUIThread();

slide-30
SLIDE 30

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** * The ui thread for WatchDog registration. From * this thread, the WatchDog program is started. */ public Runnable watchDog = new StartupUIThread();

slide-31
SLIDE 31

Self-motivated Change

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** * The ui thread for WatchDog registration. From * this thread, the WatchDog program is started. */ public Runnable watchDog = new StartupUIThread();

slide-32
SLIDE 32

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread();

slide-33
SLIDE 33

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread();

slide-34
SLIDE 34

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread();

slide-35
SLIDE 35

No Change

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread();

slide-36
SLIDE 36

No Change

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread();

slide-37
SLIDE 37

No Change

/** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ // TODO (GG) Provide a more descriptive name public Runnable watchDog = new StartupUIThread(); /** The ui thread for WatchDog registration. */ public Runnable watchDog = new StartupUIThread();

slide-38
SLIDE 38

Does it matter?

slide-39
SLIDE 39

>1,400 changes >245 tasks

slide-40
SLIDE 40

Motivation for Change

slide-41
SLIDE 41

Motivation for Change

slide-42
SLIDE 42

78-90%

Motivation for Change

slide-43
SLIDE 43

78-90%

Motivation for Change

22-10%

slide-44
SLIDE 44

78-90%

Motivation for Change

22-10%

slide-45
SLIDE 45

Motivation for No Change

slide-46
SLIDE 46

Motivation for No Change

slide-47
SLIDE 47

Motivation for No Change

7-35%

slide-48
SLIDE 48

Motivation for No Change

7-35%

slide-49
SLIDE 49

Type of Change

slide-50
SLIDE 50

Doc Visual Structure Resource Check Interface Logic Larger Support

ConQAT Gromacs

10% 20% 30% 40%

slide-51
SLIDE 51

75

Doc Visual Structure Resource Check Interface Logic Larger Support

ConQAT Gromacs

10% 20% 30% 40%

slide-52
SLIDE 52

75 :

Doc Visual Structure Resource Check Interface Logic Larger Support

ConQAT Gromacs

10% 20% 30% 40%

slide-53
SLIDE 53

75 : 25

Doc Visual Structure Resource Check Interface Logic Larger Support

ConQAT Gromacs

10% 20% 30% 40%

slide-54
SLIDE 54
slide-55
SLIDE 55

75 : 25

slide-56
SLIDE 56

What Influences Code Review?

slide-57
SLIDE 57

What Influences Code Review?

slide-58
SLIDE 58

What Influences Code Review?

slide-59
SLIDE 59

What Influences Code Review? > 2,800 changes > 970 tasks

slide-60
SLIDE 60

What Influences Code Review? > 2,800 changes > 970 tasks

slide-61
SLIDE 61

What Influences Code Review? > 2,800 changes > 970 tasks

slide-62
SLIDE 62

What Influences Code Review? > 2,800 changes > 970 tasks

slide-63
SLIDE 63

What Influences Code Review? > 2,800 changes > 970 tasks

slide-64
SLIDE 64

What Influences Code Review? > 2,800 changes > 970 tasks

slide-65
SLIDE 65

What Influences Code Review? > 2,800 changes > 970 tasks

slide-66
SLIDE 66

What Influences Code Review? > 2,800 changes > 970 tasks

slide-67
SLIDE 67

“Continuous” Integration Then (2014) +

slide-68
SLIDE 68

“Continuous” Integration Then (2014) +

slide-69
SLIDE 69

“Continuous” Integration Then (2014) +

slide-70
SLIDE 70

“Continuous” Integration Then (2014) +

slide-71
SLIDE 71

Continuous Integration Now (2017)

slide-72
SLIDE 72

Continuous Integration Now (2017)

CI Services

slide-73
SLIDE 73

Continuous Integration Now (2017)

a) Static Analysis CI Services

slide-74
SLIDE 74

Continuous Integration Now (2017)

a) Static Analysis b) Dynamic Analysis CI Services

slide-75
SLIDE 75

Continuous Integration Now (2017) +

a) Static Analysis b) Dynamic Analysis CI Services

slide-76
SLIDE 76

a) How Prevalent Is Static Analysis?

slide-77
SLIDE 77

1 2 2

a) How Prevalent Is Static Analysis?

slide-78
SLIDE 78

1 2 2

a) How Prevalent Is Static Analysis?

slide-79
SLIDE 79

1 2 2

a) How Prevalent Is Static Analysis?

slide-80
SLIDE 80

1 2 2 1 2 2

a) How Prevalent Is Static Analysis?

slide-81
SLIDE 81

1 2 2 1 2 2

a) How Prevalent Is Static Analysis?

slide-82
SLIDE 82

1 2 2 36 1 2 2

a) How Prevalent Is Static Analysis?

slide-83
SLIDE 83

1 2 2 36 1 2 2

a) How Prevalent Is Static Analysis?

slide-84
SLIDE 84

1 2 2 36 1 2 2

a) How Prevalent Is Static Analysis?

slide-85
SLIDE 85

a) How Prevalent Is Static Analysis?

slide-86
SLIDE 86

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23%

  • 36

77% 36% 36%

a) How Prevalent Is Static Analysis?

slide-87
SLIDE 87

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23%

  • 36

77% 36% 36%

a) How Prevalent Is Static Analysis?

slide-88
SLIDE 88

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23%

  • 36

77% 36% 36%

a) How Prevalent Is Static Analysis?

slide-89
SLIDE 89

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23%

  • 36

77% 36% 36%

a) How Prevalent Is Static Analysis?

slide-90
SLIDE 90

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23%

  • 36

77% 36% 36%

a) How Prevalent Is Static Analysis?

slide-91
SLIDE 91

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23%

  • 36

77% 36% 36% Moritz Beller, Radjino Bholanath, Shane McIntosh, Andy Zaidman: Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software, SANER, Osaka (Japan), 2016

a) How Prevalent Is Static Analysis?

slide-92
SLIDE 92

TODO: Add background with Sun

b) How Prevalent Is Dynamic Analysis?

slide-93
SLIDE 93

TODO: Add background with Sun

CI is the best practice in which developers not only integrate their work into a shared mainline frequently, but also verify the quality continuously through testing.

b) How Prevalent Is Dynamic Analysis?

slide-94
SLIDE 94

TODO: Add background with Sun

CI is the best practice in which developers not only integrate their work into a shared mainline frequently, but also verify the quality continuously through testing.

It is known.

b) How Prevalent Is Dynamic Analysis?

slide-95
SLIDE 95
slide-96
SLIDE 96

<TL;DR>...</TL;DR>

slide-97
SLIDE 97

<TL;DR>...</TL;DR>

  • T E S T S
  • Running nl.tudelft.watchdog.ClientVersionCheckerTest

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

slide-98
SLIDE 98

<TL;DR>...</TL;DR>

  • T E S T S
  • Running nl.tudelft.watchdog.ClientVersionCheckerTest

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>

slide-99
SLIDE 99

<TL;DR>...</TL;DR>

  • T E S T S
  • Running nl.tudelft.watchdog.ClientVersionCheckerTest

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>

slide-100
SLIDE 100

<TL;DR>...</TL;DR>

  • T E S T S
  • Running nl.tudelft.watchdog.ClientVersionCheckerTest

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>

slide-101
SLIDE 101

<TL;DR>...</TL;DR>

  • T E S T S
  • Running nl.tudelft.watchdog.ClientVersionCheckerTest

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>

slide-102
SLIDE 102

<TL;DR>...</TL;DR>

  • T E S T S
  • Running nl.tudelft.watchdog.ClientVersionCheckerTest

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>

  • Parsing Log files for Java: Maven, Gradle, Ant
slide-103
SLIDE 103

<TL;DR>...</TL;DR>

  • T E S T S
  • Running nl.tudelft.watchdog.ClientVersionCheckerTest

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>

  • Parsing Log files for Java: Maven, Gradle, Ant
slide-104
SLIDE 104

<TL;DR>...</TL;DR>

  • T E S T S
  • Running nl.tudelft.watchdog.ClientVersionCheckerTest

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>

  • Parsing Log files for Java: Maven, Gradle, Ant
slide-105
SLIDE 105

<TL;DR>...</TL;DR>

  • T E S T S
  • Running nl.tudelft.watchdog.ClientVersionCheckerTest

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>

  • Parsing Log files for Java: Maven, Gradle, Ant
  • Parsing Log files for Ruby: bundler and rake
slide-106
SLIDE 106

<TL;DR>...</TL;DR>

  • T E S T S
  • Running nl.tudelft.watchdog.ClientVersionCheckerTest

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 <TL;DR>...</TL;DR>

  • Parsing Log files for Java: Maven, Gradle, Ant
  • Parsing Log files for Ruby: bundler and rake
slide-107
SLIDE 107

b) How Prevalent Is Dynamic Analysis?

slide-108
SLIDE 108
  • 69% of Java and 87.5% of the Ruby projects:

test runs b) How Prevalent Is Dynamic Analysis?

slide-109
SLIDE 109
  • 69% of Java and 87.5% of the Ruby projects:

test runs

  • Overall, 81% of the projects:

test runs b) How Prevalent Is Dynamic Analysis?

slide-110
SLIDE 110
  • 69% of Java and 87.5% of the Ruby projects:

test runs

  • Overall, 81% of the projects:

test runs

  • 96% of builds:

test runs b) How Prevalent Is Dynamic Analysis?

slide-111
SLIDE 111

b) How Prevalent Is Dynamic Analysis?

slide-112
SLIDE 112

b) How Prevalent Is Dynamic Analysis?

slide-113
SLIDE 113

b) How Prevalent Is Dynamic Analysis?

slide-114
SLIDE 114

Testing is the single most important reason for broken builds, more prevalent than compile errors, missing dependencies, build cancellations and provisioning problems together.

b) Testing Is Central To CI

slide-115
SLIDE 115

Testing is the single most important reason for broken builds, more prevalent than compile errors, missing dependencies, build cancellations and provisioning problems together.

b) Testing Is Central To CI

Moritz Beller, Georgios Gousios, Andy Zaidman: Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub, MSR, Buenos Aires (Argentina), 2017

slide-116
SLIDE 116

What Is TravisTorrent?

slide-117
SLIDE 117

What Is TravisTorrent?

M S R M i n i n g C h a l l e n g e 2 1 7 Public Data Set

slide-118
SLIDE 118

What Is TravisTorrent? Free* SQL-like cloud query interface on Google BQ

M S R M i n i n g C h a l l e n g e 2 1 7 Public Data Set

slide-119
SLIDE 119

What Is TravisTorrent?

.testroots.org

Free* SQL-like cloud query interface on Google BQ

M S R M i n i n g C h a l l e n g e 2 1 7 Public Data Set

slide-120
SLIDE 120

TODO: Add background with Sun

Research on Continuous Code Review Enabled By TravisTorrent

slide-121
SLIDE 121

TODO: Add background with Sun

  • git_branch
  • git_all_built_commits
  • ...

Research on Continuous Code Review Enabled By TravisTorrent

slide-122
SLIDE 122

TODO: Add background with Sun

  • git_branch
  • git_all_built_commits
  • ...
  • gh_project_name
  • gh_is_pr
  • gh_team_size
  • gh_diff_tests_added
  • gh_first_commit_...

Research on Continuous Code Review Enabled By TravisTorrent

slide-123
SLIDE 123

TODO: Add background with Sun

  • git_branch
  • git_all_built_commits
  • ...
  • gh_project_name
  • gh_is_pr
  • gh_team_size
  • gh_diff_tests_added
  • gh_first_commit_...
  • tr_prev_build
  • tr_status
  • tr_log_tests_ran
  • tr_log_tests_failed
  • ...

Research on Continuous Code Review Enabled By TravisTorrent

slide-124
SLIDE 124

TODO: Add background with Sun

  • git_branch
  • git_all_built_commits
  • ...
  • gh_project_name
  • gh_is_pr
  • gh_team_size
  • gh_diff_tests_added
  • gh_first_commit_...
  • tr_prev_build
  • tr_status
  • tr_log_tests_ran
  • tr_log_tests_failed
  • ...

6 2 c

  • l

u m n s

Research on Continuous Code Review Enabled By TravisTorrent

slide-125
SLIDE 125

TODO: Add background with Sun

  • git_branch
  • git_all_built_commits
  • ...
  • gh_project_name
  • gh_is_pr
  • gh_team_size
  • gh_diff_tests_added
  • gh_first_commit_...
  • tr_prev_build
  • tr_status
  • tr_log_tests_ran
  • tr_log_tests_failed
  • ...

6 2 c

  • l

u m n s

Research on Continuous Code Review Enabled By TravisTorrent

slide-126
SLIDE 126

TODO: Add background with Sun

  • git_branch
  • git_all_built_commits
  • ...
  • gh_project_name
  • gh_is_pr
  • gh_team_size
  • gh_diff_tests_added
  • gh_first_commit_...
  • tr_prev_build
  • tr_status
  • tr_log_tests_ran
  • tr_log_tests_failed
  • ...

6 2 c

  • l

u m n s

Research on Continuous Code Review Enabled By TravisTorrent

slide-127
SLIDE 127

@Inventitech Moritz Beller, TU Delft

Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?

slide-128
SLIDE 128

@Inventitech Moritz Beller, TU Delft

Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?

slide-129
SLIDE 129

@Inventitech Moritz Beller, TU Delft

Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?

slide-130
SLIDE 130

@Inventitech Moritz Beller, TU Delft

Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?