SLIDE 1
A Conceptual Replication
- f
david widder Michael hilton bogdan vasilescu christian kastner
..
A Conceptual Replication
- f
A Conceptual Replication of Continuous Integration Pain Points A - - PowerPoint PPT Presentation
A Conceptual Replication of Continuous Integration Pain Points A - - PowerPoint PPT Presentation
A Conceptual Replication of Continuous Integration Pain Points A Conceptual Replication of Continuous Integration Pain Points david Michael christian bogdan .. widder hilton kastner vasilescu What Will I
SLIDE 2
SLIDE 3
2
What Will I talk about?
SLIDE 4
- 1. What is Continuous integration?
Why do people use it?
2
What Will I talk about?
SLIDE 5
- 1. What is Continuous integration?
Why do people use it?
- 2. Why do a replication?
2
What Will I talk about?
SLIDE 6
- 1. What is Continuous integration?
Why do people use it?
- 2. Why do a replication?
- 3. Our Mixed Method
2
What Will I talk about?
SLIDE 7
- 1. What is Continuous integration?
Why do people use it?
- 2. Why do a replication?
- 3. Our Mixed Method
- 4. Highlights of Findings:
What replicates + what doesn’t?
2
What Will I talk about?
SLIDE 8
What is continuous integration?
3
SLIDE 9
What is continuous integration?
3
SLIDE 10
What is continuous integration?
3
SLIDE 11
What is continuous integration?
3
SLIDE 12
What is continuous integration?
3
SLIDE 13
What is continuous integration?
3
SLIDE 14
What is continuous integration?
3
SLIDE 15
What is continuous integration?
3
SLIDE 16
CI is a Best practice
4
SLIDE 17
6 Reasons to Use CI Why we need CI What is CI and why use it? 7 Reasons you should use CI Why CI is important what is CI and why do you need it?
CI is a Best practice
4
SLIDE 18
5
Lots of research Showing CI Benefits
SLIDE 19
Fewer pull requests rejected
Helps catch bugs faster
Encourages people to write tests
release twice as
- ften
Higher pull request Throughput
5
Lots of research Showing CI Benefits
SLIDE 20
Also, lots of research showing CI pain points
6
SLIDE 21
Also, lots of research showing CI pain points
Hard to
configure
long builds insecure
too many
- ptions
unsupported features
6
SLIDE 22
7
We believe it is time to review and replicate CI pain points
SLIDE 23
8
Why review and replicate CI pain points?
SLIDE 24
- evaluate & synthesize
past research
8
Why review and replicate CI pain points?
SLIDE 25
- evaluate & synthesize
past research
- provide replicated
guidance to practitioners
8
Why review and replicate CI pain points?
SLIDE 26
- evaluate & synthesize
past research
- provide replicated
guidance to practitioners
- focus future research
- n areas of uncertainty
8
Why review and replicate CI pain points?
SLIDE 27
9
What’s a conceptual replication?
current CI users
Switchers & Abandoners
SLIDE 28
9
What’s a conceptual replication?
?
current CI users
Switchers & Abandoners
SLIDE 29
why study leavers instead of current users?
10
SLIDE 30
why study leavers instead of current users?
10
SLIDE 31
11
logistic regressions
Survey &
interviews
?
lit review
Pain points Replicated (or not)
TravisCI Leavers OSS
Current CI Users Industry & OSS
Triangu lation
Pain Points
SLIDE 32
11
logistic regressions
Survey &
interviews
?
lit review
Pain points Replicated (or not)
TravisCI Leavers OSS
Current CI Users Industry & OSS
Triangu lation
Pain Points
SLIDE 33
11
logistic regressions
Survey &
interviews
?
lit review
Pain points Replicated (or not)
TravisCI Leavers OSS
Current CI Users Industry & OSS
Triangu lation
Pain Points
SLIDE 34
11
logistic regressions
Survey &
interviews
?
lit review
Pain points Replicated (or not)
TravisCI Leavers OSS
Current CI Users Industry & OSS
Triangu lation
Pain Points
SLIDE 35
11
logistic regressions
Survey &
interviews
?
lit review
Pain points Replicated (or not)
TravisCI Leavers OSS
Current CI Users Industry & OSS
Triangu lation
Pain Points
SLIDE 36
11
logistic regressions
Survey &
interviews
?
lit review
Pain points Replicated (or not)
TravisCI Leavers OSS
Current CI Users Industry & OSS
Triangu lation
Pain Points
SLIDE 37
12
lit review
37 papers
35 Pain Points
SLIDE 38
13
132 Survey responses 12 interviews
?
“Why did your project stop using Travis CI, and what has the CI situation been like since then?”
SLIDE 39
14
SLIDE 40
15
132 Survey responses + 12 interviews
?
SLIDE 41
15
132 Survey responses + 12 interviews
SLIDE 42
16
People who leave CI vs Those who don’t
Why logistic regression?
SLIDE 43
Abandonment vs Switching
CI
{ .. }
17
SLIDE 44
Abandonment vs Switching
CI
{ .. }
17
SLIDE 45
18
2 logistic regressions, 6,239 Repos
Abandonment vs Switching
CI
{ .. }
SLIDE 46
18
2 logistic regressions, 6,239 Repos
Abandonment vs Switching
CI
{ .. }
SLIDE 47
?
Triangulated Results
19
SLIDE 48
20
Unsupported technologies
C#
SLIDE 49
21
Unsupported technologies
C#
SLIDE 50
21
Unsupported technologies
“I switched to Wercker, a container based CI pipeline, which means it can execute any scripts with much more flexibility.” (P89)
C#
SLIDE 51
21
Unsupported technologies
“I switched to Wercker, a container based CI pipeline, which means it can execute any scripts with much more flexibility.” (P89)
C#
needing docker: 9.5x increase in switching, 5.25x increase in abandoning
SLIDE 52
21
Unsupported technologies
“I switched to Wercker, a container based CI pipeline, which means it can execute any scripts with much more flexibility.” (P89)
C#
needing docker: 9.5x increase in switching, 5.25x increase in abandoning
Lacking Language support: 1.5x increase in switching and abandoning
SLIDE 53
22
Unsupported technologies
C#
SLIDE 54
22
Needing docker affects new TravisCI users only
Unsupported technologies
C#
SLIDE 55
22
Needing docker affects new TravisCI users only when can people hack through technology support challenges, and when are the not able to?
Unsupported technologies
C#
SLIDE 56
23
Lack
- f Tests
SLIDE 57
24
Lack
- f Tests
SLIDE 58
24
Lack
- f Tests
“The goal [of adopting Travis] was to ‘force’ myself to add some real tests (to have green Travis badge again!) but this failed so far :)” (P111)
SLIDE 59
24
Lack
- f Tests
1% increase in tests = 16% lower chance of abandoning (no effect on switching)
“The goal [of adopting Travis] was to ‘force’ myself to add some real tests (to have green Travis badge again!) but this failed so far :)” (P111)
SLIDE 60
25
CI consistency
SLIDE 61
26
CI consistency
SLIDE 62
26
“I had some open source projects running in TravisCI and some in CircleCI. I just wanted to consolidate the project to one place and I’m sorry to say that at that time TravisCI lost the battle.” (P57)
CI consistency
SLIDE 63
26
“I had some open source projects running in TravisCI and some in CircleCI. I just wanted to consolidate the project to one place and I’m sorry to say that at that time TravisCI lost the battle.” (P57)
Exposure to leavers:
~1.5x increase in chances
- f switching & abandoning
CI consistency
SLIDE 64
27
CI consistency
SLIDE 65
27
people’s past experience is very important in determining behavior on future projects!
CI consistency
SLIDE 66
27
people’s past experience is very important in determining behavior on future projects! but Not observed in past literature, so needs replication.
CI consistency
SLIDE 67
Single method or conflicting results
?
?
28
SLIDE 68
29
long build times
SLIDE 69
30
long build times
SLIDE 70
30
long build times
“We stopped using Travis CI because it was too slow for us.” (P125)
SLIDE 71
30
long build times
“We stopped using Travis CI because it was too slow for us.” (P125)
longer build times associated with a decreased chance of switching and abandoning.
SLIDE 72
30
long build times
“We stopped using Travis CI because it was too slow for us.” (P125)
longer build times associated with a decreased chance of switching and abandoning. Presence of very long builds associated with increased risk of abandoning.
SLIDE 73
31
long build times
SLIDE 74
31
So, When do long builds stop providing value, and start becoming annoying?
long build times
SLIDE 75
32
Full results in the paper
SLIDE 76
davidthewid
2 logistic regressions
Surveys, Interviews
?
Triangulation
lit review
Pain Points
SLIDE 77
we were unable to cleanly replicate some pain points, they need further study to validate their existence
davidthewid
2 logistic regressions
Surveys, Interviews
?
Triangulation
lit review
Pain Points
SLIDE 78
we were unable to cleanly replicate some pain points, they need further study to validate their existence we were able to cleanly replicate some pain points, now we can focus on solutions
davidthewid
2 logistic regressions
Surveys, Interviews
?
Triangulation
lit review
Pain Points
SLIDE 79
we were unable to cleanly replicate some pain points, they need further study to validate their existence we were able to cleanly replicate some pain points, now we can focus on solutions future work should focus more than just TravisCI, and our commit status context method can help
davidthewid
2 logistic regressions
Surveys, Interviews
?
Triangulation
lit review
Pain Points
SLIDE 80
we were unable to cleanly replicate some pain points, they need further study to validate their existence we were able to cleanly replicate some pain points, now we can focus on solutions future work should focus more than just TravisCI, and our commit status context method can help CI consistency: social ties impact a project’s tooling choices
davidthewid
2 logistic regressions
Surveys, Interviews
?
Triangulation
lit review
Pain Points