An Exploratory Study Into the Prevalence of Botched Code - - PowerPoint PPT Presentation

▶
an exploratory study into the prevalence of botched code
SMART_READER_LITE
LIVE PREVIEW

An Exploratory Study Into the Prevalence of Botched Code - - PowerPoint PPT Presentation

Nov 18, 2023 376 likes •833 views

An Exploratory Study Into the Prevalence of Botched Code Integrations @wardmuylaert @oniroi Ward Muylaert and Coen De Roover Software Languages Lab Vrije Universiteit Brussel Brussels, Belgium Textual Conflicts function foo(x) { if (x <

slide-1
SLIDE 1

An Exploratory Study Into the Prevalence of Botched Code Integrations

Ward Muylaert and Coen De Roover Software Languages Lab Vrije Universiteit Brussel Brussels, Belgium

@wardmuylaert @oniroi

slide-2
SLIDE 2

Textual Conflicts

function foo(x) { if (x < 0) { return -x; } return x; }

2

slide-3
SLIDE 3

Textual Conflicts

function foo(x) { if (x < 0) { return -x; } return x; } function foo(x) { if (x < 0) { return -x; } return 0; }

3

slide-4
SLIDE 4

Textual Conflicts

function foo(x) { if (x < 0) { return -x; } return x; } function foo(x) { if (x < 0) { return -x; } return 1; } function foo(x) { if (x < 0) { return -x; } return 0; }

?

4

slide-5
SLIDE 5

Semantic Conflicts

function foo(x) { if (x < 0) { return -x; } return x; }

5

slide-6
SLIDE 6

Semantic Conflicts

function foo(x) { if (x < 0) { return -x; } return x; } function foo(x) { if (x < 0) { return -x; } return x; } foo(-10);

6

slide-7
SLIDE 7

Semantic Conflicts

function foo(x) { if (x < 0) { return -x; } return x; } function foo(x) { if (x < 0) { return -x; } return x; } foo(10);

?

7

function foo(x) { if (x < 0) { return -x; } return 1; }

slide-8
SLIDE 8

Research Questions

RQ1 How often does code integration lead to semantic conflicts? RQ2 How much effort is needed to fix semantic conflicts after code integration? RQ3 How long does it take to fix semantic conflicts after code integration?

8 Research Questions Dataset Research Method Result

slide-9
SLIDE 9

9

GitHub

Research Questions Dataset Research Method Result

slide-10
SLIDE 10

10

GHTorrent

150GB MySQL dump, 400M commits

Georgios Gousios, The GHTorrent Dataset and Tool Suite, MSR 2013.

GitHub

Research Questions Dataset Research Method Result

slide-11
SLIDE 11

Continuous Delivery

11

Commit Build Unit tests Acceptance tests … Release

Research Questions Dataset Research Method Result

slide-12
SLIDE 12

Continuous Delivery

12

Commit Build Unit tests Acceptance tests … Release

Travis CI

Research Questions Dataset Research Method Result

slide-13
SLIDE 13

Continuous Delivery

13

TravisTorrent

1300 projects, 100+ builds, 50+ stars, recent commits

Moritz Beller et al., Oops, My Tests Broke the Build: An Analysis of Travis CI Builds with GitHub, PeerJ Preprints, 2016.

Commit Build Unit tests Acceptance tests … Release

Travis CI

Research Questions Dataset Research Method Result

slide-14
SLIDE 14

Travis CI API

14

Builds state started_at finished_at commit_id repository_id … Commits commit_id sha branch message committed_at … Repositories repository_id slug description …

Travis CI - API Reference, https://docs.travis-ci.com/api Research Questions Dataset Research Method Result

slide-15
SLIDE 15

Travis CI API

15

Builds state started_at finished_at commit_id repository_id … Commits commit_id sha branch message committed_at … Repositories repository_id slug description …

Travis CI - API Reference, https://docs.travis-ci.com/api Research Questions Dataset Research Method Result

slide-16
SLIDE 16

Travis CI API

16

Builds state started_at finished_at commit_id repository_id … Commits commit_id sha branch message committed_at … Repositories repository_id slug description …

Travis CI - API Reference, https://docs.travis-ci.com/api Research Questions Dataset Research Method Result

slide-17
SLIDE 17

Travis CI API

17

Builds state started_at finished_at commit_id repository_id … Commits commit_id sha branch message committed_at … Repositories repository_id slug description …

Travis CI - API Reference, https://docs.travis-ci.com/api

i.e., user/repo

Research Questions Dataset Research Method Result

slide-18
SLIDE 18

Travis CI API

18

Builds state started_at finished_at commit_id repository_id … Commits commit_id sha branch message committed_at … Repositories repository_id slug description …

Travis CI - API Reference, https://docs.travis-ci.com/api

Those same 1300 projects

1.1M builds

i.e., user/repo

Research Questions Dataset Research Method Result

slide-19
SLIDE 19

SELECT c.sha,dcs.state,dcs.repository_id FROM ( SELECT DISTINCT sha,state,repository_id FROM WardTravisCommits AS wtc INNER JOIN WardTravisBuilds AS wtb ON wtb.commit_id = wtc.id ) AS dcs INNER JOIN commits AS c ON c.sha = dcs.sha INNER JOIN commit_parents AS cp ON c.id = cp.commit_id GROUP BY c.id,dcs.state HAVING COUNT(*) > 1

Identify Merge Commits

19

Number of parents

Research Questions Dataset Research Method Result

slide-20
SLIDE 20

SELECT c.sha,dcs.state,dcs.repository_id FROM ( SELECT DISTINCT sha,state,repository_id FROM WardTravisCommits AS wtc INNER JOIN WardTravisBuilds AS wtb ON wtb.commit_id = wtc.id ) AS dcs INNER JOIN commits AS c ON c.sha = dcs.sha INNER JOIN commit_parents AS cp ON c.id = cp.commit_id GROUP BY c.id,dcs.state HAVING COUNT(*) > 1

Get commits and their builds

Identify Merge Commits

20

Number of parents

Research Questions Dataset Research Method Result

slide-21
SLIDE 21

SELECT c.sha,dcs.state,dcs.repository_id FROM ( SELECT DISTINCT sha,state,repository_id FROM WardTravisCommits AS wtc INNER JOIN WardTravisBuilds AS wtb ON wtb.commit_id = wtc.id ) AS dcs INNER JOIN commits AS c ON c.sha = dcs.sha INNER JOIN commit_parents AS cp ON c.id = cp.commit_id GROUP BY c.id,dcs.state HAVING COUNT(*) > 1

Get commits and their builds

Identify Merge Commits

21

Number of parents As long as they have two or more parents

Research Questions Dataset Research Method Result

slide-22
SLIDE 22

Enough Merge Commits?

584 Projects with 50+ Builds of Merge Commits

22 75 114 217.5 Research Questions Dataset Research Method Result

slide-23
SLIDE 23

Enough Merge Commits?

584 Projects with 50+ Builds of Merge Commits

23 75 114 217.5 Research Questions Dataset Research Method Result

History rewriting. Patch application.

!

slide-24
SLIDE 24

Dataset

24

100+ builds, 50+ stars, recent commits: 1300 projects

Research Questions Dataset Research Method Result

slide-25
SLIDE 25

Dataset

25

100+ builds, 50+ stars, recent commits: 1300 projects Still on Travis CI: 1248 projects

Research Questions Dataset Research Method Result

slide-26
SLIDE 26

Dataset

26

100+ builds, 50+ stars, recent commits: 1300 projects Still on Travis CI: 1248 projects 50+ builds of merge commits: 584 projects

Research Questions Dataset Research Method Result

slide-27
SLIDE 27

What Is Failure?

27

Travis CI

Research Questions Dataset Research Method Result

slide-28
SLIDE 28

What Is Failure?

28

Travis CI

✓ passed ✘ failed ✘ errored cancelled started

Research Questions Dataset Research Method Result

slide-29
SLIDE 29

Failed Merge Commit Builds per Project

29 6.7% 15.3% 29.0% 62.6% Research Questions Dataset Research Method Result

slide-30
SLIDE 30

Failed Merge Commit Builds per Project

30 6.7% 15.3% 29.0% 62.6%

½ projects: ⅙ merges fail ¼ projects: ⅓ merges fail

Research Questions Dataset Research Method Result

slide-31
SLIDE 31

Dataset revisited

31

100+ builds, 50+ stars, recent commits: 1300 projects Still on Travis CI: 1248 projects 50+ builds of merge commits: 584 projects Without outliers: 559 projects

Research Questions Dataset Research Method Result

slide-32
SLIDE 32

How Much Effort in Fixing the Build?

32

3 4 5 7 6 1 2 NBTF (Number of Builds To Fix): 4

Research Questions Dataset Research Method Result

slide-33
SLIDE 33

How Much Effort in Fixing the Build?

33 NBTF 2 4 16 Research Questions Dataset Research Method Result

slide-34
SLIDE 34

How Much Effort in Fixing the Build?

34 NBTF

Most builds are fixed easily. ¼ are not or are ignored.

2 4 16 Research Questions Dataset Research Method Result

slide-35
SLIDE 35

How Much Effort in Fixing the Build?

35 NBTF

Most builds are fixed easily. ¼ are not or are ignored.

2 4 16 Research Questions Dataset Research Method Result

NBTF only proxy for effort

!

slide-36
SLIDE 36

How Long to Fix the Build?

36

3 4 5 7 6 1 2 TTF (Time To Fix): 1h50m 13:00 13:10 14:45 14:00 13:30 15:00 15:20

Research Questions Dataset Research Method Result

slide-37
SLIDE 37

How Long to Fix the Build?

37

Time range (disjunct) Amount Relative Cumulative Less than 1 hour 8031 28.2% 28.2% Less than 12 hours 6062 21.3% 49.4% Less than 1 day 2392 8.4% 57.8% Less than 7 days 5556 19.5% 77.3% Less than 1 month (30 days) 3321 11.6% 88.9% Less than 1 year (365 days) 2967 10.4% 99.3% More than 1 year 198 0.7% 100%

Research Questions Dataset Research Method Result

slide-38
SLIDE 38

How Long to Fix the Build?

38

Time range (disjunct) Amount Relative Cumulative Less than 1 hour 8031 28.2% 28.2% Less than 12 hours 6062 21.3% 49.4% Less than 1 day 2392 8.4% 57.8% Less than 7 days 5556 19.5% 77.3% Less than 1 month (30 days) 3321 11.6% 88.9% Less than 1 year (365 days) 2967 10.4% 99.3% More than 1 year 198 0.7% 100% Most builds are fixed within a day.

Research Questions Dataset Research Method Result

slide-39
SLIDE 39

How Long to Fix the Build?

39

Time range (disjunct) Amount Relative Cumulative Less than 1 hour 8031 28.2% 28.2% Less than 12 hours 6062 21.3% 49.4% Less than 1 day 2392 8.4% 57.8% Less than 7 days 5556 19.5% 77.3% Less than 1 month (30 days) 3321 11.6% 88.9% Less than 1 year (365 days) 2967 10.4% 99.3% More than 1 year 198 0.7% 100% Most builds are fixed within a day.

Research Questions Dataset Research Method Result

Time taken ≠ time worked

  • n

!

slide-40
SLIDE 40

Number of Builds To Fix vs Time To Fix

40 Research Questions Dataset Research Method Result

slide-41
SLIDE 41

Number of Builds To Fix vs Time To Fix

41 Research Questions Dataset Research Method Result

slide-42
SLIDE 42

Answers to Research Questions

RQ1 Conflicts after code integration happen often. For half of the projects, the build did not succeed for ⅙ merge commits. RQ2 Multiple builds are needed to fix the majority of botched code integrations. RQ3 The majority of botched code integrations are fixed within a day.

42

slide-43
SLIDE 43

Future

43

This study

  • Filter out projects that do not really follow continuous delivery
  • Refine effort estimation using commit information
  • Use TravisTorrent information to refine failure categorization

Long term

  • Help developers with merging
  • Behavioural differencing with dynamic symbolic execution
slide-44
SLIDE 44

Summary

44