Evaluating How Developers Use General-Purpose Web-Search for Code - PowerPoint PPT Presentation

Evaluating How Developers Use General-Purpose Web-Search for Code Retrieval Md Masudur Rahman , Jed Barson, Sydney Paul, Joshua Kayan, Federico Andres Lois, Sebastian Fernandez Quezada, Christopher Parnin, Kathryn T. Stolee, Baishakhi Ray Date: May 29, 2018 � 1

Coding Task Convert a date string to a time object � 2

string to time � 3

string to time Search Log string to time � 4

string to time Java Search Log string to time � 5

string to time using java Search Log string to time � 6

string to time using java Search Log string to time string to time using java � 7

string to time using java Search Log string to time string to time using java � 8

string to time using java DateTime Search Log string to time string to time using java � 9

date string to DateTime using java Search Log string to time string to time using java � 10

date string to DateTime using java Search Log string to time string to time using java date string to DateTime using java � 11

date string to DateTime using java Joda Time library Search Log string to time string to time using java date string to DateTime using java � 12

date string to DateTime using Joda Time library Search Log string to time string to time using java date string to DateTime using java � 13

date string to DateTime using Joda Time library Search Log string to time string to time using java date string to DateTime using java date string to DateTime using Joda… � 14

date string to DateTime using Joda Time library � 15

date string to DateTime using Joda Time library X X X � 16

world cup fixtures Search Log string to time string to time using java date string to DateTime using java date string to DateTime using Joda … � 17

world cup fixtures string to time string to time using java date string to DateTime using java date string to DateTime using Joda … world cup fixtures � 18

place to visit in gothenburg Search Log string to time string to time using java date string to DateTime using java date string to DateTime using Joda … world cup fixtures � 19

place to visit in gothenburg Search Log string to time string to time using java date string to DateTime using java date string to DateTime using Joda … world cup fixtures place to visit in gothenburg � 20

Code Query Code Query string to time string to time using java date string to DateTime using java date string to DateTime using Joda Time library � 21

Search Task Search Task Convert a date string to a DateTime object Code using Joda Time library Query string to time string to time using java date string to DateTime using java date string to DateTime using Joda Time library � 22

Code vs Non-code Non-Code Code Query Query string to time world cup fixtures string to time using java place to visit in gothenburg date string to DateTime using java hotel in gothenburg date string to DateTime using Joda Time library � 23

General Purpose Search Engine for Code Retrieval Non-Code Code Query Query string to time world cup fixtures string to time using java place to visit in gothenburg date string to DateTime using java hotel in gothenburg date string to DateTime using Joda Time library � 24

Research Goal Non-Code Code ๏ Query characteristics ๏ User behavior Query Query string to time world cup fixtures string to time using java place to visit in gothenburg date string to DateTime using java hotel in gothenburg date string to DateTime using Joda Time library � 25

Dataset Query Search Log Chrome plugin string to time string to time using java date string to DateTime using java date string to DateTime using Joda Time library Users: 310 (mostly developer) world cup fixtures place to visit in gothenburg Consist of code and non-code queries hotel in gothenburg Total query: 150K � 26

Dataset Query Search Log No label string to time string to time using java ? date string to DateTime using java Code or Non-code date string to DateTime using Joda Time library world cup fixtures place to visit in gothenburg hotel in gothenburg � 27

Dataset Query Search Log No label string to time string to time using java ? date string to DateTime using java Code or Non-code date string to DateTime using Joda Time library world cup fixtures Query Classifier place to visit in gothenburg hotel in gothenburg � 28

Intent-based Query Classification � 29

Code Intent Analysis Query: javascript function to get mp3 play length � 30

Code Intent Analysis Query: javascript function to get mp3 play length CodeScore ? � 31

Code Intent Analysis Query: javascript function to get mp3 play length CodeScore ? 17 7 0 6 5 8 3 Token Code Intent S = set of code related tags n = popularity of a tag � 32

Code Intent Analysis Query: javascript function to get mp3 play length CodeScore 46 17 7 0 6 5 8 3 Token Code Intent Query Code Intent � 33

Query Code Score Query Code Score string to time 12 string to time using java 20 date string to DateTime using java 22.5 world cup fixtures 0 messi curly goal 2.6 place to visit in gothenburg 0 � 34

Query Code Score Query Code Score Label string to time 12 ? string to time using java 20 ? date string to DateTime using java 22.5 ? world cup fixtures 0 ? messi curly goal 2.6 ? place to visit in gothenburg 0 ? � 35

Query Code Score Classifier Evaluation Query Code Score Label string to time 12 ? Manually annotated 380 string to time using java 20 ? queries date string to DateTime using java 22.5 ? Threshold = 10 world cup fixtures 0 ? Precision: 87% messi curly goal 2.6 ? Recall: 86% place to visit in gothenburg 0 ? F1-score: 87% � 36

Query Code Score Classifier Evaluation Query Code Score Label string to time 12 Code Manually annotated 380 string to time using java 20 Code queries date string to DateTime using java 22.5 Code Threshold = 10 world cup fixtures 0 Non-code Precision: 87% messi curly goal 2.6 Non-code Recall: 86% place to visit in gothenburg 0 Non-code F1-score: 87% � 37

Query Code Score Classifier Evaluation Query Code Score Label string to time 12 Code Manually annotated 380 string to time using java 20 Code queries date string to DateTime using java 22.5 Code Threshold = 10 world cup fixtures 0 Non-code Precision: 87% messi curly goal 2.6 Non-code Recall: 86% place to visit in gothenburg 0 Non-code F1-score: 87% Code : 89K (59%) Annotated Data Non-code : 61K (41%) � 38

Research Questions Query RQ1. How do query characteristics differ for code and Characteristics non-code queries? RQ2. How do search behaviors vary for code and non-code related queries? User Behavior RQ3. How do task sessions vary for code and non- code related search tasks? � 39

Results � 40

RQ1: Query Characteristics Code Non-code date string to DateTime using java world cup fixtures date string to DateTime using Joda Time library messi curly goal javascript function to get mp3 play length hotel in gothenburg Code queries often longer (more tokens) than non-code � 41

RQ1: Query Characteristics Code Non-code world cup fixtures date string to DateTime using java messi curly goal date string to DateTime using Joda Time library javascript function to get mp3 play length hotel in gothenburg � 42

RQ1: Query Characteristics Code Non-code world cup fixtures date string to DateTime using java messi curly goal date string to DateTime using Joda Time library javascript function to get mp3 play length hotel in gothenburg � 43

RQ1: Query Characteristics Code Non-code 16K 12K 33K Code queries contain less vocabulary (unique tokens) than non-code � 44

RQ2: Query Search Behavior Query # term added # term deleted string to time - - 2 - string to time using java Code date string to DateTime using 4 2 Joda Time library Non-code hotel in gothenburg - - 1 - best hotel in gothenburg � 45

RQ2: Query Search Behavior Query # term added # term deleted string to time - - string to time using java 2 - Code date string to DateTime using 4 2 Edited query Joda Time library Non-code hotel in gothenburg - - 1 - best hotel in gothenburg � 46

RQ2: Query Search Behavior Query # term added # term deleted string to time - - 2 - string to time using java Code date string to DateTime using 4 2 Joda Time library Non-code hotel in gothenburg - - 1 - best hotel in gothenburg User often add/delete more terms (avg. 2) to a code compared to non- code (avg. 1) � 47

RQ2: Query Search Behavior Query # term added # term deleted Code Score string to time - - 12 2 - string to time using java 20 Code date string to DateTime using 4 2 30.5 Joda Time library � 48

RQ2: Query Search Behavior Query # term added # term deleted Code Score string to time - - 12 2 - string to time using java 20 Code date string to DateTime using 4 2 30.5 Joda Time library Edit query to increase code intent � 49