Path Projection For User-Centered Static Analysis Tools Khoo Yit - - PowerPoint PPT Presentation

▶
path projection
SMART_READER_LITE
LIVE PREVIEW

Path Projection For User-Centered Static Analysis Tools Khoo Yit - - PowerPoint PPT Presentation

Nov 01, 2022 35 likes •899 views

Path Projection For User-Centered Static Analysis Tools Khoo Yit Phang , Jeff Foster, Michael Hicks, Vibha Sazawal University of Maryland PASTE 2008, November 10 1 Success in Static Analysis Coverity, Fortify, Grammatech, Klocwork, many others

slide-1
SLIDE 1

Path Projection

For User-Centered Static Analysis Tools

Khoo Yit Phang, Jeff Foster, Michael Hicks, Vibha Sazawal University of Maryland PASTE 2008, November 10

1

slide-2
SLIDE 2

Success in Static Analysis

Coverity, Fortify, Grammatech, Klocwork, many others are selling static analysis tools Microsoft, Mozilla, and others are integrating static analysis into development Very active static analysis research community

2

slide-3
SLIDE 3

But there’ s a problem...

Research has focused on static analysis algorithms But, programmers use tools, not algorithms Static analysis tools are only useful if programmers can understand the results Our goal: develop ways to make static analysis tools more user-centered

3

slide-4
SLIDE 4

Path Projection

A new UI toolkit for visualizing program paths

call stacks and control-flow paths

Paths are very common in static analysis tool output

Helping users understand paths will help many static analysis tools We have applied Path Projection to Locksmith and BLAST

Experimental evaluation

Task: triaging Locksmith error reports Result: 18% improvement in completion time, similar accuracy

4

slide-5
SLIDE 5

Case Study: Locksmith

Static data race detector for C

Data race: Two or more threads access a shared variable at the same time

Locksmith reports call stacks to possibly-racing dereferences

To triage, user must decide whether multiple paths are simultaneously realizable

Polyvios Pratikakis et al. (PLDI 2006)

5

slide-6
SLIDE 6

Locksmith in Standard Viewer

6

Standard Viewer designed to mimic typical editors/IDEs

slide-7
SLIDE 7

Locksmith in Standard Viewer

6

Locksmith error report with hyperlinks

slide-8
SLIDE 8

6

slide-9
SLIDE 9

Locksmith Error Report

7

slide-10
SLIDE 10

Locksmith Error Report

7

Shared variable

slide-11
SLIDE 11

Locksmith Error Report

7

Shared variable Call stacks leading to race

slide-12
SLIDE 12

Locksmith Error Report

7

Thread creation

slide-13
SLIDE 13

Locksmith Error Report

7

Thread creation Dereference

slide-14
SLIDE 14

Locksmith Error Report

7

Thread creation Dereference w/no locks held

slide-15
SLIDE 15

Locksmith Error Report

7

Thread creation Dereference w/no locks held

slide-16
SLIDE 16

Locksmith Error Report

7

slide-17
SLIDE 17

Triaging Locksmith

8

Triage: are these call stacks simultaneously realizable?

slide-18
SLIDE 18

Triaging Locksmith

9

Begin by clicking

slide-19
SLIDE 19

Triaging Locksmith

9

Begin by clicking

slide-20
SLIDE 20

Triaging Locksmith

10

Thread creation realizable?

slide-21
SLIDE 21

Triaging Locksmith

10

Thread creation realizable? Yes, unconditionally created

slide-22
SLIDE 22

Triaging Locksmith

11

Keep around context

slide-23
SLIDE 23

Triaging Locksmith

11

Keep around context Easier than remembering

slide-24
SLIDE 24

Triaging Locksmith

12

Focus

slide-25
SLIDE 25

Triaging Locksmith

12

Call site of Next call

slide-26
SLIDE 26

Triaging Locksmith

13

Skipping a few steps

slide-27
SLIDE 27

Triaging Locksmith

13

the dereference, finally!

slide-28
SLIDE 28

Triaging Locksmith

14

slide-29
SLIDE 29

Triaging Locksmith

14

Screen is very cluttered!

slide-30
SLIDE 30

Triaging Locksmith

14

Old context is hidden!

slide-31
SLIDE 31

Triaging Locksmith

14

Old context is hidden!

Which function is this? Where was this called?

slide-32
SLIDE 32

A Thousand Cuts

15

slide-33
SLIDE 33

A Thousand Cuts

Many little distractions from actual task Seemingly straightforward task becomes complex!

Read error report Click hyperlink 1 Read code Scroll up Scroll down Split window Focus Back to error report Click hyperlink 2 Read code Scroll down Split window Focus Back to error report Click hyperlink 3 Read code Scroll down Split window Focus Back to error report Collapse splits (resize window, move window...)

15

slide-34
SLIDE 34

Path Projection

Designed for tracing paths

16

slide-35
SLIDE 35

Path Projection

Designed for tracing paths Function call inlining:

Inline function directly below call site

16

slide-36
SLIDE 36

Path Projection

Designed for tracing paths Function call inlining:

Inline function directly below call site

Path-derived code folding:

Show only implicated lines and lexical control-blocks

16

slide-37
SLIDE 37

Path Projection

Designed for tracing paths Function call inlining:

Inline function directly below call site

Path-derived code folding:

Show only implicated lines and lexical control-blocks

Show as much code as possible on one screen

16

slide-38
SLIDE 38
slide-39
SLIDE 39

Path Projection

17

Show paths side by side

slide-40
SLIDE 40

Path Projection

17

Show paths side by side

slide-41
SLIDE 41

Path Projection

18

Multiple searches (despite folds)

slide-42
SLIDE 42

Path Projection

18

Multiple searches (despite folds)

slide-43
SLIDE 43

Path Projection

19

Continuing example...

slide-44
SLIDE 44

Path Projection

19

from 1st call stack

slide-45
SLIDE 45

Path Projection

19

from 2nd call stack

slide-46
SLIDE 46

Path Projection

19

from 2nd call stack dereference

slide-47
SLIDE 47

Path Projection

19

in loop condition for dereference condition for thread creation

slide-48
SLIDE 48

Path Projection

19

not a single click or scroll!

slide-49
SLIDE 49

Path Projection

19

not a single click or scroll! no need to look here too!

slide-50
SLIDE 50

Path Projection

20

What’ s foffset?

slide-51
SLIDE 51

Path Projection

20

slide-52
SLIDE 52

Path Projection

21

What’ s in read_log?

slide-53
SLIDE 53

Path Projection

21

Reveal definition (initially folded)

slide-54
SLIDE 54

Pilot User Study

22

We discovered that static analysis is...

slide-55
SLIDE 55

Rocket Science

23

In our pilot studies, non-expert users had great trouble triaging Locksmith error reports:

ad hoc, inconsistent procedure neglected some causes of false positives sidetracked by non-causes of false positives

Even with extensive tutorials!

slide-56
SLIDE 56

Rocket Science 101

24

Our solution: triaging checklist Checklists are tool-/ error-specific

Different tools have different imprecision & error reports

Anecdotally, 41% faster at triaging using checklist

slide-57
SLIDE 57

Locksmith Triaging Checklist

To triage Locksmith:

check if any pair of paths are simultaneously realizable different cases: threads in loop, parent-child, child-child

For example:

Source of imprecision: Locksmith is path-insensitive Possible false positive: child-child threads may be mutually exclusive

25

slide-58
SLIDE 58

Locksmith Triaging Checklist

To triage Locksmith:

check if any pair of paths are simultaneously realizable different cases: threads in loop, parent-child, child-child

For example:

Source of imprecision: Locksmith is path-insensitive Possible false positive: child-child threads may be mutually exclusive

25

slide-59
SLIDE 59

Locksmith Triaging Checklist

To triage Locksmith:

check if any pair of paths are simultaneously realizable different cases: threads in loop, parent-child, child-child

For example:

Source of imprecision: Locksmith is path-insensitive Possible false positive: child-child threads may be mutually exclusive

25

slide-60
SLIDE 60

User Study

Which is better: Standard Viewer (SV) or Path Projection (PP)?

Quantitatively: completion time Qualitatively: user ratings

Data race triaging task using Locksmith

26

slide-61
SLIDE 61

User Study Issues

Large variance between participants

Participants have different skill level Are differences due to participant or UI?

Within-subjects: each participant use both interfaces

Compare UI results for each participant

27

slide-62
SLIDE 62

User Study Issues

Order and carryover effect

Participants get better over time (learning) Participants biased by initial UI or problem

Counter-balance: divide participants into two groups

SV-PP: Standard Viewer, then Path Projection PP-SV: Path Projection, then Standard Viewer

28

slide-63
SLIDE 63

User Study: Locksmith Task

6 trials from Locksmith corpus (unfamiliar to users) One warning per trial

no need to manage warnings

Only verify that paths are simultaneously realizable

No aliasing/imprecise lock state (future work)

29

slide-64
SLIDE 64

User Study: Misc.

8 student participants

3 undergraduates, 5 graduates Prior experience in C, multithreading (not necessarily C) Self-rated 3-4 (1: no experience to 5: very experienced) 2 had experience in Locksmith and Eraser

30

slide-65
SLIDE 65

Session 1 Session 2 100 200 300 400

Completion time (sec)

Standard Viewer Path Projection

Quantitative (Chart guide)

31

slide-66
SLIDE 66

Session 1 Session 2 100 200 300 400

Completion time (sec)

Standard Viewer Path Projection

Quantitative (Chart guide)

31

Time in Seconds

slide-67
SLIDE 67

Session 1 Session 2 100 200 300 400

Completion time (sec)

Standard Viewer Path Projection

Quantitative (Chart guide)

31

Time in Seconds PP-SV group SV-PP group

slide-68
SLIDE 68

Faster Completion Time

Session 1 Session 2 100 200 300 400

Completion time (sec)

Standard Viewer Path Projection

32

slide-69
SLIDE 69

Learning effect

all improved in Session 2*

Faster Completion Time

Session 1 Session 2 100 200 300 400

Completion time (sec)

Standard Viewer Path Projection

*statistically significant (p<0.05)

32

Learning effect

slide-70
SLIDE 70

Learning effect

all improved in Session 2*

SV-PP improved by 188s*

(effect size d=1.276)

Faster Completion Time

Session 1 Session 2 100 200 300 400

Completion time (sec)

Standard Viewer Path Projection

*statistically significant (p<0.05)

32

SV-PP: 188s Learning effect

slide-71
SLIDE 71

Learning effect

all improved in Session 2*

SV-PP improved by 188s*

(effect size d=1.276)

PP-SV improved by 55s*

(effect size d=0.375)

Faster Completion Time

Session 1 Session 2 100 200 300 400

Completion time (sec)

Standard Viewer Path Projection

*statistically significant (p<0.05)

32

SV-PP: 188s PP-SV: 55s Learning effect

slide-72
SLIDE 72

Learning effect

all improved in Session 2*

SV-PP improved by 188s*

(effect size d=1.276)

PP-SV improved by 55s*

(effect size d=0.375)

Similar # mistakes

10 in PP (10.9%), 9 in SV (9.8%)

18% faster on average

Faster Completion Time

Session 1 Session 2 100 200 300 400

Completion time (sec)

Standard Viewer Path Projection

*statistically significant (p<0.05)

32

SV-PP: 188s PP-SV: 55s Learning effect

slide-73
SLIDE 73

Duration where pointer is

  • ver error report

(e.g., using hyperlinks)

Less Use of Error Report

Session 1 Session 2 20 40 60 80 100 120

Duration in error report (sec)

Standard Viewer Path Projection

33

slide-74
SLIDE 74

Duration where pointer is

  • ver error report

(e.g., using hyperlinks)

On average, only 20s with PP vs. 94s with SV*

Less Use of Error Report

Session 1 Session 2 20 40 60 80 100 120

Duration in error report (sec)

Standard Viewer Path Projection

*statistically significant (p<0.05)

33

slide-75
SLIDE 75

Duration where pointer is

  • ver error report

(e.g., using hyperlinks)

On average, only 20s with PP vs. 94s with SV* “Necessary for [SV], but just a convenience in [PP]. ”

Less Use of Error Report

Session 1 Session 2 20 40 60 80 100 120

Duration in error report (sec)

Standard Viewer Path Projection

*statistically significant (p<0.05)

33

slide-76
SLIDE 76

Qualitative (Boxplot Guide)

  • Quick

to learn Confident

  • f answer

Easy to verify race Prefer Path Projection 1 2 3 4 5

Overall impression

Standard Viewer Path Projection

Strongly agree

Strongly disagree

We asked participants to rate on 1-5 scale

34

slide-77
SLIDE 77

Qualitative (Boxplot Guide)

  • Quick

to learn Confident

  • f answer

Easy to verify race Prefer Path Projection 1 2 3 4 5

Overall impression

Standard Viewer Path Projection

Strongly agree

Strongly disagree

We asked participants to rate on 1-5 scale Results summarized in boxplots

34

median quartiles max (<1.5quartile) min (<1.5quartile)

  • utlier (>1.5quartile)
slide-78
SLIDE 78

Prefer Path Projection

  • Quick

to learn Confident

  • f answer

Easy to verify race Prefer Path Projection 1 2 3 4 5

Overall impression

Standard Viewer Path Projection

Strongly agree

Strongly disagree 35

slide-79
SLIDE 79

Prefer Path Projection

  • Quick

to learn Confident

  • f answer

Easy to verify race Prefer Path Projection 1 2 3 4 5

Overall impression

Standard Viewer Path Projection

Strongly agree

Strongly disagree

Quick/confident/ easy: not statistically significant

35

slide-80
SLIDE 80

Prefer Path Projection

  • Quick

to learn Confident

  • f answer

Easy to verify race Prefer Path Projection 1 2 3 4 5

Overall impression

Standard Viewer Path Projection

Strongly agree

Strongly disagree

Quick/confident/ easy: not statistically significant Preference: all but one preferred PP to SV*

*statistically significant (p<0.05)

35

slide-81
SLIDE 81

Path Projection Features

Very useful

Not useful

  • Error

report Checklist Function inlining Code folding Multi− query Query reveals folded code

1 2 3 4 5

Usefulness of features

36

slide-82
SLIDE 82

Path Projection Features

Very useful

Not useful

Generally favorable towards PP features*

  • Error

report Checklist Function inlining Code folding Multi− query Query reveals folded code

1 2 3 4 5

Usefulness of features

36

*statistically significant (p<0.05)

slide-83
SLIDE 83

Path Projection Features

Very useful

Not useful

Checklist: “saved me from having to memorize rules”

  • Error

report Checklist Function inlining Code folding Multi− query Query reveals folded code

1 2 3 4 5

Usefulness of features

37

slide-84
SLIDE 84

Path Projection Features

Very useful

Not useful

Checklist: “saved me from having to memorize rules” Surprisingly, favored function inlining/code folding

code folding was “the best feature” or “my favorite feature”

  • Error

report Checklist Function inlining Code folding Multi− query Query reveals folded code

1 2 3 4 5

Usefulness of features

37

slide-85
SLIDE 85

Threats To Validity

Experimental design limitations

small number of users and trials not static analysis experts, unfamiliar programs statistically significant despite limitations

Standard Viewer not “real” editor

deliberate choice to avoid bias from prior experience

The checklist might bias users

checklist designed for Locksmith, not SV or PP both interfaces use the same checklist

38

slide-86
SLIDE 86

Conclusion

Path Projection: a new UI toolkit for visualizing program paths Can be used with any static analysis tools

Takes an XML path report as input

Our study showed that it improves completion time (18%) with similar accuracy and users liked it Try it at: http:/ /www.cs.umd.edu/projects/PL/PP/

39