Detection using Search-Based Application Profiling Du Sh Shen, Qi - - PowerPoint PPT Presentation

▶
detection using search based
SMART_READER_LITE
LIVE PREVIEW

Detection using Search-Based Application Profiling Du Sh Shen, Qi - - PowerPoint PPT Presentation

Aug 29, 2022 299 likes •1.1k views

Automating Performance Bottleneck Detection using Search-Based Application Profiling Du Sh Shen, Qi i Luo, , Denys Posh shyv yvanyk, Mark rk Grechanik ik* Col olle lege of of Will llia iam and and Mar ary *U *Univ iversit ity

slide-1
SLIDE 1

Automating Performance Bottleneck Detection using Search-Based Application Profiling

Du Sh Shen, Qi i Luo, , Denys Posh shyv yvanyk, Mark rk Grechanik ik*

Col

  • lle

lege of

  • f Will

llia iam and and Mar ary *U *Univ iversit ity of

  • f Il

Illi linois is at t Chi hicago

ISS ISSTA 2015 Ba Balt ltimore, MD, U.S .S.

1

slide-2
SLIDE 2

2

slide-3
SLIDE 3

Standard application profiling

Inputs Application

3

slide-4
SLIDE 4

Standard application profiling

Inputs Application

4

slide-5
SLIDE 5

Standard application profiling

  • 1. Agilefant.model.WidgetCollection.getName() 273.2s
  • 2. Agilefant.db.hibernate.UserTypeFilter.deepCopy() 213.5s
  • 3. Agilefant.model.Team.setId() 192.3s
  • 4. Agilefant.model.Backlog.setChildren() 123.9s
  • 5. ……

Inputs Application

5

slide-6
SLIDE 6

Standard application profiling

  • 1. Agilefant.model.WidgetCollection.getName() 273.2s
  • 2. Agilefant.db.hibernate.UserTypeFilter.deepCopy() 213.5s
  • 3. Agilefant.model.Team.setId() 192.3s
  • 4. Agilefant.model.Backlog.setChildren() 123.9s
  • 5. ……

Inputs Application

6

slide-7
SLIDE 7

Standard application profiling

find(int list[] [], , int n, , int key) int lo = = 0; int hi = = n n - 1; int result = = -1; hi >= = lo lo resu sult == -1 final int mid = (lo lo+hi hi) / 2; list[mid] ] > > key list[mid] == key hi hi=mid - 1; lo lo=mid + + 1; Resu sult = = mid; return result; T F F T T F F T

7

slide-8
SLIDE 8

Standard application profiling

find(int list[] [], , int n, , int key) int lo = = 0; int hi = = n n - 1; int result = = -1; hi >= = lo lo resu sult == -1 final int mid = (lo lo+hi hi) / 2; list[mid] ] > > key list[mid] == key hi hi=mid - 1; lo lo=mid + + 1; Resu sult = = mid; return result; T F F T T F F T

8

slide-9
SLIDE 9

Standard application profiling

find(int list[] [], , int n, , int key) int lo = = 0; int hi = = n n - 1; int result = = -1; hi >= = lo lo resu sult == -1 final int mid = (lo lo+hi hi) / 2; list[mid] ] > > key list[mid] == key hi hi=mid - 1; lo lo=mid + + 1; Resu sult = = mid; return result; T F F T T F F T

9

slide-10
SLIDE 10

Standard application profiling

find(int list[] [], , int n, , int key) int lo = = 0; int hi = = n n - 1; int result = = -1; hi >= = lo lo resu sult == -1 final int mid = (lo lo+hi hi) / 2; list[mid] ] > > key list[mid] == key hi hi=mid - 1; lo lo=mid + + 1; Resu sult = = mid; return result; T F F T T F F T

10

slide-11
SLIDE 11

11

slide-12
SLIDE 12

Inputs

In Input-sensitive profiling

Application

  • 1. Agilefant.model.WidgetCollection.getName() 273.2s
  • 2. Agilefant.db.hibernate.UserTypeFilter.deepCopy() 213.5s
  • 3. Agilefant.model.Team.setId() 192.3s
  • 4. Agilefant.model.Backlog.setChildren() 123.9s
  • 5. ……

12

slide-13
SLIDE 13

Inputs

In Input-sensitive profiling

Application

  • 1. Agilefant.model.WidgetCollection.getName() 273.2s
  • 2. Agilefant.db.hibernate.UserTypeFilter.deepCopy() 213.5s
  • 3. Agilefant.model.Team.setId() 192.3s
  • 4. Agilefant.model.Backlog.setChildren() 123.9s
  • 5. ……

13

slide-14
SLIDE 14

1 Input x, y, z, u 2 v = A.m (x, y) 3 if (v > z) { 4 C.h (B.m (v)) 5 } 6 else { 7 D.h (B.m (v)) 8 }

In Input-sensitive profiling

14

slide-15
SLIDE 15

1 Input x, y, z, u 2 v = A.m (x, y) 3 if (v > z) { 4 C.h (B.m (v)) 5 } 6 else { 7 D.h (B.m (v)) 8 }

In Input-sensitive profiling

unimportant input value

15

slide-16
SLIDE 16

1 Input x, y, z, u 2 v = A.m (x, y) 3 if (v > z) { 4 C.h (B.m (v)) 5 } 6 else { 7 D.h (B.m (v)) 8 }

In Input-sensitive profiling

Construct combinations of input values

16

slide-17
SLIDE 17

1 Input x, y, z, u 2 v = A.m (x, y) 3 if (v > z) { 4 C.h (B.m (v)) 5 } 6 else { 7 D.h (B.m (v)) 8 }

In Input-sensitive profiling

General-purpose methods

17

slide-18
SLIDE 18

1 Input x, y, z, u 2 v = A.m (x, y) 3 if (v > z) { 4 C.h (B.m (v)) 5 } 6 else { 7 D.h (B.m (v)) 8 }

In Input-sensitive profiling

Identify the input-sensitive bottlenecks

18

slide-19
SLIDE 19

Genetic Alg lgorithm-driven Profiler (GA-Prof)

  • Automate input-sensitive profiling
  • Explore input parameter space
  • Detect performance bottlenecks

19

slide-20
SLIDE 20

GA Analyzer Software system Profiler

20

Genetic Alg lgorithm-driven Profiler (GA-Prof)

slide-21
SLIDE 21

GA Analyzer Software system Profiler Contrast Mining Ranked list:

1. Agilefant.db.hibernate.UserTypeFilter.d eepCopy() 2. Agilefant.model.WidgetCollection.getN ame() 3. Agilefant.model.Backlog.setChildren() 4. Agilefant.model.Team.setId() 5. ………

21

Genetic Alg lgorithm-driven Profiler (GA-Prof)

slide-22
SLIDE 22

Genetic Alg lgorithms (GAs)

  • Simulate the natural selection process
  • Generate solutions to optimization problems

22

slide-23
SLIDE 23

Genetic Alg lgorithms (GAs)

  • Simulate the natural selection process
  • Generate solutions to optimization problems

23

slide-24
SLIDE 24

Why do we use GAs in in GA-Prof

  • Large input space
  • Can be formulated as a search and
  • ptimization problem

24

slide-25
SLIDE 25

Why do we use GAs in in GA-Prof

  • Large input space
  • Can be formulated as a search and
  • ptimization problem
  • Performs better than an alternative

solution

25

slide-26
SLIDE 26
  • Large input space
  • Can be formulated as a search and
  • ptimization problem
  • Perform better than an alternative

solution

Why do we use GAs in in GA-Prof

26

slide-27
SLIDE 27
  • Large input space
  • Can be formulated as a search and
  • ptimization problem
  • Perform better than an alternative

solution

Why do we use GAs in in GA-Prof

27

slide-28
SLIDE 28

GA GA Component Definitions

Genes: A chromosome/individual

Input 1: http://localhost:8080/Agilefant/editUser.action Input 2: http://localhost:8080/Agilefant/editProduct.action?productId=5 Input 3: http://localhost:8080/Agilefant/editProduct.action?productId=8 …… Individual 1: 2, 18, 36, 27, 11, 13, 6, 43, 64, 12, 85, 49, 12, 53, 44, 78, 31, 47, 6

28

slide-29
SLIDE 29

Usin ing GAs in in GA-Prof

Individual 1 Individual 2 Individual 3

……

29

slide-30
SLIDE 30

Usin ing GAs in in GA-Prof

Individual 1 Individual 2 Individual 3

…… Fitness function: Individual elapsed execution time

30

slide-31
SLIDE 31

Usin ing GAs in in GA-Prof

Individual 1 Individual 2 Individual 3

…… Fitness function: Individual elapsed execution time

  • 1. Individual 25 289.5s
  • 2. Individual 17 256.7s
  • 3. Individual 91 197.2s

……

31

slide-32
SLIDE 32

Usin ing GAs in in GA-Prof

Individual 1 Individual 2 Individual 3

……

Parent 1: 2, 18, 36, 27, 11, 13, 6, 43, 64, 12, 85, 49, 12, 53, 44, 91, 79, 23, 3, 19 Parent 2: 23, 95, 1, 67, 35, 81, 7, 17, 51, 102, 56, 39, 72, 3, 54, 37, 13, 86, 47, 76 Child 1: 2, 18, 36, 27, 11, 13, 6, 17, 51, 102, 56, 39, 72, 3, 54, 37, 13, 86, 47, 76 Child 2: 23, 95, 1, 67, 35, 81, 7, 43, 64, 12, 85, 49, 12, 53, 44, 91, 79, 23, 3, 19

  • 1. Individual 25 289.5s
  • 2. Individual 17 256.7s
  • 3. Individual 91 197.2s

…… crossover

32

slide-33
SLIDE 33

Usin ing GAs in in GA-Prof

Individual 1 Individual 2 Individual 3

……

  • 1. Individual 25 289.5s
  • 2. Individual 17 256.7s
  • 3. Individual 91 197.2s

…… crossover mutation Parent : 2, 18, 36, 27, 11, 13, 6, 43, 64, 12, 85, 49, 12, 53, 44, 91, 79, 23, 3, 19 Child : 2, 18, 36, 27, 11, 13, 6, 43, 64, 73, 85, 49, 12, 53, 44, 91, 79, 23, 3, 19

33

slide-34
SLIDE 34

Usin ing GAs in in GA-Prof

Individual 1 Individual 2 Individual 3

……

  • 1. Individual 25 289.5s
  • 2. Individual 17 256.7s
  • 3. Individual 91 197.2s

…… crossover mutation Individual 1’ Individual 2’ Individual 3’

……

34

slide-35
SLIDE 35

Usin ing GAs in in GA-Prof

Individual 1 Individual 2 Individual 3

……

  • 1. Individual 25 289.5s
  • 2. Individual 17 256.7s
  • 3. Individual 91 197.2s

…… crossover mutation Individual 1’ Individual 2’ Individual 3’

……

35

Independent variables: Crossover rate, mutation rate, number of individuals per generation

slide-36
SLIDE 36

137.92 233.2 128.48 79.6 201.8

Id Identify fying in input-sensitive bottlenecks

Traces Elapsed Execution Time

36

slide-37
SLIDE 37

137.92 233.2 128.48 79.6 201.8

Id Identify fying in input-sensitive bottlenecks

Traces Elapsed Execution Time Good Bad Bad Bad Good

37

Good

slide-38
SLIDE 38

Method 1 Method 2 Method 3 Method 4 Method 5

Id Identify fying in input-sensitive bottlenecks

Traces Elapsed Execution Time Good Bad Bad Bad Good

38

slide-39
SLIDE 39

Id Identify fying in input-sensitive bottlenecks

Method 1 Method 2 Method 3 Method 4 Method 5

Traces Elapsed Execution Time Good Bad Bad Bad Good

39

bottleneck

slide-40
SLIDE 40

In Independent Component Analysis (IC ICA)

40

slide-41
SLIDE 41

Mixed signals

In Independent Component Analysis (IC ICA)

41

slide-42
SLIDE 42

Mixed signals

In Independent Component Analysis (IC ICA)

42

slide-43
SLIDE 43

In Independent Component Analysis (IC ICA)

Feature 1 Feature 2 Trace 1 Trace 2

Mixed signals

43

slide-44
SLIDE 44

In Independent Component Analysis (IC ICA)

= × X A S

: p×m : p×k : k×m traces methods features traces methods features

44

Good traces [SGood] Bad traces [SBad]

slide-45
SLIDE 45

Contrast Min ining

For each method 𝐸 = (𝑇𝐻𝑝𝑝𝑒 − 𝑇𝐶𝑏𝑒)

= × X A S

: p×m : p×k : k×m traces methods features traces methods features

45

Good traces [SGood] Bad traces [SBad]

slide-46
SLIDE 46

Inputs App JMeter Web-based application

46

slide-47
SLIDE 47

Inputs Profiler App

Collect execution traces

MTDENT_|_http-bio-8080-exec- 7_|_131011218033882_|_fi/hut/soberit/agilefant/model/Story_|_getStoryAccesses()Ljava/util/Set;_||_ MTDRET_|_http-bio-8080-exec- 7_|_131011218036560_|_fi/hut/soberit/agilefant/model/Story_|_getStoryAccesses()Ljava/util/Set;_||_ MTDENT_|_http-bio-8080-exec-7_|_131011218074491_|_fi/hut/soberit/agilefant/model/User_|_getId()I_||_ MTDRET_|_http-bio-8080-exec-7_|_131011218076276_|_fi/hut/soberit/agilefant/model/User_|_getId()I_||_

47

slide-48
SLIDE 48

Inputs Trace Statistics Profiler GA Analyzer Execution Trace Analyzer App

Seq of inputs 1 Seq of Inputs 2 Seq of inputs 3

……

  • 1. Seq of inputs 25 289.5s
  • 2. Seq of inputs 17 256.7s
  • 3. Seq of inputs 91 197.2s

…… crossover mutation Seq of inputs 1’ Seq of inputs 2’ Seq of inputs 3’

……

48

slide-49
SLIDE 49

Inputs Trace Statistics Bad Traces Good Traces Profiler Trace Clustering GA Analyzer Execution Trace Analyzer App Traces Elapsed Execution Time

137.92 233.2 128.48 79.6 201.8

Bad Bad Bad Good Good

49

slide-50
SLIDE 50

Inputs Trace Statistics Method Statistics Bad Traces Good Traces Profiler ICA Trace Clustering GA Analyzer Execution Trace Analyzer App

Good traces [SGood] Bad traces [SBad]

= × X A S

: p×m : p×k : k×m traces methods features traces methods features

50

slide-51
SLIDE 51

Inputs Trace Statistics Method Statistics Bad Traces Bottlenecks Good Traces Profiler Contrast Mining ICA Trace Clustering GA Analyzer Execution Trace Analyzer App

For each method 𝐸 = (𝑇𝐻𝑝𝑝𝑒 − 𝑇𝐶𝑏𝑒)

  • 1. Agilefant.model.WidgetCollection.getName() 11.783
  • 2. Agilefant.db.hibernate.UserTypeFilter.deepCopy() 10.662
  • 3. Agilefant.model.Team.setId() 8.112
  • 4. Agilefant.model.Backlog.setChildren() 7.932
  • 5. ……

51

slide-52
SLIDE 52

Research Questions (RQs)

  • RQ1 - How effective is GA-Prof in

finding inputs leading to bottlenecks

  • RQ2 - How effective is GA-Prof in

identifying bottlenecks

  • RQ3 - Is GA-Prof more effective than

FOREPOST in identifying bottlenecks

52

slide-53
SLIDE 53

Research Question 1

How effective is GA-Prof in finding inputs leading to bottlenecks GA-Prof vs. Random

53

slide-54
SLIDE 54

Research Question 1

How effective is GA-Prof in finding inputs leading to bottlenecks GA-Prof vs. Random H0: There is no statistical difference between GA-Prof and Random

54

slide-55
SLIDE 55

Research Question 2

How effective is GA-Prof in identifying bottlenecks Inject nine artificial performance bottlenecks

55

slide-56
SLIDE 56

Research Question 3

Is GA-Prof more effective than FOREPOST in identifying bottlenecks GA-Prof vs. FOREPOST

56

slide-57
SLIDE 57

Experimental Design

JPetStore Dell DVD Store Agilefant

57

slide-58
SLIDE 58

Experimental Design

users URLs per user times to repeat

58

slide-59
SLIDE 59

RQ RQ1 – Finding Bottleneck-Specific Inputs

Generations Elapsed Execution Time

2 4 6 8 10 12 14 16 18 20 22 24 26 28 6 8 10

JPetStore

59

slide-60
SLIDE 60

Generations Elapsed Execution Time

1 2 3 8 10 12 14 16 18 20 22 24 26 28

Dell DVD Store

60

9.5 9.0 8.5 8.0

RQ RQ1 – Finding Bottleneck-Specific Inputs

slide-61
SLIDE 61

Generations Elapsed Execution Time

2 4 6 8 10 12 14 16 18 20 22 24 26 28

Agilefant

61

25 50 75

RQ RQ1 – Finding Bottleneck-Specific Inputs

slide-62
SLIDE 62

Elapsed Execution Time Random GA-Prof Random GA-Prof Random GA-Prof 6 8 10 9.5 9.0 8.5 8.0 JPetStore Dell DVD Store Agilefant

62

25 50 75

RQ RQ1 – Finding Bottleneck-Specific Inputs

slide-63
SLIDE 63

25 50 75 Elapsed Execution Time Random GA-Prof Random GA-Prof Random GA-Prof 6 8 10 9.5 9.0 8.5 8.0 JPetStore Dell DVD Store Agilefant 𝒒 = 𝟐. 𝟔𝒇 − 𝟑𝟐 𝒒 = 𝟑. 𝟘𝒇 − 𝟒𝟏 𝒒 = 𝟕. 𝟓𝒇 − 𝟐𝟖

63

RQ RQ1 – Finding Bottleneck-Specific Inputs

slide-64
SLIDE 64

Rankings Generations 15 10 5 10 20

RQ RQ2 – Finding In Inje jected Bottlenecks

Injected bottleneck: Jpetstore.domain.Pro duct.getName()

64

slide-65
SLIDE 65

Rankings Generations 15 10 5 10 20

RQ RQ2 – Finding In Inje jected Bottlenecks

Injected bottleneck: Jpetstore.domain.Pro duct.getName()

65

slide-66
SLIDE 66

RQ RQ2 – Finding In Inje jected Bottlenecks

66

10 20 20 40 60 10 20 50 100 250 200 150 10 20 25 50 75 10 20 5 10 15 10 20 10 30 20 10 20 10 40 30 20 10 20 10 30 20 10 20 20 40 60 80 10 20 50 100 200 150

Rankings of Nine Injected bottlenecks X axis – rankings, Y axis – number of generations

slide-67
SLIDE 67

RQ RQ3

3 – GA

GA-Prof vs. FOREPOST

1 2 3 4 5 6 JPetStore Dell DVD Store 2.2 2.6 FOREPOST

Can GA-Prof capture more injected bottlenecks? Number of captured bottlenecks

67

slide-68
SLIDE 68

Can GA-Prof capture more injected bottlenecks?

1 2 3 4 5 6 JPetStore Dell DVD Store 2.2 2.6 5.6 4.6 FOREPOST GA-Prof

RQ RQ3

3 – GA

GA-Prof vs. FOREPOST

Number of captured bottlenecks

68

slide-69
SLIDE 69

Can GA-Prof rank injected performance bottlenecks higher?

50 100 150 JPetStore Dell DVD Store 145.98 14.8 FOREPOST

RQ RQ3

3 – GA

GA-Prof vs. FOREPOST

Average rankings

69

slide-70
SLIDE 70

Can GA-Prof rank injected performance bottlenecks higher?

50 100 150 JPetStore Dell DVD Store 145.98 14.8 13.78 10.94 FOREPOST GA-Prof

RQ RQ3

3 – GA

GA-Prof vs. FOREPOST

Average rankings

70

slide-71
SLIDE 71

Online appendix: http://www.cs.wm.edu/semeru/data/ISSTA15-GAProf/

71

slide-72
SLIDE 72

72

slide-73
SLIDE 73

73

slide-74
SLIDE 74

74

slide-75
SLIDE 75

75

slide-76
SLIDE 76

Additional Sli lides for Questions

76

slide-77
SLIDE 77

GAs – In Independent Variables

  • Crossover rate – 0.3
  • Mutation rate – 0.1
  • Number of individuals per generation – 30
  • Termination Criterion

Maximum limit for the number of generations – 30 Average fitness value of every individual in one generation

77

slide-78
SLIDE 78

In Inje ject Artificial Performance Bottlenecks

  • Run applications without artificial bottlenecks
  • Obtain a ranked list of methods
  • Randomly inject nine artificial bottlenecks (each one as a

same delay)

  • Delays are chosen experimentally

78