A Method to Evaluate CFG Comparison Algorithms Patrick P.F. Chan - - PowerPoint PPT Presentation

▶
a method to evaluate cfg comparison algorithms
SMART_READER_LITE
LIVE PREVIEW

A Method to Evaluate CFG Comparison Algorithms Patrick P.F. Chan - - PowerPoint PPT Presentation

Nov 13, 2022 136 likes •655 views

A Method to Evaluate CFG Comparison Algorithms Patrick P.F. Chan Christian Collberg Research problem Which CFG similarity algorithm is better? I come up with a new algorithm, how does it compare to the existing ones? Is there a

slide-1
SLIDE 1

A Method to Evaluate CFG Comparison Algorithms

Patrick P.F. Chan Christian Collberg

slide-2
SLIDE 2

Research problem

  • Which CFG similarity algorithm is better?
  • I come up with a new algorithm, how does it

compare to the existing ones?

  • Is there a systematic way to compare CFG

similarity algorithms?

slide-3
SLIDE 3

Research outcomes

  • A methodology to evaluate and compare CFG

similarity algorithms

  • Comparison results of four CFG similarity

algorithms

  • A survey of existing CFG similarity algorithms
  • A publicly available evaluation framework
slide-4
SLIDE 4

What is CFG?

  • CFG stands for control-flow graph
  • A CFG represents all possible execution paths of a

function

  • And thus, it encodes its behavior
slide-5
SLIDE 5

a = input() if a % 2 == 0 print “even” print “odd”

Entry Exit

True False

slide-6
SLIDE 6

Why do we compare CFGs?

slide-7
SLIDE 7

Why do we compare CFGs?

  • Malware detection / classification

CFGs of malware Match

slide-8
SLIDE 8

Why do we compare CFGs?

  • Software theft detection

Original software Suspected pirated software How similar?

slide-9
SLIDE 9

Why do we compare CFGs?

  • Programming assignments grading

Assignment Submission Solution How similar?

slide-10
SLIDE 10

Why do we compare CFGs?

  • Code clones detection

How similar?

slide-11
SLIDE 11

Why do we compare CFGs?

  • Detection of changes between different versions of

a program

slide-12
SLIDE 12

Why do we compare CFGs?

  • Detection of changes between different versions of

a program

Match the nodes of the enhanced CFGs

slide-13
SLIDE 13

This leads to many algorithms to compare CFGs…

slide-14
SLIDE 14

Let’s use two existing algorithms to compare these two CFGs

1 2 3 4

1 2 3 4 5

CFG A CFG B

slide-15
SLIDE 15

Algorithm 1 from Kruegel et al.

  • Extract subgraphs that have k nodes (k-subgraphs)

from CFGs and match them

slide-16
SLIDE 16

1 2 3 4

1 2 3 4 5

CFG A CFG B

1 2 4 1 2 3 1 2 4 1 2 5 1 2 3 2 4 5

No match!

slide-17
SLIDE 17

Algorithm 2 from Hu et al.

  • Approximates the minimum number of edit
  • perations needed to transform one graph into

another graph

slide-18
SLIDE 18

Cost of matching node 1 of CFG A to node 1 of CFG B Cost of deleting node 1 of CFG B Cost of deleting node 4 of CFG B Cost of matching dummy nodes Cost of deleting nodes in CFG B Cost of deleting nodes in CFG A Cost of matching nodes

1 2 3 4

1 2 3 4 5

CFG A CFG B

slide-19
SLIDE 19

1 2 3 4

1 2 3 4 5

CFG A CFG B

Total cost = 5

slide-20
SLIDE 20

And there are many other algorithms…

  • Algorithm from Vujosˇevic

́-Janicˇic ́ et al. iteratively builds a similarity matrix between the nodes of the two CFGs, based on the similarity of their neighbor

  • Algorithm from Sokolsky et al. models the control

flow graphs using Labeled Transition Systems (LTS)

slide-21
SLIDE 21

But which one is the best?

slide-22
SLIDE 22

Evaluation of CFG similarity algorithms

  • Start by generating CFGs G1, G2,...,Gi with increasing edit

distances with respect to a seed CFG G0

  • i.e. ED(G0,Gi) = i
  • Use the algorithm under evaluation to rank the CFGs such

that the higher is the similarity score between Gi and G0 given by that algorithm, the higher Gi is ranked

  • Get a “goodness score” for the algorithm by comparing

the ranking it produces to the ground truth ⟨G1, G2, G3,...⟩, using ranking correlation algorithms such as sortedness or Pearson correlation

slide-23
SLIDE 23

Example

G0

slide-24
SLIDE 24

Example

G0 G1 G2 G3

ED = 1 ED = 2 ED = 3

slide-25
SLIDE 25

Example

G0 G1 G2 G3

ED = 1 ED = 2 ED = 3 Ranking: ⟨G1, G2, G3⟩

slide-26
SLIDE 26

Example

G0 G1 G2 G3

ED = 1 ED = 2 ED = 3 Ranking: ⟨G1, G2, G3⟩

G0 G1 G2 G3

SimA = 0.4 SimA = 0.1 SimA = 0.8

slide-27
SLIDE 27

Example

G0 G1 G2 G3

ED = 1 ED = 2 ED = 3 Ranking: ⟨G1, G2, G3⟩

G0 G1 G2 G3

SimA = 0.4 SimA = 0.1 SimA = 0.8 Ranking: ⟨G3, G1, G2⟩

slide-28
SLIDE 28

Example

G0 G1 G2 G3

ED = 1 ED = 2 ED = 3 Ranking: ⟨G1, G2, G3⟩

G0 G1 G2 G3

SimA = 0.4 SimA = 0.1 SimA = 0.8 Ranking: ⟨G3, G1, G2⟩ Pearson correlation = -0.5

slide-29
SLIDE 29

Two questions remain…

  • 1. What is the definition of the edit distance between

two CFGs?

  • 2. How to generate those CFGs such that they have

increasing edit distances with the seed CFG G0?

slide-30
SLIDE 30

What is the definition of the edit distance between two CFGs?

  • The Graph Edit Distance is a function ED : (Gi, Gj)

→ N that computes the smallest number of edit

  • perations needed to transform Gi into Gj.
  • There are four possible edit operations
slide-31
SLIDE 31

What is the definition of the edit distance between two CFGs?

  • Add a zero-degree node

1 2 3 4 1 2 3

slide-32
SLIDE 32

What is the definition of the edit distance between two CFGs?

  • Add an edge between two existing nodes

1 2 3 4 1 2 3 4

slide-33
SLIDE 33

What is the definition of the edit distance between two CFGs?

  • Delete an edge between two existing nodes

1 2 3 4 1 2 3 4

slide-34
SLIDE 34

What is the definition of the edit distance between two CFGs?

  • Delete a zero-degree node

1 2 3 4 1 2 4

slide-35
SLIDE 35

How to generate those CFGs such that they have increasing edit distances with the seed CFG G0?

a b c d

G0

slide-36
SLIDE 36

How to generate those CFGs such that they have increasing edit distances with the seed CFG G0?

a b c d a b c d a b c d

Add Edge

a b c d e

Add Node Add Edge

a b c d

Delete Edge

For every possible edit operation that can be applied to G0, apply that and generate a new graph

slide-37
SLIDE 37

How to generate those CFGs such that they have increasing edit distances with the seed CFG G0?

Do the same for the newly generated graphs Obtain the Edit Distance Graph (EDG)

slide-38
SLIDE 38

How to generate those CFGs such that they have increasing edit distances with the seed CFG G0?

a b c d a b c d a b c d a b c d

Add Edge

a b c d e

Add Node Add Edge

a b c d

Delete Edge Add Edge Add Edge

Randomly pick a CFG on each level and they become our G1, G2, G3,…

slide-39
SLIDE 39

Implementation

  • Re-coded four CFG similarity algorithms in Python
  • Implemented the evaluation framework
  • Generated an EDG with five levels
  • Picked 100 test cases (each test case comprises

five CFGs)

slide-40
SLIDE 40

Evaluation results

slide-41
SLIDE 41

Evaluation results

slide-42
SLIDE 42

Evaluation results

slide-43
SLIDE 43

Evaluation results

slide-44
SLIDE 44

Evaluation results

“Goodness score” statistics of the four algorithms

slide-45
SLIDE 45

Evaluation results

Time used by the four algorithms to finish 100 test cases

slide-46
SLIDE 46

Related work

  • An evaluation framework for text plagiarism

detection

  • Generate artificial plagiarism cases
  • Shuffling, removing, inserting, or replacing words
  • r short phrases at random
slide-47
SLIDE 47

Related work

  • An evaluation framework for code clone detection

tools

  • Inject mutated code fragments into the code

base

slide-48
SLIDE 48

Future work

  • Generate CFGs with instructions in the nodes

Edit instructions => huge EDG

slide-49
SLIDE 49

Try our framework

  • http://cfgsim.cs.arizona.edu/
  • Evaluate existing algorithms
  • Compare your own algorithm with the others
  • Fine tune your algorithm
slide-50
SLIDE 50

Summary

  • A methodology to evaluate CFG similarity

algorithms

  • Publicly available evaluation framework
  • Serves as a benchmark for CFG similarity

algorithms users / researchers

slide-51
SLIDE 51

Thank you!