END-TO-END COMPARISON OF IR-LEVEL AND ASSEMBLY-LEVEL FAULT INJECTION - - PowerPoint PPT Presentation

▶
end to end comparison of ir level and
SMART_READER_LITE
LIVE PREVIEW

END-TO-END COMPARISON OF IR-LEVEL AND ASSEMBLY-LEVEL FAULT INJECTION - - PowerPoint PPT Presentation

Feb 06, 2023 285 likes •854 views

Practical Experience Report A TALE OF TWO INJECTORS: END-TO-END COMPARISON OF IR-LEVEL AND ASSEMBLY-LEVEL FAULT INJECTION Lucas Palazzi Co-authors: Guanpeng Li, Bo Fang, and Karthik Pattabiraman DEPART M ENT OF ELECT RIC AL AN D COM PUT ER

slide-1
SLIDE 1

A TALE OF TWO INJECTORS: END-TO-END COMPARISON OF IR-LEVEL AND ASSEMBLY-LEVEL FAULT INJECTION

Lucas Palazzi

Co-authors: Guanpeng Li, Bo Fang, and Karthik Pattabiraman

DEPART M ENT OF ELECT RIC AL AN D COM PUT ER ENGINEER I NG T HE UNIVERSIT Y OF BRIT ISH COLUM BI A

Practical Experience Report

slide-2
SLIDE 2

2

MOTIVATION: SOFT ERRORS Becoming more common in processors

0111 0011

Photo source: http://aviral.lab.asu.edu/soft-error-resilience/

slide-3
SLIDE 3

3

SOFT ERROR OUTCOMES

  • 1. Benign error
  • 2. Crash
  • 3. Silent data corruption (SDC)
slide-4
SLIDE 4

4

SOFT ERROR OUTCOMES

  • 1. Benign error
  • 2. Crash
  • 3. Silent data corruption (SDC) ✲ e.g., integer sort program

1, 4, 6, 8, 10

Error-free program output:

6, 4, 1, 8, 10

SDC program output:

slide-5
SLIDE 5

5

FAULT INJECTION

Fault Injection (FI)

Program Benign error probability Crash probability SDC probability

slide-6
SLIDE 6

6

FI AT DIFFERENT LEVELS OF ABSTRACTION

Device/Circuit Gate/RTL Microarchitecture Instruction Set Architecture Software/Application

Software-implemented FI (SWiFI) Hardware-level FI

slide-7
SLIDE 7

7

SOFTWARE-IMPLEMENTED FI (SWiFI)

IR-level FI Assembly-level FI (Intermediate Representation)

slide-8
SLIDE 8

8

CODE COMPILATION EXAMPLE x86 Assembly LLVM IR C Source

slide-9
SLIDE 9

9

TRADE-OFFS OF DIFFERENT SWiFI TECHNIQUES

Accuracy Convenience

slide-10
SLIDE 10

10

TRADE-OFFS OF DIFFERENT SWiFI TECHNIQUES

Accuracy Convenience

1 1 Assembly-level FI

slide-11
SLIDE 11

11

TRADE-OFFS OF DIFFERENT SWiFI TECHNIQUES

Accuracy Convenience

1 ?? 2 ?? 1 Assembly-level FI 2 IR-level FI

slide-12
SLIDE 12

12

TRADE-OFFS OF DIFFERENT SWiFI TECHNIQUES

Accuracy Convenience

1 1 Assembly-level FI 2 IR-level FI

A A

DSN14 [1]

[1] Wei et al. DSN’14.

(SDCs)

slide-13
SLIDE 13

13

TRADE-OFFS OF DIFFERENT SWiFI TECHNIQUES

Accuracy Convenience

1 1 Assembly-level FI 2 IR-level FI

A A

DSN14 [1]

[1] Wei et al. DSN’14.

B SC17 [2]

[2] Georgakoudis et al. SC’17.

B

(SDCs)

slide-14
SLIDE 14

14

TRADE-OFFS OF DIFFERENT SWiFI TECHNIQUES

Accuracy Convenience

1 1 Assembly-level FI 2 IR-level FI

A A

DSN14 [1]

[1] Wei et al. DSN’14.

B SC17 [2]

[2] Georgakoudis et al. SC’17.

B

(SDCs)

?

slide-15
SLIDE 15

15

PRIOR WORK: SUMMARY

1 https://github.com/DependableSystemsLab/LLFI 2 https://github.com/DependableSystemsLab/PINFI

slide-16
SLIDE 16

16

PRIOR WORK: SUMMARY

  • Both studies use LLFI1 (IR-level) and PINFI2 (assembly-level)
  • SC17 uses a modified version of PINFI

1 https://github.com/DependableSystemsLab/LLFI 2 https://github.com/DependableSystemsLab/PINFI

slide-17
SLIDE 17

17

PRIOR WORK: SUMMARY

  • Both studies use LLFI1 (IR-level) and PINFI2 (assembly-level)
  • SC17 uses a modified version of PINFI
  • DSN14 (Wei et al.)
  • LLFI is as accurate as PINFI for measuring SDC probabilities

1 https://github.com/DependableSystemsLab/LLFI 2 https://github.com/DependableSystemsLab/PINFI

slide-18
SLIDE 18

18

PRIOR WORK: SUMMARY

  • Both studies use LLFI1 (IR-level) and PINFI2 (assembly-level)
  • SC17 uses a modified version of PINFI
  • DSN14 (Wei et al.)
  • LLFI is as accurate as PINFI for measuring SDC probabilities
  • SC17 (Georgakoudis et al.)
  • LLFI is not as accurate as PINFI, even for SDCs
  • Attributed differences to limitations of LLFI (e.g., back-end optimizations)

1 https://github.com/DependableSystemsLab/LLFI 2 https://github.com/DependableSystemsLab/PINFI

slide-19
SLIDE 19

19

RESEARCH QUESTIONS

slide-20
SLIDE 20

20

RESEARCH QUESTIONS

  • 1. Why does prior work come to contradictory findings?
slide-21
SLIDE 21

21

RESEARCH QUESTIONS

  • 1. Why does prior work come to contradictory findings?
  • 2. What is the accuracy of IR-level FI compared to assembly-level FI?

2.1 SDCs 2.2 Crashes

slide-22
SLIDE 22

22

RESEARCH QUESTIONS

  • 1. Why does prior work come to contradictory findings?
  • 2. What is the accuracy of IR-level FI compared to assembly-level FI?

2.1 SDCs 2.2 Crashes

slide-23
SLIDE 23

23

PRIOR WORK ANALYSIS: DSN14 VS. SC17

slide-24
SLIDE 24

24

PRIOR WORK ANALYSIS: DSN14 VS. SC17

1 https://github.com/DependableSystemsLab/PINFI 2 https://github.com/DependableSystemsLab/LLFI

Reproduce SC17 results

Assembly-level: PINFI1 IR-level: LLFI2

slide-25
SLIDE 25

25

PRIOR WORK ANALYSIS: DSN14 VS. SC17

1 https://github.com/DependableSystemsLab/PINFI 2 https://github.com/DependableSystemsLab/LLFI

Reproduce SC17 results Isolate differences

Assembly-level: PINFI1 IR-level: LLFI2 Setup, benchmarks, FI tools

slide-26
SLIDE 26

26

PRIOR WORK ANALYSIS: DSN14 VS. SC17

1 https://github.com/DependableSystemsLab/PINFI 2 https://github.com/DependableSystemsLab/LLFI

Reproduce SC17 results Isolate differences Pinpoint exact cause

Assembly-level: PINFI1 IR-level: LLFI2 Setup, benchmarks, FI tools ???

slide-27
SLIDE 27

27

PRIOR WORK ANALYSIS: DSN14 VS. SC17

LLFI Official version used by both DSN14 and SC17 PINFI Official version hosted on GitHub

SDC Probability

Benchmarks

slide-28
SLIDE 28

28

PRIOR WORK ANALYSIS: DSN14 VS. SC17

LLFI Official version used by both DSN14 and SC17 PINFI-v1 Official version hosted on GitHub (same as DSN14) PINFI-v2 Modified version used in SC17 (publicly available)

SDC Probability

Benchmarks

slide-29
SLIDE 29

29

BIT-SAMPLING METHODOLOGY

e.g., x86 double-precision floating-point instructions (addsd dsd, mulsd sd, etc.)

slide-30
SLIDE 30

30

BIT-SAMPLING METHODOLOGY

e.g., x86 double-precision floating-point instructions (addsd dsd, mulsd sd, etc.)

slide-31
SLIDE 31

31

BIT-SAMPLING METHODOLOGY

e.g., x86 double-precision floating-point instructions (addsd dsd, mulsd sd, etc.)

slide-32
SLIDE 32

32

BIT-SAMPLING METHODOLOGY

e.g., x86 double-precision floating-point instructions (addsd dsd, mulsd sd, etc.)

PINFI-v1

(DSN14)

slide-33
SLIDE 33

33

BIT-SAMPLING METHODOLOGY

e.g., x86 double-precision floating-point instructions (addsd dsd, mulsd sd, etc.)

PINFI-v2

(SC17)

slide-34
SLIDE 34

34

PRIOR WORK ANALYSIS: DSN14 VS. SC17

LLFI Official version used by both DSN14 and SC17 PINFI-v1 Official version hosted on GitHub (same as DSN14) PINFI-v2 Version used in SC17 (publicly available)

SDC Probability

Benchmarks

slide-35
SLIDE 35

35

PRIOR WORK ANALYSIS: DSN14 VS. SC17

LLFI Official version used by both DSN14 and SC17 PINFI-v1 Official version hosted on GitHub (same as DSN14) PINFI-v2 Version used in SC17 (publicly available) PINFI-v3 PINFI-v1, modified to match bit-sampling methodology of PINFI-v2

SDC Probability

Benchmarks

slide-36
SLIDE 36

36

WHY DOES THIS MATTER?

  • Affects results significantly
  • Depends on desired fault model

Important to stay consistent in comparison studies!

slide-37
SLIDE 37

37

Application [2] Device/Circuit [1]

SC17 (PINFI-v2) DSN14 (PINFI-v1, LLFI)

Photo source: https://pdfs.semanticscholar.org/c052/8c02f566d211f9bd90b7c1d3703256fad053.pdf

“fault sensitivity” [1] vs “error sensitivity” [2]

slide-38
SLIDE 38

38

RESEARCH QUESTIONS

  • 1. Why does prior work come to contradictory findings?

An invalid comparison in SC17 due to an inconsistent bit-sampling model

  • 2. What is the accuracy of IR-level FI compared to assembly-level FI?

2.1 SDCs: 2.2 Crashes:

slide-39
SLIDE 39

39

RESEARCH QUESTIONS

  • 1. Why does prior work come to contradictory findings?

An invalid comparison in SC17 due to an inconsistent bit-sampling model

  • 2. What is the accuracy of IR-level FI compared to assembly-level FI?

2.1 SDCs: 2.2 Crashes:

slide-40
SLIDE 40

40

END-TO-END EVALUATION

  • Extensive FI comparison study (LLFI vs. PINFI)
  • 25 benchmarks (incl. most from DSN14 and SC17)
  • 4 LLVM optimization levels (-O0, -O1, -O2, -O3)
  • Three statistical tests (linear reg., t-test, Spearman’s rank)

Are IR-level SDC/crash probability measurements accurate?

slide-41
SLIDE 41

41

LINEAR REGRESSION ANALYSIS Ideal case: Linear equation y = x

slide-42
SLIDE 42

42

LINEAR REGRESSION ANALYSIS

Program SDC Probabilities at –O3

0% 20% 40% 60% 80% 0% 20% 40% 60% 80%

PINFI Crash Probability LLFI Crash Probability

Program Crash Probabilities at –O3

slide-43
SLIDE 43

43

OVERALL FINDINGS

Optimization

O0 O1 O2 O3

PINFI

Accuracy (IR-Leve FI)

slide-44
SLIDE 44

44

OVERALL FINDINGS

Accuracy (IR-Leve FI) Optimization

O0 O1 O2 O3

PINFI

SDCs

Findings are consistent with DSN14 results

slide-45
SLIDE 45

45

OVERALL FINDINGS

Optimization

O0 O1 O2 O3

PINFI

SDCs

Findings are consistent with DSN14 results

Crashes Accuracy (IR-Leve FI)

slide-46
SLIDE 46

46

WHAT ABOUT CRASHES?

  • Back-end optimizations
  • Memory operations (e.g., register allocation)
  • Predominant source of crashes: segmentation faults [Fang et al., DSN16]
slide-47
SLIDE 47

47

WHAT ABOUT CRASHES? Application

Memory map

  • Back-end optimizations
  • Memory operations (e.g., register allocation)
  • Predominant source of crashes: segmentation faults [Fang et al., DSN16]
slide-48
SLIDE 48

48

WHAT ABOUT CRASHES? Application

Memory map

  • Back-end optimizations
  • Memory operations (e.g., register allocation)
  • Predominant source of crashes: segmentation faults [Fang et al., DSN16]
slide-49
SLIDE 49

49

WHAT ABOUT CRASHES? Application

Memory map

CRASH

  • Back-end optimizations
  • Memory operations (e.g., register allocation)
  • Predominant source of crashes: segmentation faults [Fang et al., DSN16]
slide-50
SLIDE 50

50

RESEARCH QUESTIONS

  • 1. Why does prior work come to contradictory findings?

An invalid comparison in SC17 due to an inconsistent bit-sampling model

  • 2. What is the accuracy of IR-level FI compared to assembly-level FI?

2.1 SDCs: IR-level FI is accurate across all optimization levels 2.2 Crashes: IR-level FI is not accurate; accuracy gets worse with optimizations

slide-51
SLIDE 51

51

RESEARCH QUESTIONS

  • 1. Why does prior work come to contradictory findings?

An invalid comparison in SC17 due to an inconsistent bit-sampling model

  • 2. What is the accuracy of IR-level FI compared to assembly-level FI?

2.1 SDCs: IR-level FI is accurate across all optimization levels 2.2 Crashes: IR-level FI is not accurate; accuracy gets worse with optimizations

slide-52
SLIDE 52

52

FUTURE WORK

  • Extending comparison to other platforms (e.g., ARM)
  • Evaluate accuracy across individual optimizations
  • Improve accuracy of IR-level FI for crashes
slide-53
SLIDE 53

53

FUTURE WORK

  • Extending comparison to other platforms (e.g., ARM)
  • Evaluate accuracy across individual optimizations
  • Improve accuracy of IR-level FI for crashes
slide-54
SLIDE 54

54

SUMMARY OF CONTRIBUTIONS

  • Settled confusion on accuracy of IR-level FI (DSN14 vs SC17)
  • Highlight the importance of clearly defining FI parameters
  • Re-establish confidence in IR-level FI for SDCs
  • Quantify accuracy (SDCs/crashes) across optimization levels
slide-55
SLIDE 55

55

SUMMARY OF CONTRIBUTIONS

  • Settled confusion on accuracy of IR-level FI (DSN14 vs SC17)
  • Highlight the importance of clearly defining FI parameters
  • Re-establish confidence in IR-level FI for SDCs
  • Quantify accuracy (SDCs/crashes) across optimization levels

Data and tools are publicly available: https://github.com/DependableSystemsLab/ISSRE19/

Thank you!

lpalazzi@ece.ubc.ca

slide-56
SLIDE 56