Enhancing Memory Error Detection for Large-Scale Applications and - - PowerPoint PPT Presentation

▶
enhancing memory error detection for large scale
SMART_READER_LITE
LIVE PREVIEW

Enhancing Memory Error Detection for Large-Scale Applications and - - PowerPoint PPT Presentation

Mar 18, 2024 177 likes •661 views

Enhancing Memory Error Detection for Large-Scale Applications and Fuzz testing Wookhyun Han , Byunggil Joe, Byoungyoung Lee * , Chengyu Song , Insik Shin KAIST, * Purdue, UCR 1 Memory error glibc: getaddrinfo Heartbleed Shellshock

slide-1
SLIDE 1

Enhancing Memory Error Detection for Large-Scale Applications and Fuzz testing

Wookhyun Han, Byunggil Joe, Byoungyoung Lee*, Chengyu Song†, Insik Shin KAIST, *Purdue, †UCR

1

slide-2
SLIDE 2

Memory error

  • Information leakage – Heartbleed
  • Privilege escalation – Shellshock
  • Remote code execution – Shellshock, glibc, Conficker

2

Heartbleed Shellshock glibc: getaddrinfo stack-based buffer

  • verflow
slide-3
SLIDE 3

Memory error detection

  • Pointer-based [SoftBound+CETS, Intel MPX]
  • Hardware support (cannot detect temporal memory errors)
  • Challenges to support complex applications
  • Redzone-based [AddressSanitizer (ASan)]
  • Compatible to complex applications
  • Most popular in practice

 Google Chrome, Mozilla Firefox, Linux Kernel  American Fuzzy Lop (AFL), ClusterFuzz, OSS-Fuzz

3

slide-4
SLIDE 4

Redzone-based memory error detection

  • Buffer overflow (spatial memory errors)

4

  • bjX

ptrX Shadow memory Check before access Accessible

Shadow memory: a bitmap to validate all addresses

slide-5
SLIDE 5

Redzone-based memory error detection

  • Buffer overflow (spatial memory errors)

4

  • bjX

ptrX Shadow memory Check before access Inaccessible (redzone) Accessible

Shadow memory: a bitmap to validate all addresses Redzone: inaccessible region between objects

slide-6
SLIDE 6

Redzone-based memory error detection

  • Buffer overflow (spatial memory errors)

4

  • bjX

ptrX Shadow memory Inaccessible (redzone) Accessible

Shadow memory: a bitmap to validate all addresses Redzone: inaccessible region between objects

Error!

slide-7
SLIDE 7

Redzone-based memory error detection

  • Use-after-free (temporal memory errors)

5

  • bjX

Inaccessible Accessible ptrX Shadow memory

slide-8
SLIDE 8

Redzone-based memory error detection

  • Use-after-free (temporal memory errors)

5

  • bjX

Inaccessible Accessible

Quarantined

ptrX ptrX free(ptrX) Shadow memory

Region is invalidated and quarantined, but not actually deallocated

slide-9
SLIDE 9

Redzone-based memory error detection

  • Use-after-free (temporal memory errors)

5

  • bjX

Inaccessible Accessible

Quarantined

ptrX ptrX free(ptrX) Shadow memory

Hold the region until quarantine zone is full (FIFO)

slide-10
SLIDE 10

Redzone-based memory error detection

  • Use-after-free (temporal memory errors)

5

  • bjX

Inaccessible Accessible

Quarantined

ptrX ptrX

  • bjY

ptrY free(ptrX) ptrY = malloc() Shadow memory

The region is actually deallocated, and can be allocated to a new object

slide-11
SLIDE 11

Limitations of redzone-based approach

6

  • bjX

ptrX

  • bjY
  • 1. What if a pointer

accesses beyond redzone?

slide-12
SLIDE 12

Limitations of redzone-based approach

6

  • bjX

ptrX

Spatial memory error

  • bjY
  • 1. What if a pointer

accesses beyond redzone?

slide-13
SLIDE 13

Limitations of redzone-based approach

6

  • bjX

ptrX

Spatial memory error

  • bjY

ptrX

  • 1. What if a pointer

accesses beyond redzone?

  • 2. What if a dangling pointer

accesses after another object is allocated in the region?

  • bjX
slide-14
SLIDE 14

Limitations of redzone-based approach

6

  • bjX

ptrX

Spatial memory error Temporal memory error

  • bjY

ptrX

  • 1. What if a pointer

accesses beyond redzone?

  • 2. What if a dangling pointer

accesses after another object is allocated in the region?

  • bjX

ptrX

  • bjZ
slide-15
SLIDE 15

Limitations of redzone-based approach

6

  • bjX

ptrX

Spatial memory error Temporal memory error

  • bjY

ptrX

  • 1. What if a pointer

accesses beyond redzone?

  • 2. What if a dangling pointer

accesses after another object is allocated in the region?

  • bjX

ptrX

  • bjZ

Cannot detect!

slide-16
SLIDE 16

Motivation

  • To enhance detectability of redzone-

based memory error detection

  • P1. Large gap to detect spatial memory errors
  • P2. Large quarantine zone to detect temporal

memory errors

7

  • bj1

P1 P1

  • bj1

P1

slide-17
SLIDE 17

Motivation

  • To enhance detectability of redzone-

based memory error detection

  • P1. Large gap to detect spatial memory errors
  • P2. Large quarantine zone to detect temporal

memory errors

7

  • bj1

P2 P1 P1

  • bj1

P1 P2

  • bj1
  • bj2
slide-18
SLIDE 18

Motivation

  • To enhance detectability of redzone-

based memory error detection

  • P1. Large gap to detect spatial memory errors
  • P2. Large quarantine zone to detect temporal

memory errors

7

  • bj1

P2 P1 P1

  • bj1

P1 P2

Huge physical memory required

  • bj1
  • bj2
slide-19
SLIDE 19

MEDS overview

  • Enhances detectability of redzone-based memory error

detection

  • Idea: Fully utilize 64-bit virtual address space to support
  • P1. Large gap to detect spatial error
  • P2. Large quarantine zone to detect temporal error
  • Approach: minimize physical memory use
  • Page aliasing allocator and page protection
  • Hierarchical memory error detection

8

slide-20
SLIDE 20

Page aliasing (P1)

  • Maps multiple virtual pages to single physical page

9

Allocated Redzone A memory page Page aliasing

  • bj1
  • bj2
  • bj4

Virtual

slide-21
SLIDE 21

Page aliasing (P1)

  • Maps multiple virtual pages to single physical page

9

Allocated Redzone A memory page Page aliasing

  • bj1
  • bj1
  • bj3
  • bj2
  • bj4

Physical Virtual

  • bj2
  • bj4
slide-22
SLIDE 22

Page aliasing (P1)

  • Maps multiple virtual pages to single physical page

9

Allocated Redzone A memory page Page aliasing

  • bj1
  • bj1
  • bj3
  • bj2
  • bj4

Physical Virtual

Redzone itself does not

  • ccupy physical memory
  • bj2
  • bj4
slide-23
SLIDE 23

Page protection (P1)

  • Redzone only pages are unmapped

10

Allocated Redzone A memory page Page aliasing

  • bj1
  • bj1
  • bj3
  • bj2

Physical Virtual

  • bj4
  • bj2

Unmapped page

slide-24
SLIDE 24

Page protection (P1)

  • Redzone only pages are unmapped

10

Allocated Redzone A memory page Page aliasing

  • bj1
  • bj1
  • bj3
  • bj2

Physical Virtual

  • bj4
  • bj2

Do not occupy shadow memory and physical memory

Unmapped page

slide-25
SLIDE 25

Page aliasing & Page protection (P2)

11

  • bj1
  • bj1
  • bj3
  • bj4

Physical Virtual

  • bj2
  • bj4

Allocated Redzone A memory page Page aliasing Unmapped page

slide-26
SLIDE 26

Page aliasing & Page protection (P2)

11

  • bj1
  • bj1
  • bj3
  • bj4

Physical Virtual

  • bj2
  • bj4
  • bj3
  • bj4

Physical Virtual

Quarantined

  • bj2
  • bj4

Allocated Redzone A memory page Page aliasing Unmapped page

slide-27
SLIDE 27

Page aliasing & Page protection (P2)

11

  • bj1
  • bj1
  • bj3
  • bj4

Physical Virtual

  • bj2
  • bj4
  • bj3
  • bj4

Physical Virtual

Quarantined

  • bj2
  • bj4
  • bjX
  • bjX

Allocated Redzone A memory page Page aliasing Unmapped page

slide-28
SLIDE 28

Page aliasing & Page protection (P2)

11

  • bj1
  • bj1
  • bj3
  • bj4

Physical Virtual

  • bj2
  • bj4
  • bj3
  • bj4

Physical Virtual

Quarantined

  • bj2
  • bj4
  • bjX
  • bjX

Allocated Redzone A memory page Page aliasing Unmapped page

Reuse physical memory immediately, while not reusing virtual addresses

slide-29
SLIDE 29

Hierarchical memory error detection

  • Many different ways to represent redzones

 Further optimizing physical memory uses

ptr

12

slide-30
SLIDE 30

Hierarchical memory error detection

  • Many different ways to represent redzones

 Further optimizing physical memory uses

ptr

12

#1. Shadow memory is invalid

slide-31
SLIDE 31

Hierarchical memory error detection

  • Many different ways to represent redzones

 Further optimizing physical memory uses

ptr

12

#1. Shadow memory is invalid #2. Virtual page is unmapped

slide-32
SLIDE 32

Hierarchical memory error detection

  • Many different ways to represent redzones

 Further optimizing physical memory uses

ptr

12

#1. Shadow memory is invalid #2. Virtual page is unmapped #3. Shadow memory is unmapped

slide-33
SLIDE 33

Evaluation

  • Configuration
  • ASan cannot use configuration for MEDS (lack of memory)
  • Compatibility
  • Performance: 2 times slowdown
  • Detection (fuzz testing): 68% more detection

13

ASan MEDS Improv. Redzone 8-1024 bytes 4MB 16,384x Quarantine 128MB 80TB 65,536x

slide-34
SLIDE 34

Compatibility

  • Unit tests from real-world applications
  • Test cases in Chrome, Firefox, Nginx
  • All Passed
  • Memory error unit tests
  • ASan unit tests
  • All Passed
  • NIST Juliet test suites
  • All Passed except random access tests

 ASan: 35% vs. MEDS: 98%

14

slide-35
SLIDE 35

Micro-scale performance overhead

  • TLB misses
  • 5 times more than ASan (more virtual pages with page aliasing)
  • Number of system calls
  • mmap(), munmap(), and mremap()
  • 32 times more than ASan (page aliasing and page protection)
  • Memory footprint
  • 218% more than baseline
  • 68% more than ASan (much larger redzone and quarantine)

15

slide-36
SLIDE 36

1 2 3 4 Chrome Firefox Apache Nginx ASan MEDS

End-to-end performance overhead

  • 108% compared to baseline, 86% to ASan

16

Baseline

slide-37
SLIDE 37

1 2 3 4 Chrome Firefox Apache Nginx ASan MEDS

End-to-end performance overhead

  • 108% compared to baseline, 86% to ASan

16

Baseline

41% to baseline 22% to ASan

slide-38
SLIDE 38

1 2 3 4 Chrome Firefox Apache Nginx ASan MEDS

End-to-end performance overhead

  • 108% compared to baseline, 86% to ASan

16

Baseline

41% to baseline 22% to ASan Large number of small objects on stack 243% to baseline 211% to ASan

slide-39
SLIDE 39

Detection (fuzz testing)

  • Run AFL (8 cores, 6 hours)
  • Despite the performance overhead, explore 68.3% more

unique crashes than ASan

17

0.5 1 1.5 2 2.5 3 3.5 4

ASan

slide-40
SLIDE 40

Detection (fuzz testing)

  • Run AFL (8 cores, 6 hours)
  • Despite the performance overhead, explore 68.3% more

unique crashes than ASan

17

0.5 1 1.5 2 2.5 3 3.5 4

ASan

MEDS finds more unique crashes in initial phase, but saturated in the end

slide-41
SLIDE 41

10 20 30 40 50 60 70

1 2 3 4 5 6 7 8

Found crashes Time spent (hrs)

ASan MEDS

Detection (fuzz testing)

  • Number of unique crashes with time spent (metacam)

18

Saturated

slide-42
SLIDE 42

How MEDS explores more crashes?

  • More input sets can be detected
  • Higher probability to detect
  • Bugs can be found earlier than ASan
  • Fuzzer can focus on the other paths
  • MEDS can detect the cases that

ASan cannot detect

  • Always bypass redzone
  • e.g., Miscalculation of structure array size
  • Size of the structure is larger than redzone

size

  • Access to certain element cannot be detected.

19

int a[10]; a[x] = x; struct A { int num[10]; }; struct A *a = malloc(sizeof(struct A)); ... (a+i)->num[8] = i;

slide-43
SLIDE 43

How MEDS explores more crashes?

  • More input sets can be detected
  • Higher probability to detect
  • Bugs can be found earlier than ASan
  • Fuzzer can focus on the other paths
  • MEDS can detect the cases that

ASan cannot detect

  • Always bypass redzone
  • e.g., Miscalculation of structure array size
  • Size of the structure is larger than redzone

size

  • Access to certain element cannot be detected.

19

int a[10]; a[x] = x; struct A { int num[10]; }; struct A *a = malloc(sizeof(struct A)); ... (a+i)->num[8] = i;

slide-44
SLIDE 44

How MEDS explores more crashes?

  • More input sets can be detected
  • Higher probability to detect
  • Bugs can be found earlier than ASan
  • Fuzzer can focus on the other paths
  • MEDS can detect the cases that

ASan cannot detect

  • Always bypass redzone
  • e.g., Miscalculation of structure array size
  • Size of the structure is larger than redzone

size

  • Access to certain element cannot be detected.

19

int a[10]; a[x] = x; struct A { int num[10]; }; struct A *a = malloc(sizeof(struct A)); ... (a+i)->num[8] = i;

slide-45
SLIDE 45

How MEDS explores more crashes?

  • More input sets can be detected
  • Higher probability to detect
  • Bugs can be found earlier than ASan
  • Fuzzer can focus on the other paths
  • MEDS can detect the cases that

ASan cannot detect

  • Always bypass redzone
  • e.g., Miscalculation of structure array size
  • Size of the structure is larger than redzone

size

  • Access to certain element cannot be detected.

19

int a[10]; a[x] = x; struct A { int num[10]; }; struct A *a = malloc(sizeof(struct A)); ... (a+i)->num[8] = i;

slide-46
SLIDE 46

Conclusion

  • Idea
  • Support large gap and large quarantine zone
  • Approach
  • Page aliasing and page protection
  • Hierarchical memory error detection
  • Despite overhead (108%), MEDS finds more crashes during

fuzz testing (68.3%)

  • Open source – will be available soon
  • https://github.com/purdue-secomp-lab/MEDS
  • Please use to detect bugs

20

slide-47
SLIDE 47

Thank you for listening!

21