Consistent, Durable, and Safe Memory Management for Byte-Addressable - - PowerPoint PPT Presentation

Consistent, Durable, and Safe Memory Management for Byte-Addressable Non-Volatile Main Memory

Iulian Moraru*, David Andersen*, Michael Kaminsky°, Niraj Tolia , Nathan Binkert, Partha Ranganathan

*Carnegie Mellon University, °Intel Labs, Maginatics, Amiato, Google

♢ ‡

♦

♢

‡

♦

Sunday, November 3, 13

New memory technologies

2

1 1 10 100 1000 10000 100000

Access time [ns] logscale DRAM Flash

x x

Sunday, November 3, 13

NVRAM: phase change, spin torque, memristor

• Memory that is both fast and non-volatile
• Will help build fast, robust systems

New memory technologies

2

1 1 10 100 1000 10000 100000

Access time [ns] logscale DRAM Flash

x x

Sunday, November 3, 13

NVRAM: phase change, spin torque, memristor

• Memory that is both fast and non-volatile
• Will help build fast, robust systems

NVRAM much faster than HDD and Flash

New memory technologies

2

1 1 10 100 1000 10000 100000

Access time [ns] logscale DRAM Flash

x x

Sunday, November 3, 13

NVRAM: phase change, spin torque, memristor

• Memory that is both fast and non-volatile
• Will help build fast, robust systems

NVRAM much faster than HDD and Flash

New memory technologies

2

1 1 10 100 1000 10000 100000

Access time [ns] logscale DRAM Flash

x x

Sunday, November 3, 13

NVRAM: phase change, spin torque, memristor

• Memory that is both fast and non-volatile
• Will help build fast, robust systems

NVRAM much faster than HDD and Flash

New memory technologies

2

x

PCM

1 1 10 100 1000 10000 100000

Access time [ns] logscale DRAM Flash

x x

Sunday, November 3, 13

NVRAM: phase change, spin torque, memristor

• Memory that is both fast and non-volatile
• Will help build fast, robust systems

NVRAM much faster than HDD and Flash

New memory technologies

2

x

PCM

1 1 10 100 1000 10000 100000

Access time [ns] logscale DRAM Flash

x x

• Placed on the memory bus (alongside DRAM)

Sunday, November 3, 13

NVRAM: phase change, spin torque, memristor

• Memory that is both fast and non-volatile
• Will help build fast, robust systems

NVRAM much faster than HDD and Flash

New memory technologies

2

x

PCM

1 1 10 100 1000 10000 100000

Access time [ns] logscale DRAM Flash

x x

• Placed on the memory bus (alongside DRAM)
• Needs new ways of handling persistent state

Sunday, November 3, 13

NV main memory - assumptions

• Mapped into process address space
• Accessed through CPU loads/stores
• Persistent namespace

3

DRAM + NVRAM Applications

OS

Sunday, November 3, 13

NV main memory - assumptions

• Mapped into process address space
• Accessed through CPU loads/stores
• Persistent namespace

3

DRAM + NVRAM Applications

OS Persistence Library

Sunday, November 3, 13

NVRAM - challenge

➡ Give developers the right tools to develop data

structures for NVRAM High performance and Robustness (no data loss)

4

Sunday, November 3, 13

Sources of data loss

5

Sunday, November 3, 13

Sources of data loss

Wear-out

5

Sunday, November 3, 13

Sources of data loss

Wear-out

5

Erroneous writes

Sunday, November 3, 13

Sources of data loss

Wear-out

5

Erroneous writes

CPU caches

Sunday, November 3, 13

Sources of data loss

Wear-out

5

Erroneous writes

CPU caches ...

Sunday, November 3, 13

Sources of data loss

Wear-out

5

Erroneous writes

CPU caches

Memory allocator

...

Sunday, November 3, 13

Sources of data loss

Wear-out

5

Erroneous writes

CPU caches

Memory allocator Virtual memory protection

...

Sunday, November 3, 13

Sources of data loss

Wear-out

5

Erroneous writes

CPU caches

Memory allocator Virtual memory protection Cache line counters

...

Sunday, November 3, 13

Malloc for NVRAM - goals

6

Sunday, November 3, 13

Malloc for NVRAM - goals

• Avoid frequently re-allocating the same block
• HW wear leveling may slow down applications that

write often to one location

6

Sunday, November 3, 13

Malloc for NVRAM - goals

• Avoid frequently re-allocating the same block
• HW wear leveling may slow down applications that

write often to one location

• Minimize # of metadata updates in NVRAM
• Wear-out & speed

6

Sunday, November 3, 13

Malloc for NVRAM - goals

• Avoid frequently re-allocating the same block
• HW wear leveling may slow down applications that

write often to one location

• Minimize # of metadata updates in NVRAM
• Wear-out & speed
• Make metadata robust to accidental corruption
• Avoids extensive loss of (persistent) data

6

Sunday, November 3, 13

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

7

1 1 1

Sunday, November 3, 13

3

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

7

1 1 1

Sunday, November 3, 13

3

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

7

1 1 1

size

Sunday, November 3, 13

3 1

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

2

7

1 1 1

size

Sunday, November 3, 13

3 1

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

2 1 1

7

1 1 1

size

Sunday, November 3, 13

3 1

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

2 1 1 1

7

1 1 1

size

Sunday, November 3, 13

3 1

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

2 1 1 1

7

1 1 1

size

Sunday, November 3, 13

3 1 1 2

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

1

7

1 1 1

size

Sunday, November 3, 13

3 1 2 1 2 3

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

7

1 1 1

size

Sunday, November 3, 13

3 1 2 1 2 3 1 2 3

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

7

1 1 1

size

Sunday, November 3, 13

3 1 2 1 2 3 1 2 3

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

7

1 1 1

• Metadata corruption can cause data loss

size

Sunday, November 3, 13

3 1 2 1 2 3 1 2 3

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

7

1 1 1

• Metadata corruption can cause data loss

size

Sunday, November 3, 13

3 1 2 1 2 3 1 2 3

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

7

1 3 1 1

• Metadata corruption can cause data loss

size

Sunday, November 3, 13

3 1 2 1 2 3 1 2 3

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

7

1 3

• Metadata corruption can cause data loss

size

Sunday, November 3, 13

3 1 2 1 2 3 1 2 3

Problems with allocators for DRAM

• Reuse recently freed blocks
• More writes than necessary
• Metadata is embedded in allocated / free blocks
• Frequent writes to one location

7

1 3

X X

• Metadata corruption can cause data loss

size

Sunday, November 3, 13

NVMalloc

8

NVRAM DRAM

Sunday, November 3, 13

NVMalloc

8

NVRAM DRAM

Sunday, November 3, 13

free / in-use bitmap

000000...

NVMalloc

8

NVRAM DRAM

Sunday, November 3, 13

free / in-use bitmap

000000... 010001...

NVMalloc

8

NVRAM DRAM

Sunday, November 3, 13

free / in-use bitmap

000000... 010001...

NVMalloc

8

NVRAM DRAM ...

size 1:

...

size 2:

...

size 3: free lists

Sunday, November 3, 13

free / in-use bitmap

000000... 010001...

NVMalloc

8

NVRAM DRAM ...

size 1:

...

size 2:

...

size 3: free lists

Sunday, November 3, 13

free / in-use bitmap

000000... 010001...

NVMalloc

8

NVRAM DRAM ...

size 1:

...

size 2:

...

size 3: free lists

Sunday, November 3, 13

free / in-use bitmap

000000... 010001...

NVMalloc

8

NVRAM DRAM ...

size 1:

...

size 2:

...

size 3: free lists header: state, size, checksum

Sunday, November 3, 13

free / in-use bitmap

000000... 010001...

NVMalloc

8

NVRAM DRAM ...

size 1:

...

size 2:

...

size 3: free lists header: state, size, checksum

Sunday, November 3, 13

free / in-use bitmap

000000... 010001...

NVMalloc

8

NVRAM DRAM ...

size 1:

...

size 2:

...

size 3: free lists header: state, size, checksum

...

Sunday, November 3, 13

free / in-use bitmap

000000... 010001...

NVMalloc

8

NVRAM DRAM ...

size 1:

...

size 2:

...

size 3: free lists header: state, size, checksum don’t allocate list:

...

Sunday, November 3, 13

• Freed blocks “don’t allocate list”
• At least T seconds (e.g. T = 0.2 s)
• Block will not be allocated more often than 1/T
• Keep most metadata in DRAM
• In-DRAM bitmap tracks blocks state
• In-DRAM free lists accelerate allocations
• Checksum in each block header
• Checksum over: state, size, location

Our solution: NVMalloc

9

Sunday, November 3, 13

Memory allocator

Building blocks

10

Memory allocator Virtual memory protection Virtual memory protection Cache line counters

Sunday, November 3, 13

Memory allocator

Building blocks

10

Virtual memory protection Cache line counters

Sunday, November 3, 13

Problem: erroneous writes

Address space code data stack Persistent data Narrow interface File system interface

11

Sunday, November 3, 13

Problem: erroneous writes

Address space code data stack Persistent data

11

Sunday, November 3, 13

Problem: erroneous writes

Address space code data stack Persistent data

11

x x x x x x x

erroneous

• verwrite

Sunday, November 3, 13

Problem: erroneous writes

Address space code data stack Persistent data

Solution: virtual memory protection

11

x x x x x x x

erroneous

• verwrite

Sunday, November 3, 13

Problem: erroneous writes

Address space code data stack Persistent data

Solution: virtual memory protection

11

x x x x x x x

erroneous

• verwrite

(if we can make it fast enough)

Sunday, November 3, 13

• Today’s sloution: mprotect
• Used for protecting in-memory databases (since

1990’s)

• Used today: in-memory databases, JVM, garbage

collecting

Virtual memory protection

12

Sunday, November 3, 13

• Today’s sloution: mprotect
• Used for protecting in-memory databases (since

1990’s)

• Used today: in-memory databases, JVM, garbage

collecting

Virtual memory protection

12

High performance penalties:

• synchronous
• syscall overhead
• one request at a time

Sunday, November 3, 13

Asynchronous mprotect

• No waiting
• No system call overhead
• Batching, sorting
• Mprotect cheaper on ranges

User process Kernel thread

13

r e q u e s t b u f

Sunday, November 3, 13

Un-protect: possible scenarios

14

time

x protect request un-protect request x

Sunday, November 3, 13

Un-protect: possible scenarios

14

time

x protect request un-protect request x Coalesce

Sunday, November 3, 13

Un-protect: possible scenarios

14

time

x protect request un-protect request x Coalesce

time

un-protect request x done un-protect x write x

Sunday, November 3, 13

Un-protect: possible scenarios

14

time

x protect request un-protect request x Coalesce

time

un-protect request x done un-protect x write x Async, if sufficiently in advance of write (e.g. GC)

Sunday, November 3, 13

Un-protect: possible scenarios

14

time

x protect request un-protect request x Coalesce

time

un-protect request x done un-protect x write x Async, if sufficiently in advance of write (e.g. GC)

time

un-protect request x write x

Sunday, November 3, 13

Un-protect: possible scenarios

14

time

x protect request un-protect request x Coalesce

time

un-protect request x done un-protect x write x Async, if sufficiently in advance of write (e.g. GC)

time

un-protect request x write x Handle page fault in kernel, transparently

Sunday, November 3, 13

Cache line counters Virtual memory protection Memory allocator

Building blocks

15

Virtual memory protection Cache line counters

Sunday, November 3, 13

Cache line counters Virtual memory protection Memory allocator

Building blocks

15

Sunday, November 3, 13

• CPU caching: important optimization for DRAM
• Even more so for NVRAM
• No guaranteed order for cache line write-back

Write Caching for NVRAM

16

Sunday, November 3, 13

• CPU caching: important optimization for DRAM
• Even more so for NVRAM
• No guaranteed order for cache line write-back

Write Caching for NVRAM

16

Sunday, November 3, 13

• CPU caching: important optimization for DRAM
• Even more so for NVRAM
• No guaranteed order for cache line write-back

Write Caching for NVRAM

16

1

Sunday, November 3, 13

• CPU caching: important optimization for DRAM
• Even more so for NVRAM
• No guaranteed order for cache line write-back

Write Caching for NVRAM

16

1 2

Sunday, November 3, 13

• CPU caching: important optimization for DRAM
• Even more so for NVRAM
• No guaranteed order for cache line write-back

Write Caching for NVRAM

16

Pointer may be flushed first

1 2

Sunday, November 3, 13

• CPU caching: important optimization for DRAM
• Even more so for NVRAM
• No guaranteed order for cache line write-back

Write Caching for NVRAM

16

• Goals:
• Avoid forcing cache line flushes (costly)
• Enforce ordering

Pointer may be flushed first

1 2

Sunday, November 3, 13

Cache line counters

17

• Make applications aware of cache state
• Enforce ordering in software

How many cache lines updated by a transaction are still in cache?

Sunday, November 3, 13

Evaluation

• Testing setup:
• Core i7 860 (2.8 GHz), 8 GB DRAM
• DRAM as proxy for NVRAM

18

Sunday, November 3, 13

Results: wear leveling

500 1000 1500 2000 2500 3000 3500 4000 4500 5000 50000 100000 150000 Total writes / 64B block Block number malloc NVMalloc

100K operations (10B-4KB, 50% deallocations)

Writes / 64 B block

19

Sunday, November 3, 13

Results: wear leveling

500 1000 1500 2000 2500 3000 3500 4000 4500 5000 50000 100000 150000 Total writes / 64B block Block number malloc NVMalloc

100K operations (10B-4KB, 50% deallocations) malloc concentrates writes & 30%-50% more writes overall than NVMalloc

Writes / 64 B block

19

Sunday, November 3, 13

20 40 60 80 100 1 10 100 1000 Percentage of cache lines Number of writes per 64 byte block NVMalloc malloc

NVMalloc in B+ tree

1M insert operations

% of all 64 B blocks

20

99 100

(log scale)

Sunday, November 3, 13

• Async. mprotect in B+ tree

2 4 6 8 Sync Async 200ms Async 0.1ms

Slowdown

9.0x 1.7x 7.9x 1M inserts, 256B values

21

(vs no mprotect)

Latency (median) 200ms 0.1ms Latency (max) 350ms 1.2ms

ideal

Sunday, November 3, 13

• Async. mprotect in B+ tree

2 4 6 8 Sync Async 200ms Async 0.1ms

Slowdown

9.0x 1.7x 7.9x 1M inserts, 256B values

21

(vs no mprotect)

different batching quanta

Latency (median) 200ms 0.1ms Latency (max) 350ms 1.2ms

ideal

Sunday, November 3, 13

DRAM as NVRAM proxy

How would our results change on NVRAM?

• Same malloc / NVMalloc access patterns
• But NVMalloc writes less frequently to NVRAM
• Same asynchronous mprotect overhead
• But lower relative overhead b/c NVRAM is slower

22

Sunday, November 3, 13

• Challenges of NVRAM on the memory bus
• Wear-out
• Erroneous writes
• CPU caches
• Building blocks for NVRAM data stores

Summary

23

Async mprotect NVMalloc Persistent data stores CLC

Sunday, November 3, 13

Code

24

github.com/efficient/nvram

Sunday, November 3, 13

Related Work

[BPFS] J. Condit, et al.! Better I/O Through Byte- Addressable, Persistent Memory. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, SOSP ’09. 2009. [Mnemosyne] H. Volos, A. J. Tack, M. M. Swift. Mnemosyne: Lightweight Persistent Memory. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS’11. 2011. [NV-Heaps] J. Coburn, et al. NV-Heaps: Making Persistent Objects Fast and Safe with Next-Generation, Non- Volatile

• Memories. In Proceeding of the 16th Inter- national Conference on Architectural Support for Programming Languages

and Operating Systems, ASPLOS’11. 2011. [Rio-Crashes] P. M. Chen, et al. The Rio File Cache: Surviving Operating System Crashes. In ASPLOS, ASPLOS- VII. 1996. [Start-Gap] M. K. Qureshi, et al. Enhancing Lifetime and Security of PCM-based Main Memory with Start-Gap Wear

• Leveling. In Proceedings ACM MICRO. 2009.

[Wear-level-malicious] M. K. Qureshi, et al. Practical and Secure PCM Systems by Online Detection of Malicious Write

• Streams. In Proceedings HPCA. 2011.

[BTree-NVBM] S.Venkataraman, et al. ConsistentandDurableData Structures for Non-Volatile Byte-Addressable

• Memory. In Proceedings of the 9th USENIX Conference on File and Storage Technologies (FAST ’11). 2011.

[FlexSC] L. Soares, M. Stumm. FlexSC: Flexible system call scheduling with exception-less system calls. In Proceedings 9th USENIX OSDI. 2010.

25

Sunday, November 3, 13