GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne , Jonathan - - PowerPoint PPT Presentation

▶

Oct 30, 2023 356 likes •535 views

GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne , Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 10 th ACM International Systems and Storage Conference Operating Systems Group, Karlsruhe Institute of

SLIDE 1

KIT – The Research University in the Helmholtz Association

Operating Systems Group, Karlsruhe Institute of Technology (KIT)

www.kit.edu

GPrioSwap: Towards a Swapping Policy for GPUs

Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 10th ACM International Systems and Storage Conference

SLIDE 2

Operating Systems Group Karlsruhe Institute of Technology 2 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Motivation

GPUs are widespread in computing

Unprecedented performance for some applications Very energy efficient

GPUs are moving to the cloud

Cost effective through oversubscription

Can safely share computational power

Even have fairness to some degree

But what about memory?

SLIDE 3

Operating Systems Group Karlsruhe Institute of Technology 3 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Our Approach: GPUSwap (VEE '15)

SLIDE 4

Operating Systems Group Karlsruhe Institute of Technology 4 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Our Approach: GPUSwap (VEE '15)

STOP

SLIDE 5

Operating Systems Group Karlsruhe Institute of Technology 5 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Our Approach: GPUSwap (VEE '15)  Achieves both fairness and good utilization

SLIDE 6

Operating Systems Group Karlsruhe Institute of Technology 6 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

GPUSwap: Swapping Policy

Choose app with most GPU memory („The Victim“)

Achieves fairness

Choose chunk of memory from victim‘s AS How do we find the right chunk? No reference bit on current GPUs! Original implementation: Random

SLIDE 7

Operating Systems Group Karlsruhe Institute of Technology 7 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Results: Runtime Overhead (lud)

1 2 3 4 5 1 2 3 4 5 Runtime [s] GPU memory [MiB]

SLIDE 8

Operating Systems Group Karlsruhe Institute of Technology 8 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Analysis: Methodology

No easy way to count page accesses

No reference bit No page faults

Performance counters only count entire application Idea: Separate single page from rest of AS Accurate access count for each page Must rerun application once per page

Virtual AS System RAM GPU memory

SLIDE 9

Operating Systems Group Karlsruhe Institute of Technology 9 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Analysis: Results (bfs)

Address space

Write Read

Large variance between application buffers Little variance within each buffer Not shown: Large stack buffer, close to zero accesses Similar results for other applications  Finding the right buffer to swap is probably enough

SLIDE 10

Operating Systems Group Karlsruhe Institute of Technology 10 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

GPrioSwap

Operates in two steps Offline step

Profile application Assign a priority to each buffer

Online step (on memory pressure)

Find set of chunks with lowest priority from victim‘s AS Select one chunk from set at random

SLIDE 11

Operating Systems Group Karlsruhe Institute of Technology 11 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Offline Step

Profile application as before

Re-run once per buffer rather than per page

Calculate avg. number of accesses per page Assign buffer priorities based on averages Pass priorities as parameter during allocation

Requires changes to application code

SLIDE 12

Operating Systems Group Karlsruhe Institute of Technology 12 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Swapping Policy (Online Step)

Select victim (application with most GPU memory) Find all chunks with lowest priority

5 3 2 5 4 2 3 5 1 2 4 1 1

SLIDE 13

Operating Systems Group Karlsruhe Institute of Technology 13 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Swapping Policy (Online Step)

Select victim (application with most GPU memory) Find all chunks with lowest priority Select one low-priority chunk at random Repeat until enough chunks have been selected

5 3 2 5 4 2 3 5 1 2 4 1 1

SLIDE 14

Operating Systems Group Karlsruhe Institute of Technology 14 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Swapping Policy (Online Step)

Select victim (application with most GPU memory) Find all chunks with lowest priority Select one low-priority chunk at random Repeat until enough chunks have been selected Swap all selected chunks Service allocation request

5 3 2 5 4 2 3 5 2 4 1 1

SLIDE 15

Operating Systems Group Karlsruhe Institute of Technology 15 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Results: Backprop

20 40 60 80 100 120 140 160 0,2 0,4 0,6 0,8 1 1,2 1,4 1,6 1,8 Random Best random Prio

GPU Memory [MiB] Runtime [s]

SLIDE 16

Operating Systems Group Karlsruhe Institute of Technology 16 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Results: Heartwall

50 100 150 200 250 0,2 0,4 0,6 0,8 1 1,2 1,4 1,6 1,8 Random Best random Prio

GPU Memory [MiB] Runtime [s] 2 out of 9 applications:

SLIDE 17

Operating Systems Group Karlsruhe Institute of Technology 17 GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 23.05.17

Conclusion

We can efficiently swap GPU data at runtime But we do not yet know what to swap Importance of pages varies by buffer Profile applications, assign buffer priorities Swap from low-priority buffers first