GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne , Jonathan - PowerPoint PPT Presentation

GPrioSwap: Towards a Swapping Policy for GPUs Jens Kehne , Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa 10 th ACM International Systems and Storage Conference Operating Systems Group, Karlsruhe Institute of Technology (KIT) www.kit.edu KIT – The Research University in the Helmholtz Association

Motivation GPUs are widespread in computing Unprecedented performance for some applications Very energy efficient GPUs are moving to the cloud Cost effective through oversubscription Can safely share computational power Even have fairness to some degree But what about memory? 2 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Our Approach: GPUSwap (VEE '15) 3 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Our Approach: GPUSwap (VEE '15) STOP 4 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Our Approach: GPUSwap (VEE '15)  Achieves both fairness and good utilization 5 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

GPUSwap: Swapping Policy Choose app with most GPU memory („The Victim“) Achieves fairness Choose chunk of memory from victim‘s AS How do we find the right chunk? No reference bit on current GPUs! Original implementation: Random 6 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Results: Runtime Overhead (lud) 5 0 4 0 Runtime [s] 3 0 2 0 1 0 0 1 0 0 2 0 0 3 0 0 4 0 0 5 0 0 GPU memory [MiB] 7 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Analysis: Methodology No easy way to count page accesses No reference bit No page faults Performance counters only count entire application Virtual AS Idea: Separate single page from rest of AS Accurate access count for each page System RAM GPU memory Must rerun application once per page 8 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Analysis: Results (bfs) Read Write Address space Large variance between application buffers Little variance within each buffer Not shown: Large stack buffer, close to zero accesses Similar results for other applications  Finding the right buffer to swap is probably enough 9 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

GPrioSwap Operates in two steps Offline step Profile application Assign a priority to each buffer Online step (on memory pressure) Find set of chunks with lowest priority from victim‘s AS Select one chunk from set at random 10 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Offline Step Profile application as before Re-run once per buffer rather than per page Calculate avg. number of accesses per page Assign buffer priorities based on averages Pass priorities as parameter during allocation Requires changes to application code 11 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Swapping Policy (Online Step) Select victim (application with most GPU memory) 1 5 3 2 5 4 2 3 1 5 2 1 4 Find all chunks with lowest priority 12 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Swapping Policy (Online Step) Select victim (application with most GPU memory) 1 5 3 2 5 4 2 3 1 5 2 1 4 Find all chunks with lowest priority Select one low-priority chunk at random Repeat until enough chunks have been selected 13 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Swapping Policy (Online Step) Select victim (application with most GPU memory) 1 5 3 2 5 4 2 3 5 2 1 4 Find all chunks with lowest priority Select one low-priority chunk at random Repeat until enough chunks have been selected Swap all selected chunks Service allocation request 14 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Results: Backprop 1,8 1,6 1,4 Runtime [s] 1,2 1 Random Best random 0,8 Prio 0,6 0,4 0,2 0 0 20 40 60 80 100 120 140 160 GPU Memory [MiB] 15 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Results: Heartwall 2 out of 9 applications: 1,8 1,6 1,4 Runtime [s] 1,2 1 Random Best random 0,8 Prio 0,6 0,4 0,2 0 0 50 100 150 200 250 GPU Memory [MiB] 16 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology

Conclusion We can efficiently swap GPU data at runtime But we do not yet know what to swap Importance of pages varies by buffer Profile applications, assign buffer priorities Swap from low-priority buffers first 17 23.05.17 GPrioSwap: Towards a Swapping Policy for GPUs Operating Systems Group Jens Kehne, Jonathan Metter, Martin Merkel, Marius Hillenbrand, Marc Rittinghaus, Frank Bellosa Karlsruhe Institute of Technology