Reduction of Operating System Jitter Caused by Page Reclaim - - PowerPoint PPT Presentation

reduction of operating system jitter
SMART_READER_LITE
LIVE PREVIEW

Reduction of Operating System Jitter Caused by Page Reclaim - - PowerPoint PPT Presentation

Reduction of Operating System Jitter Caused by Page Reclaim Yoshihiro Oyama 1,3 Shun Ishiguro 1 Jun Murakami 1 Shin Sasaki 1 Ryo Matsumiya 1 Osamu Tatebe 2,3 1. The University of Electro-Communications 2. University of Tsukuba 3. Japan Science


slide-1
SLIDE 1

Reduction of Operating System Jitter Caused by Page Reclaim

Yoshihiro Oyama1,3 Shun Ishiguro1 Jun Murakami1 Shin Sasaki1 Ryo Matsumiya1 Osamu Tatebe2,3

  • 1. The University of Electro-Communications
  • 2. University of Tsukuba
  • 3. Japan Science and Technology Agency
slide-2
SLIDE 2

Background

  • OS jitter: interference into applications by OS

– Services by OS kernel

  • E.g., interrupt handling and tasklets

– Daemon processes developed to provide OS services

  • E.g., memory management daemons
  • Jitter degrades application performance

– It deprives applications from computing resources such as CPU and memory

  • Minimizing the impact of jitter is critical in HPC
slide-3
SLIDE 3

Jitter Focused in This Study

  • We focus on jitter observed when application

frequently executes disk I/O of large data

– Footprint of file data exceeds the physical memory size – Kernel must discard page cache or swap out processes to

  • btain free memory

– Overhead is imposed on memory allocation operations

  • This jitter has not attracted much attention, but HPC

people should be aware of its potential impact

slide-4
SLIDE 4

Overview of This Study

  • 1. We clarify the impact of the jitter caused by

page reclaim

– Target OS is Linux

  • 2. We propose a mechanism for minimizing the

impact

– It increases the amount of page cache released at

  • ne time

– It reduces the number of page reclaim operations

slide-5
SLIDE 5

memory

Page Cache

application application application access read disk disk blocks

slide-6
SLIDE 6

Memory Pressure by Page Cache

application application application memory disk

slide-7
SLIDE 7

Memory Pressure by Page Cache

application application application memory disk

Page reclaim frequently occurs when:

  • Pages are consumed fast
  • Only a small number of pages are released

at one time

slide-8
SLIDE 8

Memory Pressure by Page Cache

application application application memory disk

Page reclaim frequently occurs when:

  • Pages are consumed fast
  • Only a small number of pages are released

at one time

slide-9
SLIDE 9

Page Reclaim in Linux

  • Memory pages are running short
  • > it immediately reclaims memory (direct reclaim)
  • > it awakens a kernel thread kswapd
  • kswapd reclaims pages by flushing page cache or swapping

process memory

  • Two values inside kswapd are particularly important

– Freemem threshold

  • Kswapd is awakened if the amount of free memory falls below this

threshold

– Watermark

  • Kswapd continues to reclaim pages until the amount of free pages

exceeds this value Modifiable indirectly through /proc/sys/vm/min_free_kbytes Unfortunately, it is the only parameter effective in minimizing page reclaim jitter

slide-10
SLIDE 10

Page Reclaim by kswapd

normal state page reclaim started page reclaim finished freemem threshold reclaimed pages memory objects page cache free watermark

slide-11
SLIDE 11

Proposed Mechanism (1)

  • Introduces new kernel module and kernel thread

– Starts page reclaim before kswapd – Reclaims larger # of pages at once

normal state page reclaim started page reclaim finished kswapd freemem threshold reclaimed pages kswapd watermark

  • ur

freemem threshold

  • ur

watermark

slide-12
SLIDE 12

System Structure

kernel thread kernel

memory

disk Read

Monitors disk-read requests Monitors # of free pages Invokes page reclaim function

/proc/min_free_pages /proc/reclaimed_pages

Specifies threshold

  • f free pages

Specifies # of pages reclaimed at once

slide-13
SLIDE 13

Proposed Mechanism (2)

  • It starts when both conditions are satisfied:

– Cond. 1: # of free pages < our freemem threshold – Cond. 2: our mechanism determines that memory shortage is caused by frequent I/O

  • Otherwise, our kernel thread does not start

– And eventually kswapd will be awakened

  • We expect kswapd will do a good job in minimizing

page-outs of memory objects

slide-14
SLIDE 14

Discussion

  • Q: Why introducing a kernel thread, instead of

customizing kswapd?

– Tuning kswapd parameters – Modifying kswapd code

  • A: Kswapd provides only a few parameters

– For example, kswapd users cannot directly specify the amount of reclaimed memory – But, we would like to investigate a vast space of parameters and algorithms – This inconvenience is also pointed out by another Linux engineer: https://lwn.net/Articles/422291/

slide-15
SLIDE 15

Experiments

  • We measured the impact of jitter on

the performance of a scientific application

  • Application: WRF (weather forecasting software)

– Simulated the weather around Japan in one hour (6 s x 600 steps)

  • Jitter generator

– Program that repeatedly reads a 100-GB file sequentially

  • Although it represents an extreme case, we believe that a

similar case can possibly occur in some configurations and job sets

slide-16
SLIDE 16

Condition

11 threads InfiniBand QDR 4X, MPI Node 1

...

11 threads Node 2

...

11 threads Node 3

...

11 threads Node 4

...

Jitter generator Machine specification: CPU: Intel Xeon E5645 2.4 GHz (6 cores) x 2 Memory: 48 GB HDD: SAS 15,000 rpm

slide-17
SLIDE 17

Experiment 1

  • We compared WRF performance in 3 cases

– Original – With jitter – With jitter and proposed mechanism (Jitter+Proposed)

slide-18
SLIDE 18

2 4 6 8 10 12 14 16 18 20 50 100 150 200 250

Computation time of each step (s) Step

Original

Original

Result (Not Using Proposed Mechanism)

slide-19
SLIDE 19

2 4 6 8 10 12 14 16 18 20 50 100 150 200 250

Computation time of each step (s) Step

Original Jitter

Result (Not Using Proposed Mechanism)

slide-20
SLIDE 20

Accumulated Computation Time (Not Using Proposed Mechanism)

1000 2000 3000 4000 5000 6000 7000

Original Jitter Jitter+Proposed (4 GiB reclaim) Jitter+Proposed (48 GiB reclaim) Computation time (s)

26.6% slowdown because of jitter!

slide-21
SLIDE 21

2 4 6 8 10 12 14 16 18 20 50 100 150 200 250

Computation time of each step (s) Step

Original Jitter

Result (Using Proposed Mechanism)

slide-22
SLIDE 22

2 4 6 8 10 12 14 16 18 20 50 100 150 200 250

Computation time of each step (s) Step

Original Jitter Jitter+Proposed (4 GiB reclaim)

Result (Using Proposed Mechanism)

slide-23
SLIDE 23

2 4 6 8 10 12 14 16 18 20 50 100 150 200 250

Computation time of each step (s) Step

Original Jitter Jitter+Proposed (4 GiB reclaim) Jitter+Proposed (48 GiB reclaim)

Result (Using Proposed Mechanism)

slide-24
SLIDE 24

Accumulated Computation Time (Using Proposed Mechanism)

1000 2000 3000 4000 5000 6000 7000

Original Jitter Jitter+Proposed (4 GiB reclaim) Jitter+Proposed (48 GiB reclaim) Computation time (s)

Only 1.9% slowdown

26.6% slowdown

slide-25
SLIDE 25

Experiment 2

  • In addition, we must answer

– “How good performance can we get by changing parameters of kswapd?” – “Is kswapd parameter tuning sufficient to obtain comparative performance?”

  • We measured WRF performance in Jitter case

with various kswapd parameters

slide-26
SLIDE 26

Effect of kswapd Parameter Changes

2 4 6 8 10 12 14 16 18 20 50 100 150 200 250

Computation time of each step (s) Step

Original Jitter (kswapd threshold: 88 MiB)

slide-27
SLIDE 27

Effect of kswapd Parameter Changes

2 4 6 8 10 12 14 16 18 20 50 100 150 200 250

Computation time of each step (s) Step

Original Jitter (kswapd threshold: 88 MiB) Jitter (kswapd threshold: 2 GiB) Jitter (kswapd threshold: 4 GiB)

slide-28
SLIDE 28

Effect of kswapd Parameter Changes

1000 2000 3000 4000 5000 6000 7000 Original Jitter Jitter+Proposed (4 GiB reclaim) Jitter+Proposed (48 GiB reclaim) Jitter (2 GiB threshold) Jitter (4 GiB threshold)

Computation time (s)

+12.8% +14.4%

slide-29
SLIDE 29

Related Work

  • “Core separation” approaches

– [De et al. IPDPS 2009], [Oral et al. 2010], [Rosenthal et al. 2013], [Seelam et al. IPDPS 2011] – Executes the kernel and daemons on dedicated CPU cores – Executes applications on remaining CPU cores – Prevents the kernel and daemons from depriving applications of CPU resources It is unclear how many CPU cores are sufficient for hosting kswapd threads and other system tasks – Their approach should be combined with another approach for reducing the impact of jitter

slide-30
SLIDE 30

Summary and Future Work

  • Summary

– We proposed a mechanism for reducing the impact of jitter caused by page reclaim – Jitter caused by an I/O-intensive process increased the execution time of WRF by 26.6% – The mechanism lowered the increase to 1.9%

  • Future Work

– Understanding jitter caused by reading many small files or by writing to a file – Improving the proposed mechanism in order to monitor accesses to files on remote I/O nodes – Analyzing the experimental results in more detail