A Case for Dynamically Programmable Storage Background Tasks - - PowerPoint PPT Presentation

▶
a case for dynamically programmable storage background
SMART_READER_LITE
LIVE PREVIEW

A Case for Dynamically Programmable Storage Background Tasks - - PowerPoint PPT Presentation

Oct 10, 2023 122 likes •435 views

A Case for Dynamically Programmable Storage Background Tasks Ricardo Macedo , Alberto Faria, Joo Paulo, Jos Pereira INESC TEC & University of Minho 38th IEEE International Symposium on Reliable Distributed Systems Workshops 1st Workshop

slide-1
SLIDE 1

A Case for Dynamically Programmable Storage Background Tasks

Ricardo Macedo, Alberto Faria, João Paulo, José Pereira

INESC TEC & University of Minho

38th IEEE International Symposium on Reliable Distributed Systems Workshops 1st Workshop on Distributed and Reliable Storage Systems Lyon, France October 1st 2019

slide-2
SLIDE 2

Motivation and Background

Modern storage infrastructures feature long and complex I/O paths

  • Composed by several layers, such as hypervisors, schedulers, databases, and file systems

Layers employ independent optimizations to serve applications

  • Partial visibility of the infrastructure inhibits optimal system-wide performance
  • High levels of I/O interference and performance degradation
  • Degradation amplifies when concurrent I/O services compete for shared resources

2

slide-3
SLIDE 3

Background tasks are predefined I/O tasks that can rapidly overload shared resources

  • Compaction, checkpointing, and replication
  • Introduce significant I/O interference and workload burstiness
  • Processed in best-effort manner to minimize interference with foreground workflows

The decision of when and how to execute such operations is taken by the layer itself, regardless of the overall load on the infrastructure

Motivation and Background

3

slide-4
SLIDE 4

To achieve optimal holistic performance, storage background tasks should be

dynamically programmable and their

execution handled in end-to-end fashion.

slide-5
SLIDE 5
  • Extensive evaluation of the impact of storage background tasks

○ Impact of compaction processes under HBase ○ Impact of checkpointing processes under PostgreSQL ○ Analysis of mean and tail latencies performance

  • Design of a programmable storage system to achieve optimal holistic

performance

Contributions

5

slide-6
SLIDE 6

Case Study: HBase

6

slide-7
SLIDE 7

Case Study: HBase

LSM-based design leads to the execution of background compactions

  • Merges several small-sized HFiles into fewer larger ones
  • While improving read performance, compactions introduce I/O interference and

burstiness

7

slide-8
SLIDE 8

Case Study: PostgreSQL

8

slide-9
SLIDE 9

Case Study: PostgreSQL

9

To truncate the log and allow fast recovery, PostgreSQL performs checkpoints

  • Flush dirty data pages in shared buffers to disk
  • Executed either when the WAL file is about to exceed a certain size or upon a timeout
  • Checkpoint completion target adjusts the throughput at which checkpoints are made
  • Lower bound leads to I/O burstiness and higher bound to longer recovery times
slide-10
SLIDE 10

Methodology

  • How much overhead do these background tasks impose?
  • How does their overhead vary across operation types?
  • How does their overhead vary across time?
  • How do these tasks impact tail latency?
  • How do these tasks’ configuration parameters influence their impact on

performance?

10

slide-11
SLIDE 11

Methodology

Testbed

  • HBase 2.0.5 pseudo-distributed mode, backed by HDFS 2.9.2
  • PostgreSQL 11.3 backed by an ext4 file system

11

Workloads*

  • Workload A: 50% read, 50% update, zipfian
  • Workload B: 100% update, zipfian
  • Workload C: 100% read, uniform
  • Workload D: 5% read, 95% insert, zipfian
  • Workload E: 95% scan, 5% insert, zipfian
  • Workload F: 50% read, 50% read-modify-write, zipfian

* Workloads previously used in [22]: “MeT: Workload aware elasticity for NoSQL”

Results publicly available at https://rgmacedo.github.io/drss19-website/

slide-12
SLIDE 12

Methodology

HBase deployments

  • With compaction effects. Execution phase immediately after loading phase
  • Without compaction effects. Execution phase after a waiting period

PostgreSQL deployments

  • 6 configurations with varying WAL size and checkpoint completion target
  • 128MiB and 1024MiB for maximum WAL size parameter
  • 0.1, 0.5, and 0.9 for checkpoint completion target parameter

12

slide-13
SLIDE 13

HBase Compactions

Mean latency results

13

slide-14
SLIDE 14

HBase Compactions

Mean latency results

14

Inherent dependency over files being compacted results in high I/O interference over read-oriented requests Read-oriented operations exhibit an overhead of at most 955.2% at the 99th percentile latency

53.2% 87.3%

slide-15
SLIDE 15

HBase Compactions

Mean latency results

15

Write operations are sequentially written to the WAL and then persisted at the Memstore Experienced compactions do not impose major disk overload and I/O interference Performance degradations of at most 40% at the 99th percentile latency for write operations

14.7% 27.6%

slide-16
SLIDE 16

HBase Compactions

Mean latency results

16

CPU utilization remains mostly unaltered Read and write disk throughput experience an increase of 33 and 15MiB/s, respectively Default throttling policy limits compaction throughput to reduce I/O interference

53.2% 87.3% 14.7% 27.6%

slide-17
SLIDE 17

PostgreSQL Checkpointing

Tail latency results

17

At 99th percentile latency, 0.1, 0.5, and 0.9 completion target configurations experience an

  • verhead of at most 61.9%, 33.2%, and 33.3%,

respectively Lower values result in I/O burstiness, inhibiting QoS provisioning and sustained performance Larger values throttle WAL write performance,

  • ccupying disk bandwidth for longer periods

99th percentile latency variation for the update operation for each PostgreSQL configuration, under Workload A with 1 thread

slide-18
SLIDE 18

PostgreSQL Checkpointing

Tail latency results

18

At 99th percentile latency, 0.1, 0.5, and 0.9 completion target configurations experience an

  • verhead of at most 61.9%, 33.2%, and 33.3%,

respectively Lower values result in I/O burstiness, inhibiting QoS provisioning and sustained performance Larger values throttle WAL write performance,

  • ccupying disk bandwidth for longer periods

99th percentile latency variation for the update operation for each PostgreSQL configuration, under Workload A with 1 thread

4.559 ms 4.597 ms 4.755 ms

slide-19
SLIDE 19

Discussion

Compaction and checkpointing heavily impact foreground tasks performance HBase throttles compaction throughput

  • Does not provide the building block to dynamically adapt such settings
  • Applications experience compaction effects for longer periods of time

PostgreSQL cannot dynamically adjust checkpointing activity to the overall load of the infrastructure Storage tasks should be dynamically programmable to achieve optimal holistic performance

19

slide-20
SLIDE 20

Programmable Storage Background Tasks

Following the Software-Defined Storage principles

  • Decouple background mechanisms and policies
  • At layer level, a data plane dynamically adapts background activities
  • At infrastructure-level, policy-enabled controller provides adaptable end-to-end

control

Minimize I/O variability and interference, and ensure QoS provisioning and resource fairness

20

slide-21
SLIDE 21

21

Programmable Storage Background Tasks

slide-22
SLIDE 22

22

Programmable Storage Background Tasks

slide-23
SLIDE 23

23

Programmable Storage Background Tasks

slide-24
SLIDE 24

A Case for Dynamically Programmable Storage Background Tasks

Ricardo Macedo, Alberto Faria, João Paulo, José Pereira

INESC TEC & University of Minho

38th IEEE International Symposium on Reliable Distributed Systems Workshops 1st Workshop on Distributed and Reliable Storage Systems Lyon, France October 1st 2019

slide-25
SLIDE 25

HBase Compactions

Tail latency results

25

Performance degradations of at most 40% at the 99th percentile latency for write operations Read-oriented operations exhibit an

  • verhead of at most 955.2%

Read-modify-write experiences a 281% overhead

Complementary cumulative distribution function (CCDF) for the latency of Workload E with 10 threads

509%

slide-26
SLIDE 26

PostgreSQL Checkpointing

Mean latency results

26

Read and write operations are equally exposed to performance variations 128MiB WAL size degrades performance of at most 20.7%, 13.9%, and 17.8% under 0.1, 0.5, and 0.9 checkpoint completion target

slide-27
SLIDE 27

Case Study: HBase

Highly available NoSQL database, made of RegionServers and an HMaster

  • Tables are horizontally partitioned by row key ranges into Regions
  • Writes are first persisted in a WAL and then written to a Memstore (write-oriented cache)
  • Reads hierarchically traverse the data store, accessing the Block Cache, Memstore, and

HFiles

  • Generation of different HFiles per Memstore leads read amplification

27

slide-28
SLIDE 28

Case Study: HBase

LSM-based design leads to the execution of background compactions

  • Minor compaction. Merges several small-sized HFiles into fewer larger ones
  • Major compaction. Merges all HFiles into a single larger one, removing deleted entries
  • While improving read performance, compactions introduce I/O interference and

burstiness

28

slide-29
SLIDE 29

Case Study: PostgreSQL

Relational database that handles requests from multiple applications

  • Writes are first mapped to a shared buffer and then written to a WAL buffer
  • On commit, changes are sequentially written to a WAL file on disk
  • Reads hierarchically traverse the database, accessing the shared buffer, OS cache, and disk

29

slide-30
SLIDE 30

Methodology

Testbed

  • HBase 2.0.5 pseudo-distributed mode, backed by HDFS 2.9.2
  • PostgreSQL 11.3 backed by an ext4 file system

30

Workloads*

  • Workload A: 50% read, 50% update, zipfian
  • Workload B: 100% update, zipfian
  • Workload C: 100% read, uniform
  • Workload D: 5% read, 95% insert, zipfian
  • Workload E: 95% scan, 5% insert, zipfian
  • Workload F: 50% read, 50% read-modify-write,

zipfian

* Workloads previously used in [22]: “MeT: Workload aware elasticity for NoSQL”

Execution scenario

  • Single and multi-threaded
  • Loading phase with 12.5M records (≈16GiB)
  • Runs executed 10M operations or when 17

minutes had elapsed

  • System state was recreated after each run

Results publicly available at https://rgmacedo.github.io/drss19-website/