Chenyang Lu Outline OnlineDataMigra.oninStorageServers - - PDF document

SLIDE 1

Chenyang Lu

Quality of Service in Unpredictable Computing Environments

1

Outline


• Control‐theore*c
Framework

• Service
delay
control
on
Web
servers

• On‐line
data
migra*on
in
storage
servers

• ControlWare:
adap*ve
QoS
control
middleware


39 


Online
Data
Migra.on
in
Storage
Servers


• Enterprise
storage
servers


need
to
move
data


 System
expansion
  Applica*on
changes


• Always‐on:
e‐business,


global
data
centers
 

Online
data
migra*on


40 


Storage system data 
 migration E-mail server; DB … I/Os New device

State
of
Prac.ce


41 


Script

E-mail server; DB…

Storage system

storage
 devices data 
 migration

Migration plan

Submover HP-UX LVM SAN Slow I/Oʼs!!! Need to bound impact

• n applications!

New device

The
Problem


• Execute
a
given
migra*on
plan
on‐line


• Challenges




Keep
data
consistent




Bound
impact
on
applica*on
performance




Complete
migra*on
quickly


42 


Adap.ve
solu.on


• Feedback
control
loop:
adapts
migra*on
speed
based

• n
applica*on
I/O
latency


 Enforce
latency
contract:
Bounded
average
I/O
latency
  Complete
migra*on
in
shortest
*me
allowed
by
contract


• Standard
control‐theore*c
design


 Systema*c
methodology
  Robust,
analy*cally
proven
performance


• Handle
different
workloads
and
devices


43 


Aqueduct


44 


{LCi} I/Os

Controller Actuator Monitor

{Li(k)} Rm(k)

Aqueduct

migration executor

E-mail server; DB…

Storage system

storage
 devices data 
 migration

Migration plan

Submover HP-UX LVM SAN

Application Latency Contract

SLIDE 2

Chenyang Lu

Quality of Service in Unpredictable Computing Environments

2

Monitor


• Measure
applica*ons’
average
I/O
latency
of
each
store
in
the


last
sampling
window





Current
implementa*on:
trace
replayer
directly
monitors
I/O
latencies




Can
interface
with
performance
monitoring
tools
(HP
Openview)


45 


Controller Actuator Monitor

Actuator


• Problem:
fine‐grained
control
of



migra*on
speed
using
HP‐UX
LVM


 Divide
store
into
small

(32
MB)
substores
(LVs)
  Submover
moves
substore
using
LVM
silvering
  Actuator
enforces
a
submove
rate
by
sleeping


46 


Controller Actuator Monitor

Mirror Silvering Split 1 submv/sw 2 submv/sw submv sleep submv sleep

Sampling Window Sampling Window

sleep sleep sleep sleep

Controller


• Compute
error
for
each
store
i



 
Ei(k)
=
P*LCi
‐
Li(k)
 
0<P<1:
safety
margin,
related
to
burs*ness
 
k:
represents
the
kth
sampling
window


• Compute
worst
error



 
Emin(k)
=
min{Ei(k)}


• Integral
controller
computes
new
submove
rate:



 
Rm(k)
=
Rm(k‐1)
+
K*Emin(k)
 
Control
gain
K:
aggressiveness
of
rate
change


47 


Controller Actuator Monitor

Tuning
controller
parameters


• Stability

• Tracking:
VL(k)
=
P*LC
in


steady
state


• SeOling
.me


48 


Approximate linear model VL(k+1)–VL(k)=G(Rm(k)-Rm(k-1)) System profiling: Estimate G Control Analysis Compute K Satisfy Construct transfer function Process gain G: Impact of submove rate changes on victim latency. Victim latency VL(k): highest average latency among all stores in the kth sampling window

Experimental
setup


• Enterprise‐scale
storage
server


49 


HP 9000-N4000 Server

8 440MHz processors

FC-60 disk array

(1.05 TB, 5 RAID5 Logical Units)

Aqueduct Openmail I/O Trace LU0 LUnew HP-UX 11 & LVM

emails metadata emails metadata

Fibre Channel

Experiments


• Baselines:
no
sleeping
between
(sub)moves





Whole‐store:
move
one
store
at
a
*me




Sub‐store:
move
one
substore
at
a
*me


• Constant:
steady
Poisson
streams




Replace
Logical
Unit;
migrate
three
640‐MB
stores.


• Openmail:
trace
of
an
enterprise
e‐mail
server
running
HP


Openmail




Add
Logical
Unit;
migrate
a
1854
MB
store
and
a
96
MB
store



50 


SLIDE 3

Chenyang Lu

Quality of Service in Unpredictable Computing Environments

3

Measure
G

Tune
K


51 


Constant: K = 1.09 Openmail: K = 0.36 Constant Openmail

Process gain G: the slope

• f the curves

Control gain K

Openmail:
vic.m
latency


52 


LC 0.8*LC Aqueduct Sub-store Whole-store Average Victim Latency (ms)

Openmail:
latency


53 


Aqueduct uniformly better than baselines, but … LC

Openmail:
latency
&
submove
rate


54 


 Load
highest
on
new
LU
towards
end
of
migra*on
  By
design,
submove
rate
must
be
1
or
higher

 controller
is
working
correctly


Openmail:
average
latency


55 


LC

Aqueduct Sub-store Whole-store

Openmail:
latency
CDF


56 


76% 
 91% 


SLIDE 4

Chenyang Lu

Quality of Service in Unpredictable Computing Environments

4

Related
work


• Simpler
versions
of
the
problem




Take
(parts
of)
system
offline




Migrate
data
in
“quiet
periods”



• Silvering
in
Logical
Volume
Manager
[HP‐UX
LVM,
VxVM]:


maintain
data
consistency,
no
QoS
guarantees


• Propor*onal
I/O
scheduling:
hard
to
handle
unpredictability

• MS
Manners:
no
guarantees
to
important
tasks

• Control‐theory‐based
systems:
distributed
visual
tracking,
Web


servers,
e‐mail
server,
database
real‐*me
processor
 scheduling
...



57 


Summary


• Migra*on
must
be
executed
adap*vely

• Aqueduct
is
neither
overly
aggressive




Average
I/O
latency
reduced
by
76%




Contract
viola*on
ra*o
reduced
by
78%


• Nor
overly
conserva*ve




Average
vic*m
latency
15%
lower
than
latency
contract


• Future




More
detailed
sensi*vity
analysis




Self‐tuning
controller




Mul*‐dimensional
QoS
contracts


58 


References


• C. Lu, G. A. Alvarez, J. Wilkes, Aqueduct: Online Data Migration

with Performance Guarantees, USENIX Conference on File and Storage Technologies (FAST), 2002.

• G.A. Alvarez, C. Lu and J. Wilkes, Method and System for Online

Data Migration on Storage Systems with Performance Guarantees, U. S. Patent 7,167,965, January 2007.

59