Toward a cost model for system administration Alva Couch Ning Wu - - PowerPoint PPT Presentation

▶
toward a cost model for system administration
SMART_READER_LITE
LIVE PREVIEW

Toward a cost model for system administration Alva Couch Ning Wu - - PowerPoint PPT Presentation

Sep 13, 2022 327 likes •721 views

Toward a cost model for system administration Alva Couch Ning Wu Hengky Susanto Tufts University LISA-2005 Tufts University couch@cs.tufts.edu Computer Science Executive Summary Cost of SA s includes e d u Tangible cost l c

slide-1
SLIDE 1

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Toward a cost model for system administration

Alva Couch Ning Wu Hengky Susanto Tufts University

slide-2
SLIDE 2

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Executive Summary

SA models of complexity and service Models of cost and complexity Software Engineering tickets and completions Real SA performance data quantify inspire Depends upon practice Intangible cost of SA "best practice" documents SA risk models i n s p i r e Models of task arrival and throughput Capacity planning lead to cost varies with environment SA model of troubleshooting cost utilized u t i l i z e d u t i l i z e d Estimated waiting time u t i l i z e d p r

  • p
  • r

t i

  • n

a l t

  • Out of SA's

control Tangible cost

  • f SA

Cost of SA includes i n c l u d e s

slide-3
SLIDE 3

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

System Administrator’s Summary

new metrics for complexity and process efficiency help to define new ways to improve process software engineering theory risk assessment techniques

  • perating

systems theory help to define h e l p t

  • d

e f i n e h e l p t

  • d

e f i n e suggests new ways to compute consequences of decisions lower cost, higher value leads to happily ever after leads to

slide-4
SLIDE 4

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

“Best Practices”

  • Cost the least
  • Provide the most value
  • via several intangibles

– homogeneity – consistency – repeatability – documentation – etc.

slide-5
SLIDE 5

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Patterson’s cost model

  • Cost of downtime ≈ cost of revenue lost +

cost of work lost.

  • Patterson, “A simple model of the cost of

downtime”, Proc. LISA 2002

  • Controversial: downtime cost is

“intangible”.

  • Or is it?
slide-6
SLIDE 6

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

“Best” is relative!

  • Patching systems immediately causes

more downtime than waiting for patches to stabilize.

  • Cowan et al, “Scheduling the application of

security patches for optimal uptime”, Proc. LISA 2002.

slide-7
SLIDE 7

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Time spent waiting

  • Cost of system administration = cost of

tangible assets + cost of intangibles

  • For most SA’s, cost of tangible assets is
  • ut of our control.
  • Claim 1: The intangible cost of system

administration is approximately proportional to (cumulative) time spent waiting for responses to requests

slide-8
SLIDE 8

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Learning from real data

  • Data source: RT queue, Tufts ECE/CS.
  • Data duration ≈ 400 days.
  • What is the structure of real data?
  • Is there any easy way to describe the

schedule of ticket arrivals and service?

slide-9
SLIDE 9

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Ticket history

slide-10
SLIDE 10

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Measuring time spent waiting

  • Time spent waiting is a function of

– arrival rate: number of requests coming in – service rate: how fast requests can be processed – number of “workers” available – number of “clients” affected.

  • Where

– arrivals include reconfigurations and refits – rate is reciprocal of expected service time

slide-11
SLIDE 11

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Memory

  • A process is memoryless if the next event

does not depend upon the history of prior events.

– memoryless arrivals: “Poisson process” λ = arrival rate, mean inter-arrival time = 1/λ, standard deviation of inter-arrival times = 1/λ. – memoryless service: “exponential service time”. µ = service rate, mean service time = 1/µ, standard deviation of service time = 1/µ.

slide-12
SLIDE 12

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Memoryless is nice (but perhaps impractical)

  • Memoryless arrivals: lots of identical

customers behaving independently.

  • Arrival processes with memory: bursty

behavior, such as a virus infection, spam,

  • r DDoS attack.
  • Advantage of memoryless models: closed-

form solutions to system performance (from capacity planning)

slide-13
SLIDE 13

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Multiclass systems

  • Typical site has multiple classes of

requests; some are more complex or take longer than others.

  • At first glance, no exponential service

times.

  • Throw away long times (outliers);

exponential service times emerge!

  • Claim 2: Documentation keeps

requests from waiting indefinitely.

slide-14
SLIDE 14

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Tickets filtered

slide-15
SLIDE 15

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Quandary of arrivals

  • At first glance arrivals aren’t Poisson
  • But (a month of struggling later!)

– correct for DST – sample over one-hour intervals – correct sampling for sparse event frequency – skip holidays

  • And each hour exhibits a roughly Poisson

arrival rate!

slide-16
SLIDE 16

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Ticket creation

lunch!

slide-17
SLIDE 17

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Ticket resolution

student responsible for resolving tickets starts workday! staff arrives and handles nightly buildup in queue

slide-18
SLIDE 18

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Quantifying time spent waiting

  • Our data shows that most requests are

actually accomplished at our site in (statistically) comparable times.

  • How does one estimate the time needed

for a particular request?

  • One example: troubleshooting chart.
slide-19
SLIDE 19

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Simple troubleshooting chart

got an address? no ip address got an address? yes no DHCP locally enabled? Enable DHCP no yes dhcpd running? Restart dhcpd yes yes no no end

slide-20
SLIDE 20

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Convert to program graph

B A F yes no C D no yes G H yes yes no no E D A E H C G F B flow

slide-21
SLIDE 21

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Convert from graph to tree

D A E H C G F B D A E H G F B E E E H G F E C

slide-22
SLIDE 22

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Collapse to decision tree

tD+tF +tG tH tB tC tH tF+tG

1-P(C) P(C) 1-P(D) P(D) 1-P(H«|¬D) P(H«|¬D) P(H«|D) 1-P(H«|D) D A E H G F B E E E H G F E C

slide-23
SLIDE 23

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Compute expected value

tD+tF +tG tH tB tC tH tF+tG

1-P(C) P(C) 1-P(D) P(D) 1-P(H«|¬D) P(H«|¬D) P(H«|D) P(E«|D)

expected wait = t1 + pt2+(1-p)t3 expected wait = tB+P(C) [ tC+P(D)[tD+tF+tG+P(H«|D)tH)+(1-P(D))(tF+tG+P(H«|¬D)tH] ]

t2 t3 t1 p 1-p

slide-24
SLIDE 24

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Notes on the decision tree

  • Times tX describe the capabilities of

administrative staff.

  • Probabilities P(Y) describe the site’s

characteristics and the likelihood of failures.

  • P(H«|D): probability of H happening given that D

happened in the past

  • [temporal conditional probability; not Bayesian;

Bayesian identities don’t hold! Another month of suffering to figure this out!]

slide-25
SLIDE 25

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Application: should I check the DHCP server or client first?

  • Answer: depends upon site characteristics.
  • If the likelihood is that there is a problem with X,

should check X first.

  • Consequences of incorrect choice: increased

cost.

  • Humans automatically compensate for poor

troubleshooting order.

  • Claim 3: Best practices are relative to site and

staff capabilities.

slide-26
SLIDE 26

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Bang!

  • The preceding method is “white box”; it

measures the practice directly.

  • Applying the preceding argument for a

non-trivial troubleshooting chart results in an exponential explosion in chart complexity.

  • How do we deal with huge charts or

complex processes?

  • Answer: “black box” estimation.
slide-27
SLIDE 27

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Estimators from Software Engineering

  • Time for service is approximately a function of

the number of branches in a troubleshooting chart.

  • Number of branches is approximately a function
  • f heterogeneity/diversity of site and services

provided.

  • So if we quantify diversity/complexity of service

environment, we can estimate service time.

  • “Function points”: a way of quantifying

complexity of service.

slide-28
SLIDE 28

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Non-product systems

  • We understand a great deal about

“product systems” in which components act independently.

  • System administrators are a non-product

system; they communicate and interact with each other.

  • Best way to estimate behavior of non-

product systems: discrete event simulation.

slide-29
SLIDE 29

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

A simple simulation experiment

  • Assume c administrators, four classes of

service (from extremely short to extremely long service times), independent arrival rates for classes.

  • Theory: a single class system is stable if

λ/cµ<1 and diverges to infinite wait time

  • therwise.
  • What happens when a multi-class system

approaches the saturation point?

slide-30
SLIDE 30

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Diminishing returns

slide-31
SLIDE 31

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Divergence!

slide-32
SLIDE 32

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Chaos!

slide-33
SLIDE 33

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Running near the edge

arrivals spread out bursty arrivals events in a burst, versus events spread out!

slide-34
SLIDE 34

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Summary

  • cumulative service time ≈ intangible cost
  • f operations
  • computable from practice graph: function
  • f staff expertise and site composition.
  • estimable from guesses for branch depth

and task length for each task.

  • total effect estimable via discrete event

simulation.

slide-35
SLIDE 35

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Conclusions

  • We can estimate the cost of practice by

indirect methods.

  • Best practices are always site relative!
  • Running near absolute capacity causes

chaotic increases in wait time.

slide-36
SLIDE 36

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

What’s next?

  • Simulation studies of particular aspects of the

practice:

– communication vs. documentation, – scripting vs. cfengine

  • Quantification of function point models

– various sizes and kinds of sites. – complexities of kinds of service.

  • Effects of human learning

– Insignificant for repetitive tasks. – Significant for one-time tasks.

slide-37
SLIDE 37

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Epilogue

  • More questions than answers:

– How can we best use this as a planning tool? – How much can we trust it? – How to fill in gaping holes in knowledge?

  • The potential:

– better/cheaper/more valuable administrative practices. – Ability to ask cheap “what if” questions with reasonable estimates of task complexity. – better understanding of critical capacity. – happily ever after.

slide-38
SLIDE 38

LISA-2005 couch@cs.tufts.edu Tufts University Computer Science

Questions?

Alva Couch (couch@cs.tufts.edu) Ning Wu (ningwu@cs.tufts.edu) Hengky Susanto (hsusan0a@cs.tufts.edu) Tufts University Computer Science Medford, MA 02155 Note: we plan to make the discrete event simulator

  • pen source at some future time after we clean

up the user interface.