O INRIA CNRS Universit de P a ris-Sud, F-91405 Orsa y - - PowerPoint PPT Presentation

▶
o
SMART_READER_LITE
LIVE PREVIEW

O INRIA CNRS Universit de P a ris-Sud, F-91405 Orsa y - - PowerPoint PPT Presentation

Nov 17, 2023 159 likes •680 views

Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Multi-sale Real-time Grid Monito ring with Job Stream Mining Xiangliang Zhang , Mihele Sebag, Ceile Germain-Renaud T A O INRIA CNRS

slide-1
SLIDE 1 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Multi-s ale Real-time Grid Monito ring with Job Stream Mining Xiangliang Zhang, Mi hele Sebag, Ce ile Germain-Renaud T A O − INRIA CNRS Universit de P a ris-Sud, F-91405 Orsa y Cedex, F ran e 21 Ma y 2009 Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-2
SLIDE 2 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Contents 1 Monito ring system: Grid adapted StrAP 2 Streaming Jobs 3 Monito ring Outputs Monito ring
  • n
sho rt-time s ale Clustering Qualit y Monito ring
  • n
medium-time s ale Monito ring
  • n
la rge-time s ale Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-3
SLIDE 3 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Contents 1 Monito ring system: Grid adapted StrAP 2 Streaming Jobs 3 Monito ring Outputs Monito ring
  • n
sho rt-time s ale Clustering Qualit y Monito ring
  • n
medium-time s ale Monito ring
  • n
la rge-time s ale Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-4
SLIDE 4 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Multi-s ale Realtime Grid Monito ring System Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-5
SLIDE 5 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Multi-s ale Realtime Grid Monito ring System

1 2 3 4 5 20 40 60 80 100

7 10 47 54 129 8 18 24 30 595 139 7 13 14 24 9728 19190

Percentage of jobs assigned (%)

Outliers

Clusters

exemplar shown as a job vector

Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-6
SLIDE 6 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Multi-s ale Realtime Grid Monito ring System

20 40 60 80 100 120 140 160 5 10 15 20 25 30

days percentage of jobs (%)

distirbution of jobs like [7 0 0 0 0 0]

20 40 60 80 100 120 140 160 10 20 30 40 50 60 70 80 90

days percentage of jobs (%)

distirbution of jobs like [0 0 0 0 0 0]

Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-7
SLIDE 7 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Multi-s ale Realtime Grid Monito ring System Anit y Propagation (AP) A lustering metho d: group simi l a r p
  • ints
together StrAP (Streaming AP) Online Clustering streaming data based
  • n
AP Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-8
SLIDE 8 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Why AP ?? Anit y Propagation (AP) A lustering metho d Group simi l a r p
  • ints
together Converge b y Iterations
  • f
Message passing
  • >
mo re stable results No need
  • f
K (the numb er
  • f
lusters)
  • >
less p rio r kno wledge A real p
  • int
as an exempla r to rep resent a luster
  • >
avoid meaningless averaged enters Clustering b y P assing Messages Bet w een Data P
  • ints.
B.J. F rey , D. Due k. S ien e 2007 Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-9
SLIDE 9 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Ho w AP w
  • rks
?? Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-10
SLIDE 10 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Ho w AP w
  • rks
?? Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-11
SLIDE 11 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Ho w AP w
  • rks
?? Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-12
SLIDE 12 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Ho w AP w
  • rks
?? Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-13
SLIDE 13 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Ho w AP w
  • rks
?? Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-14
SLIDE 14 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Ho w AP w
  • rks
?? Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-15
SLIDE 15 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Ho w AP w
  • rks
?? Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-16
SLIDE 16 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Ho w AP w
  • rks
?? Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-17
SLIDE 17 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Grid adapted StrAP Grid adapted StrAP (Streaming AP): Online lustering streaming jobs
  • >
  • ne-s an
  • f
the stream In remental up date
  • f
mo del
  • >
k eep tra king the stream Dete ting distributi
  • n
hanges in stream
  • >
abso rb new patterns Data streaming with Anit y p ropagation. Xiangli ang Zhang, Cyril F urtlehner, Mi hele Sebag. ECML 2008. Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-18
SLIDE 18 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Stream lustering

❡ ❡ ❡ ✐ ✐ ❡ ✐ ✐ ❡ ❡ ✐ ✐

Mo del Reservoir

❡ ❢ ❥ ✐ ❥

Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-19
SLIDE 19 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Stream lustering

❡ ❡ ❡ ✐ ✐ ❡ ✐ ✐ ❡ ❡ ✐ ✐ ❡

Mo del Reservoir

❡ ❢ ❡ ❢ ❥ ✐ ❥

Do es x t t the urrent mo del ?? if y es, up date the mo del
  • therwise,
go to reservoir Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-20
SLIDE 20 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Stream lustering

❡ ❡ ❡ ✐ ✐ ❡ ✐ ✐ ❡ ❡ ✐ ✐ ❡ ✐

Mo del Reservoir

❡ ❢ ❥ ✐ ❥ ❥ ✐ ❥

Do es x t t the urrent mo del ?? if y es, up date the mo del
  • therwise,
go to reservoir Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-21
SLIDE 21 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Stream lustering

❡ ❡ ❡ ✐ ✐ ❡ ✐ ✐ ❡ ❡ ✐ ✐ ❡ ✐ ❅

Mo del Reservoir

❡ ❢ ❥ ✐ ❥ ❅

Do es x t t the urrent mo del ?? if y es, up date the mo del
  • therwise,
go to reservoir Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-22
SLIDE 22 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Stream lustering

❡ ❡ ❡ ✐ ✐ ❡ ✐ ✐ ❡ ❡ ✐ ✐ ❡ ✐ ✐ ❡ ❅ ✐ ❡

  • ❅

❅ ❅

Mo del Reservoir

❡ ❢ ❥ ✐ ❥

  • ❅

❅ ❅

Has the distributi
  • n
hanged ?? CHANGE TEST if y es, rebuilt the mo del
  • therwise,
  • ntinue
Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-23
SLIDE 23 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Stream lustering

❡ ❡ ❡ ✐ ✐ ❡ ✐ ✐ ❡ ❡ ✐ ✐ ❡ ✐ ❅ ✐ ❡

  • ❅

❅ ❅

Mo del Reservoir

❡ ❢ ❥ ✐ ❥❅

Has the distributi
  • n
hanged ?? CHANGE TEST if y es, rebuilt the mo del
  • therwise,
  • ntinue
Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-24
SLIDE 24 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Our Mo del Output e i , the exempla r ( enter
  • f
luster) n i , size
  • f
luster

Σ

i , average distan e
  • f
p
  • ints
to their exempla r T , time stamp when the luster w as latterly visited P a rameters

ǫ,

threshold
  • f
  • mpa
ring ea h p
  • int
with mo del (set to a round value
  • f Σ
i in the initial mo del)

∆ ,

de a y windo w (de rease the w eight
  • f
  • ld
exempla rs) P age-Hinkley pa rameters ( hange dete tion) Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-25
SLIDE 25 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Contents 1 Monito ring system: Grid adapted StrAP 2 Streaming Jobs 3 Monito ring Outputs Monito ring
  • n
sho rt-time s ale Clustering Qualit y Monito ring
  • n
medium-time s ale Monito ring
  • n
la rge-time s ale Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-26
SLIDE 26 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs EGEE (Enabling Grids fo r E-s ien E) F unded b y Europ ean Commission ( ontribution: 32,000 ,0 euro) Sta rt in Ap ril 2004 Grid infrastru ture available to s ientists 24 hours-a-da y . http://publi .e u
  • e
g e e .o rg / Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-27
SLIDE 27 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs EGEE Jobs EGEE logs
  • f
39 RBs during 5 months (2006-01-01 2006-05-31)
  • lle ted
b y Real Time Monito r (RTM) system ( http://gridp
  • rtal.hep.ph.i .a .uk/rtm/)
5,268, 5 6 4 jobs fo r ea h job, its nal status (go
  • d
  • r
t yp e
  • f
erro rs) UI, RB, CE time stamps
  • f
every servi es happ ened Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-28
SLIDE 28 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Job attributes registrati
  • n_Ti
m e: time fo r registeri n g the job mat h_Ti m e: time to nd a mat hing resour e upto_s hedu led_ transfer_Time : time a eptation and transfer ( w aiting + ready time), as rep
  • rted
b y the JobControll e r (JC) upto_s hedu led_ a e p tan e_ Time : the same as Ready_fo r_T ransfer_Ti me, but as rep
  • rted
b y the LogMonito r (LM) logmonito r_ e_s heduled_Time : time job w aiting in a queue logmonito r_wn_Tim e: exe ution time Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-29
SLIDE 29 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Multi-s ale Realtime Grid Monito ring System Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-30
SLIDE 30 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Pre-p ro es s ing and No rmalization Pre-p ro essing 6 b
  • lean
attributes indi ate whether the servi es w ere rea hed
  • r
not No rmalization b y entering with standa rd deviation 1 job x i is no rmalized to x′ i = x i−µ s where, µ and s a re mean and standa rd deviation from a pa rt
  • f
streams. Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-31
SLIDE 31 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Load
  • f
jobs p er da y

20 40 60 80 100 120 140 1 2 3 4 5 6 7 8 x 10

4

Days Number of jobs per day

Sat & Sun Mon Tue Wed Thu Fri line

Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-32
SLIDE 32 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Contents 1 Monito ring system: Grid adapted StrAP 2 Streaming Jobs 3 Monito ring Outputs Monito ring
  • n
sho rt-time s ale Clustering Qualit y Monito ring
  • n
medium-time s ale Monito ring
  • n
la rge-time s ale Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-33
SLIDE 33 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Monito ring
  • n
a sho rt-t im e s ale Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-34
SLIDE 34 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Real-time Monito ring: when hange dete ted

1 2 3 4 5 10 20 30 40 50 60 70 80 90 100

Reservoir 7 10 47 54 129 8 18 24 30 595 139 7 13 14 24 9728 19190

Clusters Percentage of jobs assigned (%)

exemplar shown as a job vector

Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-35
SLIDE 35 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Real-time Monito ring: when hange dete ted

1 2 3 4 5 6 7 8 10 20 30 40 50 60 70 80 90 100

the assignment of jobs between restart 1 and restart 2

Reservoir 7 10 47 54 129 90 3 5 8 220 199 8 18 24 30 595 139 6 5 10 14 127 10854 7 13 14 24 9728 19190 7 18 34 39 50190 4619

Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-36
SLIDE 36 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Real-time Monito ring: when hange dete ted

1 2 3 4 5 6 7 8 10 20 30 40 50 60 70 80 90 100

the assignment of jobs between restart 2 and restart 3

Reservoir 7 10 47 54 129 90 3 5 8 220 199 8 18 24 30 595 139 6 5 10 14 127 10854 7 13 14 24 9728 19190 14 8 13 20 55883 16076

Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-37
SLIDE 37 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Real-time Monito ring: when hange dete ted

1 2 3 4 5 6 7 8 10 20 30 40 50 60 70 80 90 100

the assignment of jobs between restart 3 and restart 4

Reservoir 7 10 47 54 129 90 3 5 8 220 199 8 18 24 30 595 139 6 5 10 14 127 10854 50 16 23 120 36311 4081 7 18 34 39 50190 4619

Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-38
SLIDE 38 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Real-time Monito ring: when hange dete ted

1 2 3 4 5 6 7 8 10 20 30 40 50 60 70 80 90 100

the assignment of jobs between restart 4 and restart 5

Reservoir 7 10 47 54 129 24 154 190 9395 90 3 5 8 220 199 8 18 24 30 595 139 24 150 187 9392 314 611 6 5 10 14 127 10854

Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-39
SLIDE 39 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Real-time Monito ring: when hange dete ted

1 2 3 4 5 6 7 8 10 20 30 40 50 60 70 80 90 100

the assignment of jobs between restart 5 and restart 6

Reservoir 7 10 47 54 129 9 18 25 20110 8 18 24 30 595 139 6 5 10 14 127 10854 10 18 29 20091 395 276

LogMonito r is getting logged . Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-40
SLIDE 40 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Who is resp
  • nsible
fo r the logging ?? Distri buti
  • n
  • f
A ttr4/A ttr 3 Distri buti
  • n
  • f
all jobs
  • ver
39 RBs Distri buti
  • n
  • f
jobs from 9-th RB

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 x 10

6

1 2 3 4 5 6 7 8 x 10

4

jobs Att4/Att3

Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-41
SLIDE 41 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Who is resp
  • nsible
fo r the loggong ?? Whi h RB ??

5 10 15 20 25 30 35 40 0.1 0.2 0.3 0.4 0.5 0.6 0.7 RBs Correlation coefficients gdrb04.****.ch gdrb03.****.ch lappgrid07.****.fr

Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-42
SLIDE 42 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Clustering Qualit y Assess m e n t Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-43
SLIDE 43 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Clustering Purit y Purit y = 100% × ( K i= 1

|

C d i |

|

C i | )/K where K is numb er
  • f
lusters ,

|

C i| is size
  • f
luster i ,

|

C d i | is numb er
  • f
majo rit y lass items in luster i .

100 200 300 400 500 80 85 90 95 100

Averaged purity of each cluster (%) Restarts

50 100 150 200 250 300 350 400 450 500 550 50 100 150 200 250 300

Number of clusters Number of clusters Averaged purity of each cluster

Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-44
SLIDE 44 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Dis uss Real-time qualit y:
  • n
average 10000 jobs in 1 minut e vs maxim u m load: 80000 p er da y Intel 2.66GHz Dual-Co re PC with 2 GB memo ry
  • ding
in matla b
  • n
average 60000 jobs in 1 minut e
  • ding
in C/C++
  • mpa t
and live des ription
  • f
job patterns p rop
  • rtion
  • f
go
  • d
jobs and failed jobs dierent time
  • st
  • f
servi es the jobs w ent through Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-45
SLIDE 45 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Monito ring
  • n
a med ium
  • t
im e s ale Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-46
SLIDE 46 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Rupture steps

20 40 60 80 100 120 140 160 2 4 6 8 10 12 days number of restarts per day

k eep tra king the evolving
  • f
job distribution p rovides intuitive view
  • f
grid regime and its stabilit y Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-47
SLIDE 47 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Monito ring
  • n
a la rge-time s ale Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-48
SLIDE 48 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale La rge-time s ale Monito ring: Global view

20 40 60 80 100 120 140 160 5 10 15 20 25 30

days percentage of jobs (%) distirbution of jobs like [7 0 0 0 0 0]

20 40 60 80 100 120 140 160 10 20 30 40 50 60 70 80 90

days percentage of jobs (%) distirbution of jobs like [0 0 0 0 0 0]

Clustering the exempla rs > Sup er exempla rs Sup er lusters: Cluster
  • f
exempla rs the histo ry b ehavio r
  • f
these sup er lusters Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-49
SLIDE 49 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Bad Sup er Examples: da y view

Days Super Clusters

20 40 60 80 100 120 140 2 4 6 8 10 12 14 16 18 20 10% 20% 30% 40% 50% 60% 70% 80% 90%

Re- he k
  • f
ea rly stopp ed erro r t yp e
  • f
erro rs (rst ro w) Date Jan 7∼13 Jan 30 ∼ F eb 3 Ma r 16∼21 Ma y 17∼19 UI A1 A1 B1 D1 and A1 Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-50
SLIDE 50 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Dis ussion and Con lusion real-time monito ring Grid job streams p roviding multi-s al e mo dels to des ribing the status
  • f
Grid p rop
  • rtion
  • f
dierent t yp e
  • f
job patterns (realtime-view, da y-view, w eek-view ....) rupture steps
  • ine
globally analysis go
  • d
qualit y lustering is gua ranteed Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-51
SLIDE 51 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale F uture w
  • rk
mo re
  • mp
rehensive des ription
  • f
the jobs, e.g., related to UI and CE interp ret the mo del dynami s, e.g., relating the rebuild frequen y to alenda r
  • r
so ial events, in
  • llab
  • ration
with the
  • p
eration teams. Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining
slide-52
SLIDE 52 Monito ri ng system: Grid adapted StrAP Streaming Jobs Monito ri ng Outputs Monito ri ng
  • n
sho rt-time s ale Clustering Qualit y Monito ri ng
  • n
medium
  • t
i m e s ale Monito ri ng
  • n
la rge-tim e s ale Thank y
  • u
Qestions ?? Xiangli ang Zhang, Mi hele Sebag, Ce ile Germain-Renaud Grid Monito ri ng with Job Stream Mining