Analysis of Network Flow Data Gonzalo Mateos Dept. of ECE and - - PowerPoint PPT Presentation
Analysis of Network Flow Data Gonzalo Mateos Dept. of ECE and Goergen Institute for Data Science University of Rochester gmateosb@ece.rochester.edu http://www.ece.rochester.edu/~gmateosb/ April 26, 2016 Network Science Analytics Analysis of
Network Science Analytics Analysis of Network Flow Data 1
Network Science Analytics Analysis of Network Flow Data 2
◮ Networks often serve as conduits for traffic flows
◮ Commodities and people flow over transportation networks; ◮ Data flows over communication networks; and ◮ Capital flows over networks of trade relations ◮ Flow-related questions on network design, provisioning and routing
◮ Our focus: statistical analysis and modeling of network flow data
Network Science Analytics Analysis of Network Flow Data 3
◮ Let G(V , E) be a digraph. Flows are directed: origin → destination
v )
◮ Routing matrix R ∈ {0, 1}Ne×Nf states incidence of routes with links
◮ Assumed flows follow a single route from origin to destination
Network Science Analytics Analysis of Network Flow Data 4
◮ Strongly connected digraph: flows can be as many as Nv(Nv − 1)
Network Science Analytics Analysis of Network Flow Data 5
◮ Central to study of network flows is the traffic matrix Z ∈ RNv×Nv
◮ Entry zij is total volume of flow from origin vertex i to destination j
◮ Ex: net out-flow from i and net in-flow to j given by
◮ Link-level aggregate traffic vector x := [x1, . . . , xNe]T related to Z as
Network Science Analytics Analysis of Network Flow Data 6
◮ Notion of cost c associated with paths or links also important
◮ Implicitly assumed a static snapshot taken of the network flows
◮ Common assumption to treat routing matrix R as being fixed
Network Science Analytics Analysis of Network Flow Data 7
◮ Internet2 backbone: Nf = 110 flows (8 shown) over a week
Network Science Analytics Analysis of Network Flow Data 8
◮ Roadmap dictated by types of measurement and analysis goal ◮ Measure: origin-destination (OD) flow volumes zij in full ◮ Goal: model flows to understand and predict future traffic
◮ Measure: link counts xe, flow volumes unavailable ◮ Goal: traffic matrix estimation, i.e., predict unobserved OD flows zij
◮ Measure: OD costs cij for a subset of paths ◮ Goal: predict unobserved OD and link costs
Network Science Analytics Analysis of Network Flow Data 9
Network Science Analytics Analysis of Network Flow Data 10
◮ Gravity models originate in the social sciences [Stewart ’41]
◮ Ex: geography, economics, sociology, hydrology, computer networks ◮ Newton’s law of gravitation for masses m1, m2 separated by d12
12 ◮ Gravity models specify interactions among populations vary:
◮ Intuition: OD flows as “population interactions”, makes sense!
Network Science Analytics Analysis of Network Flow Data 11
◮ Sets of origins I and destinations J . Flows Zij from i ∈ I to j ∈ J ◮ Gravity models state Zij are independent, Poisson, with mean
◮ Ex: Stewart’s theory of demographic gravitation specifies
ij
◮ Unlike Netwon’s law, no empirical or theoretical support here
Network Science Analytics Analysis of Network Flow Data 12
◮ Multiple origin, destination and separation functions proposed
◮ Ex: power functions for hO(i) and hD(j), where for α, β ≥ 0
◮ Ex: power function hS(cij) = c−θ ij , θ ≥ 0. General exponential form
◮ Convenient for inference of model parameters, since
Network Science Analytics Analysis of Network Flow Data 13
◮ Q: Structure of telecommunication interactions among populations?
◮ Gravity models to model telecommunication patterns as flows ◮ Data for phone-call traffic among 32 Austrian districts in 1991
Network Science Analytics Analysis of Network Flow Data 14
| | | | | | | | | | | | | | || | | | | | 6.5 7.5 8.5 6.5 7.5 8.5 1.6 2.0 2.4 2.8 1 2 3 4 5 8.5
◮ Data (in log10 scale) suggest a gravity model of the form
◮ Typical that flow volumes vary widely in scale
Network Science Analytics Analysis of Network Flow Data 15
◮ Specified Zij as independent Poisson RVs, with means µij = E [Zij]
◮ Let αi = log hO(i), βi = log hD(j) and θ ∈ RK. Will focus on
◮ P. McCullagh and J. Nedler, Generalized Linear Models. CRC, 1989 ◮ Given flow observations Z = z, the Poisson log-likelihood for µ is
Network Science Analytics Analysis of Network Flow Data 16
◮ MLEs ˆ
Tcij, i, j ∈ I × J ⇒ log ˆ
◮ Defined ˆ
T ˆ
TT, mean flow estimates ˆ
◮ Unique MLE ˆ
◮ A. Sen, “Maximum likelihood estimation of gravity model
Network Science Analytics Analysis of Network Flow Data 17
◮ LS procedures the norm early on, based on models
◮ Beware: ordinary LS estimation doomed to yield poor results
◮ Corrective measures: replace log Zij ↔ ˜
ij
ij ) terms
ij
ij
ij
◮ LS is simple, but all things being equal ML is preferable
Network Science Analytics Analysis of Network Flow Data 18
◮ Given phone-call data, form MLEs of parameters in two models
1 2 3 4 5 2 3 4 5 Log10(Flow Volume) Log10(Fitted Value) 1 2 3 4 5 2 3 4 5 Log10(Flow Volume) Log10(Fitted Value) − − − − − − − − − − − −
◮ Prediction of traffic flows. Plot ˆ
Network Science Analytics Analysis of Network Flow Data 19
1 2 3 4 5 −3 −2 −1 1 2 3 Log10(Flow Volume) Log10(Relative Error) 1 2 3 4 5 −3 −2 −1 1 2 3 Log10(Flow Volume) Log10(Relative Error) 1 2 3 4 5 −3 −2 −1 1 2 3 Log10(Flow Volume) Log10(Relative Error) 1 2 3 4 5 −3 −2 −1 1 2 3 Log10(Flow Volume) Log10(Relative Error)
◮ Relative prection error. Plot (zij − ˆ
Network Science Analytics Analysis of Network Flow Data 20
◮ Plot empirical CDF of models’ relative prediction errors −3 −2 −1 1 2 3 0.0 0.4 0.8 Log10(Relative Error) Empirical CDF −3 −2 −1 1 2 3 0.0 0.4 0.8 Log10(Relative Error) Empirical CDF ◮ General model’s CDF lies to the left of that for the standard model
◮ Ex: Standard model errors ≤ zij for 58% of the OD pairs
Network Science Analytics Analysis of Network Flow Data 21
Network Science Analytics Analysis of Network Flow Data 22
◮ Monitoring OD flow volumes Zij fundamental to:
◮ Often difficult (even impossible) to measure the Zij. . .
◮ . . . but relatively easy to acquire link counts Xe
Network Science Analytics Analysis of Network Flow Data 23
◮ Highly underdetermined inverse problem. “Invert” known fat R in
v )
◮ Also dubbed network tomography. Taxonomy of methods:
◮ Y. Vardi, “Network tomography: Estimating source-destination traffic
Network Science Analytics Analysis of Network Flow Data 24
◮ Traffic often has units of “counts” e.g., cars per hour or Mbps
◮ Simple linear model for observed link counts X = {Xe}e∈E
◮ R ∈ {0, 1}Ne×Nf is the known routing matrix ◮ µ ∈ RNf
+ is vector of expected OD flow volumes
◮ ε is a Ne × 1 vector of i.i.d. zero-mean errors, with variance σ2
◮ Formulation suggests estimating µ via ordinary LS
Network Science Analytics Analysis of Network Flow Data 25
◮ Graph G(V , E) with Nv = 5 and Ne = 4, OD pairs {ac, ad, bc, bd}
1 4 2 3 a b c d v
◮ Although Ne = Nf = 4, rank(R) = 3 and RTR not invertible
µ x − Rµ2
Network Science Analytics Analysis of Network Flow Data 26
◮ Suppose we have initial OD flow volume measurements Z0 = z0
◮ Use z0 to constrain the LS problem. Consider the model
◮ Generalized LS estimator
µ
Network Science Analytics Analysis of Network Flow Data 27
◮ Generalized LSE is a linear combination of z0 and x, namely
◮ Typically Σ is diagonal and Ψ depends on sampling of z0
◮ Likely to obtain negative ˆ
◮ M. Bell, “The estimation of OD matrices by constrained generalized
Network Science Analytics Analysis of Network Flow Data 28
◮ Instead of historical data, regularize with prior µ ∼ N(µ0, τ 2I) ◮ Suppose X = Rµ + ε, with ε ∼ N(0, σ2I). MAP estimator
◮ Uncertainty in the estimate assessed via the covariance matrix
Network Science Analytics Analysis of Network Flow Data 29
◮ Gaussian model inappropriate even if few {µij} are small ◮ Independent, Poisson OD flows modeled as
ij
◮ Consider error-free observations X = RZ
◮ Still µ identifiable if columns of R all distinct and nonzero [Vardi ’96]
Network Science Analytics Analysis of Network Flow Data 30
◮ Subgraph induced by V ′ = {a, v, c}, OD pairs {av, vc, ac}
1 4 2 3 a b c d v
◮ Two consistent flow sets
◮ Data likelihood L(µ; x) = P
vc/2) exp(−µac − µav − µvc)
Network Science Analytics Analysis of Network Flow Data 31
◮ Q: What is the MLE ˆ
µ0 (µacµvc + µavµ2 vc/2) exp(−µac − µav − µvc)
◮ Paradox? No, solution in the boundary of the feasible set ◮ For Poisson models L(µ; x) not concave in general [Vardi ’96]
◮ EM-based MLE solver impractical (E
Network Science Analytics Analysis of Network Flow Data 32
◮ Goal: inference based on the posterior P
◮ Prior specification: Z independent, Poisson(µ); along prior P (µ)
ij
◮ Observe link counts X, conduct inference based on P
Network Science Analytics Analysis of Network Flow Data 33
◮ P
ij P (µij), yields
ij
◮ P
◮ Illustrate through an example, then give general form of P
Analysis of Network Flow Data 34
◮ Subgraph induced by V ′ = {a, v, c}, OD pairs {av, vc, ac}
1 4 2 3 a b c d v
◮ Simulate from the full joint conditional posterior P
ac
av
vc
Network Science Analytics Analysis of Network Flow Data 35
◮ If rank(R) = Ne, write R = [R1 R2] with R1 ∈ {0, 1}Ne×Ne invertible
1 , ZT 2 ]T, where Z1 = R−1 1 (X − R2Z2) ◮ The sought conditional posterior has the form
1 (x − R2z2)
ij
◮ Amenable to drawing entries of Z2 via a Gibbs sampler
Network Science Analytics Analysis of Network Flow Data 36
◮ Monroe, NC road network: Ne = 20 links and Nf = 64 flows
566 Journal
the American Statistical Association, June 1998
H A B C D E L M N
Figure
Node-Link Structure
the Monroe Network.
aXiZ and
scale parameter a for some a > 0. Small values
(MARKOVIAN) ROUTING
Y1 : A
e+B:
1980
Y2 :
B C:
Y3
C-D:
Y4 : D +E:
2600
55 :
2100
Y6:
F -E:
Y7 :
Y8 :
Yg : C +B: 1954
Y12: L-B:
Y13: H+ C:
Y14: M-C:
Y15: G-D:
Y16:
BI
Y19:
Figure
Flows for the Monroe Network
Figure 6.
◮ Network fed to the traffic simulator Integration [Van Aerde et al ’96]
◮ Data (OD flows and link counts) for 2-hour morning period
Network Science Analytics Analysis of Network Flow Data 37
◮ Estimated marginal posteriors for 8 of the 64 OD flows ( = true)
Tebaldi and West: Bayesian Inference
Traffic 569
Atol AtoG AtoF FtoM
M.11
~~~~~~~~~~~~~~~0
S I A g, a fkM 8a
50 100 150 200 250 20 40 60 80 1000 1O0 1200 1300 1400 600 700 800 900 1000 1200
ItoH ItoG ItoE LtoF
v
1
3
0~~~~~~~~~~~~~~~~~~~~~~~~~~~
0 2040 6080 100 50 100 150 0 20 4060 80 100 20 40 60 80
HtoM MtoA MtoF GtoO
20 40 60 820 02040 08000 0006080100 020 4060 80 100 120
NtoH NtoA OtoF OtoA
8 | > 8 1 t
L
8,
X0
00
200 40 60 80 100 120 20 40 80 400 450 500 550 600 650
Figure
Distributions for 16 Components
X Under Informed Gamma Priors
the A,. The true values X, appear
the x axis.
common OD pair, we add the probabilities
assignment to the columns (i.e., fixed routes) within such subsets. In this case, the
12 OD pairs generates 27 in the super- network. Now turn to the issue of modeling and inferring route counts. Consider any OD pair a = (i, j) in the
net- work, with corresponding counts
cre- ating a corresponding set
fixed routes between i and j generates some number, say ka,
such routes. Given the
probability routing matrix, we can trivially com- pute the resulting probabilities
Pa = (pa, v** Pa,ka)' of
selecting each
these fixed routes; note that these ka prob- abilities sum to unity. Write Xa,t for the number
trips that take fixed route t from i to
given Xa, the dis- aggregated counts Xa,i, ... ., Xa, k are conditionally multi- nomially distributed among the ka fixed routes
the total Xa and with route selection probabilities
the earlier assumption that Xa - Po(Aa), this trivially im- plies that the disaggregated, fixed route counts are them- selves marginally independent and Poisson distributed, with
Xa,t
VPo(p,tAa)
for
t = 1,
... I
independently
across OD pairs a, we have
P(Xa,l
v v Xa,kc lAa,
Pa)
_
(Pa,tAa)Xa,t
exp(-Pa,tAa)
n ~~~Xa,t 7
which reduces to
ka
P(Xa, 1. * Xa,kj IAa,
Pa) o Aa exp(-Aa) fJpXat (8)
t=1
as a function
(Aa, Pa), noting
that
Xa
t=1
Xa,t.
The immediate consequence
the
approach to inference about route counts in the fixed routing problem in Sections 2 and 3 applies directly to the super-network. Conditional
the parameters A and
Tebaldi and West: Bayesian Inference
Traffic 567
Ato I Ato G Ato F Fto M
~~~~~~~~~0
08
it)
U
8~~~~~~~~~~~~
? I
02i
a 0S1 . .
.~
~~~~~~~~~~~~~~g
100 200 300 400 20 40 60 80 900 1000 1200 1400 800 1000 1200 1400
ItoH ItoG ItoF LtoF
8~~~~~~~~~~
8 j&_
^
8 >
~~~~~~~~8
3.8
20 40 60 50 100 150 50 100 150 200 20 40 60
HtoM MtoA MtoF GtoO
0~~~~~~~~~~~~~~~~~~~~~ 0~~~~~~~~~~~~~~~~~~~~~
80>
8
|0
_ _ _ _ _ _0
20 40 60 80 2040 6080 100 20 40 60 80 20 60 80100120
NtoH NtoA OtoF OtoA
0"
~~~~~~~~~~~0
8 1 8 1K=SX 8 i_ 8 I"i
|
8
LO
20 40 60 80 120140 50 10 0 150 10 20 30 4 0 300 400 500 600
Figure
Distributions for 16 Components
X Under Uniform Priors
the As. The true values Xs appear
the x axis.
secondary routes used less frequently either as alternatives in times
congestion
primary routes
as
alternatives for
reasons. Primary and secondary routes between a specified OD pair will typically share a common subset
links, with a few additional links being specific to
a few routes (Figure 12). A neat extension
the development in Sections 2 and 3 provides for analysis using the Bayesian approach devel-
for the fixed routing network. We discuss this later, following a precise definition and discussion
the random routing model. Given a specific OD pair a = (i,j), messages leaving node i may now take differing routes to node
Mar- kovian routing model simply assumes that at any node
the way to the destination, each message exits
a link determined by a set
link choice probabilities, indepen- dently
the path taken to the current node and indepen- dently across messages. In some cases there will be just
exit link possible, in
cases there may be several. As in Vardi (1996), the setup can be summarized through a modified routing matrix that summaries the link choice probabilities for all possible OD pairs. As in the fixed rout- ing case, the matrix has rows indexing links in the network and columns indexing OD pairs; now, however, the entries are probabilities determining the selection
links (rows)
trips between specified OD pairs (columns). We study an example matrix from Vardi (1996) further later; with rows and columns labelled by links and OD pairs, this has the probability matrix A given in Figure 6.
Here we have 4 nodes, r = 9 directed links and c = 12
OD pairs. Consider, for example, the OD pair
trip from A to B has an 80% chance
moving directly along link A -? B and terminating there, with the com- plementary 20% chance it travels from A
that it follows this latter path, it then travels along C -? B directly and terminates with an 80% chance;
it moves along the two consecutive links
C -? D -? B and
terminates. By elementary probability computations, we can ◮ Tend to overestimate smaller flows with a uniform prior
Network Science Analytics Analysis of Network Flow Data 38
◮ Consider a prior guess µ(0) of µ, normalized such that
ij
◮ Relative entropy “distance” between µ and µ(0) given by
ij
ij /µ(0) ++}
Network Science Analytics Analysis of Network Flow Data 39
◮ Traffic matrix estimation: minimize D(µµ(0)) subject to x ≈ Rµ ◮ Dualize constraints via Lagrange multipliers λ ∈ RNe, solve
µ,λ D(µµ(0)) + λT(x − Rµ) ◮ Given λ, optimality condition yields the estimator (R = [r11, . . . , rIJ])
ij exp
Network Science Analytics Analysis of Network Flow Data 40
◮ Can view D(µµ(0)) as regularizer for x = Rz → Penalized LS
µ0 x − Rx2 + λD(µµ(0))
◮ Couple interpretations:
ij
◮ Ex: simple gravity model prior µ(0) ij
i+ µ(0) +j (more soon) ◮ Y. Zhang et al, “An information-theoretic approach to traffic matrix
Network Science Analytics Analysis of Network Flow Data 41
◮ Q: Traffic matrix estimation over time periods t = 1, . . . , τ? ◮ Given: link counts x1:τ := {x(t)}τ t=1 and routing R1:τ := {R(t)}τ t=1 ◮ Determine: OD flows z1:τ := {z(t)}τ t=1, where x(t) ≈ R(t)z(t)
Time Flow volume True Estimated
◮ Dynamic methods categorization: simultaneous or sequential ◮ A. Soule et al, “Traffic matrices: Balancing measurements, inference
Network Science Analytics Analysis of Network Flow Data 42
◮ Simultaneous methods mostly based on the linear model
◮ Penalized LS criteria employed to form ˆ
µ1:τ τ
◮ Separable penalty J(µ1:τ) =
t Jt(µ(t)) not uncommon
◮ Ex: Jt(·) based on independent Gaussian or entropy-based priors
◮ Temporal correlations in x1:τ ignored → τ decoupled static problems ◮ Over short spans can assume µ(t) = µ, treat x1:τ as replicates
Network Science Analytics Analysis of Network Flow Data 43
◮ Sequential methods leverage time correlations via Kalman filtering ◮ State µ(t) and link count (measurement) X(t) equations
◮ Kalman filter (KF) in a nutshell
◮ Prediction step: form prediction ˆ
◮ Correction step: Update ˆ
◮ Also update recursively the error covariance matrix
Network Science Analytics Analysis of Network Flow Data 44
◮ Initialize ˆ
◮ Prediction step:
◮ Kalman gain update:
◮ Correction step:
t+1
Network Science Analytics Analysis of Network Flow Data 45
◮ Model matrices Φ(t), Ψ(t) and Σ(t) must be determined
◮ Estimation depends on the model and data available
◮ Z. Ghahramani and G. Hinton, “Parameter estimation for linear
◮ KF should be periodically recalibrated → readjust Φ, Ψ and Σ
e from diagonal of R(t)Mt:tRT(t) + Σ
Network Science Analytics Analysis of Network Flow Data 46
Network Science Analytics Analysis of Network Flow Data 47
◮ Q: Why do ISPs monitor their networks routinely?
◮ Availability of traffic matrices Z(t) key to traffic monitoring ◮ While possible, rarely measure Internet flows Zij(t) at ISP level
◮ Limited z(t) to calibrate Internet traffic matrix estimation methods
Network Science Analytics Analysis of Network Flow Data 48
◮ Abilene backbone: Nv = 11 PoPs, Ne = 30 links, Nf = 110 flows ◮ Measure flows z1:τ for τ = 12 × 24 × 7 = 2, 016 time slots
◮ Abilene routing matrix R ∈ {0, 1}30×110 given, time invariant
Network Science Analytics Analysis of Network Flow Data 49
25 50 75 25 50 75 Mon Tue Wed Thu Fri Sat Sun 25 50 75
Network Science Analytics Analysis of Network Flow Data 50
◮ Compare static and dynamic methods for traffic matrix estimation ◮ Method 1: entropy-based approach termed tomogravity
z0 x − Rz2 + λ
++
ij
ij
i+ z(0) +j
◮ Method 2: KF with state and measurement equations
Network Science Analytics Analysis of Network Flow Data 51
◮ Relative error averaged over OD pairs, as a function of time
Time Spatial Relative Error Tue Wed Thu Fri Sat Sun 0.2 0.4 0.6 0.8 − − − −
◮ Tomogravity overestimates, after bias-correction comparable to KF
Network Science Analytics Analysis of Network Flow Data 52
◮ Relative error averaged over time, for each OD pair in log-log scale
−1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 Tomogravity (bias corrected) Kalman Filtering
Network Science Analytics Analysis of Network Flow Data 53
◮ True flows superimposed with tomogravity and KF predictions
10 20 30 Tue Wed Thu Fri Sat Sun 15 30 45 Tue Wed Thu Fri Sat Sun
◮ Tomogravity completely misses the dynamics of the first flow
Network Science Analytics Analysis of Network Flow Data 54
Network Science Analytics Analysis of Network Flow Data 55
◮ Consider a network graph G(V , E). Let P be the set of paths in G
◮ Network flows costs at two levels of granularity: paths and links
◮ Cost associated to path = sum of the costs of the links traversed ◮ Ex: end-to-end delay is the sum of the delays in intermediate links ◮ Our focus: a particular class of problems involving inference of costs
Network Science Analytics Analysis of Network Flow Data 56
◮ Actively inject traffic to measure cobs, e.g., multicast probing
◮ Tomography: unveil “internal” network characteristics
◮ Ex: determine link loss rates from packet loss measurements ◮ M. Coates et al, “Internet tomography,” IEEE Signal Processing
Network Science Analytics Analysis of Network Flow Data 57
◮ Kriging coined in geosciences for spatial interpolation or smoothing ◮ Key: exploit redundancies among links used by various paths ◮ D. Chua et al, “Network kriging,” IEEE J. Selected Areas in
Network Science Analytics Analysis of Network Flow Data 58
◮ Number of paths Np is much larger than Ne. Interpolation idea:
mx
◮ But in general r := rank(R) < Ne, so x not identifiable
◮ Of course do not need x to recover cmiss ⇒ xR(RT ) suffices ◮ Y. Chen et al, “An algebraic approach to practical and scalable
Network Science Analytics Analysis of Network Flow Data 59
◮ Graph G(V , E) with Nv = 4 and Ne = 3, paths {AB, AC, BC}
◮ Cannot identify x1 and x2 → Always show up summed in paths
Network Science Analytics Analysis of Network Flow Data 60
◮ Key: monitor r = rank(R) independent paths to recover xR(RT )
mxR(RT ) = RT mRo
◮ For Np = N2 v , conjecture rank(R) = O(Nv log Nv) [Chen et al ’04]
Network Science Analytics Analysis of Network Flow Data 61
◮ Interpolation appealing if we can monitor r = rank(R) paths
◮ Network kriging: recast problem as one of statistical prediction
◮ Since r = rank(R), can write the SVD of RT as
r
k ≈ s≪r
k ◮ Observation: often most of the smaller σk are close to zero
Network Science Analytics Analysis of Network Flow Data 62
◮ Singular values of the Abilene routing matrix R
5 10 15 20 25 30 10 20 30 40 50 k λk ◮ Spectral gap apparent. Effective rank s ∈ {5, 10}, even s = 2?
Network Science Analytics Analysis of Network Flow Data 63
◮ Visualize top right singular vectors {vk}4 k=1 of RT (evecs. of RRT)
◮ Northern E-W meta-path {vk}3 k=1, and southern E-W meta-path v4
Network Science Analytics Analysis of Network Flow Data 64
◮ Consider predicting an arbitrary linear summary aTc of c ◮ Ex: network-wide average path cost a = 1/Np, or cij where a = eij ◮ Let x be a realization of X, with mean µ and var [X] = Σ
◮ Given s ≤ rank(R) measured path costs cobs, find
p
mCmiss
Network Science Analytics Analysis of Network Flow Data 65
◮ Restrict attention to linear (L)MMSE predictors ˆ
mµ + aT mVmoV−1
mΣRo ◮ Estimate µ from the data via generalized LS, i.e.,
◮ Substitution of ˆ
mVmoV−1
◮ SVD-based path selection to minimize E
Network Science Analytics Analysis of Network Flow Data 66
◮ Abilene backbone: Nv = 11 PoPs, Ne = 30 links, Np = 110 paths ◮ Measure link delays x1:τ for τ = 6 × 24 × 3 = 432 time slots
◮ Abilene routing matrix R ∈ {0, 1}30×110 given, time invariant
◮ Applied the network kriging predictor to a subset cobs(t)
mVmoV−1
Network Science Analytics Analysis of Network Flow Data 67
◮ Average path delay in Abilene predicted with s = 3, 5, 7, or 9 paths
◮ Biased predictions, missing link information in approximated R
◮ Predictions capture well the delay dynamics, for all s
Network Science Analytics Analysis of Network Flow Data 68
Network Science Analytics Analysis of Network Flow Data 69
◮ Motivating reasons
◮ Assess network health ◮ Fault diagnosis ◮ Network planning
◮ Application domains
◮ Old 8-second rule for WWW ◮ Content-delivery networks ◮ Peer-to-peer networks ◮ Multiuser games ◮ Dynamic server selection
◮ Goal: infer path delays from limited end-to-end measurements
Network Science Analytics Analysis of Network Flow Data 70
◮ Consider a network graph G(V , E). Let P be the set of paths in G ◮ Several challenges in measuring all end-to-end path delays
v )
◮ Q: Can fewer measurements suffice? ◮ A: Yes! Most paths share multiple links ⇒ Correlations [Chua’06] ◮ End-to-end delay prediction problem: Given delay measurements
Network Science Analytics Analysis of Network Flow Data 71
◮ Given (cross-)covariances Vo = cov[cobs] and Vmo = cov[cmiss, cobs] ◮ The universal kriging predictor is
◮ Link delays x ∈ RNe and Σ = cov[x] ⇒ From model cov[c] is
m]⊤ known, selected heuristically
Network Science Analytics Analysis of Network Flow Data 72
◮ Network kriging prediction for a single temporal snapshot of delays ◮ D. Chua et al, “Network kriging,” IEEE J. Sel. Areas Communications,
◮ Wavelet-based approach for spatio-temporal delay prediction
◮ Diffusion wavelet matrix constructed from the topology of G ◮ Can capture temporal correlations, up to τ time slots ◮ High complexity O(τ 3P3) ⇒ Challenging for τ > 10
◮ M. Coates et al, “Compressed network monitoring for IP and all-optical
◮ Q: Should the same set of paths be measured every time slot?
◮ Low-complexity spatio-temporal inference with online path selection
Network Science Analytics Analysis of Network Flow Data 73
◮ Model delay cp(t) measured on path p ∈ P at time t as
◮ Component χp(t) captures queuing delays, traffic dependent
◮ Nonstationary: Random walk with driving noise covariance Cη
◮ Component νp(t) lumps propagation, transmission, processing delays
◮ Traffic independent, temporally white with covariance Cν = αGG⊤
◮ Measurement noise ǫp(t) i.i.d. over paths and time, var [ǫp(t)] = σ2
Network Science Analytics Analysis of Network Flow Data 74
◮ Paths measured on subset Pobs ⊂ P, use sampling matrix So(t)
◮ Kriged Kalman filter (KKF) state and measurement equations
◮ Goal: given historical data H(t) = {cobs(τ)}t τ=1, predict cmiss(t) ◮ K. Rajawat et al, “Dynamic network delay cartography,” IEEE
Network Science Analytics Analysis of Network Flow Data 75
◮ State and covariance update recursions
◮ KKF gain
◮ Kriging predictor ˆ
Network Science Analytics Analysis of Network Flow Data 76
◮ Q: How do we find the spatial covariance Cν? ◮ Idea: paths sharing multiple links should be highly correlated
◮ Similar principles used to define graph kernels ◮ Can also handle route changes, especially incremental changes
Network Science Analytics Analysis of Network Flow Data 77
◮ KKF can model and track network wide delays given sample paths ◮ Q: Practical sampling of paths? Optimal measurements? Criterion? ◮ Error covariance matrix (define Φ(t) = [M(t − 1) + Cν + Cη] /σ2)
m(t) ◮ Optimal experimental design
Pobs⊂P log det(Mmiss(t)),
p ◮ Criterion: D-optimal design, i.e., entropy of a Gaussian RV
Network Science Analytics Analysis of Network Flow Data 78
◮ Simple greedy algorithm to select observed paths Pobs ◮ Repeat |Pobs| times: Pobs ← Pobs ∪ arg maxp / ∈Pobs δPobs(p), where
p,p
◮ Increments δPobs(p) efficiently evaluated in O(|P||Pobs|3)
◮ Can be modified to handle cases when
Network Science Analytics Analysis of Network Flow Data 79
◮ Internet2 backbone: 72 paths, lightly loaded network ◮ One-way delay measurements collected using OWAMP
◮ Training phase employed to estimate Cη, α [Myers’76]
◮ Modified estimators to handle measurements on subsets of paths ◮ First 1000 samples on 50 random paths used for training Network Science Analytics Analysis of Network Flow Data 80
True Kriging Wavelet KKF
Network Science Analytics Analysis of Network Flow Data 81
◮ Normalized mean-square prediction error as figure of merit
T
Network Science Analytics Analysis of Network Flow Data 82
◮ NZ-AMP delay dataset: 186 paths, heavily loaded network ◮ Round-trip-times measured using ICMP, paths via scamper
Network Science Analytics Analysis of Network Flow Data 83
Random path selection “Optimal” path selection
◮ NMSPE order of magnitude larger than for the Internet2 data
Network Science Analytics Analysis of Network Flow Data 84
◮ Prediction of path delays. Plot ˆ
ij
ij
Network Science Analytics Analysis of Network Flow Data 85
◮ Network traffic flows ◮ Routing matrix ◮ Traffic matrix ◮ Link counts ◮ Network flow costs ◮ Network monitoring ◮ Gravity model ◮ Generalized linear model ◮ Traffic matrix estimation ◮ Network tomography ◮ Poisson traffic models ◮ Entropy minimization ◮ Tomogravity ◮ Kalman filter ◮ End-to-end measurements ◮ Active network tomography ◮ Network kriging ◮ Path-cost interpolation ◮ Identifiability ◮ Effective rank ◮ (L)MMSE predictor ◮ Path selection ◮ Diffusion wavelets ◮ Kriged Kalman filter ◮ Optimal experimental design ◮ Submodular function
Network Science Analytics Analysis of Network Flow Data 86