Diffusion Spline Adaptive Filtering Authors : Simone Scardapane, - - PowerPoint PPT Presentation

24th European Signal Processing Conference (EUSIPCO’16)

Diffusion Spline Adaptive Filtering

Authors: Simone Scardapane, Michele Scarpiniti, Danilo Comminiello and Aurelio Uncini

Contents

Introduction Overview State of the art Theoretical background Spline adaptive filter Proposed diffusion SAF Network setting Derivation of the algorithm Experimental validation Setup Results Conclusions Conclusive remarks

Content at a glance

Setting: L agents in a network received a streaming flux of data, wishing to infer a common predictive function with local communication. Problem: Standard diffusion LMS (D-LMS) is simple and efficient, however it may perform poorly when the underlying model is nonlinear. Objective: To design a novel nonlinear diffusion filter having similar computa- tional complexity as the standard D-LMS.

Current approaches and possible shortcomings

• 1. Fixed nonlinear expansion [1]: communication overhead might

be too significant.

• 2. Distributed kernel filtering [2]: resulting model is formulated in

terms of data from all agents.

• 3. Distributed machine learning/optimization [3]: might not be ad-

equate to a streaming setting with low computational power avail- able locally.

[1] Scardapane, S., Wang, D., Panella, M. and Uncini, A., 2015. Distributed learning for random vector functional-link networks. Inform. Sciences, 301, pp. 271-284. [2] Predd, J.B., Kulkarni, S.B. and Poor, H.V., 2006. Distributed learning in wireless sensor networks. IEEE Signal Process. Mag., 23(4), pp. 56-69. [3] Boyd, S., Parikh, N., Chu, E., Peleato, B. and Eckstein, J., 2011. Distributed

• ptimization and statistical learning via the alternating direction method of multipliers.
• Found. and Trends in Machine Learning, 3(1), pp. 1-122.

Contents

Introduction Overview State of the art Theoretical background Spline adaptive filter Proposed diffusion SAF Network setting Derivation of the algorithm Experimental validation Setup Results Conclusions Conclusive remarks

Data model

Denote by xn a buffer of the last M input samples. Assume that an unknown Wiener model is generating the desired response as follows: d[n] = f0

• wT

0xn

• + ν[n] ,

(1) A spline adaptive filter (SAF) computes the output similarly [1, 2], i.e. first a linear filtering operation: s[n] = wT

nxn .

(2) Then, the final output is computed via spline interpolation over s[n].

[1] Scarpiniti, M., Comminiello, D., Parisi, R. and Uncini, A., 2013. Nonlinear spline adaptive filtering. Signal Process., 93(4), pp. 772-783. [2] Scarpiniti, M., Comminiello, D., Scarano, G., Parisi, R. and Uncini, A., 2016. Steady-State Performance of Spline Adaptive Filters. IEEE Trans. on Signal Process., 64(4), pp. 816-828.

Visualization of the SAF

th tract

i ( ) s  s

( )

i u



, , 1 , 2 , 3 x i x i x i x i

q q q q

   , x Q

q

,0 x

q Control points i.c.

Spline definition

◮ The spline is defined by a set of Q control points (called knots),

denoted as Qi =

• qx,i qy,i
• .

◮ We suppose that the knots are uniformly distributed, i.e. qx,i+1 =

qx,i + ∆x, for a fixed ∆x ∈ R.

◮ We also constrain the knots to be symmetrically spaced around

the origin. The output is obtained by an interpolating polynomial of order P, passing by the closest knot to s[n] and its P successive knots.

Spline equations

Given the index i of the closest knot, we define the normalized abscissa value between qx,i and qx,i+1 as: u = s[n] ∆x − s[n] ∆x

• .

(3) From u we can compute the reference values: u =

• uP uP−1 . . . u 1

T (4) qi,n =

• qy,i qy,i+1 . . . qy,i+P

T (5) The output of the filter then given by: y[n] = ϕ(s[n]) = uTBqi,n , (6) where B ∈ R(P+1)×(P+1) is called the spline basis matrix.

Contents

Introduction Overview State of the art Theoretical background Spline adaptive filter Proposed diffusion SAF Network setting Derivation of the algorithm Experimental validation Setup Results Conclusions Conclusive remarks

Network model

◮ The connectivity of the network is represented as a real-valued

matrix C ∈ RL×L, where entry Ckl > 0 if nodes k and l are con- nected, 0 otherwise.

◮ The symbol Nk will denote the inclusive neighborhood of node k. ◮ We require the mixing coefficients to define a convex combination

for every node: Ckl ≥ 0 and

L

• l=1

Ckl = 1 k, l = 1, . . . , L . (7)

◮ At a generic time instant n, each agent receives some input/output

data denoted by

• x(k)

n , d(k)[n]

• ◮ We assume that streaming data at the local level is generated ac-

cording to: d(k)[n] = f0

• wT

0x(k) n

• + ν(k)[n] .

(8)

Visualization of the setting

k 1 3 2

( ) k n

w

( ) , k i n

q

( ) k n

x

( )[ ] k

s n

( )[ ] k

y n

k

N

Algorithm (1)

The network objective is to minimize: Jglob(w, q) =

L

• k=1

J(k)

loc(w, q) = L

• k=1

E

• e(k)[n]2

, (9) In the diffusion SAF (D-SAF), we first diffuse the linear coefficients: ψ(k)

n

=

• l∈Nk

Cklw(l)

n .

(10) w-diffusion Then, nodes perform a second diffusion step over their nonlinear co- efficients: ξ(k)

i,n =

• l∈Nk

Cklq(l)

i,n .

(11) q-diffusion This step requires combination only of q(k)

i,n , hence its complexity is

defined only by the spline order P.

Algorithm (2)

The spline output given the new span is obtained as: y(k)[n] = ϕk(s(k)[n]) = uTBξ(k)

i,n .

(12) From this, the local error is given as e(k)[n] = d(k)[n] − y(k)[n]. The two gradient descent steps are given by: w(k)

n+1 = ψ(k) n

+ µ(k)

w e(k)[n]ϕ′(s(k)[n])x(k) n ,

(13) w-adapt q(k)

i,n+1 = ξ(k) i,n + µ(k) k e(k)[n]BTu .

(14) q-adapt Note that D-LMS is a special case of the D-SAF, where each node ini- tializes its nonlinearity as the identity, and µ(k)

q

= 0, k = 1, . . . , L.

Contents

Introduction Overview State of the art Theoretical background Spline adaptive filter Proposed diffusion SAF Network setting Derivation of the algorithm Experimental validation Setup Results Conclusions Conclusive remarks

Experimental setup

◮ We set L = 10 agents with random connectivity. ◮ Weights w0 are extracted from a normal distribution. ◮ Local input signal is generated according to:

xk[n] = akxk[n − 1] +

• 1 − a2

kǫ[n] ,

(15)

◮ Local correlation coefficients, noise variances and local step-sizes

are chosen randomly in some given intervals.

◮ Experiments are repeated 15 times, by keeping fixed the topology

• f the network and the optimal parameters of the system.

◮ MATLAB code is available under open-source license.1

1https://bitbucket.org/ispamm/diffusion-spline-filtering

MSE evolution

Sample ×10 4 0.5 1 1.5 2 2.5 MSE

• 30
• 25
• 20
• 15
• 10
• 5

NC-LMS D-LMS NC-SAF D-SAF

Figure 1 : MSE evolution, averaged across the nodes.

MSD evolution

Sample ×10 4 0.5 1 1.5 2 2.5 MSD (linear part)

• 20
• 15
• 10
• 5

5

D-SAF Node 1 Node 6 Node 8

(a) Linear MSD

Sample ×10 4 0.5 1 1.5 2 2.5 MSD (non-linear part)

• 10
• 8
• 6
• 4
• 2

2

D-SAF Node 1 Node 6 Node 8

(b) nonlinear MSD Figure 2 : MSD evolution for D-SAF and 3 representative nodes running NC-SAF.

Final nonlinearity

Linear combiner output s [n]

• 2
• 1

1 2 AF output y[n]

• 2
• 1

1 2

Real NC-SAF (3 nodes)

Linear combiner output s [n]

• 2
• 1

1 2 AF output y[n]

• 2
• 1

1 2

Real D-SAF

Figure 3 : Final estimation of the nonlinear model. (a) Three representative nodes running NC-SAF. (b) Final spline of the nodes running D-SAF.

Contents

Introduction Overview State of the art Theoretical background Spline adaptive filter Proposed diffusion SAF Network setting Derivation of the algorithm Experimental validation Setup Results Conclusions Conclusive remarks

Conclusive remarks

◮ We have introduced a distributed algorithm for adapting SAFs. ◮ The algorithm allows for a flexible nonlinear estimation, with a

small increase in computational complexity.

◮ In particular, the algorithm requires in average only twice as much

computations as the standard D-LMS.

◮ Apart from a theoretical analysis, we plan to extend the algorithm

to the case of second-order adaptation with Hessian information, ATC combiners, and asynchronous networks.

◮ Additionally, we plan on investigating diffusion protocols for more

general architectures, including Hammerstein spline filters.

Questions?