Diffusion Spline Adaptive Filtering Authors : Simone Scardapane, - PowerPoint PPT Presentation

24 th European Signal Processing Conference (EUSIPCO’16) Diffusion Spline Adaptive Filtering Authors : Simone Scardapane, Michele Scarpiniti, Danilo Comminiello and Aurelio Uncini

Contents Introduction Overview State of the art Theoretical background Spline adaptive filter Proposed diffusion SAF Network setting Derivation of the algorithm Experimental validation Setup Results Conclusions Conclusive remarks

Content at a glance Setting : L agents in a network received a streaming flux of data, wishing to infer a common predictive function with local communication. Problem : Standard diffusion LMS (D-LMS) is simple and efficient, however it may perform poorly when the underlying model is nonlinear . Objective : To design a novel nonlinear diffusion filter having similar computa- tional complexity as the standard D-LMS.

Current approaches and possible shortcomings 1. Fixed nonlinear expansion [1]: communication overhead might be too significant. 2. Distributed kernel filtering [2]: resulting model is formulated in terms of data from all agents. 3. Distributed machine learning/optimization [3]: might not be ad- equate to a streaming setting with low computational power avail- able locally. [1] Scardapane, S., Wang, D., Panella, M. and Uncini, A., 2015. Distributed learning for random vector functional-link networks. Inform. Sciences , 301, pp. 271-284. [2] Predd, J.B., Kulkarni, S.B. and Poor, H.V., 2006. Distributed learning in wireless sensor networks. IEEE Signal Process. Mag. , 23(4), pp. 56-69. [3] Boyd, S., Parikh, N., Chu, E., Peleato, B. and Eckstein, J., 2011. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. and Trends in Machine Learning , 3(1), pp. 1-122.

Contents Introduction Overview State of the art Theoretical background Spline adaptive filter Proposed diffusion SAF Network setting Derivation of the algorithm Experimental validation Setup Results Conclusions Conclusive remarks

Data model Denote by x n a buffer of the last M input samples. Assume that an unknown Wiener model is generating the desired response as follows: w T � � d [ n ] = f 0 0 x n + ν [ n ] , (1) A spline adaptive filter (SAF) computes the output similarly [1, 2], i.e. first a linear filtering operation: s [ n ] = w T n x n . (2) Then, the final output is computed via spline interpolation over s [ n ] . [1] Scarpiniti, M., Comminiello, D., Parisi, R. and Uncini, A., 2013. Nonlinear spline adaptive filtering. Signal Process. , 93(4), pp. 772-783. [2] Scarpiniti, M., Comminiello, D., Scarano, G., Parisi, R. and Uncini, A., 2016. Steady-State Performance of Spline Adaptive Filters. IEEE Trans. on Signal Process. , 64(4), pp. 816-828.

Visualization of the SAF  ( ) s Control points i.c.  ( ) i u q s x Q , q x ,0 q q q q    x i , x i , 1 x i , 2 x i , 3 th tract i

Spline definition ◮ The spline is defined by a set of Q control points (called knots ), � � denoted as Q i = q x , i q y , i . ◮ We suppose that the knots are uniformly distributed, i.e. q x , i + 1 = q x , i + ∆ x , for a fixed ∆ x ∈ R . ◮ We also constrain the knots to be symmetrically spaced around the origin. The output is obtained by an interpolating polynomial of order P , passing by the closest knot to s [ n ] and its P successive knots.

Spline equations Given the index i of the closest knot, we define the normalized abscissa value between q x , i and q x , i + 1 as: u = s [ n ] � s [ n ] � ∆ x − (3) . ∆ x From u we can compute the reference values: u P u P − 1 . . . u 1 � T � u = (4) � T � q i , n = q y , i q y , i + 1 . . . q y , i + P (5) The output of the filter then given by: y [ n ] = ϕ ( s [ n ]) = u T Bq i , n , (6) where B ∈ R ( P + 1 ) × ( P + 1 ) is called the spline basis matrix.

Contents Introduction Overview State of the art Theoretical background Spline adaptive filter Proposed diffusion SAF Network setting Derivation of the algorithm Experimental validation Setup Results Conclusions Conclusive remarks

Network model ◮ The connectivity of the network is represented as a real-valued matrix C ∈ R L × L , where entry C kl > 0 if nodes k and l are con- nected, 0 otherwise. ◮ The symbol N k will denote the inclusive neighborhood of node k . ◮ We require the mixing coefficients to define a convex combination for every node: L � C kl ≥ 0 and C kl = 1 k , l = 1 , . . . , L . (7) l = 1 ◮ At a generic time instant n , each agent receives some input/output � � x ( k ) n , d ( k ) [ n ] data denoted by ◮ We assume that streaming data at the local level is generated ac- cording to: � � 0 x ( k ) d ( k ) [ n ] = f 0 w T + ν ( k ) [ n ] . (8) n

Visualization of the setting N 1 k 2 3 k ( ) [ ] k s n ( ) k ( ) k ( ) [ ] k ( ) k y n x w q n n i n ,

Algorithm (1) The network objective is to minimize: L L � e ( k ) [ n ] 2 � � J ( k ) � J glob ( w , q ) = loc ( w , q ) = E (9) , k = 1 k = 1 In the diffusion SAF (D-SAF), we first diffuse the linear coefficients: � C kl w ( l ) ψ ( k ) = (10) n . n w-diffusion l ∈N k Then, nodes perform a second diffusion step over their nonlinear co- efficients: ξ ( k ) � C kl q ( l ) i , n = (11) i , n . q-diffusion l ∈N k This step requires combination only of q ( k ) i , n , hence its complexity is defined only by the spline order P .

Algorithm (2) The spline output given the new span is obtained as: y ( k ) [ n ] = ϕ k ( s ( k ) [ n ]) = u T B ξ ( k ) (12) i , n . From this, the local error is given as e ( k ) [ n ] = d ( k ) [ n ] − y ( k ) [ n ] . The two gradient descent steps are given by: w ( k ) + µ ( k ) w e ( k ) [ n ] ϕ ′ ( s ( k ) [ n ]) x ( k ) n + 1 = ψ ( k ) (13) n , n w-adapt q ( k ) i , n + 1 = ξ ( k ) i , n + µ ( k ) k e ( k ) [ n ] B T u . (14) q-adapt Note that D-LMS is a special case of the D-SAF, where each node ini- tializes its nonlinearity as the identity, and µ ( k ) = 0 , k = 1 , . . . , L . q

Contents Introduction Overview State of the art Theoretical background Spline adaptive filter Proposed diffusion SAF Network setting Derivation of the algorithm Experimental validation Setup Results Conclusions Conclusive remarks

Experimental setup ◮ We set L = 10 agents with random connectivity. ◮ Weights w 0 are extracted from a normal distribution. ◮ Local input signal is generated according to: � 1 − a 2 x k [ n ] = a k x k [ n − 1 ] + k ǫ [ n ] , (15) ◮ Local correlation coefficients, noise variances and local step-sizes are chosen randomly in some given intervals. ◮ Experiments are repeated 15 times, by keeping fixed the topology of the network and the optimal parameters of the system. ◮ MATLAB code is available under open-source license. 1 1 https://bitbucket.org/ispamm/diffusion-spline-filtering

MSE evolution 0 NC-LMS -5 D-LMS NC-SAF D-SAF -10 MSE -15 -20 -25 -30 0 0.5 1 1.5 2 2.5 Sample × 10 4 Figure 1 : MSE evolution, averaged across the nodes.

MSD evolution 5 2 D-SAF D-SAF 0 Node 1 Node 1 0 Node 6 Node 6 Node 8 MSD (non-linear part) Node 8 MSD (linear part) -2 -5 -4 -10 -6 -15 -8 -20 -10 0 0.5 1 1.5 2 2.5 0 0.5 1 1.5 2 2.5 × 10 4 × 10 4 Sample Sample (a) Linear MSD (b) nonlinear MSD Figure 2 : MSD evolution for D-SAF and 3 representative nodes running NC-SAF.

Final nonlinearity 2 2 Real Real NC-SAF (3 nodes) D-SAF 1 1 AF output y [ n ] AF output y [ n ] 0 0 -1 -1 -2 -2 -2 -1 0 1 2 -2 -1 0 1 2 Linear combiner output s [ n ] Linear combiner output s [ n ] Figure 3 : Final estimation of the nonlinear model. (a) Three representative nodes running NC-SAF. (b) Final spline of the nodes running D-SAF.

Contents Introduction Overview State of the art Theoretical background Spline adaptive filter Proposed diffusion SAF Network setting Derivation of the algorithm Experimental validation Setup Results Conclusions Conclusive remarks

Conclusive remarks ◮ We have introduced a distributed algorithm for adapting SAFs. ◮ The algorithm allows for a flexible nonlinear estimation, with a small increase in computational complexity. ◮ In particular, the algorithm requires in average only twice as much computations as the standard D-LMS. ◮ Apart from a theoretical analysis, we plan to extend the algorithm to the case of second-order adaptation with Hessian information, ATC combiners, and asynchronous networks. ◮ Additionally, we plan on investigating diffusion protocols for more general architectures, including Hammerstein spline filters.

Questions?