Rethinking Sketching as Sampling: Efficient Approximate Solution to - PowerPoint PPT Presentation

Rethinking Sketching as Sampling: Efficient Approximate Solution to Linear Inverse Problems Fernando Gama , A. G. Marques, G. Mateos & A. Ribeiro Dept. of Electrical and Systems Engineering University of Pennsylvania fgama@seas.upenn.edu GlobalSIP, December 9, 2016 Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 1/23

PCA Classification 5 5 10 10 15 15 20 20 25 25 5 10 15 20 25 5 10 15 20 25 Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 2/23

PCA Classification 5 5 10 10 15 15 20 20 25 25 5 10 15 20 25 5 10 15 20 25 ◮ Classify images according to the digits handwritten on them Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 2/23

PCA Classification 5 5 10 10 15 15 20 20 25 25 5 10 15 20 25 5 10 15 20 25 ◮ Classify images according to the digits handwritten on them ◮ Perform PCA ⇒ Keep first few coefficients ⇒ Apply linear classifier few PCA Image coefficients { 0 , 1 } n = 784 k = 20 Linear PCA Classifier Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 2/23

PCA Classification 5 5 10 10 15 15 20 20 25 25 5 10 15 20 25 5 10 15 20 25 ◮ Few PCA coefficients ⇒ Problem is inherently lower-dimensional ◮ Improves classification task ⇒ Low-pass filter to remove noise ◮ Lower-dimensional representation can also save computational cost Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 2/23

Computational Cost 5 5 10 10 15 15 20 20 25 25 5 10 15 20 25 5 10 15 20 25 ◮ Note that in performing PCA we need the complete image ◮ However, there are pixels that do not contribute to classification ⇒ Pixels on the border of the image, for example ◮ And there are pixels that are more important for classification ⇒ Pixels that are white in one image but black in the other Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 3/23

Computational Cost 5 5 10 10 15 15 20 20 25 25 5 10 15 20 25 5 10 15 20 25 ◮ Note that in performing PCA we need the complete image ◮ However, there are pixels that do not contribute to classification ⇒ Pixels on the border of the image, for example ◮ And there are pixels that are more important for classification ⇒ Pixels that are white in one image but black in the other Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 3/23

Computational Cost 5 5 10 10 15 15 20 20 25 25 5 10 15 20 25 5 10 15 20 25 ◮ Note that in performing PCA we need the complete image ◮ However, there are pixels that do not contribute to classification ⇒ Pixels on the border of the image, for example ◮ And there are pixels that are more important for classification ⇒ Pixels that are white in one image but black in the other Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 3/23

Computational Cost 5 5 10 10 15 15 20 20 25 25 5 10 15 20 25 5 10 15 20 25 ◮ Note that in performing PCA we need the complete image ◮ However, there are pixels that do not contribute to classification ⇒ Pixels on the border of the image, for example ◮ And there are pixels that are more important for classification ⇒ Pixels that are white in one image but black in the other Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 3/23

Sampling 5 5 10 10 15 15 20 20 25 25 5 10 15 20 25 5 10 15 20 25 ◮ Few nonzero PCA coefficients ⇒ Bandlimited signal ⇒ Sampling ◮ Subspace representation on covariance graph (not all pixels are useful) ⇒ Linear combination of a few eigenvectors weighted by PCA coeff. ◮ Extend to arbitrary graphs ⇒ Sampling of bandlimited graph signals Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 4/23

Sampling ◮ Few nonzero PCA coefficients ⇒ Bandlimited signal ⇒ Sampling ◮ Subspace representation on covariance graph (not all pixels are useful) ⇒ Linear combination of a few eigenvectors weighted by PCA coeff. ◮ Extend to arbitrary graphs ⇒ Sampling of bandlimited graph signals Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 4/23

Sampling ◮ Few nonzero PCA coefficients ⇒ Bandlimited signal ⇒ Sampling ◮ Subspace representation on covariance graph (not all pixels are useful) ⇒ Linear combination of a few eigenvectors weighted by PCA coeff. ◮ Extend to arbitrary graphs ⇒ Sampling of bandlimited graph signals ◮ Design a classifier to operate on the samples ⇒ Reduce dimensionality Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 4/23

Rethinking Sketching as Sampling ◮ Sketching ⇒ Reduce dimensionality of linear transformations ◮ Projection on a lower-dimensional subspace ⇒ Smaller size matrix ⇒ Matrix sketch retains the most outstanding characteristics ◮ Smaller matrix operates on smaller vector to compute the result ⇒ Project vector on a lower-dimensional subspace ⇒ Sampling Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 5/23

Rethinking Sketching as Sampling ◮ Sketching ⇒ Reduce dimensionality of linear transformations ◮ Projection on a lower-dimensional subspace ⇒ Smaller size matrix ⇒ Matrix sketch retains the most outstanding characteristics ◮ Smaller matrix operates on smaller vector to compute the result ⇒ Project vector on a lower-dimensional subspace ⇒ Sampling ◮ Jointly design sampling of signal and sketching of linear transform ⇒ Obtain approximate solution by operating only on few samples w m × n n n × m y x y x w + H + H n m m n n ˆ y ˆ y C H s H s C m m p × n m × p m × p p × n Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 5/23

Sampling of Graph Signals ◮ Graph signals defined on top of a graph G = ( V , E , W ) with n nodes ◮ Irregular support captured by normal graph shift operator S = VΛV H ◮ Define the graph Fourier transform (GFT) ˜ x = V H x ⇒ Linear combination weighted by GFT coefficients x = V ˜ x (iGFT) Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 6/23

Sampling of Graph Signals ◮ Graph signals defined on top of a graph G = ( V , E , W ) with n nodes ◮ Irregular support captured by normal graph shift operator S = VΛV H ◮ Define the graph Fourier transform (GFT) ˜ x = V H x ⇒ Linear combination weighted by GFT coefficients x = V ˜ x (iGFT) ◮ Bandlimited graph signal ⇒ ˜ x = [˜ x k ; 0 n − k ] with k ≪ n ⇒ x = V k ˜ x k ⇒ Active eigenbasis of vectors V k = [ V k , 0 n × ( n − k ) ] ◮ Signal as a linear combination of few elements in V k ⇒ Sampling Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 6/23

Sketching ◮ Estimate the input to a linear transform by measuring the output ⇒ The model is x = Hy , with H ∈ R n × m and where n ≫ m ⇒ LS solution ⇒ Computationally costly (pseudo-)inverse Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 7/23

Sketching ◮ Estimate the input to a linear transform by measuring the output ⇒ The model is x = Hy , with H ∈ R n × m and where n ≫ m ⇒ LS solution ⇒ Computationally costly (pseudo-)inverse ◮ Traditional sketching ⇒ Reduce dimension of the linear problem ◮ Compress H and x ⇒ KH and Kx , K ∈ R p × n random, p ≪ n ⇒ Random projection on a lower-dimensional subspace ⇒ Solution of smaller problem min y � ( KH ) y − ( Kx ) � 2 2 ⇒ Faster ◮ Design K such that KH and Kx retains important traits of the problem ⇒ Then, solving for ( KH , Kx ) yields a good approximation ◮ We consider a deterministic design to obtain a smaller matrix sketch Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 7/23

Operating Conditions ◮ Sequence of signals to be processed by the same linear transform ⇒ Matrix H is big ⇒ Computationally intensive to operate with ◮ Realizations of a bandlimited graph random process ⇒ R x singular ◮ Enough computational power available prior to processing of signals ◮ Process sequence of signals fast ⇒ Apply smaller matrix to samples ◮ Traditional sampling ⇒ Ignores further processing on the signal ◮ Traditional sketching ⇒ Recomputes sketch for each realization x C H s p × n m × p ˆ y 3 y 2 ˆ y 1 ˆ x 3 x 2 x 1 Cx 3 Cx 2 Cx 1 m × 1 n × 1 p × 1 Design C , H s based on H and statistics of signal R x and noise R w Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 8/23

Problem Statement ◮ Design a sampling matrix C that selects k ≤ p ≪ n samples ◮ Design a deterministic sketch H s to be directly applied to samples ◮ Joint design of sketching and sampling prior to start of sequence ⇒ Minimize the MSE relative to using full H on the full signal x ◮ Processing of signals reduces to sampling and matrix multiplication ◮ The computational cost of processing is reduced by a factor of p / n w n × m m × n n y x w y x + H + H m n n n m ˆ y ˆ y H s C C H s m m m × p p × n p × n m × p Gama, Marques, Mateos, Ribeiro Rethinking Sketching as Sampling 9/23