compressed sensing and generative models
play

Compressed Sensing and Generative Models Ashish Bora Ajil Jalal - PowerPoint PPT Presentation

Feb 05, 2023 •408 likes •2.31k views

Compressed Sensing and Generative Models Ashish Bora Ajil Jalal Eric Price Alex Dimakis UT Austin Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 1 / 33 Talk Outline Using generative

  1. Standard Compressed Sensing Formalism “Compressible” = “sparse” Want to estimate x from y = Ax + η , for A ∈ R m × n . ◮ For this talk: ignore η , so y = Ax . Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

  2. Standard Compressed Sensing Formalism “Compressible” = “sparse” Want to estimate x from y = Ax + η , for A ∈ R m × n . ◮ For this talk: ignore η , so y = Ax . Goal: � x with k -sparse x ′ � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (1) with high probability. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

  3. Standard Compressed Sensing Formalism “Compressible” = “sparse” Want to estimate x from y = Ax + η , for A ∈ R m × n . ◮ For this talk: ignore η , so y = Ax . Goal: � x with k -sparse x ′ � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (1) with high probability. ◮ Reconstruction accuracy proportional to model accuracy. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

  4. Standard Compressed Sensing Formalism “Compressible” = “sparse” Want to estimate x from y = Ax + η , for A ∈ R m × n . ◮ For this talk: ignore η , so y = Ax . Goal: � x with k -sparse x ′ � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (1) with high probability. ◮ Reconstruction accuracy proportional to model accuracy. Theorem [Cand` es-Romberg-Tao 2006] Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

  5. Standard Compressed Sensing Formalism “Compressible” = “sparse” Want to estimate x from y = Ax + η , for A ∈ R m × n . ◮ For this talk: ignore η , so y = Ax . Goal: � x with k -sparse x ′ � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (1) with high probability. ◮ Reconstruction accuracy proportional to model accuracy. Theorem [Cand` es-Romberg-Tao 2006] ◮ m = Θ( k log( n / k )) suffices for (1). Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

  6. Standard Compressed Sensing Formalism “Compressible” = “sparse” Want to estimate x from y = Ax + η , for A ∈ R m × n . ◮ For this talk: ignore η , so y = Ax . Goal: � x with k -sparse x ′ � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (1) with high probability. ◮ Reconstruction accuracy proportional to model accuracy. Theorem [Cand` es-Romberg-Tao 2006] ◮ m = Θ( k log( n / k )) suffices for (1). ◮ Such an � x can be found efficiently with, e.g., the LASSO. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

  7. Alternatives to sparsity? MRI images are sparse in the wavelet basis. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

  8. Alternatives to sparsity? MRI images are sparse in the wavelet basis. Worldwide, 100 million MRIs taken per year. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

  9. Alternatives to sparsity? MRI images are sparse in the wavelet basis. Worldwide, 100 million MRIs taken per year. Want a data-driven model. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

  10. Alternatives to sparsity? MRI images are sparse in the wavelet basis. Worldwide, 100 million MRIs taken per year. Want a data-driven model. ◮ Better structural understanding should give fewer measurements. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

  11. Alternatives to sparsity? MRI images are sparse in the wavelet basis. Worldwide, 100 million MRIs taken per year. Want a data-driven model. ◮ Better structural understanding should give fewer measurements. Best way to model images in 2019? Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

  12. Alternatives to sparsity? MRI images are sparse in the wavelet basis. Worldwide, 100 million MRIs taken per year. Want a data-driven model. ◮ Better structural understanding should give fewer measurements. Best way to model images in 2019? ◮ Deep convolutional neural networks. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

  13. Alternatives to sparsity? MRI images are sparse in the wavelet basis. Worldwide, 100 million MRIs taken per year. Want a data-driven model. ◮ Better structural understanding should give fewer measurements. Best way to model images in 2019? ◮ Deep convolutional neural networks. ◮ In particular: generative models . Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

  14. Generative Models Random n Image k noise z Karras et al., 2018 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

  15. Generative Models Random n Image k noise z Karras et al., 2018 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

  16. Generative Models Random n Image k noise z Karras et al., 2018 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

  17. Training Generative Models Random n Image k noise z Karras et al., 2018 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

  18. Training Generative Models Random n Image k noise z Karras et al., 2018 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

  19. Training Generative Models Random n Image k noise z Karras et al., 2018 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

  20. Training Generative Models Random n Image k noise z Karras et al., 2018 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

  21. Training Generative Models Random n Image k noise z Karras et al., 2018 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

  22. Training Generative Models Random n Image k noise z Karras et al., 2018 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

  23. Training Generative Models Random n Image k noise z Karras et al., 2018 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

  24. Generative Models Want to model a distribution D of images. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

  25. Generative Models Want to model a distribution D of images. Function G : R k → R n . Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

  26. Generative Models Want to model a distribution D of images. Function G : R k → R n . When z ∼ N (0 , I k ), then ideally G ( z ) ∼ D . Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

  27. Generative Models Want to model a distribution D of images. Function G : R k → R n . When z ∼ N (0 , I k ), then ideally G ( z ) ∼ D . Generative Adversarial Networks (GANs) [Goodfellow et al. 2014]: Astronomy Particle Physics Faces Karras et al., 2018 Schawinski et al., 2017 Paganini et al., 2017 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

  28. Generative Models Want to model a distribution D of images. Function G : R k → R n . When z ∼ N (0 , I k ), then ideally G ( z ) ∼ D . Generative Adversarial Networks (GANs) [Goodfellow et al. 2014]: Astronomy Particle Physics Faces Karras et al., 2018 Schawinski et al., 2017 Paganini et al., 2017 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

  29. Generative Models Want to model a distribution D of images. Function G : R k → R n . When z ∼ N (0 , I k ), then ideally G ( z ) ∼ D . Generative Adversarial Networks (GANs) [Goodfellow et al. 2014]: Astronomy Particle Physics Faces Karras et al., 2018 Schawinski et al., 2017 Paganini et al., 2017 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

  30. Generative Models Want to model a distribution D of images. Function G : R k → R n . When z ∼ N (0 , I k ), then ideally G ( z ) ∼ D . Generative Adversarial Networks (GANs) [Goodfellow et al. 2014]: Astronomy Particle Physics Faces Karras et al., 2018 Schawinski et al., 2017 Paganini et al., 2017 Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

  31. Generative Models Want to model a distribution D of images. Function G : R k → R n . When z ∼ N (0 , I k ), then ideally G ( z ) ∼ D . Generative Adversarial Networks (GANs) [Goodfellow et al. 2014]: Astronomy Particle Physics Faces Karras et al., 2018 Schawinski et al., 2017 Paganini et al., 2017 Variational Auto-Encoders (VAEs) [Kingma & Welling 2013]. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

  32. Generative Models Want to model a distribution D of images. Function G : R k → R n . When z ∼ N (0 , I k ), then ideally G ( z ) ∼ D . Generative Adversarial Networks (GANs) [Goodfellow et al. 2014]: Astronomy Particle Physics Faces Suggestion for compressed sensing Replace “ x is k -sparse” by “ x is in range of G : R k → R n ”. Karras et al., 2018 Schawinski et al., 2017 Paganini et al., 2017 Variational Auto-Encoders (VAEs) [Kingma & Welling 2013]. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

  33. Our Results “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

  34. Our Results “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with k -sparse x ′ � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (2) Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

  35. Our Results “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (2) Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

  36. Our Results “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (2) Main Theorem I: m = O ( kd log n ) suffices for (2). Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

  37. Our Results “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (2) Main Theorem I: m = O ( kd log n ) suffices for (2). ◮ G is a d -layer ReLU-based neural network. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

  38. Our Results “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (2) Main Theorem I: m = O ( kd log n ) suffices for (2). ◮ G is a d -layer ReLU-based neural network. ◮ When A is random Gaussian matrix. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

  39. Our Results “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (2) Main Theorem I: m = O ( kd log n ) suffices for (2). ◮ G is a d -layer ReLU-based neural network. ◮ When A is random Gaussian matrix. Main Theorem II: Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

  40. Our Results “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (2) Main Theorem I: m = O ( kd log n ) suffices for (2). ◮ G is a d -layer ReLU-based neural network. ◮ When A is random Gaussian matrix. Main Theorem II: ◮ For any Lipschitz G , m = O ( k log L ) suffices. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

  41. Our Results “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ = G ( z ′ ) , � z ′ � 2 ≤ r � x − x ′ � 2 + δ � x − � x � 2 ≤ O (1) · min (2) Main Theorem I: m = O ( kd log n ) suffices for (2). ◮ G is a d -layer ReLU-based neural network. ◮ When A is random Gaussian matrix. Main Theorem II: ◮ For any Lipschitz G , m = O ( k log rL δ ) suffices. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

  42. Our Results “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ = G ( z ′ ) , � z ′ � 2 ≤ r � x − x ′ � 2 + δ � x − � x � 2 ≤ O (1) · min (2) Main Theorem I: m = O ( kd log n ) suffices for (2). ◮ G is a d -layer ReLU-based neural network. ◮ When A is random Gaussian matrix. Main Theorem II: ◮ For any Lipschitz G , m = O ( k log rL δ ) suffices. ◮ Morally the same O ( kd log n ) bound. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

  43. Our Results (II) “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (3) m = O ( kd log n ) suffices for d -layer G . Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

  44. Our Results (II) “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (3) m = O ( kd log n ) suffices for d -layer G . ◮ Compared to O ( k log n ) for sparsity-based methods. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

  45. Our Results (II) “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (3) m = O ( kd log n ) suffices for d -layer G . ◮ Compared to O ( k log n ) for sparsity-based methods. ◮ k here can be much smaller Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

  46. Our Results (II) “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (3) m = O ( kd log n ) suffices for d -layer G . ◮ Compared to O ( k log n ) for sparsity-based methods. ◮ k here can be much smaller Find � x = G ( � z ) by gradient descent on � y − AG ( � z ) � 2 . Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

  47. Our Results (II) “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (3) m = O ( kd log n ) suffices for d -layer G . ◮ Compared to O ( k log n ) for sparsity-based methods. ◮ k here can be much smaller Find � x = G ( � z ) by gradient descent on � y − AG ( � z ) � 2 . ◮ Just like for training, no proof this converges Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

  48. Our Results (II) “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (3) m = O ( kd log n ) suffices for d -layer G . ◮ Compared to O ( k log n ) for sparsity-based methods. ◮ k here can be much smaller Find � x = G ( � z ) by gradient descent on � y − AG ( � z ) � 2 . ◮ Just like for training, no proof this converges ◮ Approximate solution approximately gives (3) Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

  49. Our Results (II) “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (3) m = O ( kd log n ) suffices for d -layer G . ◮ Compared to O ( k log n ) for sparsity-based methods. ◮ k here can be much smaller Find � x = G ( � z ) by gradient descent on � y − AG ( � z ) � 2 . ◮ Just like for training, no proof this converges ◮ Approximate solution approximately gives (3) ◮ Can check that � � x − x � 2 is small. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

  50. Our Results (II) “Compressible” = “near range( G )” Want to estimate x from y = Ax , for A ∈ R m × n . Goal: � x with x ′ ∈ range( G ) � x − x ′ � 2 � x − � x � 2 ≤ O (1) · min (3) m = O ( kd log n ) suffices for d -layer G . ◮ Compared to O ( k log n ) for sparsity-based methods. ◮ k here can be much smaller Find � x = G ( � z ) by gradient descent on � y − AG ( � z ) � 2 . ◮ Just like for training, no proof this converges ◮ Approximate solution approximately gives (3) ◮ Can check that � � x − x � 2 is small. ◮ In practice, optimization error is negligible. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

  51. Related Work Model-based compressed sensing (Baraniuk-Cevher-Duarte-Hegde ’10) Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 14 / 33

  52. Related Work Model-based compressed sensing (Baraniuk-Cevher-Duarte-Hegde ’10) ◮ k -sparse + more = ⇒ O ( k ) measurements. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 14 / 33

  53. Related Work Model-based compressed sensing (Baraniuk-Cevher-Duarte-Hegde ’10) ◮ k -sparse + more = ⇒ O ( k ) measurements. Projections on manifolds (Baraniuk-Wakin ’09, Eftekhari-Wakin ’15) Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 14 / 33

  54. Related Work Model-based compressed sensing (Baraniuk-Cevher-Duarte-Hegde ’10) ◮ k -sparse + more = ⇒ O ( k ) measurements. Projections on manifolds (Baraniuk-Wakin ’09, Eftekhari-Wakin ’15) ◮ Conditions on manifold for which recovery is possible. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 14 / 33

  55. Related Work Model-based compressed sensing (Baraniuk-Cevher-Duarte-Hegde ’10) ◮ k -sparse + more = ⇒ O ( k ) measurements. Projections on manifolds (Baraniuk-Wakin ’09, Eftekhari-Wakin ’15) ◮ Conditions on manifold for which recovery is possible. Deep network models (Mousavi-Dasarathy-Baraniuk ’17, Chang et al ’17) Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 14 / 33

  56. Related Work Model-based compressed sensing (Baraniuk-Cevher-Duarte-Hegde ’10) ◮ k -sparse + more = ⇒ O ( k ) measurements. Projections on manifolds (Baraniuk-Wakin ’09, Eftekhari-Wakin ’15) ◮ Conditions on manifold for which recovery is possible. Deep network models (Mousavi-Dasarathy-Baraniuk ’17, Chang et al ’17) ◮ Train deep network to encode and/or decode. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 14 / 33

  57. Experimental Results Faces: n = 64 × 64 × 3 = 12288, m = 500 Original Lasso (DCT) (Wavelet) Lasso DCGAN Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 15 / 33

  58. Experimental Results Faces: n = 64 × 64 × 3 = 12288, m = 500 Original Lasso (DCT) (Wavelet) Lasso DCGAN Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 15 / 33

  59. Experimental Results Faces: n = 64 × 64 × 3 = 12288, m = 500 Original Lasso (DCT) (Wavelet) Lasso DCGAN Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 15 / 33

  60. Experimental Results MNIST: n = 28 x 28 = 784, m = 100. Original Lasso VAE Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 15 / 33

  61. Experimental Results Faces MNIST 0.35 0.12 Lasso (DCT) Lasso Reconstruction error (per pixel) 0.30 Reconstruction error (per pixel) Lasso (Wavelet) VAE 0.10 DCGAN 0.25 0.08 0.20 0.06 0.15 0.04 0.10 0.02 0.05 0.00 0.00 0 500 1000 1500 2000 2500 0 100 200 300 400 500 Number of measurements Number of measurements Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 16 / 33

  62. Proof Outline (ReLU-based networks) Show range( G ) lies within union of n dk k -dimensional hyperplane. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

  63. Proof Outline (ReLU-based networks) Show range( G ) lies within union of n dk k -dimensional hyperplane. � n � ≤ 2 k log( n / k ) hyperplanes. ◮ Then analogous to proof for sparsity: k Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

  64. Proof Outline (ReLU-based networks) Show range( G ) lies within union of n dk k -dimensional hyperplane. � n � ≤ 2 k log( n / k ) hyperplanes. ◮ Then analogous to proof for sparsity: k ◮ So dk log n Gaussian measurements suffice. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

  65. Proof Outline (ReLU-based networks) Show range( G ) lies within union of n dk k -dimensional hyperplane. � n � ≤ 2 k log( n / k ) hyperplanes. ◮ Then analogous to proof for sparsity: k ◮ So dk log n Gaussian measurements suffice. ReLU-based network: Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

  66. Proof Outline (ReLU-based networks) Show range( G ) lies within union of n dk k -dimensional hyperplane. � n � ≤ 2 k log( n / k ) hyperplanes. ◮ Then analogous to proof for sparsity: k ◮ So dk log n Gaussian measurements suffice. ReLU-based network: ◮ Each layer is z → ReLU( A i z ). Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

  67. Proof Outline (ReLU-based networks) Show range( G ) lies within union of n dk k -dimensional hyperplane. � n � ≤ 2 k log( n / k ) hyperplanes. ◮ Then analogous to proof for sparsity: k ◮ So dk log n Gaussian measurements suffice. ReLU-based network: ◮ Each layer is z → ReLU( A i z ). Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

  68. Proof Outline (ReLU-based networks) Show range( G ) lies within union of n dk k -dimensional hyperplane. � n � ≤ 2 k log( n / k ) hyperplanes. ◮ Then analogous to proof for sparsity: k ◮ So dk log n Gaussian measurements suffice. ReLU-based network: ◮ Each layer is z → ReLU( A i z ). � y i y i ≥ 0 ◮ ReLU( y ) i = 0 otherwise Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

  69. Proof Outline (ReLU-based networks) Show range( G ) lies within union of n dk k -dimensional hyperplane. � n � ≤ 2 k log( n / k ) hyperplanes. ◮ Then analogous to proof for sparsity: k ◮ So dk log n Gaussian measurements suffice. ReLU-based network: ◮ Each layer is z → ReLU( A i z ). � y i y i ≥ 0 ◮ ReLU( y ) i = 0 otherwise Input to layer 1: single k -dimensional hyperplane. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

  70. Proof Outline (ReLU-based networks) Show range( G ) lies within union of n dk k -dimensional hyperplane. � n � ≤ 2 k log( n / k ) hyperplanes. ◮ Then analogous to proof for sparsity: k ◮ So dk log n Gaussian measurements suffice. ReLU-based network: ◮ Each layer is z → ReLU( A i z ). � y i y i ≥ 0 ◮ ReLU( y ) i = 0 otherwise Input to layer 1: single k -dimensional hyperplane. Lemma Layer 1’s output lies within a union of n k k -dimensional hyperplanes. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

  71. Proof Outline (ReLU-based networks) Show range( G ) lies within union of n dk k -dimensional hyperplane. � n � ≤ 2 k log( n / k ) hyperplanes. ◮ Then analogous to proof for sparsity: k ◮ So dk log n Gaussian measurements suffice. ReLU-based network: ◮ Each layer is z → ReLU( A i z ). � y i y i ≥ 0 ◮ ReLU( y ) i = 0 otherwise Input to layer 1: single k -dimensional hyperplane. Lemma Layer 1’s output lies within a union of n k k -dimensional hyperplanes. Induction: final output lies within n dk k -dimensional hyperplanes. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

  72. Proof of Lemma Layer 1’s output lies within a union of n k k -dimensional hyperplanes. z is k -dimensional. Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 18 / 33

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Deep Compressed Sensing Yan Wu, Mihaela Rosca, Tim Lillicrap Compressed Sensing A Brief Review

Deep Compressed Sensing Yan Wu, Mihaela Rosca, Tim Lillicrap Compressed Sensing A Brief Review

Deep Compressed Sensing Yan Wu, Mihaela Rosca, Tim Lillicrap Compressed Sensing A Brief Review An underdetermined problem: random projection signal, e.g, vectorised image measurements M x y Reconstruction of x is possible when the signal

345 views • 6 slides
Compressed Sensing and Bayesian Experimental Design or Optimal Sensing and Reconstruction of N -

Compressed Sensing and Bayesian Experimental Design or Optimal Sensing and Reconstruction of N -

Compressed Sensing and Bayesian Experimental Design or Optimal Sensing and Reconstruction of N - Dimensional Signals by Matthias Seeger & Hannes Nickisch Presenter: Pete Trautman Outline Intro to compressive sensing Paper

206 views • 17 slides
Sparsity-optimized Harmonic Wavelets for Compressed Sensing MRI Ruediger Willenberg (ECE)

Sparsity-optimized Harmonic Wavelets for Compressed Sensing MRI Ruediger Willenberg (ECE)

Sparsity-optimized Harmonic Wavelets for Compressed Sensing MRI Ruediger Willenberg (ECE) JEB1433 Project Presentation April 22, 2010 References n M.Lustig, D.Donoho, J.M.Pauly: Sparse MRI: The Application of Compressed Sensing for Rapid

464 views • 42 slides
Structured random measurements in compressed sensing Holger Rauhut Lehrstuhl C f ur

Structured random measurements in compressed sensing Holger Rauhut Lehrstuhl C f ur

Structured random measurements in compressed sensing Holger Rauhut Lehrstuhl C f ur Mathematik (Analysis) RWTH Aachen Winter School on Compressed Sensing TU Berlin December 35, 2014 1 / 82 Too Few Data Often it is hard, expensive,

1.01k views • 82 slides
Signals gnals & S & Sys ystems ems Introduction to Compressed Sensing Adapted from:

Signals gnals & S & Sys ystems ems Introduction to Compressed Sensing Adapted from:

Lecture 17 Signals gnals & S & Sys ystems ems Introduction to Compressed Sensing Adapted from: M. Davenport, M. F. Duarte, Y. C. Eldar, G. Kutyniok , Introduction to Compressed Sensing, 2011 J. Romberg, Imaging via

371 views • 25 slides
Estimation of Mixture Subspace Models Its Algebra, Statistics, and Compressed Sensing Allen

Estimation of Mixture Subspace Models Its Algebra, Statistics, and Compressed Sensing Allen

Introduction GPCA Lossy Compression Classification Conclusion Estimation of Mixture Subspace Models Its Algebra, Statistics, and Compressed Sensing Allen Y. Yang <yang@eecs.berkeley.edu> Nov. 30, 2007. Berkeley DSP Seminar Allen

1.13k views • 76 slides
Compressed sensing in the real world - The need for a new theory Anders C. Hansen (Cambridge)

Compressed sensing in the real world - The need for a new theory Anders C. Hansen (Cambridge)

Compressed sensing in the real world - The need for a new theory Anders C. Hansen (Cambridge) Joint work with: B. Adcock (Purdue) C. Poon (Cambridge) B. Roman (Cambridge) Paris, January 13, 2014 1 / 49 Compressed Sensing in Inverse Problems

629 views • 49 slides
Compressed Sensing: Challenges and Emerging Topics Mike Davies Edinburgh Compressed Sensing

Compressed Sensing: Challenges and Emerging Topics Mike Davies Edinburgh Compressed Sensing

IDCOM, University of Edinburgh Compressed Sensing: Challenges and Emerging Topics Mike Davies Edinburgh Compressed Sensing research group (E-CoS) Institute for Digital Communications University of Edinburgh IDCOM, University of Edinburgh

342 views • 30 slides
One-bit compressed sensing with Gaussian circulant matrices Sjoerd Dirksen (RWTH Aachen

One-bit compressed sensing with Gaussian circulant matrices Sjoerd Dirksen (RWTH Aachen

One-bit compressed sensing with Gaussian circulant matrices Sjoerd Dirksen (RWTH Aachen University) based on joint work with Hans Christian Jung and Holger Rauhut (RWTH Aachen) Supported by DFG project QuaCoSS Compressed Sensing and its

410 views • 25 slides
Compressed Sensing Meets Machine Learning - Classification of Mixture Subspace Models via Sparse

Compressed Sensing Meets Machine Learning - Classification of Mixture Subspace Models via Sparse

Introduction Classification via Sparse Representation Distributed Pattern Recognition Conclusion Compressed Sensing Meets Machine Learning - Classification of Mixture Subspace Models via Sparse Representation Allen Y. Yang

861 views • 31 slides
Decoding in Compressed Sensing Ronald DeVore USC, 2008 p. 1/33 Discrete Compressed Sensing R

Decoding in Compressed Sensing Ronald DeVore USC, 2008 p. 1/33 Discrete Compressed Sensing R

Decoding in Compressed Sensing Ronald DeVore USC, 2008 p. 1/33 Discrete Compressed Sensing R N with N large x I USC, 2008 p. 2/33 Discrete Compressed Sensing R N with N large x I We are able to ask n questions about x USC, 2008

1.63k views • 145 slides
Compressed Sensing under Optimal Quantization Alon Kipnis (Stanford) Galen Reeves (Duke) Yonina

Compressed Sensing under Optimal Quantization Alon Kipnis (Stanford) Galen Reeves (Duke) Yonina

Compressed Sensing under Optimal Quantization Alon Kipnis (Stanford) Galen Reeves (Duke) Yonina Eldar (Technion) Andrea Goldsmith (Stanford) ISIT, June 2017 Table of Contents Introduction Remote Source Coding Compressed Sensing Results

761 views • 40 slides
Fast Data Driven Compressed Sensing and application to compressed quantitative MRI Mike Davies

Fast Data Driven Compressed Sensing and application to compressed quantitative MRI Mike Davies

IDCOM, University of Edinburgh Fast Data Driven Compressed Sensing and application to compressed quantitative MRI Mike Davies & Mohammad Golbabaee Joint work with Zhouye Chen, Yves Wiaux Zaid Mahbub, Ian Marshall IDCOM, University of

426 views • 29 slides
Foundations of Compressed Sensing Mike Davies Edinburgh Compressed Sensing research group (E-CoS)

Foundations of Compressed Sensing Mike Davies Edinburgh Compressed Sensing research group (E-CoS)

IDCOM, University of Edinburgh Foundations of Compressed Sensing Mike Davies Edinburgh Compressed Sensing research group (E-CoS) Institute for Digital Communications University of Edinburgh IDCOM, University of Edinburgh Part I: Foundations of

525 views • 33 slides
Introduction to Compressed Sensing Gitta Kutyniok (Institut f ur Mathematik, Technische

Introduction to Compressed Sensing Gitta Kutyniok (Institut f ur Mathematik, Technische

Introduction to Compressed Sensing Gitta Kutyniok (Institut f ur Mathematik, Technische Universit at Berlin) Winter School on Compressed Sensing, TU Berlin December 35, 2015 Gitta Kutyniok (TU Berlin) Introduction to Compressed

858 views • 54 slides
Compressed Sensing. Find x with small number of non-zeros using linear measurements. Compressed

Compressed Sensing. Find x with small number of non-zeros using linear measurements. Compressed

Fun with 1 and 2 x 1 n x 2 . Fun with 1 and 2 x 1 n x 2 . x 1 = x sgn ( x ) x 2 sgn ( x ) 2 x 2 Fun with 1 and 2 x 1

1.25k views • 123 slides
Intelligent Sem antic Web Search Service The Intute Project Speaker: Yanbo J. Wang NaCTeM,

Intelligent Sem antic Web Search Service The Intute Project Speaker: Yanbo J. Wang NaCTeM,

Intelligent Sem antic Web Search Service The Intute Project Speaker: Yanbo J. Wang NaCTeM, School of Computer Science University of Manchester Project Description The Intute project, co-funded by JISC (Joint Information Systems Committee)

322 views • 15 slides
Pillars of KDE: Flake Introduction Flake: Compound Documents Compose your documents from

Pillars of KDE: Flake Introduction Flake: Compound Documents Compose your documents from

Sebastian Kgler <sebas@kde.org>, FrOSCon 2006 Pillars of KDE: Flake Introduction Flake: Compound Documents Compose your documents from individual objects: text frames, images, vector graphics, tables, connectors Suitable for:

507 views • 16 slides
MongoDB Indexing Dr Janusz R. Getta School of Computing and Information Technology - University

MongoDB Indexing Dr Janusz R. Getta School of Computing and Information Technology - University

MongoDB Indexing file:///Users/jrg/235-2020-SPRING/SLIDES/WEEK11/26indexing/26indexing.html#1 CSCI235 Database Systems MongoDB Indexing Dr Janusz R. Getta School of Computing and Information Technology - University of Wollongong 1 of 25

1k views • 25 slides
Gaussian tutorial -Infrared spectra calculation In this tutorial Gaussian 03 program was used

Gaussian tutorial -Infrared spectra calculation In this tutorial Gaussian 03 program was used

Gaussian tutorial -Infrared spectra calculation In this tutorial Gaussian 03 program was used to perform the calculation. GaussView 5.0 was used to prepare the input file for Gaussian. Exampled compound: Aspirin Drawing the compound

456 views • 19 slides
Prompt processing and Data Quality Monitoring in the protoDUNE-SP experiment M.Potekhin NPPS

Prompt processing and Data Quality Monitoring in the protoDUNE-SP experiment M.Potekhin NPPS

Prompt processing and Data Quality Monitoring in the protoDUNE-SP experiment M.Potekhin NPPS Meeting May 24 th 2019 Overview Please look at the Backup Slides at your leisure, there is interesting material there Lots of

667 views • 31 slides
CS681: Advanced Topics in Computational Biology Can Alkan EA509 calkan@cs.bilkent.edu.tr

CS681: Advanced Topics in Computational Biology Can Alkan EA509 calkan@cs.bilkent.edu.tr

CS681: Advanced Topics in Computational Biology Can Alkan EA509 calkan@cs.bilkent.edu.tr http://www.cs.bilkent.edu.tr/~calkan/teaching/cs681/ Compression 1 Reference based Coding/decoding rather than real compression Very high

691 views • 34 slides
Continuous models of computation: computability, complexity, universality Amaury Pouly 21 mars

Continuous models of computation: computability, complexity, universality Amaury Pouly 21 mars

Continuous models of computation: computability, complexity, universality Amaury Pouly 21 mars 2017 1 / 19 Digital vs analog computers 2 / 19 Digital vs analog computers VS 2 / 19 Church Thesis Computability logic boolean circuits

907 views • 53 slides
The HELLO assignment Write a JAVA program to output the words The Legacy of Alan Turing Hello

The HELLO assignment Write a JAVA program to output the words The Legacy of Alan Turing Hello

The HELLO assignment Write a JAVA program to output the words The Legacy of Alan Turing Hello World! on the screen and halt. Space and time are not an issue. Based on slides by team of CMU s 15-251 Sanjeev Arora and Bernard Chazelle

202 views • 7 slides

More recommend