Compressed Sensing and Generative Models Ashish Bora Ajil Jalal - - PowerPoint PPT Presentation

▶
compressed sensing and generative models
SMART_READER_LITE
LIVE PREVIEW

Compressed Sensing and Generative Models Ashish Bora Ajil Jalal - - PowerPoint PPT Presentation

Feb 05, 2023 408 likes •2.31k views

Compressed Sensing and Generative Models Ashish Bora Ajil Jalal Eric Price Alex Dimakis UT Austin Ashish Bora, Ajil Jalal, Eric Price , Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 1 / 33 Talk Outline Using generative

slide-1
SLIDE 1

Compressed Sensing and Generative Models

Ashish Bora Ajil Jalal Eric Price Alex Dimakis

UT Austin

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 1 / 33

slide-2
SLIDE 2

Talk Outline

1

Using generative models for compressed sensing

2

Learning generative models from noisy data

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 2 / 33

slide-3
SLIDE 3

Talk Outline

1

Using generative models for compressed sensing

2

Learning generative models from noisy data

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 3 / 33

slide-4
SLIDE 4

Compressed Sensing

Want to recover a signal (e.g., an image) from noisy measurements.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 4 / 33

slide-5
SLIDE 5

Compressed Sensing

Want to recover a signal (e.g., an image) from noisy measurements. Medical Imaging Astronomy Single-Pixel Camera Oil Exploration

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 4 / 33

slide-6
SLIDE 6

Compressed Sensing

Want to recover a signal (e.g., an image) from noisy measurements. Medical Imaging Astronomy Single-Pixel Camera Oil Exploration Linear measurements: see y = Ax, for A ∈ Rm×n.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 4 / 33

slide-7
SLIDE 7

Compressed Sensing

Want to recover a signal (e.g., an image) from noisy measurements. Medical Imaging Astronomy Single-Pixel Camera Oil Exploration Linear measurements: see y = Ax, for A ∈ Rm×n. How many measurements m to learn the signal?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 4 / 33

slide-8
SLIDE 8

Compressed Sensing

Given linear measurements y = Ax, for A ∈ Rm×n. How many measurements m to learn the signal x?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 5 / 33

slide-9
SLIDE 9

Compressed Sensing

Given linear measurements y = Ax, for A ∈ Rm×n. How many measurements m to learn the signal x?

◮ Naively: m ≥ n or else underdetermined Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 5 / 33

slide-10
SLIDE 10

Compressed Sensing

Given linear measurements y = Ax, for A ∈ Rm×n. How many measurements m to learn the signal x?

◮ Naively: m ≥ n or else underdetermined: multiple x possible. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 5 / 33

slide-11
SLIDE 11

Compressed Sensing

Given linear measurements y = Ax, for A ∈ Rm×n. How many measurements m to learn the signal x?

◮ Naively: m ≥ n or else underdetermined: multiple x possible. ◮ But most x aren’t plausible.

5MB 36MB

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 5 / 33

slide-12
SLIDE 12

Compressed Sensing

Given linear measurements y = Ax, for A ∈ Rm×n. How many measurements m to learn the signal x?

◮ Naively: m ≥ n or else underdetermined: multiple x possible. ◮ But most x aren’t plausible.

5MB 36MB

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 5 / 33

slide-13
SLIDE 13

Compressed Sensing

Given linear measurements y = Ax, for A ∈ Rm×n. How many measurements m to learn the signal x?

◮ Naively: m ≥ n or else underdetermined: multiple x possible. ◮ But most x aren’t plausible.

5MB 36MB

◮ This is why compression is possible. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 5 / 33

slide-14
SLIDE 14

Compressed Sensing

Given linear measurements y = Ax, for A ∈ Rm×n. How many measurements m to learn the signal x?

◮ Naively: m ≥ n or else underdetermined: multiple x possible. ◮ But most x aren’t plausible.

5MB 36MB

◮ This is why compression is possible.

Ideal answer: m > (information in image) (new info. per measurement)

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 5 / 33

slide-15
SLIDE 15

Compressed Sensing

Given linear measurements y = Ax, for A ∈ Rm×n. How many measurements m to learn the signal x? m > (information in image) (new info. per measurement)

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 6 / 33

slide-16
SLIDE 16

Compressed Sensing

Given linear measurements y = Ax, for A ∈ Rm×n. How many measurements m to learn the signal x? m > (information in image) (new info. per measurement) Image “compressible” = ⇒ information in image is small.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 6 / 33

slide-17
SLIDE 17

Compressed Sensing

Given linear measurements y = Ax, for A ∈ Rm×n. How many measurements m to learn the signal x? m > (information in image) (new info. per measurement) Image “compressible” = ⇒ information in image is small. Measurements “incoherent” = ⇒ most info new.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 6 / 33

slide-18
SLIDE 18

Compressed Sensing

Want to estimate x ∈ Rn from m ≪ n linear measurements.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 7 / 33

slide-19
SLIDE 19

Compressed Sensing

Want to estimate x ∈ Rn from m ≪ n linear measurements. Suggestion: the “most compressible” image that fits measurements.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 7 / 33

slide-20
SLIDE 20

Compressed Sensing

Want to estimate x ∈ Rn from m ≪ n linear measurements. Suggestion: the “most compressible” image that fits measurements. How should we formalize that an image is “compressible”?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 7 / 33

slide-21
SLIDE 21

Compressed Sensing

Want to estimate x ∈ Rn from m ≪ n linear measurements. Suggestion: the “most compressible” image that fits measurements. How should we formalize that an image is “compressible”? Short JPEG compression

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 7 / 33

slide-22
SLIDE 22

Compressed Sensing

Want to estimate x ∈ Rn from m ≪ n linear measurements. Suggestion: the “most compressible” image that fits measurements. How should we formalize that an image is “compressible”? Short JPEG compression

◮ Intractible to compute. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 7 / 33

slide-23
SLIDE 23

Compressed Sensing

Want to estimate x ∈ Rn from m ≪ n linear measurements. Suggestion: the “most compressible” image that fits measurements. How should we formalize that an image is “compressible”? Short JPEG compression

◮ Intractible to compute.

Standard compressed sensing: sparsity in some basis

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 7 / 33

slide-24
SLIDE 24

Compressed Sensing

Want to estimate x ∈ Rn from m ≪ n linear measurements. Suggestion: the “most compressible” image that fits measurements. How should we formalize that an image is “compressible”? Short JPEG compression

◮ Intractible to compute.

Standard compressed sensing: sparsity in some basis

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 7 / 33

slide-25
SLIDE 25

Compressed Sensing

Want to estimate x ∈ Rn from m ≪ n linear measurements. Suggestion: the “most compressible” image that fits measurements. How should we formalize that an image is “compressible”? Short JPEG compression

◮ Intractible to compute.

Standard compressed sensing: sparsity in some basis

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 7 / 33

slide-26
SLIDE 26

Compressed Sensing

Want to estimate x ∈ Rn from m ≪ n linear measurements. Suggestion: the “most compressible” image that fits measurements. How should we formalize that an image is “compressible”? Short JPEG compression

◮ Intractible to compute.

Standard compressed sensing: sparsity in some basis

◮ Sparsity + other constraints (“structured sparsity”) Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 7 / 33

slide-27
SLIDE 27

Compressed Sensing

Want to estimate x ∈ Rn from m ≪ n linear measurements. Suggestion: the “most compressible” image that fits measurements. How should we formalize that an image is “compressible”? Short JPEG compression

◮ Intractible to compute.

Standard compressed sensing: sparsity in some basis

◮ Sparsity + other constraints (“structured sparsity”)

This talk: different approach, no sparsity.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 7 / 33

slide-28
SLIDE 28

Standard Compressed Sensing Formalism

“Compressible” = “sparse”

Want to estimate x from y = Ax + η, for A ∈ Rm×n.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

slide-29
SLIDE 29

Standard Compressed Sensing Formalism

“Compressible” = “sparse”

Want to estimate x from y = Ax + η, for A ∈ Rm×n.

◮ For this talk: ignore η, so y = Ax. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

slide-30
SLIDE 30

Standard Compressed Sensing Formalism

“Compressible” = “sparse”

Want to estimate x from y = Ax + η, for A ∈ Rm×n.

◮ For this talk: ignore η, so y = Ax.

Goal: x with x − x2 ≤ O(1) · min

k-sparse x′x − x′2

(1) with high probability.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

slide-31
SLIDE 31

Standard Compressed Sensing Formalism

“Compressible” = “sparse”

Want to estimate x from y = Ax + η, for A ∈ Rm×n.

◮ For this talk: ignore η, so y = Ax.

Goal: x with x − x2 ≤ O(1) · min

k-sparse x′x − x′2

(1) with high probability.

◮ Reconstruction accuracy proportional to model accuracy. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

slide-32
SLIDE 32

Standard Compressed Sensing Formalism

“Compressible” = “sparse”

Want to estimate x from y = Ax + η, for A ∈ Rm×n.

◮ For this talk: ignore η, so y = Ax.

Goal: x with x − x2 ≤ O(1) · min

k-sparse x′x − x′2

(1) with high probability.

◮ Reconstruction accuracy proportional to model accuracy.

Theorem [Cand` es-Romberg-Tao 2006]

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

slide-33
SLIDE 33

Standard Compressed Sensing Formalism

“Compressible” = “sparse”

Want to estimate x from y = Ax + η, for A ∈ Rm×n.

◮ For this talk: ignore η, so y = Ax.

Goal: x with x − x2 ≤ O(1) · min

k-sparse x′x − x′2

(1) with high probability.

◮ Reconstruction accuracy proportional to model accuracy.

Theorem [Cand` es-Romberg-Tao 2006]

◮ m = Θ(k log(n/k)) suffices for (1). Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

slide-34
SLIDE 34

Standard Compressed Sensing Formalism

“Compressible” = “sparse”

Want to estimate x from y = Ax + η, for A ∈ Rm×n.

◮ For this talk: ignore η, so y = Ax.

Goal: x with x − x2 ≤ O(1) · min

k-sparse x′x − x′2

(1) with high probability.

◮ Reconstruction accuracy proportional to model accuracy.

Theorem [Cand` es-Romberg-Tao 2006]

◮ m = Θ(k log(n/k)) suffices for (1). ◮ Such an

x can be found efficiently with, e.g., the LASSO.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 8 / 33

slide-35
SLIDE 35

Alternatives to sparsity?

MRI images are sparse in the wavelet basis.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

slide-36
SLIDE 36

Alternatives to sparsity?

MRI images are sparse in the wavelet basis. Worldwide, 100 million MRIs taken per year.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

slide-37
SLIDE 37

Alternatives to sparsity?

MRI images are sparse in the wavelet basis. Worldwide, 100 million MRIs taken per year. Want a data-driven model.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

slide-38
SLIDE 38

Alternatives to sparsity?

MRI images are sparse in the wavelet basis. Worldwide, 100 million MRIs taken per year. Want a data-driven model.

◮ Better structural understanding should give fewer measurements. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

slide-39
SLIDE 39

Alternatives to sparsity?

MRI images are sparse in the wavelet basis. Worldwide, 100 million MRIs taken per year. Want a data-driven model.

◮ Better structural understanding should give fewer measurements.

Best way to model images in 2019?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

slide-40
SLIDE 40

Alternatives to sparsity?

MRI images are sparse in the wavelet basis. Worldwide, 100 million MRIs taken per year. Want a data-driven model.

◮ Better structural understanding should give fewer measurements.

Best way to model images in 2019?

◮ Deep convolutional neural networks. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

slide-41
SLIDE 41

Alternatives to sparsity?

MRI images are sparse in the wavelet basis. Worldwide, 100 million MRIs taken per year. Want a data-driven model.

◮ Better structural understanding should give fewer measurements.

Best way to model images in 2019?

◮ Deep convolutional neural networks. ◮ In particular: generative models. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 9 / 33

slide-42
SLIDE 42

Generative Models

Random noise z Image Karras et al., 2018 n k

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

slide-43
SLIDE 43

Generative Models

Random noise z Image Karras et al., 2018 n k

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

slide-44
SLIDE 44

Generative Models

Random noise z Image Karras et al., 2018 n k

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

slide-45
SLIDE 45

Training Generative Models

Random noise z Image Karras et al., 2018 n k

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

slide-46
SLIDE 46

Training Generative Models

Random noise z Image Karras et al., 2018 n k

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

slide-47
SLIDE 47

Training Generative Models

Random noise z Image Karras et al., 2018 n k

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

slide-48
SLIDE 48

Training Generative Models

Random noise z Image Karras et al., 2018 n k

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

slide-49
SLIDE 49

Training Generative Models

Random noise z Image Karras et al., 2018 n k

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

slide-50
SLIDE 50

Training Generative Models

Random noise z Image Karras et al., 2018 n k

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

slide-51
SLIDE 51

Training Generative Models

Random noise z Image Karras et al., 2018 n k

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 10 / 33

slide-52
SLIDE 52

Generative Models

Want to model a distribution D of images.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

slide-53
SLIDE 53

Generative Models

Want to model a distribution D of images. Function G : Rk → Rn.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

slide-54
SLIDE 54

Generative Models

Want to model a distribution D of images. Function G : Rk → Rn. When z ∼ N(0, Ik), then ideally G(z) ∼ D.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

slide-55
SLIDE 55

Generative Models

Want to model a distribution D of images. Function G : Rk → Rn. When z ∼ N(0, Ik), then ideally G(z) ∼ D. Generative Adversarial Networks (GANs) [Goodfellow et al. 2014]: Karras et al., 2018 Faces Schawinski et al., 2017 Astronomy Paganini et al., 2017 Particle Physics

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

slide-56
SLIDE 56

Generative Models

Want to model a distribution D of images. Function G : Rk → Rn. When z ∼ N(0, Ik), then ideally G(z) ∼ D. Generative Adversarial Networks (GANs) [Goodfellow et al. 2014]: Karras et al., 2018 Faces Schawinski et al., 2017 Astronomy Paganini et al., 2017 Particle Physics

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

slide-57
SLIDE 57

Generative Models

Want to model a distribution D of images. Function G : Rk → Rn. When z ∼ N(0, Ik), then ideally G(z) ∼ D. Generative Adversarial Networks (GANs) [Goodfellow et al. 2014]: Karras et al., 2018 Faces Schawinski et al., 2017 Astronomy Paganini et al., 2017 Particle Physics

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

slide-58
SLIDE 58

Generative Models

Want to model a distribution D of images. Function G : Rk → Rn. When z ∼ N(0, Ik), then ideally G(z) ∼ D. Generative Adversarial Networks (GANs) [Goodfellow et al. 2014]: Karras et al., 2018 Faces Schawinski et al., 2017 Astronomy Paganini et al., 2017 Particle Physics

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

slide-59
SLIDE 59

Generative Models

Want to model a distribution D of images. Function G : Rk → Rn. When z ∼ N(0, Ik), then ideally G(z) ∼ D. Generative Adversarial Networks (GANs) [Goodfellow et al. 2014]: Karras et al., 2018 Faces Schawinski et al., 2017 Astronomy Paganini et al., 2017 Particle Physics Variational Auto-Encoders (VAEs) [Kingma & Welling 2013].

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

slide-60
SLIDE 60

Generative Models

Want to model a distribution D of images. Function G : Rk → Rn. When z ∼ N(0, Ik), then ideally G(z) ∼ D. Generative Adversarial Networks (GANs) [Goodfellow et al. 2014]: Karras et al., 2018 Faces Schawinski et al., 2017 Astronomy Paganini et al., 2017 Particle Physics Variational Auto-Encoders (VAEs) [Kingma & Welling 2013].

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 11 / 33

Suggestion for compressed sensing

Replace “x is k-sparse” by “x is in range of G : Rk → Rn”.

slide-61
SLIDE 61

Our Results

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

slide-62
SLIDE 62

Our Results

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

k-sparse x′x − x′2

(2)

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

slide-63
SLIDE 63

Our Results

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(2)

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

slide-64
SLIDE 64

Our Results

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(2) Main Theorem I: m = O(kd log n) suffices for (2).

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

slide-65
SLIDE 65

Our Results

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(2) Main Theorem I: m = O(kd log n) suffices for (2).

◮ G is a d-layer ReLU-based neural network. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

slide-66
SLIDE 66

Our Results

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(2) Main Theorem I: m = O(kd log n) suffices for (2).

◮ G is a d-layer ReLU-based neural network. ◮ When A is random Gaussian matrix. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

slide-67
SLIDE 67

Our Results

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(2) Main Theorem I: m = O(kd log n) suffices for (2).

◮ G is a d-layer ReLU-based neural network. ◮ When A is random Gaussian matrix.

Main Theorem II:

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

slide-68
SLIDE 68

Our Results

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(2) Main Theorem I: m = O(kd log n) suffices for (2).

◮ G is a d-layer ReLU-based neural network. ◮ When A is random Gaussian matrix.

Main Theorem II:

◮ For any Lipschitz G, m = O(k log L) suffices. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

slide-69
SLIDE 69

Our Results

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′=G(z′),z′2≤rx − x′2+δ

(2) Main Theorem I: m = O(kd log n) suffices for (2).

◮ G is a d-layer ReLU-based neural network. ◮ When A is random Gaussian matrix.

Main Theorem II:

◮ For any Lipschitz G, m = O(k log rL

δ ) suffices.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

slide-70
SLIDE 70

Our Results

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′=G(z′),z′2≤rx − x′2+δ

(2) Main Theorem I: m = O(kd log n) suffices for (2).

◮ G is a d-layer ReLU-based neural network. ◮ When A is random Gaussian matrix.

Main Theorem II:

◮ For any Lipschitz G, m = O(k log rL

δ ) suffices.

◮ Morally the same O(kd log n) bound. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 12 / 33

slide-71
SLIDE 71

Our Results (II)

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(3) m = O(kd log n) suffices for d-layer G.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

slide-72
SLIDE 72

Our Results (II)

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(3) m = O(kd log n) suffices for d-layer G.

◮ Compared to O(k log n) for sparsity-based methods. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

slide-73
SLIDE 73

Our Results (II)

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(3) m = O(kd log n) suffices for d-layer G.

◮ Compared to O(k log n) for sparsity-based methods. ◮ k here can be much smaller Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

slide-74
SLIDE 74

Our Results (II)

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(3) m = O(kd log n) suffices for d-layer G.

◮ Compared to O(k log n) for sparsity-based methods. ◮ k here can be much smaller

Find x = G( z) by gradient descent on y − AG( z)2.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

slide-75
SLIDE 75

Our Results (II)

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(3) m = O(kd log n) suffices for d-layer G.

◮ Compared to O(k log n) for sparsity-based methods. ◮ k here can be much smaller

Find x = G( z) by gradient descent on y − AG( z)2.

◮ Just like for training, no proof this converges Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

slide-76
SLIDE 76

Our Results (II)

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(3) m = O(kd log n) suffices for d-layer G.

◮ Compared to O(k log n) for sparsity-based methods. ◮ k here can be much smaller

Find x = G( z) by gradient descent on y − AG( z)2.

◮ Just like for training, no proof this converges ◮ Approximate solution approximately gives (3) Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

slide-77
SLIDE 77

Our Results (II)

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(3) m = O(kd log n) suffices for d-layer G.

◮ Compared to O(k log n) for sparsity-based methods. ◮ k here can be much smaller

Find x = G( z) by gradient descent on y − AG( z)2.

◮ Just like for training, no proof this converges ◮ Approximate solution approximately gives (3) ◮ Can check that

x − x2 is small.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

slide-78
SLIDE 78

Our Results (II)

“Compressible” = “near range(G)”

Want to estimate x from y = Ax, for A ∈ Rm×n. Goal: x with x − x2 ≤ O(1) · min

x′∈range(G)x − x′2

(3) m = O(kd log n) suffices for d-layer G.

◮ Compared to O(k log n) for sparsity-based methods. ◮ k here can be much smaller

Find x = G( z) by gradient descent on y − AG( z)2.

◮ Just like for training, no proof this converges ◮ Approximate solution approximately gives (3) ◮ Can check that

x − x2 is small.

◮ In practice, optimization error is negligible. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 13 / 33

slide-79
SLIDE 79

Related Work

Model-based compressed sensing (Baraniuk-Cevher-Duarte-Hegde ’10)

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 14 / 33

slide-80
SLIDE 80

Related Work

Model-based compressed sensing (Baraniuk-Cevher-Duarte-Hegde ’10)

◮ k-sparse + more =

⇒ O(k) measurements.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 14 / 33

slide-81
SLIDE 81

Related Work

Model-based compressed sensing (Baraniuk-Cevher-Duarte-Hegde ’10)

◮ k-sparse + more =

⇒ O(k) measurements.

Projections on manifolds (Baraniuk-Wakin ’09, Eftekhari-Wakin ’15)

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 14 / 33

slide-82
SLIDE 82

Related Work

Model-based compressed sensing (Baraniuk-Cevher-Duarte-Hegde ’10)

◮ k-sparse + more =

⇒ O(k) measurements.

Projections on manifolds (Baraniuk-Wakin ’09, Eftekhari-Wakin ’15)

◮ Conditions on manifold for which recovery is possible. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 14 / 33

slide-83
SLIDE 83

Related Work

Model-based compressed sensing (Baraniuk-Cevher-Duarte-Hegde ’10)

◮ k-sparse + more =

⇒ O(k) measurements.

Projections on manifolds (Baraniuk-Wakin ’09, Eftekhari-Wakin ’15)

◮ Conditions on manifold for which recovery is possible.

Deep network models (Mousavi-Dasarathy-Baraniuk ’17, Chang et al ’17)

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 14 / 33

slide-84
SLIDE 84

Related Work

Model-based compressed sensing (Baraniuk-Cevher-Duarte-Hegde ’10)

◮ k-sparse + more =

⇒ O(k) measurements.

Projections on manifolds (Baraniuk-Wakin ’09, Eftekhari-Wakin ’15)

◮ Conditions on manifold for which recovery is possible.

Deep network models (Mousavi-Dasarathy-Baraniuk ’17, Chang et al ’17)

◮ Train deep network to encode and/or decode. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 14 / 33

slide-85
SLIDE 85

Experimental Results

Faces: n = 64 × 64 × 3 = 12288, m = 500

Original Lasso (DCT) Lasso (Wavelet) DCGAN

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 15 / 33

slide-86
SLIDE 86

Experimental Results

Faces: n = 64 × 64 × 3 = 12288, m = 500

Original Lasso (DCT) Lasso (Wavelet) DCGAN

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 15 / 33

slide-87
SLIDE 87

Experimental Results

Faces: n = 64 × 64 × 3 = 12288, m = 500

Original Lasso (DCT) Lasso (Wavelet) DCGAN

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 15 / 33

slide-88
SLIDE 88

Experimental Results

MNIST: n = 28x28 = 784, m = 100.

Original Lasso VAE Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 15 / 33

slide-89
SLIDE 89

Experimental Results

100 200 300 400 500 Number of measurements 0.00 0.02 0.04 0.06 0.08 0.10 0.12 Reconstruction error (per pixel) Lasso VAE

MNIST

500 1000 1500 2000 2500 Number of measurements 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Reconstruction error (per pixel) Lasso (DCT) Lasso (Wavelet) DCGAN

Faces

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 16 / 33

slide-90
SLIDE 90

Proof Outline (ReLU-based networks)

Show range(G) lies within union of ndk k-dimensional hyperplane.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

slide-91
SLIDE 91

Proof Outline (ReLU-based networks)

Show range(G) lies within union of ndk k-dimensional hyperplane.

◮ Then analogous to proof for sparsity:

n

k

  • ≤ 2k log(n/k) hyperplanes.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

slide-92
SLIDE 92

Proof Outline (ReLU-based networks)

Show range(G) lies within union of ndk k-dimensional hyperplane.

◮ Then analogous to proof for sparsity:

n

k

  • ≤ 2k log(n/k) hyperplanes.

◮ So dk log n Gaussian measurements suffice. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

slide-93
SLIDE 93

Proof Outline (ReLU-based networks)

Show range(G) lies within union of ndk k-dimensional hyperplane.

◮ Then analogous to proof for sparsity:

n

k

  • ≤ 2k log(n/k) hyperplanes.

◮ So dk log n Gaussian measurements suffice.

ReLU-based network:

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

slide-94
SLIDE 94

Proof Outline (ReLU-based networks)

Show range(G) lies within union of ndk k-dimensional hyperplane.

◮ Then analogous to proof for sparsity:

n

k

  • ≤ 2k log(n/k) hyperplanes.

◮ So dk log n Gaussian measurements suffice.

ReLU-based network:

◮ Each layer is z → ReLU(Aiz). Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

slide-95
SLIDE 95

Proof Outline (ReLU-based networks)

Show range(G) lies within union of ndk k-dimensional hyperplane.

◮ Then analogous to proof for sparsity:

n

k

  • ≤ 2k log(n/k) hyperplanes.

◮ So dk log n Gaussian measurements suffice.

ReLU-based network:

◮ Each layer is z → ReLU(Aiz). Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

slide-96
SLIDE 96

Proof Outline (ReLU-based networks)

Show range(G) lies within union of ndk k-dimensional hyperplane.

◮ Then analogous to proof for sparsity:

n

k

  • ≤ 2k log(n/k) hyperplanes.

◮ So dk log n Gaussian measurements suffice.

ReLU-based network:

◮ Each layer is z → ReLU(Aiz). ◮ ReLU(y)i =

yi yi ≥ 0

  • therwise

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

slide-97
SLIDE 97

Proof Outline (ReLU-based networks)

Show range(G) lies within union of ndk k-dimensional hyperplane.

◮ Then analogous to proof for sparsity:

n

k

  • ≤ 2k log(n/k) hyperplanes.

◮ So dk log n Gaussian measurements suffice.

ReLU-based network:

◮ Each layer is z → ReLU(Aiz). ◮ ReLU(y)i =

yi yi ≥ 0

  • therwise

Input to layer 1: single k-dimensional hyperplane.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

slide-98
SLIDE 98

Proof Outline (ReLU-based networks)

Show range(G) lies within union of ndk k-dimensional hyperplane.

◮ Then analogous to proof for sparsity:

n

k

  • ≤ 2k log(n/k) hyperplanes.

◮ So dk log n Gaussian measurements suffice.

ReLU-based network:

◮ Each layer is z → ReLU(Aiz). ◮ ReLU(y)i =

yi yi ≥ 0

  • therwise

Input to layer 1: single k-dimensional hyperplane.

Lemma

Layer 1’s output lies within a union of nk k-dimensional hyperplanes.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

slide-99
SLIDE 99

Proof Outline (ReLU-based networks)

Show range(G) lies within union of ndk k-dimensional hyperplane.

◮ Then analogous to proof for sparsity:

n

k

  • ≤ 2k log(n/k) hyperplanes.

◮ So dk log n Gaussian measurements suffice.

ReLU-based network:

◮ Each layer is z → ReLU(Aiz). ◮ ReLU(y)i =

yi yi ≥ 0

  • therwise

Input to layer 1: single k-dimensional hyperplane.

Lemma

Layer 1’s output lies within a union of nk k-dimensional hyperplanes. Induction: final output lies within ndk k-dimensional hyperplanes.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 17 / 33

slide-100
SLIDE 100

Proof of Lemma

Layer 1’s output lies within a union of nk k-dimensional hyperplanes.

z is k-dimensional.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 18 / 33

slide-101
SLIDE 101

Proof of Lemma

Layer 1’s output lies within a union of nk k-dimensional hyperplanes.

z is k-dimensional. ReLU(A1z) is linear, within any constant region of sign(A1z).

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 18 / 33

slide-102
SLIDE 102

Proof of Lemma

Layer 1’s output lies within a union of nk k-dimensional hyperplanes.

z is k-dimensional. ReLU(A1z) is linear, within any constant region of sign(A1z). How many different patterns can sign(A1z) take?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 18 / 33

slide-103
SLIDE 103

Proof of Lemma

Layer 1’s output lies within a union of nk k-dimensional hyperplanes.

z is k-dimensional. ReLU(A1z) is linear, within any constant region of sign(A1z). How many different patterns can sign(A1z) take? k = 2 version

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 18 / 33

slide-104
SLIDE 104

Proof of Lemma

Layer 1’s output lies within a union of nk k-dimensional hyperplanes.

z is k-dimensional. ReLU(A1z) is linear, within any constant region of sign(A1z). How many different patterns can sign(A1z) take? k = 2 version: how many regions can n lines partition plane into?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 18 / 33

slide-105
SLIDE 105

Proof of Lemma

Layer 1’s output lies within a union of nk k-dimensional hyperplanes.

z is k-dimensional. ReLU(A1z) is linear, within any constant region of sign(A1z). How many different patterns can sign(A1z) take? k = 2 version: how many regions can n lines partition plane into?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 18 / 33

slide-106
SLIDE 106

Proof of Lemma

Layer 1’s output lies within a union of nk k-dimensional hyperplanes.

z is k-dimensional. ReLU(A1z) is linear, within any constant region of sign(A1z). How many different patterns can sign(A1z) take? k = 2 version: how many regions can n lines partition plane into?

◮ 1 + (1 + 2 + . . . + n) = n2+n+2

2

.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 18 / 33

slide-107
SLIDE 107

Proof of Lemma

Layer 1’s output lies within a union of nk k-dimensional hyperplanes.

z is k-dimensional. ReLU(A1z) is linear, within any constant region of sign(A1z). How many different patterns can sign(A1z) take? k = 2 version: how many regions can n lines partition plane into?

◮ 1 + (1 + 2 + . . . + n) = n2+n+2

2

.

◮ n half-spaces divide Rk into less than nk regions. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 18 / 33

slide-108
SLIDE 108

Proof of Lemma

Layer 1’s output lies within a union of nk k-dimensional hyperplanes.

z is k-dimensional. ReLU(A1z) is linear, within any constant region of sign(A1z). How many different patterns can sign(A1z) take? k = 2 version: how many regions can n lines partition plane into?

◮ 1 + (1 + 2 + . . . + n) = n2+n+2

2

.

◮ n half-spaces divide Rk into less than nk regions.

  • Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin)

Compressed Sensing and Generative Models 18 / 33

slide-109
SLIDE 109

Proof of Lemma

Layer 1’s output lies within a union of nk k-dimensional hyperplanes.

z is k-dimensional. ReLU(A1z) is linear, within any constant region of sign(A1z). How many different patterns can sign(A1z) take? k = 2 version: how many regions can n lines partition plane into?

◮ 1 + (1 + 2 + . . . + n) = n2+n+2

2

.

◮ n half-spaces divide Rk into less than nk regions.

  • Therefore d-layer network has ndk regions.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 18 / 33

slide-110
SLIDE 110

Summary (part 1)

m > (information in image) (new info. per measurement)

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 19 / 33

slide-111
SLIDE 111

Summary (part 1)

m > (information in image) (new info. per measurement) Generative models can bound information content as O(kd log n).

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 19 / 33

slide-112
SLIDE 112

Summary (part 1)

m > (information in image) (new info. per measurement) Generative models can bound information content as O(kd log n). Generative models differentiable = ⇒ can optimize in practice.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 19 / 33

slide-113
SLIDE 113

Summary (part 1)

m > (information in image) (new info. per measurement) Generative models can bound information content as O(kd log n). Generative models differentiable = ⇒ can optimize in practice. Gaussian measurements ensure independent information.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 19 / 33

slide-114
SLIDE 114

Summary (part 1)

m > (information in image) (new info. per measurement) Generative models can bound information content as O(kd log n). Generative models differentiable = ⇒ can optimize in practice. Gaussian measurements ensure independent information.

◮ O(1) approximation factor Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 19 / 33

slide-115
SLIDE 115

Summary (part 1)

m > (information in image) (new info. per measurement) Generative models can bound information content as O(kd log n). Generative models differentiable = ⇒ can optimize in practice. Gaussian measurements ensure independent information.

◮ O(1) approximation factor ⇐

⇒ O(1) SNR

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 19 / 33

slide-116
SLIDE 116

Summary (part 1)

m > (information in image) (new info. per measurement) Generative models can bound information content as O(kd log n). Generative models differentiable = ⇒ can optimize in practice. Gaussian measurements ensure independent information.

◮ O(1) approximation factor ⇐

⇒ O(1) SNR ⇐ ⇒ O(1) bits each

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 19 / 33

slide-117
SLIDE 117

Summary (part 1)

m > (information in image) (new info. per measurement) Generative models can bound information content as O(kd log n). Generative models differentiable = ⇒ can optimize in practice. Gaussian measurements ensure independent information.

◮ O(1) approximation factor ⇐

⇒ O(1) SNR ⇐ ⇒ O(1) bits each

With random weights (i.e., before training) can prove more:

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 19 / 33

slide-118
SLIDE 118

Summary (part 1)

m > (information in image) (new info. per measurement) Generative models can bound information content as O(kd log n). Generative models differentiable = ⇒ can optimize in practice. Gaussian measurements ensure independent information.

◮ O(1) approximation factor ⇐

⇒ O(1) SNR ⇐ ⇒ O(1) bits each

With random weights (i.e., before training) can prove more:

◮ The optimization has no local minima [Hand-Voroninski] Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 19 / 33

slide-119
SLIDE 119

Summary (part 1)

m > (information in image) (new info. per measurement) Generative models can bound information content as O(kd log n). Generative models differentiable = ⇒ can optimize in practice. Gaussian measurements ensure independent information.

◮ O(1) approximation factor ⇐

⇒ O(1) SNR ⇐ ⇒ O(1) bits each

With random weights (i.e., before training) can prove more:

◮ The optimization has no local minima [Hand-Voroninski] ◮ L = O(1) not nd so m = O(k log n), if k ≪ n/d. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 19 / 33

slide-120
SLIDE 120

Extensions

Inpainting:

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 20 / 33

slide-121
SLIDE 121

Extensions

Inpainting:

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 20 / 33

slide-122
SLIDE 122

Extensions

Inpainting:

◮ A is diagonal, zeros and ones. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 20 / 33

slide-123
SLIDE 123

Extensions

Inpainting:

◮ A is diagonal, zeros and ones.

Deblurring:

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 20 / 33

slide-124
SLIDE 124

Extensions

Inpainting:

◮ A is diagonal, zeros and ones.

Deblurring:

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 20 / 33

slide-125
SLIDE 125

Talk Outline

1

Using generative models for compressed sensing

2

Learning generative models from noisy data

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 21 / 33

slide-126
SLIDE 126

Where does the generative model come from?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 22 / 33

slide-127
SLIDE 127

Where does the generative model come from?

Training from lots of data.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 22 / 33

slide-128
SLIDE 128

Where does the generative model come from?

Training from lots of data.

Problem

If measuring images is hard/noisy, how do you collect a good data set?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 22 / 33

slide-129
SLIDE 129

Where does the generative model come from?

Training from lots of data.

Problem

If measuring images is hard/noisy, how do you collect a good data set?

Question

Can we learn a GAN from incomplete, noisy measurements of the desired images?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 22 / 33

slide-130
SLIDE 130

GAN Architecture

Z G Generated image Real image D Real?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 23 / 33

slide-131
SLIDE 131

GAN Architecture

Z G Generated image Real image D Real?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 23 / 33

slide-132
SLIDE 132

GAN Architecture

Z G Generated image Real image D Real?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 23 / 33

slide-133
SLIDE 133

GAN Architecture

Z G Generated image Real image D Real?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 23 / 33

slide-134
SLIDE 134

GAN Architecture

Z G Generated image Real image D Real?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 23 / 33

slide-135
SLIDE 135

GAN Architecture

Z G Generated image Real image D Real?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 23 / 33

slide-136
SLIDE 136

GAN Architecture

Z G Generated image Real image D Real?

Generator G wants to fool the discriminator D.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 24 / 33

slide-137
SLIDE 137

GAN Architecture

Z G Generated image Real image D Real?

Generator G wants to fool the discriminator D. If G, D infinitely powerful: only pure Nash equilibrium when G(Z) equals true distribution.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 24 / 33

slide-138
SLIDE 138

GAN Architecture

Z G Generated image Real image D Real?

Generator G wants to fool the discriminator D. If G, D infinitely powerful: only pure Nash equilibrium when G(Z) equals true distribution. Empirically works for G, D being convolutional neural nets.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 24 / 33

slide-139
SLIDE 139

GAN training

Z G Generated image Real image D Real? Real measurement Simulated measurement f

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 25 / 33

slide-140
SLIDE 140

GAN training

Z G Generated image Real image D Real? Real measurement Simulated measurement f

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 25 / 33

slide-141
SLIDE 141

AmbientGAN training

Z G Generated image Real image D Real? Real measurement Simulated measurement f

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 25 / 33

slide-142
SLIDE 142

AmbientGAN training

Z G Generated image Real image D Real? Real measurement Simulated measurement f

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 25 / 33

slide-143
SLIDE 143

AmbientGAN training

Z G Generated image Real image D Real? Real measurement Simulated measurement f

Discriminator must distinguish real measurements from simulated measurements of fake images

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 25 / 33

slide-144
SLIDE 144

AmbientGAN training

Z G Generated image Real image D Real? Real measurement Simulated measurement f

Discriminator must distinguish real measurements from simulated measurements of fake images Can try this for any measurement process f you understand.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 25 / 33

slide-145
SLIDE 145

AmbientGAN training

Z G Generated image Real image D Real? Real measurement Simulated measurement f

Discriminator must distinguish real measurements from simulated measurements of fake images Can try this for any measurement process f you understand. Compatible with any GAN generator architecture.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 25 / 33

slide-146
SLIDE 146

Measurement: Gaussian blur + Gaussian noise

Measured Wiener Baseline AmbientGAN Gaussian blur + additive Gaussian noise attenuates high-frequency components.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 26 / 33

slide-147
SLIDE 147

Measurement: Gaussian blur + Gaussian noise

Measured Wiener Baseline AmbientGAN Gaussian blur + additive Gaussian noise attenuates high-frequency components. Wiener baseline: deconvolve before learning GAN.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 26 / 33

slide-148
SLIDE 148

Measurement: Gaussian blur + Gaussian noise

Measured Wiener Baseline AmbientGAN Gaussian blur + additive Gaussian noise attenuates high-frequency components. Wiener baseline: deconvolve before learning GAN. AmbientGAN better preserves high-frequency components.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 26 / 33

slide-149
SLIDE 149

Measurement: Gaussian blur + Gaussian noise

Measured Wiener Baseline AmbientGAN Gaussian blur + additive Gaussian noise attenuates high-frequency components. Wiener baseline: deconvolve before learning GAN. AmbientGAN better preserves high-frequency components. Theorem: in the limit of dataset size and G, D capacity → ∞, Nash equilibrium of AmbientGAN is the true distribution.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 26 / 33

slide-150
SLIDE 150

Measurement: Obscured Square

Measured Inpainting Baseline AmbientGAN Obscure a random square containing 25% of the image.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 27 / 33

slide-151
SLIDE 151

Measurement: Obscured Square

Measured Inpainting Baseline AmbientGAN Obscure a random square containing 25% of the image. Inpainting followed by GAN training reproduces inpainting artifacts.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 27 / 33

slide-152
SLIDE 152

Measurement: Obscured Square

Measured Inpainting Baseline AmbientGAN Obscure a random square containing 25% of the image. Inpainting followed by GAN training reproduces inpainting artifacts. AmbientGAN gives much smaller artifacts.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 27 / 33

slide-153
SLIDE 153

Measurement: Obscured Square

Measured Inpainting Baseline AmbientGAN Obscure a random square containing 25% of the image. Inpainting followed by GAN training reproduces inpainting artifacts. AmbientGAN gives much smaller artifacts. No theorem: doesn’t know that eyes should have the same color.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 27 / 33

slide-154
SLIDE 154

Measurement: Limited View

Motivation: learn the distribution of panoramas from the distribution

  • f photos?

Measured AmbientGAN

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 28 / 33

slide-155
SLIDE 155

Measurement: Limited View

Motivation: learn the distribution of panoramas from the distribution

  • f photos?

Measured AmbientGAN Reveal a random square containing 25% of the image.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 28 / 33

slide-156
SLIDE 156

Measurement: Limited View

Motivation: learn the distribution of panoramas from the distribution

  • f photos?

Measured AmbientGAN Reveal a random square containing 25% of the image. AmbientGAN still recovers faces.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 28 / 33

slide-157
SLIDE 157

Measurement: Dropout

Measured Blurring Baseline AmbientGAN Drop each pixel independently with probability p = 95%.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 29 / 33

slide-158
SLIDE 158

Measurement: Dropout

Measured Blurring Baseline AmbientGAN Drop each pixel independently with probability p = 95%. Simple baseline does terribly.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 29 / 33

slide-159
SLIDE 159

Measurement: Dropout

Measured Blurring Baseline AmbientGAN Drop each pixel independently with probability p = 95%. Simple baseline does terribly. AmbientGAN can still learn faces.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 29 / 33

slide-160
SLIDE 160

Measurement: Dropout

Measured Blurring Baseline AmbientGAN Drop each pixel independently with probability p = 95%. Simple baseline does terribly. AmbientGAN can still learn faces. Theorem: in the limit of dataset size and G, D capacity → ∞, Nash equilibrium of AmbientGAN is the true distribution.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 29 / 33

slide-161
SLIDE 161

1D Projections

So far, measurements have all looked like images themselves.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 30 / 33

slide-162
SLIDE 162

1D Projections

So far, measurements have all looked like images themselves. What if we turn a 2D image into a 1D image?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 30 / 33

slide-163
SLIDE 163

1D Projections

So far, measurements have all looked like images themselves. What if we turn a 2D image into a 1D image? Motivation: X-ray scans project 3D into 2D.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 30 / 33

slide-164
SLIDE 164

1D Projections

So far, measurements have all looked like images themselves. What if we turn a 2D image into a 1D image? Motivation: X-ray scans project 3D into 2D. Face reconstruction is crude, but MNIST digits work decently:

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 30 / 33

slide-165
SLIDE 165

1D Projections

So far, measurements have all looked like images themselves. What if we turn a 2D image into a 1D image? Motivation: X-ray scans project 3D into 2D. Face reconstruction is crude, but MNIST digits work decently:

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 30 / 33

slide-166
SLIDE 166

Compressed sensing

Compressed sensing: learn an image x from low-dimensional linear projection Ax.

10 50 100 200 300 400 500 750 Number of measurements (m) 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 Reconstruction error (per pixel)

AmbientGAN (ours) Lasso

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 31 / 33

slide-167
SLIDE 167

Compressed sensing

Compressed sensing: learn an image x from low-dimensional linear projection Ax. AmbientGAN can learn the generative model from a dataset of projections {(Ai, Aixi)}.

10 50 100 200 300 400 500 750 Number of measurements (m) 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 Reconstruction error (per pixel)

AmbientGAN (ours) Lasso

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 31 / 33

slide-168
SLIDE 168

Compressed sensing

Compressed sensing: learn an image x from low-dimensional linear projection Ax. AmbientGAN can learn the generative model from a dataset of projections {(Ai, Aixi)}. Beats standard sparse recovery (e.g. Lasso).

10 50 100 200 300 400 500 750 Number of measurements (m) 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 Reconstruction error (per pixel)

AmbientGAN (ours) Lasso

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 31 / 33

slide-169
SLIDE 169

Compressed sensing

Compressed sensing: learn an image x from low-dimensional linear projection Ax. AmbientGAN can learn the generative model from a dataset of projections {(Ai, Aixi)}. Beats standard sparse recovery (e.g. Lasso).

10 50 100 200 300 400 500 750 Number of measurements (m) 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 Reconstruction error (per pixel)

AmbientGAN (ours) Lasso

Theorem about unique Nash equilibrium in the limit.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 31 / 33

slide-170
SLIDE 170

Summary

Z G Generated image D Real measurement Simulated measurement f

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 32 / 33

slide-171
SLIDE 171

Summary

Z G Generated image D Real measurement Simulated measurement f

Plug the measurement process into the GAN architecture of your choice.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 32 / 33

slide-172
SLIDE 172

Summary

Z G Generated image D Real measurement Simulated measurement f

Plug the measurement process into the GAN architecture of your choice. The generator learns the pre-measurement ground truth better than if you denoise before training.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 32 / 33

slide-173
SLIDE 173

Summary

Z G Generated image D Real measurement Simulated measurement f

Plug the measurement process into the GAN architecture of your choice. The generator learns the pre-measurement ground truth better than if you denoise before training. Could let us learn distributions we have no data for.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 32 / 33

slide-174
SLIDE 174

Summary

Z G Generated image D Real measurement Simulated measurement f

Plug the measurement process into the GAN architecture of your choice. The generator learns the pre-measurement ground truth better than if you denoise before training. Could let us learn distributions we have no data for. Read the paper (“AmbientGAN”) for lots more experiments.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 32 / 33

slide-175
SLIDE 175

Conclusion and open questions

Main results:

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-176
SLIDE 176

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-177
SLIDE 177

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

◮ Can use a generative model to reconstruct from lossy measurements. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-178
SLIDE 178

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

◮ Can use a generative model to reconstruct from lossy measurements.

Finite-sample theorems for learning the generative model?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-179
SLIDE 179

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

◮ Can use a generative model to reconstruct from lossy measurements.

Finite-sample theorems for learning the generative model?

◮ Take Gaussian blur plus Gaussian noise. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-180
SLIDE 180

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

◮ Can use a generative model to reconstruct from lossy measurements.

Finite-sample theorems for learning the generative model?

◮ Take Gaussian blur plus Gaussian noise. ◮ Wiener filter before GAN: lose frequencies beyond O(1) standard

deviations.

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-181
SLIDE 181

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

◮ Can use a generative model to reconstruct from lossy measurements.

Finite-sample theorems for learning the generative model?

◮ Take Gaussian blur plus Gaussian noise. ◮ Wiener filter before GAN: lose frequencies beyond O(1) standard

deviations.

◮ With N data points, can we learn log N standard deviations? Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-182
SLIDE 182

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

◮ Can use a generative model to reconstruct from lossy measurements.

Finite-sample theorems for learning the generative model?

◮ Take Gaussian blur plus Gaussian noise. ◮ Wiener filter before GAN: lose frequencies beyond O(1) standard

deviations.

◮ With N data points, can we learn log N standard deviations?

Better upper bound on complexity of generative models?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-183
SLIDE 183

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

◮ Can use a generative model to reconstruct from lossy measurements.

Finite-sample theorems for learning the generative model?

◮ Take Gaussian blur plus Gaussian noise. ◮ Wiener filter before GAN: lose frequencies beyond O(1) standard

deviations.

◮ With N data points, can we learn log N standard deviations?

Better upper bound on complexity of generative models?

◮ Lipschitz parameter at initialization is much smaller than nd... Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-184
SLIDE 184

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

◮ Can use a generative model to reconstruct from lossy measurements.

Finite-sample theorems for learning the generative model?

◮ Take Gaussian blur plus Gaussian noise. ◮ Wiener filter before GAN: lose frequencies beyond O(1) standard

deviations.

◮ With N data points, can we learn log N standard deviations?

Better upper bound on complexity of generative models?

◮ Lipschitz parameter at initialization is much smaller than nd... ◮ ...but we don’t actually expect it to be small after training. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-185
SLIDE 185

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

◮ Can use a generative model to reconstruct from lossy measurements.

Finite-sample theorems for learning the generative model?

◮ Take Gaussian blur plus Gaussian noise. ◮ Wiener filter before GAN: lose frequencies beyond O(1) standard

deviations.

◮ With N data points, can we learn log N standard deviations?

Better upper bound on complexity of generative models?

◮ Lipschitz parameter at initialization is much smaller than nd... ◮ ...but we don’t actually expect it to be small after training.

Can the reconstruction incorporate density over the manifold?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-186
SLIDE 186

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

◮ Can use a generative model to reconstruct from lossy measurements.

Finite-sample theorems for learning the generative model?

◮ Take Gaussian blur plus Gaussian noise. ◮ Wiener filter before GAN: lose frequencies beyond O(1) standard

deviations.

◮ With N data points, can we learn log N standard deviations?

Better upper bound on complexity of generative models?

◮ Lipschitz parameter at initialization is much smaller than nd... ◮ ...but we don’t actually expect it to be small after training.

Can the reconstruction incorporate density over the manifold?

◮ Computational problem: pseudodeterminant of Jacobian matrix. Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-187
SLIDE 187

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

◮ Can use a generative model to reconstruct from lossy measurements.

Finite-sample theorems for learning the generative model?

◮ Take Gaussian blur plus Gaussian noise. ◮ Wiener filter before GAN: lose frequencies beyond O(1) standard

deviations.

◮ With N data points, can we learn log N standard deviations?

Better upper bound on complexity of generative models?

◮ Lipschitz parameter at initialization is much smaller than nd... ◮ ...but we don’t actually expect it to be small after training.

Can the reconstruction incorporate density over the manifold?

◮ Computational problem: pseudodeterminant of Jacobian matrix. ◮ Speed-up with linear sketching? Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-188
SLIDE 188

Conclusion and open questions

Main results:

◮ Can use lossy measurements to learn a generative model of the

underlying distribution.

◮ Can use a generative model to reconstruct from lossy measurements.

Finite-sample theorems for learning the generative model?

◮ Take Gaussian blur plus Gaussian noise. ◮ Wiener filter before GAN: lose frequencies beyond O(1) standard

deviations.

◮ With N data points, can we learn log N standard deviations?

Better upper bound on complexity of generative models?

◮ Lipschitz parameter at initialization is much smaller than nd... ◮ ...but we don’t actually expect it to be small after training.

Can the reconstruction incorporate density over the manifold?

◮ Computational problem: pseudodeterminant of Jacobian matrix. ◮ Speed-up with linear sketching?

More uses of differentiable compression?

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 33 / 33

slide-189
SLIDE 189

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 34 / 33

slide-190
SLIDE 190

Ashish Bora, Ajil Jalal, Eric Price, Alex Dimakis (UT Austin) Compressed Sensing and Generative Models 35 / 33