On Inferences from Completed Data Jamie Haddock February 14, 2019 - - PowerPoint PPT Presentation

On Inferences from Completed Data

Jamie Haddock February 14, 2019

Computational and Applied Mathematics, UCLA

joint with 2019 UCLA REU group (D. Molitor, D. Needell, S. Sambandam, J. Song, S. Sun)

1

Motivation

MyLymeData is a large collection of Lyme disease patient survey data collected by LymeDisease.org (∼12,000 patients, 100s of questions)

2

Motivation

MyLymeData is a large collection of Lyme disease patient survey data collected by LymeDisease.org (∼12,000 patients, 100s of questions)

• data is highly incomplete due to branching structure of surveys and

missing responses

2

Motivation

MyLymeData is a large collection of Lyme disease patient survey data collected by LymeDisease.org (∼12,000 patients, 100s of questions)

• data is highly incomplete due to branching structure of surveys and

missing responses

• research questions of interest do not require individual entries

2

Motivation

MyLymeData is a large collection of Lyme disease patient survey data collected by LymeDisease.org (∼12,000 patients, 100s of questions)

• data is highly incomplete due to branching structure of surveys and

missing responses

• research questions of interest do not require individual entries

Question: Can we perform statistical inferences on imputed data?

2

Main Question

3

Sampling and Imputation Techniques

Uniform Sampling: Sample each entry with uniform probability p.

4

Sampling and Imputation Techniques

Uniform Sampling: Sample each entry with uniform probability p. Structured Sampling: Sample zero and nonzero entries with p0 and p1.

4

Sampling and Imputation Techniques

Uniform Sampling: Sample each entry with uniform probability p. Structured Sampling: Sample zero and nonzero entries with p0 and p1. Nuclear Norm Minimization (NNM): min X∗ s.t. Xij = Mij for all (i, j) ∈ Ω

4

Sampling and Imputation Techniques

Uniform Sampling: Sample each entry with uniform probability p. Structured Sampling: Sample zero and nonzero entries with p0 and p1. Nuclear Norm Minimization (NNM): min X∗ s.t. Xij = Mij for all (i, j) ∈ Ω ℓ1-Regularized Nuclear Norm Minimization (ℓ1-NNM): min X∗ + αXΩC 1 s.t. Xij = Mij for all (i, j) ∈ Ω

4

Simple Inferences

Entrywise Mean λ(M): mean of the entries of M

• Entrywise mean error:

Eλ = |λ( ˆ M) − λ(M)|. Row Mean µ(M): average row of M

• Normalized row mean error:

Eµ = µ( ˆ M) − µ(M)2 µ(M)2 . ⊲ original matrix, M ⊲ recovered matrix, ˆ M

5

Experimental Design - Synthetic Data

⊲ 30 × 30 rank 5 matrix generated as product of sparse matrices with nonzero entries sampled uniformly from [0, 1]

6

Experimental Design - Synthetic Data

⊲ 30 × 30 rank 5 matrix generated as product of sparse matrices with nonzero entries sampled uniformly from [0, 1] ⊲ each trial consists of sampling, completion, and inference on original and completed matrices

6

Experimental Design - Synthetic Data

⊲ 30 × 30 rank 5 matrix generated as product of sparse matrices with nonzero entries sampled uniformly from [0, 1] ⊲ each trial consists of sampling, completion, and inference on original and completed matrices

• matrix is sampled via uniform sampling and structured sampling

(with listed p0), and completed with NNM and ℓ1-NNM respectively

6

Experimental Design - Synthetic Data

⊲ 30 × 30 rank 5 matrix generated as product of sparse matrices with nonzero entries sampled uniformly from [0, 1] ⊲ each trial consists of sampling, completion, and inference on original and completed matrices

• matrix is sampled via uniform sampling and structured sampling

(with listed p0), and completed with NNM and ℓ1-NNM respectively

• ℓ1 regularization parameter α is chosen in {0.05, 0.1, 0.2, . . . , 0.5} to

minimize matrix recovery error

6

Experimental Design - Synthetic Data

⊲ 30 × 30 rank 5 matrix generated as product of sparse matrices with nonzero entries sampled uniformly from [0, 1] ⊲ each trial consists of sampling, completion, and inference on original and completed matrices

• matrix is sampled via uniform sampling and structured sampling

(with listed p0), and completed with NNM and ℓ1-NNM respectively

• ℓ1 regularization parameter α is chosen in {0.05, 0.1, 0.2, . . . , 0.5} to

minimize matrix recovery error

⊲ matrix recovery error and inference errors averaged over 10 trials

6

Synthetic Data

⊲ p0 = 0 ⊲ ω is proportion of entries sampled

7

Synthetic Data

⊲ p0 = 0.2 ⊲ ω is proportion of entries sampled

8

Synthetic Data

⊲ p0 = 0.4 ⊲ ω is proportion of entries sampled

9

Experimental Design - MyLymeData

⊲ complete 30 × 16 submatrix of MyLymeData

10

Experimental Design - MyLymeData

⊲ complete 30 × 16 submatrix of MyLymeData ⊲ each trial consists of sampling, completion, and inference on original and completed matrices

10

Experimental Design - MyLymeData

⊲ complete 30 × 16 submatrix of MyLymeData ⊲ each trial consists of sampling, completion, and inference on original and completed matrices

• matrix is sampled via uniform sampling and structured sampling

(with listed p0), and completed with NNM and ℓ1-NNM respectively

10

Experimental Design - MyLymeData

⊲ complete 30 × 16 submatrix of MyLymeData ⊲ each trial consists of sampling, completion, and inference on original and completed matrices

• matrix is sampled via uniform sampling and structured sampling

(with listed p0), and completed with NNM and ℓ1-NNM respectively

• ℓ1 regularization parameter α is chosen in {0.05, 0.1, 0.2, . . . , 0.5} to

minimize matrix recovery error

10

Experimental Design - MyLymeData

⊲ complete 30 × 16 submatrix of MyLymeData ⊲ each trial consists of sampling, completion, and inference on original and completed matrices

• matrix is sampled via uniform sampling and structured sampling

(with listed p0), and completed with NNM and ℓ1-NNM respectively

• ℓ1 regularization parameter α is chosen in {0.05, 0.1, 0.2, . . . , 0.5} to

minimize matrix recovery error

⊲ matrix recovery error and inference errors averaged over 10 trials

10

MyLyme Data

⊲ p0 = 0 ⊲ ω is proportion of entries sampled

11

MyLyme Data

⊲ p0 = 0.2 ⊲ ω is proportion of entries sampled

12

MyLyme Data

⊲ p0 = 0.4 ⊲ ω is proportion of entries sampled

13

Preliminary Error Bounds

Inference Error Bound Entrywise Mean |λ(M) − λ( ˆ M)| ≤ (mn)

− 1 q M − ˆ

Mq Row Mean µ(M) − µ( ˆ M)q ≤

• nq−1

m

1

q M − ˆ

Mq ⊲ M ∈ Rm×n ⊲ recovered matrix, ˆ M

14

Entrywise Mean Simulation

15

Row Mean Simulation

16

Conclusions and Future Directions

• inference errors can be smaller than the associated matrix recovery

errors

17

Conclusions and Future Directions

• inference errors can be smaller than the associated matrix recovery

errors

• structured sampling and ℓ1-NNM often results in better matrix and

inference recovery than uniform sampling and NNM

17

Conclusions and Future Directions

• inference errors can be smaller than the associated matrix recovery

errors

• structured sampling and ℓ1-NNM often results in better matrix and

inference recovery than uniform sampling and NNM

• develop exact recovery guarantees for ℓ1-NNM on matrices with
• bserved entries selected via structured sampling

17

References and Acknowledgements

[Cand` es and Recht, 2009] Emmanuel J. Cand`

es and Benjamin Recht (2009) Exact Matrix Completion via Convex Optimization Foundations of Computational Mathematics 9, 771 – 772. [Molitor and Needell, 2018] Denali Molitor and Deanna Needell (2018) Matrix Completion for Structured Observations arXiv preprint arXiv:1801.09657 [Eld` en, 2007] Lars Eld` en Matrix Methods in Data Mining and Pattern Recognition, 69 Society for Industrial and Applied Mathematics, Philadelphia, 2007 Thank you to Professor Andrea Bertozzi, Dr. Anna Ma, Lorraine Johnson (LDo CEO), and the patients who contributed to the MyLymeData database!

18

Thanks!

Questions?

19