Recover Reco verin ing S Struct cture o e of N Nois isy D y - - PowerPoint PPT Presentation

Reco Recover verin ing S Struct cture o e of N Nois isy D y Data thro throug ugh h Hypo pothe thesis Te Testi ting ng

1

2020 IEEE International Symposium on Information Theory

The work of M. Jeong and M. Cardone was supported in part by the U.S. National Science Foundation under Grant CCF-1849757. The work of A. Dytso and H. V. Poor was supported in part by the U.S. National Science Foundation under Grant CCF-1908308.

Minoh Jeong?, Alex Dytso†, Martina Cardone?, and H. Vincent Poor†

?University of Minnesota †Princeton University

Motivation

2

The problem of re recoveri ring ng da data struc ructure ure is becoming a prevailing task of modern communication and computing systems.

Motivation - Example

3

The problem of re recoveri ring ng da data struc ructure ure is becoming a prevailing task of modern communication and computing systems. Privatizing personal data

User

Pe Persona nal da data ta

Recommendation based on noisy data

Noi Noise

• Recommender system

Related Work

4

• Linear regression with shuffled data

○

• A. Pananjady, M. J. Wainwright, and T. A. Courtade, “Linear regression with shuffled data: Statistical and computational

limits of permutation recovery,” IEEE Transactions on Information Theory, vol. 64, no. 5, pp. 3286–3300, May 2018.

○

• D. Hsu, K. Shi, and X. Sun, “Linear regression without correspondence,” in Proceedings of the 31st International Conference
• n Neural Information Processing Systems (NIPS), December 2017, pp. 1530–1539.
• Unlabeled sensing

○

• J. Unnikrishnan, S. Haghighatshoar, and M. Vetterli, “Unlabeled sensing with random linear measurements,” IEEE

Transactions on Information Theory, vol. 64, no. 5, pp. 3237–3253, May 2018.

○

• S. Haghighatshoar and G. Caire, “Signal recovery from unlabeled samples,” IEEE Transactions on Signal Processing, vol. 66,
• no. 5, pp. 1242–1257, March 2018.

Related Work

5

• Linear regression with shuffled data

○

• A. Pananjady, M. J. Wainwright, and T. A. Courtade, “Linear regression with shuffled data: Statistical and computational

limits of permutation recovery,” IEEE Transactions on Information Theory, vol. 64, no. 5, pp. 3286–3300, May 2018.

○

• D. Hsu, K. Shi, and X. Sun, “Linear regression without correspondence,” in Proceedings of the 31st International Conference
• n Neural Information Processing Systems (NIPS), December 2017, pp. 1530–1539.
• Unlabeled sensing

○

• J. Unnikrishnan, S. Haghighatshoar, and M. Vetterli, “Unlabeled sensing with random linear measurements,” IEEE

Transactions on Information Theory, vol. 64, no. 5, pp. 3237–3253, May 2018.

○

• S. Haghighatshoar and G. Caire, “Signal recovery from unlabeled samples,” IEEE Transactions on Signal Processing, vol. 66,
• no. 5, pp. 1242–1257, March 2018.

Bayesian perspective with prior distribution on the data

Main difference:

Generator Y Data

⊕

N ∼ N(0n, KN)

Ground

Truth

Hˆ

π, ˆ

π ∈ P

H⇡?, π? ∈ P

X ∼ N(0n, In)

Decoder

Problem Formulation

• System model

6

Data Hypothesis

X ∈ Rn is the unknown data

Observation

Hπ is the hypothesis for the permutation of X

Decision

Given Y, according to which permutation was X sorted?

Problem Formulation

• System model

7

Generator Y Data

⊕

N ∼ N(0n, KN)

Ground

Truth

Hˆ

π, ˆ

π ∈ P

H⇡?, π? ∈ P

X ∼ N(0n, In)

Decoder

Data Hypothesis

X ∈ Rn is the unknown data

Observation

Hπ is the hypothesis for the permutation of X

Decision

Given Y, according to which permutation was X sorted?

Each hypothesis indicates the order of elements of X

Problem Formulation

• Example

8

  −4 −1 3  

  3 −4 −2  

  3 −4 −2   ∈ H{2,3,1}

  −4 −1 3   ∈ H{1,2,3}

Let n = 3, then we have 6 hypotheses Hπ, π ∈ P defined as H{1,2,3} : x1 ≤ x2 ≤ x3, H{1,3,2} : x1 ≤ x3 ≤ x2, H{2,1,3} : x2 ≤ x1 ≤ x3, H{2,3,1} : x2 ≤ x3 ≤ x1, H{3,1,2} : x3 ≤ x1 ≤ x2, H{3,2,1} : x3 ≤ x2 ≤ x1, where xi, i ∈ [1 : 3] is the i-th element of X.

Hπ has an n-dimensional cone shape

Problem Formulation

• Example

9

  −4 −1 3  

  3 −4 −2  

  3 −4 −2   ∈ H{2,3,1}

  −4 −1 3   ∈ H{1,2,3}

M-ary hypothesis testing

Let n = 3, then we have 6 hypotheses Hπ, π ∈ P defined as H{1,2,3} : x1 ≤ x2 ≤ x3, H{1,3,2} : x1 ≤ x3 ≤ x2, H{2,1,3} : x2 ≤ x1 ≤ x3, H{2,3,1} : x2 ≤ x3 ≤ x1, H{3,1,2} : x3 ≤ x1 ≤ x2, H{3,2,1} : x3 ≤ x2 ≤ x1, where xi, i ∈ [1 : 3] is the i-th element of X.

Hπ has an n-dimensional cone shape

Optimal Decision

• Neyman-Pearson lemma

10

Decision Observation

• J. Neyman and E. S. Pearson, “On the problem of the most efficient tests of statistical hypotheses,” Philosophical Transactions of the Royal Society of London.

Series A, Containing Papers of a Mathematical or Physical Character, vol. 231, pp. 289–337, 1933.

Optimal Decision

• Neyman-Pearson lemma

11

Optimal decision criterion in terms of error probability

Decision Observation

• J. Neyman and E. S. Pearson, “On the problem of the most efficient tests of statistical hypotheses,” Philosophical Transactions of the Royal Society of London.

Series A, Containing Papers of a Mathematical or Physical Character, vol. 231, pp. 289–337, 1933.

Optimal Decision Regions

• Optimal decision regions
• Example

12

If y ∈ Rα,KN, then Hˆ

π = Hα

Collection of y such that given y the optimal decision is Hπ

Rπ,KN =   y ∈ Rn : fY(y, Hπ) > max

τ2P τ6=π

fY(y, Hτ)    , ∀π ∈ P

Optimal Decision Regions (Main Result 1)

• Optimal decision regions

13

Rπ,KN = Hπ, ∀π ∈ P

Theorem. When KN = σ2In (noise is memoryless and isotropic),

Optimal Decision Regions (Main Result 1)

• Example

14

When KN = σ2In

y =   3 −4 −2  

Observation

Hˆ

π = H{2,3,1}

Optimal decision

4 2   ∈ H{2,3,1}

Rπ,KN = Hπ, ∀π ∈ P

Optimal Decision Regions (Main Result 1)

• Example

15

When KN = σ2In

y =   3 −4 −2  

Observation

Hˆ

π = H{2,3,1}

Optimal decision

4 2   ∈ H{2,3,1}

Rπ,KN = Hπ, ∀π ∈ P

Computational complexity is O(n log n)

Probability of Error (Main Result 2)

• Probability of error in terms of the ratio of two volumes

16

When KN = σ2In

A =  In 0n×n In σIn

• <latexit sha1_base64="E/dA+q4a1c9CzlkvWBjdWbCDyCQ=">ACOXicbVDLSsNAFJ34Nr6iLt0MtoqrklRQN0LFje4qWC0IUymt3VwMgkzE7GE/JYb/8Kd4MaFIm79Aac14KMeGDhz7j3ce0+Ucqa06z5aE5NT0zOzc/P2wuLS8oqzunahkxSaNGEJ7IdEQWcCWhpjm0UwkjhcRtfHw/rlDUjFEnGuBykEMekL1mOUaCOFTrNatY/wIb9CPpM5FMtGS3hX0aCryN3TAXvmYxKCwK7Pul7CvWjwk2H9sH0f12VauhU3Fr7gh4nHglqaASzdB58LsJzWIQmnKiVMdzUx3kRGpGORS2nylICb0mfegYKohZJshHlxd4yhd3EukeULjkfrTkZNYqUEcmU6z4pX6WxuK/9U6me4dBDkTaZB0K9BvYxjneBhjLjLJFDNB4YQKpnZFdMrIgnVJmzbhOD9PXmcXNRr3m6tflavNPbKObQBtpEO8hD+6iBTlATtRBFd+gJvaBX6956t6s96/WCav0rKNfsD4+ASJiqZE=</latexit>

CHπ = Hπ × Hπ

B2n(02n, 1)

2n-dimensional ball centered at 02n with radius 1

2n-dimensional cone

Rπ,KN = Hπ, ∀π ∈ P

Probability of Error (Main Result 2)

• Probability of error in terms of the ratio of two volumes

17

When KN = σ2In

A =  In 0n×n In σIn

• <latexit sha1_base64="E/dA+q4a1c9CzlkvWBjdWbCDyCQ=">ACOXicbVDLSsNAFJ34Nr6iLt0MtoqrklRQN0LFje4qWC0IUymt3VwMgkzE7GE/JYb/8Kd4MaFIm79Aac14KMeGDhz7j3ce0+Ucqa06z5aE5NT0zOzc/P2wuLS8oqzunahkxSaNGEJ7IdEQWcCWhpjm0UwkjhcRtfHw/rlDUjFEnGuBykEMekL1mOUaCOFTrNatY/wIb9CPpM5FMtGS3hX0aCryN3TAXvmYxKCwK7Pul7CvWjwk2H9sH0f12VauhU3Fr7gh4nHglqaASzdB58LsJzWIQmnKiVMdzUx3kRGpGORS2nylICb0mfegYKohZJshHlxd4yhd3EukeULjkfrTkZNYqUEcmU6z4pX6WxuK/9U6me4dBDkTaZB0K9BvYxjneBhjLjLJFDNB4YQKpnZFdMrIgnVJmzbhOD9PXmcXNRr3m6tflavNPbKObQBtpEO8hD+6iBTlATtRBFd+gJvaBX6956t6s96/WCav0rKNfsD4+ASJiqZE=</latexit>

CHπ = Hπ × Hπ

B2n(02n, 1)

Pe = 1 − n!Vol

• CHπ ∩ AB2n(02n, 1)
• σnVol (B2n (02n, 1))

Rπ,KN = Hπ, ∀π ∈ P

Theorem. When KN = σ2In (noise is memoryless and isotropic),

Large 𝑜 Behavior (Main Result 3)

• Lower and Upper bounds

18

When KN = σ2In

1 n!  Pc  1 n! kAk2n σn

* kAk2 σ 2 hp 2 + 1, 1 ⌘

kAk = (σ4 + 4)

1 2

2 + σ2 2 + 1 ! 1

2

Lemma.

Large 𝑜 Behavior (Main Result 3)

• Lower and Upper bounds

19

When KN = σ2In

1 n!  Pc  1 n! kAk2n σn

* kAk2 σ 2 hp 2 + 1, 1 ⌘

Very noisy problem

Need to have small σ for reliable detection

kAk = (σ4 + 4)

1 2

2 + σ2 2 + 1 ! 1

2

Lemma.

Main Take-Away Results

20

When KN = σ2In

Rπ,KN = Hπ, ∀π ∈ P

Optimal decision regions

Pe = 1 − n!Vol

• CHπ ∩ AB2n(02n, 1)
• σnVol (B2n (02n, 1))

Probability of error

lim

n→∞

log 1

Pc

log(n!) = 1

Large n regime

Future Work

• Characterizing the optimal decision regions with general noise covariance

matrix.

• Characterizing the probability of error with general noise covariance.

21

Rπ,KN = L (Hπ) , π ∈ P

→ Results can be found in arXiv: 2005.07812

→ Necessary and sufficient conditions such that the optimal decision regions Rπ,KN’s are a linear transformation L(·) of the Hπ’s

Thank you

Email: jeong316@umn.edu

22