3.26pt Randomized Projections for Corrupted Linear Systems Jamie - - PowerPoint PPT Presentation

3.26pt

Randomized Projections for Corrupted Linear Systems

Jamie Haddock1, Deanna Needell2

1Graduate Group in Applied Mathematics, University of California, Davis 2Department of Mathematics, University of California, Los Angeles

jhaddock@math.ucdavis.edu January 28, 2018

Problem

Solve large-scale, highly overdetermined, corrupted system of equations for solution to uncorrupted subsystem.

1

Problem

Solve large-scale, highly overdetermined, corrupted system of equations for solution to uncorrupted subsystem. Problem: Ax = b + e, A ∈ Rm×n, m >> n (Corrupted) Error (e): sparse, arbitrarily large entries Solution (x∗): x∗ ∈ {x : Ax = b}

1

Problem

Solve large-scale, highly overdetermined, corrupted system of equations for solution to uncorrupted subsystem. Problem: Ax = b + e, A ∈ Rm×n, m >> n (Corrupted) Error (e): sparse, arbitrarily large entries Solution (x∗): x∗ ∈ {x : Ax = b} Applications: logic programming, error correction in telecommunications

1

Problem

Solve large-scale, highly overdetermined, corrupted system of equations for solution to uncorrupted subsystem. Problem: Ax = b + e, A ∈ Rm×n, m >> n (Corrupted) Error (e): sparse, arbitrarily large entries Solution (x∗): x∗ ∈ {x : Ax = b} Applications: logic programming, error correction in telecommunications Problem: Ax = b + e, A ∈ Rm×n, m >> n (Noisy) Error (e): small, evenly distributed entries Solution (xLS): xLS ∈ argminAx − b − e2

1

Why not least-squares?

x∗ xLS

2

Randomized Kaczmarz

RK

• 1. Start with initial guess x0
• 2. xk+1 = xk +

bik −aT

ik xk

aik 2 aik where ik ∈ [m] is chosen randomly

• 3. Repeat (2)

x x0

3

Randomized Kaczmarz

RK

• 1. Start with initial guess x0
• 2. xk+1 = xk +

bik −aT

ik xk

aik 2 aik where ik ∈ [m] is chosen randomly

• 3. Repeat (2)

x x0 x1

3

Randomized Kaczmarz

RK

• 1. Start with initial guess x0
• 2. xk+1 = xk +

bik −aT

ik xk

aik 2 aik where ik ∈ [m] is chosen randomly

• 3. Repeat (2)

x x0 x1 x2

3

Randomized Kaczmarz

RK

• 1. Start with initial guess x0
• 2. xk+1 = xk +

bik −aT

ik xk

aik 2 aik where ik ∈ [m] is chosen randomly

• 3. Repeat (2)

x x0 x1 x2 x3

3

Randomized Kaczmarz

RK

• 1. Start with initial guess x0
• 2. xk+1 = xk +

bik −aT

ik xk

aik 2 aik where ik ∈ [m] is chosen randomly

• 3. Repeat (2)

Theorem (Strohmer-Vershynin, 2008) If Ax = b is consistent and RK is used with P[ik = j] = aj2/A2

F then

iterates converge linearly in expectation with Exk − x2 ≤

• 1 −

1 A2

FA−12

k x0 − x2.

4

Proposed Method

Goal: Use RK to detect the corrupted equations with high probability.

5

Proposed Method

Goal: Use RK to detect the corrupted equations with high probability. Lemma (H.-Needell) Let ǫ∗ = mini∈[m] |Ax∗ − b|i = |ei| and suppose |supp(e)| = s. If ||ai|| = 1 for i ∈ [m] and ||x − x∗|| < 1

2ǫ∗ we have that the d ≤ s indices

• f largest magnitude residual entries are contained in supp(e). That is,

we have D ⊂ supp(e), where D = argmax

D⊂[A],|D|=d

• i∈D

|Ax − b|i.

5

Proposed Method

Goal: Use RK to detect the corrupted equations with high probability. x∗ xk

5

Proposed Method

Goal: Use RK to detect the corrupted equations with high probability. x∗ xk We call ǫ∗/2 the detection horizon.

5

Proposed Method

Method 1 Windowed Kaczmarz

1: procedure WK(A, b, k, W , d) 2:

S = ∅

3:

for i = 1, 2, ...W do

4:

xi

k = kth iterate produced by RK with x0 = 0, A, b.

5:

D = d indices of the largest entries of the residual, |Axi

k − b|.

6:

S = S ∪ D

7:

return x, where ASC x = bSC

6

Example

WK(A,b,k = 2,W = 3,d = 1): j = 1, i = 1, S = ∅ x∗ x1 H1 H2 H3 H4 H5 H6 H7

7

Example

WK(A,b,k = 2,W = 3,d = 1): j = 1, i = 1, S = ∅ x∗ x1 x1

1

H1 H2 H3 H4 H5 H6 H7

7

Example

WK(A,b,k = 2,W = 3,d = 1): j = 2, i = 1, S = {7} x∗ x1 x1

1

x1

2

H1 H2 H3 H4 H5 H6 H7

7

Example

WK(A,b,k = 2,W = 3,d = 1): j = 1, i = 2, S = {7} x∗ x2 H1 H2 H3 H4 H5 H6 H7

7

Example

WK(A,b,k = 2,W = 3,d = 1): j = 1, i = 2, S = {7} x∗ x2 H1 H2 H3 H4 H5 H6 H7 x2

1 7

Example

WK(A,b,k = 2,W = 3,d = 1): j = 2, i = 2, S = {7, 5} x∗ x2 H1 H2 H3 H4 H5 H6 H7 x2

1

x2

2 7

Example

WK(A,b,k = 2,W = 3,d = 1): j = 1, i = 3, S = {7, 5} x∗ x3 H1 H2 H3 H4 H5 H6 H7

7

Example

WK(A,b,k = 2,W = 3,d = 1): j = 1, i = 3, S = {7, 5} x∗ x3 H1 H2 H3 H4 H5 H6 H7 x3

1 7

Example

WK(A,b,k = 2,W = 3,d = 1): j = 2, i = 3, S = {7, 5, 6} x∗ x3 H1 H2 H3 H4 H5 H6 H7 x3

1

x3

2 7

Example

Solve ASC x = bSC . x∗ H1 H2 H3 H4

7

Theoretical Guarantees

Lemma (H.-Needell) Let ǫ∗ = mini∈[m] |Ax∗ − b|i = |ei| and suppose |supp(e)| = s. Assume that ||ai|| = 1 for all i ∈ [m] and let 0 < δ < 1. Define k∗ =

• log
• δ(ǫ∗)2

4||x∗||2

• log
• 1 −

σ2

min(Asupp(e)C )

m−s

• .

Then in window i of the Windowed Kaczmarz method, the iterate produced by the RK iterations, xi

k∗ satisfies

P

• ||xi

k∗ − x∗|| ≤ 1

2ǫ∗ ≥ p := (1 − δ) m − s m k∗ .

8

Theoretical Guarantee Values (Gaussian A ∈ R50000×100)

k∗ =

• log
• δ(ǫ∗)2

4||x∗||2

• log
• 1−

σ2 min(Asupp(e)C ) m−s

• 9

Theoretical Guarantee Values (Correlated A ∈ R50000×100)

k∗ =

• log
• δ(ǫ∗)2

4||x∗||2

• log
• 1−

σ2 min(Asupp(e)C ) m−s

• 10

Theoretical Guarantee Values (Gaussian A ∈ R50000×100)

P

• ||xi

k∗ − x∗|| ≤ 1 2ǫ∗

≥ p := (1 − δ)

• m−s

m

k∗

0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

p

s = 1 s = 10 s = 50 s = 100 s = 200 s = 300 s = 400 11

Theoretical Guarantee Values (Correlated A ∈ R50000×100)

P

• ||xi

k∗ − x∗|| ≤ 1 2ǫ∗

≥ p := (1 − δ)

• m−s

m

k∗

12

Theoretical Guarantees

Theorem (H.-Needell) Assume that ||ai|| = 1 for all i ∈ [m] and let 0 < δ < 1. Suppose d ≥ s = |supp(e)|, W ≤ ⌊ m−n

d ⌋ and k∗ is as given in lemma 2. Then the

Windowed Kaczmarz method on A, b will detect the corrupted equations (supp(e) ⊂ S) and the remaining equations given by A[m]−S, b[m]−S will have solution x∗ with probability at least pW := 1 −

• 1 − (1 − δ)

m − s m k∗W .

13

Theoretical Guarantee Values (Gaussian A ∈ R50000×100)

pW := 1 −

• 1 − (1 − δ)
• m−s

m

k∗W

0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

pW

s = 1 s = 10 s = 50 s = 100 s = 200 s = 300 s = 400 14

Experimental Values (Gaussian A ∈ R50000×100)

0.2 0.4 0.6 0.8 1 0.4 0.5 0.6 0.7 0.8 0.9 1

Success ratio

s = 100 s = 200 s = 500 s = 750 s = 1000

15

Experimental Values (Gaussian A ∈ R50000×100)

500 1000 1500 2000

k

0.2 0.4 0.6 0.8 1

Success ratio

s = 100 s = 200 s = 500 s = 750 s = 1000

16

Experimental Values (Gaussian A ∈ R50000×100)

17

Experimental Values (Gaussian A ∈ R50000×100)

18

Theoretical Guarantee Values (Correlated A ∈ R50000×100)

pW := 1 −

• 1 − (1 − δ)
• m−s

m

k∗W

19

Experimental Values (Correlated A ∈ R50000×100)

20

Experimental Values (Correlated A ∈ R50000×100)

21

Conclusions and Future Work

• randomized projection methods are able to detect corruption
• often experimental results far outperform theoretical guarantees
• performance on real data
• reduce dependence on artificial parameters

22

The End

Thanks! Questions?

• E. Amaldi, P. Belotti, and R. Hauser.

Randomized relaxation methods for the maximum feasible subsystem problem. In International Conference on Integer Programming and Combinatorial Optimization, pages 249–264. Springer, 2005.

• N. Jamil, X. Chen, and A. Cloninger.

Hildreths algorithm with applications to soft constraints for user interface layout. Journal of Computational and Applied Mathematics, 288:193–202, 2015.

• S. Kaczmarz.

Angen¨ aherte aufl¨

• sung von systemen linearer gleichungen.

Bull.Internat.Acad.Polon.Sci.Lettres A, pages 335–357, 1937.

• T. Strohmer and R. Vershynin.

A randomized Kaczmarz algorithm with exponential convergence. Journal of Fourier Analysis and Applications, 15(2):262–278, 2009.

23