A Randomized Likelihood Method for Data Reduction in Large-scale - - PowerPoint PPT Presentation

A Randomized Likelihood Method for Data Reduction in Large-scale Inverse Problems

Ellen B. Le Tan Bui-Thanh Aaron Myers

Institute for Computational Engineering and Sciences (ICES) The University of Texas at Austin SIAM CSE 15 Salt Lake City

π-day, 2015

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 1 / 31

Big data, big (inverse) problems

An inverse problem: find parameters of a model given real

• bservations.

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 2 / 31

Big data, big (inverse) problems

An inverse problem: find parameters of a model given real

• bservations.

yobs is our (noisy) data vector,

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 2 / 31

Big data, big (inverse) problems

An inverse problem: find parameters of a model given real

• bservations.

yobs is our (noisy) data vector, ex. temperature measurements on a thermal fin ǫ ∼ N(0, Γ) yobs

j

:= w(xj) + ǫj, j = 1, . . . N

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 2 / 31

Big data, big (inverse) problems

An inverse problem: find parameters of a model given real

• bservations.

yobs is our (noisy) data vector, ex. temperature measurements on a thermal fin ǫ ∼ N(0, Γ) yobs

j

:= w(xj) + ǫj, j = 1, . . . N u is the parameter we want

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 2 / 31

Big data, big (inverse) problems

An inverse problem: find parameters of a model given real

• bservations.

yobs is our (noisy) data vector, ex. temperature measurements on a thermal fin ǫ ∼ N(0, Γ) yobs

j

:= w(xj) + ǫj, j = 1, . . . N u is the parameter we want Physics model: −∇ · (eu∇w) = 0 in Ω −eu∇w · n = Bi w

• n ∂Ω \ ΓR,

−eu∇w · n = −1

• n ΓR,

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 2 / 31

Big data, big (inverse) problems

An inverse problem: find parameters of a model given real

• bservations.

yobs is our (noisy) data vector, ex. temperature measurements on a thermal fin ǫ ∼ N(0, Γ) yobs

j

:= w(xj) + ǫj, j = 1, . . . N u is the parameter we want Physics model: −∇ · (eu∇w) = 0 in Ω −eu∇w · n = Bi w

• n ∂Ω \ ΓR,

−eu∇w · n = −1

• n ΓR,

     ⇒ G (u) = w, Our forward map

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 2 / 31

Big data, big (inverse/optimization) problems

Minimize the cost.

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 3 / 31

Big data, big (inverse/optimization) problems

Minimize the cost. Cost is J

• u; yobs, u0
• s.t.

−∇ · (eu∇w) = 0 in Ω −eu∇w · n = Bi w

• n ∂Ω \ ΓR,

−eu∇w · n = −1

• n ΓR,

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 3 / 31

Big data, big (inverse/optimization) problems

Minimize the cost. Cost is J

• u; yobs, u0
• := 1

2

• yobs − G (u)
• 2

Γ

• data misfit

s.t. −∇ · (eu∇w) = 0 in Ω −eu∇w · n = Bi w

• n ∂Ω \ ΓR,

−eu∇w · n = −1

• n ΓR,

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 3 / 31

Big data, big (inverse/optimization) problems

Minimize the cost. Cost is J

• u; yobs, u0
• := 1

2

• yobs − G (u)
• 2

Γ

• data misfit

+ 1 2 u − u02

C

• some regularization

s.t. −∇ · (eu∇w) = 0 in Ω −eu∇w · n = Bi w

• n ∂Ω \ ΓR,

−eu∇w · n = −1

• n ΓR,

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 3 / 31

More data, higher numerical rank

−∇ · (eu∇w) = 0 in Ω −eu∇w · n = Bi w

• n ∂Ω \ ΓR,

−eu∇w · n = −1

• n ΓR,

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 4 / 31

More data, higher numerical rank

−∇ · (eu∇w) = 0 in Ω −eu∇w · n = Bi w

• n ∂Ω \ ΓR,

−eu∇w · n = −1

• n ΓR,

0.2 0.4 0.6 0.8 1 1 2 3 4 5 6

• bservation locations

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 4 / 31

More data, higher numerical rank

−∇ · (eu∇w) = 0 in Ω −eu∇w · n = Bi w

• n ∂Ω \ ΓR,

−eu∇w · n = −1

• n ΓR,

0.2 0.4 0.6 0.8 1 1 2 3 4 5 6

• bservation locations

10 20 30 40 50 60 10

−4

10

−2

10 10

2

Misfit Hessian singular values index 2001 data 993 data 497 data 249 data 125 data 63 data Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 4 / 31

More data, higher numerical rank

−∇ · (eu∇w) = 0 in Ω −eu∇w · n = Bi w

• n ∂Ω \ ΓR,

−eu∇w · n = −1

• n ΓR,

0.2 0.4 0.6 0.8 1 1 2 3 4 5 6

• bservation locations

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 5 / 31

More data, higher numerical rank

−∇ · (eu∇w) = 0 in Ω −eu∇w · n = Bi w

• n ∂Ω \ ΓR,

−eu∇w · n = −1

• n ΓR,

0.2 0.4 0.6 0.8 1 1 2 3 4 5 6

• bservation locations

10 20 30 40 50 60 10

−2

10

−1

10 10

1

10

2

Misfit Hessian singular values index 2001 data 1001 data 501 data 251 data 126 data 63 data Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 5 / 31

Big data issues in large-scale inverse problems

Big data issues

1

More data

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 6 / 31

Big data issues in large-scale inverse problems

Big data issues

1

More data → higher rank

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 6 / 31

Big data issues in large-scale inverse problems

Big data issues

1

More data → higher rank → more PDE solves

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 6 / 31

Big data issues in large-scale inverse problems

Big data issues

1

More data → higher rank → more PDE solves → more $$$

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 6 / 31

Big data issues in large-scale inverse problems

Big data issues

1

More data → higher rank → more PDE solves → more $$$

2

There’s a lot of redundancy in big data

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 6 / 31

Big data issues in large-scale inverse problems

Big data issues

1

More data → higher rank → more PDE solves → more $$$

2

There’s a lot of redundancy in big data

3

Furthermore, carrying big data along (I/O, data moving, etc) large-scale inversion is costly

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 6 / 31

Big data issues in large-scale inverse problems

Big data issues

1

More data → higher rank → more PDE solves → more $$$

2

There’s a lot of redundancy in big data

3

Furthermore, carrying big data along (I/O, data moving, etc) large-scale inversion is costly

Challenge

How to reduce the cost for big-data-meets-big-inverse-problems?

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 6 / 31

Big data issues in large-scale inverse problems

Big data issues

1

More data → higher rank → more PDE solves → more $$$

2

There’s a lot of redundancy in big data

3

Furthermore, carrying big data along (I/O, data moving, etc) large-scale inversion is costly

Challenge

How to reduce the cost for big-data-meets-big-inverse-problems? An answer: reduce the amount of data.

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 6 / 31

Big data issues in large-scale inverse problems

Big data issues

1

More data → higher rank → more PDE solves → more $$$

2

There’s a lot of redundancy in big data

3

Furthermore, carrying big data along (I/O, data moving, etc) large-scale inversion is costly

Challenge

How to reduce the cost for big-data-meets-big-inverse-problems? An answer: reduce the amount of data. BUT HOW?

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 6 / 31

A Randomized Likelihood Method for Big Data

misfit = 1 2

• Γ− 1

2

• yobs − G (u)
• 2

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 7 / 31

A Randomized Likelihood Method for Big Data

misfit = 1 2

• Γ− 1

2

• yobs − G (u)
• 2

1 2

• Eε
• εεT

Γ− 1

2

• yobs − G (u)
• 2

, e.g. ε ∼ N (0, I)

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 7 / 31

A Randomized Likelihood Method for Big Data

misfit = 1 2

• Γ− 1

2

• yobs − G (u)
• 2

= 1 2

• Eε
• εεT

Γ− 1

2

• yobs − G (u)
• 2

, e.g. ε ∼ N (0, I)

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 7 / 31

A Randomized Likelihood Method for Big Data

misfit = 1 2

• Γ− 1

2

• yobs − G (u)
• 2

= 1 2

• Eε
• εεT

Γ− 1

2

• yobs − G (u)
• 2

, e.g. ε ∼ N (0, I) = 1 2Eε

• εTΓ− 1

2

• yobs − G (u)

2

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 7 / 31

A Randomized Likelihood Method for Big Data

misfit = 1 2

• Γ− 1

2

• yobs − G (u)
• 2

= 1 2

• Eε
• εεT

Γ− 1

2

• yobs − G (u)
• 2

, e.g. ε ∼ N (0, I) = 1 2Eε

• εTΓ− 1

2

• yobs − G (u)

2 Thus, the cost function becomes J = 1 2Eε

• εTΓ− 1

2

• yobs − G (u)

2 + 1 2 u − u02

C

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 7 / 31

Randomized Likelihood Method for Big Data

J =

1 2

• Γ− 1

2

yobs − G (u)

• 2

+ 1

2 u − u02 C

= 1 2Eε

• εTΓ− 1

2

• yobs − G (u)

2 + 1 2 u − u02

C

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 8 / 31

Randomized Likelihood Method for Big Data

J =

1 2

• Γ− 1

2

yobs − G (u)

• 2

+ 1

2 u − u02 C

= 1 2Eε

• εTΓ− 1

2

• yobs − G (u)

2 + 1 2 u − u02

C Monte Carlo

≈ 1 2N

N

• j=1
• εT

j Γ− 1

2

• yobs − G (u)

2 + 1 2 u − u02

C

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 8 / 31

Randomized Likelihood Method for Big Data

J =

1 2

• Γ− 1

2

yobs − G (u)

• 2

+ 1

2 u − u02 C

= 1 2Eε

• εTΓ− 1

2

• yobs − G (u)

2 + 1 2 u − u02

C Monte Carlo

≈ 1 2N

N

• j=1
• εT

j Γ− 1

2

• yobs − G (u)

2 + 1 2 u − u02

C

=

1 2

• ΣTΓ− 1

2

yobs − G (u)

• 2

+ 1

2 u − u02 C =: ˜

J where Σ := 1 √ N [ε1, . . . , εN] ∈ IRd×N

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 8 / 31

Randomized Likelihood Method for Big Data

J =

1 2

• Γ− 1

2

yobs − G (u)

• 2

+ 1

2 u − u02 C

= 1 2Eε

• εTΓ− 1

2

• yobs − G (u)

2 + 1 2 u − u02

C Monte Carlo

≈ 1 2N

N

• j=1
• εT

j Γ− 1

2

• yobs − G (u)

2 + 1 2 u − u02

C

=

1 2

• ΣTΓ− 1

2

yobs − G (u)

• 2

+ 1

2 u − u02 C =: ˜

J where Σ := 1 √ N [ε1, . . . , εN] ∈ IRd×N if N ≪ d ⇒ substantially reducing the data

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 8 / 31

Randomized Likelihood Method for Big Data

Numerical demonstration for 1D elliptic inverse problem

Plot J (κ) := J(u0 + κ∇J (u0)) and ˜ J (κ) := ˜ J(u0 + κ∇J (u0))

−10 −5 5 10

1

10

2

10

3

10

4

10

5

κ cost J J ˜ J, N=1 ˜ J, N=5 ˜ J, N=10 ˜ J, N=20

ε is Normal

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 9 / 31

Randomized Likelihood Method for Big Data

Numerical demonstration for 1D elliptic inverse problem

Plot J (κ) := J(u0 + κ∇J (u0)) and ˜ J (κ) := ˜ J(u0 + κ∇J (u0))

−10 −5 5 10

1

10

2

10

3

10

4

10

5

κ cost J J ˜ J, N=1 ˜ J, N=5 ˜ J, N=10 ˜ J, N=20

ε is Normal

−10 −5 5 10

2

10

3

10

4

10

5

κ cost J J ˜ J, N=1 ˜ J, N=5 ˜ J, N=10 ˜ J, N=20

ε is Rademacher

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 9 / 31

Randomized Likelihood Method for Big Data

Numerical demonstration for 1D elliptic inverse problem

Plot J (κ) := J(u0 + κ∇J (u0)) and ˜ J (κ) := ˜ J(u0 + κ∇J (u0))

−10 −5 5 10

1

10

2

10

3

10

4

10

5

κ cost J J ˜ J, N=1 ˜ J, N=5 ˜ J, N=10 ˜ J, N=20

ε is Normal

−10 −5 5 10

2

10

3

10

4

10

5

κ cost J J ˜ J, N=1 ˜ J, N=5 ˜ J, N=10 ˜ J, N=20

ε is Rademacher Even N = 1 random vector has a very good approximate minimizer

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 9 / 31

Randomized Likelihood Method for Big Data

Numerical demonstration for 1D elliptic inverse problem

Plot J (κ) := J(u0 + κ∇J (u0)) and ˜ J (κ) := ˜ J(u0 + κ∇J (u0))

−10 −5 5 10

1

10

2

10

3

10

4

10

5

κ cost J J ˜ J, N=1 ˜ J, N=5 ˜ J, N=10 ˜ J, N=20

Σ is sparse Achlioptas

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 10 / 31

Randomized Likelihood Method for Big Data

Numerical demonstration for 1D elliptic inverse problem

Plot J (κ) := J(u0 + κ∇J (u0)) and ˜ J (κ) := ˜ J(u0 + κ∇J (u0))

−10 −5 5 10

1

10

2

10

3

10

4

10

5

κ cost J J ˜ J, N=1 ˜ J, N=5 ˜ J, N=10 ˜ J, N=20

Σ is sparse Achlioptas

˜ J := 1 2

• ΣTΓ− 1

2

• yobs − G (u)
• 2

+ 1 2 u − u02

C Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 10 / 31

Randomized Likelihood Method for Big Data

Numerical results: MAP point of 1D Elliptic inverse problem

d = 1025 and N = 1

0.2 0.4 0.6 0.8 1 −2 −1.5 −1 −0.5 0.5 x MAP J ˜ J, N=1

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 11 / 31

Randomized Likelihood Method for Big Data

Numerical results: MAP point of 1D Elliptic inverse problem

d = 1025 and N = 2

0.2 0.4 0.6 0.8 1 −2 −1.5 −1 −0.5 0.5 x MAP J ˜ J, N=2

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 12 / 31

Randomized Likelihood Method for Big Data

Numerical results: MAP point of 1D Elliptic inverse problem

d = 1025 and N = 5

0.2 0.4 0.6 0.8 1 −2 −1.5 −1 −0.5 0.5 x MAP J ˜ J, N=5

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 13 / 31

Randomized Likelihood Method for Big Data

Numerical results: MAP point of 1D Elliptic inverse problem

d = 1025 and N = 10

0.2 0.4 0.6 0.8 1 −2 −1.5 −1 −0.5 0.5 x MAP J ˜ J, N=10

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 14 / 31

Randomized Likelihood Method for Big Data

Numerical results: MAP point of 1D Elliptic inverse problem

d = 1025 and N = 20

0.2 0.4 0.6 0.8 1 −2 −1.5 −1 −0.5 0.5 x MAP J ˜ J, N=20

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 15 / 31

Randomized Likelihood Method for Big Data

Numerical demonstration for 2D elliptic inverse problem

Plot J (κ) := J(u0 + κ∇J (u0)) and ˜ J (κ) := ˜ J(u0 + κ∇J (u0))

−5 5 10 x 10

−4

10

3

10

4

10

5

10

6

10

7

κ cost J J ˜ J, N=1 ˜ J, N=10 ˜ J, N=25 ˜ J, N=50 ˜ J, N=100

ε is Normal

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 16 / 31

Randomized Likelihood Method for Big Data

Numerical demonstration for 2D elliptic inverse problem

Plot J (κ) := J(u0 + κ∇J (u0)) and ˜ J (κ) := ˜ J(u0 + κ∇J (u0))

−5 5 10 x 10

−4

10

3

10

4

10

5

10

6

10

7

κ cost J J ˜ J, N=1 ˜ J, N=10 ˜ J, N=25 ˜ J, N=50 ˜ J, N=100

ε is Normal

−5 5 10 x 10

−4

10

3

10

4

10

5

10

6

10

7

κ cost J J ˜ J, N=1 ˜ J, N=10 ˜ J, N=25 ˜ J, N=50 ˜ J, N=100

ε is Rademacher

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 16 / 31

Randomized Likelihood Method for Big Data

Numerical demonstration for 2D elliptic inverse problem

Plot J (κ) := J(u0 + κ∇J (u0)) and ˜ J (κ) := ˜ J(u0 + κ∇J (u0))

−5 5 10 x 10

−4

10

3

10

4

10

5

10

6

10

7

κ cost J J ˜ J, N=1 ˜ J, N=10 ˜ J, N=25 ˜ J, N=50 ˜ J, N=100

ε is Normal

−5 5 10 x 10

−4

10

3

10

4

10

5

10

6

10

7

κ cost J J ˜ J, N=1 ˜ J, N=10 ˜ J, N=25 ˜ J, N=50 ˜ J, N=100

ε is Rademacher Even N = 1 random vector has a very good approximate minimizer

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 16 / 31

Randomized Likelihood Method for Big Data

Numerical demonstration for 2D elliptic inverse problem

Plot J (κ) := J(u0 + κ∇J (u0)) and ˜ J (κ) := ˜ J(u0 + κ∇J (u0))

−5 5 10 x 10

−4

10

3

10

4

10

5

10

6

10

7

κ cost J J ˜ J, N=1 ˜ J, N=10 ˜ J, N=25 ˜ J, N=50 ˜ J, N=100

ε is Normal

−5 5 10 x 10

−4

10

3

10

4

10

5

10

6

10

7

κ cost J J ˜ J, N=1 ˜ J, N=10 ˜ J, N=25 ˜ J, N=50 ˜ J, N=100

ε is Rademacher Even N = 1 random vector has a very good approximate minimizer Rademacher looks better

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 16 / 31

Randomized Likelihood Method for Big Data

Numerical demonstration for 2D elliptic inverse problem - using theory of random projection

Plot J (κ) := J(u0 + κ∇J (u0)) and ˜ J (κ) := ˜ J(u0 + κ∇J (u0))

−5 5 10 x 10

−4

10

2

10

3

10

4

10

5

10

6

κ cost J J ˜ J, N=1 ˜ J, N=10 ˜ J, N=25 ˜ J, N=50 ˜ J, N=100

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 17 / 31

Randomized Likelihood Method for Big Data

Numerical demonstration for 2D elliptic inverse problem - using theory of random projection

Plot J (κ) := J(u0 + κ∇J (u0)) and ˜ J (κ) := ˜ J(u0 + κ∇J (u0))

−5 5 10 x 10

−4

10

2

10

3

10

4

10

5

10

6

κ cost J J ˜ J, N=1 ˜ J, N=10 ˜ J, N=25 ˜ J, N=50 ˜ J, N=100

Σ is sparse Achlioptas - preserves distances (aka is a JL transform)

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 17 / 31

Randomized Likelihood Method for Big Data

Numerical results for 2D elliptic inverse problem

˜ uMAP = arg min

u

˜ J N = 31

−0.05 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

uMAP = arg min

u

J d = 1333

−0.05 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 18 / 31

Randomized Likelihood Method for Big Data

Numerical results for 2D elliptic inverse problem

˜ uMAP = arg min

u

˜ J N = 51

−0.05 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

uMAP = arg min

u

J d = 1333

−0.05 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 19 / 31

Randomized Likelihood Method for Big Data

Numerical results for 2D elliptic inverse problem

˜ uMAP = arg min

u

˜ J N = 101

−0.05 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

uMAP = arg min

u

J d = 1333

−0.05 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 20 / 31

Randomized Likelihood Method for Big Data

Numerical results for 2D elliptic inverse problem

˜ uMAP = arg min

u

˜ J N = 301

−0.05 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

uMAP = arg min

u

J d = 1333

−0.05 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 21 / 31

Randomized Likelihood Method for Big Data

Numerical results for 2D elliptic inverse problem

˜ uMAP = arg min

u

˜ J N = 601

−0.05 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

uMAP = arg min

u

J d = 1333

−0.05 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 22 / 31

An Analysis of Randomized Likelihood Method

Define I (u, ε) :=

• εTΓ− 1

2

• yobs − G (u)

2 + 1 2 u − u02

C

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 23 / 31

An Analysis of Randomized Likelihood Method

Define I (u, ε) :=

• εTΓ− 1

2

• yobs − G (u)

2 + 1 2 u − u02

C

˜ J (u, [ε1, . . . , εN]) = 1 N

N

• j=1

I

• u, εj
• and J (u) = Eε [I (u, ε)]

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 23 / 31

An Analysis of Randomized Likelihood Method

Define I (u, ε) :=

• εTΓ− 1

2

• yobs − G (u)

2 + 1 2 u − u02

C

˜ J (u, [ε1, . . . , εN]) = 1 N

N

• j=1

I

• u, εj
• and J (u) = Eε [I (u, ε)]

Lemma (Uniform law of large numbers for ˜ J)

Suppose u ∈ U, a compact subset of IRp, and let I (u, ε) : U × IRd → IR be continuous in u for each ε and measurable in ε for each u. Assume Eε sup

u∈U

|I (u, ε)| < ∞, then, as N → ∞ sup

u∈U

• ˜

J (u, [ε1, . . . , εN]) − J (u)

• a.s.

→ 0

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 23 / 31

An Analysis of Randomized Likelihood Method

Define I (u, ε) :=

• εTΓ− 1

2

• yobs − G (u)

2 + 1 2 u − u02

C

˜ J (u, [ε1, . . . , εN]) = 1 N

N

• j=1

I

• u, εj
• and J (u) = Eε [I (u, ε)]

Theorem (Convergence of the randomized likelihood optimizers)

Moreover, assume that J (u) : U → IR is continuous and uMAP = arg min

u

J (u) is the unique minimizer of J (u), then ˜ uMAP = arg min

u

˜ J (u, [ε1, . . . , εN]) P → uMAP

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 24 / 31

But also...

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 25 / 31

But also...

We have bounds on our errors from recent work in random projections.

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 25 / 31

But also...

We have bounds on our errors from recent work in random projections. NOTICE: ˜ J := 1 2

• ΣTΓ− 1

2

• yobs − G (u)
• 2

+ 1 2 u − u02

C

Theorem (Convergence of minimizer of cost function with data misfit reduced by a JLT)

Suppose G (u) = Au i.e. is linear map A ∈ Rd×K. Let uMAP = arg min

u

b − Au, and ˜ uMAP = arg min

u

S(b − Au), where S ∈ RN×d is a Johnson-Lindenstrauss Transform. Then with high probability,

• uMAP − ˜

uMAP

• ≤

ǫ σmin(A)

• b − AuMAP
• , where

N ≥ K log K ǫ2

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 25 / 31

And

We also have other theoretical results from stochastic programming not mentioned in this talk for brevity.

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 26 / 31

Numerical results of convergence of optimizer: motivation for proving tighter bounds

• uMAP −

˜ uMAP

N

• plotted against 1/sqrt(N)

Σ is Normal Σ is Rademacher Σ is Achlioptas

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 27 / 31

Conclusions

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 28 / 31

Conclusions

We can view our method from either the stochastic approximation framework (Statistics) or from the random projection framework (CS).

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 28 / 31

Conclusions

We can view our method from either the stochastic approximation framework (Statistics) or from the random projection framework (CS). We can trade accuracy for efficiency (and therefore gain scalability) - in inverse problems we are limited by Mozorov anyway, so we are not interested in convergence to exact minimizer.

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 28 / 31

Future work

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 29 / 31

Future work

Connect stochastic programming and statistical theory and random projections/JL theory to get sharper bounds and a more complete analysis

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 29 / 31

Future work

Connect stochastic programming and statistical theory and random projections/JL theory to get sharper bounds and a more complete analysis Extend this deterministic problem to a Bayesian framework

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 29 / 31

Future work

Connect stochastic programming and statistical theory and random projections/JL theory to get sharper bounds and a more complete analysis Extend this deterministic problem to a Bayesian framework Test on 3D/large-scale data sets

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 29 / 31

Future work

Connect stochastic programming and statistical theory and random projections/JL theory to get sharper bounds and a more complete analysis Extend this deterministic problem to a Bayesian framework Test on 3D/large-scale data sets This research is supported by DOE grants DE-SC0010518 and DE-SC0011118. We are grateful for the support.

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 29 / 31

An Analysis of Randomized Likelihood Method

Related work

Krebs, J. R, Anderson, J. E., Hinkley, D., Neelamani, R., Lee, S., Baumstein, A., and Lacasse, M. Fast full-wavefield seismic inversion using encoded sources, Geophys., 74, pp. WCC177–WCC188, 2009. (Heuristic approach)

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 30 / 31

An Analysis of Randomized Likelihood Method

Related work

Krebs, J. R, Anderson, J. E., Hinkley, D., Neelamani, R., Lee, S., Baumstein, A., and Lacasse, M. Fast full-wavefield seismic inversion using encoded sources, Geophys., 74, pp. WCC177–WCC188, 2009. (Heuristic approach) Haber, E., Chung, M. , and Herrmann, F . An effective method for parameter estimation with PDE constrains with multiple right-hand sides, S ¯ IAM J. Optim., 22(3), pp. 739–757, 2012. (Make connection with stochastic programming to deal with multi-source problems)

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 30 / 31

An Analysis of Randomized Likelihood Method

Related work

Krebs, J. R, Anderson, J. E., Hinkley, D., Neelamani, R., Lee, S., Baumstein, A., and Lacasse, M. Fast full-wavefield seismic inversion using encoded sources, Geophys., 74, pp. WCC177–WCC188, 2009. (Heuristic approach) Haber, E., Chung, M. , and Herrmann, F . An effective method for parameter estimation with PDE constrains with multiple right-hand sides, S ¯ IAM J. Optim., 22(3), pp. 739–757, 2012. (Make connection with stochastic programming to deal with multi-source problems) Our work Deal with big data

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 30 / 31

An Analysis of Randomized Likelihood Method

Related work

Krebs, J. R, Anderson, J. E., Hinkley, D., Neelamani, R., Lee, S., Baumstein, A., and Lacasse, M. Fast full-wavefield seismic inversion using encoded sources, Geophys., 74, pp. WCC177–WCC188, 2009. (Heuristic approach) Haber, E., Chung, M. , and Herrmann, F . An effective method for parameter estimation with PDE constrains with multiple right-hand sides, S ¯ IAM J. Optim., 22(3), pp. 739–757, 2012. (Make connection with stochastic programming to deal with multi-source problems) Our work Deal with big data Provide theoretical analysis using statistical theory

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 30 / 31

Tan Bui-Thanh Ellen B. Le Aaron Myers ICES and Dept of Aero. Eng., UT Austin ICES, UT Austin ICES, UT Austin tanbui@ices.utexas.edu ellenle@ices.utexas.edu aaron@ices.utexas.edu

Questions? Please ask/email/talk to us!

Bui-Thanh, Le, Myers (ICES, UT Austin) Big data meets big inversions 31 / 31