Data driven regularization by projection Andrea Aspri Joint work - - PowerPoint PPT Presentation

logo.png

Data driven regularization by projection

Andrea Aspri

Joint work with Y. Korolev and O. Scherzer

Joint meeting Fudan University and RICAM

Shanghai & Linz - 10, June 2020

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Motivation

Given Au = y and yδ noisy measurements s.t. y − yδ ≤ δ. Goal: Given measurements yδ, reconstruct the unknown quantity u. Assume we have {ui, yi}i=1,··· ,n s.t. Aui = yi

Questions

How can we use these pairs in the reconstruction process? How can we use these pairs when A is not explicitly known?

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Learning an operator: is it possible?

Main goal: develop stable algorithms for finding u such that Au = y without explicit knowledge of A, but having only training pairs: {ui, yi}i=1,··· ,n s.t. Aui = yi noisy measurements yδ, s.t. y − yδ ≤ δ

Question

Is there a regularization method capable of learning a linear operator? Spoiler: Yes, there is...

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Learning an operator: is it possible?

Main goal: develop stable algorithms for finding u such that Au = y without explicit knowledge of A, but having only training pairs: {ui, yi}i=1,··· ,n s.t. Aui = yi noisy measurements yδ, s.t. y − yδ ≤ δ

Question

Is there a regularization method capable of learning a linear operator? Spoiler: Yes, there is...

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Terminology & Notation

ui are “training images” Un := Span{ui}i=1,··· ,n PUn orthogonal projection onto Un yi are “training data” Yn := Span{yi}i=1,··· ,n PYn orthogonal projection onto Yn u† solution to Au = y.

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Main assumptions

• 1. Operator

◮ A : U → Y, with U, Y Hilbert spaces. ◮ A is bounded, linear and injective (but A−1 is unbounded).

• 2. Data

◮ Linear independence of {ui}i=1,··· ,n, ∀n ∈ N. ◮ Uniform boundedness: ∃Cu > 0 s.t. ui ≤ Cu, ∀i ∈ N. ◮ Sequentiality: training pairs are nested, i.e.

{ui, y i}i=1,··· ,n+1 = {ui, y i}i=1,··· ,n ∪ {un+1, y n+1}, hence Un ⊂ Un+1 and Yn ⊂ Yn+1, ∀n ∈ N.

◮ Density: training images spaces are dense in U, i.e.,

n∈N Un = U.

Consequences: Training data y i are linearly independent and uniformly bounded as well; Training data spaces are dense in R(A).

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Main assumptions

• 1. Operator

◮ A : U → Y, with U, Y Hilbert spaces. ◮ A is bounded, linear and injective (but A−1 is unbounded).

• 2. Data

◮ Linear independence of {ui}i=1,··· ,n, ∀n ∈ N. ◮ Uniform boundedness: ∃Cu > 0 s.t. ui ≤ Cu, ∀i ∈ N. ◮ Sequentiality: training pairs are nested, i.e.

{ui, y i}i=1,··· ,n+1 = {ui, y i}i=1,··· ,n ∪ {un+1, y n+1}, hence Un ⊂ Un+1 and Yn ⊂ Yn+1, ∀n ∈ N.

◮ Density: training images spaces are dense in U, i.e.,

n∈N Un = U.

Consequences: Training data y i are linearly independent and uniformly bounded as well; Training data spaces are dense in R(A).

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Main assumptions

• 1. Operator

◮ A : U → Y, with U, Y Hilbert spaces. ◮ A is bounded, linear and injective (but A−1 is unbounded).

• 2. Data

◮ Linear independence of {ui}i=1,··· ,n, ∀n ∈ N. ◮ Uniform boundedness: ∃Cu > 0 s.t. ui ≤ Cu, ∀i ∈ N. ◮ Sequentiality: training pairs are nested, i.e.

{ui, y i}i=1,··· ,n+1 = {ui, y i}i=1,··· ,n ∪ {un+1, y n+1}, hence Un ⊂ Un+1 and Yn ⊂ Yn+1, ∀n ∈ N.

◮ Density: training images spaces are dense in U, i.e.,

n∈N Un = U.

Consequences: Training data y i are linearly independent and uniformly bounded as well; Training data spaces are dense in R(A).

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Regularization by projection

Approximate u† using the Minimum Norm Solution (MNS) of finite dimensional problems Least-Squares Proj. (1) APnu = y

Pn = orthogonal projection onto a finite dimensional space of U.

Dual Least-Squares Proj. (2) QnAu = Qny

Qn orthogonal projection onto a finite dimensional space of Y.

Our idea to use training pairs

Choose Pn = PUn. Choose Qn = PYn

It can be proven

MNS : uU

n = A−1PYny

In general uU

n u†.

It can be proven

MNS : uY

n = PA∗YnuU n

In this case uY

n → u†.

Engl, H. W. and Hanke, M. and Neubauer, A., Regularization of Inverse Problems, Springer (1996). Seidman T. I., Nonconvergence results for the application of least-squares estimation to ill-posed problems, J. Opt. Th.

• Appl. (1980).

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Regularization by projection

Approximate u† using the Minimum Norm Solution (MNS) of finite dimensional problems Least-Squares Proj. (1) APnu = y

Pn = orthogonal projection onto a finite dimensional space of U.

Dual Least-Squares Proj. (2) QnAu = Qny

Qn orthogonal projection onto a finite dimensional space of Y.

Our idea to use training pairs

Choose Pn = PUn. Choose Qn = PYn

It can be proven

MNS : uU

n = A−1PYny

In general uU

n u†.

It can be proven

MNS : uY

n = PA∗YnuU n

In this case uY

n → u†.

Engl, H. W. and Hanke, M. and Neubauer, A., Regularization of Inverse Problems, Springer (1996). Seidman T. I., Nonconvergence results for the application of least-squares estimation to ill-posed problems, J. Opt. Th.

• Appl. (1980).

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Regularization by projection

Approximate u† using the Minimum Norm Solution (MNS) of finite dimensional problems Least-Squares Proj. (1) APnu = y

Pn = orthogonal projection onto a finite dimensional space of U.

Dual Least-Squares Proj. (2) QnAu = Qny

Qn orthogonal projection onto a finite dimensional space of Y.

Our idea to use training pairs

Choose Pn = PUn. Choose Qn = PYn

It can be proven

MNS : uU

n = A−1PYny

In general uU

n u†.

It can be proven

MNS : uY

n = PA∗YnuU n

In this case uY

n → u†.

Engl, H. W. and Hanke, M. and Neubauer, A., Regularization of Inverse Problems, Springer (1996). Seidman T. I., Nonconvergence results for the application of least-squares estimation to ill-posed problems, J. Opt. Th.

• Appl. (1980).

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Regularization by projection

Approximate u† using the Minimum Norm Solution (MNS) of finite dimensional problems Least-Squares Proj. (1) APnu = y

Pn = orthogonal projection onto a finite dimensional space of U.

Dual Least-Squares Proj. (2) QnAu = Qny

Qn orthogonal projection onto a finite dimensional space of Y.

Our idea to use training pairs

Choose Pn = PUn. Choose Qn = PYn

It can be proven

MNS : uU

n = A−1PYny

In general uU

n u†.

It can be proven

MNS : uY

n = PA∗YnuU n

In this case uY

n → u†.

Engl, H. W. and Hanke, M. and Neubauer, A., Regularization of Inverse Problems, Springer (1996). Seidman T. I., Nonconvergence results for the application of least-squares estimation to ill-posed problems, J. Opt. Th.

• Appl. (1980).

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Regularization by projection

Approximate u† using the Minimum Norm Solution (MNS) of finite dimensional problems Least-Squares Proj. (1) APnu = y

Pn = orthogonal projection onto a finite dimensional space of U.

Dual Least-Squares Proj. (2) QnAu = Qny

Qn orthogonal projection onto a finite dimensional space of Y.

Our idea to use training pairs

Choose Pn = PUn. Choose Qn = PYn

It can be proven

MNS : uU

n = A−1PYny

In general uU

n u†.

It can be proven

MNS : uY

n = PA∗YnuU n

In this case uY

n → u†.

Engl, H. W. and Hanke, M. and Neubauer, A., Regularization of Inverse Problems, Springer (1996). Seidman T. I., Nonconvergence results for the application of least-squares estimation to ill-posed problems, J. Opt. Th.

• Appl. (1980).

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Regularization by projection

Approximate u† using the Minimum Norm Solution (MNS) of finite dimensional problems Least-Squares Proj. (1) APnu = y

Pn = orthogonal projection onto a finite dimensional space of U.

Dual Least-Squares Proj. (2) QnAu = Qny

Qn orthogonal projection onto a finite dimensional space of Y.

Our idea to use training pairs

Choose Pn = PUn. Choose Qn = PYn

It can be proven

MNS : uU

n = A−1PYny

In general uU

n u†.

It can be proven

MNS : uY

n = PA∗YnuU n

In this case uY

n → u†.

Engl, H. W. and Hanke, M. and Neubauer, A., Regularization of Inverse Problems, Springer (1996). Seidman T. I., Nonconvergence results for the application of least-squares estimation to ill-posed problems, J. Opt. Th.

• Appl. (1980).

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Projection onto Image Space (least-squares proj.)

We consider APUnu = y and MNS uU

n = A−1PYny.

Goal: give an explicit expression of uU

n in terms of training pairs only and

find conditions to guarantee convergence of uU

n to u†.

Key ingredient: the Gram-Schmidt orthonormalization procedure {yi}1=1,··· ,n

G−S

− − − → {¯ yi}i=1,··· ,n {¯ yi}i=1,··· ,n is a orthonormal basis of Yn. We identify ¯ ui s.t. A¯ ui = ¯ yi, for i = 1, · · · , n.

Remark: By Gram-Schmidt it is easy to verify that ¯ ui = ui − i−1

k=1(y i, ¯

y k)¯ uk y i − PYi−1y i , ∀i ∈ N

Explicit Representation

uU

n = A−1PYny = n i=1(y, ¯

yi)¯ ui.

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Projection onto Image Space (least-squares proj.)

We consider APUnu = y and MNS uU

n = A−1PYny.

Goal: give an explicit expression of uU

n in terms of training pairs only and

find conditions to guarantee convergence of uU

n to u†.

Key ingredient: the Gram-Schmidt orthonormalization procedure {yi}1=1,··· ,n

G−S

− − − → {¯ yi}i=1,··· ,n {¯ yi}i=1,··· ,n is a orthonormal basis of Yn. We identify ¯ ui s.t. A¯ ui = ¯ yi, for i = 1, · · · , n.

Remark: By Gram-Schmidt it is easy to verify that ¯ ui = ui − i−1

k=1(y i, ¯

y k)¯ uk y i − PYi−1y i , ∀i ∈ N

Explicit Representation

uU

n = A−1PYny = n i=1(y, ¯

yi)¯ ui.

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Question

What about convergence of uU

n ?

Need to require some restrictive assumptions on training pairs and exact data y. Weak convergence iff uU

n ≤ C1, C1 > 0, for all n ∈ N.

To introduce sufficients conditions on the data to guaranteee weak convergence we use Gram-Schmidt, {ui}

G−S

− − − → {ui} and Aui = y i. Then (not optimal) sufficient conditions are

◮ +∞

j=1 |(u†, uj)| < ∞

◮ for all i ∈ N, given PYny i = n

j=1 βi,n j y j, assume that n

• j=1

(βi,n

j )2 ≤ C.

Strong convergence: ∞

i=1 |(y, ¯

yi)|¯ ui < ∞.

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Question

What about convergence of uU

n ?

Need to require some restrictive assumptions on training pairs and exact data y. Weak convergence iff uU

n ≤ C1, C1 > 0, for all n ∈ N.

To introduce sufficients conditions on the data to guaranteee weak convergence we use Gram-Schmidt, {ui}

G−S

− − − → {ui} and Aui = y i. Then (not optimal) sufficient conditions are

◮ +∞

j=1 |(u†, uj)| < ∞

◮ for all i ∈ N, given PYny i = n

j=1 βi,n j y j, assume that n

• j=1

(βi,n

j )2 ≤ C.

Strong convergence: ∞

i=1 |(y, ¯

yi)|¯ ui < ∞.

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Noisy data

When we have yδ = y + ∆, with ∆ / ∈ R(A) and ∆ ≤ δ, we have uU

n = n

• i=1

(yδ, ¯ yi)¯ ui We cannot expect convergence (since A−1 is unbounded), in fact u† − uU

n = A−1(I − PYn)Au† − A−1PYn∆.

Moral

The number of training pairs plays the role of a regularisation parameter.

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Projection onto Data Space (dual least-squares proj.)

PYnAu = PYny (1) MNS : uY

n = PA∗YnuU n

(2) Main issue: to implement (2) we need the training pairs (vi, yi)i=1,··· ,n where vi = A∗yi, i = 1, · · · , n.

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Question

What happens when restrictive assumptions on the traning pairs are not satisfied? data of the adjoint operator are missing?

(A possible) Answer

Use a regularization of the projected problem.

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Variational regularization

Let’s start again from APUnu = y and {ui, yi}i=1,··· ,n. We study

Optimization problem

min

u∈U 1 2APUnu − yδ2 + αJ (u)

α= regularization parameter, J a regularization term. Key ingredient: the Gram-Schmidt orthonormalization procedure {ui}1=1,··· ,n

G−S

− − − → {ui}i=1,··· ,n − → yi := Aui Hence APUnu = n

i=1(u, ui)yi.

Analysis: existence of a minimiser and convergence rates comes from classical results.

Neubauer, A. and Scherzer, O., Finite-dimensional approximation of Tikhonov regularized solutions of nonlinear ill-posed problems, Num. Func. Anal. Opt. (1990). Pöschl, C., Resmerita, E. and Scherzer, O., Discretization of variational regularization in Banach spaces, Inv. Prob. (2010). Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Variational regularization

Let’s start again from APUnu = y and {ui, yi}i=1,··· ,n. We study

Optimization problem

min

u∈U 1 2APUnu − yδ2 + αJ (u)

α= regularization parameter, J a regularization term. Key ingredient: the Gram-Schmidt orthonormalization procedure {ui}1=1,··· ,n

G−S

− − − → {ui}i=1,··· ,n − → yi := Aui Hence APUnu = n

i=1(u, ui)yi.

Analysis: existence of a minimiser and convergence rates comes from classical results.

Neubauer, A. and Scherzer, O., Finite-dimensional approximation of Tikhonov regularized solutions of nonlinear ill-posed problems, Num. Func. Anal. Opt. (1990). Pöschl, C., Resmerita, E. and Scherzer, O., Discretization of variational regularization in Banach spaces, Inv. Prob. (2010). Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Tests - Setting

Forward operator: Radon transform Ru = y

Figure: An example of training pairs. Figure: Examples of triplets used for projection

• nto data space.

In variational regularization, we choose (for example) J = TV .

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Noise 1% (in yδ)

Original Least-Sq. Dual Least-Sq. Variational Regul.

Figure: Reconstruction using 1000 training pairs and with 1% of noise. Figure: Reconstruction using 2000 training pairs and with 1% of noise.

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Increasing the number of training pairs...

Original Least-Squares Dual Least-Squares

Figure: Reconstruction using 2000 training pairs and 1% of noise. Figure: Reconstruction using 5000 training pairs and 1% of noise.

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Increasing the number of training pairs...

Original Variational Regul. (data-driven) Variational Regul. (model-based)

Figure: Reconstruction using 7000 training pairs and 1% of noise in the data driven variational

regularization.

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Conclusions

We have seen: Model-based regularization theory can be extended to purely data driven setting; No numerical access to the forward operator (in the injective case); About the size of the traning set:

◮ variational regularization: having more training data is better; ◮ regularization by projection: size of training set as regularization

parameter − → too many training pairs comprise stability;

Regularization by projection is capable of learning linear (injective)

• perators.

Andrea Aspri (RICAM) Data driven regularization by projection

logo.png

Thank you for your attention

Andrea Aspri (RICAM) Data driven regularization by projection