recent developments of approximation theory and greedy algorithms - PowerPoint PPT Presentation

recent developments of approximation theory and greedy algorithms Peter Binev Department of Mathematics and Interdisciplinary Mathematics Institute University of South Carolina R educed O rder M odeling in G eneral R elativity Pasadena, CA June 6-7, 2013

Table of Contents Outline Greedy Algorithms Initial Remarks Greedy Bases Examples for Greedy Algorithms Tree Approximation Initial Setup Binary Partitions Near-Best Approximation Near-Best Tree Approximation Parameter Dependent PDEs Reduced Basis Method Kolmogorov Widths Results Robustness

Greedy Algorithms Initial Remarks Polynomial Approximations find the best L 2 -approximation via polynomials to a function in an interval [ a, b ] space X = L 2 [ a, b ] of functions: f ∈ X � � f � < ∞ ◮ � � � 1 2 | f ( x ) | 2 dx norm � f � = � f � X = � f � 2 := ◮ [ a,b ] basis ϕ 1 = 1 , ϕ 2 = x , ϕ 3 = x 2 , ... , ϕ n = x n − 1 , ... ◮ space of polynomials of degree n − 1 Φ n := span { ϕ 1 , ϕ 2 , ..., ϕ n } ◮ approximation p n := argmin � f − p � ◮ p ∈ Φ n n � representation p n = c n,j ( f ) ϕ j ◮ j =1 in general, the coefficients c n,j ( f ) are not easy to find ◮ in this case we can use orthogonality Hilbert spaces ◮ �

Greedy Algorithms Initial Remarks Polynomial Approximations find the best L 2 -approximation via polynomials to a function in a domain Ω space X = L 2 (Ω) of functions: f ∈ X � � f � < ∞ ◮ � � � 1 2 | f ( x ) | 2 dx norm � f � = � f � X = � f � 2 := ◮ Ω basis ϕ 1 , ϕ 2 , ϕ 3 , ... , ϕ n , ... ◮ space of polynomials of degree n Φ n := span { ϕ 1 , ϕ 2 , ..., ϕ n } ◮ approximation p n := argmin � f − p � ◮ p ∈ Φ n n � representation p n = c n,j ( f ) ϕ j ◮ j =1 in general, the coefficients c n,j ( f ) are not easy to find ◮ in this case we can use orthogonality Hilbert spaces ◮ �

Greedy Algorithms Initial Remarks Hilbert Space Setup Banach space X � normed linear space � � f � X ◮ Hilbert space H � Banach space with a scalar product � � f, g � ◮ � L 2 (Ω) is a Hilbert space � � f, g � := f ( x ) g ( x ) dx ◮ g - complex conjugation ¯ Ω � � 1 2 � f � := � f, f � (induced) norm ◮ orthogonality f ⊥ g � � f, g � = 0 ◮ n − 1 � orthogonal basis ψ n = ϕ n + q j ϕ j and ψ n ⊥ Φ n − 1 ◮ Gramm-Schmidt j =1 space of polynomials Φ n = span { ϕ 1 , ϕ 2 , ..., ϕ n } = span { ψ 1 , ψ 2 , ..., ψ n } ◮ n � representation p n = C j ( f ) ψ j := argmin � f − p � ◮ C j do not depend on n p ∈ Φ n j =1 n � C j ( f ) := � f, ψ j � � in case � ψ j � = 1 , we have p n = � f, ψ j � ψ j ◮ � ψ j , ψ j � j =1

Greedy Algorithms Initial Remarks Approximation in Hilbert Spaces � � ∞ orthonormal basis of H : ψ 1 , ψ 2 , ψ 3 , ... , ψ n , ... � ψ j ◮ � 0 j =1 � � ∞ if j � = k � ψ j , ψ k � = δ j,k := H = span ψ j � 1 if j = k j =1 n � linear approximation p n = � f, ψ j � ψ j ∈ Φ n ◮ j =1 ∞ � � � Parseval’s Identity � f − p n � 2 = � 2 � � f, ψ j � ∞ approximation error ◮ � f � 2 = � 2 � � � � � f, ψ j � j = n +1 j =1 n � nonlinear approximation g n = � f, ψ k j � ψ k j ◮ j =1 � � ⊂ I Λ n = Λ n − 1 ∪ { k n } index set Λ n = k 1 , k 2 , ..., k n N � ◮ How to find Λ n ? ◮ � � � � � � � � f, ψ k 1 � � ≥ � � f, ψ k 2 � � ≥ � � f, ψ k 3 � � ≥ ... �

Greedy Algorithms Initial Remarks Nonlinear Approximation � � ∞ given a basis ψ j j =1 , choose any n elements from it and form the linear n � combinations C j ψ k j j =1 � � � C k ψ k : Λ ⊂ I N, #Λ ≤ n approximation class (not a space!) Σ n := g = ◮ k ∈ Λ �� � � ∞ Σ n = Σ n ψ j can be defined for any basis in a Banach space X ◮ j =1 approximate f ∈ X via functions from Σ n ◮ best approximation σ n ( f ) := inf g ∈ Σ n � f − g � ◮ basic question: how to find g n ∈ Σ n such that � f − g n � ≤ Cσ n ? ◮ � � ∞ � � ∞ Note that although ψ j j =1 and ϕ j j =1 might yield the same polynomial spaces Φ n , it is usually the case that �� � ∞ � �� � ∞ � Σ n ψ j � = Σ n ϕ j for all n and even the rates of σ n can be completely different j =1 j =1

Greedy Algorithms Greedy Bases Greedy Approximation how to find efficiently g n ∈ Σ n that approximates f well? � � ∞ the case of an orthonormal basis ψ j j =1 in a Hilbert space X ◮ � incremental algorithm for finding g n = � f, ψ k � ψ k k ∈ Λ n Λ 0 = ∅ and Λ j = Λ j − 1 ∪ { k j } , where k j = argmax � f, ψ k � = argmax � f − g j − 1 , ψ k � k ∈ I N \ Λ j − 1 k ∈ I N for a general basis in a Banach space X , let f ∈ X has the ◮ ∞ � representation f = c j ( f ) ψ j j =1 � Greedy Algorithm : define g n = c k ( f ) ψ k k ∈ Λ n for Λ 0 = ∅ and Λ j = Λ j − 1 ∪ { k j } , where k j = argmax c k ( f ) k ∈ I N \ Λ j − 1 Note that in the general case g n is no longer the best approximation from Σ n to f

Greedy Algorithms Greedy Bases Greedy Basis the bases, for which g n is a good approximation � � ∞ Greedy Basis ψ j j =1 � for any f ∈ X the greedy approximation ◮ g n = g n ( f ) to f satisfies � f − g n � ≤ Gσ n ( f ) with a constant G independent on f and n � � ∞ � � ∞ Unconditional Basis ψ j j =1 � for any sign sequence θ j j =1 , ◮ � ∞ � ∞ � � θ j = ± 1 , the operator M θ defined by M θ a j ψ j = θ j a j ψ j j =1 j =1 is bounded � � ∞ Democratic Basis ψ j j =1 � there exists a constant D such that for ◮ any two finite sets of indeces P and Q with the same cardinality � � � � ∞ ∞ � � � � � � # P = # Q we have ψ k � ≤ D ψ k � � � k ∈ P k ∈ Q Theorem [Konyagin, Temlyakov] A basis is greedy if and only if it is unconditional and democratic.

Greedy Algorithms Greedy Bases Weak Greedy Algorithm often it is difficult (or even impossible) to find the maximizing element ψ k ◮ settle for an element which is at least γ times the best with 0 < γ ≤ 1 ◮ � define g n ( f ) := c k ( f ) ψ k ◮ k ∈ Λ n for Λ 0 = ∅ and Λ j = Λ j − 1 ∪ { k j } , where c k j ( f ) ≥ γ max N \ Λ j − 1 c k ( f ) k ∈ I Theorem [Konyagin, Temlyakov] For any greedy basis of a Banach space X and any γ ∈ (0 , 1] there is a basis-specific constant C ( γ ) , independent on f and n , such that � f − g n ( f ) � ≤ C ( γ ) σ n ( f )

Greedy Algorithms Examples for Greedy Algorithms General Greedy Strategy start with g 0 = 0 and Λ 0 = ∅ ◮ set j = 1 and loop through the next items ◮ analyze the element f − g 0 with the possible improvements related to ◮ � � ∞ (some of) the elements from ψ k k =1 by calculating a decision functional λ j ( f − g j − 1 , ψ k ) for each possible ψ k • in tree approximation the number of possible elements is bounded by (a multiple of) j • in the classical settings λ j is usually related to k,C � f − g j − 1 − Cψ k � inf be greedy, use the element ψ k j with the largest λ j or at least the one, ◮ for which λ j ( f − g j − 1 , ψ k j ) ≥ γ sup λ j ( f − g j − 1 , ψ k ) k set Λ j = Λ j − 1 ∪ { k j } ◮ � � calculate the next approximation g j based on ψ k ◮ k ∈ Λ j • in the classical settings g j is found in the form g j − 1 + Cψ kj set j := j + 1 and continue the loop ◮

Greedy Algorithms Examples for Greedy Algorithms Greedy Algorithms for Dictionaries in Hilbert Spaces � � ∞ ψ k k =1 is a dictionary (not a basis!) in a Hilbert space with � ψ k � = 1 Pure Greedy Algorithm ◮ k j := argmax |� f − g j − 1 , ψ k �| and g j = g j − 1 + � f − g j − 1 , ψ k j � ψ k j k

Greedy Algorithms Examples for Greedy Algorithms Greedy Algorithms for Dictionaries in Hilbert Spaces � � ∞ ψ k k =1 is a dictionary (not a basis!) in a Hilbert space with � ψ k � = 1 Pure Greedy Algorithm ◮ k j := argmax |� f − g j − 1 , ψ k �| and g j = g j − 1 + � f − g j − 1 , ψ k j � ψ k j k Orthogonal Greedy Algorithm ◮ |� f − g j − 1 , ψ k �| and g j = P { ψ k } k ∈ Λ j f k j := argmax k where P Ψ f is the orthogonal projection of f on the space span { Ψ }

Greedy Algorithms Examples for Greedy Algorithms Greedy Algorithms for Dictionaries in Hilbert Spaces � � ∞ ψ k k =1 is a dictionary (not a basis!) in a Hilbert space with � ψ k � = 1 Pure Greedy Algorithm ◮ k j := argmax |� f − g j − 1 , ψ k �| and g j = g j − 1 + � f − g j − 1 , ψ k j � ψ k j k Orthogonal Greedy Algorithm ◮ |� f − g j − 1 , ψ k �| and g j = P { ψ k } k ∈ Λ j f k j := argmax k where P Ψ f is the orthogonal projection of f on the space span { Ψ } Weak Greedy Algorithm with 0 < γ ≤ 1 ◮ |� f − g j − 1 , ψ k j �| ≥ γ sup |� f − g j − 1 , ψ k �| k and g j = g j − 1 + � f − g j − 1 , ψ k j � ψ k j