Approximation and Non-parametric Estimation of ResNet-type - - PowerPoint PPT Presentation

Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networks

Kenta Oono1,2 Taiji Suzuki1,3

Oono and Suzuki, Jun 13th #77 1

Poster: 13th June, Pacific Ballroom #77

↑Paper Link

{kenta_oono, taiji}@mist.i.u-tokyo.ac.jp

• 1. The University of Tokyo 2. Preferred Networks, Inc. 3. RIKEN AIP

Thirty-sixth International Conference on Machine Learning (ICML 2019) June 13th 2019, Long Beach, CA, U.S.A.

Key Takeaway

• Q. Why ResNet-type CNNs work well?

Oono and Suzuki, Jun 13th #77 2

Key Takeaway

• A. Hidden sparse structure promotes

good performance.

• Q. Why ResNet-type CNNs work well?

Oono and Suzuki, Jun 13th #77 3

Problem Setting

Oono and Suzuki, Jun 13th #77 4

We consider a non-parametric regression problem:

! = #°(&) + )

#°: True function (e.g., Hölder, Barron, Besov class), ): Gaussian noise

Problem Setting

Oono and Suzuki, Jun 13th #77 5

Given ! i.i.d. samples, we pick an estimator " # from the hypothesis class ℱ, which is a set of functions realized by CNNs with a specified architecture.

% = #°()) + ,

#°: True function (e.g., Hölder, Barron, Besov class), ,: Gaussian noise

We consider a non-parametric regression problem:

Problem Setting

Oono and Suzuki, Jun 13th #77 6

! = #°(&) + )

#°: True function (e.g., Hölder, Barron, Besov class), ): Gaussian noise

Goal: Evaluate the estimation error ℛ + # ∶= -& | + #(&) − #°(&)|0

We consider a non-parametric regression problem: Given 1 i.i.d. samples, we pick an estimator + # from the hypothesis class ℱ, which is a set of functions realized by CNNs with a specified architecture.

Prior Work

ℛ " # ≾ inf(∈ℱ ∥ # − #° ∥.

/ + 1

2(4ℱ/6) Approximation Error Model Complexity

Oono and Suzuki, Jun 13th #77 7

6: Sample size ℱ: Set of functions realizable by CNNs with a specified architecture #°: True function (e.g., Hölder, Barron, Besov etc.) 1 2(9): 2-notation ignoring logarithmic terms.

CNN type Parameter Size !ℱ Minimax Optimality Discrete Optimization General # of all weights Sub-optimal L

• Prior Work

ℛ $ % ≾ inf*∈ℱ ∥ % − %° ∥/

0 + 2

3(!ℱ/6) Approximation Error Model Complexity

Oono and Suzuki, Jun 13th #77 8

6: Sample size ℱ: Set of functions realizable by CNNs with a specified architecture %°: True function (e.g., Hölder, Barron, Besov etc.) 2 3(9): 3-notation ignoring logarithmic terms.

CNN type Parameter Size !ℱ Minimax Optimality Discrete Optimization General # of all weights Sub-optimal L

• Sparse*

# of non-zero weights Optimal J Needed L

Prior Work

ℛ $ % ≾ inf*∈ℱ ∥ % − %° ∥/

0 + 2

3(!ℱ/6) Approximation Error Model Complexity

* e.g., Hölder case: [Yarotsuky, 17; Schmidt-Hieber, 17; Petersen & Voigtlaender, 18]

Oono and Suzuki, Jun 13th #77 9

6: Sample size ℱ: Set of functions realizable by CNNs with a specified architecture %°: True function (e.g., Hölder, Barron, Besov etc.) 2 3(9): 3-notation ignoring logarithmic terms.

CNN type Parameter Size !ℱ Minimax Optimality Discrete Optimization General # of all weights Sub-optimal L

• Sparse*

# of non-zero weights Optimal J Needed L ResNet # of all weights Optimal J Not Needed J

Prior Work

ℛ $ % ≾ inf*∈ℱ ∥ % − %° ∥/

0 + 2

3(!ℱ/6) Approximation Error Model Complexity

* e.g., Hölder case: [Yarotsuky, 17; Schmidt-Hieber, 17; Petersen & Voigtlaender, 18]

Oono and Suzuki, Jun 13th #77 10

6: Sample size ℱ: Set of functions realizable by CNNs with a specified architecture %°: True function (e.g., Hölder, Barron, Besov etc.) 2 3(9): 3-notation ignoring logarithmic terms.

CNN type Parameter Size !ℱ Minimax Optimality Discrete Optimization General # of all weights Sub-optimal L

• Sparse*

# of non-zero weights Optimal J Needed L ResNet # of all weights Optimal J Not Needed J

Contribution

ResNet-type CNNs can achieve minimax-optimal rates without unrealistic constraints.

* e.g., Hölder case: [Yarotsuky, 17; Schmidt-Hieber, 17; Petersen & Voigtlaender, 18]

Oono and Suzuki, Jun 13th #77 11

CNN type Parameter Size !ℱ Minimax Optimality Discrete Optimization General # of all weights Sub-optimal L

• Sparse*

# of non-zero weights Optimal J Needed L ResNet # of all weights Optimal J Not Needed J

Contribution

ResNet-type CNNs can achieve minimax-optimal rates without unrealistic constraints.

Known optimal FNNs have block-sparse structures Key Observation

* e.g., Hölder case: [Yarotsuky, 17; Schmidt-Hieber, 17; Petersen & Voigtlaender, 18]

Oono and Suzuki, Jun 13th #77 12

Block-sparse FNN

Oono and Suzuki, Jun 13th #77 13

b

FCσ

W1,b1

FCσ

WM,bM

wM

Forward

FNN ∶= %

&'( )

*&

+ FC&(.) − 1

w1

Block-sparse FNN

Oono and Suzuki, Jun 13th #77 14

b

FCσ

W1,b1

FCσ

WM,bM

w1

wM

Forward

FNN ∶= %

&'( )

*&

+ FC&(.) − 1

Barron [Klusowski & Barron, 18] Hölder [Yarotsky, 17; Schmidt-Hieber, 17] Besov [Suzuki, 19]. Known best approximating FNNs are block-sparse when the true function is ---

Block-sparse FNN to ResNet-type CNN

Oono and Suzuki, Jun 13th #77 15

b

FCσ

W1,b1

FCσ

WM,bM

wM

FNN ∶= %

&'( )

*&

+ FC&(.) − 1

+ +

id

id

FCid

W,b

Convσ

w1,b1

Convσ

wM,bM

CNN: = FC ∘ Conv) + id ∘ ⋯ ∘ (Conv( + id) ∘ ;

Barron [Klusowski & Barron, 18] Hölder [Yarotsky, 17; Schmidt-Hieber, 17] Besov [Suzuki, 19]. Transform w1

Known best approximating FNNs are block-sparse when the true function is ---

Block-sparse FNN to ResNet-type CNN

Oono and Suzuki, Jun 13th #77 16

b

FCσ

W1,b1

FCσ

WM,bM

wM

+ +

id

id

FCid

W,b

Convσ

w1,b1

Convσ

wM,bM

Barron [Klusowski & Barron, 18] Hölder [Yarotsky, 17; Schmidt-Hieber, 17] Besov [Suzuki, 19].

↑ Minimax Optimal

CNN: = FC ∘ Conv* + id ∘ ⋯ ∘ (Conv0 + id) ∘ 2

w1

Transform Known best approximating FNNs are block-sparse when the true function is ---

Block-sparse FNN to ResNet-type CNN

Oono and Suzuki, Jun 13th #77 17

b

FCσ

W1,b1

FCσ

WM,bM

wM

Barron [Klusowski & Barron, 18] Hölder [Yarotsky, 17; Schmidt-Hieber, 17] Besov [Suzuki, 19].

+ +

id

id

FCid

W,b

Convσ

w1,b1

Convσ

wM,bM

↑ Minimax Optimal ↑ Minimax Optimal, too !

w1

Transform Known best approximating FNNs are block-sparse when the true function is ---

For any block-sparse FNN with ! blocks, there exists a ResNet- type CNN with ! residual blocks which has "(!) more parameters and which is identical (as a function) to the FNN.

Oono and Suzuki, Jun 13th #77 18

Block-sparse FNN ResNet-type CNN

Theorem

Block-sparse FNN to ResNet-type CNN

b

FCσ

W1,b1

FCσ

WM,bM

w1

wM

+ + id

id

FCid

W,b

Convσ

w1,b1

Convσ

wM,bM

Optimality of ResNet-type CNNs

Oono and Suzuki, Jun 13th #77 19

Suppose the true function !° is #-Hölder. There exists a set of ResNet-type CNNs ℱ such that:

Theorem (e.g., Hölder Case)

Optimality of ResNet-type CNNs

Oono and Suzuki, Jun 13th #77 20

Suppose the true function !° is #-Hölder. There exists a set of ResNet-type CNNs ℱ such that:

• ℱ does NOT have sparse constraints
• the estimator %

! of ℱ achieves the minimax-optimal estimation error rate (up to log factors).

Theorem (e.g., Hölder Case)

Optimality of ResNet-type CNNs

Oono and Suzuki, Jun 13th #77 21

Suppose the true function !° is #-Hölder. There exists a set of ResNet-type CNNs ℱ such that:

• ℱ does NOT have sparse constraints
• the estimator %

! of ℱ achieves the minimax-optimal estimation error rate (up to log factors). J Minimax optimal ! J No discrete optimization !

Theorem (e.g., Hölder Case)

Optimality of ResNet-type CNNs

Oono and Suzuki, Jun 13th #77 22

J Minimax optimal ! J No discrete optimization !

• Using the same strategy, we can prove that ResNet-type CNNs can achieve the same

rate as FNNs for the Barron class etc.

• We remove unrealistic constraints on channels size, too (see the paper).

Note

Suppose the true function !° is #-Hölder. There exists a set of ResNet-type CNNs ℱ such that:

• ℱ does NOT have sparse constraints
• the estimator %

! of ℱ achieves the minimax-optimal estimation error rate (up to log factors).

Theorem (e.g., Hölder Case)

CNN type Parameter Size !ℱ Minimax Optimality Discrete Optimization General # of all weights Sub-optimal L

• Sparse*

# of non-zero weights Optimal J Needed L ResNet # of all weights Optimal J Not Needed J

Conclusion

ResNet-type CNNs can achieve minimax-optimal rates in several function classes without implausible constraints.

Oono and Suzuki, Jun 13th #77 23

Poster: 13th June, Pacific Ballroom #77

↑ Minimax Optimal ↑ Minimax Optimal, too ! ↑Paper Link

b

FCσ

W1,b1

FCσ

WM,bM

w1

wM

+ + id

id

FCid

W,b

Convσ

w1,b1

Convσ

wM,bM