Feature Grouping as a Stochastic Regularizer for High Dimensional - - PowerPoint PPT Presentation

▶
feature grouping as a stochastic regularizer for high
SMART_READER_LITE
LIVE PREVIEW

Feature Grouping as a Stochastic Regularizer for High Dimensional - - PowerPoint PPT Presentation

Sep 12, 2022 122 likes •281 views

Feature Grouping as a Stochastic Regularizer for High Dimensional Structured Data Sergl Aydre Bertrand Thirion Gal Varoquaux (Stevens Ins3tute of Technology, USA) (INRIA, France) (INRIA, France) POSTER : Pacific Ballroom #121, 06/11,

slide-1
SLIDE 1

Feature Grouping as a Stochastic Regularizer for High Dimensional Structured Data

Bertrand Thirion

(INRIA, France)

Gaël Varoquaux

(INRIA, France)

Sergül Aydöre

(Stevens Ins3tute of Technology, USA)

POSTER: Pacific Ballroom #121, 06/11, Tuesday

slide-2
SLIDE 2
  • Brain imaging, Genomics, Seismology, Astronomy, Chemistry, etc.

High Dimensional and Small-Sample Data Situations

PET acquisition process wikipedia MRI Scanner and rs-fMRI time series acquisition [NVIDIA] Genomics

Integrative Genomics Viewer, 2012

Seismology

hKps://www.mapnagroup.com

Astronomy

Astronomy Magazine, 2015

POSTER: Pacific Ballroom #121, 06/11, Tuesday

A typical MEG equipment [BML2001]

slide-3
SLIDE 3

Fitting Complex Models in These Situations

  • 1. Large feature dimension: due to rich temporal and spatial resolution
  • 2. Noise in the data: due to artifacts unrelated to the effect of interest
  • 3. Small sample size: due to logistics and cost of data acquisition

Challenges Regularization Strategies

  • Early Stopping: [Yao, 2007]
  • ℓ𝟐 and ℓ𝟑 penalties: [Tibshirami 1996]
  • Pooling Layers in CNNs: [Hinton 2012]
  • Group LASSO: [Yuan 2006]
  • Dropout: [Srivastana 2014]

POSTER: Pacific Ballroom #121, 06/11, Tuesday

slide-4
SLIDE 4

Fitting Complex Models in These Situations

  • 1. Large feature dimension: due to rich temporal and spatial resolution
  • 2. Noise in the data: due to artifacts unrelated to the effect of interest
  • 3. Small sample size: due to logistics and cost of data acquisition

Challenges Regularization Strategies

  • Early Stopping: [Yao, 2007]
  • ℓ𝟐 and ℓ𝟑 penalties: [Tibshirami 1996]
  • Pooling Layers in CNNs: [Hinton 2012]………………….. TRANSLATION INVARIANCE
  • Group LASSO: [Yuan 2006]…………………………………… STRUCTURE + SPARSITY
  • Dropout: [Srivastana 2014]…………………………………………….…. STOCHASTICITY

POSTER: Pacific Ballroom #121, 06/11, Tuesday

slide-5
SLIDE 5

Fitting Complex Models in These Situations

  • 1. Large feature dimension: due to rich temporal and spatial resolution
  • 2. Noise in the data: due to artifacts unrelated to the effect of interest
  • 3. Small sample size: due to logistics and cost of data acquisition

Challenges Regularization Strategies

  • Early Stopping: [Yao, 2007]
  • ℓ𝟐 and ℓ𝟑 penalties: [Tibshirami 1996]
  • Pooling Layers in CNNs: [Hinton 2012]………………….. TRANSLATION INVARIANCE
  • Group LASSO: [Yuan 2006]…………………………………… STRUCTURE + SPARSITY
  • Dropout: [Srivastana 2014]…………………………………………….…. STOCHASTICITY
  • PROPOSED: Use STRUCTURE & STOCHASTICITY

POSTER: Pacific Ballroom #121, 06/11, Tuesday

slide-6
SLIDE 6

Feature Grouping to Capture Structure

POSTER: Pacific Ballroom #121, 06/11, Tuesday

Training Data Recursive Nearest AgglomeraJon (ReNA) [Hoyos et al 2016] Iteration 2 Iteration 1 Iteration N

Number of clusters = 5

  • ReNA: a data-driven, graph constrained feature

grouping algorithm

  • Each feature (pixel) is assigned to a cluster.

Clusters are then recursively merged until the desired number of clusters remain.

  • Benefits of ReNA: (i) a fast clustering algorithm

(ii) leads to good signal approximations. Algorithm

Φ ∈ Rk×p

<latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit><latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit><latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit><latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit>

Φ =       α1 · · · α1 0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 α2 · · · α2 0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 α3 · · · α3 0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 α4 · · · α4 0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 α5 · · · α5      

<latexit sha1_base64="fFMxY1HAqzjxFw4rJL83tBDhvS8=">ADqXicnZJda9swFIYVex+d95Wul7s5LKwMBsF2UrqbQqE3u+hFCk0aFoUgy0oiKktGksuCyW/rf9hd/82U1P2Im8HoAcF73nP0SJZPkgtubBjeNDz/xctXr3feBG/fvf/wsbn7aWBUoSnrUyWUHibEMEl61tuBRvmpEsEewiuTxZ1S+umDZcyXO7yNk4IzPJp5wS6zJbuMaJ0qkGbHzEvfmfAlHADhMy7LxLma/14CJiKfk0kEmKbKmod8H8I7L4TN7N8JxsHjUgWLa/D4f3F13kZfBevU4J3n8rbBuzV497m8bfCDGvwAIMBMpvc/Z9Jshe1wHfBURJVoSp6k+YfnCpaZExaKogxoyjM7bgk2nIq2DLAhWE5oZdkxkZOSpIxMy7Xk7aEr85JYaq0W9LC2n28oySZMYscZ2rkTL12srcVhsVdvpjXHKZF5ZJenvQtBgFazGFlKuGbVi4QShmru7Ap0Tah1wx24R4jqn/xUDOJ21GnHZ93WcVw9xw76jL6gbyhCh+gY/UQ91EfU2/dOvb438L/7Z/7Q/3Xb6jWqPXtoI3z6F28VEs8=</latexit>

Φ

<latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit><latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit><latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit><latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit>

Feature Grouping Matrix

Each row captures a different structure

Reduction and Low-rank Approximation

slide-7
SLIDE 7

Proposed Approach

POSTER: Pacific Ballroom #121, 06/11, Tuesday Consider fully connected neural network with 𝑰 layers

slide-8
SLIDE 8

Proposed Approach

POSTER: Pacific Ballroom #121, 06/11, Tuesday Pre-compute a bank of feature grouping matrices

slide-9
SLIDE 9

Proposed Approach

POSTER: Pacific Ballroom #121, 06/11, Tuesday Sample from the training set

slide-10
SLIDE 10

Proposed Approach

POSTER: Pacific Ballroom #121, 06/11, Tuesday Sample from the bank of feature grouping matrices

Φ

<latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit><latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit><latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit><latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit>
slide-11
SLIDE 11

Proposed Approach

POSTER: Pacific Ballroom #121, 06/11, Tuesday Re-define parameter space and project input onto lower dimensional space Forward Propagation

slide-12
SLIDE 12

Proposed Approach

POSTER: Pacific Ballroom #121, 06/11, Tuesday Apply back propagation Back Propagation

slide-13
SLIDE 13

Proposed Approach

POSTER: Pacific Ballroom #121, 06/11, Tuesday Update parameters

To update , project gradients back to the

  • riginal space.

W0

<latexit sha1_base64="7h/gwNMsHiXm01r2cU3YjWrPJNU=">AB83icbVDLSsNAFL3xWeur6tLNYBFclaQKuiy6cVnBPqApZTKdtEMnkzBzI5TQ3DjQhG3/ow7/8ZJm4W2Hhg4nHMv98wJEikMu63s7a+sbm1Xdop7+7tHxWjo7bJk414y0Wy1h3A2q4FIq3UKDk3URzGgWSd4LJXe53nrg2IlaPOE14P6IjJULBKFrJ9yOK4yDMOrOBO6hU3Zo7B1klXkGqUKA5qHz5w5ilEVfIJDWm57kJ9jOqUTDJZ2U/NTyhbEJHvGepohE3/WyeUbOrTIkYaztU0jm6u+NjEbGTKPATuYZzbKXi/95vRTDm34mVJIiV2xKEwlwZjkBZCh0JyhnFpCmRY2K2FjqilDW1PZluAtf3mVtOs17JWf7iqNm6LOkpwCmdwAR5cQwPuoQktYJDAM7zCm5M6L86787EYXOKnRP4A+fzB/e0kaI=</latexit>

Other terms are updated in a standard way.

slide-14
SLIDE 14

Experimental Results

POSTER: Pacific Ballroom #121, 06/11, Tuesday

Feature Grouping performs best as the sample size decreases

Performance in terms of sample size for fMRI data Performance in terms of computa3on 3me for OliveZ Faces

Feature Grouping is computaJonally efficient and robust to noise

Noisy Settings Small-sample Settings

slide-15
SLIDE 15

Thank You!

Visit our POSTER TODAY at Pacific Ballroom #121!