Feature Grouping as a Stochastic Regularizer for High Dimensional - PowerPoint PPT Presentation

Feature Grouping as a Stochastic Regularizer for High Dimensional Structured Data Sergül Aydöre Bertrand Thirion Gaël Varoquaux (Stevens Ins3tute of Technology, USA) (INRIA, France) (INRIA, France) POSTER : Pacific Ballroom #121, 06/11, Tuesday

High Dimensional and Small-Sample Data Situations • Brain imaging, Genomics, Seismology, Astronomy, Chemistry, etc. A typical MEG equipment [BML2001] PET acquisition process wikipedia MRI Scanner and rs-fMRI time series acquisition [NVIDIA] Astronomy Astronomy Magazine, 2015 Seismology Genomics hKps://www.mapnagroup.com Integrative Genomics Viewer, 2012 POSTER : Pacific Ballroom #121, 06/11, Tuesday

Fitting Complex Models in These Situations Challenges 1. Large feature dimension : due to rich temporal and spatial resolution 2. Noise in the data : due to artifacts unrelated to the effect of interest 3. Small sample size : due to logistics and cost of data acquisition Regularization Strategies • Early Stopping : [Yao, 2007] • ℓ 𝟐 and ℓ 𝟑 penalties: [Tibshirami 1996] • Pooling Layers in CNNs: [Hinton 2012] • Group LASSO: [Yuan 2006] • Dropout: [Srivastana 2014] POSTER : Pacific Ballroom #121, 06/11, Tuesday

Fitting Complex Models in These Situations Challenges 1. Large feature dimension : due to rich temporal and spatial resolution 2. Noise in the data : due to artifacts unrelated to the effect of interest 3. Small sample size : due to logistics and cost of data acquisition Regularization Strategies • Early Stopping : [Yao, 2007] • ℓ 𝟐 and ℓ 𝟑 penalties: [Tibshirami 1996] • Pooling Layers in CNNs: [Hinton 2012]………………….. TRANSLATION INVARIANCE • Group LASSO: [Yuan 2006]…………………………………… STRUCTURE + SPARSITY • Dropout: [Srivastana 2014]…………………………………………….…. STOCHASTICITY POSTER : Pacific Ballroom #121, 06/11, Tuesday

Fitting Complex Models in These Situations Challenges 1. Large feature dimension : due to rich temporal and spatial resolution 2. Noise in the data : due to artifacts unrelated to the effect of interest 3. Small sample size : due to logistics and cost of data acquisition Regularization Strategies • Early Stopping : [Yao, 2007] • ℓ 𝟐 and ℓ 𝟑 penalties: [Tibshirami 1996] • Pooling Layers in CNNs: [Hinton 2012]………………….. TRANSLATION INVARIANCE • Group LASSO: [Yuan 2006]…………………………………… STRUCTURE + SPARSITY • Dropout: [Srivastana 2014]…………………………………………….…. STOCHASTICITY • PROPOSED : Use STRUCTURE & STOCHASTICITY POSTER : Pacific Ballroom #121, 06/11, Tuesday

<latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit> <latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit> <latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit> <latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit> <latexit sha1_base64="fFMxY1HAqzjxFw4rJL83tBDhvS8=">ADqXicnZJda9swFIYVex+d95Wul7s5LKwMBsF2UrqbQqE3u+hFCk0aFoUgy0oiKktGksuCyW/rf9hd/82U1P2Im8HoAcF73nP0SJZPkgtubBjeNDz/xctXr3feBG/fvf/wsbn7aWBUoSnrUyWUHibEMEl61tuBRvmpEsEewiuTxZ1S+umDZcyXO7yNk4IzPJp5wS6zJbuMaJ0qkGbHzEvfmfAlHADhMy7LxLma/14CJiKfk0kEmKbKmod8H8I7L4TN7N8JxsHjUgWLa/D4f3F13kZfBevU4J3n8rbBuzV497m8bfCDGvwAIMBMpvc/Z9Jshe1wHfBURJVoSp6k+YfnCpaZExaKogxoyjM7bgk2nIq2DLAhWE5oZdkxkZOSpIxMy7Xk7aEr85JYaq0W9LC2n28oySZMYscZ2rkTL12srcVhsVdvpjXHKZF5ZJenvQtBgFazGFlKuGbVi4QShmru7Ap0Tah1wx24R4jqn/xUDOJ21GnHZ93WcVw9xw76jL6gbyhCh+gY/UQ91EfU2/dOvb438L/7Z/7Q/3Xb6jWqPXtoI3z6F28VEs8=</latexit> <latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit> <latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit> <latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit> <latexit sha1_base64="/01EjcQ24nCKW+XD2jV2+sBlJ4E=">ACF3icbVC7TsMwFHV4lvIKMLJYVEhMUYKQYKxgYSyIPqQmVI7jtFYdO7IdpCrKX7DwKywMIMQKG3+D02aAlitZPjrnXt1zT5gyqrTrfltLyura+u1jfrm1vbOr231Eik5i0sWBC9kKkCKOctDXVjPRSVASMtINx1el3n0gUlHB7/QkJUGChpzGFCNtqIHt+KFgkZok5sv91ogW0Kc+gnSozDMb4v7fAx9TROiYFrAgd1wHXdacBF4FWiAqloD+8uPBM4SwjVmSKm+56Y6yJHUFDNS1P1MkRThMRqSvoEcmUVBPr2rgMeGiWAspHlcwyn7eyJHiSqtm87Sr5rXSvI/rZ/p+CLIKU8zTieLYozBrWAZUgwopJgzSYGICyp8QrxCEmEtYmybkLw5k9eBJ1Tx3Md7+as0bys4qiBQ3AEToAHzkETXIMWaAMHsEzeAVv1pP1Yr1bH7PWJauaOQB/yvr8AXvMoBU=</latexit> Feature Grouping to Capture Structure Algorithm Training Data • ReNA: a data-driven, graph constrained feature Recursive Nearest AgglomeraJon (ReNA) grouping algorithm [Hoyos et al 2016] Iteration N • Each feature (pixel) is assigned to a cluster. Iteration 1 Iteration 2 Clusters are then recursively merged until the desired number of clusters remain. Number of clusters = 5 • Benefits of ReNA: (i) a fast clustering algorithm (ii) leads to good signal approximations. Reduction and Low-rank Approximation Φ ∈ R k × p Feature Grouping Matrix   0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 α 1 · · · α 1 0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 α 2 · · · α 2   Φ   Φ = 0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 α 3 · · · α 3     0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 α 4 · · · α 4   0 · · · 0 0 · · · 0 0 · · · 0 0 · · · 0 α 5 · · · α 5 Each row captures a different structure POSTER : Pacific Ballroom #121, 06/11, Tuesday

Proposed Approach Consider fully connected neural network with 𝑰 layers POSTER : Pacific Ballroom #121, 06/11, Tuesday

Proposed Approach Pre-compute a bank of feature grouping matrices POSTER : Pacific Ballroom #121, 06/11, Tuesday

Proposed Approach Sample from the training set POSTER : Pacific Ballroom #121, 06/11, Tuesday