Outline Background Sparse Coding Semi-supervised Learning with - - PowerPoint PPT Presentation

▶
outline
SMART_READER_LITE
LIVE PREVIEW

Outline Background Sparse Coding Semi-supervised Learning with - - PowerPoint PPT Presentation

Dec 10, 2022 227 likes •417 views

Sparse-coded Net Model and Applications Y. Gwon, M. Cha, W. Campbell, H.T. Kung, C. Dagli IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2016) September 16, 2016 This work is sponsored by the

slide-1
SLIDE 1
  • Y. Gwon, M. Cha, W. Campbell, H.T. Kung, C. Dagli

IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2016) September 16, 2016

Sparse-coded Net Model and Applications

This ¡work ¡is ¡sponsored ¡by ¡the ¡Defense ¡Advanced ¡Research ¡Projects ¡Agency ¡under ¡Air ¡Force ¡Contract ¡FA8721-­‑05-­‑C-­‑0002. ¡Opinions, ¡ interpretations, ¡conclusions, ¡and ¡recommendations ¡are ¡those ¡of ¡the ¡authors ¡and ¡are ¡not ¡necessarily ¡endorsed ¡by ¡the ¡United ¡States ¡Government. ¡

slide-2
SLIDE 2

2 MLSP 2016

  • Background – Sparse Coding
  • Semi-supervised Learning with Sparse Coding
  • Sparse-coded Net
  • Experimental Evaluation
  • Conclusions and Future Work

Outline

slide-3
SLIDE 3

3 MLSP 2016

Background: Sparse Coding

  • *

'%

  • *

&(

  • 0.5 *

%'"

  • ≈ 1.2 ×

+ 0.9 × + 0.5 ×

x y101 d101 y208 d208 y263 d263 X ¡≈ ¡D ¡ Y ¡ ¡

Sparse ¡ Coding ¡ Data ¡

X Y D

… … Feature ¡dic-onary ¡

  • Unsupervised method to learn representation of data

– Decompose data into sparse linear combination of learned basis vectors – Domain transform: raw data ⟶ feature vectors

slide-4
SLIDE 4

4 MLSP 2016

Background: Sparse Coding (cont.) min{D,y} ǁ‗x – Dyǁ‗2

2 + λǁ‗yǁ‗1

X ¡≈ ¡D ¡ Y ¡ ¡

Sparse ¡ Coding ¡ Data ¡

X Y D

… … Feature ¡dic-onary ¡

min{D,y} ǁ‗x – Dyǁ‗2

2 + λǁ‗yǁ‗0

  • Popularly solved as L1-regularized optimization (LASSO/LARS)

– Optimizing on L0 pseudo-norm is intractable ⟹ greedy-L0 algorithm (OMP) can be used instead

Convex relaxation

slide-5
SLIDE 5

5 MLSP 2016

  • Background – Sparse Coding
  • Semi-supervised Learning with Sparse Coding
  • Sparse-coded Net
  • Experimental Evaluation
  • Conclusions and Future Work

Outline

slide-6
SLIDE 6

6 MLSP 2016

  • Semi-supervised learning

– Unsupervised stage: learn feature representation using unlabeled data – Supervised stage: optimize task objective using learned feature representations

  • f labeled data
  • Semi-supervised learning with sparse coding

– Unsupervised stage: sparse coding and dictionary learning with unlabeled data – Supervised stage: train classifier/regression using sparse codes of labeled data

Semi-supervised Learning with Sparse Coding

Preprocessing (optional) Raw data (unlabeled) Sparse coding & dictionary learning

D

(learned dictionary) Preprocessing (optional) Raw data (labeled) Sparse coding with D Feature pooling Classifier/ regression

Unsupervised stage Supervised stage

slide-7
SLIDE 7

7 MLSP 2016

  • Background – Sparse Coding
  • Semi-supervised Learning with Sparse Coding
  • Sparse-coded Net
  • Experimental Evaluation
  • Conclusions and Future Work

Outline

slide-8
SLIDE 8

8 MLSP 2016

  • Semi-supervised learning with sparse coding cannot jointly
  • ptimize feature representation learning and task objective
  • Sparse codes used as feature vectors for task cannot be

modified to induce correct data labels

– No supervised dictionary learning ⟹ sparse coding dictionary is learned using only unlabeled data

Sparse-coded Net Motivations

slide-9
SLIDE 9

9 MLSP 2016

Sparse-coded Net

Sparse coding

D

x(1)

Sparse coding

x(2)

Sparse coding

x(3)

Sparse coding

x(M)

. . .

Pooling (nonlinear rectification) Softmax

y(1) y(2) y(3) y(M) z p(l | z)

  • Feedforward model with sparse coding, pooling, softmax layers

– Pretrain: semi-supervised learning with sparse coding – Finetune: SCN backpropagation

slide-10
SLIDE 10

10 MLSP 2016

  • When predicted output does not

match ground truth, hold softmax weights constant and adjust pooled sparse code by gradient descent

– z ⟶ z*

  • Adjust sparse codes from adjusted

pooled sparse code by putback

– z* ⟶ Y*

  • Adjust sparse coding dictionary by

rank-1 updates or gradient descent

– D ⟶ D*

  • Redo feedforward path with adjusted

dictionary and retrain softmax

  • Repeat until convergence

SCN Backpropagation

Softmax

z

Rewrite softmax loss as function of z

Putback

slide-11
SLIDE 11

11 MLSP 2016

  • Background – Sparse Coding
  • Semi-supervised Learning with Sparse Coding
  • Sparse-coded Net
  • Experimental Evaluation
  • Conclusions and Future Work

Outline

slide-12
SLIDE 12

12 MLSP 2016

  • Audio and Acoustic Signal Processing (AASP)

– 30-second WAV files recorded in 44.1kHz 16-bit stereo – 10 classes such as bus, busy street, office, and open-air market – For each class, 10 labeled examples

  • CIFAR-10

– 60,000 32x32 color images – 10 classes such as airplane, automobile, cat, and dog – We sample 2,000 images to form train and test datasets

  • Wikipedia

– 2,866 documents – Annotated with 10 categorical labels – Text-document is represented as 128 LDA features

Experimental Evaluation

slide-13
SLIDE 13

13 MLSP 2016

  • Sparse-coded net model for LARS achieves the best accuracy

performance of 78%

– Comparable to the best AASP scheme (79%) – Significantly better than the AASP baseline† (57%)

Results: AASP Sound Classification

Sound Classification Performance on AASP dataset

Method Accuracy Semi-supervised via sparse coding (LARS) 73.0% Semi-supervised via sparse coding (OMP) 69.0% GMM-SVM 61.0% Deep SAE NN (4 layers) 71.0% Sparse-coded net (LARS) 78.0% Sparse-coded net (OMP) 75.0%

†D.Stowell, D.Giannoulis, E.Benetos, M.Lagrange, and M.D.Plumbley, “Detection and

Classification of Acoustic Scenes and Events,” IEEE Trans. on Multimedia, vol. 17, no. 10, 2015.

slide-14
SLIDE 14

14 MLSP 2016

  • Again, sparse-coded net model for LARS achieves the best

accuracy performance of 87.9%

– Superior to RBM and CNN pipelines evaluated by Coates et al.†

Results: CIFAR Image Classification

Image Classification performance on CIFAR-10

Method Accuracy Semi-supervised via sparse coding (LARS) 84.0% Semi-supervised via sparse coding (OMP) 81.3% GMM-SVM 76.8% Deep SAE NN (4 layers) 81.9% Sparse-coded net (LARS) 87.9% Sparse-coded net (OMP) 85.5%

†A. Coates, A. Ng, and H. Lee, “An Analysis of Single-layer Networks in Unsupervised Feature Learning,” in AISTATS, 2011.

slide-15
SLIDE 15

15 MLSP 2016

Results: Wikipedia Category Classification

Text Classification performance on Wikipedia dataset

Method Accuracy Semi-supervised via sparse coding (LARS) 69.4% Semi-supervised via sparse coding (OMP) 61.1% Deep SAE NN (4 layers) 67.1% Sparse-coded net (LARS) 70.2% Sparse-coded net (OMP) 62.1%

  • We achieve the best accuracy of 70.2% with sparse-coded

net on LARS

– Superior to 60.5 – 68.2% by existing approaches†1,†2

†1K. Duan, H. Zhang, and J. Wang, “Joint learning of cross-modal classifier and factor analysis for multimedia data

classification,” Neural Computing and Applications, vol. 27, no. 2, 2016.

†2 L. Zhang, Q. Zhang, L. Zhang, D. Tao, X. Huang, and B. Du, “Ensemble Manifold Regularized Sparse Low-rank Approximation for

Multi- view Feature Embedding,” Pattern Recognition, vol. 48, no. 10, 2015.

slide-16
SLIDE 16

16 MLSP 2016

  • Background – Sparse Coding
  • Semi-supervised Learning with Sparse Coding
  • Sparse-coded Net
  • Experimental Evaluation
  • Conclusions and Future Work

Outline

slide-17
SLIDE 17

17 MLSP 2016

Conclusions

  • Introduced sparse-coded net model that jointly optimizes sparse

coding and dictionary learning with supervised task at output layer

  • Proposed SCN backpropagation algorithm that can handle mix-up of

feature vectors related to pooling nonlinearity

  • Demonstrated superior classification performance on sound (AASP),

image (CIFAR-10), and text (Wikipedia) data

Future Work

  • More realistic larger-scale experiments necessary
  • Generalize hyperparameter optimization techniques for various

datasets (e.g., audio, video, text)

Conclusions and Future Work