Biologically-Inspired Sparse Restricted Boltzmann Machines Pablo - - PowerPoint PPT Presentation

▶
biologically inspired sparse restricted boltzmann machines
SMART_READER_LITE
LIVE PREVIEW

Biologically-Inspired Sparse Restricted Boltzmann Machines Pablo - - PowerPoint PPT Presentation

Jun 25, 2023 124 likes •543 views

Biologically-Inspired Sparse Restricted Boltzmann Machines Pablo Tostado Michael Wiest Alice Yepremyan 1 Content 1. Motivation 2. Background a. Restricted Boltzmann Machines b. Sparsity 3. Methods a. Pruning algorithm b.

slide-1
SLIDE 1

Biologically-Inspired Sparse Restricted Boltzmann Machines

Pablo Tostado Michael Wiest Alice Yepremyan

1

slide-2
SLIDE 2

Content

1. Motivation 2. Background

a. Restricted Boltzmann Machines b. Sparsity

3. Methods

a. Pruning algorithm b. Evaluation criteria

4. Results 5. Discussion 6. Future directions

2

slide-3
SLIDE 3

Motivation

3

slide-4
SLIDE 4

4

  • Reduces computational

complexity

  • Increases speed
  • Yield to higher probabilities
  • Total energy consumed

decreases with increasing sparsity

Sparsity

  • Increases computational

efficiency

  • Less prone to overfitting
  • Can often lead to better

solutions in neural networks (AlexNet and dropout)

In Biological systems:1 In Artificial systems:2

slide-5
SLIDE 5

Background

5

slide-6
SLIDE 6

6

  • Generative, stochastic Artificial Neural Networks
  • Fully connected bipartite graphs
  • Able to learn probability distributions over its set of inputs
  • Building block of deeper neural networks
  • Hebbian nature in the learning algorithm

Restricted Boltzmann Machines

slide-7
SLIDE 7

7

RBM: Structure

[3]

slide-8
SLIDE 8

8

Z =

(v,h) v,h

RBM: Energy of Network

slide-9
SLIDE 9

Methods

9

slide-10
SLIDE 10

RBM architecture

10

Hidden Units: 1) 100 nodes 2) 500 nodes Visible Units: 784 units MNIST (28x28)

[3]

slide-11
SLIDE 11

Our Algorithm

11

1. Do an initial round of training (1000 epochs). 2. Prune the P* lowest weight connection (set weight to zero) 3. Train again (400 epochs) 4. Repeat 2 and 3 until desired amount of pruning done 5. Do a final round of training (1000 epochs)

slide-12
SLIDE 12

Data

12

  • Training: 1000 MNIST images (100 of each character)
  • Testing: 100 MNIST images (10 of each character)

*Due to computing time.

slide-13
SLIDE 13

13

  • Accuracy* of image reconstruction from:

1. Noisy image 2. Occluded image

  • Altering the parameters

○

Hidden nodes: 100 and 500

○

Percent noisy/occluded: 5, 10, 25, 50%

  • Visual nodes represented by hidden nodes
  • Pruning over time

Evaluation Criteria for Pruned and Unpruned

slide-14
SLIDE 14

Evaluation Criteria: Noise

14

20% Noise

slide-15
SLIDE 15

Evaluation Criteria: Square Occlusion

15

20% occlusion

slide-16
SLIDE 16

Image Recovery Example

16

N Steps

slide-17
SLIDE 17

Noise Convergence (20% noise, 25% pruning)

slide-18
SLIDE 18

Occlusion Convergence (20% occlude, 25% prune)

18

slide-19
SLIDE 19

Image recovery Scoring

19

  • 10 examples of each image
  • For each image do 100 iterations of random noise/occlusion
  • For some iteration i:
slide-20
SLIDE 20

Results

20

slide-21
SLIDE 21

Pruning preference (100 hidden nodes)

21

0% 5% 10% 25% 50% 80% Percentage Pruned

slide-22
SLIDE 22

Pruning preference (500 hidden nodes)

22

0% 5% 10% 25% 50% Percentage Pruned

slide-23
SLIDE 23

23

slide-24
SLIDE 24

24

slide-25
SLIDE 25

25

100 Hidden Nodes

slide-26
SLIDE 26

26

100 Hidden Nodes

slide-27
SLIDE 27

27

500 Hidden Nodes

slide-28
SLIDE 28

28

500 Hidden Nodes

slide-29
SLIDE 29

Discussion: Example of Denoising

29

Input Img 50% noise

0% pruning 10% pruning

slide-30
SLIDE 30

Number of Training Epochs

30

slide-31
SLIDE 31

Training Error

31

slide-32
SLIDE 32

32

  • Train over more images
  • Alternate pruning heuristics (L1 norm)
  • Train over more hidden nodes
  • Computational efficiency evaluation
  • Build a classifier on top of MNIST pixels

Future directions:

slide-33
SLIDE 33

Thanks!

33

slide-34
SLIDE 34

Additional Slides

34

slide-35
SLIDE 35

35

100 Hidden Nodes

slide-36
SLIDE 36

36

100 Hidden Nodes

slide-37
SLIDE 37

37

500 Hidden Nodes

slide-38
SLIDE 38

38

500 Hidden Nodes

slide-39
SLIDE 39

Works Cited

[1] B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research, 37:3311–3325, 1997. [2] S. Changpinyo, M. Sandler, and A. Zhmoginov. The power of sparsity in convolutional neural networks. CoRR, abs/1702.06257, 2017. [3] Chris Nicholson, Adam Gibson, Skymind team. “A Beginner's Tutorial for Restricted Boltzmann Machines.” A

Beginner's Tutorial for Restricted Boltzmann Machines - Deeplearning4j: Open-Source, Distributed Deep Learning for the JVM, deeplearning4j.org/restrictedboltzmannmachine.

39

slide-40
SLIDE 40

40

Measure of Error: KL-Divergence