SLIDE 1
Neural Computing Journal Club 14/10/2020 Presented by Edward Jones
1
Overview of the Review
- Introduction of the paper and authors
- Central idea of the Lottery Ticket Hypothesis
- Discussion of the experiments
- Relation to other work
2
Neural Computing Journal Club 14/10/2020 Presented by Edward - - PowerPoint PPT Presentation
Neural Computing Journal Club 14/10/2020 Presented by Edward - - PowerPoint PPT Presentation
Neural Computing Journal Club 14/10/2020 Presented by Edward Jones 1 Overview of the Review Introduction of the paper and authors Central idea of the Lottery Ticket Hypothesis Discussion of the experiments Relation to
SLIDE 2
SLIDE 3
The Authors
3
SLIDE 4
The Lottery Ticket Hypothesis
“A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.”
4
SLIDE 5
5
SLIDE 6
6
SLIDE 7
7
SLIDE 8
8
SLIDE 9
9
SLIDE 10
SLIDE 11
SLIDE 12
“A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.”
12
SLIDE 13
“A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.”
13
SLIDE 14
“A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.”
14
SLIDE 15
“A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.”
15
SLIDE 16
“A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.”
16
≤
SLIDE 17
17
SLIDE 18
Results - Lenet
18
SLIDE 19
19
Results - Resnet-18
SLIDE 20
Other findings
- Dropout works synergistically with pruning
- These sparse networks appear to generalise better
- Winning tickets are robust to weight noise
20
SLIDE 21
Follow-up work
21
SLIDE 22
Summary
- Winning tickets (subnetworks) exists
- Both structure and intialisation matter for winning tickets
- This could lead to optimisation of training
22
SLIDE 23
References
Bellec, G., Kappel, D., Maass, W., and Legenstein, R. (2017). Deep Rewiring: Training very sparse deep networks. arXiv:1711.05136 [cs, stat]. Available at: http://arxiv.org/abs/1711.05136 [Accessed October 14, 2020]. Bogdan, P. A., Rowley, A. G. D., Rhodes, O., and Furber, S. B. (2018). Structural Plasticity on the SpiNNaker Many-Core Neuromorphic
- System. Front. Neurosci. 12. doi:10.3389/fnins.2018.00434.
Frankle, J., and Carbin, M. (2019). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. arXiv:1803.03635 [cs]. Available at: http://arxiv.org/abs/1803.03635 [Accessed October 13, 2020]. Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M. (2020a). Linear Mode Connectivity and the Lottery Ticket Hypothesis. arXiv:1912.05671 [cs, stat]. Available at: http://arxiv.org/abs/1912.05671 [Accessed October 14, 2020]. Frankle, J., Schwab, D. J., and Morcos, A. S. (2020b). The Early Phase of Neural Network Training. arXiv:2002.10365 [cs, stat]. Available at: http://arxiv.org/abs/2002.10365 [Accessed October 14, 2020].
23
Lange, R. (2020). The Lottery Ticket Hypothesis: A Survey. Medium. Available at: https://towardsdatascience.com/the-lottery-ticket-hypothesis-a-survey- d1f0f62f8884 [Accessed October 14, 2020]. LeCun, Y., Denker, J. S., and Solla, S. A. Optimal Brain Damage. 8. The Lottery Ticket Hypothesis with Jonathan Frankle (2020). Available at: https://www.youtube.com/watch?v=SfjJoevBbjU [Accessed October 14, 2020]. Yu, H., Edunov, S., Tian, Y., and Morcos, A. S. (2020). Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP. arXiv:1906.02768 [cs, stat]. Available at: http://arxiv.org/abs/1906.02768 [Accessed October 14, 2020].