SLIDE 1
Splitting yields adaptive net structure optimization
Questions
- Why splitting?
- What neurons should be split first?
- How to split a neuron optimally?
With Splitting Steepest Descent Splitting yields adaptive net - - PowerPoint PPT Presentation
With Splitting Steepest Descent Splitting yields adaptive net - - PowerPoint PPT Presentation
Energy-Aware Neural Architecture Optimization With Splitting Steepest Descent Splitting yields adaptive net structure optimization Questions Why splitting? What neurons should be split first? How to split a neuron optimally?
SLIDE 2
SLIDE 3
SGD local minima
Intuition: escaping local minima
- A simple network:
- Smooth loss change:
- Splitting 𝜄 into 𝑛 copies :
saddle point?
SLIDE 4
Splitting Steepest Descent
- How to choose 𝑛 and {𝜄𝑗, 𝑥𝑗} optimally?
Splitting-matrix Splitting-index, minimum eigenvalue
𝜄
𝜄1
𝜄2
- Optimal splitting strategy
no splitting
SLIDE 5
Our Algorithm
- 0.2
12
- 0.1
1
- 0.9
3
- 0.2
1
- 1.2
4 gain flops budget
SLIDE 6