SLIDE 1
Xavier Snelgrove, CTO & Co-Founder, Whirlscape @wxswxs March 2017
Low memory RNNs... for emoji! Xavier Snelgrove , CTO & - - PowerPoint PPT Presentation
Low memory RNNs... for emoji! Xavier Snelgrove , CTO & - - PowerPoint PPT Presentation
Low memory RNNs... for emoji! Xavier Snelgrove , CTO & Co-Founder, Whirlscape @wxswxs March 2017 Me, me, me! Me, me, me! Me, me, me! Minuum Dango http:/ /minuum.com http:/ /getdango.com Me, me, me! Minuum Dango http:/
SLIDE 2
SLIDE 3
Me, me, me!
SLIDE 4
Me, me, me!
Dango
http:/ /getdango.com
Minuum
http:/ /minuum.com
SLIDE 5
Me, me, me!
Dango
http:/ /getdango.com
Minuum
http:/ /minuum.com
SLIDE 6
Title Text
SLIDE 7
Title Text
SLIDE 8
Title Text
SLIDE 9
Title Text
SLIDE 10
Title Text
SLIDE 11
Title Text
SLIDE 12
Title Text
SLIDE 13
SLIDE 14
SLIDE 15
With Dango
SLIDE 16
With Dango
SLIDE 17
With Dango
SLIDE 18
With Dango
SLIDE 19
With Dango
SLIDE 20
100s of Millions of Examples
Hi prince 👒 Never mind. I forgot I’m single 😓😪 that's what I like to hear 😈❤ Highway driving in the morning 🌆👍 happy bro bro it was cool chilling with you in line for Travis gotta catch another show turn up one time🙐😝
SLIDE 21
100s of Millions
- f Examples
SLIDE 22
100s of Millions
- f Examples
GPUs crunch away for days
💮
💮
💮
SLIDE 23
100s of Millions
- f Examples
Trained Model
💭
💮
📛 GPUs crunch away for days
💮
💮
💮
SLIDE 24
Let’s eat lunch later
SLIDE 25
Let’s eat lunch later
SLIDE 26
Let’
Let’s eat lunch later
SLIDE 27
Let’
Let’s eat lunch later
SLIDE 28
Let’
Let’s eat lunch later
🍵😌🍞
SLIDE 29
Emoji in semantic-space
SLIDE 30
How can we run this on device?
SLIDE 31
Let’s eat lunch later
Word Embedding Recurrent Layers Dense Output Layers
SLIDE 32
Embedding memory
the and cat yesterday eggplant . . . alchemist missspellling
SLIDE 33
Embedding memory
the and cat yesterday eggplant . . . alchemist missspellling
}
}
512 100,000
SLIDE 34
Embedding memory
512 100,000 4 bytes × ×
SLIDE 35
Embedding memory
512 100,000 4 bytes × ×
=200 MB
SLIDE 36
Embedding memory
512 100,000 4 bytes × ×
=200 MB
SQLite
SLIDE 37
Embedding memory
512 100,000 4 bytes × ×
=200 MB
SQLite
3 bits
20 MB
Quantize
SLIDE 38
Embedding memory
SQLite
Distribution of embedding values
Hufgman coding? Depends on quantization
SLIDE 39
Let’s eat lunch later
Word Embedding Recurrent Layers Dense Output Layers
SLIDE 40
Recurrent Layer Memory
Input Vector Previous State Next State
+
Output Vector
SLIDE 41
Recurrent Layer Memory
SLIDE 42
Recurrent Layer Memory
}
768
}
768
SLIDE 43
Recurrent Layer Memory
768 768 4 bytes × × 3 ×
= 14MB
2 layers ×
SLIDE 44
Recurrent Layer Memory
768 768 4 bytes × × 3 ×
= 14MB
2 layers × 2 bytes
Quantize (float16)
7MB
SLIDE 45
Recurrent Layer Memory
Distribution of weight values
SLIDE 46
Recurrent Layer Memory
Distribution of weight values
Many near-zero values
SLIDE 47
Recurrent Layer Memory
SLIDE 48
Recurrent Layer Memory
Prune 50% of weights closest to 0
SLIDE 49
Recurrent Layer Memory
Prune 50% of weights closest to 0 Train the rest of the network
SLIDE 50
Recurrent Layer Memory
Prune 50% of weights closest to 0 Train the rest of the network Repeat, pruning more each iteration
SLIDE 51
Recurrent Layer Memory
Prune 50% of weights closest to 0 Train the rest of the network Repeat, pruning more each iteration
90% prune 7MB × 0.1 = 700 kB
SLIDE 52
Recurrent Layer Memory
Prune 50% of weights closest to 0 Train the rest of the network Repeat, pruning more each iteration
90% prune 7MB × 0.1 = 700 kB
SLIDE 53
Questions?
http:/ /getdango.com
Xavier Snelgrove, CTO & Co-Founder @wxswxs