Learning to map between ferns with differentiable binary embedding - PowerPoint PPT Presentation

DIFFERENTIABLE BINARY EMBEDDING USING FERNS Learning to map between ferns with differentiable binary embedding networks Maximilian Blendowski & Mattias P. Heinrich Institute of Medical Informatics University of Lübeck Short paper @ MIDL 2020

DIFFERENTIABLE BINARY EMBEDDING USING FERNS Random Fern Basics Ozuysal , Mustafa, et al. "Fast keypoint recognition using random ferns." IEEE transactions on pattern analysis and machine intelligence 32.3 (2009): 448-461. Dimension Threshold depth binary code 𝐺𝑓𝑠𝑜 𝑙 , … , 𝑒 𝑛 𝐺𝑓𝑠𝑜 𝑙 𝐺𝑓𝑠𝑜 𝑙 , … , 𝑢 𝑛 𝐺𝑓𝑠𝑜 𝑙 0 1 2 3 4 5 𝑛 = 2 𝑒 1 𝑢 1 𝐺𝑓𝑠𝑜 𝑙 𝒈 𝐸𝑗𝑛 +4 -2 +10 -6 +8 +1 −2 < −3 ? , +10 < 1? 𝑔𝑓𝑏𝑢𝑣𝑠𝑓 𝒈 𝐺𝑓𝑠𝑜 1 1,2 −3,1 𝟏𝟏 1 𝟏 𝐺𝑓𝑠𝑜 2 5,0 2,0 +1 < +2 ? , +4 < 0? +8 < +3 ? , −6 < −3? 𝐺𝑓𝑠𝑜 3 4,3 3, −1 𝟏𝟐 𝑮𝒇𝒔𝒐 𝟐 Class A 𝑮𝒇𝒔𝒐 𝟑 Class A 𝑮𝒇𝒔𝒐 𝟒 Class A 0.5 0.5 0.5 0.25 0.25 0.25 𝑄 𝒈|𝐵 = 0.5 ∗ 0.25 ∗ 0.5 = 0.0625 0.0 0.0 0.0 𝑄 𝒈|𝐶 = 0.25 ∗ 0.4 ∗ 0.125 = 0.0125 0 0 1 1 0 0 1 1 0 0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 𝑮𝒇𝒔𝒐 𝟐 Class B 𝑮𝒇𝒔𝒐 𝟑 Class B 𝑮𝒇𝒔𝒐 𝟒 Class B 0.5 0.5 0.5 𝑄 𝒈|𝐵 > 𝑄 𝒈|𝐶 → 𝑑𝑚𝑏𝑡𝑡𝑗𝑔𝑧 𝒈 𝑏𝑡 𝐵 0.25 0.25 0.25 0.0 0.0 0.0 0 0 1 1 0 0 1 1 0 0 1 1 0 1 0 1 0 1 0 1 0 1 0 1

DIFFERENTIABLE BINARY EMBEDDING USING FERNS Standard convolution 𝑐 𝑗 (: ) 𝑏 𝑗 (: ) 𝑐 𝑗 1) unfold 𝐷𝑝𝑜𝑤𝑝𝑚𝑣𝑢𝑗𝑝𝑜 𝑐 𝐺𝑗𝑚𝑢𝑓𝑠 𝑏 𝑗 𝑘 𝑏 𝑘 (: ) 𝑐 𝑘 (: ) 𝑙 2 ∙ 𝑑 𝑗𝑜 𝑜 𝑣 𝑋𝑓𝑗𝑕ℎ𝑢𝑡 𝑏 𝑘 𝑙 𝑉𝐺𝑁 ℎ 𝑗𝑜 𝑙 𝑙 2 ∙ 𝑑 𝑗𝑜 𝑥 𝑗𝑜 𝑑 𝑝𝑣𝑢 𝑝𝑣𝑢 𝑔𝑓𝑏𝑢 𝑗 Matrix Multiplication 𝑝𝑣𝑢 𝑔𝑓𝑏𝑢 𝑘 4) fold ℎ 𝑝𝑣𝑢 𝑑 𝑝𝑣𝑢 𝑥 𝑝𝑣𝑢 3

DIFFERENTIABLE BINARY EMBEDDING USING FERNS Drop-in replacement > 0 ? → 01 … 0 + 𝑡 𝑞 = 𝑗𝑒𝑦 𝑗 𝑐 𝑗 (: ) 𝑞 = tanh 𝑉𝐺𝑁 𝑗 𝑏 𝑗 (: ) 2) Generate 𝑞 , … , 𝑒 𝑛 𝑞 𝑞 , … , 𝑢 𝑛 𝑞 𝑞 𝑑 𝑗 𝑒 1 − 𝑢 1 𝑐 𝑗 Binary codes on 𝑗 LUT 1) unfold > 0 ? → 00 … 1 + 𝑡 𝑟 = 𝑗𝑒𝑦 𝑗 𝑟 = tanh 𝑉𝐺𝑁 𝑗 𝑟 , … , 𝑒 𝑛 𝑟 𝑟 , … , 𝑢 𝑛 𝑟 𝑟 𝑑 𝑗 𝑒 1 − 𝑢 1 codes 𝑐 > 0 ? → 01 … 0 + 𝑡 𝑞 = 𝑗𝑒𝑦 𝑘 𝑏 𝑗 𝑘 𝑞 = tanh 𝑉𝐺𝑁 𝑘 𝑞 , … , 𝑒 𝑛 𝑞 𝑞 , … , 𝑢 𝑛 𝑞 𝑞 𝑑 𝑒 1 − 𝑢 1 𝐺𝑓𝑠𝑜 𝑞 𝑏 𝑘 (: ) 𝑐 𝑘 (: ) 𝑘 Binary codes on 𝑘 𝑜 𝑣 > 0 ? → 01 … 1 + 𝑡 𝑟 = 𝑗𝑒𝑦 𝑘 𝑟 = tanh 𝑉𝐺𝑁 𝑘 𝑟 , … , 𝑒 𝑛 𝑟 𝑟 , … , 𝑢 𝑛 𝑟 𝑟 𝐺𝑓𝑠𝑜 𝑟 𝑑 𝑒 1 − 𝑢 1 𝑏 𝑘 𝑘 𝑙 𝑉𝐺𝑁 combination 3) weigthed ℎ 𝑗𝑜 LUT entry 𝑙 𝑟 𝑞 𝑞 𝑞 𝑒 1 𝑒 𝑛 𝑒 1 𝑒 𝑛 𝑙 2 ∙ 𝑑 𝑗𝑜 Differentiable ferns 𝑥 𝑗𝑜 𝑞 𝑞 + ⋯ + 𝑀𝑉𝑈 𝑗𝑒𝑦 𝑗 𝑞 𝑟 𝑟 𝑗𝑒𝑦 𝑗 = 𝑀𝑉𝑈 𝑗𝑒𝑦 𝑗 ∙ 𝑥 𝑗 ∙ 𝑥 𝑗 𝑝𝑣𝑢 𝑔𝑓𝑏𝑢 𝑗 𝑞 𝑗𝑒𝑦 𝑘 #𝑔𝑓𝑠𝑜𝑡 ∙ 2 𝑒𝑓𝑞𝑢ℎ 𝑞 + ⋯ + 𝑀𝑉𝑈 𝑗𝑒𝑦 𝑘 𝑀𝑉𝑈 𝑞 𝑟 𝑟 𝑝𝑣𝑢 = 𝑀𝑉𝑈 𝑗𝑒𝑦 𝑘 ∙ 𝑥 ∙ 𝑥 𝑔𝑓𝑏𝑢 𝑘 𝑘 𝑘 4) fold 𝑟 𝑗𝑒𝑦 𝑘 ℎ 𝑝𝑣𝑢 𝑟 𝑗𝑒𝑦 𝑗 trainable 𝑑 𝑝𝑣𝑢 𝑥 𝑝𝑣𝑢 weights 𝑑 𝑝𝑣𝑢 4

DIFFERENTIABLE BINARY EMBEDDING USING FERNS Evaluation c in : 3 c in : 64 c in : 64 c in : 64 Global Input c out : 64 c out : 64 c out : 64 c out : 2 Tumor: Average Patch Yes/no? Kernelsize: 5 Kernelsize: 3 Kernelsize: 3 Kernelsize: 1 Pooling Stride: 2 Stride: 2 Stride: 2 Stride: 1 BatchNorm BatchNorm BatchNorm --- [1] [2] [3] [1] Hubara, Itay, et al. "Binarized neural networks." Advances in neural information processing systems . 2016. [2] Rastegari, Mohammad, et al. "Xnor-net: Imagenet classification using binary convolutional neural networks." European conference on computer vision . Springer, Cham, 2016. [3] Veta, Mitko, et al. "Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge." Medical image analysis 54 (2019): 111-121. 5