Brain-Generated Labels Sergey Vaisman, VP R&D, InnerEye S9554 - - PowerPoint PPT Presentation

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, VP R&D, InnerEye

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 2

“ Without humans, artificial intelligence is still pretty stupid.. ” (The Wall Street Journal) New Data Neural Neural Data Dataset Dataset Network Deployment Network Collection Annotation Exploration Design Training Optimization Image: The New York Times S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 3

AI Training Challenges Classification Performance 1 0.9 AUC (Area Under the Curve) 0.8 0.7 0.6 0.5 0.4 0 20 40 60 80 100 120 140 160 180 200 220 240 260 Time [min] S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 4

Brain In The Loop - Iterative AI Training Framework S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 5

Image Classification Average Performance 10x times faster Trained on NVIDIA GeForce GTX1080 ti GPU S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 6

InnerEye – The Company Combining Human Intelligence with Artificial Intelligence ▪ Founded in 2014 - Technology spin-off from Israel ’ s The Hebrew and Ben-Gurion Universities ▪ Offices in Herzliya, Israel and Tokyo, Japan ▪ Over $6M of funding provided so far ▪ Products: Visual content review, AI Training and Validation, Connected Human ▪ Management Team: Uri Antman Prof. Leon Y. Deouell Sergey Vaisman Prof. Amir B. Geva CEO Founder and CSO VP R&D Founder and CTO S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 7

Agenda ▪ Iterative AI Training Framework ▪ Performance ▪ Use Cases 8 S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye

InnerEye AI Training Framework Image Classification Brainwaves Network Classification Network Soft Labels Convergence Trained Criterion Network Images Images for human review S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 9

InnerEye AI Training Framework Image Classification Network Brainwaves Classification Soft Labels Network Trained Network Convergence Deployment Criterion Images Images for human review 1 S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 0

Tapping Into The Brain Distraction Visual Decision Response caused by Motor Sensation Processing Making motor Execution Planning User response Image Response InnerEye Bypass Brain activity measurement S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 11

Brain Activity Measurement - EEG t = 1s t = 1.33s t = 0 t = 0.33s t = 0.66s t = 1.66s t = 2s 12 S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye

t = 1s t = 1.33s t = 0 t = 0.33s t = 0.66s t = 1.66s t = 2s S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 13

Single Trial Spatio-Temporal Activity Single Trial 1000 Spatio-Temporal Matrix 2000 3000 T Time samples x D Channels 4000 Trial # 5000 6000 7000 8000 9000 0 200 400 600 800 time (ms) t = 1s t = 0 t = 0.33s t = 0.66s S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 14

Single Trial Classification Targets Non Targets 1000 Pz 10 100 Non target 2000 Target 3000 200 5 4000 m V Trial # Trial # 300 5000 0 6000 400 7000 500 -5 0 100 200 300 400 500 600 700 800 900 8000 time (ms) 9000 600 0 200 400 600 800 0 200 400 600 800 time (ms) time (ms) t = 200ms t = 300ms t = 450ms t = 550ms t = 700ms t = 200ms t = 300ms t = 450ms t = 550ms t = 700ms t = 1s t = 0 S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 15

InnerEye Brainwaves Classification Network OUTPUT: Trial Classification INPUT: score Single Trial Spatio-Temporal Matrix T Time samples x D Channels Deep Neural Network Ensemble S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 16

Brainwaves Classification Scores Distribution Score Sports Cars Regular Cars Image 17 S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye

Soft Labels Concept ▪ These images contain flowers but not all of them should contribute equally to the learning process ▪ Can we create more informative labels to address the diversity and improve classification accuracy? Label: FLOWER Label: FLOWER Label: FLOWER Image Source: Google Open Images Dataset S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 18

EEG Classification Scores Are Used to Generate Soft Labels ▪ Images that received high or low scores from the EEG classifier are given higher weight ▪ Images that received intermediate (inconclusive) scores from the EEG classifier are given lower weight 𝑑 𝑗 = EEG Classification Score 𝑧 𝑗 = ቊ1, 𝑑 𝑗 ≥ 𝑈𝐼𝑆 0, 𝑑 𝑗 < 𝑈𝐼𝑆 𝑧 𝑗 = Sample Label 𝑥 𝑗 = Sample Weight (Soft Label) 𝑑 𝑗 , 𝑧 𝑗 = 1 𝑥 𝑗 = ቊ 1 − 𝑑 𝑗 , 𝑧 𝑗 = 0 THR = Classification Threshold S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 19

Soft Labels Are Correlated with Human Confidence Level S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 20

InnerEye AI Training Framework Image Classification Network Brainwaves Classification Network Soft Labels Convergence Deployment Criterion Images Images for human review S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 21

Image Classification Network ▪ In each iteration, image classification layers are trained using the new generated soft labels 64 filters 128 filters X 2 blocks X 2 blocks 256 filters 512 filters X 3 blocks 512 filters X 3 blocks 32 X 3 blocks neurons Sigmoid output S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 22

Soft Labels Are Used As Sample Weights in Loss Function ▪ We add sample weights to the cross-entropy loss function: 𝑀(𝑦 𝑗 ) = Cross Entropy Loss Function 𝑦 𝑗 = Sample 𝑀 𝑦 𝑗 = −𝒙 𝒋 𝑧 𝑗 𝑚𝑝𝑕 𝑞 𝑗 + 1 − 𝑧 𝑗 𝑚𝑝𝑕(1 − 𝑞 𝑗 ) 𝑧 𝑗 = Sample Label 𝑥 𝑗 = Sample Weight (Soft Label) 𝑞 𝑗 = Sample Prediction S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 23

Convergence and Active Learning Iterations ▪ The learning algorithm selects new samples to learn from in the next iteration based on Confidence criterion: 𝐷 𝑗 = Confidence for Sample i j = Class index 𝐷 𝑗 = max 𝑄 𝑧 𝑗 = 𝑘|𝑦 𝑗 𝑘 𝑧 𝑗 = Predicted Sample label 𝑦 𝑗 = Sample i ▪ Only the least confident samples ( 𝐷 𝑗 < THR) will be sent for human review ▪ Also used as Convergence Condition S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 24

Soft Labels Improve Neural Network Performance (permuted sample weights of the labels) (*) AUC shown after the first iteration S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 25

Active Learning Combined With Soft Labels Improves Neural Network Performance (*) AUC shown after the first iteration S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 26

InnerEye AI Training Framework Image Classification Brainwaves Network Classification Network Soft Labels Convergence Trained Criterion Network Images Images for human review S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 27

Agenda ▪ Iterative AI Training Framework ▪ Performance ▪ Use Cases S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 28

We started with 2,800 Cars images to be classified. Initial network performance for Sports cars detection: AUC= 0.5 S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 29

After Iteration 1 (T=4.2 min): Human expert reviewed 200 images. Network performance: AUC=0.8 S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 30

After Iteration 2 (T=8.7 min): Human expert reviewed 400 images. Network performance: AUC=0.86 S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 31

After Iteration 3 (T=13.3 min): Human expert reviewed 589 images. Network performance: AUC=0.88 S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 32