Brain-Generated Labels Sergey Vaisman, VP R&D, InnerEye S9554 - - - PowerPoint PPT Presentation

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels

Sergey Vaisman, VP R&D, InnerEye

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 2

“Without humans, artificial intelligence is still pretty stupid..”

Image: The New York Times

(The Wall Street Journal)

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 3

Data Collection Dataset Annotation Neural Network Design Deployment New Data Neural Network Training Optimization Dataset Exploration

AI Training Challenges

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 4

0.4 0.5 0.6 0.7 0.8 0.9 1 20 40 60 80 100 120 140 160 180 200 220 240 260

AUC (Area Under the Curve) Time [min] Classification Performance

Brain In The Loop - Iterative AI Training Framework

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 5

Image Classification Average Performance

10x times faster

Trained on NVIDIA GeForce GTX1080 ti GPU

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 6

InnerEye – The Company

Combining Human Intelligence with Artificial Intelligence

▪ Founded in 2014 - Technology spin-off from Israel’s The Hebrew and Ben-Gurion Universities ▪ Offices in Herzliya, Israel and Tokyo, Japan ▪ Over $6M of funding provided so far ▪ Products: Visual content review, AI Training and Validation, Connected Human ▪ Management Team:

Uri Antman

CEO

• Prof. Amir B. Geva

Founder and CTO

• Prof. Leon Y. Deouell

Founder and CSO

Sergey Vaisman

VP R&D

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 7

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye

Agenda

▪ Iterative AI Training Framework ▪ Performance ▪ Use Cases

8

Images Images for human review

Brainwaves Classification Network Image Classification Network

Trained Network

InnerEye AI Training Framework

Soft Labels S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 9 Convergence Criterion

InnerEye AI Training Framework

Images

Images for human review

Convergence Criterion

Image Classification Network

Trained Network

Deployment

Brainwaves Classification Network

Soft Labels

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 1

Visual Processing Response Planning Motor Execution Distraction caused by motor response Sensation Image User Response Decision Making Brain activity measurement

Tapping Into The Brain

InnerEye Bypass

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 11

Brain Activity Measurement - EEG

t = 0.33s t = 0.66s t = 1s t = 1.33s t = 1.66s t = 2s t = 0 S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 12

t = 0.33s t = 0.66s t = 1s t = 1.33s t = 1.66s t = 2s t = 0 S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 13

t = 0.33s t = 0.66s t = 1s t = 0

Single Trial Spatio-Temporal Activity

time (ms) Trial # 200 400 600 800 1000 2000 3000 4000 5000 6000 7000 8000 9000

T Time samples x D Channels

Single Trial Spatio-Temporal Matrix

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 14

t = 1s t = 0

Single Trial Classification

Targets

time (ms) Trial # 200 400 600 800 100 200 300 400 500 600

Non Targets

time (ms) Trial # 200 400 600 800 1000 2000 3000 4000 5000 6000 7000 8000 9000 100 200 300 400 500 600 700 800 900

• 5

5 10

time (ms) m V

Pz

Non target Target

t = 200ms t = 300ms t = 450ms t = 550ms t = 700ms t = 200ms t = 300ms t = 450ms t = 550ms t = 700ms

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 15

InnerEye Brainwaves Classification Network

Deep Neural Network Ensemble

T Time samples x D Channels

INPUT: Single Trial Spatio-Temporal Matrix OUTPUT: Trial Classification score

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 16

Brainwaves Classification Scores Distribution

Score

Sports Cars Regular Cars

Image

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 17

Soft Labels Concept

▪ These images contain flowers but not all of them should contribute equally to the learning process ▪ Can we create more informative labels to address the diversity and improve classification accuracy?

Label: FLOWER Label: FLOWER Label: FLOWER

Image Source: Google Open Images Dataset

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 18

EEG Classification Scores Are Used to Generate Soft Labels

▪ Images that received high or low scores from the EEG classifier are given higher weight ▪ Images that received intermediate (inconclusive) scores from the EEG classifier are given lower weight

𝑥𝑗 = ቊ 𝑑𝑗, 𝑧𝑗 = 1 1 − 𝑑𝑗, 𝑧𝑗 = 0 𝑑𝑗 = EEG Classification Score 𝑧𝑗 = Sample Label 𝑥𝑗 = Sample Weight (Soft Label)

THR = Classification Threshold

𝑧𝑗 = ቊ1, 𝑑𝑗 ≥ 𝑈𝐼𝑆 0, 𝑑𝑗 < 𝑈𝐼𝑆

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 19

Soft Labels Are Correlated with Human Confidence Level

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 20

InnerEye AI Training Framework

Images

Images for human review

Convergence Criterion

Image Classification Network

Brainwaves Classification Network

Deployment

Soft Labels S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 21

Image Classification Network

▪ In each iteration, image classification layers are trained using the new generated soft labels

64 filters X 2 blocks 128 filters X 2 blocks 256 filters X 3 blocks 512 filters X 3 blocks 512 filters X 3 blocks 32 neurons Sigmoid

• utput

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 22

Soft Labels Are Used As Sample Weights in Loss Function

▪ We add sample weights to the cross-entropy loss function:

𝑀(𝑦𝑗) = Cross Entropy Loss Function 𝑦𝑗 = Sample 𝑧𝑗 = Sample Label 𝑥𝑗 = Sample Weight (Soft Label) 𝑞𝑗 = Sample Prediction

𝑀 𝑦𝑗 = −𝒙𝒋 𝑧𝑗𝑚𝑝𝑕 𝑞𝑗 + 1 − 𝑧𝑗 𝑚𝑝𝑕(1 − 𝑞𝑗)

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 23

▪ The learning algorithm selects new samples to learn from in the next iteration based on Confidence criterion: ▪ Only the least confident samples (𝐷𝑗 < THR) will be sent for human review ▪ Also used as Convergence Condition

Convergence and Active Learning Iterations

𝐷𝑗 = max

𝑘

𝑄 𝑧𝑗 = 𝑘|𝑦𝑗

𝐷𝑗 = Confidence for Sample i

j = Class index

𝑧𝑗 = Predicted Sample label 𝑦𝑗 = Sample i

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 24

Soft Labels Improve Neural Network Performance

(*) AUC shown after the first iteration

(permuted sample weights of the labels)

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 25

Active Learning Combined With Soft Labels Improves Neural Network Performance

(*) AUC shown after the first iteration S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 26

Images Images for human review

Brainwaves Classification Network Image Classification Network

Trained Network

InnerEye AI Training Framework

Soft Labels S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye Convergence Criterion 27

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye

Agenda

▪ Iterative AI Training Framework ▪ Performance ▪ Use Cases

28

We started with 2,800 Cars images to be classified. Initial network performance for Sports cars detection: AUC=0.5

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 29

After Iteration 1 (T=4.2 min): Human expert reviewed 200 images. Network performance: AUC=0.8

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 30

After Iteration 2 (T=8.7 min): Human expert reviewed 400 images. Network performance: AUC=0.86

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 31

After Iteration 3 (T=13.3 min): Human expert reviewed 589 images. Network performance: AUC=0.88

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 32

After Iteration 4 (T=17.2 min): Human expert reviewed 718 images. Network performance: AUC=0.9

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 33

After Iteration 5 (T=20.6 min): Human expert reviewed 795 images. Network performance: AUC=0.92

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 34

After 20.6 minutes the network is trained to detect Sports Cars. The human expert was required to review only 28% of the images

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 35

Examples of correctly classified sports cars:

Images with the highest score from the InnerEye system trained to classify Sport Cars:

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 36

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 37

Image Classification Performance vs. % of Training Data

38 (*) AUC shown after the first iteration

Subject 1 Subject 2 Subject 3

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 38

Image Classification Performance vs. Training Time

(*) Trained on NVIDIA GeForce GTX 1080 Ti

0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 25 50 75 100 125 150 175 200 225 250 275

AUC Time [min]

Subject 1

InnerEye Training Method Manual Training Method 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 25 50 75 100 125 150 175 200 225 250 275

AUC Time [min]

Subject 2

InnerEye Training Method Manual Training Method 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 25 50 75 100 125 150 175 200 225 250 275

AUC Time [min]

Subject 3

InnerEye Training Method Manual Training Method

▪ Time savings come from combination of fast presentation rate and faster convergence

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 39

Image Classification Average Performance

10x times faster

Trained on NVIDIA GeForce GTX1080 ti GPU

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 40

Soft Labels Compensate for EEG Misclassified Samples

Correctly classified sports cars Weight: 0.93 Weight: 0.81 Weight: 0.87 False Positives Weight: 0.26 Weight: 0.3 Weight:0.22 Weight: 0.4 Weight: 0.43 Weight: 0.31 Missed sports cars Weight: 0.85 Weight: 0.79 Weight: 0.91 Correctly classified regular cars

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 41

Soft Labels Compensate for EEG Misclassified Samples

(*) AUC shown after the first iteration S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 42

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye

Agenda

▪ Iterative AI Training Framework ▪ Performance ▪ Use Cases

4 3

Multiclass Classification

Accuracy: 0.81

Event Related Potentials measured on electrode T5

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 44

Validation of Annotation Quality

▪Fast Screening and amendment of low confidence annotated data

Screenshot of the output from InnerEye system output of detecting images wrongly labeled as “flowers”

Source: Google Open Images Dataset

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 45

Combined Brain-Computer Visual Network

IMG Net EEG Net

Score: Classification:

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 46

InnerEye Driver Car Sensors Road Conditions

Environment: Driving Agent

State Reward Action

Screenshot of the output from “InnerEye driver” brain responding to seeing pedestrian crossing the road, measuring Hazard Detection, Attention and Emotion

Reinforcement Learning for Autonomous Driving

▪Incorporating brain insights in the training process of the AI driving agent

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 47

Efficiency & Throughput Cost

Personalization Real-Time Interface Privacy

Summary

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 48

THANK YOU! COME SEE OUR DEMO AT BOOTH 335

Contact me at: sergey@innereye.ai

S9554 - Fast Training of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 49