and Challenges DAVIDE BACCIU DIPARTIMENTO DI INFORMATICA UNIVERSIT - - PowerPoint PPT Presentation

Deep Learning: Trends and Challenges

DAVIDE BACCIU DIPARTIMENTO DI INFORMATICA UNIVERSITÀ DI PISA

BioMedical IoT

Challenges

Applications

Trends

Structured* HPC Knowledge Transfer*

Input Hard-coded expert reasoning Prediction Expert- designed features Trainable predictor Learned features

AI ML

Learned feature hierarchy

Deep Learning

Neural Net Machinery in 1 Slide

w1 …

f

wn

Synaptic weights Free parameters of the model Neuron Activation Weighted input summation + thresholding function (often differentiable and nonlinear) Network input Network prediction Learning Ground-truth predictions in training data can be used to adapt the synaptic weights of all neurons

BioMedical IoT

Challenges

Applications

Trends

Structured* HPC Knowledge Transfer*

Structured Data

Compound information whose atomic components provide informative content when considered in their surrounding context

Sequences Trees Graphs

Learning with Structured Data

Vectorial dataset Structured dataset

Learning from a population where each individual is a fixed-size vector

Learning from a population where each individual is a variable size graph (vectorial information as node labels)

ML@UNIPI (since 1993)

Recursive Neural Networks

A neural model that can unfold on the structure of the sample

d c b a c

d c b c a

Prediction for the whole structure Neural encoding of the nodes

From Image to Graph Convolutions

Image

… …

Graph Learn hidden neurons responsive to visual patterns Learning hidden neurons responsive to structural patterns

• Node labels
• Connectivity

Community Detection

Community detection in social graphs

Kipf & Welling, Semi-Supervised Classification with Graph Convolutional Networks, ICLR 2017

BioMedical IoT

Challenges

Applications

Trends

Structured* HPC Knowledge Transfer*

The Rise of Deep Learning...

…and biomedical applications slowly starting to catch up

Source: query on Scopus abstracts on Sept. 2017

some 50 review papers

500 1000 1500 2000 2500 3000 3500 2005 2007 2009 2011 2013 2015 2017 Deep Learning Deep Learning + Life Sciences

CNN for DNA/RNA Sequences (DeepBind)

T A G A C A T C T … …

927 CNN models predicting a binding score for transcription factors and RNA-binding proteins

1D convolutions on the input sequence train to respond to task-specific motifs

Alipanahi, Babak, et al. "Predicting the sequence specificities of DNA-and RNA- binding proteins by deep learning." Nature biotechnology 33.8 (2015): 831-838. http://tools.genes.toronto.edu/deepbind/

CNN for DNA Sequences

Deep learning visual training system designed for machine vision applications GPU accelerated CNN training

Digits ML@UNIPI

cag gcc taa cac atg caa gtc gaa cgg taa nag att gat agc ttg cta tca atg ctg acg anc ggc gga cgg gtg agt aat gcc tgg gaa tat acc ctg atg tgg gg gat aac tat tgg aaa cga tag cta ata… Triplet ID aaa 1 aac 2 … … taa 59 … … ttt 64

Triplet vocabulary Use ID as graylevel of the corresponding pixel

500K DNA sequences from 18 bacteria species transformed into images

Convolutions have to be 1D even if it is an image!

ML@UNIPI

Testing Deep Learning Acceleration

500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Dna100K Dna500K

CNN Training Time

P100 M40

Dell PowerEdge C4130

• 4xM40 12Gb
• 2 Xeon E5-2670v3
• 128GB RAM

Dell PowerEdge C4130

• 4xP100 16Gb PCIE
• 2 Xeon E5-2690v4
• 256GB RAM

3h.30m 3d.3h ML@UNIPI

Exploiting Clonal Diversity for Personalized Cancer Treatment

primary tumor Metastasis 1 Metastasis 2

Predicting the effect of chemioterapic drugs from patients clonal trees Non-Isomorph tree transduction

ML@UNIPI

Allele frequency information

BioMedical IoT

Challenges

Applications

Trends

Structured* HPC Knowledge Transfer*

Internet of Streams

Enormous amounts of heterogeneous sequential data

+

Adding actuation calls for increased adaptivity

Cloud Intelligence

Deep learning for sequences (LSTM,GRU,…) Do we really need:

• To transfer all our data to the

could for analytics

• Complex DL models for all our

tasks

Edge Intelligence

• Learning models that scale from

tiny (8KB) to large (or deep)

• Reservoir computing and

randomized networks ML@UNIPI

Distributed Intelligence as an IoT Service

ML@UNIPI Multiple learning primitives within the same neural machinery

• Supervised, anomaly detection

& feature selection Embedded learning, management and

• ver-the-air deployment

tuning to normality Identifying anomalies/novelties

Automating medical screening (from 30mins to 10secs)

BioMedical IoT

Challenges

Applications

Trends

Structured* HPC Knowledge Transfer*

Are We Really Building Adaptive Applications?

Probably yes.. if we consider agents and reinforcement learning Otherwise we use pre- programmed adaptation

Predictor created at development time

The Adaptivity Challenge

Learning Automation Standardization & Protocols Learning as a primitive

BioMedical IoT

Challenges

Applications

Trends

Structured* HPC Knowledge Transfer*

Different Forms of Parallelism?

Current deep learning accelerations based on stream/data parallelism Structures are irregular and require synchronization Branch&bound?

BioMedical IoT

Challenges

Applications

Trends

Structured* HPC Knowledge Transfer*

Sharing Learned Knowledge

A scalable approach for IoT applications Impacting also biomedical applications Reusing trained models

Hidden neural representation as a unifying language?

Deep Learning…

• …or learning representations from data
• Effective for the machine to perform predictions
• Not necessarily helping humans understand the underlying biological

process

• Structured information as a means to supply relational

knowledge

Upcoming life-science and IoT applications Success will depend on how key challenges will be addressed