Deep Learning on Graphs and Manifolds 1 Yuan YAO HKUST Based on - - PowerPoint PPT Presentation

Deep Learning on Graphs and Manifolds

Yuan YAO HKUST Based on Xavier Bresson et al.

1

Acknowledgement

A following-up course at HKUST: https://deeplearning-math.github.io/

Non-Euclidean Data?

Al Also chemistry, physics, social science, communication netwo works, etc. Gr Graphs/ Ne Netwo works

=

Graphs and Manifolds

Graphs Manifolds

Social Networks as Graphs and Features on Edges and Vertices

vertex gender age ... edge friendship frequency ...

Domain structure Data on a domain

Graphs and Manifolds may vary

3D shapes (different manifolds)

Community Detection

Molecule graph

Challenges

´ What geometric structure in images, speech, video, text, is exploited by CNNs? ´ How to leverage such structure on non-Euclidean domains?

Convolutional Networks on Euclidean Domain (e.g. LeNet for Images)

An An architecture for hi high-di dimens nsiona nal learni ning ng : Cu Curse of dimensionality : di dim(image) = 1024 x x 1024 ≈ 10 106 Fo For N=10 samples/dim ⇒ 10 101,

1,000, 000,000 000 po

point nts Co ConvNe Nets are po powerful ul to to solve e high-di dimens nsiona nal learni ning ng pr probl blems.

ConvNets on Euclidean Domains

Ma Main assumption : Da Data (ima mage, video, sound) is com compos

• siti

tion

• nal

al, it , it is is fo formed o

• f p

f patterns t that a are: Lo Local St Stationa nary Mu Multi-sc scale (hierarchical) Co ConvNe Nets le leverage th the e co compositi tionality ty str tructu cture e : Th They extract compositional features and feed them to classifier, re recommender, r, etc etc (en (end-to to-en end). ).

Co Computer Vision NL NLP Dr Drug disc scovery Ga Games

Key Property: Locality

Ne Neocognitron Fuk Fukus ushi hima 1980

Lo Locality y : Pr Property inspired by the human visual cortex system. Lo Local recept ptive ve fields ds (H (Hubel el, Wies esel el 1962) ) : Ac Activate in in t the p presence o

• f

f lo local fe features.

ier Bresson 9

Key Property: Stationarity (Invariance)

St Stationa narity y ⇔ Tr Translation invariance Gl Global invariance Lo Local stationa narity y ⇔ Si Similar pa patche hes are sha hared d acr acros

• ss the

e dat ata a dom

• mai

ain Lo Local inva nvarianc nce, e , essentia ial fo l for in intra-cl clas ass va variations ns

Key Property: Multiscale Representation

Mu Multi-sc scale : Si Simpl ple st structures s combine to compose se sl slightly more ab abstract act st structures, a , and s so o

• n,

, in in a a h hie ierarchic ical w l way. In Inspir ired b by b brain in vi visua ual pr primary y cortex x (V1 V1 and V2 V2 neurons).

Fe Featur ures learne ned d by by Co Conv nvNet be become inc ncreasing ngly more compl plex at de deepe per layers (Zeiler Zeiler, , Ferg ergus 2013)

How to avoid the curse of dimensionality?

Lo Locality y : Co Compact support kernels ⇒ O( O(1) parame meters pe per filter. St Stationa narity y : Co Convolutional operators ⇒ O( O(nlo logn) ) in gen gener eral al (FFT) an and O(n) ) for co compact ct ker ernel els. Mu Multi-sc scale : Do Downsamp mpling + + pooling ⇒ O( O(n)

Bresson

Implementation: Compositional Maps

fl = l-th image feature (R,G,B channels), dim(fl) = n × 1 g(k)

l

= l-th feature map, dim(g(k)

l

) = n(k)

l

× 1 Compositional features consist of multiple convolutional + pooling layers. Convolutional layer g(k)

l

= ⇠ qk1 X

l0=1

W(k)

l,l0 ? ⇠

qk2 X

l0=1

W(k1)

l,l0

? ⇠ · · · fl0 !!! Activation, e.g. ⇠(x) = max{x, 0} rectified linear unit (ReLU) Pooling g(k)

l

(x) = kg(k1)

l

(x0) : x0 2 N(x)kp p = 1, 2, or 1

Summary of ConvNets

Fi Filters localized in space (lo localit lity) Co Convolutional filters (st stationarity) Mu Multiple layers (mu multi-sc scale) O( O(1) pa parameters pe per filter (inde ndepe pende ndent nt of input nput image size n) O( O(n) com complex exity ty per er lay ayer er (f (filter tering g don

• ne

e in th the e spati atial al dom

• mai

ain)

Generalization to ConvNets on Graphs?

Ho How w to extend Co ConvNe Nets to gr grap aph-st structured data? As Assumption : No Non-Eu Euclidean da data is locally y stationa nary y and nd mani nifest hi hierarchi hical struc uctur ures. Ho How w to define com compos

• siti

tion

• nal

ality ty on

• n gr

grap aphs? ? (con (convol

• luti

tion

• n an

and pool

• oling

g on

• n gr

grap aphs) Ho How w to make them fa fast? ? (l (linear ear com complex exity ty)

Next:

´ Prof. Xavier Bresson, NTU

´ IPAM talk on Convolutional Neural Networks on Graphs ´ https://www.youtube.com/watch?v=v3jZRkvIOIM

´ Prof. Zhizhen ZHAO, UIUC

´ Seminar: Multi-Scale and Multi-Representation Learning on Graphs and Manifolds

Thank you!