CMP722 ADVANCED COMPUTER VISION Lecture #9 Graph Networks Aykut - PowerPoint PPT Presentation

Illustration: Kevin Hong // Quanta Magazine CMP722 ADVANCED COMPUTER VISION Lecture #9 – Graph Networks Aykut Erdem // Hacettepe University // Spring 2019

Illustration: StyleGAN trained on Portrait by Yuli-Ban Previously on CMP722 • image synthesis via generative models • conditional generative models • structured vs unstructured prediction • image-to-image translation • generative adversarial networks • cycle-consistent adversarial networks

Lecture overview • graph structured data • graph neural nets (GNNs) • GNNs for ”classical” network problems • Disclaimer: Much of the material and slides for this lecture were borrowed from — Yujia Li and Oriol Vinyals' tutorial on Graph Nets — Thomas Kipf’s talk on structured deep models: deep Learning on graphs and beyond 3

Deep Learning Speech data Grid games Natural language processing (NLP) Deep neural al nets s that at exp xploit: • translation equivariance (weight sharing) • hierarchical compositionality 4

Modeling Structured Data Data with Rigid Structure Unstructured Data Graph Structured Data sequences output visual data 5

Modeling Structured Data Data with Rigid Structure Unstructured Data Graph Structured Data sequences output visual data 6

Graph structured data • A lot of real-world data does not “live” on grids Social networks Knowledge graphs Citation networks Communication networks Multi-agent systems Molecules Protein interaction networks Standard deep learning architectures like CNNs and RNNs don’t work here! Road maps 7

Recipe for a good model for graphs • Handle different types of graph prediction problems Requires: Representations for graphs, nodes and edges • Handle graphs of varying sizes and structure Requires: A parametrization independent of graph size and structure • Handle arbitrary node ordering Requires: A model invariant to node permutations • Utilize graph structure Requires: A mechanism to communicate information on graphs 8

<latexit sha1_base64="zubxHmSomD0kX0Gah9ugDAcKCiM=">ACD3icbVDLSgMxFL1TX7W+Rl26CRbFVZmpgi6LgriSKvYBnVoyaYNzWSGJCOUYf7Ajb/ixoUibt26829MHwtPRA4Oede7r3HjzlT2nG+rdzC4tLySn61sLa+sblb+/UVZRIQmsk4pFs+lhRzgStaY5bcaS4tDntOEPLkZ+4FKxSJxp4cxbYe4J1jACNZG6tiHXoh13w/SZoY8JtDk6e32X16jTzNQqrQZdaxi07JGQPNE3dKijBFtWN/ed2IJCEVmnCsVMt1Yt1OsdSMcJoVvETRGJMB7tGWoQKbOe10fE+GDozSRUEkzRMajdXfHSkOlRqGvqkcratmvZH4n9dKdHDWTpmIE0FmQwKEo50hEbhoC6TlGg+NAQTycyuiPSxESbCAsmBHf25HlSL5fc41L5qRYOZ/GkYc92IcjcOEUKnAFVagBgUd4hld4s56sF+vd+piU5qxpzy78gfX5A8EhnHg=</latexit> <latexit sha1_base64="tK0GwrpjZkzyEoIt4K5bmTeux0o=">ACDXicbVDLSsNAFJ3UV62vqEs3g1WoICWpgm6EqgtdVrAPaEOZTCft0MkzEyEvIDbvwVNy4UcevenX/jJI2g1QMDZ865l3vcUNGpbKsT6MwN7+wuFRcLq2srq1vmJtbLRlEApMmDlgOi6ShFOmoqRjqhIMh3GWm748vUb98RIWnAb9UkJI6Phpx6FCOlpb651/ORGmHE4qvkrJ9XC8+Tw7hN+8kB32zbFWtDPAvsXNSBjkafOjNwhw5BOuMENSdm0rVE6MhKYkaTUiyQJER6jIelqypFPpBNn1yRwXysD6AVCP65gpv7siJEv5cR3dW6opz1UvE/rxsp79SJKQ8jRTieDvIiBlUA02jgAqCFZtogrCgeleIR0grHSAJR2CPXvyX9KqVe2jau3muFy/yOMogh2wCyrABiegDq5BAzQBvfgETyDF+PBeDJejbdpacHIe7bBLxjvXw/Ym4s=</latexit> <latexit sha1_base64="iX9ruAlInZe7ksrDjGls+70X4No=">ACD3icbVC7TsMwFHV4lvIKMLJYVCmKilIMBZYmKqC6ENqSuW4TmvVcSL7BqmK8gcs/AoLAwixsrLxN7iPAVqOZOn4nHt17z1+LgGx/m2FhaXldWc2v59Y3NrW17Z7euo0RVqORiFTJ5oJLlkNOAjWjBUjoS9Ywx9cjfzGA1OaR/IOhjFrh6QnecApASN17CMvJND3g/Qiwx6XePL109vsPq1gD3jINK5kHbvgFJ0x8Dxp6SApqh27C+vG9EkZBKoIFq3XCeGdkoUcCpYlvcSzWJCB6THWoZKYua0/E9GT40ShcHkTJPAh6rvztSEmo9DH1TOVpXz3oj8T+vlUBw3k65jBNgk4GBYnAEOFROLjLFaMghoYQqrjZFdM+UYSCiTBvQnBnT54n9VLRPSmWbk4L5ctpHDm0jw7QMXLRGSqja1RFNUTRI3pGr+jNerJerHfrY1K6YE179tAfWJ8/p9KcaQ=</latexit> Graph Neural Networks (GNNs) Notation: G = ( A , X ) Hidden layer Hidden layer A ∈ R N × N Adjacency matrix X ∈ R N × F Feature matrix Input Output ReLU ReLU a: Pass messages between pairs of nodes & agglomerate • Mai ain idea: 9

Recap: Convolutional neural networks (on grids) Single CNN layer with 3x3 filter: (Animation by Vincent Dumoulin) 10

<latexit sha1_base64="qbGjBof6uWNMbDuRhmy/fSLnPs=">AB9XicbVDLSgMxFL3xWeur6tJNsAiuykwVdFl047KCfUA7lkyaUMzmSHJKGWY/3DjQhG3/os7/8ZMOwtPRA4nHMv9+T4seDaOM43WldW9/YLG2Vt3d29/YrB4dtHSWKshaNRKS6PtFMcMlahvBurFiJPQF6/iTm9zvPDKleSTvzTRmXkhGkgecEmOlh35IzNgP0nE2SJ1sUKk6NWcGvEzcglShQHNQ+eoPI5qETBoqiNY914mNlxJlOBUsK/cTzWJCJ2TEepZKEjLtpbPUGT61yhAHkbJPGjxTf2+kJNR6Gvp2Mk+pF71c/M/rJSa48lIu48QwSeHgkRgE+G8AjzkilEjpYQqrjNiumYKEKNLapsS3AXv7xM2vWae16r31UG9dFHSU4hM4AxcuoQG30IQWUFDwDK/whp7QC3pH/PRFVTsHMEfoM8f3d+Svw=</latexit> <latexit sha1_base64="phTuK/mdfKyPkPZcnwtHvB/ft8=">AB9XicbVDLSgMxFL3xWeur6tJNsAiuykwVdFl047KCfUA7lkyaUMzmSHJKGWY/3DjQhG3/os7/8ZMOwtPRA4nHMv9+T4seDaOM43WldW9/YLG2Vt3d29/YrB4dtHSWKshaNRKS6PtFMcMlahvBurFiJPQF6/iTm9zvPDKleSTvzTRmXkhGkgecEmOlh35IzNgP0nE2SN1sUKk6NWcGvEzcglShQHNQ+eoPI5qETBoqiNY914mNlxJlOBUsK/cTzWJCJ2TEepZKEjLtpbPUGT61yhAHkbJPGjxTf2+kJNR6Gvp2Mk+pF71c/M/rJSa48lIu48QwSeHgkRgE+G8AjzkilEjpYQqrjNiumYKEKNLapsS3AXv7xM2vWae16r31UG9dFHSU4hM4AxcuoQG30IQWUFDwDK/whp7QC3pH/PRFVTsHMEfoM8f32SwA=</latexit> <latexit sha1_base64="Jr5tVz/PSl7Bu7UvRtELReweTFI=">AB9XicbVDLSgMxFL3xWeur6tJNsAiuykwVdFl047KCfUA7lkyaUMzmSHJKGWY/3DjQhG3/os7/8ZMOwtPRA4nHMv9+T4seDaOM43WldW9/YLG2Vt3d29/YrB4dtHSWKshaNRKS6PtFMcMlahvBurFiJPQF6/iTm9zvPDKleSTvzTRmXkhGkgecEmOlh35IzNgP0nE2SHk2qFSdmjMDXiZuQapQoDmofPWHEU1CJg0VROue68TGS4kynAqWlfuJZjGhEzJiPUslCZn20lnqDJ9aZYiDSNknDZ6pvzdSEmo9DX07mafUi14u/uf1EhNceSmXcWKYpPNDQSKwiXBeAR5yxagRU0sIVdxmxXRMFKHGFlW2JbiLX14m7XrNPa/V7y6qjeuijhIcwmcgQuX0IBbaEILKCh4hld4Q0/oBb2j/noCip2juAP0OcPNIuS+A=</latexit> <latexit sha1_base64="D8NZwhGRc3SadH+lq9NyH2X2S6M=">AB7HicbVBNS8NAFHzxs9avqkcvi0XwVJIq6LHoxWMF0xbaUDbTbt0swm7L0IJ/Q1ePCji1R/kzX/jts1BWwcWhpk37HsTplIYdN1vZ219Y3Nru7RT3t3bPzisHB23TJpxn2WyER3Qmq4FIr7KFDyTqo5jUPJ2+H4bua3n7g2IlGPOEl5ENOhEpFgFK3k9wYJmn6l6tbcOcgq8QpShQLNfuXL5lgWc4VMUmO6nptikFONgk+Lfcyw1PKxnTIu5YqGnMT5PNlp+TcKgMSJdo+hWSu/k7kNDZmEod2MqY4MsveTPzP62Y3QS5UGmGXLHFR1EmCSZkdjkZCM0ZyoklGlhdyVsRDVlaPsp2xK85ZNXSate8y5r9YerauO2qKMEp3AGF+DBNTgHprgAwMBz/AKb45yXpx352MxuYUmRP4A+fzB/K2jsY=</latexit> Recap: Convolutional neural networks (on grids) Single CNN layer with 3x3 filter: h 0 h 1 . . . h i (Animation by Vincent Dumoulin) 11

<latexit sha1_base64="phTuK/mdfKyPkPZcnwtHvB/ft8=">AB9XicbVDLSgMxFL3xWeur6tJNsAiuykwVdFl047KCfUA7lkyaUMzmSHJKGWY/3DjQhG3/os7/8ZMOwtPRA4nHMv9+T4seDaOM43WldW9/YLG2Vt3d29/YrB4dtHSWKshaNRKS6PtFMcMlahvBurFiJPQF6/iTm9zvPDKleSTvzTRmXkhGkgecEmOlh35IzNgP0nE2SN1sUKk6NWcGvEzcglShQHNQ+eoPI5qETBoqiNY914mNlxJlOBUsK/cTzWJCJ2TEepZKEjLtpbPUGT61yhAHkbJPGjxTf2+kJNR6Gvp2Mk+pF71c/M/rJSa48lIu48QwSeHgkRgE+G8AjzkilEjpYQqrjNiumYKEKNLapsS3AXv7xM2vWae16r31UG9dFHSU4hM4AxcuoQG30IQWUFDwDK/whp7QC3pH/PRFVTsHMEfoM8f32SwA=</latexit> <latexit sha1_base64="lLk1F27F8DWO3iEmWaQeT2KfeA=">ACnicbVDLSsNAFJ34rPUVdelmtAiuSlIFXRYFcVnFPqCJZTKdtEMnkzAzEcqQtRt/xY0LRdz6Be78GydtFtp64MLhnHu5954gYVQqx/m2FhaXldWS2vl9Y3NrW17Z7cl41Rg0sQxi0UnQJIwyklTUcVIJxERQEj7WB0mfvtByIkjfmdGifEj9CA05BipIzUsw+8CKlhEOph1tM0gx7lcCoF+ja71dZz64VWcCOE/cglRAgUbP/vL6MU4jwhVmSMqu6yTK10goihnJyl4qSYLwCA1I1COIiJ9PXklg0dG6cMwFqa4ghP194RGkZTjKDCd+ZVy1svF/7xuqsJzX1OepIpwPF0UpgyqGOa5wD4VBCs2NgRhQc2tEA+RQFiZ9MomBHf25XnSqlXdk2rt5rRSvyjiKIF9cAiOgQvOQB1cgwZoAgwewTN4BW/Wk/VivVsf09YFq5jZA39gf4A7OubCg=</latexit> <latexit sha1_base64="D8NZwhGRc3SadH+lq9NyH2X2S6M=">AB7HicbVBNS8NAFHzxs9avqkcvi0XwVJIq6LHoxWMF0xbaUDbTbt0swm7L0IJ/Q1ePCji1R/kzX/jts1BWwcWhpk37HsTplIYdN1vZ219Y3Nru7RT3t3bPzisHB23TJpxn2WyER3Qmq4FIr7KFDyTqo5jUPJ2+H4bua3n7g2IlGPOEl5ENOhEpFgFK3k9wYJmn6l6tbcOcgq8QpShQLNfuXL5lgWc4VMUmO6nptikFONgk+Lfcyw1PKxnTIu5YqGnMT5PNlp+TcKgMSJdo+hWSu/k7kNDZmEod2MqY4MsveTPzP62Y3QS5UGmGXLHFR1EmCSZkdjkZCM0ZyoklGlhdyVsRDVlaPsp2xK85ZNXSate8y5r9YerauO2qKMEp3AGF+DBNTgHprgAwMBz/AKb45yXpx352MxuYUmRP4A+fzB/K2jsY=</latexit> <latexit sha1_base64="qbGjBof6uWNMbDuRhmy/fSLnPs=">AB9XicbVDLSgMxFL3xWeur6tJNsAiuykwVdFl047KCfUA7lkyaUMzmSHJKGWY/3DjQhG3/os7/8ZMOwtPRA4nHMv9+T4seDaOM43WldW9/YLG2Vt3d29/YrB4dtHSWKshaNRKS6PtFMcMlahvBurFiJPQF6/iTm9zvPDKleSTvzTRmXkhGkgecEmOlh35IzNgP0nE2SJ1sUKk6NWcGvEzcglShQHNQ+eoPI5qETBoqiNY914mNlxJlOBUsK/cTzWJCJ2TEepZKEjLtpbPUGT61yhAHkbJPGjxTf2+kJNR6Gvp2Mk+pF71c/M/rJSa48lIu48QwSeHgkRgE+G8AjzkilEjpYQqrjNiumYKEKNLapsS3AXv7xM2vWae16r31UG9dFHSU4hM4AxcuoQG30IQWUFDwDK/whp7QC3pH/PRFVTsHMEfoM8f3d+Svw=</latexit> <latexit sha1_base64="Jr5tVz/PSl7Bu7UvRtELReweTFI=">AB9XicbVDLSgMxFL3xWeur6tJNsAiuykwVdFl047KCfUA7lkyaUMzmSHJKGWY/3DjQhG3/os7/8ZMOwtPRA4nHMv9+T4seDaOM43WldW9/YLG2Vt3d29/YrB4dtHSWKshaNRKS6PtFMcMlahvBurFiJPQF6/iTm9zvPDKleSTvzTRmXkhGkgecEmOlh35IzNgP0nE2SHk2qFSdmjMDXiZuQapQoDmofPWHEU1CJg0VROue68TGS4kynAqWlfuJZjGhEzJiPUslCZn20lnqDJ9aZYiDSNknDZ6pvzdSEmo9DX07mafUi14u/uf1EhNceSmXcWKYpPNDQSKwiXBeAR5yxagRU0sIVdxmxXRMFKHGFlW2JbiLX14m7XrNPa/V7y6qjeuijhIcwmcgQuX0IBbaEILKCh4hld4Q0/oBb2j/noCip2juAP0OcPNIuS+A=</latexit> Recap: Convolutional neural networks (on grids) Single CNN layer with 3x3 filter: h 0 h 1 . . . h i h i ∈ R F are (hidden layer) activations of a pixel/node (Animation by Vincent Dumoulin) 12