Understanding Geometry of Encoder-Decoder CNNs (E-D CNNs) Jong - PowerPoint PPT Presentation

Understanding Geometry of Encoder-Decoder CNNs (E-D CNNs) Jong Chul Ye & Woon Kyoung Sung BISPL - BioImaging, Signal Processing and Learning Lab. Dept. Bio & Brain Engineering Dept. of Mathematical Sciences KAIST, Korea

E-D CNN for Inverse Problems CNN

E-D CNN for Inverse Problems CNN

E-D CNN for Inverse Problems CNN Successful applications to various inverse problems

Why Same Architecture Works for Different Inverse Problems ?

<latexit sha1_base64="1xOFabKTay95z7c6ZUXCSJWXZAE=">ACE3icbVDLSsNAFJ3UV62vqEs3g0UQkZKoBuh4MZlC/YBTQiTyaQdOpmEmYm0hIKf4MZfceNCEbdu3Pk3TpMutPXAhTPn3Mvce/yEUaks69soLS2vrK6V1ysbm1vbO+buXlvGqcCkhWMWi6PJGUk5aipFuIgiKfEY6/vBm6nfuiZA05ndqnBA3Qn1OQ4qR0pJnoyuHZlGHoXQYj3GYG+R0/hyBHFy1GUBbnomVWrZuWAi8SekSqYoeGZX04Q4zQiXGpOzZVqLcDAlFMSOTipNKkiA8RH3S05SjiEg3y2+awCOtBDCMhS6uYK7+nshQJOU48nVnhNRAzntT8T+vl6rwys0oT1JFOC4+ClMGVQynAcGACoIVG2uCsKB6V4gHSCsdIwVHYI9f/IiaZ/V7POa1byo1psPRxlcAOwTGwSWog1vQAC2AwSN4Bq/gzXgyXox346NoLRmzCPfBHxifP6mndg=</latexit> Classical Methods for Inverse Problems Step 1: Signal Representation coefficients X h b i , x i ˜ x = b i Synthesis i basis Analysis basis

<latexit sha1_base64="dRyoeK2luEuIb3X8ywuvlvflFPU=">ACEnicbZBNS8MwGMdTX+d8q3r0EhyCgoxWBb0IQy8eJ7gXWEtJ03QLS9OSpLJR9hm8+FW8eFDEqydvfhvTrgfd/EPgl/zPCTP308Ylcqyvo2FxaXldXKWnV9Y3Nr29zZbcs4FZi0cMxi0fWRJIxy0lJUMdJNBEGRz0jH97k9c4DEZLG/F6NE+JGqM9pSDFS2vLM49GVI9PIo9BRlAUE+jkyxPus4BM4ckRx8yaVbcKwXmwS6iBUk3P/HKCGKcR4QozJGXPthLlZkgoihmZVJ1UkgThIeqTnkaOIiLdrFhpAg+1E8AwFvpwBQv390SGIinHka87I6QGcraWm/VeqkKL92M8iRVhOPpQ2HKoIphng8MqCBYsbEGhAXVf4V4gATCSqdY1SHYsyvPQ/u0bp/VrbvzWuO6jKMC9sEBOAI2uANcAuaoAUweATP4BW8GU/Gi/FufExbF4xyZg/8kfH5AyCNnR8=</latexit> Classical Methods for Inverse Problems Step 2: Basis Search by Optimization Eg. Compressed Sensing x = X ˜ b i h b i , x i i

Why do They Look so Different ? Any Link between Them ?

<latexit sha1_base64="DaFmbtzayW3V2tBvW3rbADydJY=">ACGXicbZDLSsNAFIYnXmu9RV26GSxCBSmJCroRim5cVrAXaEKYTCbt0MkzEykIfQ13Pgqblwo4lJXvo3TNoK2/jDw851zOHN+P2FUKsv6MhYWl5ZXVktr5fWNza1tc2e3JeNUYNLEMYtFx0eSMpJU1HFSCcRBEU+I21/cD2ut+JkDTmdypLiBuhHqchxUhp5JlWdunINPIohA5DvMcI9D1aHR4dw6EjpsBRlAU/3DMrVs2aCM4buzAVUKjhmR9OEOM0IlxhqTs2lai3BwJRTEjo7KTSpIgPEA90tWo4hIN59cNoKHmgQwjIV+XMEJ/T2Ro0jKLPJ1Z4RUX87WxvC/WjdV4YWbU56kinA8XRSmDKoYjmOCARUEK5Zpg7Cg+q8Q95FAWOkwyzoEe/bkedM6qdmnNev2rFK/KuIogX1wAKrABuegDm5AzQBg/gCbyAV+PReDbejPdp64JRzOyBPzI+vwEaXJ8Y</latexit> Our Theoretical Findings h b i ( x ) , x i ˜ y = X b i ( x ) i

<latexit sha1_base64="DaFmbtzayW3V2tBvW3rbADydJY=">ACGXicbZDLSsNAFIYnXmu9RV26GSxCBSmJCroRim5cVrAXaEKYTCbt0MkzEykIfQ13Pgqblwo4lJXvo3TNoK2/jDw851zOHN+P2FUKsv6MhYWl5ZXVktr5fWNza1tc2e3JeNUYNLEMYtFx0eSMpJU1HFSCcRBEU+I21/cD2ut+JkDTmdypLiBuhHqchxUhp5JlWdunINPIohA5DvMcI9D1aHR4dw6EjpsBRlAU/3DMrVs2aCM4buzAVUKjhmR9OEOM0IlxhqTs2lai3BwJRTEjo7KTSpIgPEA90tWo4hIN59cNoKHmgQwjIV+XMEJ/T2Ro0jKLPJ1Z4RUX87WxvC/WjdV4YWbU56kinA8XRSmDKoYjmOCARUEK5Zpg7Cg+q8Q95FAWOkwyzoEe/bkedM6qdmnNev2rFK/KuIogX1wAKrABuegDm5AzQBg/gCbyAV+PReDbejPdp64JRzOyBPzI+vwEaXJ8Y</latexit> Our Theoretical Findings h b i ( x ) , x i ˜ y = X b i ( x ) i

<latexit sha1_base64="DaFmbtzayW3V2tBvW3rbADydJY=">ACGXicbZDLSsNAFIYnXmu9RV26GSxCBSmJCroRim5cVrAXaEKYTCbt0MkzEykIfQ13Pgqblwo4lJXvo3TNoK2/jDw851zOHN+P2FUKsv6MhYWl5ZXVktr5fWNza1tc2e3JeNUYNLEMYtFx0eSMpJU1HFSCcRBEU+I21/cD2ut+JkDTmdypLiBuhHqchxUhp5JlWdunINPIohA5DvMcI9D1aHR4dw6EjpsBRlAU/3DMrVs2aCM4buzAVUKjhmR9OEOM0IlxhqTs2lai3BwJRTEjo7KTSpIgPEA90tWo4hIN59cNoKHmgQwjIV+XMEJ/T2Ro0jKLPJ1Z4RUX87WxvC/WjdV4YWbU56kinA8XRSmDKoYjmOCARUEK5Zpg7Cg+q8Q95FAWOkwyzoEe/bkedM6qdmnNev2rFK/KuIogX1wAKrABuegDm5AzQBg/gCbyAV+PReDbejPdp64JRzOyBPzI+vwEaXJ8Y</latexit> Our Theoretical Findings h b i ( x ) , x i ˜ y = X b i ( x ) i

<latexit sha1_base64="DaFmbtzayW3V2tBvW3rbADydJY=">ACGXicbZDLSsNAFIYnXmu9RV26GSxCBSmJCroRim5cVrAXaEKYTCbt0MkzEykIfQ13Pgqblwo4lJXvo3TNoK2/jDw851zOHN+P2FUKsv6MhYWl5ZXVktr5fWNza1tc2e3JeNUYNLEMYtFx0eSMpJU1HFSCcRBEU+I21/cD2ut+JkDTmdypLiBuhHqchxUhp5JlWdunINPIohA5DvMcI9D1aHR4dw6EjpsBRlAU/3DMrVs2aCM4buzAVUKjhmR9OEOM0IlxhqTs2lai3BwJRTEjo7KTSpIgPEA90tWo4hIN59cNoKHmgQwjIV+XMEJ/T2Ro0jKLPJ1Z4RUX87WxvC/WjdV4YWbU56kinA8XRSmDKoYjmOCARUEK5Zpg7Cg+q8Q95FAWOkwyzoEe/bkedM6qdmnNev2rFK/KuIogX1wAKrABuegDm5AzQBg/gCbyAV+PReDbejPdp64JRzOyBPzI+vwEaXJ8Y</latexit> Our Theoretical Findings Encoder analysis basis h b i ( x ) , x i ˜ y = X b i ( x ) i

<latexit sha1_base64="DaFmbtzayW3V2tBvW3rbADydJY=">ACGXicbZDLSsNAFIYnXmu9RV26GSxCBSmJCroRim5cVrAXaEKYTCbt0MkzEykIfQ13Pgqblwo4lJXvo3TNoK2/jDw851zOHN+P2FUKsv6MhYWl5ZXVktr5fWNza1tc2e3JeNUYNLEMYtFx0eSMpJU1HFSCcRBEU+I21/cD2ut+JkDTmdypLiBuhHqchxUhp5JlWdunINPIohA5DvMcI9D1aHR4dw6EjpsBRlAU/3DMrVs2aCM4buzAVUKjhmR9OEOM0IlxhqTs2lai3BwJRTEjo7KTSpIgPEA90tWo4hIN59cNoKHmgQwjIV+XMEJ/T2Ro0jKLPJ1Z4RUX87WxvC/WjdV4YWbU56kinA8XRSmDKoYjmOCARUEK5Zpg7Cg+q8Q95FAWOkwyzoEe/bkedM6qdmnNev2rFK/KuIogX1wAKrABuegDm5AzQBg/gCbyAV+PReDbejPdp64JRzOyBPzI+vwEaXJ8Y</latexit> Our Theoretical Findings Decoder Encoder analysis basis synthesis basis h b i ( x ) , x i ˜ y = X b i ( x ) i

<latexit sha1_base64="bo3reUJLRgiLys4OrWvNpVArY=">ACJ3icbVBNSwMxEM36bf2qevQSLIHKbsq6KUievGoaK3QrSWbzrah2eySzIpl8d948a94EVREj/4T03YP2joQePePCbzgkQKg675UxMTk3PzM7NFxYWl5ZXiqtr1yZONYcqj2WsbwJmQAoFVRQo4SbRwKJAQi3onvb12h1oI2J1hb0EGhFrKxEKztBSzeJRr+KjkC2gJ/Tk1sc4ofe0Qn2TRk1BfclUWwK936FBv9XDNndYqlksuWV3UHQceDkokbzOm8VXvxXzNAKFXDJj6p6bYCNjGgWX8FDwUwMJ413WhrqFikVgGtngzge6ZkWDWNtn0I6YH87MhYZ04sCOxkx7JhRrU/+p9VTDA8bmVBJiqD4cFGYSox7YdGW0IDR9mzgHEt7F8p7zDNONpoCzYEb/TkcXC9W/b2yu7Ffun4Mo9jmyQTbJNPHJAjskZOSdVwskjeSZv5N15cl6cD+dzODrh5J518qec7x9ZY6R3</latexit> Linear E-D CNN X y = ˜ BB > x = h x, b i i ˜ b i i un-pooling pooling Learned filters

<latexit sha1_base64="bo3reUJLRgiLys4OrWvNpVArY=">ACJ3icbVBNSwMxEM36bf2qevQSLIHKbsq6KUievGoaK3QrSWbzrah2eySzIpl8d948a94EVREj/4T03YP2joQePePCbzgkQKg675UxMTk3PzM7NFxYWl5ZXiqtr1yZONYcqj2WsbwJmQAoFVRQo4SbRwKJAQi3onvb12h1oI2J1hb0EGhFrKxEKztBSzeJRr+KjkC2gJ/Tk1sc4ofe0Qn2TRk1BfclUWwK936FBv9XDNndYqlksuWV3UHQceDkokbzOm8VXvxXzNAKFXDJj6p6bYCNjGgWX8FDwUwMJ413WhrqFikVgGtngzge6ZkWDWNtn0I6YH87MhYZ04sCOxkx7JhRrU/+p9VTDA8bmVBJiqD4cFGYSox7YdGW0IDR9mzgHEt7F8p7zDNONpoCzYEb/TkcXC9W/b2yu7Ffun4Mo9jmyQTbJNPHJAjskZOSdVwskjeSZv5N15cl6cD+dzODrh5J518qec7x9ZY6R3</latexit> Linear E-D CNN w/ Skipped Connection X y = ˜ BB > x = h x, b i i ˜ b i i more redundant expression Learned filters

<latexit sha1_base64="9EuOyjKGC2x9hAgBpajvIdywLlA=">ACJ3icbVBNSwMxEM36bf2qevQSLIHKbsq6EWRevGoaK3QXZdsOq2h2eySzEpL6b/x4l/xIqiIHv0npu0etDoQePePCbzolQKg676UxMTk3PzM7NFxYWl5ZXiqtr1ybJNIcqT2SibyJmQAoFVRQo4SbVwOJIQi1qnw702j1oIxJ1hd0Ugpi1lGgKztBSYfG4c+SjkA2gFVq59TFJaYceUd9kcSioL5lqSaCdHRoNWj1qc4elwmLJLbvDon+Bl4MSyes8L74jYRnMSjkhlT9wUgx7TKLiEfsHPDKSMt1kL6hYqFoMJesM7+3TLMg3aTLR9CumQ/enosdiYbhzZyZjhnRnXBuR/Wj3D5mHQEyrNEBQfLWpmkmJCB6HRhtDAUXYtYFwL+1fK75hmHG20BRuCN37yX3C9W/b2yu7FfunkMo9jmyQTbJNPHJATsgZOSdVwskDeSKv5M15dJ6d+djNDrh5J518qucr29XoqR2</latexit> Deep Convolutional Framelets Perfect reconstruction X h x, b i i ˜ x = ˜ BB > x = b i i Frame conditions w/o skipped connection w skipped connection Ye et al, SIAM J. Imaging Science, 2018

<latexit sha1_base64="T/1m1u26m8O8vLHErH3u6EKQhAM=">ACM3icbVDLSgMxFM3Ud31VXboJFkFByowKuhGKbsSVotVCpw6Z9LYNZjJDckdaSv/JjT/iQhAXirj1H0zbWaj1QsLhPEjuCRMpDLrui5ObmJyanpmdy8vLC4tF1ZWr02cag4VHstYV0NmQAoFRQoZpoYFEo4Sa8OxnoN/egjYjVFXYTqEespURTcIaWCgpn3SMfhWwAPd7qbA+vWx/jhHboEfVNGgWC+pKplgTa2aFhIAY2X4+YLDpig0LRLbnDoePAy0CRZHMeFJ78RszTCBRyYypeW6C9R7TKLiEft5PDSM37EW1CxULAJT7w137tNyzRoM9b2KRD9meixyJjulFonRHDtvmrDcj/tFqKzcN6T6gkRVB89FAzlRjOiQNoQGjrJrAeNa2L9S3macbQ1520J3t+Vx8H1bsnbK7kX+8XyZVbHLFknG2SLeOSAlMkpOScVwskDeSZv5N15dF6dD+dzZM05WaN/Brn6xvQsKgT</latexit> <latexit sha1_base64="1HS4n8UkvGQcnzeL2YdPrnXeg=">ACkXicbVHLSgMxFM2M7/qunQTLIpuyowK6kIpuhHcKFoVmjpkMrdtMkMSUYsQ/H73Hn35i2g4/qhQvnNfuYkzwY0Ng/Pn5qemZ2bX6gsLi2vrFbX1u9NmsGTZaKVD/G1IDgCpqWwGPmQYqYwEP8fPFUH94AW14qu5sP4O2pF3FO5xR6io+kZueVfSJ7H7uodPMYmhy1URS2o1fx1gYoZqFOIdHDgnLEmtGQWEjJlxwv6kWCEvZfQNkt+MSwom+5btChkJNxU8rVKVK0F9WBk+C8IS1BDpV1H1XeSpCyXoCwT1JhWGS2XVBtORMwqJDcQEbZM+1Cy0FJZh2MbroAG87JsGdVDtXFo/YnxUFlcb0Zewy3X49M6kNyf+0Vm47x+2Cqy3oNh4UCcX2KZ4+D04RqYFX0HKNPc7YpZj2rKrPvEijtCOPnkv+B+vx4e1IObw1rjvDzHPNpEW2gXhegINdAlukZNxLwV79A79c78Df/Eb/hlru+VNRvol/lXn1/Lv9k=</latexit> Role of ReLUs? Generator for Multiple Expressions y = ˜ h x, b i ( x ) i ˜ X B ( x ) B ( x ) > x = b i ( x ) i   0 0 σ 1 · · · 0 0 σ 2 Input dependent {0,1} matrix · · ·   Σ l ( x ) = . . .   ... . . .   . . . --> Input adaptivity   0 0 σ m l · · ·

Input Space Partitioning for Multiple Expressions

Expressivity of E-D CNN # of representation # of network elements

Expressivity of E-D CNN # of channel # of representation # of network elements

Expressivity of E-D CNN # of channel Network depth # of representation # of network elements

Expressivity of E-D CNN Skipped connection # of channel Network depth # of representation # of network elements