deep neural architectures for structured output problems
play

Deep neural architectures for structured output problems Clment - PowerPoint PPT Presentation

Dec 20, 2023 •760 likes •1.37k views

Deep neural architectures for structured output problems Clment Chatelain Soufiane Belharbi soufiane.belharbi@litislab.fr clement.chatelain@insa-rouen.fr LITIS - INSA de Rouen Joint work with: J.Lerouge, R.Herault, S.Adam, R.Modzelewski,

  1. Deep neural architectures for structured output problems Clément Chatelain Soufiane Belharbi soufiane.belharbi@litislab.fr clement.chatelain@insa-rouen.fr LITIS - INSA de Rouen Joint work with: J.Lerouge, R.Herault, S.Adam, R.Modzelewski, F.Jardin, B.Labbe May 19, 2015 images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems

  2. Plan 1 Structured Output Problems 2 Input/Output Deep Architecture (IODA) 3 Application of IODA to medical image labeling 4 Application of IODA to Facial Landmark Detection 5 Conclusion 6 Future Work on IODA images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 1/46

  3. Structured Output Problems Traditional Machine Learning Problems f : X → y Inputs X ∈ R d : any type of input Outputs y ∈ R for the task: classification, regression, . . . Machine Learning for Structured Output Problems f : X → Y Inputs X ∈ R d : any type of input Outputs Y ∈ R d ′ , d ′ > 1 a structured object ( dependencies ) See C. Lampert slides [3]. images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 2/46

  4. Structured Output Problems Traditional Machine Learning Problems f : X → y Inputs X ∈ R d : any type of input Outputs y ∈ R for the task: classification, regression, . . . Machine Learning for Structured Output Problems f : X → Y Inputs X ∈ R d : any type of input Outputs Y ∈ R d ′ , d ′ > 1 a structured object ( dependencies ) See C. Lampert slides [3]. images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 2/46

  5. Structured Output Problems Data = representation ( values ) + structure ( dependencies ) Text: part-of-speech tagging, translation speech ⇄ text Protein folding Image Structured data images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 3/46

  6. Structured Output Problems Approaches that Deal with Structured Output Data ◮ Kernel based methods: Kernel Density Estimation (KDE) ◮ Discriminative methods: Structure output SVM ◮ Graphical methods: HMM, CRF , MRF , . . . Drawbacks Perform one single data transformation Difficult to deal with high dimensional data Ideal approach ◮ Structured output problems ◮ High dimension data ◮ Multiple data transformation (complex mapping functions) Deep neural networks? images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 4/46

  7. Structured Output Problems Approaches that Deal with Structured Output Data ◮ Kernel based methods: Kernel Density Estimation (KDE) ◮ Discriminative methods: Structure output SVM ◮ Graphical methods: HMM, CRF , MRF , . . . Drawbacks Perform one single data transformation Difficult to deal with high dimensional data Ideal approach ◮ Structured output problems ◮ High dimension data ◮ Multiple data transformation (complex mapping functions) Deep neural networks? images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 4/46

  8. Structured Output Problems Traditional Deep neural Network Input layer Hidden layer 1 Hidden layer 2 Hidden layer 3 Hidden layer 4 Output layer x 1 x 2 x 3 y 1 car x 4 y 2 bus y 3 x 5 bike x 6 x 7 ◮ High dimension data OK ◮ Multiple data transformation (complex mapping functions) OK ◮ Structured output problems NO images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 5/46

  9. Input/Output Deep Architecture (IODA) Plan 1 Structured Output Problems 2 Input/Output Deep Architecture (IODA) 3 Application of IODA to medical image labeling 4 Application of IODA to Facial Landmark Detection 5 Conclusion 6 Future Work on IODA images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 6/46

  10. Input/Output Deep Architecture (IODA) IODA: ◮ Incorporate the output structure by learning ◮ Discover hidden dependencies in the outputs Input layer Hidden layer 1 Hidden layer 2 Hidden layer 3 Hidden layer 4 Output layer x 1 y 1 x 2 y 2 x 3 y 3 x 4 y 4 Structured object x 5 y 5 x 6 y 6 y 7 x 7 images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 7/46

  11. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 x 1 x 2 x 3 x 4 x 5 x 6 x 7 images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 8/46

  12. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 x 1 x 2 x 3 x 4 x 5 x 6 x 7 images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 8/46

  13. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 x 1 x 2 x 3 x 4 x 5 x 6 x 7 images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 9/46

  14. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 x 1 x 2 x 3 x 4 x 5 x 6 x 7 R in ( x i ; θ in ) = ˆ x i images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 10/46

  15. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 Output layer x 1 y 1 y 2 x 2 x 3 y 3 x 4 y 4 x 5 y 5 x 6 y 6 x 7 y 7 R in ( x i ; θ in ) = ˆ x i images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 11/46

  16. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 Hidden layer 4 Output layer x 1 y 1 x 2 y 2 x 3 y 3 x 4 y 4 x 5 y 5 x 6 y 6 y 7 x 7 R in ( x i ; θ in ) = ˆ x i images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 12/46

  17. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 Hidden layer 3 Hidden layer 4 Output layer x 1 y 1 x 2 y 2 x 3 y 3 x 4 y 4 x 5 y 5 x 6 y 6 x 7 y 7 R in ( x i ; θ in ) = ˆ x i images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 13/46

  18. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 Hidden layer 3 Hidden layer 4 Output layer x 1 y 1 x 2 y 2 x 3 y 3 x 4 y 4 x 5 y 5 x 6 y 6 x 7 y 7 R in ( x i ; θ in ) = ˆ R out ( y i ; θ out ) = ˆ x i y i images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 14/46

  19. Input/Output Deep Architecture (IODA) Training IODA Input layer Hidden layer 1 Hidden layer 2 Hidden layer 3 Hidden layer 4 Output layer x 1 y 1 x 2 y 2 x 3 y 3 x 4 y 4 Back-propagation x 5 y 5 x 6 y 6 x 7 y 7 R in ( x i ; θ in ) = ˆ R out ( y i ; θ out ) = ˆ x i y i M ( x i ; θ , θ in , θ out ) = ˆ y i images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 15/46

  20. Input/Output Deep Architecture (IODA) IODA framework: min θ L ( θ , D ( x , y )) � n L ( θ , D ( x , y )) = 1 � C ( M ( x i ; θ , θ in , θ out ) , y i ) n � �� � i = 1 Learn ( input → output ) dependencies) + ℓ in ( R in ( x i ; θ in ) , x i ) � �� � Learn input dependencies � + ℓ out ( R out ( y i ; θ out ) , y i ) � �� � Learn output dependencies C ( . ) , ℓ in ( . ) , ℓ out ( . ) : defined costs. min θ L ( θ , D ( x , y )) is hard to solve ⇒ split L ( θ , D ( x , y )) images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 16/46

  21. Input/Output Deep Architecture (IODA) IODA framework: min θ L ( θ , D ( x , y )) � n L ( θ , D ( x , y )) = 1 � C ( M ( x i ; θ , θ in , θ out ) , y i ) n � �� � i = 1 Learn ( input → output ) dependencies) + ℓ in ( R in ( x i ; θ in ) , x i ) � �� � Learn input dependencies � + ℓ out ( R out ( y i ; θ out ) , y i ) � �� � Learn output dependencies C ( . ) , ℓ in ( . ) , ℓ out ( . ) : defined costs. min θ L ( θ , D ( x , y )) is hard to solve ⇒ split L ( θ , D ( x , y )) images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 16/46

  22. Input/Output Deep Architecture (IODA) Relaxed-simplified version of IODA Unsupervised training: 1 � n 1 → Input dependencies : min i = 1 ℓ in ( R in ( x i ; θ in ) , x i ) n θ in 1 � n → output dependencies: min i = 1 ℓ out ( R out ( y i ; θ out ) , y i ) n θ out Standard supervised learning: 2 � n 1 min i = 1 C ( M ( x i ; θ , θ in , θ out ) , y i ) n θ , θ in , θ out Open source implementation Implemented using our library: Crino [1] [ Python-Theano based] . images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 17/46

  23. Input/Output Deep Architecture (IODA) Relaxed-simplified version of IODA Unsupervised training: 1 � n 1 → Input dependencies : min i = 1 ℓ in ( R in ( x i ; θ in ) , x i ) n θ in 1 � n → output dependencies: min i = 1 ℓ out ( R out ( y i ; θ out ) , y i ) n θ out Standard supervised learning: 2 � n 1 min i = 1 C ( M ( x i ; θ , θ in , θ out ) , y i ) n θ , θ in , θ out Open source implementation Implemented using our library: Crino [1] [ Python-Theano based] . images/logos LITIS - INSA de Rouen Deep neural architectures for structured output problems 17/46

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Deep neural networks and structured output problems presentation of my current PhD work ISP

Deep neural networks and structured output problems presentation of my current PhD work ISP

Deep neural networks and structured output problems presentation of my current PhD work ISP seminar. UCL, Louvain-la-Neuve 2016 Soufiane Belharbi Romain Hrault Clment Chatelain Sbastien Adam soufiane.belharbi@insa-rouen.fr LITIS lab.,

1.27k views • 93 slides
Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating Deep (large) Neural Network (DNN) classifiers Neurons and DNN architectures Numerical training of DNNs (supervised deep learning) Spiking

630 views • 33 slides
Neural Map Structured Memory for Deep RL Emilio Parisotto eparisot@andrew.cmu.edu PhD Student

Neural Map Structured Memory for Deep RL Emilio Parisotto eparisot@andrew.cmu.edu PhD Student

Neural Map Structured Memory for Deep RL Emilio Parisotto eparisot@andrew.cmu.edu PhD Student Machine Learning Department Carnegie Mellon University Supervised Learning Deep Most deep learning problems are posed as supervised Neural

562 views • 53 slides
Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural

Optimizing Deep Neural Networks Leena Chennuru Vankadara 26-10-2015 Table of Contents Neural Networks and loss surfaces Problems of Deep Architectures Optimization in Neural Networks Under fitting Proliferation of Saddle

1.67k views • 41 slides
Complex Prediction Problems A novel approach to multiple Structured Output Prediction Yasemin

Complex Prediction Problems A novel approach to multiple Structured Output Prediction Yasemin

Complex Prediction Problems A novel approach to multiple Structured Output Prediction Yasemin Altun Max-Planck Institute ECML HLIE08 Yasemin Altun Complex Prediction Information Extraction Extract structured information from unstructured

826 views • 27 slides
Neural Map: Structured Memory for Deep Reinforcement Learning Emilio Parisotto and Ruslan

Neural Map: Structured Memory for Deep Reinforcement Learning Emilio Parisotto and Ruslan

CS 885 Reinforcement Learning Neural Map: Structured Memory for Deep Reinforcement Learning Emilio Parisotto and Ruslan Salakhutdinov, ICLR 2018 Presented by Andreas Stckel July 6, 2018 motivation: navigating partially observable

1.49k views • 53 slides
Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane

Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane

Unsupervised Learning and Inverse Problems with Deep Neural Networks J oan Bruna, Stphane Mallat, Ivan Dokmanic, Martin de Hoop cole Normale Suprieure www.di.ens.fr/data Deep Convolutional Networks ( x ) x ( u ) L j L 1 f M (

520 views • 47 slides
Deep Residual Output Layers for Neural Language Generation Nikolaos Pappas, James Henderson June

Deep Residual Output Layers for Neural Language Generation Nikolaos Pappas, James Henderson June

Deep Residual Output Layers for Neural Language Generation Nikolaos Pappas, James Henderson June 13, 2019 Neural language generation Cat sat on ? Probability distribution at time t given context vector h t R d , weights W R d |V|

474 views • 14 slides
8. Other Deep Architectures CS 519 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt

8. Other Deep Architectures CS 519 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt

8. Other Deep Architectures CS 519 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt Kira and Ian Goodfellow A brief overview of other architectures Unsupervised Architectures Deep Belief Networks Autoencoders GANs

439 views • 25 slides
Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural

Neural Information Retrieval Wassila Lalouani 1 Plan Neural network architectures Neural IR tasks Neural IR architecture Feature Representations Neural IR query auto completion Neural IR query suggestion Neural IR document

1.48k views • 18 slides
Learning Deep Architectures Using Kernel Modules Li Deng Microsoft Research, Redmond (thanks

Learning Deep Architectures Using Kernel Modules Li Deng Microsoft Research, Redmond (thanks

MLSLP 2012 Learning Deep Architectures Using Kernel Modules Li Deng Microsoft Research, Redmond (thanks collaborations/discussions with many people) Introduction Deep neural net (modern multilayer perceptron) Hard to

330 views • 15 slides
Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1

Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1

Training Neural Networks: Normalization, Regularization etc. Intro to Deep Learning, Fall 2020 1 Quick Recap: Training a network Divergence between desired output and Average over all actual output of net for a given input training instances

1.75k views • 140 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
Training Strategies CS 6355: Structured Prediction 1 So far we saw What is structured output

Training Strategies CS 6355: Structured Prediction 1 So far we saw What is structured output

Training Strategies CS 6355: Structured Prediction 1 So far we saw What is structured output prediction? Different ways for modeling structured prediction Conditional random fields, factor graphs, constraints What we only

986 views • 60 slides
What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured Prediction? Graham Neubig @ ICLR Workshop on Deep Reinforcement Learning Meets Structured Prediction 5/6/2019 Types of Prediction Types of

1.73k views • 127 slides
Tutorial on Neural Network Optimization Problems presentation by Ian Goodfellow Deep Learning

Tutorial on Neural Network Optimization Problems presentation by Ian Goodfellow Deep Learning

Tutorial on Neural Network Optimization Problems presentation by Ian Goodfellow Deep Learning Summer School Montreal August 9, 2015 Google Proprietary Optimization -Exhaustive search -Random search (genetic algorithms) -Analytical solution

1.1k views • 42 slides
BEYOND STANDARDS: A NEW GPU-AWARE IMAGE CODING SYSTEM Pablo Enfedaque GPU Technology Conference

BEYOND STANDARDS: A NEW GPU-AWARE IMAGE CODING SYSTEM Pablo Enfedaque GPU Technology Conference

BEYOND STANDARDS: A NEW GPU-AWARE IMAGE CODING SYSTEM Pablo Enfedaque GPU Technology Conference April 4 - 7, 2016 Ph.D. Advisors Francesc Aul-Llins Juan C. Moure Universitat Autnoma de Barcelona Table of contents 4. RESULTS PROBLEM

454 views • 19 slides
Ithaca Green Building Policy N ICK G OLDSMITH , C ITY OF I THACA & T OWN OF I THACA , NY P

Ithaca Green Building Policy N ICK G OLDSMITH , C ITY OF I THACA & T OWN OF I THACA , NY P

Ithaca Green Building Policy N ICK G OLDSMITH , C ITY OF I THACA & T OWN OF I THACA , NY P RESENTATION FOR T OMPKINS C OUNTY E NERGY T ASK F ORCE A UGUST 16, 2019 Table of Contents Background 1. 2. Overview of Requirements Details of the

762 views • 47 slides
C++11 CSP Because we havent built enough CSP libraries Dr Kevin Chalmers School of Computing

C++11 CSP Because we havent built enough CSP libraries Dr Kevin Chalmers School of Computing

C++11 CSP Because we havent built enough CSP libraries Dr Kevin Chalmers School of Computing Edinburgh Napier University Edinburgh k.chalmers@napier.ac.uk Before we get started... Yes you can pull it (although it needs more testing)

590 views • 25 slides
COMMUNITY-BASED MAINTENANCE MANAGEMENT PLATFORM www.mobility-work.com A NEW APPROACH TO

COMMUNITY-BASED MAINTENANCE MANAGEMENT PLATFORM www.mobility-work.com A NEW APPROACH TO

COMMUNITY-BASED MAINTENANCE MANAGEMENT PLATFORM www.mobility-work.com A NEW APPROACH TO MAINTENANCE OUR GOAL: TO PUT MAINTENANCE AT THE HEART OF YOUR BUSINESS. Simplify the daily life of your Join the first maintenance social Collect and

423 views • 9 slides
High thRoughput Optical Netw ork ESA HydRON Team European Space Agency, ESA-ESTEC I NDUSTRY DAY

High thRoughput Optical Netw ork ESA HydRON Team European Space Agency, ESA-ESTEC I NDUSTRY DAY

High thRoughput Optical Netw ork ESA HydRON Team European Space Agency, ESA-ESTEC I NDUSTRY DAY , 26Jun / 1Jul 2020 - BELGI UM ESA UNCLASSIFIED - For Official Use ESA UNCLASSIFIED - For Official Use 26Jun / 1Jul 2020 Slide 1 Agenda

831 views • 58 slides
I nte llie tc h Vide o & Image Pr oc e ssing F or any que r y c ontac t us at sale

I nte llie tc h Vide o & Image Pr oc e ssing F or any que r y c ontac t us at sale

1 Fo I nte llie tc h Vide o & Image Pr oc e ssing F or any que r y c ontac t us at sale s@inte llite c hglobal.c om 2 Vide o Wall Contr olle r s T he Mo st Ve rsa tile Arc hite c ture 1. Signal Ac quisition: VGA,

500 views • 36 slides
CSS codes Shyam Sundhar R November 12, 2017 Shyam Sundhar R CSS codes November 12, 2017 1 /

CSS codes Shyam Sundhar R November 12, 2017 Shyam Sundhar R CSS codes November 12, 2017 1 /

CSS codes Shyam Sundhar R November 12, 2017 Shyam Sundhar R CSS codes November 12, 2017 1 / 10 Outline Codes as a group 1 Code Construction 2 Hadamard Transform 3 Error Modelling 4 Error Correction 5 Shyam Sundhar R CSS codes

545 views • 10 slides
Turbulence Visualization at the Terascale on Desktop PCs Marc Treib*, Kai Brger*, Florian

Turbulence Visualization at the Terascale on Desktop PCs Marc Treib*, Kai Brger*, Florian

Turbulence Visualization at the Terascale on Desktop PCs Marc Treib*, Kai Brger*, Florian Reichl*, Charles Meneveau + , Alex Szalay + , and Rdiger Westermann* * Technische Universitt Mnchen, Munich, Germany + Johns Hopkins University,

798 views • 21 slides

More recommend