learning quantities from vision and language
play

Learning quantities from vision and language Raffaella Bernardi - PowerPoint PPT Presentation

Sep 14, 2022 •458 likes •1.09k views

Learning quantities from vision and language Raffaella Bernardi University of Trento March 23, 2017 Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 1 / 44 Cardinals and Quantifiers Three of the animals are dogs.

  1. Learning quantities from vision and language Raffaella Bernardi University of Trento March 23, 2017 Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 1 / 44

  2. Cardinals and Quantifiers Three of the animals are dogs. vs. Most of the animals are dogs Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 2 / 44

  3. Cardinals and Quantifiers Three of the animals are dogs. vs. Most of the animals are dogs Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 2 / 44

  4. Quantifiers: are they in a scale? Expected abstract scale: < no, few, some, most, all > Q. How do we learn they are in this order? Q. Do we take this order into account when using them? Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 3 / 44

  5. Quantifiers: are they in a scale? Expected abstract scale: < no, few, some, most, all > Q. How do we learn they are in this order? Q. Do we take this order into account when using them? Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 3 / 44

  6. Quantifiers: are they in a scale? Expected abstract scale: < no, few, some, most, all > Q. How do we learn they are in this order? Q. Do we take this order into account when using them? Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 3 / 44

  7. Litteral vs. Pragmatic meaning What do we learn from language, what from vision, what from both? Conjecture 1: we can learn their litteral meaning (respecting the abstract scale) from images . Conjecture 2: they can be represented by a cross-modal function . Conjecture 3: text corpora could help learning their use . Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 4 / 44

  8. Litteral vs. Pragmatic meaning What do we learn from language, what from vision, what from both? Conjecture 1: we can learn their litteral meaning (respecting the abstract scale) from images . Conjecture 2: they can be represented by a cross-modal function . Conjecture 3: text corpora could help learning their use . Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 4 / 44

  9. Litteral vs. Pragmatic meaning What do we learn from language, what from vision, what from both? Conjecture 1: we can learn their litteral meaning (respecting the abstract scale) from images . Conjecture 2: they can be represented by a cross-modal function . Conjecture 3: text corpora could help learning their use . Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 4 / 44

  10. New Challenge for CV From content words to Function words Most tasks considered so far involve processing of objects and lexicalised relations amongst objects ( content words ). Humans (even pre-school children) can abstract over raw data to perform certain types of higher-level reasoning, expressed in natural language by function words . Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 5 / 44

  11. Operations inolved in quatifying A logical strategy Quantifiers require: 1 an approximate number estimation mechanism, acting over the relevant sets in the image; 2 a quantification comparison step. A “logical” strategy: 1 from raw data to abstract set representation 2 from the latter to quantifiers. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 6 / 44

  12. Operations inolved in quatifying A logical strategy Quantifiers require: 1 an approximate number estimation mechanism, acting over the relevant sets in the image; 2 a quantification comparison step. A “logical” strategy: 1 from raw data to abstract set representation 2 from the latter to quantifiers. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 6 / 44

  13. Comparison step Look, some green circles!: Learning to quantify from images (Sorodoc et al., 2016): Very high results: NNs should be able to learn the second subtask quite easily. Is the “logical” strategy a good one? Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 7 / 44

  14. Comparison step Look, some green circles!: Learning to quantify from images (Sorodoc et al., 2016): Very high results: NNs should be able to learn the second subtask quite easily. Is the “logical” strategy a good one? Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 7 / 44

  15. Learning quantification from images Layout Learning quantification from images 1 Quantifiers vs. Cardinals 2 Behavioral Study 3 Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 8 / 44

  16. Learning quantification from images Learning quantification from images Pay attention to those sets! Learning quantification from images Sorodoc et. al. just submitted. (a) (b) (c) (d) (e) Query: fish are red. Answers: (a) All, (b) Most, (c) Some, (d) Few, (e) No. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 9 / 44

  17. Learning quantification from images Learning quantification from images Pay attention to those sets! Learning quantification from images Sorodoc et. al. just submitted. (a) (b) (c) (d) (e) Query: fish are red. Answers: (a) All, (b) Most, (c) Some, (d) Few, (e) No. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 9 / 44

  18. Learning quantification from images Not raw data: All sorts of variances in place The system cannot memorize correlations between type of objects and quantifiers property of objects and quantifiers number of objects and quantifiers Quite challenging! Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 10 / 44

  19. Learning quantification from images Quantifiers as proportions Q of the fish are red . ���� ���� restrictor scope We take quantifiers to be a fiexed relation: | scope ∩ restrictor | ( e . g . | red ∩ fish | ) | restrictor | | fish | Prevalence estimates (Khemlain et al 2009): No: 0% Few: 1% - 17% (inc) Some: 17 % - 70% Most: 70% (inc) - 99% (inc.) All: 100% Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 11 / 44

  20. Learning quantification from images Quantifiers as proportions Q of the fish are red . ���� ���� restrictor scope We take quantifiers to be a fiexed relation: | scope ∩ restrictor | ( e . g . | red ∩ fish | ) | restrictor | | fish | Prevalence estimates (Khemlain et al 2009): No: 0% Few: 1% - 17% (inc) Some: 17 % - 70% Most: 70% (inc) - 99% (inc.) All: 100% Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 11 / 44

  21. Learning quantification from images Quantifiers as proportions Q of the fish are red . ���� ���� restrictor scope We take quantifiers to be a fiexed relation: | scope ∩ restrictor | ( e . g . | red ∩ fish | ) | restrictor | | fish | Prevalence estimates (Khemlain et al 2009): No: 0% Few: 1% - 17% (inc) Some: 17 % - 70% Most: 70% (inc) - 99% (inc.) All: 100% Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 11 / 44

  22. Learning quantification from images Computer Vision Models Start simple: concatenation. CNN+BOW Zhou et al. Simple Baseline for Visual Question Answering 2015 (iBOWIMG) Memorize correlations, no higher level abstraction Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 12 / 44

  23. Learning quantification from images Computer Vision Models Lesson learned from SoA: Memory and Attention Memory process new information based on previous ones. (LSTM, GRU) Attention Mechanism Use language to help making the representation of the image more focused Stacked Attention use language to focus the visual representation and use the later to focus the linguistic representation. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 13 / 44

  24. Learning quantification from images Computer Vision Models Lesson learned from SoA: Memory and Attention Memory process new information based on previous ones. (LSTM, GRU) Attention Mechanism Use language to help making the representation of the image more focused Stacked Attention use language to focus the visual representation and use the later to focus the linguistic representation. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 13 / 44

  25. Learning quantification from images Computer Vision Models Lesson learned from SoA: Memory and Attention Memory process new information based on previous ones. (LSTM, GRU) Attention Mechanism Use language to help making the representation of the image more focused Stacked Attention use language to focus the visual representation and use the later to focus the linguistic representation. Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 13 / 44

  26. Learning quantification from images Sequential Processing CNN+LSTM model Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 14 / 44

  27. Learning quantification from images Attention Mechanism: SAN’s attention layer Yang, Z., et al. (CVPR 2016). Stacked attention networks (SAN) for image question answering. Linguistic Nonlinear input transformation Linear Tanh Softmax transformation transformation transformation + + Visual + input + + + Gist Raffaella Bernardi (University of Trento) LaVi: quantifiers March 23, 2017 15 / 44

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Vision, Language, Interaction and Generation Qi Wu Australian Institute for Machine Learning

Vision, Language, Interaction and Generation Qi Wu Australian Institute for Machine Learning

Vision, Language, Interaction and Generation Qi Wu Australian Institute for Machine Learning Australia Centre for Robotic Vision University of Adelaide Vision-and-Language Computer Vision (CV) Natural Language Processing (NLP) Image

618 views • 57 slides
Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning

Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning

Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning Jiasen Lu April 21, 2020 1 1 Vision and Language Visual Question Answering Image Captioning Visual Commonsense Reasoning Refer Expression 2

1.13k views • 53 slides
Language and Vision at UniTN Raffaella Bernardi University of Trento LaVi @ UniTn Learning the

Language and Vision at UniTN Raffaella Bernardi University of Trento LaVi @ UniTn Learning the

Language and Vision at UniTN Raffaella Bernardi University of Trento LaVi @ UniTn Learning the meaning of Quantifiers from Language, Vision (and Audio): https://quantit-clic.github.io/ none almost none few the smaller part some many most

753 views • 57 slides
Vision, Language, Learning Communication, Participation: An Approach to AAC for Students with CVI

Vision, Language, Learning Communication, Participation: An Approach to AAC for Students with CVI

Vision, Language, Learning Communication, Participation: An Approach to AAC for Students with CVI Christine Roman-Lantzy, Ph.D. Sarah W. Blackstone, Ph.D., CCC-SLP 1 1 Material from: The Bridge School &amp; CATIC Children with CVI and CCN

1.02k views • 65 slides
Introduction to Deep Learning Principles and applications in vision and natural language

Introduction to Deep Learning Principles and applications in vision and natural language

Introduction to Deep Learning Principles and applications in vision and natural language processing Jakob Verbeek (INRIA) Slides in collaboration with Laurent Besacier (Univ. Grenoble Alpes) 2018 Introduction Convolutional Neural Networks

517 views • 28 slides
Multimodality Learning from Text, Speech, and Vision CMU 11-4/611 Natural Language Processing

Multimodality Learning from Text, Speech, and Vision CMU 11-4/611 Natural Language Processing

Multimodality Learning from Text, Speech, and Vision CMU 11-4/611 Natural Language Processing Lecture 28 April 14, 2020 Shruti Palaskar Outline I. What is multimodality? II. Types of modalities III. Commonly used Models IV. Multimodal

1.48k views • 78 slides
From Activity to Language: Learning to recognise the meaning of motion Centre for Vision, Speech

From Activity to Language: Learning to recognise the meaning of motion Centre for Vision, Speech

From Activity to Language: Learning to recognise the meaning of motion Centre for Vision, Speech and Signal Processing Prof Rich Bowden 20 June 2011 Centre for Vision Speech and Signal Processing Overview Talk is about recognising spatio

538 views • 50 slides
Deep learning in computer vision and natural language processing Yifeng Tao School of Computer

Deep learning in computer vision and natural language processing Yifeng Tao School of Computer

Introduction to Machine Learning Deep learning in computer vision and natural language processing Yifeng Tao School of Computer Science Carnegie Mellon University Slides adapted from Matt Gormley, Russ Salakhutdinov Yifeng Tao Carnegie

415 views • 40 slides
Vision and Language Learning with Graph Neural Networks Linchao Zhu 22 Apr, 2020 Recognition,

Vision and Language Learning with Graph Neural Networks Linchao Zhu 22 Apr, 2020 Recognition,

Vision and Language Learning with Graph Neural Networks Linchao Zhu 22 Apr, 2020 Recognition, LEarning, Reasoning UTS CRICOS 00099F Overview RNNs for Image Captioning Transformer for Image Captioning Graph Network for Visual

385 views • 19 slides
ENGLISH LANGUAGE DEPARTMENT ENGLISH LANGUAGE LEARNING STELLAR SUPPORT PROGRAMME PROGRAMMES

ENGLISH LANGUAGE DEPARTMENT ENGLISH LANGUAGE LEARNING STELLAR SUPPORT PROGRAMME PROGRAMMES

ENGLISH LANGUAGE DEPARTMENT ENGLISH LANGUAGE LEARNING STELLAR SUPPORT PROGRAMME PROGRAMMES HOLISTIC &amp; ASSESSMENT EVENTS ST STrate tegi gies for or Englis lish Langua guage Learni rning ng And nd Readi ding ng VISION :

401 views • 19 slides
Self-supervised Learning for Vision-and-Language Licheng Yu, Yen-Chun Chen, Linjie Li Nowadays

Self-supervised Learning for Vision-and-Language Licheng Yu, Yen-Chun Chen, Linjie Li Nowadays

Self-supervised Learning for Vision-and-Language Licheng Yu, Yen-Chun Chen, Linjie Li Nowadays Machine Learning Algorithm Data Compute Nowadays Machine Learning Algorithm Data Compute Datasets + Labels MS COCOs Image Captioning:

3.23k views • 115 slides
Impact of Deep Learning Speech Recogni4on Computer Vision

Impact of Deep Learning Speech Recogni4on Computer Vision

Impact of Deep Learning Speech Recogni4on Computer Vision Recommender Systems Language Understanding Drug Discovery and Medical Image

526 views • 21 slides
WELCOME TO VISION NEXT BASE CAMP! Vision Next Learning Journey A city of informed, engaged

WELCOME TO VISION NEXT BASE CAMP! Vision Next Learning Journey A city of informed, engaged

WELCOME TO VISION NEXT BASE CAMP! Vision Next Learning Journey A city of informed, engaged philanthropists accelerating meaningful change for all. Vision Next Learning Goals The New Philanthropy Explore the collective and personal values

147 views • 14 slides
vision &amp; language CS 685, Fall 2020 Introduction to Natural Language Processing

vision &amp; language CS 685, Fall 2020 Introduction to Natural Language Processing

vision &amp; language CS 685, Fall 2020 Introduction to Natural Language Processing http://people.cs.umass.edu/~miyyer/cs685/ Mohit Iyyer College of Information and Computer Sciences University of Massachusetts Amherst some slides adapted from

737 views • 57 slides
VISION, MISSION, AND VALUE STATEMENT VISION: Transforming lives through learning. M ISSION :

VISION, MISSION, AND VALUE STATEMENT VISION: Transforming lives through learning. M ISSION :

VISION, MISSION, AND VALUE STATEMENT VISION: Transforming lives through learning. M ISSION : Provide outstanding learning opportunities that prepare students to meet community needs and future challenges of a complex, global society. V ALUE S

506 views • 21 slides
Neural Program Synthesis Rishabh Singh, Google Brain Great Collaborators! Deep Learning and

Neural Program Synthesis Rishabh Singh, Google Brain Great Collaborators! Deep Learning and

Neural Program Synthesis Rishabh Singh, Google Brain Great Collaborators! Deep Learning and Evolutionary Progression Vision Speech Language Deep Learning and Evolutionary Progression Vision Speech Language Programming Deep Learning

1.44k views • 142 slides
Structure: Exposition General problem Purpose / stated goal(s) Experimental setup

Structure: Exposition General problem Purpose / stated goal(s) Experimental setup

Structure: Exposition General problem Purpose / stated goal(s) Experimental setup summary Result summary 1 Experimental Evaluation: Bias and Generalization in Deep Generative Models Nicholay Topin MLD, Carnegie Mellon

352 views • 22 slides
Bias and Generalization in Deep Generative Models Shengjia Zhao*, Hongyu Ren*, Arianna Yuan,

Bias and Generalization in Deep Generative Models Shengjia Zhao*, Hongyu Ren*, Arianna Yuan,

Bias and Generalization in Deep Generative Models Shengjia Zhao*, Hongyu Ren*, Arianna Yuan, Jiaming Song, Noah Goodman and Stefano Ermon *equal contribution Success in Generative Modeling of Images Brock A, et al. &quot;Large scale gan

350 views • 19 slides
Data Preprocessing Week 2 Topics Topics Data Types Data Repositories Data

Data Preprocessing Week 2 Topics Topics Data Types Data Repositories Data

Data Preprocessing Week 2 Topics Topics Data Types Data Repositories Data Preprocessing Present homework assignment #1 Team Homework Assignment #2 Team Homework Assignment #2 Read pp 227 Read pp. 227 240, pp. 250

926 views • 67 slides
Motivation Garbage-in, garbage-out Cannot get good mining results from bad data Data

Motivation Garbage-in, garbage-out Cannot get good mining results from bad data Data

Motivation Garbage-in, garbage-out Cannot get good mining results from bad data Data Preprocessing Need to understand data properties to select the right technique and parameter values Data cleaning Mirek Riedewald Data

511 views • 7 slides
Prototype Selection Using Polyhedron Curvature Benyamin Ghojogh, Fakhri Karray, Mark Crowley

Prototype Selection Using Polyhedron Curvature Benyamin Ghojogh, Fakhri Karray, Mark Crowley

Anomaly Detection and Prototype Selection Using Polyhedron Curvature Benyamin Ghojogh, Fakhri Karray, Mark Crowley Canadian AI conference, 2020 1 Anomaly Detection finding outliers or anomalies which differ significantly from the normal

391 views • 23 slides
SCATTERPLOTS: TASKS, DATA AND DESIGN A. Sarikaya and M. Gleicher Presented By: IEEE Transaction

SCATTERPLOTS: TASKS, DATA AND DESIGN A. Sarikaya and M. Gleicher Presented By: IEEE Transaction

SCATTERPLOTS: TASKS, DATA AND DESIGN A. Sarikaya and M. Gleicher Presented By: IEEE Transaction on Visualization and Computer Graphics Shareen Mahmud 1 WHAT IS A TRADITIONAL SCATTERPLOT? Encodes two quantitative variables using the

670 views • 18 slides
Money illusion Jean-Robert Tyran U Vienna Outline Introduction Evidence Surveys

Money illusion Jean-Robert Tyran U Vienna Outline Introduction Evidence Surveys

The Econom ics of Money illusion Jean-Robert Tyran U Vienna Outline Introduction Evidence Surveys (Stephens and Tyran WP 2012) Neuroscience Field (labor, housing, stock markets ) Tyran (Science 2007), Fehr and

802 views • 34 slides
Quantified Sentences as a Window into Prediction and Priming: An ERP Study Aniello De Santo Jon

Quantified Sentences as a Window into Prediction and Priming: An ERP Study Aniello De Santo Jon

Quantified Sentences as a Window into Prediction and Priming: An ERP Study Aniello De Santo Jon Rawski Amanda Yazdani John E. Drury Stony Brook University aniello.desanto@stonybrook.edu aniellodesanto.github.io CLS 54 April 26, 2018 The

939 views • 71 slides

More recommend