learning effective and interpretable semantic models
play

Learning Effective and Interpretable Semantic Models using - PowerPoint PPT Presentation

Feb 16, 2023 •160 likes •1.03k views

1 Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding (NNSE) Brian Murphy, Partha Talukdar, Tom Mitchell Machine Learning Department Carnegie Mellon University 2 Distributional Semantic Modeling 2

  1. 1 Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding (NNSE) Brian Murphy, Partha Talukdar, Tom Mitchell Machine Learning Department Carnegie Mellon University

  2. 2 Distributional Semantic Modeling

  3. 2 Distributional Semantic Modeling • Words are represented in a high dimensional vector space

  4. 2 Distributional Semantic Modeling • Words are represented in a high dimensional vector space • Long history: • (Deerwester et al., 1990), (Lin, 1998), (Turney, 2006), ...

  5. 2 Distributional Semantic Modeling • Words are represented in a high dimensional vector space • Long history: • (Deerwester et al., 1990), (Lin, 1998), (Turney, 2006), ... • Although effective, these models are often not interpretable

  6. 2 Distributional Semantic Modeling • Words are represented in a high dimensional vector space • Long history: • (Deerwester et al., 1990), (Lin, 1998), (Turney, 2006), ... • Although effective, these models are often not interpretable Examples of top 5 words from 5 randomly chosen dimensions from SVD 300

  7. Why interpretable dimensions? 3 Semantic Decoding: (Mitchell et al., Science 2008)

  8. Why interpretable dimensions? 3 Semantic Decoding: (Mitchell et al., Science 2008) “motorbike” Input stimulus word

  9. Why interpretable dimensions? 3 Semantic Decoding: (Mitchell et al., Science 2008) (0.87, ride) (0.29, see) “motorbike” . . . (0.00, rub) Input (0.00, taste) stimulus word Semantic representation

  10. Why interpretable dimensions? 3 Semantic Decoding: (Mitchell et al., Science 2008) (0.87, ride) (0.29, see) “motorbike” . . . (0.00, rub) Input (0.00, taste) stimulus word Semantic Mapping learned representation from fMRI data

  11. Why interpretable dimensions? 3 Semantic Decoding: (Mitchell et al., Science 2008) (0.87, ride) (0.29, see) predicted “motorbike” . activity . for . “motorbike” (0.00, rub) Input (0.00, taste) stimulus word Semantic Mapping learned representation from fMRI data

  12. 4 Why interpretable dimensions? (Mitchell et al., Science 2008)

  13. 4 Why interpretable dimensions? (Mitchell et al., Science 2008)

  14. 4 Why interpretable dimensions? (Mitchell et al., Science 2008) • Interpretable dimension reveals insightful brain activation patterns!

  15. 4 Why interpretable dimensions? (Mitchell et al., Science 2008) • Interpretable dimension reveals insightful brain activation patterns! • But, features in the semantic representation were based on 25 hand-selected verbs

  16. 4 Why interpretable dimensions? (Mitchell et al., Science 2008) • Interpretable dimension reveals insightful brain activation patterns! • But, features in the semantic representation were based on 25 hand-selected verbs ‣ can’t represent arbitrary concepts

  17. 4 Why interpretable dimensions? (Mitchell et al., Science 2008) • Interpretable dimension reveals insightful brain activation patterns! • But, features in the semantic representation were based on 25 hand-selected verbs ‣ can’t represent arbitrary concepts ‣ need data-driven, broad coverage semantic representations

  18. 5 What is an Interpretable, Cognitively-plausible Representation?

  19. 5 What is an Interpretable, Cognitively-plausible Representation? features words

  20. 5 What is an Interpretable, Cognitively-plausible Representation? features words w ... 0 0 0 ... 1.2 ...

  21. 5 What is an Interpretable, Cognitively-plausible Representation? features words w ... 0 0 0 ... 1.2 ... 1. Compact representation: Sparse , many zeros

  22. 5 What is an Interpretable, Cognitively-plausible Representation? features words w ... 0 0 0 ... 1.2 ... 1. Compact representation: 2. Uneconomical to store negative Sparse , many zeros (or inferable) characteristics: Non-Negative

  23. 5 What is an Interpretable, Cognitively-plausible Representation? features words 3. Meaningful Dimensions: w ... 0 0 0 ... 1.2 ... Coherent 1. Compact representation: 2. Uneconomical to store negative Sparse , many zeros (or inferable) characteristics: Non-Negative

  24. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative

  25. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative Hand Tuned

  26. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative Hand Tuned Corpus-derived (existing)

  27. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative Hand Tuned Hand Coded 83.5 Corpus-derived Corpus-derived 83.1 (existing) Prediction accuracy (on Neurosemantic Decoding) (Murphy, Talukdar, Mitchell, StarSem 2012)

  28. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative Hand Tuned Corpus-derived (existing)

  29. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative Hand Tuned Corpus-derived (existing) NNSE Our proposal

  30. 6 Properties of Different Semantic Representations Broad Non- Effective Sparse Coherent Coverage Negative Hand Tuned Corpus-derived (existing) NNSE Our proposal

  31. 7 Non-Negative Sparse Embedding (NNSE)

  32. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) X

  33. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) X w i input representation for word w i

  34. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) X A D = x w i input representation for word w i

  35. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) Basis X A D = x w i input representation NNSE representation for word w i for word w i

  36. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) Basis X A D = x w i input representation NNSE representation for word w i for word w i • matrix A is non-negative

  37. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) Basis X A D = x w i input representation NNSE representation for word w i for word w i • matrix A is non-negative • sparsity penalty on the rows of A

  38. 7 Non-Negative Sparse Embedding (NNSE) Input Matrix (Corpus Cooc + SVD) Basis X A D = x w i input representation NNSE representation for word w i for word w i • matrix A is non-negative • sparsity penalty on the rows of A • alternating minimization between A and D, using SPAMS package

  39. 8 NNSE Optimization X A D = x

  40. 9 Experiments

  41. 9 Experiments • Three main questions

  42. 9 Experiments • Three main questions 1. Are NNSE representations effective in practice?

  43. 9 Experiments • Three main questions 1. Are NNSE representations effective in practice? 2. What is the degree of sparsity of NNSE?

  44. 9 Experiments • Three main questions 1. Are NNSE representations effective in practice? 2. What is the degree of sparsity of NNSE? 3. Are NNSE dimensions coherent?

  45. 9 Experiments • Three main questions 1. Are NNSE representations effective in practice? 2. What is the degree of sparsity of NNSE? 3. Are NNSE dimensions coherent? • Setup

  46. 9 Experiments • Three main questions 1. Are NNSE representations effective in practice? 2. What is the degree of sparsity of NNSE? 3. Are NNSE dimensions coherent? • Setup • partial ClueWeb09, 16bn tokens, 540m sentences, 50m documents

  47. 9 Experiments • Three main questions 1. Are NNSE representations effective in practice? 2. What is the degree of sparsity of NNSE? 3. Are NNSE dimensions coherent? • Setup • partial ClueWeb09, 16bn tokens, 540m sentences, 50m documents • dependency parsed using Malt parser

  48. 10 Baseline Representation: SVD

  49. 10 Baseline Representation: SVD • For about 35k words (~adult vocabulary) , extract

  50. 10 Baseline Representation: SVD • For about 35k words (~adult vocabulary) , extract • document co-occurrence

  51. 10 Baseline Representation: SVD • For about 35k words (~adult vocabulary) , extract • document co-occurrence • dependency features from the parsed corpus

  52. 10 Baseline Representation: SVD • For about 35k words (~adult vocabulary) , extract • document co-occurrence • dependency features from the parsed corpus • Reduce dimensionality using SVD. Subsets of this reduced dimensional space is the baseline

  53. 10 Baseline Representation: SVD • For about 35k words (~adult vocabulary) , extract • document co-occurrence • dependency features from the parsed corpus • Reduce dimensionality using SVD. Subsets of this reduced dimensional space is the baseline • This is also the input (X) to NNSE

  54. 10 Baseline Representation: SVD • For about 35k words (~adult vocabulary) , extract • document co-occurrence • dependency features from the parsed corpus • Reduce dimensionality using SVD. Subsets of this reduced dimensional space is the baseline • This is also the input (X) to NNSE • Other representations were also compared (e.g., LDA, Collobert and Weston, etc.), details in the paper

  55. 11 Is NNSE effective in Neurosemantic Decoding?

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead Cynthia Rudin, Duke University Presenters : Sreya Dutta Roy, Ziqian Lin 1 Overview Introduction Explainable ML Vs

1.04k views • 39 slides
From ML Successes to Applications ICIP18 Tutorial on Interpretable Deep Learning 2 Black Box

From ML Successes to Applications ICIP18 Tutorial on Interpretable Deep Learning 2 Black Box

From ML Successes to Applications ICIP18 Tutorial on Interpretable Deep Learning 2 Black Box Models Huge volumes of data Solve task Computing power Deep Neural Network Information (implicit) ICIP18 Tutorial on Interpretable Deep

540 views • 33 slides
Learning interpretable continuous-time models of latent stochastic dynamical systems Lea Duncker,

Learning interpretable continuous-time models of latent stochastic dynamical systems Lea Duncker,

Learning interpretable continuous-time models of latent stochastic dynamical systems Lea Duncker, Gerg o Bohner, Julien Boussard, Maneesh Sahani Gatsby Computational Neuroscience Unit University College London ICML June 12, 2019 nonlinear

671 views • 32 slides
Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation Outline Exploring Context with Deep Structured Models Guosheng Lin, Chunhua Shen, Ian Reid, Anton van dan Hengel; Efficient Piecewise Training

992 views • 45 slides
Learning Interpretable Models Expressed in Linear Temporal Logic Alberto Camacho 1 , 2 and Sheila

Learning Interpretable Models Expressed in Linear Temporal Logic Alberto Camacho 1 , 2 and Sheila

Learning Interpretable Models Expressed in Linear Temporal Logic Alberto Camacho 1 , 2 and Sheila McIlraith 1 , 2 1 Department of Computer Science, University of Toronto 2 Vector Institute { acamacho, sheila } @cs.toronto.edu ICAPS 2019 July 13,

457 views • 19 slides
IMLI: An Incremental Framework for MaxSAT-Based Learning of Interpretable Classification Rules

IMLI: An Incremental Framework for MaxSAT-Based Learning of Interpretable Classification Rules

IMLI: An Incremental Framework for MaxSAT-Based Learning of Interpretable Classification Rules Bishwamittra Ghosh Joint work with Kuldeep S. Meel 1 Applications of Machine Learning 2 Example Dataset 3 Representation of an interpretable

1.24k views • 55 slides
Toward Predictive Digital Twins via component-based reduced-order models and interpretable

Toward Predictive Digital Twins via component-based reduced-order models and interpretable

Toward Predictive Digital Twins via component-based reduced-order models and interpretable machine learning Michael Kapteyn*, Dr. David Knezevic, Prof. Karen Willcox AIAA SciTech | Paper AIAA-2020-0418 | January 6, 2020 Outline 1 Motivation

426 views • 27 slides
Deep Visual Models with Interpretable Features and Modularized Structures Quanshi Zhang John

Deep Visual Models with Interpretable Features and Modularized Structures Quanshi Zhang John

Deep Visual Models with Interpretable Features and Modularized Structures Quanshi Zhang John Hopcroft Center Shanghai Jiao Tong University Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu, "Interpretable Convolutional Neural Networks"

694 views • 42 slides
Layer-wise Relevance Propagation in Neural Neural Networks Deep Learning Shortcomings Networks

Layer-wise Relevance Propagation in Neural Neural Networks Deep Learning Shortcomings Networks

LRP Ariyan Zarei Motivation Having More interpretable Layer-wise Relevance Propagation in Neural Neural Networks Deep Learning Shortcomings Networks to have more interpretable Papers and Demo Introduction Machine Learning models

373 views • 35 slides
Toward In Interpretable De Deep Re Reinforcement Lea Learning g wi with Li Linea ear Model

Toward In Interpretable De Deep Re Reinforcement Lea Learning g wi with Li Linea ear Model

Toward In Interpretable De Deep Re Reinforcement Lea Learning g wi with Li Linea ear Model el U-Tr Trees Guiliang Liu, Oliver Schulte, Wang Zhu, Qingcan Li Machine Learning Lab, ECML-PKDD 2018 Presentation PROBLEM DEFINITION

372 views • 15 slides
Ralph Hodgson Realizing a semantic solution: Ontologies are like and unlike other IT models

Ralph Hodgson Realizing a semantic solution: Ontologies are like and unlike other IT models

Enterprise Architecture in SOA: Coverage Models and Methodologies Going Semantic Semantic technology May 23 rd , 2005 Enterprise architecture and semantic technology Enterprise architecture maturity model Capabilities of semantic

244 views • 9 slides
RDF, RDFS and OWL: Graph Data Models for the Semantic Web Semantic Web: The Idea Semantic

RDF, RDFS and OWL: Graph Data Models for the Semantic Web Semantic Web: The Idea Semantic

RDF, RDFS and OWL: Graph Data Models for the Semantic Web Semantic Web: The Idea Semantic Web does not mean computers are going to understand the meaning of everything on the web does mean that we can create some standards ways of

344 views • 33 slides
Compositional Distributional Semantic Models for Semantic Relatedness and Entailment Sidharth

Compositional Distributional Semantic Models for Semantic Relatedness and Entailment Sidharth

Compositional Distributional Semantic Models for Semantic Relatedness and Entailment Sidharth Gupta Sai Krishna Prasad Guided By:- Amitabha Mukherjee Distributional Semantic Models (DSMs) Distributional hypothesis - words that occur in

375 views • 11 slides
Computational Phenotyping in Polysomnography : Using Interpretable Physiology-Based Machine

Computational Phenotyping in Polysomnography : Using Interpretable Physiology-Based Machine

Computational Phenotyping in Polysomnography : Using Interpretable Physiology-Based Machine Learning Models to Predict Health Outcomes Chris Fernandez 1,2 Sam Rusk 1,2 Nick Glattard ,1,2 Mehdi Shokoueinejad. 1,3 1 Department of Population Health

609 views • 21 slides
Interpretable & Transparent Deep Learning Fraunhofer HHI, Machine Learning Group Wojciech

Interpretable & Transparent Deep Learning Fraunhofer HHI, Machine Learning Group Wojciech

Fraunhofer Image Processing Heinrich Hertz Institute Interpretable & Transparent Deep Learning Fraunhofer HHI, Machine Learning Group Wojciech Samek Northern Lights Deep Learning Workshop (NLDL19) Troms, Norway 8th January 2019

1.15k views • 51 slides
Not Just a Black Box: Interpretable Deep Learning for Genomics Avan> Shrikumar, Peyton

Not Just a Black Box: Interpretable Deep Learning for Genomics Avan> Shrikumar, Peyton

Not Just a Black Box: Interpretable Deep Learning for Genomics Avan> Shrikumar, Peyton Greenside, Anshul Kundaje Peyton Anshul 1 With great power comes really poor interpretability Deep Interpretable Deep Power Learning Learning

783 views • 41 slides
Incremental Parsing in Bounded Memory William Schuler Department of Linguistics The Ohio State

Incremental Parsing in Bounded Memory William Schuler Department of Linguistics The Ohio State

Incremental Parsing in Bounded Memory William Schuler Department of Linguistics The Ohio State University September 16, 2010 William Schuler Incremental Parsing in Bounded Memory Motivation Goal: simple processing model, matches observations

1.27k views • 80 slides
A Survey On Automated Dynamic Malware Analysis Evasion and Counter-Evasion: PC, Mobile, and Web

A Survey On Automated Dynamic Malware Analysis Evasion and Counter-Evasion: PC, Mobile, and Web

A Survey On Automated Dynamic Malware Analysis Evasion and Counter-Evasion: PC, Mobile, and Web Alexei Bulazel & Blent Yener River Loop Security Rensselaer Polytechnic Institute (RPI) Introduction Automated dynamic malware analysis is

684 views • 33 slides
Towards a syntactically motivated analysis of modifiers in German Ines Rehbein & Hagen

Towards a syntactically motivated analysis of modifiers in German Ines Rehbein & Hagen

Tagset Annotation Experiment Parsing Experiments Towards a syntactically motivated analysis of modifiers in German Ines Rehbein & Hagen Hirschmann KONVENS 2014 October 8, 2014 Tagset Annotation Experiment Parsing Experiments Modifying

592 views • 36 slides
Good Morning and Welcome Back! We trust you have a productive and an enjoyable time here in

Good Morning and Welcome Back! We trust you have a productive and an enjoyable time here in

ITS Research IT Good Morning and Welcome Back! We trust you have a productive and an enjoyable time here in Utrecht! Let's collaborate! Ton Smeele ITS/ResearchIT, Utrecht University ITS Research IT Agenda University Utrecht's use

891 views • 19 slides
On the (Im)possibility of Privately Outsourcing Linear Programming 26.10.13 1 / 25 Linear

On the (Im)possibility of Privately Outsourcing Linear Programming 26.10.13 1 / 25 Linear

On the (Im)possibility of Privately Outsourcing Linear Programming 26.10.13 1 / 25 Linear programming Suppose a brewery produces ale and beer . It uses three type of resources: corn , hops , and malt . Each beverage requires

592 views • 44 slides
T H E L LV M PA S S M A N AG E R PA RT 2 F R O M T H E P R E V I O U S TA L K : A pass

T H E L LV M PA S S M A N AG E R PA RT 2 F R O M T H E P R E V I O U S TA L K : A pass

C H A N D L E R CA R R U T H L LV M D E V M TG 2 0 1 4 T H E L LV M PA S S M A N AG E R PA RT 2 F R O M T H E P R E V I O U S TA L K : A pass operates on some unit of IR (Function, Module, ) Generally to transform it

666 views • 52 slides
1 Not only does a great mom (or grandma) MODEL, but she also... OBSERVE What do I mean by

1 Not only does a great mom (or grandma) MODEL, but she also... OBSERVE What do I mean by

Influential M.O.M.s - 05.13.10 Intro: Where Is God? joke: http://www.freetimejokes.com/ mothers-day-jokes/where-is-god Were in BIG trouble this time. God is missing and they think WE did it! Moms have so much more influence than

399 views • 3 slides
MAM: where we are, where we go panel discussion MAM 9, June. 30, 2016, Budapest, Hungary panel

MAM: where we are, where we go panel discussion MAM 9, June. 30, 2016, Budapest, Hungary panel

MAM: where we are, where we go panel discussion MAM 9, June. 30, 2016, Budapest, Hungary panel discussion MAM: where we are, where we go Reseach fields queueing/ruin models: EXP Markov modulated (Kronecker algebra) Markov modulted

266 views • 7 slides

More recommend