computational linguistics hci evaluation measures
play

Computational Linguistics: HCI, Evaluation Measures R AFFAELLA B - PowerPoint PPT Presentation

Jan 16, 2023 •36 likes •856 views

Computational Linguistics: HCI, Evaluation Measures R AFFAELLA B ERNARDI U NIVERSIT ` A DEGLI S TUDI DI T RENTO E - MAIL : BERNARDI @ DISI . UNITN . IT Contents First Last Prev Next Contents 1 Today main topics . . . . . . . . . . . .

  1. Computational Linguistics: HCI, Evaluation Measures R AFFAELLA B ERNARDI U NIVERSIT ` A DEGLI S TUDI DI T RENTO E - MAIL : BERNARDI @ DISI . UNITN . IT Contents First Last Prev Next ◭

  2. Contents 1 Today main topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2 HCI via Natural Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3 Natural Language Interfaces to Data Bases . . . . . . . . . . . . . . . . . . . . . . . . 9 3.1 Sample Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2 Restricted NL input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4 From the ’90 .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5 Question Answering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 5.1 History of ideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 5.2 Re-emergence of QA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 5.3 Sample of QA architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 5.3.1 Question Classification: Why and How? . . . . . . . . . . . 18 5.3.2 Question Classification: Problems . . . . . . . . . . . . . . . . 19 5.3.3 Document Retrieval in QA . . . . . . . . . . . . . . . . . . . . . . 20 5.3.4 Document Retrieval in QA: Approaches . . . . . . . . . . . 21 5.3.5 Candidate Answer Extraction . . . . . . . . . . . . . . . . . . . . 22 5.3.6 Answer Re-ranking and Selection . . . . . . . . . . . . . . . . 23 5.4 Enriched QA Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Contents First Last Prev Next ◭

  3. 5.4.1 Fact Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 5.4.2 Other QA issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.5 Understanding in QA?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.6 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 5.7 Today research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 6 Interactive Question Answering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 6.1 Characteristic of IQA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 6.2 Context in IQA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 6.3 Context in Open Domain IQA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 6.4 System-initiated Interactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 6.4.1 Type of system initiatives . . . . . . . . . . . . . . . . . . . . . . . 35 6.4.2 System-initiated contributions . . . . . . . . . . . . . . . . . . . 36 6.5 System initiative in DB IQA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 6.6 Ontology in IQA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 6.7 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 6.8 Characteristics for the Ontology . . . . . . . . . . . . . . . . . . . . . . . . . . 40 6.9 Today research on Dialogues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 7.1 New trend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Contents First Last Prev Next ◭

  4. 7.2 Evaluation campaigns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 8 Information Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 8.1 History of IR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 8.2 IR challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 8.3 Retrieval: Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 8.4 Index: example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 8.5 Problems with index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 8.6 Inverted Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 8.7 Indexing and Inverted Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 8.8 Index size and space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 8.9 Retrieval Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 9 Query Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 9.1 Query processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 9.2 Query processing: Boolean Model . . . . . . . . . . . . . . . . . . . . . . . . 57 10 Problem with Boolean search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 11 Ranking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 11.1 Ranking: boolean query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 11.2 Full text queries (non boolean) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 11.3 Document and Query as a binary vector . . . . . . . . . . . . . . . . . . . . 62 Contents First Last Prev Next ◭

  5. 11.4 Similarity Measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 12 Query-document matching scores I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 12.1 Summing up: Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 13 Recall: Vector Space Model: Document as vectors . . . . . . . . . . . . . . . . . . 66 13.1 Assumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 14 Query-document matching scores II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 14.1 Proximity: Angle and Cosine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 14.2 Example: Length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 14.3 Example: Cosine Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 14.4 Example: Query-Document Matching . . . . . . . . . . . . . . . . . . . . . 72 15 Summary: Ranked retrieval in the vector space model . . . . . . . . . . . . . . . 73 16 Evaluation Measures: Contingeny Matrix . . . . . . . . . . . . . . . . . . . . . . . . . 74 16.1 Evaluation Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 16.2 Evaluation Measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 16.3 Precision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 16.4 Precision, Recall and F-Measure . . . . . . . . . . . . . . . . . . . . . . . . . . 79 16.5 Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 16.6 Problem with Recall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 16.7 Trade-off between Recall & Precision. . . . . . . . . . . . . . . . . . . . . . 83 Contents First Last Prev Next ◭

  6. 16.8 Precision/Recall: at position . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Contents First Last Prev Next ◭

  7. 1. Today main topics We will see: • HCI: NLIDB, QA, IQA, IR • Evaluation Measures: Accuracy, Precision, Recall. Contents First Last Prev Next ◭

  8. 2. HCI via Natural Language In the ’50 Machine Translation work pointed out serious problems in trying to deal with unrestricted, extended text in open domains. This led researchers in the ’60 and early ’70 to focus on question-answering dialogues in restricted domain. Attention shifted from developing NL systems to solving individual language-related problems, e.g., to develop faster, and more efficient parsers. Now, researchers are back to deal with unrestricted extended text and dialogues. 1. NLIDB: NL interface to DB 2. Dialogue Systems 3. QA: Question Answering 4. IQA: Interactive Question Answering 5. IR: Information Retrieval All of them aim at assisting users to access data from some source. Contents First Last Prev Next ◭

  9. 3. Natural Language Interfaces to Data Bases NLIDB refers to systems that allow the user to access information stored in a database by typing requests in some natural language. Its history (see Androutsopoulos for more details): ’60/’70 they were built having a particular DB in mind. No interest in portability issues. E.g., LUNAR late ’70 Dialogues; large DB; semantic grammars (domain dependent - no portable). E.g. LADDER early ’80 From English into Prolog evaluated against Prolog DB. Eg., CHAT-80 mid ’80 popular research area. Research focused on portability issues. E.g. TEAM ’90 NLIDBs did not gain the expected commercial acceptance. Alternative solutions were successful (graphical or form-based interface). Decrease in the nubmer of papers on the topic. Contents First Last Prev Next ◭

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Computational Linguistics: Evaluation Methods Raffaella Bernardi University of Trento Contents

Computational Linguistics: Evaluation Methods Raffaella Bernardi University of Trento Contents

Computational Linguistics: Evaluation Methods Raffaella Bernardi University of Trento Contents First Last Prev Next 1. Admin Perusall sends email reminders to students 3, 2, and 1 day before the deadline of an assignment. Only

477 views • 34 slides
Topics in Computational Linguistics Topics in Computational Linguistics March 28, 2014 GIL,

Topics in Computational Linguistics Topics in Computational Linguistics March 28, 2014 GIL,

. . . . .. . . .. . . .. . . .. . . .. .. . . Topics in Computational Linguistics Topics in Computational Linguistics March 28, 2014 GIL, National Taiwan University Lab of Ontologies, Language Processing and e-Humanities

783 views • 56 slides
Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H AUSSER Computational Linguistics Universitt Erlangen Nrnberg Germany Foundations of Computational Linguistics viii Part IV Semantics and

1.53k views • 130 slides
Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H AUSSER Computational Linguistics Universitt Erlangen Nrnberg Germany Foundations of Computational Linguistics iv Part II Theory of Grammar 7.

1.29k views • 107 slides
Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H AUSSER Computational Linguistics Universitt Erlangen Nrnberg Germany Foundations of Computational Linguistics vi Part III Morphology and

1.64k views • 141 slides
Computational linguistics and NLP: How far from generic linguistics? Andrey Kutuzov University

Computational linguistics and NLP: How far from generic linguistics? Andrey Kutuzov University

Computational linguistics and NLP: How far from generic linguistics? Andrey Kutuzov University of Oslo Language Technology Group with thanks to Joachim Nivre and Abigail See January 17, 2018 Contents What is NLP? 1 Case 1: Redefining parts

2.56k views • 189 slides
Computational Linguistics I CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu

Computational Linguistics I CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu

Computational Linguistics I CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu What is Computational Linguistics? Study of computer processing of natural languages Interdisciplinary field Roots in linguistics and

310 views • 27 slides
Computational Linguistics I CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu

Computational Linguistics I CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu

Computational Linguistics I CMSC 723 / LING 723 / INST 725 M ARINE C ARPUAT marine@cs.umd.edu What is Computational Linguistics? Study of computer processing of natural languages Interdisciplinary field Roots in linguistics and

749 views • 28 slides
Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H

Foundations of Computational Linguistics man-machine communication in natural language R OLAND H AUSSER Computational Linguistics Universitt Erlangen Nrnberg Germany Foundations of Computational Linguistics ii Part I Theory of Language 1.

957 views • 92 slides
Evaluation measures in NLP Zdenk abokrtsk October 30, 2020 NPFL070 Language Data Resources

Evaluation measures in NLP Zdenk abokrtsk October 30, 2020 NPFL070 Language Data Resources

Evaluation measures in NLP Zdenk abokrtsk October 30, 2020 NPFL070 Language Data Resources Charles University Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics unless otherwise stated Outline Evaluation

470 views • 45 slides
Convergences: Convergences Bitext + morph = IGT concern with data bilingual text

Convergences: Convergences Bitext + morph = IGT concern with data bilingual text

Fieldwork as a Computational Problem three data types The Human Language Project: Uniting Computational Linguistics three kinds of metadata with Documentary Linguistics relations Steven Bird computational challenge

398 views • 10 slides
One-Shot Learning: Language Acquisition for Machine SS16 Computational Linguistics for

One-Shot Learning: Language Acquisition for Machine SS16 Computational Linguistics for

One-Shot Learning: Language Acquisition for Machine SS16 Computational Linguistics for Low-Resource Languages Mayumi Ohta July 6, 2016 Institute for Computational Linguistics Heidelberg University Table of contents 1. Introduction 2.

985 views • 70 slides
Computational Linguistics Statistical NLP Aurlie Herbelot 2020 Centre for Mind/Brain Sciences

Computational Linguistics Statistical NLP Aurlie Herbelot 2020 Centre for Mind/Brain Sciences

Computational Linguistics Statistical NLP Aurlie Herbelot 2020 Centre for Mind/Brain Sciences University of Trento 1 Table of Contents 1. Probabilities and language modeling 2. Naive Bayes algorithm 3. Evaluation issues 4. The feature

973 views • 78 slides
Evaluation measures in NLP Zdenk abokrtsk 8th April 2020 NPFL124 Natural Language

Evaluation measures in NLP Zdenk abokrtsk 8th April 2020 NPFL124 Natural Language

Evaluation measures in NLP Zdenk abokrtsk 8th April 2020 NPFL124 Natural Language Processing Charles University Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics unless otherwise stated Title of your

597 views • 32 slides
Why does NLP need linguistics? Julia Hockenmaier juliahmr@illinois.edu NLP and Linguistics:

Why does NLP need linguistics? Julia Hockenmaier juliahmr@illinois.edu NLP and Linguistics:

Why does NLP need linguistics? Julia Hockenmaier juliahmr@illinois.edu NLP and Linguistics: Finding the Common Ground 48th ACL, July 2010 Uppsala, Sweden 1 Why do computational linguists need to know about linguistics? 2 Why do

838 views • 44 slides
Penalty Functions for Evaluation Measures of Unsegmented Speech Retrieval Petra Galukov,

Penalty Functions for Evaluation Measures of Unsegmented Speech Retrieval Petra Galukov,

Penalty Functions for Evaluation Measures of Unsegmented Speech Retrieval Petra Galukov, Pavel Pecina, Jan Haji Institute of Formal and Applied Linguistics Charles University in Prague {galuscakova,pecina,hajic}@ufal.mff.cuni.cz

520 views • 20 slides
TOEIC speaking and writingrespond to a request Fri 01 Nov 2019 11:48:17 AM CST

TOEIC speaking and writingrespond to a request Fri 01 Nov 2019 11:48:17 AM CST

Respond to a request TOEIC speaking and writingrespond to a request Fri 01 Nov 2019 11:48:17 AM CST http://github.com/drbean/curriculum/master/speaking TOEIC speaking and writingrespond to a request Dr Bean ( ) Respond to a

396 views • 4 slides
OpenCms Hispano s Modules OpenCms Hispano Sergio Raposo Vargas Alejandro Alves Caldern

OpenCms Hispano s Modules OpenCms Hispano Sergio Raposo Vargas Alejandro Alves Caldern

OpenCms Hispano s Modules OpenCms Hispano Sergio Raposo Vargas Alejandro Alves Caldern Modules overview Index Forum module Components Classes Future Demo Advanced direct edit Classes How it works Future

768 views • 18 slides
Enterprise AOP With the Spring Framework Jrgen Hller VP & Distinguished Engineer,

Enterprise AOP With the Spring Framework Jrgen Hller VP & Distinguished Engineer,

Enterprise AOP With the Spring Framework Jrgen Hller VP & Distinguished Engineer, Interface21 Agenda Spring Core Container Spring AOP Framework AOP in Spring 2.0 Example: Transaction Advice What's Coming in Spring

558 views • 30 slides
Acoustic and perceptual features of eating-places . PerMagnus Lindborg Nanyang Technological

Acoustic and perceptual features of eating-places . PerMagnus Lindborg Nanyang Technological

Acoustic and perceptual features of eating-places . PerMagnus Lindborg Nanyang Technological University, Singapore (asst. prof.) KTH Royal Institute of Technology, Stockholm (doct. cand.) permagnus@ntu.edu.sg www.permagnus.net Monday, July 29,

235 views • 20 slides
Mapletree Commercial Trust 1H FY20/21 Financial Results 22 October 2020 0 Important Notice

Mapletree Commercial Trust 1H FY20/21 Financial Results 22 October 2020 0 Important Notice

Mapletree Commercial Trust 1H FY20/21 Financial Results 22 October 2020 0 Important Notice This presentation is for information only and does not constitute an offer or solicitation of an offer to sell or invitation to subscribe for or acquire

806 views • 29 slides
Service Layer Patterns Introduction to Software Architecture Jay Urbain, Ph.D. urbain@msoe.edu

Service Layer Patterns Introduction to Software Architecture Jay Urbain, Ph.D. urbain@msoe.edu

Service Layer Patterns Introduction to Software Architecture Jay Urbain, Ph.D. urbain@msoe.edu References: Please review last slide 1 2 Service-oriented architecture (SOA) Style of software design where services are provided to the

1.46k views • 47 slides
Possessive Marking in Idioms Manfred Sailer Goethe University, Frankfurt a.M. November 16, 2015

Possessive Marking in Idioms Manfred Sailer Goethe University, Frankfurt a.M. November 16, 2015

Possessive Marking in Idioms Manfred Sailer Goethe University, Frankfurt a.M. November 16, 2015 Sailer (GU Frankfurt) European HPSG Meeting November 16, 2015 1 / 45 Overview Introduction 1 External possessor in German 2 Additional

676 views • 49 slides
Whats the smallest part of spinach? A new experimental approach to the count/mass distinction

Whats the smallest part of spinach? A new experimental approach to the count/mass distinction

Whats the smallest part of spinach? A new experimental approach to the count/mass distinction Sea Hee Choi & Tania Ionin University of Illinois at Urbana-Champaign Experiments in Linguistic Meaning Conference University of Pennsylvania

559 views • 38 slides

More recommend