Evaluation of Rich and Explicit Feedback for Exploratory Search - PowerPoint PPT Presentation

Nov 24, 2022 •483 likes •573 views

Evaluation of Rich and Explicit Feedback for Exploratory Search Esben Srig 1 , Nicolas Collignon 2 , Rebecca Fiebrink 1 , and Noriko Kando 3 1 Goldsmiths, University of London 2 University of Edinburgh 3 National Institute of Informatics, Japan

Evaluation of Rich and Explicit Feedback for Exploratory Search Esben Sørig 1 , Nicolas Collignon 2 , Rebecca Fiebrink 1 , and Noriko Kando 3 1 Goldsmiths, University of London 2 University of Edinburgh 3 National Institute of Informatics, Japan
Annotation feedback for exploratory IR • Current search systems have limited support exploratory searchers • Document annotation is a well- known strategy for active reading • We investigate the usage of annotations as an explicit feedback signal for IR systems. Alighieri, Dante. 1993. Convivio. Ed. Giorgio Inglese. Milano: Rizzoli
Previous work • Relevance feedback is a well-studied interactive, explicit feedback mechanism. User marks documents as relevant or irrelevant • Variants of relevance feedback such as interactive query expansion • Golovchinsky et al 1 studied the effect of annotation feedback on retrieval performance • Significant improvement in retrieval performance in single iteration of feedback • Fixed query, fixed search results, non-interactive, no user experience 1 G. Golovchinsky, M. N. Price, B. N. Schilit. 1999. From reading to retrieval: freeform ink annotations as queries. In Prooceedings of ACM SIGIR '99. ACM. 19-25.
Simulation retrieval performance • TREC Dynamic Domain 2017 user simulator • New York Times Annotated Corpus • Performance averaged over 60 search tasks
Simulation retrieval performance • TREC Dynamic Domain 2017 user simulator • New York Times Annotated Corpus • Performance averaged over 60 search tasks
Evaluation platform for annotation feedback
Experiment with human subjects • Goal : measure retrieval performance and user experience of real search users of annotation feedback • Two conditions: 1) Relevance feedback only. 2) Annotation feedback only • Between-subjects design • Amazon Mechanical Turk participants • Tasks chosen from user simulator • Pre- and post-task questionnaires to measure user experience
Discussion and questions • Is evaluation on TREC dataset representative of multi-session exploratory search contexts? • Are questionnaires sufficient to measure hypothesized benefit to user exploration, understanding, and engagement? • Questions?

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

FedRec: Federated Recommendation with Explicit Feedback Guanyu Lin 1 , 2 # , Feng Liang 1 , 2 # ,

FedRec: Federated Recommendation with Explicit Feedback Guanyu Lin 1 , 2 # , Feng Liang 1 , 2 # , Weike Pan 1 , 2 and Zhong Ming 1 , 2 { linguanyu20161, liangfeng2018 } @email.szu.edu.cn, { panweike, mingz } @szu.edu.cn 1 National

801 views • 38 slides

Student Evaluation of Learning and Feedback for Instructors Student Evaluation of Learning and

Student Evaluation of Learning and Feedback for Instructors Student Evaluation of Learning and Feedback for Instructors SELFI Initial USAT Charge Phase 1 USAT committee (2014 15) Adopt a new set of quantitative (Likert style) questions

610 views • 31 slides

An Explicit Formula for the Zero-Error Feedback Capacity of a Class of Finite-State Additive

An Explicit Formula for the Zero-Error Feedback Capacity of a Class of Finite-State Additive Noise Channels Amir Saberi *, Farhad Farokhi and Girish Nair EEE Department, The University of Melbourne Email: a.saberi @ student.unimelb.edu.au ISIT

491 views • 32 slides

Evaluation or check ? Evaluation purpose? Feedback from exercise-lessons or best practise

Evaluation or check ? Evaluation purpose? Feedback from exercise-lessons or best practise Test the performance/capacity. Improving manuals-implementing result Improving the evaluation itself Goals from the old HELCOM

189 views • 16 slides

Explicit-State Abstraction: A New Method Abstractions for Generating Heuristic Functions

Explicit-State Abstraction Explicit-State Abstraction: A New Method Abstractions for Generating Heuristic Functions Projections Explicit-State Abstractions Evaluation Malte Helmert 1 Patrik Haslum 2 org Hoffmann 3 J Conclusion 1

485 views • 25 slides

Exploratory Analysis of Teaching Evaluation Data John Jordi, Ph.D., May 7, 2018 Learning &

Exploratory Analysis of Teaching Evaluation Data John Jordi, Ph.D., May 7, 2018 Learning & Organizational Development Specialist Office of Faculty Development & Teaching Excellence Introduction & Research Questions Student

154 views • 11 slides

NSF L.I.F.E. form Introduction February 4, 2020 Todd Abraham, PhD, MBA The Feedback Process

NSF L.I.F.E. form Introduction February 4, 2020 Todd Abraham, PhD, MBA The Feedback Process Level-of-Interest Feedback Evaluation aka L.I.F.E. Level-of-Interest Feedback Evaluation On-Line Feedback for Scientists on project proposals

245 views • 22 slides

Matrix Co-factorization for Recommendation with Rich Side Information and Implicit Feedback Yi

Matrix Co-factorization for Recommendation with Rich Side Information and Implicit Feedback Yi Fang and Luo Si Department of Computer Science Purdue University West Lafayette, IN 47906, USA fangy@cs.purdue.edu HetRec 2011 Yi Fang and Luo Si

367 views • 23 slides

Explicit and Implicit Discourse Relations: An Extrinsic Evaluation Peter Bourgonje and Manfred

Explicit and Implicit Discourse Relations: An Extrinsic Evaluation Peter Bourgonje and Manfred Stede Applied Computational Linguistics Universitt Potsdam Workshop on Coherence Relations Humboldt-Universitt zu Berlin January 17-18, 2020

342 views • 21 slides

Unbiased Offline Recommender Evaluation for Missing-Not-At-Random Implicit Feedback Serge

Unbiased Offline Recommender Evaluation for Missing-Not-At-Random Implicit Feedback Serge Belongie Deborah Estrin Lo Longqi Yang Yuan Xuan Chenyang Wang Yin Cui Funders: 1 Offline Evaluation of Recommendation Algorithm user-item

619 views • 36 slides

Course Feedback Uses: Improvement and Formal Evaluation Fall Faculty Conference 2018 Student

Course Feedback Uses: Improvement and Formal Evaluation Fall Faculty Conference 2018 Student Course Feedback Student Course Feedback Conversations with Students The Basics Tell your students: You value honest and constructive

664 views • 13 slides

Announcements: - Thank you for participating in our mid-quarter evaluation Thank you for

Announcements: - Thank you for participating in our mid-quarter evaluation Thank you for participating in our homework feedback polls! - - Course project - Average was ~80% Dont worry about grade but take feedback seriously - -

423 views • 41 slides

An Exploratory Study of How Developers Exploratory Study Seek, Relate, and Collect Relevant

Code Investigation Z ephyrin Soh Context and Paper An Exploratory Study of How Developers Exploratory Study Seek, Relate, and Collect Relevant Results Activities Information during Software Strategy Search Relate Maintenance Tasks

397 views • 23 slides

Exploratory Monitoring at Bing AUTOMATED SYNTHETIC EXPLORATORY MONITORING OF DYNAMIC WEB SITES

Exploratory Monitoring at Bing AUTOMATED SYNTHETIC EXPLORATORY MONITORING OF DYNAMIC WEB SITES USING SELENIUM Outline 1. Modern Engineering Principles 2. Monitoring Approaches 3. Statistical Models 4. Selenium 5. Exploratory Principles 6.

618 views • 26 slides

Health benefits of biofortification an economic ex-ante evaluation of iron-rich rice and wheat

Health benefits of biofortification an economic ex-ante evaluation of iron-rich rice and wheat in India Alexander J. Stein, J.V. Meenakshi, Matin Qaim, Penelope Nestel, H.P.S. Sachdev and Zulfiqar A. Bhutta HarvestPlus/IFPRI, Washington,

531 views • 40 slides

The main messages emerging from our analysis of data, self-evaluation and feedback from key

The main messages emerging from our analysis of data, self-evaluation and feedback from key stakeholders is: Keep going you are doing the right things, We are having an impact on learner outcomes and sustaining improvement where it is

729 views • 29 slides

Welcome to the 4th Annual Honoree Athletic Banquet Athletic Support Team Transportation :

Welcome to the 4th Annual Honoree Athletic Banquet Athletic Support Team Transportation : Anne Wright Lynn Shadrix Finance: Yolanda Chisolm Equipment Maintenance: Marty Monette Patrick Halstead Patrick Matthews Field

763 views • 49 slides

Overview of the sufficiency of measures analysis Co-funded by the European Union 18 June 2019

Overview of the sufficiency of measures analysis Co-funded by the European Union 18 June 2019 Business as Business as usual usual (BAU) U) state state GES GES Environmental status gap Example business Possible BAU states as

520 views • 13 slides

Council Deliberations November 25, 2019 2020 Proposed Business Plan and Budget 2020 Budget

2020 Proposed Business Plan and Budget Council Deliberations November 25, 2019 2020 Proposed Business Plan and Budget 2020 Budget Objectives 1. Maintain base programs and services aligned with inflation Incorporate Establish link 2. Provide

660 views • 20 slides

Data structure Mapping data What data you need to entry Exact location (case specific)

Attributes and features Data structure Mapping data What data you need to entry Exact location (case specific) Aggregated data by administrative levels (aggregated) Aggregated data are usually represented by polygons A field for the

673 views • 44 slides

Techniques to improve Dictionary Based CLIR Sai Madhurya Peyyeti KX48810 Different Techniques

Techniques to improve Dictionary Based CLIR Sai Madhurya Peyyeti KX48810 Different Techniques in IR Translation is the key problem in CLIR Query Translation u Dictionary Based Corpora Based Dis-Adv: lack of resources Machine

452 views • 11 slides

QUERY AND DOCUMENT EXPANSION IN TEXT RETRIEVAL Clara Isabel Cabezas University of Maryland

QUERY AND DOCUMENT EXPANSION IN TEXT RETRIEVAL Clara Isabel Cabezas University of Maryland College Park May, 2 nd 2000 1 1.- Definition 2.- Query expansion a) Different approaches b) Application example: Ballesteros and Croft c)

689 views • 37 slides

Reinforcing patient relevance in evidence generation Feedback from breakout session 5C EMAs

Reinforcing patient relevance in evidence generation Feedback from breakout session 5C EMAs Regulatory Science Strategy to 2025 Human Stakeholder Workshop Presented by Bruno Sepodes, CHMP and Koenraad Norga, PDCO on 19 November 2019 An

115 views • 8 slides

Classification and Machine Learning techniques for CBIR: introduction to the RETIN system

Classification and Machine Learning techniques for CBIR: introduction to the RETIN system Matthieu Cord ETIS CNRS UMR 8051 Classification and Machine Learning techniques for CBIR: introduction to the RETIN system p.1/81 Content-Based Image

1.37k views • 81 slides