IR in Context of the User: Interactive IR Evaluation Peter Ingwersen Royal School of LIS Denmark pi@iva.dk – http://www.iva.dk/pi Oslo University College, Norway Essir2011 Ingwersen 1
Agenda - 1 Introduction (20 min) Research Frameworks vs. Models Central components of Interactive IR (IIR) The Integrated Cognitive Research Framework for IR From Simulation to ‘Ultra-light’ IIR (20 min) Short-term IR interaction experiments Sample study – Diane Kelly (2005/2007) Essir2011 Ingwersen 2
Agenda - 2 Experimental Research Designs with Test persons (25 min) Interactive-light session-based IR studies Request types Test persons Design of task-based simulated search situations Relevance and evaluation measures in IIR Sample study – Pia Borlund (2000; 2003b) Essir2011 Ingwersen 3
Agenda - 3 Naturalistic Field Investigations of IIR (20 min) Integrating context variables Live systems & (simulated) work tasks Sample Study – Marianne Lykke (Nielsen) (2001; 2004) Wrapping up (5 min) Questions are welcome during the sessions Advanced Information Retrieval / Ricardo Baetza Yeates & Massimo Melucci (eds.). Springer, 2011, p. 91-118. Essir2011 Ingwersen 4
Frameworks & Models – difference? Frameworks describe The concept model Essential objects to study A precise (often formal) representation of objects and The relationships of objects relationships (or processes) The changes in the objects / within a framework relationships that affect the Modeling may also in principle functioning of the system encompass human actors and Promising goals and methods organizations of research Frameworks may lead to Frameworks contain (tacit) shared assumptions Research Designs, incl. ontological, conceptual, factual, Research Questions; epistemological, and Experimental Setting; methodological Methodology Essir2011 Ingwersen 5
The Lab. Research Framework – cave with central variables ( The Turn , 2005) Docu- Search ments request Relevance Represen- Represen- assessment tation tation Pseudo RF Database Query Matching Query Result Recall Evaluation Evaluation base Result 7 Essir2011 Ingwersen 6 Essir2009 Ingwersen
User-centered (contextual) MODELS Examples (in display order) Wilson, 1999 (conceptual: Info. Behavior; Seek; IR) Byström & Järvelin, 1995 (flow chart: Info. Seek) Saracevic, 1996 ( conceptual, stratified: IR) Vakkari, 2000 (flow chart, Online Search; Relevance) Wang & Soergel, 1998 (conceptual: Relevance Assessment Process & Criteria) Essir2011 Ingwersen 7
Information behaviour and IR T. Wilson´s Onion Model, 1999 - extended: Job-related Non-job-related Work Tasks Tasks and Interests Interests Daily-life behavior Seeking Interactive IR IR Information behaviour Behaviour 2011 8 Peter Ingwersen
Perceived Perceived Task Task Situational Situational Personal Personal Factors Factors Factors Factors Information Information Need analysis Need analysis Organization Organization Choice of Action Choice of Action - identification of - identification of IS&R model, alternatives alternatives Personal Personal - ranking them - ranking them 1995: Bystöm & Seeking style Seeking style - choosing an action - choosing an action Järvelin, fig. 2 Implementation Implementation Evaluation Evaluation a) needs satisfied, task may a) needs satisfied, task may be completed be completed b) needs cannot be satisfied b) needs cannot be satisfied c) further information is c) further information is needed needed (From: The Turn, p. 69 ) 2011 Peter Ingwersen 9
Saracevic´ stratified model for IIR (1996) 2011 Peter Ingwersen 10
Wang & Soergel 1998 Decision Rules Knowledge of Elimination topic Multiple criteria person Dominance organization Scarcity journal Satisfice document type Chain Document DIEs Criteria Values Decision processing combining deciding Title Topicality Epistemic Acceptance Author Orientation Functional Maybe Abstract Conditional Rejection Social Quality Emotional Journal Novelty Series Availability Date Authority Type Relation (From: The Turn, p. 201) DIEs: Document Information Elements Values: Document Values/Worth 2011 11 Peter Ingwersen
IR and relevance in Seeking context – Seeking into IS&R: Vakkari 2000 2011 Peter Ingwersen 12
Research Setting Types Laboratory experiments – no test persons, but Simulations – Log analyses (not treated in presentation) Laboratory study – with test persons: ‘ Ultra-light ’ (short-term interaction: 1-2 retrieval runs) – or ‘ Interactive light ’ (session-based multi-run interaction) Field experiment – experimental (artificial) situation in natural setting with test persons Field study – study of natural performance or behavior in natural setting with test persons Longitudinal studies Case study – (qualitative) study with few test persons Essir2011 Ingwersen 13
Variables involved in a test: Independent (the ‘cause’) , e.g., Interface functionalities; Different IR models; Searcher knowledge Dependent (the ‘effect’) , e.g., Performance measures of output (recall/prec.; CumGain; usability) Controlled (held constant; statistically neutralized; randomized): Database; Information objects Search algorithms Simulated work task situations – Assigned TREC topics Test persons Hidden variables (Moderating or Intervening) , e.g., Variation of test persons’ levels of experience …!!! – see the Integrated Research Framework for IR Essir2011 Ingwersen 14
Agenda - 1 Introduction to Tutorial (20 min) Research Frameworks vs. Models Central components of Interactive IR (IIR) The Integrated Cognitive Research Framework for IR From Simulation to ‘Ultra-light’ IIR (20 min) Short-term IR interaction experiments Sample study – Diane Kelly (2005/2007) Essir2009 Essir2011 Ingwersen 15 15
Central Components of Interactive IR – the basic integrated framework Information Information objects objects Org. Org. Cognitive Cognitive Social Social Interface Interface Actor(s) Actor(s) Context Context (team) (team) IT: Engines IT: Engines Cultural Cultural Logics Logics Algorithms Algorithms Essir2011 Ingwersen 16
Central Components of Interactive IR – the basis of the integrated framework Information objects Org. Cognitive Social Interface Actor(s) Context (team) IT: Engines Cultural Logics Algorithms = Cognitive transformation and influence = Interactive communication of cognitive structures = Cognitive transformation and influence over time The Lab. Framework Essir2011 Ingwersen 17
Dimensions and Range of Variables in the Integrated IIR framework: 9 dimensions from 6 components Information Information objects objects Org. Org. Cognitive Cognitive Interaction Social Social Interface Interface Actor(s) Actor(s) Context Context (team) (team) IT: Engines IT: Engines Cultural Cultural Logics Logics Algorithms Algorithms Essir2011 Essir2009 Ingwersen Ingwersen 18 18
Categories of Dimensions i the Cognitive Research Framework Natural work task dimension Socio-org. task 1. dimensions Natural search task dimension 2. Actor characteristics dimension 3. Actor dimensions Perceived work task dimension 4. Perceived search task 5. Each containing multiple Document dimension 6. variables Algorithmic search engine dimension 7. Algorithmic dimensions Algorithmic interface dimension 8. Access and interaction dimension 9. Essir2011 Essir2009 Ingwersen Ingwersen 19 19
Natural Work Natural Search Actor Perceived Work Perceived Tasks (WT) & Tasks (ST) Tasks Search Tasks Org WT Structure ST Structure Domain Perceived WT Perceived Knowledge Structure Information Need Content WT Strategies & ST Strategies & IS&R Knowledge Perceived WT Perceived ST Practices Practices Strategies & Structure/Type Practices WT Granularity, ST Granularity, Experience on Perceived WT Perceived ST Size & Complexity Size & Complexity Work Task Granularity, Size Strategies & & Complexity Practices WT Dependencies ST Dependencies Experience on Perceived WT Perceived ST Search Task Dependencies Specificity & Complexity WT Requirements ST Requirements Stage in Work Perceived WT Perceived ST Task Execution Requirements Dependencies WT Domain & ST Domain & Perception of Perceived WT Perceived ST Context Context Socio-Org. Domain & Context Stability Context Variables Sources of Perceived ST with Difficulty Domain & Context Motivation & values Emotional State Essir2011 Essir2009 Ingwersen Ingwersen 20 20
Recommend
More recommend
