Considering Qualitative Evaluation University of Calgary: Sheelagh - - PowerPoint PPT Presentation

Considering Qualitative Evaluation

University of Calgary: University of St. Andrews: Cork Institute of Technology: Tableau: Sheelagh Carpendale, Alice Thudt, Jo Vermeulen, Jagoda Walny Uta Hinrichs Trevor Hogan Melanie Tory

Overview

Talk 1: Setting the stage: qualitative Talk 2: Observation as a skill Activity 1: Practicing observation Activity 2: Interviewing 1 Activity 3: Interviewing 2 Talk 3: Interviewing as a skill Activity 4: Interviewing 3 Panel: open discussion Talk 4: In summary – next step analysis

Choosing an evaluative approach

start with a question relate the question to existing research

in contrast in agreement as extension or variation

choose a methodology

Goals of Empirical Research

McGrath, J.: Methodology Matters: Doing Research in the Social and Behavioural Sciences. In: Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, San Francisco (1995)

REALISM GENERALIZABILITY PRECISION

Goals of Empirical Research

REALISM GENERALIZABILITY PRECISION

results can apply to other people or situations

McGrath, J.: Methodology Matters: Doing Research in the Social and Behavioural Sciences. In: Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, San Francisco (1995)

Goals of Empirical Research

REALISM GENERALIZABILITY PRECISION

control of factors that were not studied

McGrath, J.: Methodology Matters: Doing Research in the Social and Behavioural Sciences. In: Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, San Francisco (1995)

Goals of Empirical Research

REALISM GENERALIZABILITY PRECISION

context of study is like the context of use

McGrath, J.: Methodology Matters: Doing Research in the Social and Behavioural Sciences. In: Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, San Francisco (1995)

abstract

experimental simulation field experiment

REALISM

unobtrusive concrete

lab experiment judgment study sample survey formal theory computer simulation field study

• btrusive

PRECISION GENERALIZABILITY

McGrath, J.: Methodology Matters: Doing Research in the Social and Behavioural Sciences. In: Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, San Francisco (1995)

today focus is on realism and qualitative data gathering methods

Albert Einstein

‘Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted’

Qualitative Research

• study aspects that can not necessarily be counted
• rich in-depth understanding
• considers the interplay among factors
• helps investigate open-ended questions
• can provide insights beyond the expected
• can generate design criteria and more specific

hypotheses

Lofland, J., Lofland, L.: Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Wadsworth Publishing Company, CA, USA (1995) Seidman, I.: Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences. Teachers’ College Press, New York (1998)

Qualitative Data

• primary methods for gathering data:

– observations – interviews

• types of data records:

– field notes – journal entries – artefacts (e.g. sketches, diagrams, annotations) – audio & video tapes – screen captures – computer logs – …

Lofland, J., Lofland, L.: Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Wadsworth Publishing Company, CA, USA (1995) Seidman, I.: Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences. Teachers’ College Press, New York (1998)

Data Collection as a Skill

• qualitative data collection requires skill and sensitivity
• observation and interviewing skills can develop with

practice and can be learnt

• follows rigorous methods

Lofland, J., Lofland, L.: Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Wadsworth Publishing Company, CA, USA (1995) Seidman, I.: Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences. Teachers’ College Press, New York (1998)

Qualitative Methodologies

• primarily qualitative methodologies (focus of this tutorial)
• mixed methodologies: qualitative methods in

conjunction with quantitative methods

• heuristic inspection methodologies (not discussed here)

Qualitative Methods as Primary

• develop a rich in-depth understanding
• used at any time in the development life cycle

– finished visualization or prototype assessed e.g. via field studies – during the design process – formative studies to inform the initial design

• variety of methods to choose from

– in-situ observations – immersive observations – laboratory observational studies – contextual interviews – …

Mixed Methods

• Qualitative methods can be used in conjunction with

quantitative methods to:

– provide context for quantitative results – gather participants opinions and preferences – raise new questions – aid the interpretation of quantitative results

• Common mixed methods approaches:

– experimenter observations – think aloud protocol – collecting opinions (e.g. through interviews or Likert scales)

Gorard, S.: Combining Methods in Educational Research. McGraw-Hill, New York (2004)

Characteristics of Qualitative Methods

• involved and rigorous process that requires practice

– gathering data and rich note taking – transcription – in-depth analysis

Strauss, A.L., Corbin, J.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications, London (1998) Auerbach, C.: Qualitative Data: An Introduction to Coding and Analysis. University Press, New York (2003)

Characteristics of Qualitative Methods

• sample sizes can differ from quantitative experiments

– not concerned with making statistically significant statements – focussed on ‘saturation’ – can be determined during the study

Strauss, A.L., Corbin, J.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications, London (1998) Auerbach, C.: Qualitative Data: An Introduction to Coding and Analysis. University Press, New York (2003)

• U. Hinrichs, S. Forlini, and B. Moynihan. Speculative Practices: Utilizing InfoVis to Explore Untapped Literary Collections. IEEE

Transactions on Visualizations and Computer Graphics, 22(1), 2016.

• C. Neustaedter and P. Sengers. Autobiographical Design in HCI Research:

Designing and Learning through Use-It-Yourself. In Proc. of DIS, pages 514–523, 2012.

Characteristics of Qualitative Methods

• subjective perspective as an asset

– awareness and explicit discussion of one’s own ‘lens’

Strauss, A.L., Corbin, J.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications, London (1998) Auerbach, C.: Qualitative Data: An Introduction to Coding and Analysis. University Press, New York (2003)

Qualitative Methods in Visualization

Qualitative methods can help to enhance our understanding of:

– Experiences beyond usability – Environments and work practices – Visual data analysis and reasoning – Communication through visualization – Collaborative visual analysis

Lam, H., Bertini, E., Isenberg, P., Plaisant, C., & Carpendale, S. (2012). Empirical studies in information visualization: Seven scenarios. IEEE Transactions on Visualization and Computer Graphics, 18(9), 1520-1536.

Summary

• carefully choose method to fit the research goal
• consider trade-offs between realism,

generalizability and precision

• borrow and adapt methods from other fields
• conduct research with rigor specific to the

chosen method

• publish sufficient details
• make claims appropriate to the chosen method