Experimentation in Virtual Environments Will Steptoe 29 th January - - PowerPoint PPT Presentation

Experimentation in Virtual Environments

Will Steptoe 29th January 2010

What’s in this lecture?

• 1. Introduction to experimentation in VEs
• 2. Experimental Design
• 3. Case study

What is an Experiment?

• A set of actions and observations, performed in

the context of solving a particular problem or question, to support or falsify a hypothesis or research concerning phenomena.

• The experiment is a cornerstone of the empirical

approach to acquiring deeper knowledge about the physical world.

Why Experiment?

• Causal explanation vs. Informal description
• Parameters and parameter estimation
• Counterfactual evidence to existing theory
• Problem solving
• Technology exploration, development, and

refinement

Why Experiment in VEs?

• Free from physical implications and limitations

– Dangerous situations including training and military – Ethical considerations are different – “Transformed” social interaction allows one to break normal rules of physical interaction because users do not actually have to share the same “reality”.

• Applications

– Opportunities afforded by VEs – Ubiquity of HCI – Increasing transition to 3D displays and interfaces featuring natural input gesture in daily work and entertainment

Why Experiment in VEs?

Why Experiment in VEs?

Collecting data Identifying a purpose and stating questions Reporting and evaluating the study Identifying a research problem

Big picture of experimentation

Defining Research Methods Reviewing the literature Analyzing and interpreting data

Purpose Statement

• “Why you want to do the study and what you

intend to accomplish”

Locke, Spirduso, and Silverman, 2000.

• Most important aspect of an experiment and
• rients the reader to the central intent of the study

Review of the literature

• What has been studied?
• How was it done?
• What implications arise?
• Identify gaps
• Relate to research method
• http://scholar.google.co.uk
• http://academic.research.microsoft.com
• http://liinwww.ira.uka.de/bibliography/index.html

Research Questions

• 3-7 questions that extend from purpose statement.
• Contextualise research with existing literature.
• Present scope of study (i.e. what you are and are

not considering).

• Will usually mention attributes of what you are

studying (variables).

Experimental Hypotheses

• Predictions of relationship among variables.
• Must be ‘falsifiable’:

– Must be possible to obtain evidence that could lead to its rejection

• Carry out some procedure that may be used to

test the hypothesis (i.e. gather ‘data’).

• May be:

– Rejected – Confirm – Neither

Variables

• Characteristic or attribute of an individual or an
• rganisation that can be measured or observed

and that varies among the people or organisation being studied.

– Creswell 2002

• Distinguished by temporal order and

measurement.

– Temporal order means that one variable precedes another in time, and because of this order, affects or “causes” another variable.

Independent Variables

• The ones we manipulate or change.
• Variables that (probably) cause, influence, or

affect outcomes.

• Also called treatment, manipulated, antecedent, or

predictor variables.

Dependent Variables

• The ones we measure.
• Variables that depend on the independent

variables.

• Outcomes or results of the influence of the

independent variables.

• Also known as criterion, outcome, and effect

variables.

Control Variables

• The ones we keep the same.
• Variables that if they were not controlled, would

influence the dependent variable.

Intervening or Mediating Variables

• The ones we must acknowledge.
• Variables that “stand between” the independent

and dependent variables.

• Mediate the effects of the independent variables
• n the dependent variable.

Example of Variables

• Pit experiment

– Independent

• Rendering fidelity, use of haptics.

– Dependent

• Heart rate, body movement.

– Control

• everything apart from the independent variables! Includes

procedure prior to experiment.

– Mediating

• Previous experience of VR or games, age, sex, personality (i.e.

“openness” to experience).

Purpose Statement Revisited

• Design of purpose statement begins with

identifying the proposed variables for a study, and specifying how the variables will be measured or observed.

Approaches to research

• Qualitative
• Quantitative
• Mixed

Qualitative Approach

• Non-numerical data collection and explanation.
• Themes developed from open-ended emerging data

For example, if you are asked to explain in qualitative terms a thermal image displayed in multiple colours, then you would explain the colour differences rather than the heat's numerical value.

• Constructivist perspectives: multiple meanings of

individual experiences that are socially and historically constructed, with intent to developing a theory or pattern.

Qualitative Data Collection Methods

– Questionnaires

• May be recorded on numerical scale (i.e. Likert scale).

– Phenomenologies

• “Essence” of human experience concerning a phenomenon, as

described by participants (i.e. post-experimental interviews).

– Ethnographies

• Observing a cultural group generally over extended time.

– Grounded theory studies

• Derivation of general, abstract theory of a process or interaction

grounded in the views of participants (i.e. interviews)

– Narratives

• Worded descriptions over extended time formed into narrative

chronology.

Quantitative Approach

• Numerical data collection and statistical analysis.
• Measurement is central to quantitative research

because it provides the fundamental connection between empirical observation and mathematical expression of quantitative relationships.

• Postpositivist claims for developing knowledge:

cause and effect, reduction to specific variables and hypotheses, use of measurement and

• bservation, and the test of theories.

Quantitative Data Collection Methods

• Devices able to gather numbers of anything

measurable:

– Body Tracking (head, hand, eye ... ) – Physiological response (heart rate, skin conductance, pupil dilation) – Performance data (time taken, errors, motion paths, decisions made)

Mixed methods

• Collecting, analysing, and integrating quantitative and

qualitative research and data.

• The purpose of this form of research is that both

qualitative and quantitative research, in combination, provide a better understanding of a research problem

• r issue than either research approach alone.
• Mixing the data (merging of quantitative and

qualitative data must be kept distinct but connected).

• Raises issues: how, what, where, why, value (of

mixing).

Mixed methods is particularly suited to VEs

• VEs mediate users’ sensory input channels with digital

stimuli.

– Suited to quantitative data collection as the data is delivered in numeric form.

• However, a key feature of VEs is that they are able to

elicit natural behaviour due to high levels of immersion and presence. This means that we are studying natural human perception, behaviour, and interaction.

– Suited to subjective data relating to an individual’s experience in a VE.

Selecting an approach

• Match between problem and approach

– Quantitative suited to test theory, identifying factors that influence an outcome, the use of a technology, or understanding best predictors of outcomes. – Qualitative often suited to studies that are exploratory in nature, in order to identify potential variables for future investigation. – Mixed methods can provide best of both, generalising findings (quantitative) and developing detailed view of the meaning of a phenomenon or concept for individuals (qualitative).

Ethical Considerations

• Standard Principles in Ethics

– British Psychological Society http://www.bps.org.uk/

• For ethical reasons some areas of human

experience and behaviour may be beyond the reach of experiment, observation or other form of psychological investigation.

• The investigation should be considered from the

standpoint of all participants; foreseeable threats to their psychological well-being, health, values or dignity should be eliminated.

Ethical Considerations

• Where possible inform participants about the

purposes of the experiment.

– When would it not be possible?

• The investigator should inform the participants of

all aspects of the research or intervention that might reasonably be expected to influence willingness to participate.

• Consent, deception, withdrawal, payment, debrief,

confidentiality, physical protection.

Experimental Design

• A within-groups design is an experiment in which

the participants serve in more than one treatment.

– Advantages include reduction in error variance associated with individual differences, fewer subjects required, thereby increasing statistical power. – Weaknesses include “carryover effects” arising from subjects performing multiple experimental sessions, thereby introducing effects of learning and fatigue. This can be minimised by randomising orders of treatment

• ver subjects.

Experimental Design

• A between-groups design is an experiment in

which participants are exposed to only one treatment.

– Advantages include no confounding effects of learning

• r fatigue.

– Weaknesses include the data being more susceptible to individual differences in participants, more participants required, possibly not suitable for studies intended to gather comparison data.

Experimental Design

• Selection of within-groups or between-groups

design is often a natural decision based on restrictions of variables and focus of hypotheses.

Experiment!

• (Following detailed preparation, often involving

pilot studies and refining experimental procedure.)

Case Study

• Two-part study investigating:
• 1. Social presence in avatar- and video-mediated

telecommunication

• 2. Media richness in avatar-mediated telecommunication

Case Study

• Purpose statement:

– Avatar-mediated communication in which users are embodied by graphical humanoids within a shared virtual environment (VE), has rapidly increased in prevalence and popularity as an emerging form of visual remote interaction. – As this new mediator of social interaction develops, we need to investigate potentials and limitations particularly in terms of nonverbal expressivity. – We frame our work in the social domain of interpersonal trust and deception, which we consider to present a compelling array of issues by which to investigate social interaction in visual telecommunication systems.

Case Study

• Literature review:

– Theories of social presence and media richness. Intended to contextualise in broader domain of HCI. – State-of-art in avatar-mediated communication (AMC) and video-mediated communication (VMC). – Deception, detection of deception. Emphasis on

• culesics (eye behaviour including gaze, blinking and

pupil dilation) and eye tracking, as this was the primary data collection technique.

Case Study

• Hypotheses for experiment 1 (E1):

– During AMC and VMC, participants will exhibit similar

• culesic behaviours of gaze, blink rate, and pupil size.

However, psychological arousal measured by questionnaire data will be greater following VMC. – When communicating deceptive messages in both mediums: participants’ proportion of gaze directed at their partner will contrast to that measured when truth telling; participants’ blink rate will decrease, followed by compensatory blinks after speech has ended; participants’ pupils will dilate to a larger size than when truth telling.

Case Study

• Independent variables:

– AMC and VMC – Truth and Deception

• Dependent variables:

– Eye tracking

• Gaze, blinks, pupil dilation

– Head tracking – Hand tracking – Voice detection – Questionnaire data

• Mediating variables:

– The participants themselves!

Case Study

• Research approach:

– Mixed methods approach as both quantitative and qualitative data were collected and used to describe the experimental interactions.

Case Study

• Ethical considerations:

– We need to record participants on video, audio, and

• behaviour. If they say “no” to collection of behaviour

then we can’t perform the experiment. – The study explicitly investigated deception, so this raises ethical and moral considerations. Each participant may react differently. – Experimentation almost always operates under a facade of truth in order to control aspects of the procedure.

Case Study

• Experiment design:

– Between-subjects design with regards to mediation type (i.e. Participants performed only in AMC or only in VMC).

• Groups were balanced using a pre-experiment personality

questionnaire

– Within-subjects repeated measures design with regards to truth telling and deception (i.e. Participants told the truth in multiple stages and told lies in multiple stages).

• Orders of lying and truth telling were randomised

Case Study – Data collection

Case Study

• Results and Analysis

– Read the paper! http://www.cs.ucl.ac.uk/staff/w.steptoe/files/chi2010- steptoe.pdf – Data processing in preparation for statistical analysis – Different data types require different processing and statistical tests. – Analysis tests for significant differences between data relating to multiple independent variables. – Topic for another lecture / course of its own!

Summary

• VEs provide a powerful platform for

experimentation capable of investigating a diverse range of domains including human perception, behaviour and response, social interaction, HCI technology.

• Experimental design in VEs is complex and

influenced by many factors.

• Findings and approaches in the VE literature

slowly filter through to commercial and home application.

Creswell, 2003

Case Study

• Hypotheses for experiment 2 (E2):

– When assessing the veracity of E1’s participants performing AMC, judgments will be more accurate and confident when observing avatars featuring faithful reproduction of oculesic behavior than judgments of avatars with no oculesic expression, or audio-only

• stimuli. Similarly, higher ratings of engagement, and

confidence in this rating, will be elicited when observing avatars featuring oculesic behaviour