Experimentation in Virtual Environments Will Steptoe 29 th January 2010
What’s in this lecture? 1. Introduction to experimentation in VEs 2. Experimental Design 3. Case study
What is an Experiment? • A set of actions and observations, performed in the context of solving a particular problem or question, to support or falsify a hypothesis or research concerning phenomena. • The experiment is a cornerstone of the empirical approach to acquiring deeper knowledge about the physical world.
Why Experiment? • Causal explanation vs. Informal description • Parameters and parameter estimation • Counterfactual evidence to existing theory • Problem solving • Technology exploration, development, and refinement
Why Experiment in VEs? • Free from physical implications and limitations – Dangerous situations including training and military – Ethical considerations are different – “Transformed” social interaction allows one to break normal rules of physical interaction because users do not actually have to share the same “reality”. • Applications – Opportunities afforded by VEs – Ubiquity of HCI – Increasing transition to 3D displays and interfaces featuring natural input gesture in daily work and entertainment
Why Experiment in VEs?
Why Experiment in VEs?
Big picture of experimentation Identifying a research problem Reviewing the literature Identifying a purpose and stating questions Defining Research Methods Collecting data Analyzing and interpreting data Reporting and evaluating the study
Purpose Statement • “Why you want to do the study and what you intend to accomplish” Locke, Spirduso, and Silverman, 2000. • Most important aspect of an experiment and orients the reader to the central intent of the study
Review of the literature • What has been studied? • How was it done? • What implications arise? • Identify gaps • Relate to research method • http://scholar.google.co.uk • http://academic.research.microsoft.com • http://liinwww.ira.uka.de/bibliography/index.html
Research Questions • 3-7 questions that extend from purpose statement. • Contextualise research with existing literature. • Present scope of study (i.e. what you are and are not considering). • Will usually mention attributes of what you are studying (variables).
Experimental Hypotheses • Predictions of relationship among variables. • Must be ‘falsifiable’: – Must be possible to obtain evidence that could lead to its rejection • Carry out some procedure that may be used to test the hypothesis (i.e. gather ‘data’). • May be: – Rejected – Confirm – Neither
Variables • Characteristic or attribute of an individual or an organisation that can be measured or observed and that varies among the people or organisation being studied. – Creswell 2002 • Distinguished by temporal order and measurement. – Temporal order means that one variable precedes another in time, and because of this order, affects or “causes” another variable.
Independent Variables • The ones we manipulate or change. • Variables that (probably) cause, influence, or affect outcomes. • Also called treatment, manipulated, antecedent, or predictor variables.
Dependent Variables • The ones we measure. • Variables that depend on the independent variables. • Outcomes or results of the influence of the independent variables. • Also known as criterion, outcome, and effect variables.
Control Variables • The ones we keep the same. • Variables that if they were not controlled, would influence the dependent variable.
Intervening or Mediating Variables • The ones we must acknowledge. • Variables that “stand between” the independent and dependent variables. • Mediate the effects of the independent variables on the dependent variable.
Example of Variables • Pit experiment – Independent • Rendering fidelity, use of haptics. – Dependent • Heart rate, body movement. – Control • everything apart from the independent variables! Includes procedure prior to experiment. – Mediating • Previous experience of VR or games, age, sex, personality (i.e. “openness” to experience).
Purpose Statement Revisited • Design of purpose statement begins with identifying the proposed variables for a study, and specifying how the variables will be measured or observed.
Approaches to research • Qualitative • Quantitative • Mixed
Qualitative Approach • Non-numerical data collection and explanation. • Themes developed from open-ended emerging data For example, if you are asked to explain in qualitative terms a thermal image displayed in multiple colours, then you would explain the colour differences rather than the heat's numerical value. • Constructivist perspectives: multiple meanings of individual experiences that are socially and historically constructed, with intent to developing a theory or pattern.
Qualitative Data Collection Methods – Questionnaires • May be recorded on numerical scale (i.e. Likert scale). – Phenomenologies • “Essence” of human experience concerning a phenomenon, as described by participants (i.e. post-experimental interviews). – Ethnographies • Observing a cultural group generally over extended time. – Grounded theory studies • Derivation of general, abstract theory of a process or interaction grounded in the views of participants (i.e. interviews) – Narratives • Worded descriptions over extended time formed into narrative chronology.
Quantitative Approach • Numerical data collection and statistical analysis. • Measurement is central to quantitative research because it provides the fundamental connection between empirical observation and mathematical expression of quantitative relationships. • Postpositivist claims for developing knowledge: cause and effect, reduction to specific variables and hypotheses, use of measurement and observation, and the test of theories.
Quantitative Data Collection Methods • Devices able to gather numbers of anything measurable: – Body Tracking (head, hand, eye ... ) – Physiological response (heart rate, skin conductance, pupil dilation) – Performance data (time taken, errors, motion paths, decisions made)
Mixed methods • Collecting, analysing, and integrating quantitative and qualitative research and data. • The purpose of this form of research is that both qualitative and quantitative research, in combination, provide a better understanding of a research problem or issue than either research approach alone. • Mixing the data (merging of quantitative and qualitative data must be kept distinct but connected). • Raises issues: how, what, where, why, value (of mixing).
Mixed methods is particularly suited to VEs • VEs mediate users’ sensory input channels with digital stimuli. – Suited to quantitative data collection as the data is delivered in numeric form. • However, a key feature of VEs is that they are able to elicit natural behaviour due to high levels of immersion and presence. This means that we are studying natural human perception, behaviour, and interaction. – Suited to subjective data relating to an individual’s experience in a VE.
Selecting an approach • Match between problem and approach – Quantitative suited to test theory, identifying factors that influence an outcome, the use of a technology, or understanding best predictors of outcomes. – Qualitative often suited to studies that are exploratory in nature, in order to identify potential variables for future investigation. – Mixed methods can provide best of both, generalising findings (quantitative) and developing detailed view of the meaning of a phenomenon or concept for individuals (qualitative).
Ethical Considerations • Standard Principles in Ethics – British Psychological Society http://www.bps.org.uk/ • For ethical reasons some areas of human experience and behaviour may be beyond the reach of experiment, observation or other form of psychological investigation. • The investigation should be considered from the standpoint of all participants; foreseeable threats to their psychological well-being, health, values or dignity should be eliminated.
Ethical Considerations • Where possible inform participants about the purposes of the experiment. – When would it not be possible? • The investigator should inform the participants of all aspects of the research or intervention that might reasonably be expected to influence willingness to participate. • Consent, deception, withdrawal, payment, debrief, confidentiality, physical protection.
Experimental Design • A within-groups design is an experiment in which the participants serve in more than one treatment. – Advantages include reduction in error variance associated with individual differences, fewer subjects required, thereby increasing statistical power. – Weaknesses include “carryover effects” arising from subjects performing multiple experimental sessions, thereby introducing effects of learning and fatigue. This can be minimised by randomising orders of treatment over subjects.
Recommend
More recommend
