WEB-BASED VR EXPERIMENTS POWERED BY THE CROWD Xiao Ma [1,2] Megan - - PowerPoint PPT Presentation
WEB-BASED VR EXPERIMENTS POWERED BY THE CROWD Xiao Ma [1,2] Megan Cackett [2] Leslie Park [2] Eric Chien [1,2] Mor Naaman [1,2] The Web Conference 2018 [1] Social Technologies Lab, Cornell Tech [2] Cornell University xiao@jacobs.cornell.edu |
[1] Social Technologies Lab, Cornell Tech [2] Cornell University
xiao@jacobs.cornell.edu | maxiao.info
The Web Conference 2018 Mor Naaman[1,2] Megan Cackett[2] Leslie Park[2] Xiao Ma[1,2] Eric Chien[1,2]
VIRTUAL REALITY
3
The Big Bang Theory S9E20
A VISION
4
Crowdsourcing Virtual Reality (VR)
COMBINING THE BEST OF BOTH WORLDS
5
Virtual Reality (as of 2017) Crowdsourcing Crowdsourced VR (this work) Manipulation Realism Measurement Granularity Participant Diversity Reproducibility
COMBINING THE BEST OF BOTH WORLDS
6
Virtual Reality (as of 2017) Crowdsourcing Crowdsourced VR (this work) Manipulation Realism + High
Measurement Granularity + High
Participant Diversity
+ High Reproducibility
+ High
COMBINING THE BEST OF BOTH WORLDS
7
Virtual Reality (as of 2017) Crowdsourcing Crowdsourced VR (this work) Manipulation Realism + High
+ High Measurement Granularity + High
+ High Participant Diversity
+ High + High Reproducibility
+ High + High
HOW
8
RESEARCH QUESTIONS
9
RQ1: Are there VR-eligible reachable workers? RQ2: What is a good user flow? RQ3: What types of experiment manipulations can we deliver remotely? RQ4: What are the limitations and challenges?
CONTRIBUTIONS
10
experiment manipulation remotely
Source code and data are available at: bit.ly/VRCrowdExperiments
OUTLINE
11
experiment manipulation remotely
Source code and data are available at: bit.ly/VRCrowdExperiments
Please take a picture of your device with lasts 4 digits of your worker ID handwritten on a piece of paper in view.
CONSTRUCTING VR-READY PANEL
12
N = 242
(MORE) DIVERSE PARTICIPANTS
13
N = 242
(MORE) DIVERSE PARTICIPANTS
14
WHITE ASIAN BLACK OTHER
0% 25% 50% 75% 100%
10% 6% 14% 70% MALE FEMALE*
0% 25% 50% 75% 100%
39% 61% SUBURBAN URBAN RURAL
0% 25% 50% 75% 100%
18% 30% 52% U.S. OTHER
0% 25% 50% 75% 100%
10% 90% HIGH SCHOOL OR BELOW SOME / 2-YEAR COLLEGE BACHELOR'S MASTER'S AND ABOVE
10% 32.5%
13% 29% 29% 14% BELOW $30K $30K - $80K ABOVE $80K
0% 25% 50% 75%
22% 57% 21%
Age: 18 - 78 (Median 32)
* One worker self-identified as “other”.
OUTLINE
15
experiment manipulation remotely
Source code and data are available at: bit.ly/VRCrowdExperiments
DESIGN GOALS OF USER FLOW
16
Usable
Direct workers to participate in VR and complete survey on desktop
Web-Based
Allows for online data collection and thus remote participation
Low Technical Barrier
Allows for easier replication and adaptation
17
https://facebook.github.io/react-vr/ https://nodejs.org/en/
+
TECHNICAL PLATFORM CHOICE
Source code and data are available at: bit.ly/VRCrowdExperiments
FLOW
18
Worker accepts task through Amazon Mechanical Turk Worker completes experiment in VR
VR Web App URL
Worker answers survey in Qualtrics
Verification Code 1
Experimenter approves payment
Verification Code 2
OUTLINE
19
experiment manipulation remotely
Source code and data are available at: bit.ly/VRCrowdExperiments
MODELS OF ILLUSIONS IN VR
20
MODELS OF ILLUSIONS IN VR
21
Gonzalez-Franco, M., & Lanier, J. (2017). Model of Illusions and Virtual Reality. Frontiers in psychology, 8, 1125.
Place Illusion
A user’s feeling of being transported into the rendered environment
Embodiment Illusion
A user’s feeling of experiencing the virtual world through an avatar
Plausibility Illusion
A user’s feeling that events happening in the virtual world are real
MODELS OF ILLUSIONS IN VR
22
Place Illusion
A user’s feeling of being transported into the rendered environment
Embodiment Illusion
A user’s feeling of experiencing the virtual world through an avatar
Plausibility Illusion
A user’s feeling that events happening in the virtual world are real Study 1: Restorative Effects of Virtual Environments [Valtchanov et al. 2010], N = 22 (original) v.s. 55 (ours) Study 2: Proteus Effect [Yee and Bailenson, 2007], N = 50 (original) v.s. 59 (ours) Study 3: Drawing Power of Crowds [Milgram et al. 1969], N = 1,424 (original) v.s. 56 (ours)
Gonzalez-Franco, M., & Lanier, J. (2017). Model of Illusions and Virtual Reality. Frontiers in psychology, 8, 1125.
MODELS OF ILLUSIONS IN VR
23
Place Illusion
A user’s feeling of being transported into the rendered environment
Embodiment Illusion
A user’s feeling of experiencing the virtual world through an avatar
Plausibility Illusion
A user’s feeling that events happening in the virtual world are real
Gonzalez-Franco, M., & Lanier, J. (2017). Model of Illusions and Virtual Reality. Frontiers in psychology, 8, 1125.
Study 1: Restorative Effects of Virtual Environments [Valtchanov et al. 2010], N = 22 (original) v.s. 55 (ours) Study 2: Proteus Effect [Yee and Bailenson, 2007], N = 50 (original) v.s. 59 (ours) Study 3: Drawing Power of Crowds [Milgram et al. 1969], N = 1,424 (original) v.s. 56 (ours)
24
STUDY 2: PROTEUS EFFECT
25
Van Der Heide, Brandon, et al. "The Proteus effect in dyadic communication: Examining the effect of avatar appearance in computer-mediated dyadic interaction." Communication Research 40.6 (2013): 838-860.
26
STUDY 3: DRAWING POWER OF CROWDS
27
Milgram, S., Bickman, L., & Berkowitz, L. (1969). Note on the drawing power of crowds of different size. Journal
N = 1,424
STUDY 3: DRAWING POWER OF CROWDS
28
Milgram, S., Bickman, L., & Berkowitz, L. (1969). Note on the drawing power of crowds of different size. Journal
N = 1,424
29
RESULTS
30
Front Default Field of View (101°) Divided Into Four Zones Zone 1 Zone 2 Zone 3 Zone 4
FOUR CONDITIONS
31
Size of stimulus crowd Zero Low Medium High
Participant Male avatar Female avatar
GAZE* DISTRIBUTION
32
Size of stimulus crowd Zero (N = 15) Low (N = 15) Medium (N = 13) High (N = 13)
Participant Male avatar Female avatar
0% 100%
* Head rotation used as a proxy for gaze.
GAZE* DISTRIBUTION
33
Size of stimulus crowd
Participant Male avatar Female avatar
0% 100%
60% 8% 14% 18%
* Head rotation used as a proxy for gaze.
Zero (N = 15) Low (N = 15) Medium (N = 13) High (N = 13)
GAZE* DISTRIBUTION
34
Size of stimulus crowd
Participant Male avatar Female avatar
0% 100%
60% 8% 14% 18% 54% 12% 12% 22% 47% 13% 16% 24% 47% 14% 16% 23% *** statistically significant
* Head rotation used as a proxy for gaze.
Zero (N = 15) Low (N = 15) Medium (N = 13) High (N = 13)
RECAP
35
Place Illusion
A user’s feeling of being transported into the rendered environment
Embodiment Illusion
A user’s feeling of experiencing the virtual world through an avatar
Plausibility Illusion
A user’s feeling that events happening in the virtual world are real Study 1: Restorative Effects of Virtual Environments [Valtchanov et al. 2010], 22 (original) v.s. 55 (ours) Study 2: Proteus Effect [Yee and Bailenson, 2007], 50 (original) v.s. 59 (ours) Study 3: Drawing Power of Crowds [Milgram et al. 1969], 1,424 (original) v.s. 56 (ours)
Source code and data are available at: bit.ly/VRCrowdExperiments
DISCUSSION
36
COMBINING THE BEST OF BOTH WORLDS
37
Virtual Reality (as of 2017) Crowdsourcing Crowdsourced VR (this work) Manipulation Realism + High
+ High Measurement Granularity + High
+ High Participant Diversity
+ High + High Reproducibility
+ High + High
38
Place Illusion Embodiment Illusion Plausibility Illusion
Illusion Delivered Illusion Delivered Illusion Delivered Study 1: Restorative Effects of Virtual Environments [Valtchanov et al. 2010], N = 22 (original) v.s. 55 (ours) Study 2: Proteus Effect [Yee and Bailenson, 2007], N = 50 (original) v.s. 59 (ours) Study 3: Drawing Power of Crowds [Milgram et al. 1969], N = 1,424 (original) v.s. 56 (ours)
39
Place Illusion Embodiment Illusion Plausibility Illusion
Illusion Delivered Illusion Delivered Illusion Delivered Head Rotation Study 1: Restorative Effects of Virtual Environments [Valtchanov et al. 2010], N = 22 (original) v.s. 55 (ours) Study 2: Proteus Effect [Yee and Bailenson, 2007], N = 50 (original) v.s. 59 (ours) Study 3: Drawing Power of Crowds [Milgram et al. 1969], N = 1,424 (original) v.s. 56 (ours)
N = 242
40
WHITE ASIAN BLACK OTHER
0% 25% 50% 75% 100%
10% 6% 14% 70% MALE FEMALE*
0% 25% 50% 75% 100%
39% 61% SUBURBAN URBAN RURAL
0% 25% 50% 75% 100%
18% 30% 52% U.S. OTHER
0% 25% 50% 75% 100%
10% 90% HIGH SCHOOL OR BELOW SOME / 2-YEAR COLLEGE BACHELOR'S MASTER'S AND ABOVE
10% 32.5%
13% 29% 29% 14% BELOW $30K $30K - $80K ABOVE $80K
0% 25% 50% 75%
22% 57% 21%
Age: 18 - 78 (Median 32)
* One worker self-identified as “other”.
41
Place Illusion Embodiment Illusion Plausibility Illusion
Illusion Delivered Illusion Delivered Illusion Delivered Replicated Partially replicated Didn’t replicate Head Rotation Study 1: Restorative Effects of Virtual Environments [Valtchanov et al. 2010], N = 22 (original) v.s. 55 (ours) Study 2: Proteus Effect [Yee and Bailenson, 2007], N = 50 (original) v.s. 59 (ours) Study 3: Drawing Power of Crowds [Milgram et al. 1969], N = 1,424 (original) v.s. 56 (ours)
42
reduction, estimating $3,600 for the same in-lab studies)
Cheaper
(67% reduction, estimating 10 in-lab studies per day)
Faster
Easier
Source code and data are available at: bit.ly/VRCrowdExperiments
CONTRIBUTIONS
43
experiment manipulation remotely
Source code and data are available at: bit.ly/VRCrowdExperiments
THE (LONG-DUE) PROMISE OF VIRTUAL REALITY
44
Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., & Bailenson, J. N. (2002). Immersive virtual environment technology as a methodological tool for social psychology. Psychological Inquiry, 13(2), 103-124.
… replications, or at least near-perfect replications, become quite possible.
… allow for more representative sampling. Whole experiments can be carried out concurrently in multiple laboratories via networked collaboratories. ... provide a compelling sense of personal, social, and environmental presence for users, while allowing the investigator near-perfect control over the experimental environment and actions within it.
A VISION
45
Scaling Replicable VR Studies
Source code and data are available at: bit.ly/VRCrowdExperiments
Crowdsourcing Virtual Reality (VR)
THANK YOU
46
Mor Naaman[1,2]
[1] Social Technologies Lab, Cornell Tech [2] Cornell University
Megan Cackett[2] Leslie Park[2] Xiao Ma[1,2] Eric Chien[1,2]
We thank the crowdworkers who participated in our studies, and Oculus for providing the equipment used for developing the studies. We thank Dan Goldstein, Jake Hofman and Sid Suri for early feedback and direction.
Source code and data are available at: bit.ly/VRCrowdExperiments
This work is partially supported by Oath through the Connected Experiences Lab at Cornell Tech.