Conducting web-based experiments for numerical cognition research - - PowerPoint PPT Presentation

Conducting web-based experiments for numerical cognition research

Arnold Kochari

Institute for Logic, Language, and Computation, University of Amsterdam
 Donders Institute for Brain, Cognition, and Behaviour, Radboud University in Nijmegen

Web-based data collection for psychology

What is possible:

• surveys
• reaction times
• mouse tracking
• audio/video recording

Advantages:

• fast:

• simultaneous data collection

• no need for appointment management
• reaching diverse populations; large samples
• cheaper (not because of underpaying)
• easy sharing of the experiment scripts (no special software needed)

Issues:


• experiment programming and participant recruitment

• no control of the environment in which experiment is completed
• timing less accurate than in labs

Timings of RT experiments in web-browsers

Replications of classical effects in web-based experiments:

• Crump et al 2013 (stroop task, task switching, flanker task, simon task, attentional blink,

masked priming etc.)


• Zwaan and Pecher 2012 (mental simulation in language comprehension)

• Barnhoorn et al 2014 (stroop, masked priming, attentional blink)

not a big issue for within-participant designs between-participant effects can still be reliable, just with more participants

• More variability in RTs due to imprecise timings (different monitors, keyboards,

browsers)

• Clear lag in RTs measured by JavaScript (2-45 ms - Reimers & Stewart 2015; 25 ms - de

Leeuw & Motz 2016)
 However:


• it is random

• this lag is stable across conditions / within-systems

• we can compensate for that by having more participants

Technical questions

Programming experiments for web-browsers

Importantly: same scripts can be used on lab computers Free and open source:

• jsPsych - scripting manually
• lab.js - graphical interface + scripting manually
• PsychoPy/PsychoJS - graphical interface + scripting manually
• PsyToolkit - also takes care of data collection

Commercial:

• Gorilla - graphical interface

Data collection tools

Participant recruitment:

• Amazon MTurk (not available from every country)
• Prolific.ac
• Qualtrics
• thers

Hosting data collection:

• combined with the experiment programming platform (e.g. Gorilla.sc)
• special dedicated service (e.g. JATOS, psiTurk)
• personal or university web hosting space

Results with 2 classical paradigms

Size-congruity effect

close replication of Henik & Tzelgov 1982

Materials: 8 digit pairs
 Experimental factors:
 congruity (congruent vs. incongruent) 
 X numerical distance (2 vs 4)
 X font size distance (small vs. large)


• 64 unique exp. trials + neutral and empty trials

• trial: fixation cross for 150 ms followed by digit pairs for max. 1850 ms

• participants used P and Q as response keys

Experiment 1a: number comparison task; N participants = 23
 Experiment 1b: size comparison task; N participants = 24


• average time spent on task: 6-8 minutes

Size-congruity effect: numerical distance

Henik & Tzelgov, Exp 2: Current study:

Semantic comparison

* main effect of congruity
 * main effect of numerical distance

Experiment scripts, data and analysis code are available at: http://osf.io/dy8kf

* main effect of congruity
 * interaction of congruity and physical 
 size distance

Physical comparison

Size-congruity effect: physical size distance

Henik & Tzelgov, Exp 2: Current study:

Experiment scripts, data and analysis code are available at: http://osf.io/dy8kf

relevant dimension

Physical comparison

+ * main effect of physical size distance
 + * interaction of congruity and physical 
 size distance

Semantic comparison

+ * interaction of congruity and physical 
 size distance

to be ignored dimension

Distance and priming effects

replication of Van Opstal, Gevers, de Moor, & Verguts 2008

Materials: all digits from 1 to 9 except 5 included as primes and targets;
 64 unique combinations

500 ms 100 ms 83 ms 100 ms

• max. 2000 ms

Experimental factors: 
 distance of the target from the standard (1-4)
 distance of the prime from the standard (1-4)
 congruity


• 256 trials in total

• participants N = 72
• participants used P and Q as response keys


2 different possible mappings


• average time spent on task: 15 minutes

Distance and priming effects: 


distance of the target

• only trials with identical prime and target analysed
• 2 (size: before/after the standard) X 4 ( abs. comparison distance: 1, 2, 3, or 4) within-

subjects ANOVA on median correct RTs


• main effect of comparison distance: F (3, 213) = 10.6, p < 0.001.

Van Opstal et al (exp 1): Current study:

Experiment scripts, data and analysis code are available at: http://osf.io/dy8kf

• only trials with non-identical prime and target analysed
• 2 (size: before/after the standard) X 2 (congruency) within-subjects ANOVA on median

correct RTs


• main effect of congruency: F (1, 71) = 58.4, p < 0.001.

Van Opstal et al (Exp 1):

before standard: after standard: congruent: incongruent: congruent: incongruent: 405 427 399 429 =22 ms (median) =30 ms

Current study:

before standard: after standard: congruent: incongruent: congruent: incongruent: 528 552 538 560 =24 ms =22 ms

Distance and priming effects: 


congruity

Experiment scripts, data and analysis code are available at: http://osf.io/dy8kf

• only congruent trials with non-identical prime and target analysed
• 2 (size: before/after the standard) X 4 (abs. priming distance: 1, 2, 3, or 4) within-subjects

ANOVA on median correct RTs


• main effect of prime distance: F (3, 213) = 13.9, p < 0.001.

Van Opstal et al (exp 1):

Current study:

Distance and priming effects: 


distance of the prime

Experiment scripts, data and analysis code are available at: http://osf.io/dy8kf

NB: Primes were not actually displayed for exactly 83 ms!

Lessons learnt

Some tips

• Fair pay is important
• Not too many trials. I try to have experiments for max 15-20 minutes.
• Pre-register participant exclusion criteria: a lot of researcher degrees of freedom here.
• Ensuring participants put effort:

• I automatically exclude everyone who spent less than X s on reading the instructions

• 50% error rate

• a question at the end where they can given an honest answer
• Ensuring participants do not get distracted:
• tell them their data will be lost if they switch windows/tabs

• experiment has an automatic pace - no opportunity to decide to do something else

• inspect the duration of breaks (>3 minutes means they got distracted)
• Ensuring participant naiveté:

• put a cap on the number of previous studies

Experiment scripts, data and analysis code are available at: 
 http://osf.io/dy8kf

References

• Barnhoorn, J. S., Haasnoot, E., Bocanegra, B. R., & van Steenbergen, H. (2015). QRTEngine: An

easy solution for running online reaction time experiments using Qualtrics. Behavior research methods, 47(4), 918-929.

• Crump, M. J., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon's Mechanical Turk as

a tool for experimental behavioral research. PloS one, 8(3), e57410.

• de Leeuw, J. R., & Motz, B. A. (2016). Psychophysics in a Web browser? Comparing response

times collected with JavaScript and Psychophysics Toolbox in a visual search task. Behavior Research Methods, 48(1), 1-12.

• Henik, A., & Tzelgov, J. (1982). Is three greater than five: The relation between physical and

semantic size in comparison tasks. Memory & cognition, 10(4), 389-395.

• Reimers, S., & Stewart, N. (2015). Presentation and response timing accuracy in Adobe Flash and

HTML5/JavaScript Web experiments. Behavior research methods, 47(2), 309-327.

• Van Opstal, F

., Gevers, W., De Moor, W., & Verguts, T. (2008). Dissecting the symbolic distance effect: Comparison and priming effects in numerical and nonnumerical orders. Psychonomic Bulletin & Review, 15(2), 419-425.

• Zwaan, R. A., & Pecher, D. (2012). Revisiting mental simulation in language comprehension: Six

replication attempts. PloS one, 7(12), e51382.