Conducting web-based experiments for numerical cognition research - PowerPoint PPT Presentation

Conducting web-based experiments for numerical cognition research Arnold Kochari Institute for Logic, Language, and Computation, University of Amsterdam 
 Donders Institute for Brain, Cognition, and Behaviour, Radboud University in Nijmegen

Web-based data collection for psychology What is possible: - surveys - reaction times - mouse tracking - audio/video recording Advantages: - fast: 
 - simultaneous data collection 
 - no need for appointment management - reaching diverse populations; large samples - cheaper (not because of underpaying) - easy sharing of the experiment scripts (no special software needed) Issues: 
 - experiment programming and participant recruitment 
 - no control of the environment in which experiment is completed - timing less accurate than in labs

Timings of RT experiments in web-browsers - More variability in RTs due to imprecise timings (different monitors, keyboards, browsers) - Clear lag in RTs measured by JavaScript (2-45 ms - Reimers & Stewart 2015; 25 ms - de Leeuw & Motz 2016) 
 However: 
 - it is random 
 - this lag is stable across conditions / within-systems 
 - we can compensate for that by having more participants not a big issue for within-participant designs between-participant effects can still be reliable, just with more participants Replications of classical effects in web-based experiments: - Crump et al 2013 (stroop task, task switching, flanker task, simon task, attentional blink, masked priming etc.) 
 - Zwaan and Pecher 2012 (mental simulation in language comprehension) 
 - Barnhoorn et al 2014 (stroop, masked priming, attentional blink)

Technical questions

Programming experiments for web-browsers Free and open source: • jsPsych - scripting manually • lab.js - graphical interface + scripting manually • PsychoPy/PsychoJS - graphical interface + scripting manually • PsyToolkit - also takes care of data collection Commercial: • Gorilla - graphical interface Importantly: same scripts can be used on lab computers

Data collection tools Hosting data collection: • combined with the experiment programming platform (e.g. Gorilla.sc) • special dedicated service (e.g. JATOS, psiTurk) • personal or university web hosting space Participant recruitment: • Amazon MTurk (not available from every country) • Prolific.ac • Qualtrics • others

Results with 2 classical paradigms


 
 
 Size-congruity effect close replication of Henik & Tzelgov 1982 Materials: 8 digit pairs 
 Experimental factors: 
 congruity (congruent vs. incongruent) 
 X numerical distance (2 vs 4) 
 X font size distance (small vs. large) 
 - 64 unique exp. trials + neutral and empty trials 
 - trial: fixation cross for 150 ms followed by digit pairs for max. 1850 ms 
 - participants used P and Q as response keys Experiment 1a: number comparison task ; N participants = 23 
 Experiment 1b: size comparison task ; N participants = 24 
 - average time spent on task: 6-8 minutes

Size-congruity effect: numerical distance Current study: Henik & Tzelgov, Exp 2: Semantic comparison * main e ff ect of congruity 
 * main e ff ect of numerical distance Physical comparison * main e ff ect of congruity 
 * interaction of congruity and physical 
 size distance Experiment scripts, data and analysis code are available at: http://osf.io/dy8kf

Size-congruity effect: physical size distance Henik & Tzelgov, Exp 2: Current study: Physical comparison + * main e ff ect of physical size distance 
 + * interaction of congruity and physical 
 relevant dimension size distance Semantic comparison to be ignored dimension + * interaction of congruity and physical 
 size distance Experiment scripts, data and analysis code are available at: http://osf.io/dy8kf


 Distance and priming effects replication of Van Opstal, Gevers, de Moor, & Verguts 2008 500 ms 100 ms 83 ms 100 ms max. 2000 ms Materials: all digits from 1 to 9 except 5 included as primes and targets; 
 64 unique combinations Experimental factors: 
 distance of the target from the standard (1-4) 
 distance of the prime from the standard (1-4) 
 congruity 
 - 256 trials in total 
 - participants N = 72 - participants used P and Q as response keys 
 2 di ff erent possible mappings 
 - average time spent on task: 15 minutes

Distance and priming effects: 
 distance of the target Van Opstal et al (exp 1): Current study: - only trials with identical prime and target analysed - 2 (size: before/after the standard) X 4 ( abs. comparison distance: 1, 2, 3, or 4) within- subjects ANOVA on median correct RTs 
 - main e ff ect of comparison distance: F (3, 213) = 10.6, p < 0.001. Experiment scripts, data and analysis code are available at: http://osf.io/dy8kf

Distance and priming effects: 
 congruity before standard: Van Opstal et al (Exp 1): 405 congruent: =22 ms (median) 427 incongruent: after standard: congruent: 399 =30 ms incongruent: 429 Current study: before standard: 528 congruent: =24 ms 552 incongruent: after standard: congruent: 538 =22 ms incongruent: 560 - only trials with non-identical prime and target analysed - 2 (size: before/after the standard) X 2 (congruency) within-subjects ANOVA on median correct RTs 
 - main e ff ect of congruency: F (1, 71) = 58.4, p < 0.001. Experiment scripts, data and analysis code are available at: http://osf.io/dy8kf

Distance and priming effects: 
 distance of the prime Current study: Van Opstal et al (exp 1): - only congruent trials with non-identical prime and target analysed - 2 (size: before/after the standard) X 4 (abs. priming distance: 1, 2, 3, or 4) within-subjects ANOVA on median correct RTs 
 - main e ff ect of prime distance: F (3, 213) = 13.9, p < 0.001. NB: Primes were not actually displayed for exactly 83 ms! Experiment scripts, data and analysis code are available at: http://osf.io/dy8kf

Lessons learnt

Some tips - Fair pay is important - Not too many trials. I try to have experiments for max 15-20 minutes. - Pre-register participant exclusion criteria: a lot of researcher degrees of freedom here. - Ensuring participants put e ff ort: 
 - I automatically exclude everyone who spent less than X s on reading the instructions 
 - 50% error rate 
 - a question at the end where they can given an honest answer - Ensuring participants do not get distracted: - tell them their data will be lost if they switch windows/tabs 
 - experiment has an automatic pace - no opportunity to decide to do something else 
 - inspect the duration of breaks (>3 minutes means they got distracted) - Ensuring participant naiveté: 
 - put a cap on the number of previous studies

Experiment scripts, data and analysis code are available at: 
 http://osf.io/dy8kf

References • Barnhoorn, J. S., Haasnoot, E., Bocanegra, B. R., & van Steenbergen, H. (2015). QRTEngine: An easy solution for running online reaction time experiments using Qualtrics. Behavior research methods, 47(4), 918-929. • Crump, M. J., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PloS one, 8(3), e57410. • de Leeuw, J. R., & Motz, B. A. (2016). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. Behavior Research Methods, 48(1), 1-12. • Henik, A., & Tzelgov, J. (1982). Is three greater than five: The relation between physical and semantic size in comparison tasks. Memory & cognition, 10(4), 389-395. • Reimers, S., & Stewart, N. (2015). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behavior research methods, 47(2), 309-327. • Van Opstal, F ., Gevers, W., De Moor, W., & Verguts, T. (2008). Dissecting the symbolic distance e ff ect: Comparison and priming e ff ects in numerical and nonnumerical orders. Psychonomic Bulletin & Review, 15(2), 419-425. • Zwaan, R. A., & Pecher, D. (2012). Revisiting mental simulation in language comprehension: Six replication attempts. PloS one, 7(12), e51382.