Motivation of Crowd Workers Does it matter? Babak Naderi Quality - - PowerPoint PPT Presentation
Motivation of Crowd Workers Does it matter? Babak Naderi Quality and Usability Lab, Telekom Innovation Laboratories Technische Universitt Berlin 1 Agenda Theoretical Model Measurement of Motivation Influence of Task Design Trapping
Quality and Usability Lab, Telekom Innovation Laboratories Technische Universität Berlin
1
2
3
– I am enjoying doing tasks in MTurk very much. [Intrinsic] – Because this job is a part of my life. [Identified] – Because I want people (requesters or others) to like me. [Introjected] – Because it allows me to earn money. [External] – I don’t know why, we are provided with unrealistic working conditions. [Amotivation]
4
5
No high school
PhD+ 1.40% 3.90%
Retired
4.20% Yes 5.30% Age 18-26 27-32 33-40 41-55 Gender Male 43% Female 57% 56+ 12.70% 32% 28.20% 20.01% 7% Employment Status $100,000 + 15.50% Education Level Some college Bachelor's degree
Master's degree
34.50% 32.70% 10.90% High school/GED 16.50% 12.30% 8.10% 6% Household Income $10,000 - $19,999 $20,000 - $39,999 $40,000 - $59,999 $60,000 - $79,999 No 94.70% Mturk Master 16.20% HIT Approval Rate (%) [0,85] (85,90] (90,95] (95,98] 12.30% 19% 21.50% Perceived Weekly Working on Mturk <4 Hours 10-15 H 15-25 H
Going school
5.60% 24.30% (98,100] 22.90% 25 H+ 21.80% 11.30% 20.40%
$80,000 - $99,999
4-10 Hours 30.30%
< $10,000
7.40% 29.20% 21.50% Other 10.20%
Unable to work
7.40% Keeping house 10.90% Working part time 16.50% Working full time 45.10%
6
Adjusted χ² by the degrees of freedom (df) (
𝜓² 𝑒𝑔) ≤ 3
Root mean square error of approximation (RMSEA) ≤ .08 Comparative fit index (CFI) ≥ .95 Item reliability (IR) ≥ .4
7
8
9
Groups (based on Approval Rate) Study1 Range [%] [0,70] (70,80] (80,90] (90,95] (95,100] # workers 11 2 29 110 70 Study2 Range [%] [0,85] (85,90] (90,95] (95,98] (98,100] # workers 58 80 92 93 78
10
𝑘 𝑂 𝑘=1
𝑂 𝑗=1
Where:
N # of duplicated items Ii a reliability check Item Ii´ duplicate of item Ii M # of participants mi # of participants with deviated answer to items i T Cut-off threshold
11
Inconsistency Scores of Study1 (Mdn = 1) were significantly different from the Inconsistency Scores of Study2 (Mdn =.65), 𝑞 < .001, 𝑠 = −.31.
12
13
Herzberg’s two factory theory of job satisfaction.
14
T0 No-trapping T1 Motivating msg
This is an interruption. We - the team of Crowdee - like to ensure that people work conscientiously and attentively on our tasks. Please select the answer ‘< x>’ to confirm your attention now.
15
T2 Easy to cheat T3 Hard to cheat
16
Performing Qualification Job (Q1)
New Crowd Workers
Job T1 Motivation Message Job T2 Low Effort
Entrance Level Study Level
Job T0 No Trapping
Training Job (T1)
Job T3 High Effort
Group Assigned
Participants: 179 workers 87 female and 92 male Mage= 27.9 y., SDage=8.1 y
17
18
*
Group N of CIs lower N of CIs higher N of CIs overlapping RMSD
Trapping T0 – No Trapping 17 6 27 0.426 Trapping T1 –Motivating message 13 2 35 0.375 Trapping T2 – Easy to cheat 17 3 30 0.411 Trapping T3 – Hard to Cheat 16 4 30 0.390
19
20
http://crowdee.de
21
http://crowdee.de
22
23
24