Measuring Workers Pre-task Interactions by Jason T. Jacques - - PowerPoint PPT Presentation
Measuring Workers Pre-task Interactions by Jason T. Jacques supervised by Per Ola Kristensson jtj21@cam.ac.uk people.ds.cam.ac.uk/jtj21 Spotting Advertising Color Outline 100 US-based participants 200 US-based participants 10 per study
jtj21@cam.ac.uk people.ds.cam.ac.uk/jtj21
by Jason T. Jacques supervised by Per Ola Kristensson
100 US-based participants 10¢ per study Q4 2012
200 US-based participants 12¢ per study Q3 2014
06:00 UTC
http://aa.usno.navy.mil/data/docs/earthview.php
18:00 UTC
http://aa.usno.navy.mil/data/docs/earthview.php
† Estimated based on a linear trend of 100,000 crowd workers in March 2007 (http://goo.gl/opQH10);
400,000 in September 2010 (http://goo.gl/G1g9AC); and 500,000 in January 2011 (http://goo.gl/vno0GW).
$ turkmill Usage: turkmill [-updategeoip] [-novisitors] [-postback] webpath logfile ... $ turkmill web/path/to/hit /var/log/apache2/access_log /var/log/apache2/access_log2
http://tr.im/turkmill
Worker Preview Complete Conversion Rate 1 09:48 09:48 1÷1 = 1.00 2 10:12 10:13 2÷2 = 1.00 3 10:17 2÷3 = 0.66 4 11:31 11:31 3÷4 = 0.75 5 11:06 11:07 4÷5 = 0.80 6 12:22 12:23 5÷6 = 0.83
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Proportion Proportion of Time Elapsed
Color Outline
0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Proportion Proportion of Time Elapsed
Conversion Rate Previews Completions
Conversion rate graph for Color
A B C 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 48 96 144 192 240 288 336 384 432 480 528 576 Proportion Hours Passed
Conversion Rate Previews Completions
Demographic study
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Proportion Proportion of Time Elapsed
Color Outline
An Assessment of Intrinsic and Extrinsic Motivation
Jakob Rogstadiusa, Vassilis Kostakosa, Aniket Kitturb, Boris Smusa, Jim Laredoc, Maja Vukovicc
a Madeira Interactive Technologies InstituteUniversity of Madeira 9000390 Funchal, Portugal {jakob,vk}@m-iti.org
b Carnegie Mellon University5000 Forbes Avenue Pittsburgh, PA 15213, USA nkittur@cs.cmu.edu
c IBM T.J. Watson Research CenterHawthorne NY 10532, USA {laredoj,maja}@us.ibm.com
Abstract
Crowdsourced labor markets represent a powerful new pa- radigm for accomplishing work. Understanding the motivat- ing factors that lead to high quality work could have signifi- cant benefits. However, researchers have so far found that motivating factors such as increased monetary reward gen- erally increase workers’ willingness to accept a task or the speed at which a task is completed, but do not improve the quality of the work. We hypothesize that factors that in- crease the intrinsic motivation of a task – such as framing a task as helping others – may succeed in improving output quality where extrinsic motivators such as increased pay do
pothesis along with a novel experimental design that enables controlled experimentation with intrinsic and extrinsic mo- tivators in Amazon’s Mechanical Turk, a popular crowd- sourcing task market. Results suggest that intrinsic motiva- tion can indeed improve the quality of workers’ output, con- firming our hypothesis. Furthermore, we find a synergistic interaction between intrinsic and extrinsic motivators that runs contrary to previous literature suggesting “crowding
retical implications for crowd work.
tion, audio transcribing and various types of surveys. In return, the people who carry out the work are paid money for each completed task, often in small amounts: tagging an image, for example, may pay a few cents. Crowdsourcing work involves a number of challenges different from those faced in traditional work settings. Crowd workers in general purpose markets like MTurk may have highly varying expertise, skills, and motivations. Employers (“requesters” in MTurk) have very little visibil- ity into these characteristics, especially compared to a traditional organization in which workers are vetted during recruitment, have work histories, have reputations within and outside the organization, and may go through organi- zational socialization methods such as training to ensure they can appropriately satisfy their job requirements. Fur- thermore, workers can easily return work for a given job with no repercussions or even create an entirely new pro- file with a clear reputation. These challenges mean that employers have more limited means of eliciting high quali- ty output than in traditional organizations. This study experimentally assesses the interaction of ex- trinsic and intrinsic motivators in crowdsourcing markets using a novel experimental methodology that controls for
Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media Blood cell COUNT these, including partially visible cells Malaria parasite in ring-form with double chromatin dots. COUNT these. Malaria parasite in other growth stage. IGNORE these.
Figure 1. Instructions given to participants on how to complete the experimental task. Figure 2. A sample image of medium complexity from the experi- mental task.
Global Health Council (high intrinsic motivation)
No sponsor (medium intrinsic motivation)
Rimek International (low intrinsic motivation)
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Conversion Rate Proportion of Time Elapsed
None (Medium) Rimek (Low) GHC (High)
jtj21@cam.ac.uk people.ds.cam.ac.uk/jtj21
by Jason T. Jacques supervised by Per Ola Kristensson
http://tr.im/turkmill