Planning to Control Crowd-Sourced Workflows Daniel S. Weld - - PDF document
3/4/20 Planning to Control Crowd-Sourced Workflows Daniel S. Weld University of Washington 1 30,000 View Crowdsourcing is huge & growing rapidly Virtual organizations Flash teams with mixed human & machine members
3/4/20 1
1
– Virtual organizations – Flash teams with mixed human & machine members
– Reduce labor required by 30-85%
2
2
3/4/20 2
3
3
Universal reference for anything Answers to 7.1 M prog. questions 190 M reviews of 4.4 M businesses
4
4
3/4/20 3
Game to find 3D structure of proteins. Solved 15 year outstanding AIDS puzzle 800,000 volunteers – Hubble images Discovered “Hanny’s Voorwerp” black-hole “Pea galaxies” Crowdsourced bird count & identification Migration shift -> effect of climate change
5
5
Will Grow to $5B by 2018 [Staffing Industry Analysts]
6
Charts from Panos Ipierotis’ blog; phone from pixabay
. Nationality and gender of
60% Growth Hours / week Nationality
6
3/4/20 4
Write a descriptive caption for this picture, then submit.
7
Submit $0.05
Figure from [Little et al. 2010]
A parial view of a pocket calculator together with some coins and a pen.
7
8
8
3/4/20 5
9
[Little et al, 2010]
9
10
[Little et al, 2010]
10
3/4/20 6
11
[Little et al, 2010]
11
[Little et al, 2010]
12
12
3/4/20 7
A CASIO multi-function, solar powered scientific calculator. A blue ball point pen with a blue rubber grip and the tip extended. Six British coins; two of £1 value, three of 20p value and one of 1p value. Seems to be a theme illustration for a brochure or document cover treating finance - probably personal finance.
13
A parial view of a pocket calculator together with some coins and a pen.
[Little et al, 2010]
Figure from [Little et al. 2010]
13
[Little et al, 2010]
14
14
3/4/20 8
[Little et al, 2010]
15
15
16
16
3/4/20 9
18
Agent Sensors ? Actuators Environment
Percepts Actions
19
POMDP
19
3/4/20 10
20
Figure from Dan Klein & Pieter Abbeel - UC Berkeley CS188: http://ai.berkeley.edu.] And Sarah Reeves (http://dear-theophilus.deviantart.com/)
World State s = <x, y> Actions P(s’ | s, a) Cost c Observe: Next State s’ = <x’, y’> Reward = f(s, a, s’) Input: Output: While learning action & reward probabilities (Reinforcement learning) Construct policy, π : SàA, that chooses best action for each state I.e., actions that maximize expected reward – costs over time
20
21
Belief State P(s) Actions P(s’ | s, a) Cost c Observe: Noisy Sensor = f(s’) Input: Output: While learning action & reward probabilities (Reinforcement learning) Construct policy, π : SàA, that chooses best action for each state I.e., actions that maximize expected reward – costs over time Reward
Figure from Dan Klein & Pieter Abbeel - UC Berkeley CS188: http://ai.berkeley.edu.]
21
3/4/20 11
22
22
23
Robot figure from Dan Klein & Pieter Abbeel - UC Berkeley CS188: http://ai.berkeley.edu.]
23
3/4/20 12
Qualitya1 Quality a2 Worker creates (hopefully) improved artifact
27
P P
27
28
Qualitya1 Quality a2 P P
28
3/4/20 13
29
Qualitya1 Quality a2 P P Qualitya1 Quality a2 P P
Worker votes that artifact 1 is better
29
35
need improving ? make ballot job update quality estimates submit Y N Y N better of α and α’ initial artifact (α) estimate quality
α’
more voting ?
α α’ α’
generate improve job
35
3/4/20 14
36
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α’
better of α and α’
α
36
37
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α’
better of α and α’
α
37
3/4/20 15
38
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α'
better of α and α’
α
38
39
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α'
better of α and α’
α
39
3/4/20 16
40
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α'
better of α and α’
α
40
41
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α'
better of α and α’
α
41
3/4/20 17
42
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α'
better of α and α’
α
42
43
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α'
better of α and α’
α
43
3/4/20 18
44
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α'
better of α and α’
α
44
45
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α'
better of α and α’
α
45
3/4/20 19
46
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α'
better of α and α’
α
46
47
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α'
better of α and α’
α
47
3/4/20 20
48
need improving ? generate improve job make ballot job update quality estimates submit Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α'
better of α and α’
α
48
49
need improving ? generate improve job make ballot job update quality estimates submit α Y N Y N initial artifact ( ) estimate quality
α’
more voting ?
α α’ α'
better of α and α’
α
49
3/4/20 21
0.5 0.55 0.6 0.65 0.7 0.75 0.8
POMDP Hand Coded Quality
[Dai, Mausam & W, AAAI-11] [Dai et al. AIJ 2013]
50
50
POMDP (TurKontrol) Hand Coded
51
51
3/4/20 22
POMDP Hand Coded
52
52
117
117