SLIDE 1
Collaborative Signal Detection: Human-human and Human-computer - - PowerPoint PPT Presentation
Collaborative Signal Detection: Human-human and Human-computer - - PowerPoint PPT Presentation
Collaborative Signal Detection: Human-human and Human-computer teams Jason S. McCarley Ali Enright Megan Bartlett Signal d Detection Noise Signal p(evidence | N) or p(evidence | S+N) + Noise Theory Decision variable / Evidence d
SLIDE 2
SLIDE 3
Decision variable / Evidence p(evidence | N) or p(evidence | S+N) Noise Signal Noise +
Signal Detection Theory
d’
SLIDE 4
d’1 d’2 d’team Information pooling
(Bahrami et al., 2010; Sorkin & Dai, 1994)
The Uniform Weighting (UW) Model
SLIDE 5
Noise Signal + Noise
SLIDE 6
The ROC & the Z-ROC
z (False alarm rate) z (Hit rate)
0,0
Slope = σN/σS False alarm rate Hit rate
0.0 1.0 0.0 1.0
d’e
SLIDE 7
- Fig. 2. The effects of aggregation on zROC curves. The straight line labeled
“True” corresponds to a signal detection model with d0 = 2 and σ = 1. The curve labeled “Case I” results from aggregating over items with different d0
- values. The curve labeled “Case II” results from aggregating over items with
different criteria.
A problem A solution
Hierarchical ROC modeling
µGrand
µ1 = µGrand + α1 µ2 = µGrand + α2 …
(Morey et al., 2008) Fit ROC with Bayesian sampling procedure from vague priors. Produces a posterior distribution for model parameters.
SLIDE 8
d’1 d’2 d’team Information pooling
The Uniform Weighting (UW) Model
(Bahrami et al., 2010; Sorkin & Dai, 1994)
SLIDE 9
S1’s variance S2’s variance
Shared variance
SLIDE 10
d’1 d’2 d’team Information pooling r > 0
The Uniform Weighting (UW) Model
(Bahrami et al., 2010; Sorkin & Dai, 1994)
SLIDE 11
How efficiently do people collaborate in a naturalistic search task?
SLIDE 12
Time
Definitely Yes Probably Yes Guess Yes Guess No Probably No Definitely No
You found a threat! +
SLIDE 13
500 trials
250 trials (150-, 100+) Single Observer Worked alone Same order of trials to extract correlations 250 trials (150-, 100+) Team Worked together
SLIDE 14
Comparisons
- Single-person search
- Team search
- UW, ρ = 0
- Predicted from formula based on
individual searchers d’e scores
- Mock UW
- (RatingS1 + RatingS2) / 2
- Incorporates correlations between
judgments
SLIDE 15
Experiment 1
Room 1 Room 2
Individual search
Room 1 Room 2
Team search Free viewing
SLIDE 16
SLIDE 17
SLIDE 18
Room 1 Room 2
Individual search
Room 1 Room 2
Team search Free viewing
Experiment 2
SLIDE 19
SLIDE 20
Experiment 3
Room 1 Room 2
Individual search
Room 1 Room 2
Team search Free viewing
SLIDE 21
SLIDE 22
Experiment 4
Room 1 Room 2
Individual search
Room 1 Room 2
Team search Viewing time = 3 sec
SLIDE 23
SLIDE 24
Experiment 5
SLIDE 25
Experiment 5
Room 1 Room 2
Individual search
Room 1 Room 2
Team search Free viewing
SLIDE 26
SLIDE 27
d’1 d’2 d’team Information pooling r Teams outperform statistical expectations in a collaborative visual search. Collaboration increases team members’ d’?
SLIDE 28
d’1 d’2 d’team Information pooling r Teams outperform statistical expectations in a collaborative visual search. Collaboration increases team members’ d’? Collaboration de-correlates team members’ judgments?
SLIDE 29
How well does a person collaborate with a computerized aid?
Bartlett & McCarley (2017)