Collaborative Signal Detection: Human-human and Human-computer - PowerPoint PPT Presentation

Collaborative Signal Detection: Human-human and Human-computer teams Jason S. McCarley Ali Enright Megan Bartlett

Signal d’ Detection Noise Signal p(evidence | N) or p(evidence | S+N) + Noise Theory Decision variable / Evidence

d’ 1 Information d’ team pooling d’ 2 The Uniform Weighting (UW) Model (Bahrami et al., 2010; Sorkin & Dai, 1994)

Noise Signal + Noise

The ROC & the Z-ROC 1.0 Slope = σ N / σ S d’ e z (Hit rate) Hit rate 0,0 0.0 0.0 1.0 False alarm rate z (False alarm rate)

A problem A solution Hierarchical ROC modeling µ Grand µ 1 = µ Grand + α 1 µ 2 = µ Grand + α 2 … Fit ROC with Bayesian sampling procedure from vague priors. Produces a posterior distribution for model parameters. Fig. 2. The effects of aggregation on z ROC curves. The straight line labeled “True” corresponds to a signal detection model with d 0 = 2 and σ = 1. The curve labeled “Case I” results from aggregating over items with different d 0 values. The curve labeled “Case II” results from aggregating over items with different criteria. (Morey et al., 2008)

d’ 1 Information d’ team pooling d’ 2 The Uniform Weighting (UW) Model (Bahrami et al., 2010; Sorkin & Dai, 1994)

S1’s S2’s variance variance Shared variance

d’ 1 Information d’ team pooling r > 0 d’ 2 The Uniform Weighting (UW) Model (Bahrami et al., 2010; Sorkin & Dai, 1994)

How efficiently do people collaborate in a naturalistic search task?

+ Time Definitely Yes Probably Yes Guess Yes Guess No Probably No Definitely No You found a threat!

500 trials 250 trials (150 - , 100 + ) 250 trials (150 - , 100 + ) Single Observer Team Worked alone Worked together Same order of trials to extract correlations

Comparisons • Single-person search • Team search • UW, ρ = 0 - Predicted from formula based on individual searchers d’ e scores • Mock UW - (Rating S1 + Rating S2 ) / 2 - Incorporates correlations between judgments

Experiment 1 Individual search Team search Room 1 Room 2 Room 1 Room 2 Free viewing

Experiment 2 Individual search Team search Room 1 Room 2 Room 1 Room 2 Free viewing

Experiment 3 Individual search Team search Room 1 Room 2 Room 1 Room 2 Free viewing

Experiment 4 Individual search Team search Room 1 Room 2 Room 1 Room 2 Viewing time = 3 sec

Experiment 5

Experiment 5 Individual search Team search Room 1 Room 2 Room 1 Room 2 Free viewing

Teams outperform statistical expectations in a collaborative visual search. Collaboration increases team d’ 1 members’ d’ ? Information d’ team pooling r d’ 2

Teams outperform statistical expectations in a collaborative visual search. Collaboration increases team d’ 1 members’ d’ ? Information d’ team Collaboration pooling r de-correlates team members’ judgments? d’ 2

How well does a person collaborate with a computerized aid? Bartlett & McCarley (2017)