Expectancy bias and Bias and forensic evidence Bias and speech - - PDF document

expectancy bias and
SMART_READER_LITE
LIVE PREVIEW

Expectancy bias and Bias and forensic evidence Bias and speech - - PDF document

Overview Bias effects Expectancy bias and Bias and forensic evidence Bias and speech research forensic speech research Blind forensic speech research Maartje Schreuder TMFI / Maastricht University I AFPA 2011 Bundeskrim


slide-1
SLIDE 1

Expectancy bias and forensic speech research

Maartje Schreuder TMFI / Maastricht University I AFPA 2011

Bundeskrim inalam t Vienna, Austria July 27, 2011

Overview

  • Bias effects
  • Bias and forensic evidence
  • Bias and speech research
  • Blind forensic speech research

Bias effects

  • Bias
  • Expectancy
  • Confirmation bias (“tunnel vision”)

‘The most obvious danger in forensic science is that an examiner's observations and conclusions will be influenced by extraneous, potentially biasing information’ (Risinger et al., 2002)

Merckelbach, Crombach & Van Koppen, 2003

  • Madrid bombings 11 maart 2004
  • Finger prints on bag with detonating devices
  • Several FBI- and other finger print experts:

– 100% match Brandon Mayfield, sollicitor in USA, converted to Muslim – Mayfield arrested

Bias effects, expectancy effects

Bias effects, expectancy effects

Bias-effects, expectancy effects

  • Spanish police: match with Algerian

Ouhnane Daoud

  • Mayfield released
slide-2
SLIDE 2

Bias-effects, expectancy effects

Experiment Dror and colleagues (2006a):

  • 5 finger print experts (USA, UK, Israel, Netherlands,

Australia)

  • 2 finger prints from case they had done before: match
  • Information: “these are the finger prints that were falsely

matched by the FBI as belonging to the Madrid bomber”

– non-match – Instruction: ignore all context information

Bias-effects, expectancy effects

  • Experts are susceptible to irrelevant and

misleading information!

Results:

DNA and interpretation

Electropherograms in an easy-to-interpret case. Thompson, 2009

DNA and interpretation

  • Problematic DNA cases: partial or mixed DNA

profiles

  • Thompson, as part of lecture for DNA experts

– Profile of evidentiary material – Profile of suspect Tom – Not all peaks of profile evidentiary material in material

  • f suspect, nor victim
  • Wrong suspect?

– No 12 on D3, no OL on FGA

  • Experts: “Obvious that these peaks are artefacts, can be

ignored”

Thompson, 2009

DNA and interpretation

Electropherogram of a saliva sample and four suspect profiles.

DNA and interpretation

– 2nd lecture, same case: profile of Dick labeled as suspect. “I’m not sure”.

– Is 12 peak on locus D3 true allele or a drop-in artefact? – No 20 peak at locus FGA.

  • Experts: “How can you doubt that? Morphology,

peak height disparity, stochastic effects, …”

  • Thompson: “suspect was actually Tom”
  • Experts: “Oops”

Thompson, 2009

slide-3
SLIDE 3

DNA and interpretation

Electropherogram of a saliva sample and four suspect profiles.

DNA and interpretation

– 3rd lecture: profile of Harry labeled as suspect. “Problematic: D3 14, 17”.

  • Experts: “no problem at all. Allelic dropout, masking

by an artefact, …”

Thompson, 2009

DNA and interpretation

Electropherogram of a saliva sample and four suspect profiles.

DNA and interpretation

– 4th lecture: profile Sally labeled as suspect. “Do you agree that this defendant should be excluded?”

  • Experts: “no, evidentiary profile could be a mixed

profile”

  • (mix-theory was not mentioned before, when

‘suspect’ had the other profiles.)

Thompson, 2009

DNA and interpretation

Electropherogram of a saliva sample and four suspect profiles.

DNA and interpretation

  • Standards may shift to encompass the

profile of the suspects!

slide-4
SLIDE 4

Bias-effects, expectancy effects

  • Unintentional, subconscious process
  • Psychological phenomenon: fooled by our

brains!

  • Almost impossible to disregard background

information

  • Science: (double) blind to control for observer

effects and expectancy/confirmation bias

  • Forensic labs: regulations

– against contamination of samples – for reporting and conclusions, etc. – hardly any precautions against bias!

Bias-effects, expectancy effects

  • And we’re not even talking about speech

research yet!

– Inherently variable!

Expectancy Bias and Speech intelligibility

Experiment:

  • Noisy speech fragment (music)
  • Target sentence follows given cue sentence
  • Participant writes orthographic transcription
  • 2 conditions:

– Introduction from crime reporter (television) – No introduction

Expectancy Bias and Speech intelligibility

Cue sentence: “En toen zei ze: ik heb er geen zin meer in, ik wil naar huis”

[And then she said: “I don’t feel like it anymore, I want to go home”]

Write the sentence down that follows the cue sentence.

Expectancy Bias and Speech intelligibility

Target sentence: “En eeh toen heb ik haar gebracht”

[And eh then I brought her]

Some phonological similarity:

gebracht [brought] verkracht [raped]

  • gepakt

[grabbed]

Participants

  • 87 participants

– 20 excluded: understood ‘other’ word (wrong + not crime-related)

  • Of which 14 in Control condition
  • not ‘aided’ by context
  • 67 participants remained

– 38 Context condition – 29 Control condition

slide-5
SLIDE 5

Results

  • Understood by the 67 (remaining) participants:

Context condition

17 (45%) 21 (55%)

Correct word Crime-related word Thanks to my students of research practicum 2011 Non-context condition

3 (10%) 26 (90%)

2(2) = 9.291, (p = 0.002)

Results

Transcriptions (totals):

  • Correct transcription

– gebracht [brought] 49

  • Crime-related transcriptions

– Verkracht [raped] 11 – Gepakt [grabbed] 7 – Vastgepakt [grabbed] 1 – Geraakt [hit] 1

  • Other transcriptions

– Gepakt* [grasped, hugged] 6 – Hard [hard] 2 – Ontmoet [met] 2 – Bekijk het maar [whatever] 1 – … …

Context, transcripts, degradation

Lange, Tomas, Dana, & Dawes, 2011:

  • 2 experiments with degraded recordings

– Contextual information: criminal justice system – Dubious transcripts along with ‘evidentiary’ recordings

  • Results:

– systematic misinterpretations – Confidence in expectation-induced misinterpretations – Information leads to miscalibration to quality of recording: poor quality goes undetected

Lange, Tomas, Dana, & Dawes, 2011

Experiment with ‘wire-tapping interpreters’

1

06- 56991078 Mart in

2

06- 56991078 Mart in

3

06- 56991078 Mart in

4

06- 56991078 Mart in

5

06-3635406 Robe rt

6

06-3635406 Robe rt

7

06- 56991078 Mart in

Police interview

Robert Martin

1

06- 56991078 Mart in

2

06- 56991078 Mart in

3

06- 56991078 Mart in

4

06- 56991078 Mart in

5

06-13635406 Robe rt

6

06-3635406 Robe rt

7

06- 56991078 Mart in

Police interviews

False positives with Martin & False negative with Robert False negatives with Martin & False positive with Robert Martin Robert

slide-6
SLIDE 6

Speaker identification (auditory)

  • Stadia of analysis
  • 1. ‘Blind’ analysis
  • 2. Analysis of the questioned material
  • 3. Comparison of questioned material with

reference material

Blind analysis

  • 1. Blind analysis

– “Evidence line-up”:

  • anonimized material
  • addition of control group: “foils”
  • presented in random order

– Expert has no knowledge of the source of the materials, and searches unprejudiced for salient similarities/differences.

33/ 21

Vergelijkend spraakonderzoek Speaker identification

  • 1. Blind analysis

– No a priori expectations – Unprejudiced – Objective – “foils”: test of the expert + ‘ground truth’

Speaker identification

Limitations of (blind) analysis:

  • Speaker’s variation range
  • Comparison complicated when language

use situation and/or emotional/physical condition of speaker differ

– Limited basis for comparison

Speaker identification

  • 2. Non-blind analysis:

– Full materials – Speaker’s variation range – Context – Communicative circumstances – Line quality – Educated Native Speaker – But: bias effects

  • Only influences strength of conclusion, conclusion should stay

the same after blind analysis

slide-7
SLIDE 7

Acoustic / phonetic methods

  • Blind research!
  • Two co-workers, P1 & P2
  • Materials anonymized by P1

– Handles all information and material – Adds foils in case of speaker identification/verification – Randomizes the order of presentation of materials

  • questioned and reference material, and foils all together

– Numbers the materials, without reference to source

Acoustic / phonetic methods

  • P2 runs the first – blind –

acoustic/phonetic analyses

  • P2: first conclusion about grouping of

materials

  • P1 provides the key
  • P2 runs a second analysis with all material

and (necessary) information present

  • Strength of conclusions

Implications for forensic research

  • Forensic research: highly responsible

– Unintentionally prone to biases

  • Forensic experts international: Not only

critical on conclusions and reporting

  • evaluate methodology!
  • Also analytic phase: scrutiny.

– Improve scientific objectivity – Take precautions!

IAFPA

Standards

to avoid case information to contaminate evidence?

Thank you for your attention! Literature

  • Broeders, A.P.A. (2002). Het Herkennen van Stemmen, in: P.J. van Koppen, D. Hessing, H. Merckelbach

& H.F.M. Crombag (red.) Het Recht van Binnen: Psychologie van het Recht. Kluwer: Deventer, 573-596.

  • Broeders, A.P.A. (2009). De blinde onderzoeker. Trema Tijdschrift voor de Rechterlijke Macht, 6, 237-

243.

  • Broeders, A.P.A.(2010). Het Herkennen van Stemmen, in: P.J. van Koppen, H.L.G.J. Merckelbach, M.

Jelicic & J.W. de Keijser (red.) Reizen met mijn rechter: Psychologie van het Recht. Kluwer: Deventer, 305-332.

  • Dror, Itiel E., David Charlton, Ailsa E. Péron (2006a). Contextual information renders experts vulnerable

to making erroneous identifications. Forensic Science International, 156, 74–78.

  • Dror, I.E. (2006b). Why Experts make Errors. Journal of Forensic Identification, 600, 56-60.
  • Giard, R. & Merckelbach, H.L.G.J. (2009). Nietzsches gelijk: waarom wijsheid achteraf onbillijk is.

Nederlands Juristenblad, 16, 1014 – 1020.

  • Kerstholt, J.H., Raaijmakers, J.G.W., & Valeton, J.M. (1992). The Effect of Expectation on the

Identification of Known and Unknown Persons. Applied Cognitive Psychology, 6, 173-180.

  • Kerstholt, J.H. & Jackson, J.L. (1998). Judicial Decision Making: Order of Evidence Presentation and

Availability of Background Information. Applied Cognitive Psychology, 12, 445-454.

  • Lange, N.D., Thomas, R.P., Dana, J., & Dawes, R.M. (2011). Contextual Biases in the Interpretation of

Auditory Eveidence. Law and Human Behavior, 35, 178-187.

  • Merckelbach, H.L.G.J., Crombag, H.F.M. & Koppen, van P.J. (2003). ‘Hoge verwachtingen: over het

corrumperend effect van verwachtingen op forensische expertise’, Nederlands Juristenblad, 14, 710-716.

  • Risinger, D.M., Saks, M.J., Thompson, C.T. & Rosenthal, R. (2002). The DaubertIKumho implications of
  • bserver effects in forensic science: Hidden problems of expectation and suggestion. California Law

Review, 90, 1-56.

  • Thompson W.C. (2009). Painting the target around the matching profile (the Texas sharpshooter fallacy

in forensic DNA interpretation). Law, Probability, and Risk, 8, 257276.

  • Tobin, W.A. & Thompson, W.C. (2006). Evaluating and Challenging Forensic Identification Evidence.

Champion Magazine. National Association of Criminal Defense Lawyers, July, 12-23.