Recent Guidelines Related to the Implementation of Speaker - PowerPoint PPT Presentation

Forensic Panel Odyssey 2016 23 June 2016, Bilbao, Spain Recent Guidelines Related to the Implementation of Speaker Recognition In Court in Europe Daniel Ramos ATVS – Biometric Recognition Group Universidad Autonoma de Madrid http://atvs.ii.uam.es

Three Recent Guidelines… Released in 2015 Forensic Panel, OdysseBilbao, 23 June, 2016. 2

Three Recent Guidelines… Released in 2015 Released in 2016 Forensic Panel, OdysseBilbao, 23 June, 2016. 3

Three Recent Guidelines… Released in 2015 Released in 2016 Released in 2016 Forensic Panel, OdysseBilbao, 23 June, 2016. 4

ENFSI Guideline for Evaluative Reporting http://www.enfsi.eu/documents/forensic-guidelines Forensic Panel, OdysseBilbao, 23 June, 2016. 5

ENFSI Guideline: Critical Recommendations n Probabilistic expressions of conclusions q Verbal Scales are recommended q Avoid absolute statements (identification/exclusion)! Forensic Panel, OdysseBilbao, 23 June, 2016. 6

ENFSI Guideline: Critical Recommendations n Probabilistic expressions of conclusions q Verbal Scales are recommended q Avoid absolute statements (identification/exclusion)! n Hypotheses in a case always must consider at least one alternative q Clear definition of the population of potential alternative authors Forensic Panel, OdysseBilbao, 23 June, 2016. 7

ENFSI Guideline: Critical Recommendations n Probabilistic expressions of conclusions q Verbal Scales are recommended q Avoid absolute statements (identification/exclusion)! n Hypotheses in a case always must consider at least one alternative q Clear definition of the population of potential alternative authors n Findings must be evaluated given each of all hypotheses q Likelihood ratios (LR) Forensic Panel, OdysseBilbao, 23 June, 2016. 8

ENFSI Guideline: Critical Recommendations n Probabilistic expressions of conclusions q Verbal Scales are recommended q Avoid absolute statements (identification/exclusion)! n Hypotheses in a case always must consider at least one alternative q Clear definition of the population of potential alternative authors n Findings must be evaluated given each of all hypotheses q Likelihood ratios (LR) n Conclusions expressed in terms of support to the hypotheses (LR) q Avoid probabilities of the hypotheses (posteriors) Forensic Panel, OdysseBilbao, 23 June, 2016. 9

ENFSI Guideline: Critical Recommendations n Probabilistic expressions of conclusions q Verbal Scales are recommended q Avoid absolute statements (identification/exclusion)! n Hypotheses in a case always must consider at least one alternative q Clear definition of the population of potential alternative authors n Findings must be evaluated given each of all hypotheses q Likelihood ratios (LR) n Conclusions expressed in terms of support to the hypotheses (LR) q Avoid probabilities of the hypotheses (posteriors) n Towards data-driven testable and repeatable methodologies, but… q Subjective judgements (probabilities, LRs) are allowed Forensic Panel, OdysseBilbao, 23 June, 2016. 10

A Simple Example (Automatic Systems) Forensic Panel, OdysseBilbao, 23 June, 2016. 11

Guideline: Validation of Forensic LR Methods Forensic Panel, OdysseBilbao, 23 June, 2016. 12

Guideline: Validation of Forensic LR Methods n Objective q Determine if a LR method is valid to be used in casework q All the validation process should be documented for transparency Forensic Panel, OdysseBilbao, 23 June, 2016. 13

Guideline: Validation of Forensic LR Methods n Objective q Determine if a LR method is valid to be used in casework q All the validation process should be documented for transparency q Towards standardization of procedures (for biometrics) Forensic Panel, OdysseBilbao, 23 June, 2016. 14

Guideline: Validation of Forensic LR Methods n Objective q Determine if a LR method is valid to be used in casework q All the validation process should be documented for transparency q Towards standardization of procedures (for biometrics) q Proposed some relevant performance characteristics Performance Performance Graphical Validation Characteristic Metric Representation criteria Experiment Data Analytical result Validation Decision ECE plot According to the Data +/- [%] compared Pass / fail Accuracy Cllr, EER NBE plot definition Description used to the baseline ECE plot According to the Data +/- [%] compared Cllr min Discriminating power NBE plot definition Description used to the baseline Pass / fail Tippett plot ECE plot According to the Data +/- [%] compared Cllr cal Calibration NBE plot definition Description used to the baseline Pass / fail ECE plot Variation of DET plot According to the Data +/- [%] compared Robustness Cllr, EER Tippett plot definition Description used to the baseline Pass / fail ECE plot Coherence Variation of DET plot According to the Data +/- [%] compared Cllr, EER Tippett plot definition Description used to the baseline Pass / fail ECE plot Variation of DET plot According to the Data +/- [%] compared Generalization Cllr, EER Tippett plot definition Description used to the baseline Pass / fail Forensic Panel, OdysseBilbao, 23 June, 2016. 15

Guideline: Validation of Forensic LR Methods n Objective q Determine if a LR method is valid to be used in casework q All the validation process should be documented for transparency q Towards standardization of procedures (for biometrics) q Proposed some relevant performance characteristics Performance Performance Graphical Validation Characteristic Metric Representation criteria Experiment Data Analytical result Validation Decision NOT ECE plot According to the Data +/- [%] compared Pass / fail Accuracy Cllr, EER NBE plot definition Description used to the baseline LIMITED ECE plot According to the Data +/- [%] compared Cllr min Discriminating power NBE plot definition Description used to the baseline Pass / fail Tippett plot TO THESE ECE plot According to the Data +/- [%] compared Cllr cal Calibration NBE plot definition Description used to the baseline Pass / fail (Guideline ECE plot Variation of DET plot According to the Data +/- [%] compared Robustness Cllr, EER Tippett plot definition Description used to the baseline Pass / fail is open in ECE plot Coherence Variation of DET plot According to the Data +/- [%] compared Cllr, EER Tippett plot definition Description used to the baseline Pass / fail this sense) ECE plot Variation of DET plot According to the Data +/- [%] compared Generalization Cllr, EER Tippett plot definition Description used to the baseline Pass / fail ! Forensic Panel, OdysseBilbao, 23 June, 2016. 16

Guideline: Validation of Forensic LR Methods n Objective q Determine if a LR method is valid to be used in casework q All the validation process should be documented for transparency q Towards standardization of procedures (for biometrics) q Proposed some relevant performance characteristics Performance Performance Graphical Validation Characteristic Metric Representation criteria Experiment Data Analytical result Validation Decision ECE plot According to the Data +/- [%] compared Pass / fail Accuracy Cllr, EER NBE plot definition Description used to the baseline Emphasis on the use of ECE plot According to the Data +/- [%] compared Cllr min Discriminating power NBE plot definition Description used to the baseline Pass / fail Forensically Relevant Data Tippett plot ECE plot According to the Data +/- [%] compared Cllr cal Calibration NBE plot definition Description used to the baseline Pass / fail • Lab performance first ECE plot Variation of DET plot According to the Data +/- [%] compared • Followed by forensic Robustness Cllr, EER Tippett plot definition Description used to the baseline Pass / fail ECE plot Coherence Variation of performance (CRITICAL!) DET plot According to the Data +/- [%] compared Cllr, EER Tippett plot definition Description used to the baseline Pass / fail ECE plot Variation of DET plot According to the Data +/- [%] compared Generalization Cllr, EER Tippett plot definition Description used to the baseline Pass / fail ! Forensic Panel, OdysseBilbao, 23 June, 2016. 17

ENFSI Guideline: Forensic Automatic and Semi- Automatic Speaker Recognition http://www.enfsi.eu/documents/forensic-guidelines Forensic Panel, OdysseBilbao, 23 June, 2016. 18

ENFSI Guideline: Forensic Automatic and Semi- Automatic Speaker Recognition n Compatible with ENFSI Guideline for reporting q Reporting likelihood ratios http://www.enfsi.eu/documents/forensic-guidelines Forensic Panel, OdysseBilbao, 23 June, 2016. 19

ENFSI Guideline: Forensic Automatic and Semi- Automatic Speaker Recognition n Compatible with ENFSI Guideline for reporting q Reporting likelihood ratios n Compatible with Guideline for Validation q Similar performance characteristics q Emphasis in forensically relevant data http://www.enfsi.eu/documents/forensic-guidelines Forensic Panel, OdysseBilbao, 23 June, 2016. 20

ENFSI Guideline: Forensic Automatic and Semi- Automatic Speaker Recognition n Compatible with ENFSI Guideline for reporting q Reporting likelihood ratios n Compatible with Guideline for Validation q Similar performance characteristics q Emphasis in forensically relevant data n Addresses many other issues q Most used technologies and methods q Most used features q Audio pre-processing q … http://www.enfsi.eu/documents/forensic-guidelines Forensic Panel, OdysseBilbao, 23 June, 2016. 21