Intro Calibration Gaussian scores Applications The End The distribution of calibrated likelihood-ratios in speaker recognition David van Leeuwen and Niko Br¨ ummer d.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University Nijmegen, Agnitio Research 15 October 2013 1 1 First published at Interspeech 2013 David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 1 / 20
Intro Calibration Gaussian scores Applications The End Inspiration for this work Target Non − target 0.6 Score distribution mean 0.5 • We had these 0.4 badly-behaving 0.3 scores 2 depending on 0.2 0.1 utterance duration Test duration 0 (sec) Train 20 duration 40 • We tried to design (sec) 80 60 60 40 80 20 universal calibration (a) Cosine Kernel transformations Target Non − target 60 • Question arose: Score distribution mean 50 40 where do calibrated 30 scores hang out? 20 10 • What is their Test 0 duration Train 20 (sec) distribution? duration 40 (sec) 60 80 60 80 40 20 (b) Normalized Cosine Kernel 2 Mandasari et al. , Interspeech 2011 David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 2 / 20
Intro Calibration Gaussian scores Applications The End What is calibration? Traditionally: • The capability to set a threshold correctly Nowadays: • The ability to give a proper probabilistic statement about identity • . . . to produce (log) likelihood ratio scores for every comparison • . . . that lead to optimal Bayes’ decisions David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 3 / 20
Intro Calibration Gaussian scores Applications The End What is calibration? Traditionally: • The capability to set a threshold correctly Nowadays: • The ability to give a proper probabilistic statement about identity • . . . to produce (log) likelihood ratio scores for every comparison • . . . that lead to optimal Bayes’ decisions Bayes’ decision Priors + likelihoods → posteriors Posteriors + costs → expected costs Minimize expected costs → decision David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 3 / 20
Intro Calibration Gaussian scores Applications The End The forensic motivation of the Likelihood Ratio Use the log Likelihood Ratio as weight of evidence in court • Using Bayes’s rule, separate contributions • Forensic Expert, w.r.t. the material they know about • The other evidence / circumstances of the case to compute the posterior probability that suspect is the perpetrator David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 4 / 20
Intro Calibration Gaussian scores Applications The End The forensic motivation of the Likelihood Ratio Use the log Likelihood Ratio as weight of evidence in court • Using Bayes’s rule, separate contributions • Forensic Expert, w.r.t. the material they know about E • The other evidence / circumstances of the case I to compute the posterior probability that suspect is the perpetrator H p = ¬ H d • Mathematically, P ( H p | E , I ) P ( E | H p , I ) P ( H p , I ) × = P ( H d | E , I ) P ( E | H d , I ) P ( H d , I ) � �� � � �� � � �� � given by expert other evidence judge/jury wants to know David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 4 / 20
Intro Calibration Gaussian scores Applications The End From scores to likelihood ratios • A likelihood ratio can be treated like a score • All analysis tricks work: ROC, DET, EER, decision cost functions. . . • But can we transform a score into a LR? • This is a process known as calibration: giving meaning to probabilistic statements David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 5 / 20
Intro Calibration Gaussian scores Applications The End From scores to likelihood ratios • A likelihood ratio can be treated like a score • All analysis tricks work: ROC, DET, EER, decision cost functions. . . • But can we transform a score into a LR? • This is a process known as calibration: giving meaning to probabilistic statements problem statement But what is the definition of calibrated scores / LRs? David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 5 / 20
Intro Calibration Gaussian scores Applications The End Definition of Calibrated Likelihood Ratios Our definition 3 The LR of the LR is the LR or, for the mathematically inclined LR = P ( LR | H p ) P ( LR | H d ) 3 Proof in paper, short version in Mandasari et al. , IEEE-TASLP (2013, accepted) David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 6 / 20
Intro Calibration Gaussian scores Applications The End Definition of Calibrated Likelihood Ratios Our definition 3 The LR of the LR is the LR or, for the mathematically inclined LR = P ( LR | H p ) P ( LR | H d ) which happens to be equivalent to log LR = log P (log LR | H p ) P (log LR | H d ) The LLR of the LLR is the LLR 3 Proof in paper, short version in Mandasari et al. , IEEE-TASLP (2013, accepted) David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 6 / 20
Intro Calibration Gaussian scores Applications The End More inspiration: Why are DET curves straight? • If score distributions are Gaussian, then DET curve is straight 40 d’= 1 • Slope is ratio of minimum DCF operating point 20 standard- miss probability (%) DCF operating point deviations of the 10 EER score distributions 5 • If DET is straight, d’= 4 2 score distributions are 1 not necessarily d’= 5 0.5 Gaussian d’= 6 • but can be made 0.1 Gaussian by 0.1 0.5 1 2 5 10 20 40 warping of score false alarm probability (%) axis David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 7 / 20
Intro Calibration Gaussian scores Applications The End For reference: these are the score distributions Probability density Non − targets probability of false alarm • Clearly not Gaussian miss 0.15 • But still leading to a threshold straight DET curve Density 0.10 • non-targets: d ( x ) Targets (different) 0.05 • targets: e ( x ) (equal) 0.00 − 10 0 10 20 score David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 8 / 20
Intro Calibration Gaussian scores Applications The End Can Gaussian Scores be Well Calibrated? Let’s try • Gaussian non-targets d ( x ) = N ( x | µ d , σ 2 d ) • calibration definition for LLR: x = log e ( x ) d ( x ) targets e ( x ) = e x d ( x ) Now use the expression for the normal distribution N , and see what the targets e ( x ) look like 1 e x − ( x − µ d ) 2 / 2 σ 2 e ( x ) = e x d ( x ) = √ d 2 πσ d David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 9 / 20
Intro Calibration Gaussian scores Applications The End Math 101 Expanding the exponent for target distribution e ( x ): − x 2 − 2 µ d x + µ 2 + 2 σ 2 d x d 2 σ 2 2 σ 2 d d = − x 2 − 2( µ d + σ 2 d ) x + µ 2 d 2 σ 2 d � � 2 x − ( µ d + σ 2 2 µ d σ 2 d + σ 4 d ) d = − + 2 σ 2 2 σ 2 d d � �� � � �� � Gaussian form Normalisation constant David van Leeuwen and Niko Br¨ ummerd.vanleeuwen@let.ru.nl, nbrummer@agnito.es Netherlands Forensic Institute / Radboud University BTFS 2013 The distribution of calibrated likelihood-ratios in speaker recognition 10 / 20
Recommend
More recommend
