How to Evaluate Accuracy of Biometric Systems Peter Vojtek - - PowerPoint PPT Presentation

How to Evaluate Accuracy of Biometric Systems

Peter Vojtek

peter.vojtek@innovatrics.com

Intro

• PeWe member

○ 2006 - 2010

• Innovatrics

○ fingerprint-based biometrics

■ SDKs ■ large-scale fingerprint matching (AFIS) ■ end-products

○ 300M+ people

Typical Biometric Modalities

• Face
• Iris
• Retina
• Signature
• …
• Fingerprints

Examples of Biometric Systems

• National ID
• Social Insurance
• Health Insurance
• Border Control
• Driving Licenses
• Voter’s Lists
• Ghost Workers

Identity Management Systems

Accuracy

• Enrollment:

○ FTE: Fail to Enroll

• Verification:

○ FMR: False Match Rate ○ FNMR: False Non-Match Rate

• Identification:

○ FPIR: False Positive Identification Rate ○ FNIR: False Negative Identification Rate

AFIS

• Automated Fingerprint Identification System

○ CAFIS, MegaMatcher, ExpressID AFIS

• 100 000 000+ fingerprint comparisons / second / CPU

How to Compute Accuracy

• 1. Enroll 1000 different records for the first time

Every record must be unique, we label them A = {a1, a2, …, a1000}

• 2. Enroll same people again

We label them B = {b1, b2, …, b1000}. We know that a and b with the same index are from the same person

• 3. Perform verification of all records from A against all records from B

In total we will have 1 million matching results for every pair.

• 4. Analyze 1000 scores having the same index

This is called genuine distribution.

• 5. Analyze 999 000 scores having different index

This is called impostor distribution.

• 6. Calculate FNR and FNMR for different scores to get ROC curve

How to Compute Accuracy

Should: Accept Should: Reject Reality: Accepted TA FA Reality: Rejected FR TR

How to Compute Accuracy

Should: Accept Should: Reject Reality: Accepted TA (1000) FA (0) Reality: Rejected FR (0) TR (999 000) The false non-match rate is the expected probability that Ai will be falsely declared not to match to Bi.

FNMR = FR / (FR + TA) = 0 : 1000

How to Compute Accuracy

Should: Accept Should: Reject Reality: Accepted TA (999) FA (0) Reality: Rejected FR (1) TR (999 000) The false non-match rate is the expected probability that Ai will be falsely declared not to match to Bi.

FNMR = FR / (FR + TA) = 1 : 1000

How to Compute Accuracy

Should: Accept Should: Reject Reality: Accepted TA (1000) FA (0) Reality: Rejected FR (0) TR (999 000) The false match rate is the expected probability that a sample will be falsely declared to match a single randomly-selected “non-self”.

FMR = FA / (FA + TR) = 0 : 999 000

FNMR FMR Similarity score

ROC Curve

FNMR FMR

Examples of Real-life Accuracies

• iPhone 5S

○ Verification, 1 finger, FMR 1:50 000

• Time Attendance System

○ Verification, population 10-1000, 1 finger, FMR < 1:1000

• Population 4.5M, 6 fingers

○ Identification, FPIR < 1:100 000, FNIR < 2% (1:50)

How to Influence Accuracy

• Threshold

○ Security vs. comfort

• Fingerprints

○ how many, which positions ○ quality

○

position anonymization

• Template extractor
• Matching speed
• Discriminative ability of bio. modality

○ Dataset size ~ FMR

Customer and Accuracy

• Not aware
• Aware, but ignoring
• Cooperating
• Demanding

Datasets

• physical access
• huge difference in accuracy due to quality of fingerprints
• annotated datasets

Independent Accuracy Tests

• NIST PFT

○ Proprietary Fingerprint Template Evaluation ○ Verification

• NIST FpVTE

○ Fingerprint Vendor Technology Evaluation ○ Identification

• NIST MINEX

○ Minutia Exchange

NIST PFT II

Resources

• INDIA UID
• Introduction to Biometrics

○ Springer, 2011

• Best Practices in Testing and Reporting Performance of Biometric Devices

○ http://ftp.sas.ewi.utwente.nl/open/courses/intro_biometrics/Mansfield02.pdf

Other Keywords

• Deterrence effect
• Fingerprint quality (NFIQ)
• Speed
• Template extraction

○ basic pattern, minutiae points, pattern

• Segmentation
• ABIS
• Positive/Negative identification
• Criminal/Civil AFIS
• India UID, Indonesia eKTP
• iPhone
• FAR, FRR