Summary of the 2015 NIST Language Recognition i-Vector Machine - PowerPoint PPT Presentation

Summary of the 2015 NIST Language Recognition i-Vector Machine Learning Challenge Craig Greenberg Audrey Tong, Alvin Martin, George Doddington National Institute of Standards and Technology Douglas Reynolds, Elliot Singer MIT Lincoln Laboratory Desire Banse, John Howard, Hui Zhao NIST contractors Daniel Garcia-Romero, Alan McCree HLT Center of Excellence, JHU Jaime Hernandez-Cordero, Lisa Mason U.S. Department of Defense Odyssey 2016 Special Session June 24, 2016

Motivation  Attract researchers outside of the speech processing community to work on language recognition  Explore new ideas in machine learning for use in language recognition  Improve the performance of language recognition technology 2

The Task: Open Set Language Identification Language 1 Language 2 ? Audio . . . Segment Language N Unknown Languages 3

The Data & Language Selection  Telephone conversations and narrowband broadcasts  Previous NIST LREs 1996 – 2011  Selected data from the IARPA Babel Program  Data driven selection  Only languages with multiple sources were select to reduce source to language effect  Language pairs with higher confusability were preferred 4

Data Size Number of Number of Set Total Segments Languages Segments per Language Train 50 300 1500 65 Dev. • 50 target ≈ 100 6431 • 15 out-of-set 6500 • 5000 target 65 – 1500 progress Test • 50 target 100 – 3500 evaluation • 15 out-of-set • 1500 out -of-set – 450 progress – 1050 evaluation  Training set did not include out-of-set data  Development set included an unlabeled out-of-set  Test set was divided into a progress and evaluation subsets 5

Data Sources Distribution of sources across languages similar 6

Segment Speech Durations Distribution of segment speech durations relatively similar 7

Performance Metric • k = target language • oos = out-of-set language • n = total number of target languages = 50 • P oos = prior probably of oos language = 0.23 8

Participation  Worldwide participation  6 continents  31 countries  78 participants downloaded data  56 participants submitted results  44 unique organizations  3773 valid submissions during the challenge period (May 1 - September 1, 2015)  4021 submissions as of January 2016 9

Contrast with Traditional LREs i-Vector Traditional Input i-vector representations of audio audio segments segments Task Identification (n-class problem) Detection (2-class problem) Cost based on miss and false alarm Metric Cost based on error rates rates Target Language 50 10 – 25 Segment Speech Log normal distribution with a Uniform distribution of 3, 10, 30 Duration mean of 35 secs secs Challenge Duration 4 months 1 week Scoring Results feedback on portion of the No results feedback test set Evaluation Platform Web-based Manual 10

Web-based Evaluation Platform  Goal was to facilitate the evaluation process with limited human involvement  All evaluation activities were conducted via a web interface  Download training and evaluation data  Upload submissions for validation and scoring  Track submission status  View results and site ranking 11

Daily Best Cost During the Challenge 12

Best Cost Per Participant at End of Challenge 13

Number of Submissions Per Participant 14

Results by Target Language 15

Results by Speech Duration 16

Lessons Learned  Record participation, more than all previous LREs  56 participants from 44 unique organizations from 31 countries in 6 continents  46 of 56 systems were better than the baseline system  6 were better than the oracle system  Half of improvement made in the first few weeks of the four-month challenge  Top systems did well on Burmese, not so on English or Hindi  Performance on OOS language was in the middle  Did not receive many system descriptions so not clear if new methods were investigated  Top system did develop novel technique to improve out-of- set detection (talk upcoming) 17

Benchmark Your System  NIST plans to keep the web-based evaluation platform up for the foreseeable future as a system development tool  Visit http://ivectorchallenge.nist.gov  Download the training and development data  Test your system on the test data used in the Challenge 18

Upcoming Activities  SRE16 & Workshop – 15 th edition  Speaker detection of telephone speech recorded over variety of handsets  Introduction of fixed training condition  Test segments with more speech duration variability  Data collected outside North America  Inclusion of trials using same and different phone numbers  http://www.nist.gov/itl/iad/mig/sre16.cfm  2016 LRE Analysis Workshop  In-depth analysis of results from LRE15  Co-located with SLT16 in San Juan, Puerto Rico 19