iiit h system submission for fire2014 shared task on
play

IIIT-H System Submission for FIRE2014 Shared Task on Transliterated - PowerPoint PPT Presentation

Oct 29, 2023 •236 likes •1.34k views

IIIT-H System Submission for FIRE2014 Shared Task on Transliterated Search Irshad Ahmad Bhat Vandan Mujadia Aniruddha Tammewar Riyaz Ahmad Bhat Manish Shrivastava Language Technologies Research Centre, International Institute of Information

  1. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Description Language Identification (LID) of query words in code-mixed queries Code-mixing - A socio-linguistic phenomenon prominent among multi-lingual speakers switch back and forth between two or more languages or language-varieties spoken and written communication sudden rise due to increase in social networking channels Why LID? Pre-requisite for various NLP tasks ∵ Performance of any NLP task ∝ amount and level of code-mixing e.g. Parsing, MT, ASR, IR & IE, Semantic Processing, etc. 3 / 18

  2. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Description Language Identification (LID) of query words in code-mixed queries Code-mixing - A socio-linguistic phenomenon prominent among multi-lingual speakers switch back and forth between two or more languages or language-varieties spoken and written communication sudden rise due to increase in social networking channels Why LID? Pre-requisite for various NLP tasks ∵ Performance of any NLP task ∝ amount and level of code-mixing e.g. Parsing, MT, ASR, IR & IE, Semantic Processing, etc. 3 / 18

  3. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Description Language Identification (LID) of query words in code-mixed queries Code-mixing - A socio-linguistic phenomenon prominent among multi-lingual speakers switch back and forth between two or more languages or language-varieties spoken and written communication sudden rise due to increase in social networking channels Why LID? Pre-requisite for various NLP tasks ∵ Performance of any NLP task ∝ amount and level of code-mixing e.g. Parsing, MT, ASR, IR & IE, Semantic Processing, etc. 3 / 18

  4. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Description Language Identification (LID) of query words in code-mixed queries Code-mixing - A socio-linguistic phenomenon prominent among multi-lingual speakers switch back and forth between two or more languages or language-varieties spoken and written communication sudden rise due to increase in social networking channels Why LID? Pre-requisite for various NLP tasks ∵ Performance of any NLP task ∝ amount and level of code-mixing e.g. Parsing, MT, ASR, IR & IE, Semantic Processing, etc. 3 / 18

  5. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Description Back transliteration of Indic words to their native scripts. Challenge - Enormous noise/variation in transliterated form particularly in social media. Importance - Retrieval of relevant documents in native script for a Roman transliterated query. Example queries and their expected system output Input query Outputs sachin \ H tendulkar \ H number \ E of \ E sachin tendulkar number of centuries centuries \ E palak paneer recipe palak \ H= ���к paneer \ H= ���� recipe \ E mungeri \ H= � ��� ��� lal \ H= ��� ke \ H= к � mungeri lal ke haseen sapney haseen \ H= ���� sapney \ H= ��� iguazu \ E water \ E fall \ E argentina \ E iguazu water fall argentina Table 1: Input query with desired outputs, where L is Hindi and has to be labeled as H 4 / 18

  6. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Description Back transliteration of Indic words to their native scripts. Challenge - Enormous noise/variation in transliterated form particularly in social media. Importance - Retrieval of relevant documents in native script for a Roman transliterated query. Example queries and their expected system output Input query Outputs sachin \ H tendulkar \ H number \ E of \ E sachin tendulkar number of centuries centuries \ E palak paneer recipe palak \ H= ���к paneer \ H= ���� recipe \ E mungeri \ H= � ��� ��� lal \ H= ��� ke \ H= к � mungeri lal ke haseen sapney haseen \ H= ���� sapney \ H= ��� iguazu \ E water \ E fall \ E argentina \ E iguazu water fall argentina Table 1: Input query with desired outputs, where L is Hindi and has to be labeled as H 4 / 18

  7. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Description Back transliteration of Indic words to their native scripts. Challenge - Enormous noise/variation in transliterated form particularly in social media. Importance - Retrieval of relevant documents in native script for a Roman transliterated query. Example queries and their expected system output Input query Outputs sachin \ H tendulkar \ H number \ E of \ E sachin tendulkar number of centuries centuries \ E palak paneer recipe palak \ H= ���к paneer \ H= ���� recipe \ E mungeri \ H= � ��� ��� lal \ H= ��� ke \ H= к � mungeri lal ke haseen sapney haseen \ H= ���� sapney \ H= ��� iguazu \ E water \ E fall \ E argentina \ E iguazu water fall argentina Table 1: Input query with desired outputs, where L is Hindi and has to be labeled as H 4 / 18

  8. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Description Back transliteration of Indic words to their native scripts. Challenge - Enormous noise/variation in transliterated form particularly in social media. Importance - Retrieval of relevant documents in native script for a Roman transliterated query. Example queries and their expected system output Input query Outputs sachin \ H tendulkar \ H number \ E of \ E sachin tendulkar number of centuries centuries \ E palak paneer recipe palak \ H= ���к paneer \ H= ���� recipe \ E mungeri \ H= � ��� ��� lal \ H= ��� ke \ H= к � mungeri lal ke haseen sapney haseen \ H= ���� sapney \ H= ��� iguazu \ E water \ E fall \ E argentina \ E iguazu water fall argentina Table 1: Input query with desired outputs, where L is Hindi and has to be labeled as H 4 / 18

  9. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Description Back transliteration of Indic words to their native scripts. Challenge - Enormous noise/variation in transliterated form particularly in social media. Importance - Retrieval of relevant documents in native script for a Roman transliterated query. Example queries and their expected system output Input query Outputs sachin \ H tendulkar \ H number \ E of \ E sachin tendulkar number of centuries centuries \ E palak paneer recipe palak \ H= ���к paneer \ H= ���� recipe \ E mungeri \ H= � ��� ��� lal \ H= ��� ke \ H= к � mungeri lal ke haseen sapney haseen \ H= ���� sapney \ H= ��� iguazu \ E water \ E fall \ E argentina \ E iguazu water fall argentina Table 1: Input query with desired outputs, where L is Hindi and has to be labeled as H 4 / 18

  10. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  11. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  12. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  13. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  14. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  15. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  16. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  17. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  18. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  19. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  20. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  21. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  22. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  23. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Data Word Query Labeling is meant for 6 language-pairs: Hindi-English (H-E) Gujarati-English (G-E) Bengali-English (B-E) Tamil-English (T-E) Kannada-English (K-E) Malayalam-English (M-E). Data released contain the following: Monolingual corpora of English, Hindi and Gujarati. Word lists with corpus frequencies for English, Hindi, Bengali and Gujarati. Word transliteration pairs for Hindi-English, Bengali-English and Gujarati-English. A development set of 1000 transliterated code-mixed queries for each language pair. A separate test set of ∼ 1000 queries for the evaluation of of results. 5 / 18

  24. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Token Level Language Identification Query word labeling is a similar problem to Document-level Language Identification task [1] Query word labeling is a token level language identification problem while Document language identification is about deciphering the language a document is written in. More complex than Document-level Language Identification ∵ #features Document-level > #features Word-level Features available for Query word labeling are mostly restricted to word level like: word morphology syllable structure phonemic (letter) inventory n -gram models best suited for the task [2], [3], [5], [7], [6] 6 / 18

  25. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Token Level Language Identification Query word labeling is a similar problem to Document-level Language Identification task [1] Query word labeling is a token level language identification problem while Document language identification is about deciphering the language a document is written in. More complex than Document-level Language Identification ∵ #features Document-level > #features Word-level Features available for Query word labeling are mostly restricted to word level like: word morphology syllable structure phonemic (letter) inventory n -gram models best suited for the task [2], [3], [5], [7], [6] 6 / 18

  26. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Token Level Language Identification Query word labeling is a similar problem to Document-level Language Identification task [1] Query word labeling is a token level language identification problem while Document language identification is about deciphering the language a document is written in. More complex than Document-level Language Identification ∵ #features Document-level > #features Word-level Features available for Query word labeling are mostly restricted to word level like: word morphology syllable structure phonemic (letter) inventory n -gram models best suited for the task [2], [3], [5], [7], [6] 6 / 18

  27. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Token Level Language Identification Query word labeling is a similar problem to Document-level Language Identification task [1] Query word labeling is a token level language identification problem while Document language identification is about deciphering the language a document is written in. More complex than Document-level Language Identification ∵ #features Document-level > #features Word-level Features available for Query word labeling are mostly restricted to word level like: word morphology syllable structure phonemic (letter) inventory n -gram models best suited for the task [2], [3], [5], [7], [6] 6 / 18

  28. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Token Level Language Identification Query word labeling is a similar problem to Document-level Language Identification task [1] Query word labeling is a token level language identification problem while Document language identification is about deciphering the language a document is written in. More complex than Document-level Language Identification ∵ #features Document-level > #features Word-level Features available for Query word labeling are mostly restricted to word level like: word morphology syllable structure phonemic (letter) inventory n -gram models best suited for the task [2], [3], [5], [7], [6] 6 / 18

  29. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Token Level Language Identification Query word labeling is a similar problem to Document-level Language Identification task [1] Query word labeling is a token level language identification problem while Document language identification is about deciphering the language a document is written in. More complex than Document-level Language Identification ∵ #features Document-level > #features Word-level Features available for Query word labeling are mostly restricted to word level like: word morphology syllable structure phonemic (letter) inventory n -gram models best suited for the task [2], [3], [5], [7], [6] 6 / 18

  30. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Token Level Language Identification Query word labeling is a similar problem to Document-level Language Identification task [1] Query word labeling is a token level language identification problem while Document language identification is about deciphering the language a document is written in. More complex than Document-level Language Identification ∵ #features Document-level > #features Word-level Features available for Query word labeling are mostly restricted to word level like: word morphology syllable structure phonemic (letter) inventory n -gram models best suited for the task [2], [3], [5], [7], [6] 6 / 18

  31. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Token Level Language Identification Query word labeling is a similar problem to Document-level Language Identification task [1] Query word labeling is a token level language identification problem while Document language identification is about deciphering the language a document is written in. More complex than Document-level Language Identification ∵ #features Document-level > #features Word-level Features available for Query word labeling are mostly restricted to word level like: word morphology syllable structure phonemic (letter) inventory n -gram models best suited for the task [2], [3], [5], [7], [6] 6 / 18

  32. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Token Level Language Identification Query word labeling is a similar problem to Document-level Language Identification task [1] Query word labeling is a token level language identification problem while Document language identification is about deciphering the language a document is written in. More complex than Document-level Language Identification ∵ #features Document-level > #features Word-level Features available for Query word labeling are mostly restricted to word level like: word morphology syllable structure phonemic (letter) inventory n -gram models best suited for the task [2], [3], [5], [7], [6] 6 / 18

  33. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Query Word Classification Language Identification as a classification problem For each query word, predict its class from a finite set of classes. In our case classes labels are: English Any of the six Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam and Tamil Ambiguous Named Entity Other Features for classification Letter-based n-gram posterior probabilities Use of Dictionaries 7 / 18

  34. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Query Word Classification Language Identification as a classification problem For each query word, predict its class from a finite set of classes. In our case classes labels are: English Any of the six Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam and Tamil Ambiguous Named Entity Other Features for classification Letter-based n-gram posterior probabilities Use of Dictionaries 7 / 18

  35. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Query Word Classification Language Identification as a classification problem For each query word, predict its class from a finite set of classes. In our case classes labels are: English Any of the six Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam and Tamil Ambiguous Named Entity Other Features for classification Letter-based n-gram posterior probabilities Use of Dictionaries 7 / 18

  36. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Query Word Classification Language Identification as a classification problem For each query word, predict its class from a finite set of classes. In our case classes labels are: English Any of the six Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam and Tamil Ambiguous Named Entity Other Features for classification Letter-based n-gram posterior probabilities Use of Dictionaries 7 / 18

  37. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Query Word Classification Language Identification as a classification problem For each query word, predict its class from a finite set of classes. In our case classes labels are: English Any of the six Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam and Tamil Ambiguous Named Entity Other Features for classification Letter-based n-gram posterior probabilities Use of Dictionaries 7 / 18

  38. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Query Word Classification Language Identification as a classification problem For each query word, predict its class from a finite set of classes. In our case classes labels are: English Any of the six Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam and Tamil Ambiguous Named Entity Other Features for classification Letter-based n-gram posterior probabilities Use of Dictionaries 7 / 18

  39. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Query Word Classification Language Identification as a classification problem For each query word, predict its class from a finite set of classes. In our case classes labels are: English Any of the six Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam and Tamil Ambiguous Named Entity Other Features for classification Letter-based n-gram posterior probabilities Use of Dictionaries 7 / 18

  40. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Query Word Classification Language Identification as a classification problem For each query word, predict its class from a finite set of classes. In our case classes labels are: English Any of the six Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam and Tamil Ambiguous Named Entity Other Features for classification Letter-based n-gram posterior probabilities Use of Dictionaries 7 / 18

  41. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Query Word Classification Language Identification as a classification problem For each query word, predict its class from a finite set of classes. In our case classes labels are: English Any of the six Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam and Tamil Ambiguous Named Entity Other Features for classification Letter-based n-gram posterior probabilities Use of Dictionaries 7 / 18

  42. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Query Word Classification Language Identification as a classification problem For each query word, predict its class from a finite set of classes. In our case classes labels are: English Any of the six Indian languages: Bengali, Hindi, Gujarati, Marathi, Malayalam and Tamil Ambiguous Named Entity Other Features for classification Letter-based n-gram posterior probabilities Use of Dictionaries 7 / 18

  43. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Posterior Probabilities Train separate letter-based smoothed n -gram LMs for each language in a language pair N -gram LMs Compute the conditional probability corresponding to k 1 classes c 1 , c 2 , ... , c k as: p ( c i | w ) = p ( w | c i ) ∗ p ( c i ) (1) Prior distribution p ( c ) of a class is estimated from the respective training sets shown below. Language Data Size Average Token Length Hindi 32,9091 9.19 English 94,514 4.78 Gujarati 40,889 8.84 Tamil 55,370 11.78 Malayalam 12,8118 13.18 Bengali 29,3240 11.08 Kannada 579736 12.74 1 k = 2 for each LP 8 / 18

  44. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Posterior Probabilities Train separate letter-based smoothed n -gram LMs for each language in a language pair N -gram LMs Compute the conditional probability corresponding to k 1 classes c 1 , c 2 , ... , c k as: p ( c i | w ) = p ( w | c i ) ∗ p ( c i ) (1) Prior distribution p ( c ) of a class is estimated from the respective training sets shown below. Language Data Size Average Token Length Hindi 32,9091 9.19 English 94,514 4.78 Gujarati 40,889 8.84 Tamil 55,370 11.78 Malayalam 12,8118 13.18 Bengali 29,3240 11.08 Kannada 579736 12.74 1 k = 2 for each LP 8 / 18

  45. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Posterior Probabilities Train separate letter-based smoothed n -gram LMs for each language in a language pair N -gram LMs Compute the conditional probability corresponding to k 1 classes c 1 , c 2 , ... , c k as: p ( c i | w ) = p ( w | c i ) ∗ p ( c i ) (1) Prior distribution p ( c ) of a class is estimated from the respective training sets shown below. Language Data Size Average Token Length Hindi 32,9091 9.19 English 94,514 4.78 Gujarati 40,889 8.84 Tamil 55,370 11.78 Malayalam 12,8118 13.18 Bengali 29,3240 11.08 Kannada 579736 12.74 1 k = 2 for each LP 8 / 18

  46. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Posterior Probabilities Train separate letter-based smoothed n -gram LMs for each language in a language pair N -gram LMs Compute the conditional probability corresponding to k 1 classes c 1 , c 2 , ... , c k as: p ( c i | w ) = p ( w | c i ) ∗ p ( c i ) (1) Prior distribution p ( c ) of a class is estimated from the respective training sets shown below. Language Data Size Average Token Length Hindi 32,9091 9.19 English 94,514 4.78 Gujarati 40,889 8.84 Tamil 55,370 11.78 Malayalam 12,8118 13.18 Bengali 29,3240 11.08 Kannada 579736 12.74 1 k = 2 for each LP 8 / 18

  47. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results LM p ( w ) is implemented as an n -gram model using the IRSTLM-Toolkit[4] with Kneser-Ney smoothing as: n � p ( l i | l i − 1 p ( w ) = i − j ) (2) i =1 where l is a letter and j 2 is a parameter indicating the amount of context used 2 j=4 = ⇒ 5-gram model 9 / 18

  48. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Lib-linear SVM classifier Trained separate SVM classifiers for each language pair Low dimensional feature vectors: Posterior probabilities from both the language models in a language pair Presence of a word in English dictionaries as a boolean feature. We use python’s PyEnchant-package with the following dictionaries: en GB: British English en US: American English de DE: German fr FR: French 10 / 18

  49. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Lib-linear SVM classifier Trained separate SVM classifiers for each language pair Low dimensional feature vectors: Posterior probabilities from both the language models in a language pair Presence of a word in English dictionaries as a boolean feature. We use python’s PyEnchant-package with the following dictionaries: en GB: British English en US: American English de DE: German fr FR: French 10 / 18

  50. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Lib-linear SVM classifier Trained separate SVM classifiers for each language pair Low dimensional feature vectors: Posterior probabilities from both the language models in a language pair Presence of a word in English dictionaries as a boolean feature. We use python’s PyEnchant-package with the following dictionaries: en GB: British English en US: American English de DE: German fr FR: French 10 / 18

  51. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Lib-linear SVM classifier Trained separate SVM classifiers for each language pair Low dimensional feature vectors: Posterior probabilities from both the language models in a language pair Presence of a word in English dictionaries as a boolean feature. We use python’s PyEnchant-package with the following dictionaries: en GB: British English en US: American English de DE: German fr FR: French 10 / 18

  52. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Lib-linear SVM classifier Trained separate SVM classifiers for each language pair Low dimensional feature vectors: Posterior probabilities from both the language models in a language pair Presence of a word in English dictionaries as a boolean feature. We use python’s PyEnchant-package with the following dictionaries: en GB: British English en US: American English de DE: German fr FR: French 10 / 18

  53. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Lib-linear SVM classifier Trained separate SVM classifiers for each language pair Low dimensional feature vectors: Posterior probabilities from both the language models in a language pair Presence of a word in English dictionaries as a boolean feature. We use python’s PyEnchant-package with the following dictionaries: en GB: British English en US: American English de DE: German fr FR: French 10 / 18

  54. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Lib-linear SVM classifier Trained separate SVM classifiers for each language pair Low dimensional feature vectors: Posterior probabilities from both the language models in a language pair Presence of a word in English dictionaries as a boolean feature. We use python’s PyEnchant-package with the following dictionaries: en GB: British English en US: American English de DE: German fr FR: French 10 / 18

  55. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Lib-linear SVM classifier Trained separate SVM classifiers for each language pair Low dimensional feature vectors: Posterior probabilities from both the language models in a language pair Presence of a word in English dictionaries as a boolean feature. We use python’s PyEnchant-package with the following dictionaries: en GB: British English en US: American English de DE: German fr FR: French 10 / 18

  56. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Back Transliteration of Indic Words Transliteration of Indic words from Roman to the respective native scripts Learn a classification model that can predict a phonetically equivalent letter sequence from target script for each letter sequence in a source script. Transliteration of the said 6 Indian languages is carried out in the following manner: Convert Indic words in training data to WX for readability. WX is a transliteration scheme for representing Indian languages in ASCII. In WX every consonant and every vowel has a single mapping into Roman, that means there is no loss of information while conversion. 11 / 18

  57. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Back Transliteration of Indic Words Transliteration of Indic words from Roman to the respective native scripts Learn a classification model that can predict a phonetically equivalent letter sequence from target script for each letter sequence in a source script. Transliteration of the said 6 Indian languages is carried out in the following manner: Convert Indic words in training data to WX for readability. WX is a transliteration scheme for representing Indian languages in ASCII. In WX every consonant and every vowel has a single mapping into Roman, that means there is no loss of information while conversion. 11 / 18

  58. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Back Transliteration of Indic Words Transliteration of Indic words from Roman to the respective native scripts Learn a classification model that can predict a phonetically equivalent letter sequence from target script for each letter sequence in a source script. Transliteration of the said 6 Indian languages is carried out in the following manner: Convert Indic words in training data to WX for readability. WX is a transliteration scheme for representing Indian languages in ASCII. In WX every consonant and every vowel has a single mapping into Roman, that means there is no loss of information while conversion. 11 / 18

  59. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Back Transliteration of Indic Words Transliteration of Indic words from Roman to the respective native scripts Learn a classification model that can predict a phonetically equivalent letter sequence from target script for each letter sequence in a source script. Transliteration of the said 6 Indian languages is carried out in the following manner: Convert Indic words in training data to WX for readability. WX is a transliteration scheme for representing Indian languages in ASCII. In WX every consonant and every vowel has a single mapping into Roman, that means there is no loss of information while conversion. 11 / 18

  60. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Back Transliteration of Indic Words Transliteration of Indic words from Roman to the respective native scripts Learn a classification model that can predict a phonetically equivalent letter sequence from target script for each letter sequence in a source script. Transliteration of the said 6 Indian languages is carried out in the following manner: Convert Indic words in training data to WX for readability. WX is a transliteration scheme for representing Indian languages in ASCII. In WX every consonant and every vowel has a single mapping into Roman, that means there is no loss of information while conversion. 11 / 18

  61. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Back Transliteration of Indic Words Transliteration of Indic words from Roman to the respective native scripts Learn a classification model that can predict a phonetically equivalent letter sequence from target script for each letter sequence in a source script. Transliteration of the said 6 Indian languages is carried out in the following manner: Convert Indic words in training data to WX for readability. WX is a transliteration scheme for representing Indian languages in ASCII. In WX every consonant and every vowel has a single mapping into Roman, that means there is no loss of information while conversion. 11 / 18

  62. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Learn a transliteration model using ID3 Decision trees from the transformed training data of each language. The models are character based, mapping each character in Roman script to WX based on their context of previous 3 and next 3 characters. Training data available only for Hindi, Bengali and Gujarati. Use the transliteration model to predict the equivalent of Romanized word in WX. Use Indic converter to convert WX to native script. For Telugu, Tamil and Malayalam, use Hindi WX transliteration model to predict WX forms. Use Indic converter to convert WX to Devanagari. Use Unicode encoding tables of these languages to extract the corresponding letters. Mapping Hindi Hexadecimal encoding to the encoding of other Indian languages is trivial. 12 / 18

  63. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Learn a transliteration model using ID3 Decision trees from the transformed training data of each language. The models are character based, mapping each character in Roman script to WX based on their context of previous 3 and next 3 characters. Training data available only for Hindi, Bengali and Gujarati. Use the transliteration model to predict the equivalent of Romanized word in WX. Use Indic converter to convert WX to native script. For Telugu, Tamil and Malayalam, use Hindi WX transliteration model to predict WX forms. Use Indic converter to convert WX to Devanagari. Use Unicode encoding tables of these languages to extract the corresponding letters. Mapping Hindi Hexadecimal encoding to the encoding of other Indian languages is trivial. 12 / 18

  64. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Learn a transliteration model using ID3 Decision trees from the transformed training data of each language. The models are character based, mapping each character in Roman script to WX based on their context of previous 3 and next 3 characters. Training data available only for Hindi, Bengali and Gujarati. Use the transliteration model to predict the equivalent of Romanized word in WX. Use Indic converter to convert WX to native script. For Telugu, Tamil and Malayalam, use Hindi WX transliteration model to predict WX forms. Use Indic converter to convert WX to Devanagari. Use Unicode encoding tables of these languages to extract the corresponding letters. Mapping Hindi Hexadecimal encoding to the encoding of other Indian languages is trivial. 12 / 18

  65. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Learn a transliteration model using ID3 Decision trees from the transformed training data of each language. The models are character based, mapping each character in Roman script to WX based on their context of previous 3 and next 3 characters. Training data available only for Hindi, Bengali and Gujarati. Use the transliteration model to predict the equivalent of Romanized word in WX. Use Indic converter to convert WX to native script. For Telugu, Tamil and Malayalam, use Hindi WX transliteration model to predict WX forms. Use Indic converter to convert WX to Devanagari. Use Unicode encoding tables of these languages to extract the corresponding letters. Mapping Hindi Hexadecimal encoding to the encoding of other Indian languages is trivial. 12 / 18

  66. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Learn a transliteration model using ID3 Decision trees from the transformed training data of each language. The models are character based, mapping each character in Roman script to WX based on their context of previous 3 and next 3 characters. Training data available only for Hindi, Bengali and Gujarati. Use the transliteration model to predict the equivalent of Romanized word in WX. Use Indic converter to convert WX to native script. For Telugu, Tamil and Malayalam, use Hindi WX transliteration model to predict WX forms. Use Indic converter to convert WX to Devanagari. Use Unicode encoding tables of these languages to extract the corresponding letters. Mapping Hindi Hexadecimal encoding to the encoding of other Indian languages is trivial. 12 / 18

  67. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Learn a transliteration model using ID3 Decision trees from the transformed training data of each language. The models are character based, mapping each character in Roman script to WX based on their context of previous 3 and next 3 characters. Training data available only for Hindi, Bengali and Gujarati. Use the transliteration model to predict the equivalent of Romanized word in WX. Use Indic converter to convert WX to native script. For Telugu, Tamil and Malayalam, use Hindi WX transliteration model to predict WX forms. Use Indic converter to convert WX to Devanagari. Use Unicode encoding tables of these languages to extract the corresponding letters. Mapping Hindi Hexadecimal encoding to the encoding of other Indian languages is trivial. 12 / 18

  68. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Learn a transliteration model using ID3 Decision trees from the transformed training data of each language. The models are character based, mapping each character in Roman script to WX based on their context of previous 3 and next 3 characters. Training data available only for Hindi, Bengali and Gujarati. Use the transliteration model to predict the equivalent of Romanized word in WX. Use Indic converter to convert WX to native script. For Telugu, Tamil and Malayalam, use Hindi WX transliteration model to predict WX forms. Use Indic converter to convert WX to Devanagari. Use Unicode encoding tables of these languages to extract the corresponding letters. Mapping Hindi Hexadecimal encoding to the encoding of other Indian languages is trivial. 12 / 18

  69. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Learn a transliteration model using ID3 Decision trees from the transformed training data of each language. The models are character based, mapping each character in Roman script to WX based on their context of previous 3 and next 3 characters. Training data available only for Hindi, Bengali and Gujarati. Use the transliteration model to predict the equivalent of Romanized word in WX. Use Indic converter to convert WX to native script. For Telugu, Tamil and Malayalam, use Hindi WX transliteration model to predict WX forms. Use Indic converter to convert WX to Devanagari. Use Unicode encoding tables of these languages to extract the corresponding letters. Mapping Hindi Hexadecimal encoding to the encoding of other Indian languages is trivial. 12 / 18

  70. Outline Description Introduction Data Query Word Labeling Methodology Hindi Song Lyrics Retrieval Results Language Pair BengaliEnglish GujaratiEnglish HindiEnglish KannadaEnglish MalayalamEnglish TamilEnglish LP 0.835 0.986 0.83 0.939 0.895 0.983 LR 0.83 0.868 0.749 0.926 0.963 0.987 LF 0.833 0.923 0.787 0.932 0.928 0.985 EP 0.819 0.078 0.718 0.804 0.796 0.991 ER 0.907 1 0.887 0.911 0.934 0.98 EF 0.861 0.145 0.794 0.854 0.86 0.986 TP 0.011 0.28 0.074 0 0.095 0 TR 0.181 0.243 0.357 0 0.102 0 TF 0.021 0.261 0.122 0 0.098 0 LA 0.85 0.856 0.792 0.9 0.891 0.986 EQMF All(NT) 0.383 0.387 0.143 0.429 0.383 0.714 EQMF − NE(NT) 0.479 0.413 0.255 0.555 0.525 0.714 EQMF − Mix(NT) 0.383 0.387 0.143 0.437 0.492 0.714 EQMF − Mix and NE(NT) 0.479 0.413 0.255 0.563 0.675 0.714 EQMF All 0.004 0.007 0.001 0 0.008 0 EQMF − NE 0.004 0.007 0.001 0 0.008 0 EQMF − Mix 0.004 0.007 0.001 0 0.008 0 EQMF − Mix and NE 0.004 0.007 0.001 0 0.008 0 ETPM 72/288 259/911 907/2004 0/751 90/852 0/0 Table : Subtask-I: Token Level Results 3 3 LP, LR, LF: Token level precision, recall and F-measure for the Indian language in the language pair. EP, ER, EF: Token level precision, recall and F-measure for English tokens. TP, TR, TF: Token level transliteration precision, recall, and F-measure. LA: Token level language labeling accuracy. EQMF: Exact query match fraction. − : without transliteration. ETPM: Exact transliterated pair match 13 / 18

  71. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Description Hindi Song Lyrics Retrieval - A Information Retrial plus linguistic phenomenon - also prominent among multi-lingual specific Indian speaker - switch back and forth between language scripts - rise due to increase in multi script same language content Shared Task - Multi-script Ad hoc retrieval for Hindi Song Lyrics Why? - ∵ To improve retrieval and relevance of IR systems ∵ To increase search space 14 / 18

  72. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Description Hindi Song Lyrics Retrieval - A Information Retrial plus linguistic phenomenon - also prominent among multi-lingual specific Indian speaker - switch back and forth between language scripts - rise due to increase in multi script same language content Shared Task - Multi-script Ad hoc retrieval for Hindi Song Lyrics Why? - ∵ To improve retrieval and relevance of IR systems ∵ To increase search space 14 / 18

  73. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Description Hindi Song Lyrics Retrieval - A Information Retrial plus linguistic phenomenon - also prominent among multi-lingual specific Indian speaker - switch back and forth between language scripts - rise due to increase in multi script same language content Shared Task - Multi-script Ad hoc retrieval for Hindi Song Lyrics Why? - ∵ To improve retrieval and relevance of IR systems ∵ To increase search space 14 / 18

  74. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Description Hindi Song Lyrics Retrieval - A Information Retrial plus linguistic phenomenon - also prominent among multi-lingual specific Indian speaker - switch back and forth between language scripts - rise due to increase in multi script same language content Shared Task - Multi-script Ad hoc retrieval for Hindi Song Lyrics Why? - ∵ To improve retrieval and relevance of IR systems ∵ To increase search space 14 / 18

  75. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Description Hindi Song Lyrics Retrieval - A Information Retrial plus linguistic phenomenon - also prominent among multi-lingual specific Indian speaker - switch back and forth between language scripts - rise due to increase in multi script same language content Shared Task - Multi-script Ad hoc retrieval for Hindi Song Lyrics Why? - ∵ To improve retrieval and relevance of IR systems ∵ To increase search space 14 / 18

  76. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Description Hindi Song Lyrics Retrieval - A Information Retrial plus linguistic phenomenon - also prominent among multi-lingual specific Indian speaker - switch back and forth between language scripts - rise due to increase in multi script same language content Shared Task - Multi-script Ad hoc retrieval for Hindi Song Lyrics Why? - ∵ To improve retrieval and relevance of IR systems ∵ To increase search space 14 / 18

  77. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Description Hindi Song Lyrics Retrieval - A Information Retrial plus linguistic phenomenon - also prominent among multi-lingual specific Indian speaker - switch back and forth between language scripts - rise due to increase in multi script same language content Shared Task - Multi-script Ad hoc retrieval for Hindi Song Lyrics Why? - ∵ To improve retrieval and relevance of IR systems ∵ To increase search space 14 / 18

  78. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Description Hindi Song Lyrics Retrieval - A Information Retrial plus linguistic phenomenon - also prominent among multi-lingual specific Indian speaker - switch back and forth between language scripts - rise due to increase in multi script same language content Shared Task - Multi-script Ad hoc retrieval for Hindi Song Lyrics Why? - ∵ To improve retrieval and relevance of IR systems ∵ To increase search space 14 / 18

  79. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Data and Data Normalization Documents (?60000) contain lyrics both in Devanagari and Roman scripts Data Normalization - - ∵ Cleaning of unwanted content and specific word handling (i.e. jahaa.N, jahaan,mann , D, etc.) ∵ Converted all document in uniform Roman script 15 / 18

  80. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Data and Data Normalization Documents (?60000) contain lyrics both in Devanagari and Roman scripts Data Normalization - - ∵ Cleaning of unwanted content and specific word handling (i.e. jahaa.N, jahaan,mann , D, etc.) ∵ Converted all document in uniform Roman script 15 / 18

  81. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Data and Data Normalization Documents (?60000) contain lyrics both in Devanagari and Roman scripts Data Normalization - - ∵ Cleaning of unwanted content and specific word handling (i.e. jahaa.N, jahaan,mann , D, etc.) ∵ Converted all document in uniform Roman script 15 / 18

  82. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Data and Data Normalization Documents (?60000) contain lyrics both in Devanagari and Roman scripts Data Normalization - - ∵ Cleaning of unwanted content and specific word handling (i.e. jahaa.N, jahaan,mann , D, etc.) ∵ Converted all document in uniform Roman script 15 / 18

  83. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Posting list and Relevancy Build index from the scratch on unified roman scripted song data Use conventional TF-IDF metric Parse song lyric document for relevancy measure Title of the song ¿ First line of song ¿ First line of stanzas ¿ Each line of chorus ¿ etc. 16 / 18

  84. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Posting list and Relevancy Build index from the scratch on unified roman scripted song data Use conventional TF-IDF metric Parse song lyric document for relevancy measure Title of the song ¿ First line of song ¿ First line of stanzas ¿ Each line of chorus ¿ etc. 16 / 18

  85. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Posting list and Relevancy Build index from the scratch on unified roman scripted song data Use conventional TF-IDF metric Parse song lyric document for relevancy measure Title of the song ¿ First line of song ¿ First line of stanzas ¿ Each line of chorus ¿ etc. 16 / 18

  86. Outline Description Introduction Data Methodology Query Word Labeling Hindi Song Lyrics Retrieval Results Posting list and Relevancy Build index from the scratch on unified roman scripted song data Use conventional TF-IDF metric Parse song lyric document for relevancy measure Title of the song ¿ First line of song ¿ First line of stanzas ¿ Each line of chorus ¿ etc. 16 / 18

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

LAW-MWE-CxG 2018 Shared task poster boosters 1. DEEP-BGT AT PARSEME SHARED TASK 2018:

LAW-MWE-CxG 2018 Shared task poster boosters 1. DEEP-BGT AT PARSEME SHARED TASK 2018:

LAW-MWE-CxG 2018 Shared task poster boosters 1. DEEP-BGT AT PARSEME SHARED TASK 2018: BIDIRECTIONAL LSTM- CRF MODEL FOR VERBAL MULTIWORD EXPRESSION IDENTIFICATION, Gzde Berk, Berna Erden and Tunga Gungor 2. GBD-NER AT PARSEME SHARED TASK

555 views • 14 slides
WMT 10 Shared Tasks: Translation Task System Combination Task Chris Callison-Burch, Philipp

WMT 10 Shared Tasks: Translation Task System Combination Task Chris Callison-Burch, Philipp

WMT 10 Shared Tasks: Translation Task System Combination Task Chris Callison-Burch, Philipp Koehn, Christof Monz, Omar Zaidan 15 July 2010 Philipp Koehn WMT10 Shared Tasks 15 July 2010 Translation Task 1 Open benchmark for machine

662 views • 37 slides
TUPA at MRP 2019 A Multi-Task Baseline System CoNLL Shared Task 3 November 2019 1 / 9 Daniel

TUPA at MRP 2019 A Multi-Task Baseline System CoNLL Shared Task 3 November 2019 1 / 9 Daniel

TUPA at MRP 2019 A Multi-Task Baseline System CoNLL Shared Task 3 November 2019 1 / 9 Daniel Hershcovich 1 , Ofjr Arviv 2 1 University of Copenhagen 2 Hebrew University of Jerusalem Transition-Based UCCA Parser look (Hershcovich et al., 2017)

405 views • 12 slides
The SDL Language Weaver Systems in the WMT12 QE Shared Task Team : Radu Soricut, Nguyen Bach,

The SDL Language Weaver Systems in the WMT12 QE Shared Task Team : Radu Soricut, Nguyen Bach,

The SDL Language Weaver Systems in the WMT12 QE Shared Task Team : Radu Soricut, Nguyen Bach, Ziyuan Wang System 1 : M5P-based QE system with 15FFs, directly optimized for DeltaAvg System 2 : SVR-based QE system with 20FFs, manual FF selection

148 views • 11 slides
Shared Governance Task Force Report https://web.ramapo.edu/shared-governance-task-force/ 1

Shared Governance Task Force Report https://web.ramapo.edu/shared-governance-task-force/ 1

4/4/2018 Ramapo College of New Jersey Shared Governance Task Force Report https://web.ramapo.edu/shared-governance-task-force/ 1 Overview of Preliminary Report How Did We Get Here? The Charge to the Shared Governance Task Force (SGTF)

228 views • 9 slides
Query Word Labeling and Back Transliteration for Indian Languages: MSRI Shared task system

Query Word Labeling and Back Transliteration for Indian Languages: MSRI Shared task system

Query Word Labeling and Back Transliteration for Indian Languages: MSRI Shared task system description Spandana Gella 1 , 2 , Jatin Sharma 1 , Kalika Bali 1 1 Microsoft Research, India 2 University of Melbourne, Australia December 4, 2013

355 views • 13 slides
Query Word Labeling and Transliteration for Indian Languages: IITP TS Shared Task system

Query Word Labeling and Transliteration for Indian Languages: IITP TS Shared Task system

Query Word Labeling and Transliteration for Indian Languages: IITP TS Shared Task system description Shubham Kumar Deepak Kumar Gupta Dr. Asif Ekbal Department of Computer Science & Engineering Indian Institute of Technology Patna 1st

696 views • 30 slides
Elites Tweet? Characterizing Verified Twitter Users Indraneil Paul (IIIT Hyderabad), Abhinav

Elites Tweet? Characterizing Verified Twitter Users Indraneil Paul (IIIT Hyderabad), Abhinav

Elites Tweet? Characterizing Verified Twitter Users Indraneil Paul (IIIT Hyderabad), Abhinav Khattar (IIIT Delhi), Shaan Chopra (IIIT Delhi), Ponnurangam Kumaraguru (IIIT Delhi), Manish Gupta (Microsoft India) Outline A: PROBLEM AND MOTIVATION

525 views • 33 slides
The PARSEME Shared Task on Automatic Identification of Verbal Multiword Expressions (1.1) C.

The PARSEME Shared Task on Automatic Identification of Verbal Multiword Expressions (1.1) C.

Context Annotation methodology Corpora Shared task Wrapping up The PARSEME Shared Task on Automatic Identification of Verbal Multiword Expressions (1.1) C. Ramisch 1 , S. Ricardo 1 , A. Savary 2 , V. Vincze 3 , V. Barbu 4 , A. Bhatia 5 , M.

1.02k views • 25 slides
The SIGMORPHON 2016 shared task morphological reinflection Ryan Cotterell, Christo Kirov,

The SIGMORPHON 2016 shared task morphological reinflection Ryan Cotterell, Christo Kirov,

The SIGMORPHON 2016 shared task morphological reinflection Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, Mans Hulden 1 SIGMORPHON 2016 Shared task - morphological reinflection Shared task

1.28k views • 73 slides
HAU at the GermEval 2019 Shared Task on the Identification of Offensive Language in Microposts

HAU at the GermEval 2019 Shared Task on the Identification of Offensive Language in Microposts

HAU at the GermEval 2019 Shared Task on the Identification of Offensive Language in Microposts System Description of Word List, Statistical and Hybrid Approaches afer 1 , Tom De Smedt 2 , and Sylvia Jaki 3 Johannes Sch 1 Institute for

493 views • 28 slides
What sets Verified Users apart? Insights, Analysis and Prediction of Verified Users on Twitter

What sets Verified Users apart? Insights, Analysis and Prediction of Verified Users on Twitter

What sets Verified Users apart? Insights, Analysis and Prediction of Verified Users on Twitter Indraneil Paul (IIIT Hyderabad), Abhinav Khattar (IIIT Delhi), Shaan Chopra (IIIT Delhi), Ponnurangam Kumaraguru (IIIT Delhi), Manish Gupta

731 views • 39 slides
Modeling Unrestricted Coreference in OntoNotes CoNLL-2011 Shared Task Sameer S Pradhan 1 Lance

Modeling Unrestricted Coreference in OntoNotes CoNLL-2011 Shared Task Sameer S Pradhan 1 Lance

CoNLL Shared Task OntoNotes Evaluation Modeling Unrestricted Coreference in OntoNotes CoNLL-2011 Shared Task Sameer S Pradhan 1 Lance Ramshaw 1 Mitchell Marcus 2 Martha Palmer 3 Ralph Weischedel 1 Nianwen Xue 4 1 BBN Technologies, Cambridge, MA

1.17k views • 70 slides
Medicare Shared Savings Program: Application Submission Review For Initial Applicants June 13,

Medicare Shared Savings Program: Application Submission Review For Initial Applicants June 13,

Medicare Shared Savings Program: Application Submission Review For Initial Applicants June 13, 2017 Medicare Shared Savings Program DISCLAIMER This presentation was current at the time it was published or uploaded onto the web. Medicare

914 views • 72 slides
Results of the WMT19 Metrics Shared Task Segment-Level and Strong MT Systems Pose Big Challenges

Results of the WMT19 Metrics Shared Task Segment-Level and Strong MT Systems Pose Big Challenges

Results of the WMT19 Metrics Shared Task Segment-Level and Strong MT Systems Pose Big Challenges Qingsong Ma Johnny Tian-Zheng Wei Ond rej Bojar Yvette Graham 1 / 31 Overview Overview of Metrics Task Updates to Metric Task in

554 views • 40 slides
Medicare Shared Savings Program Accountable Care Organizations: Application Submission Review for

Medicare Shared Savings Program Accountable Care Organizations: Application Submission Review for

Medicare Shared Savings Program Accountable Care Organizations: Application Submission Review for ACOs not Currently Participating in the Program (Initial Applicants) Presented by: Centers for Medicare & Medicaid Services June 7, 2016

819 views • 56 slides
Generating Computer Music from Skeletal Notation for Carnatic Music Compositions (M.

Generating Computer Music from Skeletal Notation for Carnatic Music Compositions (M.

2nd Comp-Music Workshop, Istanbul, 12-13 July, 2012 Generating Computer Music from Skeletal Notation for Carnatic Music Compositions (M. Subramanian) manianms@yahoo.com (Click here for a Web based presentation with Audio links) Genererating

331 views • 21 slides
Agenda Announcements List comprehension Set Password example 1/14/2013 CompSci101

Agenda Announcements List comprehension Set Password example 1/14/2013 CompSci101

Agenda Announcements List comprehension Set Password example 1/14/2013 CompSci101 Peter Lorensen 1 List comprehension [ element for element in sequence if element ] [ [ ] x for loop if pay = [850.0, 1250.0, 900.0, 1400.0,

453 views • 10 slides
Java Fundamentals Marco Piccioni Course Goals Teach Java to programmers having previous

Java Fundamentals Marco Piccioni Course Goals Teach Java to programmers having previous

Chair of Software Engineering Languages in Depth Series: Java Programming Prof. Dr. Bertrand Meyer Java Fundamentals Marco Piccioni Course Goals Teach Java to programmers having previous programming knowledge/experience Show

463 views • 35 slides
Kinetic Typography 2 Background CHI2015:TextAlive CHI2015:TextAlive Kinetic Typography A

Kinetic Typography 2 Background CHI2015:TextAlive CHI2015:TextAlive Kinetic Typography A

CHI2015:TextAlive TextAlive : Integrated Design Environment for Kinetic Typography Jun Kato, Tomoyasu Nakano, Masataka Goto AIST, Japan 1 Background CHI2015:TextAlive CHI2015:TextAlive Kinetic Typography 2 Background CHI2015:TextAlive

734 views • 29 slides
Approximate Joins for Data-Centric XML Nikolaus Augsten 1 ohlen 1 Curtis Dyreson 2 Johann Gamper 1

Approximate Joins for Data-Centric XML Nikolaus Augsten 1 ohlen 1 Curtis Dyreson 2 Johann Gamper 1

Approximate Joins for Data-Centric XML Nikolaus Augsten 1 ohlen 1 Curtis Dyreson 2 Johann Gamper 1 Michael B 1 Free University of Bozen-Bolzano Bolzano, Italy { augsten,boehlen,gamper } @inf.unibz.it 2 Utah State University Logan, UT, U.S.A.

1.19k views • 105 slides
Clova Music: DJ AI (Adrian Kim), M.S. Clova AI

Clova Music: DJ AI (Adrian Kim), M.S. Clova AI

Clova Music: DJ AI (Adrian Kim), M.S. Clova AI Research(CLAIR), Naver Corp. Clova: Cloud-based Virtual Assistant General Purpose AI platform Clova: Cloud-based Virtual Assistant https://clova.ai Clova:

808 views • 78 slides
Jones, J. (2017, February 1). Scenery, machinery, people Rethinking our view of humans. The

Jones, J. (2017, February 1). Scenery, machinery, people Rethinking our view of humans. The

Jones, J. (2017, February 1). Scenery, machinery, people Rethinking our view of humans. The Culture Blend . Retrieved from http://www.thecultureblend.com/scenery-machinery-people-rethinking-our-view-of-humans/ SCENERY encompassed everything

220 views • 4 slides
Single mask technology implementation Piotr Bielwka 10 th RD51 Stony Brook Single mask

Single mask technology implementation Piotr Bielwka 10 th RD51 Stony Brook Single mask

Single mask technology implementation Piotr Bielwka 10 th RD51 Stony Brook Single mask technology implementation a) New machinery: Old Kapton etching machine New Kapton etching machine 10 th RD51 Stony Brook Single mask technology

239 views • 6 slides

More recommend