ICT-Crossn: The System of Cross- lingual Information Retrieval of - - PowerPoint PPT Presentation

▶
ict crossn the system of cross lingual information
SMART_READER_LITE
LIVE PREVIEW

ICT-Crossn: The System of Cross- lingual Information Retrieval of - - PowerPoint PPT Presentation

Jul 29, 2023 145 likes •624 views

ICT-Crossn: The System of Cross- lingual Information Retrieval of ICT in NTCIR-7 INSTI INSTITUT Weihua Luo, Tian Xia, Ji Guo, Qun Liu UTE E OF COMPUTING T COMPUTING TECH Multilingual Interaction Technology Laboratory (MITEL) Institute of

slide-1
SLIDE 1

INSTI INSTITUT UTE E OF COMPUTING T COMPUTING TECH CHNOLOGY NOLOGY

ICT-Crossn: The System of Cross- lingual Information Retrieval of ICT in NTCIR-7

Weihua Luo, Tian Xia, Ji Guo, Qun Liu Multilingual Interaction Technology Laboratory (MITEL) Institute of Computing Technology, Chinese Academy of Sciences presented in NTCIR-7 2008-12-18

slide-2
SLIDE 2

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 2

INSTITUTE OF COMPUTING TECHNOLOGY

Outline

Background The system architecture Query translation Document retrieval Experiments on the dry-run set Results on the formal run Conclusion

slide-3
SLIDE 3

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 3

INSTITUTE OF COMPUTING TECHNOLOGY

Outline

Background The system architecture Query translation Document retrieval Experiments on the dry-run set Results on the formal run Conclusion

slide-4
SLIDE 4

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 4

INSTITUTE OF COMPUTING TECHNOLOGY

Background

Tasks in NTCIR-7

  • Advanced Cross-lingual Information Access (ACLIA)

Information Retrieval for Question Answering (IR4QA)

  • A novel task in ACLIA task cluster in NTCIR-7
  • Motivation
  • CLIR has made a great progress
  • Find out which IR technique would help CCLQA
  • Subtasks
  • CS-CS
  • EN-CS √
  • CT-CT √
  • EN-CT
  • JA-JA
  • EN-JA
slide-5
SLIDE 5

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 5

INSTITUTE OF COMPUTING TECHNOLOGY

Difference between CLIR and IR4QA

Query

<TOPIC> <NUM>001</NUM> <SLANG>CH</SLANG> <TLANG>EN</TLANG> <TITLE>Time Warner, American Online (AOL), Merger, Impact</TITLE> <DESC>Find reports about the impact of AOL/Time Warner merger.</DESC> <NARR> <BACK>Time Warner and American Online (AOL) announced a merger on January 10th, 2000. The market value was estimated at $US350 billion making it the biggest merger in the US.</BACK> <REL>Comments on AOL/Time Warner merger's effects on Internet and entertainment media businesses are relevant. Descriptions of the development of the AOL/Time Warner merger are partially relevant. Information about the total amount and the transformation of ownership structure are irrelevant.</REL> </NARR> <CONC>Time Warner, American Online, AOL, Gerald Levin, merger, M&A, Merger and Acquisition, media, entertainment business</CONC> </TOPIC>

CLIR

<TOPIC ID="ACLIA1-CS-T41"> <QUESTION LANG="EN"> <![CDATA[ List the hazards of global warming.]]> </QUESTION> <QUESTION LANG="CS"> <![CDATA[ 列举全球气候变暖的危害。]]> </QUESTION> <NARRATIVE LANG="EN"> <![CDATA[ Users need to know the harm of global warming to human beings and the environment.]]> </NARRATIVE> <NARRATIVE LANG="CS"> <![CDATA[ 用户需要知道全球气候变暖对人类和环境有什么危害。]]> </NARRATIVE> </TOPIC>

IR4QA

slide-6
SLIDE 6

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 6

INSTITUTE OF COMPUTING TECHNOLOGY

Difference between CLIR and IR4QA

Query

  • CLIR

Monolingual description of Information need (IN) Different grained description of IN

  • TITLE DESC NARR

Accurate keywords Detailed background and relevance judgement

  • IR4QA

Bilingual description of IN One grained description of IN

  • QUESTION ≈ NARRATIVE

No keywords available No background and relevance judgement in detail Question type provided with topics

slide-7
SLIDE 7

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 7

INSTITUTE OF COMPUTING TECHNOLOGY

Difference between CLIR and IR4QA

Metrics of evaluation

CLIR

Traditional IR metrics: AP

IR4QA

IR evaluation

  • AP--primary
  • Q
  • nDCG

End-to-end evaluation with a QA system

  • F-score
slide-8
SLIDE 8

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 8

INSTITUTE OF COMPUTING TECHNOLOGY

slide-9
SLIDE 9

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 9

INSTITUTE OF COMPUTING TECHNOLOGY

Outline

Background The system architecture Query translation Document retrieval Experiments on the dry-run set Results on the formal run Conclusion

slide-10
SLIDE 10

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 10

INSTITUTE OF COMPUTING TECHNOLOGY

Our Implementation

Basic idea

A traditional CLIR framework still works

IR4QA is still an IR task

Major concerned issues

Shorter query

  • Improper translation of a few words may lead to retrieval of

more irrelevant documents

Different evaluation metrics (reference set)

  • The judgement of document relevance is vague for a new

task

  • Which IR model is more suitable?

Modification should be made!

slide-11
SLIDE 11

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 11

INSTITUTE OF COMPUTING TECHNOLOGY

Traditional CLIR Flowchart

EN Query Query Translation CS Query Formal Representation CS Document Formal Representation Relevance Estimation Result Reranking document list

Dictionary- based, corpus- based, MT module based… VSM model, Probabilistic model,LM model…

slide-12
SLIDE 12

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 12

INSTITUTE OF COMPUTING TECHNOLOGY

Our Improvement for IR4QA

EN Query Query Translation CS Query Formal Representation CS Document Formal Representation Relevance Estimation Result Reranking document list

use of intermed iate data

  • f SMT

Borrow Idea of system combination

  • f SMT
slide-13
SLIDE 13

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 13

INSTITUTE OF COMPUTING TECHNOLOGY

ICT-Crossn

slide-14
SLIDE 14

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 14

INSTITUTE OF COMPUTING TECHNOLOGY

Outline

Background The system architecture Query translation Document retrieval Experiments on the dry-run set Results on the formal run Conclusion

slide-15
SLIDE 15

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 15

INSTITUTE OF COMPUTING TECHNOLOGY

Query Translation

Basic ideas

Phrase translation

Full text translation is unfit for short questions Phrases resolve some ambiguities Phrase based SMT creates huge phrase tables

OOV words translation

Many OOV words provide key information of questions Any dictionary has a limited coverage Corresponding translation may be found with search

engines

slide-16
SLIDE 16

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 16

INSTITUTE OF COMPUTING TECHNOLOGY

Phrase Translation

Phrase table

Trained on 5M pairs of EN-CS sentence Word alignment with Giza++ Phrase extraction and probabilities estimation with

a tool in Mencius (our phrase-based SMT decoder)(F.J. Och, Hermann Ney, 2008)

Size up to 17GB The phrase table is filtered by the test set

slide-17
SLIDE 17

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 17

INSTITUTE OF COMPUTING TECHNOLOGY

Sample of CS-EN Bilingual Phrase Table

! 三 井 住友 ! Sumitomo Mitsui 0:0 1:2 2:2 3:1 3:2 1 0.018866 1 0.120792 2.718 1 ! 三 井 住友 金融 ! Sumitomo Mitsui Financial 0:0 1:2 2:2 3:1 3:2 4:3 1 0.0129453 1 0.0073817 2.718 1 ! 三 井 住友 金融 集团 ! Sumitomo Mitsui Financial Group 0:0 1:2 2:2 3:1 3:2 4:3 5:4 1 0.00932347 1 0.00318336 2.718 1 " , 以 对付 " to deal with the 0:0 2:2 3:2 3:3 1 0.000132428 0.5 0.000141711 2.718 1 " , 以 对付 " to deal with 0:0 2:2 3:2 3:3 1 0.000132428 0.5 0.000511601 2.718 1 " , 以 协调 to coordinate its own 2:0 3:1 3:2 3:3 0.333333 8.20519e-06 1 1.46173e-06 2.718 1 " , 以便 使 " so that the 0:0 2:1 2:2 3:1 0.333333 6.48644e-05 0.5 0.000702031 2.718 1 " , 以便 使 " so that 0:0 2:1 2:2 3:1 0.333333 6.48644e-05 0.5 0.00253444 2.718 1 广播 电视 Broadcast 0:0 1:0 0.444444 0.105186 0.166667 0.024001 2.718 1 广播 电视 broadcast and tv 0:0 1:2 1 0.106304 0.0833333 0.000460165 2.718 1 广播 电视 broadcasting 0:0 1:0 0.142857 0.0856858 0.166667 0.0846071 2.718 1 广播 电视台 TV station 0:0 1:0 1:1 0.0754717 0.00252064 0.5 0.0257354 2.718 1 广播 电视台 feeding radio and television 0:0 0:1 1:3 1 0.0685612 0.5 2.82344e-05 2.718 1 国际 发展 局 International Development Bureau 0:0 1:1 2:2 1 0.280224 1 0.00156186 2.718 1 国际 发展 局 援助 International Development Bureau Assistance 0:0 1:1 2:2 3:3 1 0.0941678 1 2.42523e-05 2.718 1 国际 发展 局 援助 局 International Development Bureau Assistance Department 0:0 1:1 2:2 3:3 4:4 1 0.00106589 1 1.82349e-07 2.718 1 国际 发展署 代表团 Delegation Visits 0:0 1:0 1:1 2:0 0.117647 0.000173761 1 0.091007 2.718 1 国际 法 international law 0:0 1:1 0.2 0.19695 0.75 0.0735244 2.718 1 国际 法 the international law 0:1 1:2 0.5 0.19695 0.25 0.020366 2.718 1 国际 法庭 International Court of Justice , 0:0 1:1 1:3 1 0.111924 0.0238096 8.55327e-06 2.718 1 国际 法庭 International Court of Justice 0:0 1:1 1:3 0.6 0.111924 0.0714286 5.25604e-05 2.718 1 国际 反 恐 international anti terrorism 0:0 1:1 2:2 1 0.0935168 0.166667 0.0128866 2.718 1 国际 反 恐 international counter terrorism 0:0 1:1 2:2 1 0.0077033 0.166667 0.000422512 2.718 1 国际 反 恐 international terrorism 0:0 1:1 2:1 0.0384615 0.0118484 0.25 0.069292 2.718 1 外事

  • f the Foreign Affairs

0:2 0:3 0.75 0.0421837 0.0769231 0.00190038 2.718 1 外事 the Foreign Affairs 0:1 0:2 0.818182 0.0421837 0.115385 0.0142804 2.718 1 外事 the foreign affairs 0:1 0:2 0.214286 0.00529871 0.0384615 0.0015063 2.718 1 外事 人才 personnel of foreign affairs 0:2 0:3 1:0 1 0.000712337 1 9.76949e-05 2.718 1 外事 委员会 Affairs Committee 0:0 1:1 0.4 0.042495 0.0833333 0.0592912 2.718 1 外事 委员会 Foreign Affairs Committee , 0:0 0:1 1:2 0.8 0.0268476 0.0833333 0.00196545 2.718 1 外事 委员会 Foreign Affairs Committee of the 0:0 0:1 1:2 0.666667 0.0268476 0.0185185 0.000445208 2.718 1 外事 委员会 Foreign Affairs Committee of 0:0 0:1 1:2 0.777778 0.0268476 0.0324074 0.00160727 2.718 1 外事 委员会 Foreign Affairs Committee 0:0 0:1 1:2 0.763285 0.0268476 0.365741 0.0120779 2.718 1

slide-18
SLIDE 18

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 18

INSTITUTE OF COMPUTING TECHNOLOGY

What is human genetic sequence determination?

“human genetic sequence determination” Algorithm of phrase identification and translation

Template based phrase extraction words ≤3 Search for translation In the phrase table Not found found OOV words translation Output top N translations by p(c|e)

Human, genetic sequence, determination, Human genetic, sequence, determination, Human genetic sequence, determination… Human genetic sequence, determination

Sub-phrases enumeration Choose the combination minimizing the joining times

人类基因组序列 测定，确定，决心 人类基因组序列测定 人类基因组序列确定…

Output top N translations by p(c|e) Translation assemble

slide-19
SLIDE 19

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 19

INSTITUTE OF COMPUTING TECHNOLOGY

Phrase translation

Additional comments

The particle phrase/word is removed The phrase occurring in many questions is removed A word sequence with the first letter of its each word

capitalized is viewed as a whole phrase

Balkan Syndrome Edmund Ho Hau Wah Parade of The Century …

slide-20
SLIDE 20

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 20

INSTITUTE OF COMPUTING TECHNOLOGY

OOV words translation

OOV words

Non-exists in the phrase table Retrieve candidate translation with search engine

(SE) (Ying Zhang, Phil Vines, 2004)

Rank candidates with a linear function

Parenthesis Similarity of transliteration co-occurrence probability

Weights are tuned in a greedy way

( , ) ( , )

i i i

score c e f c e ω = ∑

slide-21
SLIDE 21

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 21

INSTITUTE OF COMPUTING TECHNOLOGY

English query construction Search documents with a Chinese SE Select top N snippets Duplication deletion & word segmenation Extract candidate translation in a window around key words Rank candidates by score(c,e)

slide-22
SLIDE 22

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 22

INSTITUTE OF COMPUTING TECHNOLOGY

Outline

Background The system architecture Query translation Document retrieval Experiments on the dry-run set Results on the formal run Conclusion

slide-23
SLIDE 23

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 23

INSTITUTE OF COMPUTING TECHNOLOGY

Document Retrieval

Basic ideas

Different models have different performance on

different task

System combination in proper way improves over

a single model (L.X.Shi, J.Y.Nie, 2007)

Various models are complementary

Lemur toolkit

slide-24
SLIDE 24

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 24

INSTITUTE OF COMPUTING TECHNOLOGY

Document Retrieval

Index models? Retrieval models?

Index models Various models with minor difference is easy to

combine

Index models have less parameters to be tuned A linear function and the objective function

slide-25
SLIDE 25

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 25

INSTITUTE OF COMPUTING TECHNOLOGY

IR model selection

The results of different IR models and index units for Chinese monolingual IR on the NTCIR-5 test set

Mean Average Precision T run D run N run IR model Unigram Bigram Word Unigram Bigram Word Unigram Bigram Word tf_idf 0.2497 0.2664 0.2515 0.2164 0.2465 0.2052 0.2716 0.3032 0.2729 fb_tf_idf 0.2724 0.3162 0.2950 0.2569 0.3088 0.2608 0.2844 0.3362 0.2837

  • kapi

0.2609 0.2646 0.2582 0.2065 0.2393 0.1769 0.2835 0.3172 0.2737 fb_okapi 0.2931 0.3005 0.3277 0.1946 0.3130 0.2345 0.1678 0.3390 0.2773 kl_jm 0.2369 0.2366 0.2499 0.2218 0.2151 0.1851 0.2916 0.2970 0.2685 kl_dir 0.3031 0.2615 0.2706 0.2588 0.2270 0.2013 0.3154 0.3019 0.2790 kl_abs 0.3045 0.2519 0.2578 0.2302 0.2217 0.1823 0.3183 0.3013 0.2624 mixfb_kl_dir 0.3443 0.2860 0.3057 0.2933 0.2983 0.2513 0.3343 0.3378 0.2992

slide-26
SLIDE 26

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 26

INSTITUTE OF COMPUTING TECHNOLOGY

Parameter optimaztion

The results of different index units and feedback document number

0.27 0.285 0.3 0.315 0.33 0.345 0.36 2 3 4 5 6 7 8 10 15 20 25 30

feedback document number MAP Unigram Bigram Word

slide-27
SLIDE 27

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 27

INSTITUTE OF COMPUTING TECHNOLOGY

Parameter optimaztion

The results of different index units and feedback term count

0.27 0.285 0.3 0.315 0.33 0.345 0.36 4 6 9 1 2 1 6 2 feedback term count MAP Unigram Bigram Word

slide-28
SLIDE 28

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 28

INSTITUTE OF COMPUTING TECHNOLOGY

Index Unit Combination

Retrieval model

language model, Dirrichlet smoothing, with feedback

Score normalization

divide by the score of top 1 doc[0,1]

Index model combination

Each index model finishes retrieval and feedback

respectively

the system combines their final score

slide-29
SLIDE 29

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 29

INSTITUTE OF COMPUTING TECHNOLOGY

Outline

Background The system architecture Query translation Document retrieval Experiments on the dry-run set Results on the formal run Conclusion

slide-30
SLIDE 30

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 30

INSTITUTE OF COMPUTING TECHNOLOGY

Development set

Construction

Motivation

Evaluation metric of IR4QA ≠ Evaluation metric of CLIR Parameters tuned on previous test sets may well be

unfit for IR4QA

Process

Download topics and answers in the dry run set Convert according to specified XML format Mark documents containing answers as relevant

slide-31
SLIDE 31

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 31

INSTITUTE OF COMPUTING TECHNOLOGY

Experiments on the dry-run set

Development set

Task Question Type Question number Event 14 Relationship 10 Biography 21 EN-CS Definition 33 Event 8 Relationship 10 Biography 24 CT-CT Definition 29

slide-32
SLIDE 32

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 32

INSTITUTE OF COMPUTING TECHNOLOGY

Sample topics in development data set

Topics Relevant documents What is Mayday? za2001_038799 za2001_041818 za2001_040008 za1999_028092 za2001_042120 za2001_003573 za2000_014785 za2001_033102 List events related to Russian submarine Kursk explosion. XIN20010312.0245 XIN20000815.0073 XIN20011009.0072 za2001_000149 What is the International Criminal Court? XIN19980728.0164 XIN19980617.0100 za2000_061113 What is the relationship between Shintaro Ishihara and the literary circle? za1999_010451 za1999_007192 za1999_012952 za1999_007192 XIN19990425.0127 印尼暴動與峇里島觀光有何關連？ (What is the relationship between riots in Indonesia and tourism in Bali?) udn_xxx_19980911_0015 udn_xxx_19980517_0264 udn_xxx_19980521_0269 udn_xxx_19990512_0009 udn_xxx_19981012_0046 禽流感是在說什麼呢? (What is Avian influenza?) udn_xxx_20010926_1110898 mhn_xxx_20010901_1073944 mhn_xxx_20010926_1110315 edn_xxx_20011203_1214182 誰是連戰？ (Who is Lien Chan?) udn_xxx_20010521_0913048 udn_xxx_19990322_0237 udn_xxx_19990902_0408 udn_xxx_20000309_0049710 udn_xxx_19990507_0272 神舟號太空船 (List events about Shenzhou spacecrafts.) udn_xxx_19991123_0131 udn_xxx_19991130_0249 udn_xxx_19991124_0194 udn_xxx_20000108_0255269 udn_xxx_19991122_0061 udn_xxx_19991123_0131 udn_xxx_20010111_0719983 udn_xxx_20010118_0730638 udn_xxx_19991122_0147 udn_xxx_20001106_0618907

slide-33
SLIDE 33

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 33

INSTITUTE OF COMPUTING TECHNOLOGY

The Results of Index Models Combination in EN-CS Task

0.185 0.19 0.195 0.2 0.205 0.21 0.215 0.22 . 5 . 1 5 . 2 5 . 3 5 . 4 5 . 5 5 . 6 5 . 7 5 . 8 5 . 9 5

MAP

U+B U+W W+B 0.5U+0.5W+B

slide-34
SLIDE 34

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 34

INSTITUTE OF COMPUTING TECHNOLOGY

The Results of Index Models Combination in CT-CT Task

0.21 0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29 0.3 0.31 0.32 . 5 . 1 5 . 2 5 . 3 5 . 4 5 . 5 5 . 6 5 . 7 5 . 8 5

MAP U+B U+W B+W 0.3U+0.7B+W

slide-35
SLIDE 35

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 35

INSTITUTE OF COMPUTING TECHNOLOGY

Outline

Background The system architecture Query translation Document retrieval Experiments on the dry-run set Results on the formal run Conclusion

slide-36
SLIDE 36

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 36

INSTITUTE OF COMPUTING TECHNOLOGY

Results on the formal run

Our runs

Run ID Description MITEL-EN-CS-01-T 0.5U+0.5W+0.15B, question part is used MITEL-EN-CS-02-T 0.5U+0.5W, question part is used MITEL-EN-CS-03-T 0.45U+0.55B, question part is used MITEL-EN-CS-04-D 0.5U+0.5W+0.15B, narrative part is used MITEL-EN-CS-05-TD 0.5U+0.5W+0.15B, both question and narrative part are used MITEL-CT-CT-01-T 0.3U+0.7B, question part is used MITEL-CT-CT-02-T 0.3U+0.7B+0.05W, question part is used MITEL-CT-CT-03-D 0.3U+0.7B, narrative part is used MITEL-CT-CT-04-T Bigram, question part is used

slide-37
SLIDE 37

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 37

INSTITUTE OF COMPUTING TECHNOLOGY

Performance in CLIR

The performance based on pseudo-qrels in the NTCIR-7 formal result (EN-CS)

0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 AP Q nDCG metric score MITEL-EN-CS-01-T MITEL-EN-CS-02-T MITEL-EN-CS-03-T MITEL-EN-CS-04-D MITEL-EN-CS-05-TD

slide-38
SLIDE 38

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 38

INSTITUTE OF COMPUTING TECHNOLOGY

Performance in CLIR

The performance based on real qrels in the NTCIR-7 formal result (EN-CS)

0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 AP Q nDCG metric score MITEL-EN-CS-01-T MITEL-EN-CS-02-T MITEL-EN-CS-03-T MITEL-EN-CS-04-D MITEL-EN-CS-05- TD

slide-39
SLIDE 39

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 39

INSTITUTE OF COMPUTING TECHNOLOGY

Performance in CLIR

The performance based on pseudo-qrels in the NTCIR-7 formal result (CT-CT)

0.5 0.55 0.6 0.65 0.7 0.75 0.8 AP Q nDCG metric score MITEL-CT-CT-01-T MITEL-CT-CT-02-T MITEL-CT-CT-03-D MITEL-CT-CT-04-T

slide-40
SLIDE 40

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 40

INSTITUTE OF COMPUTING TECHNOLOGY

Performance in CLIR

The performance based on real qrels in the NTCIR-7 formal result (CT-CT)

0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 AP Q nDCG metric score MITEL-CT-CT-01-T MITEL-CT-CT-02-T MITEL-CT-CT-03-D MITEL-CT-CT-04-T

slide-41
SLIDE 41

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 41

INSTITUTE OF COMPUTING TECHNOLOGY

Performance in CCLQA with a QA system

Performance in CCLQA with a QA system

CCLQA IR4QA ATR/NiCT CMUJAV KECIR MITEL 0.2750 0.1982 (0.2168) 0.1829

EN-CS Collaboration Task (F3 scores based on automatic evaluation)

slide-42
SLIDE 42

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 42

INSTITUTE OF COMPUTING TECHNOLOGY

Analysis

Overall recall with top 1000 over 90% (dry run) Precision relatively low Model combination helps recall more Evaluation metric of IR4QA differs from the one of

CLIR

“Who is Hu Jintao?” Bibliography of Hu is more appropriate than a meeting he

attends

Model combination doesn’t change this!

slide-43
SLIDE 43

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 43

INSTITUTE OF COMPUTING TECHNOLOGY

Outline

Background The system architecture Query translation Document retrieval Experiments on the dry-run set Results on the formal run Conclusion

slide-44
SLIDE 44

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 44

INSTITUTE OF COMPUTING TECHNOLOGY

Conclusion

IR4QA is a novel but challenging task ICT-Crossn achieves a good performance

A phrase table of SMT + OOV words translation

based on SE

Index model combination The construction of Development set based on

the dry run set

slide-45
SLIDE 45

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 45

INSTITUTE OF COMPUTING TECHNOLOGY

Conclusion

Problems

identification and translation of phrases “natural gas hydrates” reranking A reranking method for CLIR of I2R doesn’t work in

IR4QA(L.P.Yang, D.H.Ji, 2005)

Future work

Phrases translation High recall & low precision mean… Reranking should work A better objection function for IR4QA should be explored

slide-46
SLIDE 46

12/18/2008 IR4QA_NTCIR7_MITEL_Luo_Xia_Guo_Liu 46

INSTITUTE OF COMPUTING TECHNOLOGY

Information retrieval for question answering: a challenging task but a long way to go

Thank you!

Many thanks to my co-authors Tian Xia, Ji Guo and Qun Liu Many thanks to Dr. Zhongjun He for his SMT system -- Mencius