S3K Seeking Statement-Supporting top-K Witnesses Steffen Metzger*, - PowerPoint PPT Presentation

S3K Seeking Statement-Supporting top-K Witnesses Steffen Metzger*, Shady Elbassuoni*, Katja Hose*, Ralf Schenkel* ⁺ *Max-Planck-Institute for Informatics ⁺ Saarland University CIKM 2011 25 th October 2011

Statements Statements = „factual statements“ • – Obama was born in Hawaii – Obama was born in Kenya – Heath Ledger played a role in The Dark Knight – Maggie Gyllenhaal appeared in The Dark Knight – … „statement“ used interchangeably with „fact“, but no • truthness assumed in general S3K: Seeking Statement-Supporting top-K Witnesses 2/25

Document Search by Statements Heath Ledger played a role in The Dark Knight Witnesses for the Statement(s) Doc A Statement Search Doc Engine B Doc C S3K: Seeking Statement-Supporting top-K Witnesses 3/25

Document Search by Statements Heath Ledger played a role in The Dark Knight Witnesses for the Statement(s) Doc A Statement Search Doc Engine B Doc C S3K: Seeking Statement-Supporting top-K Witnesses 4/25

Statement Expressions One statement, many forms of verbal expression • Ledger's performance in The Dark Knight... ...Ledger's latest movie The Dark Knight... Heath was great in The Dark Knight... ...Ledger playing the Joker in The Dark Knight... S3K: Seeking Statement-Supporting top-K Witnesses 5/25

Statement Expressions One statement, many forms of verbal expression • Did Heath Ledger Ledger's performance in The Dark Knight... play a role in The Dark Knight? ...Ledger's latest movie The Dark Knight... Ledger acted in Heath was great in The Dark Knight... the new Batman movie? ...Ledger playing the Joker in The Dark Knight... Unify into single representation: RDF triples • HeathLedger actedIn TheDarkKnight Subject Relation Object S3K: Seeking Statement-Supporting top-K Witnesses 6/25

Document Search by Statements HeathLedger actedIn TheDarkKnight Doc A Statement Search Doc Engine B Appearance Index: HeathLedger actedIn TheDarkKnight Doc - Doc A But wait! C What is it good for? - Doc B - Doc C S3K: Seeking Statement-Supporting top-K Witnesses 7/25

Information Need: Verification Hey, I heard O bama is born in Kenya ! What?! No, he's from Hawaii ! Isn't that part of Kenya? Obama wasBornIn Kenya Obama wasBornIn Hawaii S3K: Seeking Statement-Supporting top-K Witnesses 8/25

Information Need: Verification Hey, I heard O bama is born in Kenya ! What?! No, he's from Hawaii ! Isn't that part of Kenya? Obama wasBornIn Kenya Obama wasBornIn Hawaii S3K: Seeking Statement-Supporting top-K Witnesses 9/25

Information Need: Context HeathLedger actedIn TheDarkKnight MaggieGyllenhaal actedIn TheDarkKnight What role did Did they have he play? an affair? What's the relation What's his interpretation of their characters? of that role? S3K: Seeking Statement-Supporting top-K Witnesses 10/25

Ranked Witness List Ranked list of source documents Statement Set: Doc MaggieGyllenhal actedIn TheDarkKnight A HeathLedger actedIn TheDarkKnight Doc B Doc C How should we rank? S3K: Seeking Statement-Supporting top-K Witnesses 11/25

Not every witness is equally appropriate S3K: Seeking Statement-Supporting top-K Witnesses 12/25

Not every witness is equally appropriate Clearly in front of the line VS Some witnesses have more Authority than others (witness trust) S3K: Seeking Statement-Supporting top-K Witnesses 13/25

Testimonies are not equally convincing Obama visited his home country Kenya. Barack Obama was born in Honolulu, Hawaii in 1961 Ledger's latest movie The Dark Knight topped the box office ... Ledger playing the Joker in The Dark Knight... S3K: Seeking Statement-Supporting top-K Witnesses 14/25

Testimonies are not equally convincing Obama visited his home country Kenya. Barack Obama was born in Honolulu, Hawaii in 1961 Ledger's latest movie The Dark Knight topped the box office ... Ledger playing the Joker Persuasiveness of testimonies varies in The Dark Knight... (strength of statement support) S3K: Seeking Statement-Supporting top-K Witnesses 15/25

The more the better Obama wasBornIn Hawaii Hawaii isPartOf USA The more statements covered the better (coverage) S3K: Seeking Statement-Supporting top-K Witnesses 16/25

The more the better Obama wasBornIn Hawaii S3K: Seeking Statement-Supporting top-K Witnesses 17/25

The more the better Obama wasBornIn Hawaii The more additional information about the topic (entities involved) the better (on-topicness) S3K: Seeking Statement-Supporting top-K Witnesses 18/25

Criteria for ranking • Coverage – how many of the statements are covered in the document? • Persuasiveness – how convincing are the sources – Authority of the document – Strength of statement support • On-Topicness – is the focus of the document on the topic expressed by the statements? S3K: Seeking Statement-Supporting top-K Witnesses 19/25

Criteria for ranking • Coverage – how many of the statements are covered in the document? 1) How to connect documents • Persuasiveness – how convincing are the with statements? 2) And how can we measure a sources document's persuasiveness? – Authority of the document – Strength of statement support • On-Topicness – is the focus of the document on the topic expressed by the statements? „Easy“: Appearances of the entities occurring in the statement(s) S3K: Seeking Statement-Supporting top-K Witnesses 20/25

Leveraging IE Methods Information Ontology Extraction HeathLedger wasBornIn Pattern p1 : Heath Ledger was born Perth(Australia) X was born in Y in Perth, Australia... HeathLedger actedIn ...Ledger's latest movie Pattern p2 : TheDarkKnight The Dark Knight... X 's latest movie Y Knowledge Extraction Rules, Information Patterns S3K: Seeking Statement-Supporting top-K Witnesses 21/25

Leveraging IE Methods Information Ontology Extraction HeathLedger wasBornIn Pattern p1 : Heath Ledger was born Perth(Australia) X was born in Y in Perth, Australia... HeathLedger actedIn ...Ledger's latest movie Pattern p2 : TheDarkKnight The Dark Knight... X 's latest movie Y Pattern reliability differs → Confidence values p1 expresses wasBornIn : 0.9 p2 expresses actedIn : 0.4 p2 expresses directed : 0.5 S3K: Seeking Statement-Supporting top-K Witnesses 22/25

Witness Ranking: Trust Rank of witness d based on P  isPerfect  d , g  Probability to be a perfect witness of a given set f i = s i ,r i ,o i  g ={ f 1 , ... , f n } of statements with S3K: Seeking Statement-Supporting top-K Witnesses 23/25

Witness Ranking: Trust Rank of witness d based on P  isPerfect  d , g  Probability to be a perfect witness of a given set f i =( s i ,r i ,o i ) g ={ f 1 , ... , f n } of statements with P  isPerfect  d ,g ∝ P  trust  d  P  g ∣ d  P  isPerfect  d ,g ∝ pr  d  P  g ∣ d  trust estimated by pagerank pr(d) n Independence P  g ∣ d = ∏  P  f i ∣ d  1 − P  f i ∣ Col  Collection (Col) i = 1 based Smoothing S3K: Seeking Statement-Supporting top-K Witnesses 24/25

Support and On-Topicness f i =( s i ,r i ,o i ) n P  g ∣ d = ∏  P  f i ∣ d  1 − P  f i ∣ Col  X ∈{ d ,Col } i = 1 P  f i ∣ X = s P e  s i ∣ X  o P e  o i ∣ X  1 − s − o  P f  f i ∣ X  Pattern instance with s i ,o i count  s,d  m P f  f i ∣ X = ∑ P e  s ∣ d = s i ,o i ∣ X  P  r i ∣ p j   P  p j ∑ e ∈ d count  e,d  j = 0 m Pattern P f  f i ∣ X = ∑ s i ,o i ∣ X  conf  p j ,r i   P  p j confidence conf  p j ,r i  j = 0 S3K: Seeking Statement-Supporting top-K Witnesses 25/25

Statement Presence Estimation f i =( s i ,r i ,o i ) P  f i ∣ X = s P e  s i ∣ X  o P e  o i ∣ X  1 − s − o  P f  f i ∣ X  m X ∈{ d ,Col } P f  f i ∣ X = ∑ s i ,o i ∣ X  conf  p j ,r i  P  p j j = 0 s i ,o i , X  s i ,o i ∣ X = count  p j P  p j Pattern Importance ∑ p count  p, X  Estimation S3K: Seeking Statement-Supporting top-K Witnesses 26/25

Evaluation Documents: ~180.000 from ClueWeb09 ⁺ • IE Parser: Modified SOFIE/PROSPERA system • Evaluation 1: • – Comparing different ranking approaches on prefiltered witness set • naive : only filtering • topic : mainly enity occurrences • persuade : aiming for persuasiveness • mix : balancing both aims • lucene : keyword search by entities on filtered subset ⁺ http://lemurproject.org/clueweb09.php/ S3K: Seeking Statement-Supporting top-K Witnesses 27/25

Evaluation Evaluation 2: • – Comparing extraction based methods against plain keyword search on full document corpus • unfluc: e e – based on entities • unfluc: e r e – based on entities & manual relation formulation • unfluc: e p e – based on entities & best pattern for relation (highest confidence) S3K: Seeking Statement-Supporting top-K Witnesses 28/25