S3K Seeking Statement-Supporting top-K Witnesses Steffen Metzger*, - - PowerPoint PPT Presentation

▶
s3k
SMART_READER_LITE
LIVE PREVIEW

S3K Seeking Statement-Supporting top-K Witnesses Steffen Metzger*, - - PowerPoint PPT Presentation

Jun 05, 2023 280 likes •616 views

S3K Seeking Statement-Supporting top-K Witnesses Steffen Metzger*, Shady Elbassuoni*, Katja Hose*, Ralf Schenkel* *Max-Planck-Institute for Informatics Saarland University CIKM 2011 25 th October 2011 Statements Statements =

slide-1
SLIDE 1

S3K

Seeking Statement-Supporting top-K Witnesses

Steffen Metzger*, Shady Elbassuoni*, Katja Hose*, Ralf Schenkel*⁺

*Max-Planck-Institute for Informatics ⁺Saarland University CIKM 2011 25th October 2011

slide-2
SLIDE 2

S3K: Seeking Statement-Supporting top-K Witnesses

2/25

Statements

  • Statements = „factual statements“

– Obama was born in Hawaii – Obama was born in Kenya – Heath Ledger played a role in The Dark Knight – Maggie Gyllenhaal appeared in The Dark Knight

– …

  • „statement“ used interchangeably with „fact“, but no

truthness assumed in general

slide-3
SLIDE 3

S3K: Seeking Statement-Supporting top-K Witnesses

3/25

Document Search by Statements

Heath Ledger played a role in The Dark Knight Statement Search Engine Doc A Doc B Doc C

Witnesses for the Statement(s)

slide-4
SLIDE 4

S3K: Seeking Statement-Supporting top-K Witnesses

4/25

Document Search by Statements

Heath Ledger played a role in The Dark Knight Statement Search Engine Doc A Doc B Doc C

Witnesses for the Statement(s)

slide-5
SLIDE 5

S3K: Seeking Statement-Supporting top-K Witnesses

5/25

Statement Expressions

  • One statement, many forms of verbal expression

Ledger's performance in The Dark Knight... ...Ledger's latest movie The Dark Knight... Heath was great in The Dark Knight... ...Ledger playing the Joker in The Dark Knight...

slide-6
SLIDE 6

S3K: Seeking Statement-Supporting top-K Witnesses

6/25

Statement Expressions

  • One statement, many forms of verbal expression
  • Unify into single representation: RDF triples

HeathLedger actedIn TheDarkKnight

Ledger's performance in The Dark Knight... ...Ledger's latest movie The Dark Knight... Heath was great in The Dark Knight... ...Ledger playing the Joker in The Dark Knight...

Subject Relation Object

Did Heath Ledger play a role in The Dark Knight? Ledger acted in the new Batman movie?

slide-7
SLIDE 7

S3K: Seeking Statement-Supporting top-K Witnesses

7/25

Document Search by Statements

Statement Search Engine Doc A Doc B Doc C

HeathLedger actedIn TheDarkKnight HeathLedger actedIn TheDarkKnight

Appearance Index:

  • Doc A
  • Doc B
  • Doc C

But wait! What is it good for?

slide-8
SLIDE 8

S3K: Seeking Statement-Supporting top-K Witnesses

8/25

Obama wasBornIn Kenya

Information Need: Verification

Obama wasBornIn Hawaii

Hey, I heard Obama is born in Kenya! What?! No, he's from Hawaii! Isn't that part of Kenya?

slide-9
SLIDE 9

S3K: Seeking Statement-Supporting top-K Witnesses

9/25

Obama wasBornIn Kenya

Information Need: Verification

Obama wasBornIn Hawaii

Hey, I heard Obama is born in Kenya! What?! No, he's from Hawaii! Isn't that part of Kenya?

slide-10
SLIDE 10

S3K: Seeking Statement-Supporting top-K Witnesses

10/25

Information Need: Context

HeathLedger actedIn TheDarkKnight MaggieGyllenhaal actedIn TheDarkKnight Did they have an affair? What's his interpretation

  • f that role?

What role did he play? What's the relation

  • f their characters?
slide-11
SLIDE 11

S3K: Seeking Statement-Supporting top-K Witnesses

11/25

Ranked Witness List

Statement Set:

Doc A Doc B Doc C

Ranked list of source documents

How should we rank?

HeathLedger actedIn TheDarkKnight MaggieGyllenhal actedIn TheDarkKnight

slide-12
SLIDE 12

S3K: Seeking Statement-Supporting top-K Witnesses

12/25

Not every witness is equally appropriate

slide-13
SLIDE 13

S3K: Seeking Statement-Supporting top-K Witnesses

13/25

Not every witness is equally appropriate

Clearly in front

  • f the line

VS

Some witnesses have more Authority than others (witness trust)

slide-14
SLIDE 14

S3K: Seeking Statement-Supporting top-K Witnesses

14/25

Testimonies are not equally convincing

Obama visited his home country Kenya. Barack Obama was born in Honolulu, Hawaii in 1961 Ledger's latest movie The Dark Knight topped the box office ... Ledger playing the Joker in The Dark Knight...

slide-15
SLIDE 15

S3K: Seeking Statement-Supporting top-K Witnesses

15/25

Testimonies are not equally convincing

Obama visited his home country Kenya. Barack Obama was born in Honolulu, Hawaii in 1961 Ledger's latest movie The Dark Knight topped the box office ... Ledger playing the Joker in The Dark Knight...

Persuasiveness of testimonies varies (strength of statement support)

slide-16
SLIDE 16

S3K: Seeking Statement-Supporting top-K Witnesses

16/25

The more the better

The more statements covered the better (coverage)

Obama wasBornIn Hawaii Hawaii isPartOf USA

slide-17
SLIDE 17

S3K: Seeking Statement-Supporting top-K Witnesses

17/25

The more the better

Obama wasBornIn Hawaii

slide-18
SLIDE 18

S3K: Seeking Statement-Supporting top-K Witnesses

18/25

The more the better

The more additional information about the topic (entities involved) the better (on-topicness)

Obama wasBornIn Hawaii

slide-19
SLIDE 19

S3K: Seeking Statement-Supporting top-K Witnesses

19/25

Criteria for ranking

  • Coverage – how many of the statements are

covered in the document?

  • Persuasiveness – how convincing are the

sources – Authority of the document – Strength of statement support

  • On-Topicness – is the focus of the document
  • n the topic expressed by the statements?
slide-20
SLIDE 20

S3K: Seeking Statement-Supporting top-K Witnesses

20/25

Criteria for ranking

  • Coverage – how many of the statements are

covered in the document?

  • Persuasiveness – how convincing are the

sources – Authority of the document – Strength of statement support

  • On-Topicness – is the focus of the document
  • n the topic expressed by the statements?

1) How to connect documents with statements? 2) And how can we measure a document's persuasiveness? „Easy“: Appearances of the entities occurring in the statement(s)

slide-21
SLIDE 21

S3K: Seeking Statement-Supporting top-K Witnesses

21/25

Leveraging IE Methods

Information Extraction Ontology

Heath Ledger was born in Perth, Australia... ...Ledger's latest movie The Dark Knight... HeathLedger actedIn TheDarkKnight HeathLedger wasBornIn Perth(Australia) Pattern p1: X was born in Y Pattern p2: X 's latest movie Y

Information Knowledge Extraction Rules, Patterns

slide-22
SLIDE 22

S3K: Seeking Statement-Supporting top-K Witnesses

22/25

Leveraging IE Methods

Information Extraction Ontology

Heath Ledger was born in Perth, Australia... ...Ledger's latest movie The Dark Knight... HeathLedger actedIn TheDarkKnight HeathLedger wasBornIn Perth(Australia) Pattern p1: X was born in Y Pattern p2: X 's latest movie Y

Pattern reliability differs

→ Confidence values

p1 expresses wasBornIn : 0.9 p2 expresses actedIn : 0.4 p2 expresses directed : 0.5

slide-23
SLIDE 23

S3K: Seeking Statement-Supporting top-K Witnesses

23/25

Witness Ranking: Trust

Rank of witness d based on

Probability to be a perfect witness of a given set

  • f statements with

g={f 1,..., f n}

PisPerfect d , g

f i=si ,ri ,oi

slide-24
SLIDE 24

S3K: Seeking Statement-Supporting top-K Witnesses

24/25

Witness Ranking: Trust

Rank of witness d based on

Probability to be a perfect witness of a given set

  • f statements with

g={f 1,..., f n}

PisPerfect d , g

trust estimated by pagerank pr(d)

PisPerfectd ,g∝Ptrust d Pg∣d PisPerfectd ,g∝ prd Pg∣d

Independence Collection (Col) based Smoothing

Pg∣d=∏

i=1 n

 Pf i∣d1− Pf i∣Col f i=(si,ri,oi)

slide-25
SLIDE 25

S3K: Seeking Statement-Supporting top-K Witnesses

25/25

Support and On-Topicness

Pf i∣X=s Pesi∣Xo Peoi∣X1−s−oPf f i∣X Pg∣d=∏

i=1 n

 Pf i∣d1− Pf i∣Col X∈{d ,Col} Pes∣d= count s,d

∑e∈d count e,d

Pf f i∣X=∑

j=0 m

P p j

si,oi∣XPri∣p j

Pf f i∣X=∑

j=0 m

P p j

si,oi∣Xconf  p j,ri

Pattern confidence conf  p j,ri

Pattern instance with si,oi

f i=(si,ri,oi)

slide-26
SLIDE 26

S3K: Seeking Statement-Supporting top-K Witnesses

26/25

Statement Presence Estimation

Pf i∣X=s Pesi∣Xo Peoi∣X1−s−oPf f i∣X

Pf f i∣X=∑

j=0 m

P p j

si,oi∣Xconf  p j,ri

P p j

si,oi∣X= count  p j si,oi, X

∑p count  p, X

Pattern Importance Estimation

X∈{d ,Col} f i=(si,ri,oi)

slide-27
SLIDE 27

S3K: Seeking Statement-Supporting top-K Witnesses

27/25

Evaluation

  • Documents: ~180.000 from ClueWeb09

⁺

  • IE Parser: Modified SOFIE/PROSPERA system
  • Evaluation 1:

– Comparing different ranking approaches on prefiltered witness set

  • naive: only filtering
  • topic: mainly enity occurrences
  • persuade: aiming for persuasiveness
  • mix: balancing both aims
  • lucene: keyword search by entities on filtered subset

⁺http://lemurproject.org/clueweb09.php/

slide-28
SLIDE 28

S3K: Seeking Statement-Supporting top-K Witnesses

28/25

Evaluation

  • Evaluation 2:

– Comparing extraction based methods against plain keyword search on full document corpus

  • unfluc: e e

– based on entities

  • unfluc: e r e

– based on entities & manual relation formulation

  • unfluc: e p e

– based on entities & best pattern for relation (highest confidence)

slide-29
SLIDE 29

S3K: Seeking Statement-Supporting top-K Witnesses

29/25

Evaluation

  • 56 queries

– witness pooling from various rankings (top-50, top-20)

  • Graded Human Assessments

– On-topicness: focus on entities of statements – Persuasiveness: how well does the document convince me of the statement(s) – 3 grades on both scales

⁺http://lemurproject.org/clueweb09.php/

slide-30
SLIDE 30

S3K: Seeking Statement-Supporting top-K Witnesses

30/25

Evaluation

Overall comparison over all queries

slide-31
SLIDE 31

S3K: Seeking Statement-Supporting top-K Witnesses

31/25

Evaluation – Single vs. Multi-Statement

Comparing single-statement vs. multi-statement Persuasiveness performance by mrr

slide-32
SLIDE 32

S3K: Seeking Statement-Supporting top-K Witnesses

32/25

Thanks for your attention

  • Any questions?
slide-33
SLIDE 33

S3K: Seeking Statement-Supporting top-K Witnesses

33/25

Image sources used

  • Mostly public domain, some CC licenses, some

from free collections/fun pages assuming public domain or a fair use based usage

  • Wikimedia Commons
  • Cliparts (opencliparts.org, MS ClipArt Collection)
  • Heath Ledger image from „Doctor Hyde“ @ Flickr,

CC attribution: http://www.flickr.com/photos/indieflickr/2214583962/

  • Charts generated with latex and pstricks
  • Football pictures from random fun pages, no license

information found. If you hold the rights on any of them and you feel infringed on your rights, please contact me.