Staging User Feedback toward Rapid Conflict Resolution in Data - - PowerPoint PPT Presentation

▶
staging user feedback toward rapid conflict resolution in
SMART_READER_LITE
LIVE PREVIEW

Staging User Feedback toward Rapid Conflict Resolution in Data - - PowerPoint PPT Presentation

Sep 27, 2023 219 likes •463 views

Staging User Feedback toward Rapid Conflict Resolution in Data Fusion Romila P Pradhan* , Siarhei Bykau , Sunil Prabhakar* *Purdue University, Bloomberg L.P. 1 Fusing data from multiple sources Data It Item S 1 S 2 S 3 S 4 Zootopia Howard

slide-1
SLIDE 1

Staging User Feedback toward Rapid Conflict Resolution in Data Fusion

Romila P Pradhan*, Siarhei Bykau , Sunil Prabhakar* *Purdue University, Bloomberg L.P.

1

slide-2
SLIDE 2

Fusing data from multiple sources

2

Data It Item S1 S2 S3 S4 Zootopia Howard Spencer Spencer Kung Fu Panda Stevenson Nelson Inside Out leFauve Docter Finding Dory Stanton Minions Coffin Renaud Rio Jones Saldanha

slide-3
SLIDE 3

Data It Item Correctness o

  • f c

claims Zootopia Howard (0.000) Spencer (1.000) Kung Fu Panda Stevenson (0.015) Nelson (0.985) Inside Out leFauve (0.001) Docter (0.999) Finding Dory Stanton (1.000) Minions Coffin (0.921) Renaud (0.079) Rio Jones (0.015) Saldanha (0.985) Data It Item Correctness o

  • f c

claims Zootopia Howard (0.000) Spencer ( (1.0 .000) Kung Fu Panda Stevenson (0.015) Nelson ( (0.9 .985) Inside Out leFauve (0.001) Do Docter (0.9 .999) Finding Dory Stanton ( (1.0 .000) Minions Coffin ( (0.9 .921) Renaud (0.079) Rio Jones (0.015) Sa Saldanha (0.9 .985)

Data fusion systems

3

Data It Item S1 S2 S3 S4 Zootopia Howard Spencer Spencer Kung Fu Panda Stevenson Nelson Inside Out leFauve Docter Finding Dory Stanton Minions Coffin Renaud Rio Jones Saldanha

Source accuracy Correctness

  • f claims

iterative computation So Source Ac Accuracy S1 0.317 S2 0.027 S3 0.992 S4 1.000

ACCU1

[1] Xin Luna Dong, Laure Berti-Equille, Divesh Srivastava. Data Fusion: Resolving Conflicts from Multiple Sources. WAIM 2013.

slide-4
SLIDE 4

Data It Item Correctness o

  • f c

claims Zootopia Howard (0.000) Spencer ( (1.0 .000) Kung Fu Panda Stevenson (0.015) Nelson ( (0.9 .985) Inside Out leFauve (0.001) Do Docter (0.9 .999) Finding Dory Stanton ( (1.0 .000) Minions Coffin ( (0.9 .921) Renaud (0.079) Rio Jones (0.015) Sa Saldanha (0.9 .985)

Comparison with ground truth

4

Data It Item S1 S2 S3 S4 Zootopia Howard Spencer Spencer Kung Fu Panda Stevenson Nelson Inside Out leFauve Docter Finding Dory Stanton Minions Coffin Renaud Rio Jones Saldanha

Source accuracy Correctness

  • f claims

iterative computation So Source Ac Accuracy S1 0.317 S2 0.027 S3 0.992 S4 1.000

ACCU1

[1] Xin Luna Dong, Laure Berti-Equille, Divesh Srivastava. Data Fusion: Resolving Conflicts from Multiple Sources. WAIM 2013.

Data It Item Tr Truth Zootopia Howard Kung Fu Panda Stevenson Inside Out Docter Finding Dory Stanton Minions Coffin Rio Saldanha

slide-5
SLIDE 5

Involve the User

5

D

Data Fusion Model Correctness

  • f claims

Validate data item Labels User feedback to fusion model

slide-6
SLIDE 6

6

How to be most effective with user feedback?

slide-7
SLIDE 7

This talk

7

Query-by-committee

Uncertainty Sampling

4 ranking strategies Maximum Expected Utility Approximate MEU

Item-level ranking

Feedback Errors Evaluation Non-expert

  • Confidence
  • Error-rate
  • Conflicting

feedback

  • 80
  • 60
  • 40
  • 20

20 40 60 80 100 ∆ distance_to_ground_truth (%) data items validated (%)

Random QBC US ApproxMEU MEU GUB

Holistic ranking

slide-8
SLIDE 8

Data It Item S1 S2 S3 S4 Zootopia Howard Spencer Spencer Kung Fu Panda Stevenson Nelson Inside Out leFauve Docter Finding Dory Stanton Minions Coffin Renaud Rio Jones Saldanha

Query-by-committee (QBC)

8

item-level ranking holistic ranking feedback errors evaluation

most sources agree

Zootopia Howard Spencer Spencer Rio Jones Saldanha

sources disagree

slide-9
SLIDE 9

Data It Item Correctness o

  • f c

f claims Zootopia Howard (0.000) Spencer (1.000) Kung Fu Panda Stevenson (0.015) Nelson (0.985) Inside Out leFauve (0.001) Docter (0.999) Finding Dory Stanton (1.000) Minions Coffin (0.921) Renaud (0.079) Rio Jones (0.015) Saldanha (0.985)

Uncertainty Sampling (US)

9

item-level ranking holistic ranking feedback errors evaluation

Data It Item Correctness o

  • f c

f claims Zootopia Howard (0.000) Spencer (1.000) Kung Fu Panda Stevenson (0.015) Nelson (0.985) Inside Out leFauve (0.001) Docter (0.999) Finding Dory Stanton (1.000) Minions Coffin (0.921) Renaud (0.079) Rio Jones (0.015) Saldanha (0.985) Kung Fu Panda Stevenson (0.015) Nelson (0.985) Minions Coffin (0.921) Renaud (0.079)

slide-10
SLIDE 10

Data It Item S1 S2 S3 S4 Zootopia Howard Spencer Spencer Kung Fu Panda Stevenson Nelson Inside Out leFauve Docter Finding Dory Stanton Minions Coffin Renaud Rio Jones Saldanha

Implication of a validation

10

item-level ranking holistic ranking feedback errors evaluation

Data Item S1 S2 S3 S4 Zootopia Howard Spencer Spencer Kung Fu Panda Stevenson Nelson Inside Out leFauve Docter Finding Dory Stanton Minions Coffin Renaud Rio Jones Saldanha S2 S3 S4 Renaud Stanton leFauve Saldanha Nelson Docter

slide-11
SLIDE 11

Data Item S1 S2 S3 S4 Zootopia Howard Spencer Spencer Kung Fu Panda Stevenson Nelson Inside Out leFauve Docter Finding Dory Stanton Minions Coffin Renaud Rio Jones Saldanha

11

item-level ranking holistic ranking feedback errors evaluation

S4 Spencer

Implication of a validation

slide-12
SLIDE 12

Ideal utility function

12

item-level ranking holistic ranking feedback errors evaluation

data fusion model truth function average correctness

  • f true claims

Utility Function

truth function ?

slide-13
SLIDE 13

13

  • ver correctness
  • f all claims

item-level ranking holistic ranking feedback errors evaluation

Practical utility function

Entropy Utility Function

entropies of all data items

slide-14
SLIDE 14

§ Value of perfect information

14

item-level ranking holistic ranking feedback errors evaluation

Maximum Expected Utility (MEU)

entropy utility if claim is true

Best alternative in the absence of ground truth

slide-15
SLIDE 15

Approximate-MEU

15

item-level ranking holistic ranking feedback errors evaluation

removed bottleneck iterative computation of MEU

  • Key idea: Propagation of changes

no need to fuse for every claim! correctness of claims of validated data item accuracies of sources correctness of claims

  • f unvalidated data

items

slide-16
SLIDE 16

Users can be wrong

16

item-level ranking holistic ranking feedback errors evaluation

  • Honest but unsure user

80% certain about a claim user is correct 85% of the time

Claim3 Claim1 Claim2

  • Error-rate of user
  • Conflicting feedback from a crowd of workers

6/10 3/10 1/10

slide-17
SLIDE 17

Real-world datasets

17

item-level ranking holistic ranking feedback errors evaluation

  • 1. X. L. Dong, L. Berti-Equille, and D. Srivastava. Integrating conflicting data: The role of source dependence. PVLDB, 2009
  • 2. X. Li, X. L. Dong, K. Lyons, W. Meng, and D. Srivastava. Truth finding on the deep web: Is the problem solved? PVLDB, 2012
  • 3. J. Pasternack and D. Roth. Knowing what to believe(when you already know something). COLING, 2010
  • 4. http://lunadong.com/fusionDataSets.htm

Books1 FlightsDay2 Population3 Flights2 Items 1263 5836 40696 121567 Sources 894 38 2545 38 Claims 24303 80452 46734 1931701

Feedback Simulation

  • Books: silver standard provided in [4]
  • Flight information: data provided by carrier websites considered ground truth
  • Population: manually identified the true claim for data items having multiple claims
slide-18
SLIDE 18

Competing methods

18

item-level ranking holistic ranking feedback errors evaluation

  • It

Item-level r ranking m methods

§ QBC / US

  • De

Decision-theoretic r ranking m methods

§ MEU / Approx-MEU § Greedy Upper Bound (GUB)

  • Ra

Random

§ all data items equally beneficial

ground-truth-utility-based

slide-19
SLIDE 19

Large number of sources, few claims: holistic ranking

19

item-level ranking holistic ranking feedback errors evaluation

  • 80
  • 60
  • 40
  • 20

20 40 60 80 100 ∆ distance_to_ground_truth (%) data items validated (%)

Random QBC US ApproxMEU MEU GUB

Books

slide-20
SLIDE 20

20

item-level ranking holistic ranking feedback errors evaluation

  • 12
  • 9
  • 6
  • 3

20 40 60 80 100 120 ∆ distance_to_ground_truth (%) # data items validated

QBC US ApproxMEU MEU

Population

Large number of sources, few claims: holistic ranking

slide-21
SLIDE 21

Large number of claims, few sources: either QBC/holistic

21

item-level ranking holistic ranking feedback errors evaluation

FlightsDay

  • 60
  • 40
  • 20

20 40 60 80 100 ∆ distance_to_ground_truth (%) data items validated (%)

Random QBC US ApproxMEU MEU GUB

slide-22
SLIDE 22

22

item-level ranking holistic ranking feedback errors evaluation

Flights

  • 6
  • 4
  • 2

0.5 1 1.5 2 ∆ distance_to_ground_truth (%) data items validated (%)

QBC US ApproxMEU10

Large number of claims, few sources: either QBC/holistic

slide-23
SLIDE 23

Contributions

23

  • Integrating user feedback to improve the performance of

existing data fusion systems

  • Designed strategies to generate an effective ordering for

validating claims

  • scalable decision-theoretic solution for iterative fusion
  • explored imperfect feedback scenarios
  • Evaluation on real-world datasets confirmed that guided

feedback rapidly increases the effectiveness of data fusion