[PPT] - Autonomously Reviewing and Validating the Knowledge Base of a PowerPoint Presentation

SLIDE 1

Autonomously Reviewing and Validating the Knowledge Base of a Never-Ending Learning System

Saulo D. S. Pedro 1, Ana Paula Appel 2 and Estevam R. Hruschka Jr. 1

1 Department of Computer Science – Federal University of S˜

ao Carlos, Brazil

2 IBM Research Brazil

May, 2013

1 / 22

SLIDE 2

Outline

1

Introduction

2

Motivation

3

Proposed Work

4

Experiments

5

Conclusion

2 / 22

SLIDE 3

Introduction

Outline

1

Introduction

2

Motivation

3

Proposed Work

4

Experiments

5

Conclusion

3 / 22

SLIDE 4

Introduction

NELL (Never-Ending Language Learner)

A computer system that runs 24/7; Gather knowledge from web pages to acquire knowledge to become a better learner each day; The content available on the web is not always reliable - can lead a false beliefs propagation because of noisy data; Part of the knowledge extracted by NELL should be supervised by humans to be incorporated definitely in KB.

4 / 22

SLIDE 5

Introduction

Prophet

Implements link prediction on NELL to finding new relations in the NELL’s KB and identifying the anomalies, → misplaced edges The relations and categories extracted by NELL are mapped as an

ntology → complex network

Use graph properties to investigate if the knowledge learned by NELL is correct or not.

5 / 22

SLIDE 6

Introduction

Prophet Example

Sport Team Team Plays in League P l a y e r s Sport Uses Stadium

A t h l e t e P l a y s i n L e a g u e

Stadium Home to League Basketball NBA Milwaukee Bucks Michael Redd Madison Square Garden

6 / 22

SLIDE 7

Introduction

Prophet Rules

R12a(sport, sportsleague):- players(sport, athlete), athleteplaysinleague(athlete, sportsleague), numberof(athlete) ≥ 10; R12b(sport, sportsleague):- sportteam(sport, sportsteam), teamplaysinleague(sportsteam, sportsleague), numberof(sportsteam) ≥ 10; R12c(sport, sportsleague):- sportusesstadium(sport, stadiumoreventvenue), stadiumhometoleague(stadiumoreventvenue, sportsleague), numberof(stadiumoreventvenue) ≥ 10 R12d(sport, sportsleague):- players(sport, athlete), athleteplaysinleague(athlete, sportsleague),sportteam(sport, sportsteam), teamplaysinleague(sportsteam, sportsleague), sportusesstadium(sport, stadiumoreventvenue), stadiumhometoleague(stadiumoreventvenue, sportsleague);

7 / 22

SLIDE 8

Introduction

Prophet Misplaced Edges

P l a y e r s

Athlete Plays in League

Soccer NBA Cristiano Ronaldo When Prophet identifies an outliers, it means that the its algorithm was able to determine a new rule but there are a few instances that do not match all the requirements of rule found by Prophet → misplaced edges.

8 / 22

SLIDE 9

Motivation

Outline

1

Introduction

2

Motivation

3

Proposed Work

4

Experiments

5

Conclusion

9 / 22

SLIDE 10

Motivation

There are two possible scenarios for the anomalies: at least one relation (edge) in the anomaly should be wrong the two rules are right but because of combination made by Prophet the relation predicted is wrong

10 / 22

SLIDE 11

Motivation

There are two possible scenarios for the anomalies: at least one relation (edge) in the anomaly should be wrong the two rules are right but because of combination made by Prophet the relation predicted is wrong The information gathered by Prophet could be just sent to human

supervision. But we want to take:

best profit from these anomalies advantage human opinion through Web communities thus configuring a self-supervision approach

10 / 22

SLIDE 12

Proposed Work

Outline

1

Introduction

2

Motivation

3

Proposed Work

4

Experiments

5

Conclusion

11 / 22

SLIDE 13

Proposed Work

Prophet + SS-Crowd

Problem Description

How Conversing Learning techniques can be used to help reviewing and validating facts that were learned by NELL and were flagged as possible mistakes by Prophet.

12 / 22

SLIDE 14

Proposed Work

Prophet + SS-Crowd

Problem Description

How Conversing Learning techniques can be used to help reviewing and validating facts that were learned by NELL and were flagged as possible mistakes by Prophet.

Proposed Work

A method to combine the knowledge gathered from web communities through the SS-Crowd component with the outliers identified by Prophet, i.e., use web QA users opinion to validate the anomalies.

12 / 22

SLIDE 15

Proposed Work

Conversing Learning

Based on Active Learning and Interactive Learning Allow machines to convert knowledge into content understandable by humans Autonomously ask people to take part in the knowledge acquisition and labelling process

13 / 22

SLIDE 16

Proposed Work

Reaching web users assessment through SS-Crowd

The proposed approach can be summarized by the following steps: Converting KB’s facts into human understandable sentences; Generating questions that will prompt users to decide whether the facts are correct or not; Receiving all the answers for an specific question; Combining the answers to produce a single result; Returning to Prophet that will use it as a parameter to create or not a new link in NELL’s KB.

14 / 22

SLIDE 17

Proposed Work

Experiment with SS-Crowd

Edges of an outlier identified by Prophet

TeamPlaysSport(Manchester United, basketball) TeamWonTrophy(Manchester United, UEFA Champions League)

Edges converted into human understandable questions:

Manchester United is a team that plays sport basketball Manchester United is a team that won trophy UEFA Champions League:

15 / 22

SLIDE 18

Proposed Work

Experiment with SS-Crowd

Edges of an outlier identified by Prophet

TeamPlaysSport(Manchester United, basketball) TeamWonTrophy(Manchester United, UEFA Champions League)

Edges converted into human understandable questions:

Manchester United is a team that plays sport basketball Manchester United is a team that won trophy UEFA Champions League:

Expectation:

At least one of the edges is wrong, confirming the health of outliers identification algorithm.

15 / 22

SLIDE 19

Experiments

Outline

1

Introduction

2

Motivation

3

Proposed Work

4

Experiments

5

Conclusion

16 / 22

SLIDE 20

Experiments

Experiments Set up

We used NELL’s KB at the 100th iteration → undirected graph 9,419 nodes and 24,132 edges.; We ran Prophet that found new rules and instances and misplace edges; all misplaced edges were sent to SS-Crowd to start the human assessment process;

17 / 22

SLIDE 21

Experiments

Experiments Set up

We used NELL’s KB at the 100th iteration → undirected graph 9,419 nodes and 24,132 edges.; We ran Prophet that found new rules and instances and misplace edges; all misplaced edges were sent to SS-Crowd to start the human assessment process;

Table : Distribution of the relations considered in our tests Relations # of outliers # of answers AthletePlaysInLeague & Players 9 72 TeamPlaysSport & TeamPlaysInLeague 20 144 TeamPlaysSport & TeamWonTrophy 53 386

17 / 22

SLIDE 22

Experiments

Results

The rate of outliers with at least one wrong edge indicates the health of the anomalies detection algorithm;

18 / 22

SLIDE 23

Experiments

Results

The rate of outliers with at least one wrong edge indicates the health of the anomalies detection algorithm;

Table : Numbers for edges evaluated as suitable or not to the real world through the web community eyes. Outliers at least one wrong edge 39 (47.56%) both edges correct 40 (48.19%) unresolved edges 3 (03.65%)

18 / 22

SLIDE 24

Experiments

Experiment with SS-Crowd

Edges of an outlier identified by Prophet

TeamPlaysSport(Manchester United, basketball) TeamWonTrophy(Manchester United, UEFA Champions League)

Edges converted into human understandable questions:

Manchester United is a team that plays sport basketball; Manchester United is a team that won trophy UEFA Champions League:

19 / 22

SLIDE 25

Experiments

Experiment with SS-Crowd

Edges of an outlier identified by Prophet

TeamPlaysSport(Manchester United, basketball) TeamWonTrophy(Manchester United, UEFA Champions League)

Edges converted into human understandable questions:

Manchester United is a team that plays sport basketball; Manchester United is a team that won trophy UEFA Champions League:

Both relations are right!!

19 / 22

SLIDE 26

Experiments

Experiment with SS-Crowd

Edges of an outlier identified by Prophet

TeamPlaysSport(Manchester United, basketball) TeamWonTrophy(Manchester United, UEFA Champions League)

Edges converted into human understandable questions:

Manchester United is a team that plays sport basketball; Manchester United is a team that won trophy UEFA Champions League:

Both relations are right!! Manchester United is a basketball team and also a soccer team NELL was not successful to decide which one it should chose;

19 / 22

SLIDE 27

Conclusion

Outline

1

Introduction

2

Motivation

3

Proposed Work

4

Experiments

5

Conclusion

20 / 22

SLIDE 28

Conclusion

The results obtained in the performed experiments have shown that the combination of Prophet and SS-Crowd allows a never-ending learning system - NELL - to identify which edges are really wrong and which edges needs more time (NELL iterations) to fill the gaps on information to be considered valid. The experiments show that Prophet has a great accuracy. Most of the combination of edges that produce a misplaced connection are related to a co-reference problem restricted to NELL and is not a misbehaviour of Prophet itself. The validation of a learning machine with SS-Crowd is a useful approach to help self-supervision and self-revision in NELL.

21 / 22

SLIDE 29

Autonomously Reviewing and Validating the Knowledge Base of a Never-Ending Learning System

Saulo D. S. Pedro 1, Ana Paula Appel 2 and Estevam R. Hruschka Jr. 1

1 Department of Computer Science – Federal University of S˜

ao Carlos, Brazil

2 IBM Research Brazil

May, 2013

22 / 22