AutomationinInformation ExtractionandIntegration SunitaSarawagi - - PowerPoint PPT Presentation

AutomationinInformation ExtractionandIntegration

SunitaSarawagi IITBombay sunita@it.iitb.ac.in

Dataintegration

Theprocessofintegratingdatafrommultiple, heterogeneous,looselystructuredinformation sourcesintoasinglewell-definedstructured database

Atediousexerciseinvolving

schemamapping,

structure/informationextraction,

duplicateelimination,

missingvaluesubstitution,

errordetection

standardization

Applicationscenarios

Largeenterprises:

Phenomenalamountoftimeandresourcesspenton datacleaning

Example:Segmentingandmergingname-address listsduringdatawarehousing

Web:

Creatingstructureddatabasesfromdistributed unstructuredweb-pages

Citationdatabases:CiteseerandCora

Otherscientificapplications

Bio-informatics

Extractinggenerelationsfrommedicaltext(KDDcup2002)

Casestudy:CiteSeer

Paperlocation:

Extractinformationfromspecificpublisherwebsites

Extractps/pdf filesbysearchingthewebwithterms like“publications”

Informationextractedfrompapers:

Title,authorfromheader

Extractcitationentries

Bibliographysection

Separateintoindividualrecords

Segmentintotitle,author,date,pagenumbersetc

Duplicateeliminationacrossseveralcitationsto apaper(de-duplication)

Recenttrends

Classicalproblemthathasbotheredresearchersand practitionersfordecades

Severalexistingcommercialsolutionsforenterprisedata integration[mid-80s]

Manual,domain-specific,data-drivenscript-basedtools

Example:Name/addresscleaning

Requirehigh-expertisetocodeandmaintain

Desiretoview“Webasadatabase”gotmachinelearning researchersworkingoncleaning

Scopeofthetutorial

Novelapplicationofdataminingandmachine learningtechniquestoautomatedatacleaning

• perations.

Distillrecentresearchresultsfromvariousareas:

Machinelearning,datamining,informationretrieval, naturallanguageprocessing,webwrapperextraction

Focusontwooperations

InformationExtraction

Duplicateelimination

Outline

InformationExtraction

Rule-basedmethods

Probabilisticmethods

Duplicateelimination

Reducingtheneedfortrainingdata:

Activelearning

Bootstrappingfromstructureddatabases

Semi-supervisedlearning

Summaryandresearchproblems

InformationExtraction(IE)

TheIEtask:Given,

E:asetofstructuredelements(Targetschema)

S:unstructuredsourceS

extractallinstancesofEfromS

Varyinglevelsofdifficultydependingoninput andkindofextractedpatterns

Textsegmentation:Extractionbysegmentingtext

HTMLwrapper:Extractionfromformattedtext

ClassicalIE:Extractionfromfree-formattext

• IEbytextsegmentation

Source:concatenationofstructuredelementswith limitedreorderingandsomemissingfields

Example:Addresses,bibrecords

House number Building Road City Zip

Author Year Title Journal VolumePage State

• ý

• ö

HTMLWrappers

Recordlevel:

Extractingelementsofasinglelistofhomogeneous recordsfromapage

Discoveringrecordboundarybydetectingregularity

Page-level:

Extractingelementsofmultiplekindsofrecords

Example:name,courses,publicationsfromhomepages

Site-level:

Example:populatingauniversitydatabasefrompages

• fauniversitywebsite

IEfromfree-formattext

Examples:

Geneinteractionsfrommedicalarticles

Partnumber,problemdescriptionfromemailsinhelp centers

Structuredrecordsdescribinganaccidentfrom insuranceclaims,

Mergingcompanies,theirrolesandamountfrom newsarticles FocusofNLresearchers[MessageUnderstanding Conferences(MUC)] Requiresdeeplinguisticsandsemanticanalysis Wewilldiscuss:ShallowIEbasedonsyntacticcues

IEviamachinelearning

Givenseveralexamplesshowingpositionof structuredelementsintext, Trainamodeltoidentifytheminunseentext Issues:

Whataretheinputfeatures?

Buildper-elementclassifiersorasinglejointclassifier?

Whichtypeofclassifiertouse?

Howmuchtrainingdataisrequired?

Canonetellwhentheextractorislikelywrong?

Inputfeatures

Contentoftheelement

Specifickeywordslikestreet,zip,vol,pp,

Propertiesofwordslikecapitalization,partsof speech,number?

Formattinginformation:e.g.,font,size

Inter-elementsequencing

Intra-elementsequencing

Elementlength

Externaldatabase

Dictionarywords

Semanticrelationshipbetweenwords

Richerstructure:tree,tables

• ❋

StructureofIEmodels

Rule-basedIEmodels

Stalker (Musleaetal2001)

Modeltype:Ruleswithconjunctsanddisjuncts

Foreachelement,tworules:startrulesR1andendruleR2

Features:

htmltagsprimarily

punctuations

predefinedtextfeatures:isNumber,isCapitalized

Relationshipbetweenelements:

Independentwithinsamelevelofhierarchy

Trainingmethod:basicsequentialrulecovering algorithm

Example

Author:

R1:skipTo(<li> )

R2:skipTo(( )

Title:

R1:skipTo(<B> )ORskipTo(“ )

disjunction

R2:skipTo(</B> )OR skipTo(“ )

Volume:

R1: skipTo(<B> )skipuntil(Number)

conjunction

R2: skipTo(</B> )

• ✄
• ☎

Limitationsofrule-basedapproach

AsinWEIN,Stalker

Noorderingdependencybetweenelements

Non-overlapofelementsnotexploited

Positioninformationignored

Contentlargelyignored

Heuristicstoorderrulefiringandordering

• ✄
• ☎

Finitestatemachines

Modelorderingrelationshipbetweenelements( Softmealy,Hsu1998)

Node:elementstobeextracted

Transitionedge:rulesmarkingstartofelement.

RulesaresimilartothoseinSTALKER.

Whenmorethanonerulefiresapplymorespecificrule Allallowablepermutationsmustappearintrainingdata

IEwithHiddenMarkovModels

ProbabilisticmodelsforIE

0.9 0.5 0.5 0.8 0.2 0.1

A B C 0.6 0.3 0.1 X B Z 0.4 0.2 0.4 Y A C 0.1 0.1 0.8

dddd dd 0.8 0.2

HMMStructure

• NaïveModel:Onestateperelement
• Nestedmodel

Eachelementanother HMM

HMMDictionary

Foreachword(=feature),associatethe probabilityofemittingthatword

Multinomialmodel

Moreadvancedmodelswithoverlapping featuresofaword,

example,

partofspeech,

capitalizedornot

type:number,letter,wordetc

Maximumentropymodels(McCallum2000)

Learningmodelparameters

Whentrainingdatadefinesuniquepaththrough HMM

Transitionprobabilities

Probabilityoftransitioningfromstateitostatej=

numberoftransitionsfromitoj totaltransitionsfromstatei

Emissionprobabilities

Probabilityofemittingsymbolkfromstatei=

numberoftimeskgeneratedfromi numberoftransitionfromI

Whentrainingdatadefinesmultiplepath:

AmoregeneralEMlikealgorithm(Baum-Welch)

UsingtheHMMtosegment

FindhighestprobabilitypaththroughtheHMM.

Viterbi:quadraticdynamicprogrammingalgorithm

• t

• t

ComparativeEvaluation

Naïvemodel– Onestateperelementinthe HMM

IndependentHMM– OneHMMperelement;

RuleLearningMethod– Rapier

NestedModel– EachstateintheNaïvemodel replacedbyaHMM

Results:ComparativeEvaluation

TheNestedmodeldoesbestinallthreecases

(from Borkar 2001) Elem ents insta nces Dataset 6 740 US Addresses 6 769 Company Addresses 17 2388 IITBstudent Addresses

HMMapproach:summary

Inter-elementsequencing Intra-elementsequencing Elementlength Characteristicwords Non-overlappingtags

OuterHMMtransitions

InnerHMM

Multi-stateInnerHMM

Dictionary

Globaloptimization

InformationExtraction:summary

Rule-based

And/orcombinationwith heuristicstocontrolfiring

Brittletovariationsindata

Requirelessertrainingdata, wrappersreportedtolearn with<10examples

UsedinHTMLwrappers

Probabilistic

Jointprobability distribution,moreelegant

Mightgethardingeneral

Canhandlevariations

Usedfortextsegmentation andNEextraction

Featureengineeringiskey:havetomodelhowto combinethemwithoutunduecomplexity

Outline

InformationExtraction

Rule-basedmethods

Probabilisticmethods

Duplicateelimination

Reducingtheneedfortrainingdata:

Activelearning

Bootstrappingfromstructureddatabases

Semi-supervisedlearning

Summaryandresearchproblems

Thede-duplicationproblem

Givenalistofsemi-structuredrecords, findallrecordsthatrefertoasameentity

Exampleapplications:

Datawarehousing:mergingname/addresslists

Entity: a) Person b) Household

Automaticcitationdatabases(Citeseer):references

Entity:paper

Challenges

Errorsandinconsistenciesindata

Spottingduplicatesmightbehardastheymay bespreadfarapart:

maynotbegroup-ableusingobviouskeys

Domain-specific

Existingmanualapproachesrequirere-tuningwith everynewdomain

Example:citationsfromCiteseer

Ourprior:

duplicatewhenauthor,title,booktitleandyearmatch..

• L. Breiman,L.Friedman,andP.Stone,(1984).

LeoBreiman,JeromeH.Friedman,RichardA. Olshen,andCharlesJ.Stone.

InVLDB-94

InProc.ofthe20thInt'lConferenceonVery LargeDatabases,Santiago,Chile,September1994.

JohnsonLaird,PhilipN.(1983).Mentalmodels. Cambridge,Mass.:HarvardUniversityPress.

P.N.Johnson-Laird.MentalModels:Towardsa CognitiveScienceofLanguage,Inference,and Consciousness.CambridgeUniversityPress,1983

• H. Balakrishnan,S. Seshan,andR.H.Katz.,

ImprovingReliableTransportandHando PerformanceinCellularWirelessNetworks,ACM WirelessNetworks,1(4),December1995.

• H. Balakrishnan,S. Seshan,E. Amir,R.H.

Katz,"ImprovingTCP/IPPerformanceover WirelessNetworks,"Proc.1stACMConf.on MobileComputingandNetworking,November1995.

Learningthede-duplicationfunction

Givenexamplesofduplicatesandnon-duplicate pairs,learntopredictifpairisduplicateornot. Inputfeatures:

Variouskindsofsimilarityfunctionsbetweenattributes

Editdistance, Soundex,N-gramsontextattributes

Absolutedifferenceonnumericattributes

Someattributesimilarityfunctionsareincompletely specified

Example:weighteddistanceswithparameterizedweights

Needtolearntheweightsfirst.

Thelearningapproach

Learningattributesimilarityfunctions

Stringeditdistancewithparameters:

C(x,y):costofreplacingxwithy

d:costofdeletingacharacter

i:costofinsertingacharacter

Learningparametersfromexamples showingmatchings

TransformedExamples:

sequenceof

Match

Insert

Deletes

[Bilenko&Mooney,2002]

[Ristad&Yianilos1998

Trainastochasticmodelonsequence

• ❍❏■

• ◆P❖

Summary:De-deduplication

Previousworkconcentratedondesigninggood static,domain-specificstringsimilarityfunctions

Recentspateofworkondynamiclearning-based approachappearspromising

Twolevels:

Attribute-level:Tuningparametersofexistingstring similarityfunctionstomatchexamples

Record-level:ClassifierslikeSVMsanddecision treesusedtocombinethesimilarityalongvarious attributessavingtheeffortoftuningthresholdsand conditions

Outline

InformationExtraction

Rule-basedmethods

Probabilisticmethods

Duplicateelimination

Reducingtheneedfortrainingdata:

Activelearning

Bootstrappingfromstructureddatabases

Semi-supervisedlearning

Summaryandresearchproblems

Activelearning

Ordinarylearner:

learnsfromafixedsetoflabeledtrainingdata

Activelearner:

Selectsunlabeledexamplesfromalargepooland interactivelyseekstheirlabelsfromauser

Carefulselectionofexamplescouldleadtofaster convergence

Usefulwhenunlabeledexamplesareabundantand labelingthemrequireshumaneffort

Example:activelearning

Measuringpredictioncertainty

Classifier-specificmethods

Supportvectormachines:

Distancefromseparator

NaïveBayes classifier:

Posteriorprobabilityofwinningclass

Decisiontreeclassifier:

Weightedsumofdistancefromdifferentboundaries,errorof theleaf,depthoftheleaf,etc

Committee-basedapproach: (Seung, Opper,andSompolinsky1992)

Disagreementsamongstmembersofacommittee

Mostsuccessfullyusedmethod

Formingaclassifiercommittee

Randomlyperturblearntparameters

Probabilisticclassifiers:.

Samplefromposteriordistributiononparameters giventrainingdata.

Example:binomialparameterphasabeta distributionwithmeanp

Discriminativeclassifiers:

Randomboundaryinuncertaintyregion

Committee-basedalgorithm

TrainkclassifiersC1,C2,..Ckontrainingdata

Foreachunlabeledinstancex

Findpredictiony1,..,yk fromthekclassifiers

ComputeuncertaintyU(x)asentropyofabovey-s

Pickinstancewithhighestuncertainty

• ✁

• ✟

• ✝✞

Activelearningindeduplication withdecisiontrees

Formingcommitteeoftreesbyrandomperturbation

Selectingsplitattribute

Normally:attributewithlowestentropy

Perturbed:randomattributewithincloserangeoflowest

Selectingasplitpoint

Normally:midpointofrangewithlowestentropy

Perturbed:arandompointanywhereintherangewith lowestentropy

Speedofconvergence

With100pairs:

Activelearning:97% (peak)

Random:only30% (fromSarawagi2002)

• ❍

ActivelearninginIEwithHMM

FormingcommitteeofHMMsbyrandom perturbation

Emissionandtransitionprobabilitiesare independentmultinomialdistributions.

PosteriordistributionforMultinomialparameters:

Dirichlet withmeanestimatedasusingmaximum likelihood

Resultsonpartofspeechtagging(Dagan1999)

92.6%accuracyusingactivelearningwith20,000 instancesasagainst100,000random

Activelearninginrule-basedIE

Stalker(Musleaetal2000)

Learntwoclassifiers:

• nebasedonaforwardtraversalofthedocument,

secondbasedonabackwardtraversal

Selectforlabelingthoserecordsthatget conflictingpredictionfromthetwo

Performance:85%accuracywithoutactive learningyield94%withactivelearning

Bootstrappingfromstructured databases

Givenadatabaseofstructuredelements

Example:collectionofstructured bibtex entries

Segmenttobestmatchwiththedatabase

HMM:

Initializedictionaryusingdatabase

LearntransitionsusingBaumWelchonunlabeled data

Assigningprobabilitieshard

Stillopentoinvestigation

Rule-basedIE:(Snowball,Agichtein2000)

Semi-supervisedlearning

Canunlabeleddataimproveclassifieraccuracy?

Possibly,forprobabilisticclassifierslikeHMMs

Uselabeleddatatotrainaninitialmodel

UseBaumWelchonunlabeleddatatorefinemodelto maximizedatalikelihood

Unfortunately,nogaininaccuracyreported(Seymore 1999)

Needsfurtherinvestigation

Unsupervisedlearninginduplicate elimination

[Tailor2002]

Clusterthesimilarityvectorsofrecordpairsinto threegroups

Labeltheclustersbasedondistancetotheideal duplicateandnon-duplicatevector.

(optional)Trainclassifieronthislabeleddata

Results:79.8%accuracyonWalmart’s items table.

Summary

InformationExtraction

Variouslevelsofcomplexitydependingoninput

Segmentation,HTMLwrappers,free-format

Model-type:

Rule-basedandprobabilistic(HMM)

Independentorsimultaneous

Severalresearchprototypesineachtype

Duplicateelimination

Challengingbecauseofvariationsindataformat

Learningappliedtodesigndeduplicationfunction

ManualVslearningapproach

Manual

Inspectpatterns

Codescripts

Requireshigh-skill programmer Learning

Labelexamples

Choose&trainmodel

Low-skill,cheaperlabor formostpart

Featuredesignand modelselection requiresveryhighskill

Summary

Reducingneedforlabeleddata

Activelearning

Variousmethodsproposed

Committee-basedsamplingmostpopular

Applicationwith

HMMforIE

Decisiontreesfordeduplication

Topicsoffurtherresearch

InformationExtraction:

Exploitinghigher-levelstructuresininputdata,e.g.trees,tables

IntegratedlearninginthepresenceofalargestructuredDB, smalllabeleddataandlargeunlabeleddata

Efficiencyinthepresenceofalargedatabase/dictionary

Wrappersatthewebsitelevelinvolvingseveralstructuredtables

Duplicateelimination

Multi-tablede-duplication

Integratingsemi-supervisedandactivelearning

Efficientactivelearningwithoutrequiringmaterializationofall possiblepairs

Efficientevaluationofade-duplicationfunction

Topicsoffurtherresearch

Combiningmachinelearningofextraction patternswithhumangeneratedscripts

Updatingmodelsasdataarrives:continuous learning

Goingfromresearchprototypestorobust productsandtoolkits

References

General

• H. Galhardas,D. Florescu,D. Shasha,E.Simon,andC. Saita.Declarativedatacleaning:

Language,modelandalgorithms.VLDB,2001.

S.Lawrence,C.L.Giles,andK. Bollacker.Digitallibrariesandautonomouscitationindexing. IEEEComputer,32(6):67-71,1999.

A.McCallum,K. Nigam,J.Reed,J. Rennie,andK. Seymore.Cora:Computerscience researchpapersearchengine.http://cora.whizbang.com/,2000.

IEEEDataEngineeringspecialissueonDataCleaning. http://www.research.microsoft.com/research/db/debull/A00dec/issue. htm,December2000.

M.A.HernandezandS.J. Stolfo.Real-worlddataisdirty:Datacleansingandthe merge/purgeproblem.DataMiningandKnowledgeDiscovery,2(1), 1998.

Information extraction

• E. Agichtein,L.Gravano,“Snowball:Extractingrelationsfromlargeplaintextcollections",

ACMIntl.Conf.onDigitalLibraries“2000

DM.Bikel,SMiller,RSchwartzandR. Weischedel,"Nymble:ahigh-performancelearning name-finder",ANLP1997,

Vinayak R. Borkar, KaustubhDeshmukh,and SunitaSarawagi.Automatictextsegmentation forextractingstructuredrecords.SIGMOD2001.

MaryElaine Calif andR.J.Mooney.Relationallearningofpattern-matchrulesforinformation extraction.AAAI1999.

DFreitag andAMcCallum,InformationExtractionwithHMMStructuresLearnedby StochasticOptimization,AAAI2000

A.McCallumandD. Freitag andF.Pereira,MaximumentropyMarkovmodelsforinformation extractionandsegmentation,ICML-2000

References

K Seymore,AMcCallum,RRosenfeld.LearningHiddenMarkovModelstructureforinformation extraction.AAAIWorkshoponMachineLearningforInformationExtraction,1999.

• S. Soderland.Learninginformationextractionrulesforsemi-structuredandfreetext.Machine

Learning,34,1999.

Wrappers

C.Y.Chung,M. Gertz,andN. Sundaresan.Reverseengineeringforwebdata:From

visualtosemanticstructures.ICDE2002.

WilliamW.Cohen,MatthewHurst,andLeeS.Jensen.A exible learningsystemforwrapping tablesandlistsinhtmldocuments.WWW2002.

• DavidW. Embley,Y.S. Jiang,and Yiu-KaiNg.Record-boundarydiscoveryinwebdocuments.In

SIGMOD1999.

C.-N.HsuandM.-T.Dung.Generatingfinite-statetransducersfor semistructureddataextraction fromtheweb.InformationSystemsSpecialIssueon Semistructured Data,23(8),1998.

• N. Kushmerick,D.S.Weld,andR. Doorenbos.Wrapperinductionforinformationextraction.

IJCAI,1997.

L.Liu,C. Pu,andW.Han. Xwrap:AnXML-enabledwrapperconstructionsystemforweb informationsources.ICDE,2000.

Ion Muslea,StevenMintonandCraigA. Knoblock,HierarchicalWrapperInductionfor Semistructured InformationSources,"AutonomousAgentsandMulti-AgentSystems",2001.

JussiMyllymaki.EffectivewebdataextractionwithstandardXMLtechnologies.WWW,2001.

References

Duplicateelimination

AZ.Broder,SC.Glassman,MS.Manasse,GeoffreyZweig,“SyntacticClusteringoftheWeb”, WWW1997

M.G. Elfeky,V.S. Verykios,A.K. Elmagarmid,“Tailor:Arecordlinkagetoolkit”,ICDE2002.

SSarawagi and AnuradhaBhamidipaty,Interactivededuplication usingactivelearning,ACM SIGKDD2002

W.E.Winkler.Matchingandrecordlinkage.InB.G.C.etal,editor,BusinessSurveyMethods, pages355-384.NewYork:J.Wiley,1995.

Activeandsemi-supervisedlearning

ShlomoArgamon-Engelson and IdoDagan.Committee-basedsampleselectionforprobabilistic classififers.J.ofArtificialIntelligenceResearch,11:335--360,1999.

Yoav Freund,H.Sebastian Seung,Eli Shamir,andNaftaliTishby.Selectivesamplingusingthe querybycommitteealgorithm.MachineLearning,28(2-3):133-168,1997.

Ion Muslea,SteveMinton,andCraig Knoblock.“Selectivesamplingwithredundantviews".AAAI, 2000

H.S. Seung,M. Opper,andH. Sompolinsky.Querybycommittee.InComputationalLearing Theory,pages287-294,1992.

T.ZhangandF.J. Oles.Aprobabilityanalysisonthevalueofunlabeleddataforclassification problems.ICML,2000