TheGameTheoreticWeb TheGameTheoreticWeb - - PowerPoint PPT Presentation

Web(2.0)Mining:AnalyzingSocial Media Web(2.0)Mining:AnalyzingSocial Media

AnupamJoshi

• EbiquityGroup,UMBC

joshi@cs.umbc.edu http://ebiquity.umbc.edu/ AnupamJoshi

• EbiquityGroup,UMBC

joshi@cs.umbc.edu http://ebiquity.umbc.edu/

TheGameTheoreticWeb TheGameTheoreticWeb

SocialMedia SocialMedia

Socialmediadescribesthe

• nlinetoolsandplatforms

thatpeopleusetoshare

• pinions,insights,

experiences,and perspectives - wikipedia

• Levelofuserparticipation

andthoughtsharingacross variedtopics

Twitterment beta

StateoftheBlogosphere StateoftheBlogosphere

“Blogosphere isthecollectivetermencompassingallblogsasacommunityor socialnetwork’’ WikipediaNov06

Knowing&InfluencingyourAudience Knowing&InfluencingyourAudience

• Yourgoalistocampaignforapresidential

candidate

• Howcanyoutrackthebuzzabouthim/her?
• Whataretherelevantcommunitiesand

bogs?

• Whichcommunitiesaresupporters,whichareskeptical,

whichareputoffbythehype?

• Isyourcampaignhavinganeffect?Thedesiredeffect?
• Whichbloggersareinfluentialwithpoliticalaudience?Of

these,whicharealreadyonboardandwhicharelost causes?

• Towhomshouldyousenddetailsortalkto?

Knowing&InfluencingyourMarket Knowing&InfluencingyourMarket

• YourgoalistomarketApple’siPhone
• Howcanyoutrackthebuzzaboutit?
• Whataretherelevantcommunitiesand

blogs?

• Whichcommunitiesarefans,whichare

suspicious,whichareputoffbythehype?

• Isyouradvertisinghavinganeffect?The

desiredeffect?

• Whichbloggersareinfluentialinthis

market?Ofthese,whicharealready

• nboardandwhicharelostcauses?
• Towhomshouldyousenddetailsor

evaluationsamples?

OpinionsinSocialMedia OpinionsinSocialMedia

“LastnightinBostonatamid-dollarfundraiser

JohnEdwardsgaveafantasticspeech.Itwas

• netheloosest mostcharismatic speechesIhave

seenhimgive.Manyofthepointsandlinewere fromhisstandardstumpspeechbuttherewasa definiteconfidenceandsenseofhumorinhis delivery. HealsodwelledontheenvironmentmorethanI haveseenhimdoinotherspeeches.The environmentalsectionkickedoffwithwithagood andtrue linethatgotabigovation:“Onglobal warming:AlGorewasright.”……..”1

http://www.dailykos.com/storyonly/2007/10/4/71218/3740

Expressed Opinions Narrative Reader’sPerspective

• Opinionscan

influencethevotesof

• thers

WhatisInfluence? WhatisInfluence?

• MeasurableInfluence

Theabilityofabloggertopersuadeanotherbloggerto

• Takeactionbymeansofcreatinganewpostaboutthetopic

andcommentingontheoriginal.

• Quotetheblogger’sviewsinherpost.
• Linktotheoriginalpostviatrackbacks,comments

.

• Linktothebloggerthroughothermeanslikedel.icio.us,digg,

citeULike,Connotea,etc.

• Subscribetotheblogfeed .

Epidemic-basedInfluenceModels Epidemic-basedInfluenceModels

• LinearThresholdModel

Σ bwv≥ θv

wistheactiveneighborofv,

θv intrinsicthresholdforanode

• GreedyHeuristic
• Assignrandomθv
• Computeapproxinfluencedset
• Ateachstep,addthenodethat

increasesthemarginalgaininthesize

• ftheinfluencedset
• 4

3 2 1 5

2/5 1/5 2/5 1/3 1/3 1/3 1 1/2 1/2 1

θv Active Active Inactive Otherapproaches:Latane’,iRank,

• Kempetal.

LimitationsofExistingApproaches LimitationsofExistingApproaches

• Selectednodesmaybelongto

differenttopics

• Opinionsorbiasnot

considered

• Informationisspread

throughoutthenetworkwithout consideringsocialstructure

• Intrinsicthresholdθvisbasedona

pseudorandomfunction

• Staticviewofthenetwork,no

temporalevidence

First10nodesselectedusing GreedyHill%ClimbingHeuristic http://www.engadget.com http://www.boingboing.net http://www.dailykos.com http://postsecret.blogspot.com http://slashdot.org http://www.albinoblacksheep.com http://www.opinionjournal.com http://profiles.blogdrive.com http://godlessmom.blogspot.com http://thinkprogress.org TECH, POLITICS,DAILY/NEWS

FindingCommunities(andFeeds)ThatMatter FindingCommunities(andFeeds)ThatMatter

• !!!!

"!#$%%&'( ")!!*$+,!!&'- !!'! .&&'& "%%) TopAdvertisingFeeds

1.Adrants» MarketingandAdvertisingNewsWithAttitude 2.Adverblog:advertisingandnewmediamarketing 3.http://ad-rag.com 4.adfreak 5.AdJab 6.MITAdvertisingLab:futureofadvertisingandadvertisingtechnology 7.AdPulp:DailyJuicefromtheAdBiz 8.Advertising/DesignGoodness

RelatedTags:advertising marketing media news design

FeedsThatMatter FeedsThatMatter

TopFeedsfor“Politics”

Mergedfolders:“political”,“politicalblogs”

• TalkingPointsMemo:byJoshuaMicah

Marshall

• DailyKos:StateoftheNation
• Eschaton
• TheWashingtonMonthly
• Wonkette,PoliticsforPeoplewithDirtyMinds
• http://instapundit.com/
• InformedComment
• PowerLine
• AMERICAblog:Becauseagreatnation

deservesthetruth

• CrooksandLiars

TopFeedsfor“Knitting”

Mergedfolders“knittingblogs”

• YarnHarlotknitting
• WendyKnits!
• SeeEunnyKnit!
• theblueblog
• Grumperinagoestolocalyarnshopsand

HomeDepot

• YouKnitWhat??
• Mason-DixonKnitting
• knitandtonic
• CrazyAuntPurl
• http://www.lollygirl.com/blog/

http://instapundit.com http://michellemalkin.com/ http://dailykos.com http://crooksandliars.com http://volokh.com http://rightwingnews.com

InfluenceinCommunities InfluenceinCommunities

Communitiesdetectedusing“Fastalgorithmfordetecting communitystructureinnetworks”,M.E.J.Newman

AuthorityandPopularity AuthorityandPopularity

Authority

• contributestoinfluence
• Influencemaybesubjective.
• Asource,authoritativeinone

communitycouldinfluence anothercommunitynegatively. Withinacommunity,an authoritativesourcewouldbe influential.

Popularity

• Authorityandpopularityoften

treatedequally

• Onblogsearchengines,authority

ismeasuredusinginlinks,which isatbestpopularity

• Popularitydoesn’tmeaninfluence
• Dilbertisextremelypopularbut

notinfluential

LinkPolarity/Bias LinkPolarity/Bias

• Linkingaloneisnotindicatorofinfluence
• Polaritycanindicatethetypeofinfluence
• Consistentnegative/positiveopinionoveraperiodoftime

canindicatebias

• Linkpolarity/citationsignalcanalsobehelpfulindetermining

trust

DemocratBlog RepublicanBlog StrongNegative Opinion MildlyNegative

• pinion

StronglyPositive

• pinion

OurApproachtoLinkPolarity OurApproachtoLinkPolarity

• ShallowSentimentAnalysis
• Calculatethenumberofpositivelyoriented(!)and

Negativelyorientedwords(!)inthetext-windowaround thelink

• ApplyStemming,basiccanonicalization
• Corpusincludessimplebi-gramsoftheform“"
• Polarity=(Np– Nn)/(Np+Nn)
• Denominatoractsasanormalizationmechanism
• NaturalLanguageProcessingis,yetlarge-

scaleeffectshelp!

LinkPolarityExample LinkPolarityExample

• “StephenColbert'sperformanceattheWhiteHouseCorrespondents' Association

dinnerhasgarneredhimhuge applause intheblogosphereandalsoonC-Span whereitwasshownmorethanonce.Thoseofuswhohavebeenangry withBush forquitesometimebecauseofhisarrogant andfecklesscorruption ofour countrywereevenmorethrilledtoseeandknowthathehadnorecoursebuttosit thereandwatchhisaspirationsforgreatnessbedestroyedbyamaster ofirony. This willbehislegacy:Istandbythisman.Istandbythismanbecausehestands forthings.Notonlyforthings,hestandsonthings.Thingslikeaircraftcarriersand rubbleandrecentlyfloodedcitysquares.Andthatsendsastrong message,that nomatterwhathappenstoAmerica,shewillalwaysrebound-- withthemost powerfully stagedphotoopsintheworld.WewhohavebeenwatchingStephen Colbertevisceratepoliticiansthathavecomeonhisshowknewhewasagifted comedian.ButittookSaturday'sdinnertodemonstratehowincrediblyeffective the artformColberthaschosenisforexposingthePotemkinRegimeBushandhis henchmenhavecreated.Roveandtherightwingmachinehavenoanswertothe performancebuttosay"itbombed", "itwasn'tfunny",andtohopethatbyignoring it,thecausticcleansingagentithaslobbedintotheircampcanbecontained.Yet, theRepublicanspinmeistersarethemastersofspin.”[2] This% http://dailykos.com/storyonly/2006/4/30/1441/59811 Np=8,Nn=4 ;Polarity=Np – Nn /Np +Nn =0.33

[2]http://www.pacificviews.org/weblog/archives/001989.html

PropagatingInfluence PropagatingInfluence

• BasedonworkofGuhaetal[1] formodelingpropagationof

trustanddistrust

• Framework
• Mijrepresentsinfluence/biasfromuseritoj.(0<=Mij <=1)
• Mij isinitializedtothepolarityfromitoj.
• BeliefMatrix#representstheinitialsetofknownbeliefs,andis

sparse

• GoalistocomputeallunknownvaluesinM
• BeliefMatrixafterith atomicpropagation
• Mi+1 =Mi *Ci
• CombinedOperator
• Ci=a1 *M+a2 *MT*M+a3 *MT +a4 *M*MT
• a{0.4,0.4,0.1,0.1}representsweighingfactor

[1]GuhaR,KumarR,RaghavanP,TomkinsA.Propagationoftrust anddistrust.In:$ %&&&'(!)!)*+,#-.//01,(#$.//01

Experiments Experiments

• Domain
• PoliticalBlogosphere
• DatasetfromBuzzmetrics[2] providespost-postlinkstructureover14million

posts

• Fewoff-the-topicpostshelpaggregation
• Potentialbusinessvalue
• ReferenceDataset
• Hand-labeleddatasetfromLadaAdamicetal[3] classifyingpoliticalblogsinto

rightandleftleaningbloggers

• Timeframe:2004presidentialelections,over1500blogsanalyzed
• Overlapof300blogsbetweenBuzzmetricsandreferencedataset
• Goal
• ClassifytheblogsinBuzzmetricsdatasetasdemocratandrepublicanand

comparewithreferencedataset

[2]LadaA.AdamicandNatalieGlance,"Thepoliticalblogosphereandthe2004USElection",inProceedingsoftheWWW-2005Workshop Buzzmetrics– www.buzzmetrics.com

EvaluationMetrics EvaluationMetrics

Confusion Matrix

• Accuracy=73%
• TruePositiveRate(Recall)=

78%

• FalsePositiveRate(FP)=

31%

• TrueNegativeRate(Recall)

=69%

• FalseNegativeRate(FN)=

21%

• Precision(R)=75%
• Precision(D)=72%
• 2

PolarityImprovesClassificationbyalmost26%

SampleData SampleData

• Trustpropagation

compensatesforinitial incorrect polarity(DK– AT)

• Trustpropagationdoesnot

changecorrect polarity(AT- DK)

• Trustpropagationassigns

correctpolarityfornon- existent directlinks(AT-IP)

• Numbersin

problematic(MM-AT)

• Improvesentimentdetection?

MSMClassificationResults MSMClassificationResults

InterestingObservations InterestingObservations

• 24outof27sourcesclassified

“correctly”

• guardian,foxnews,

humaneventsonline,mediamatters

• MainOutliers-- “thenation” and

“bostonglobe”

• Bothleftandrightleaningblogs

talknegativelyabout“nytimes” and“abcnews” andpositively about“rawstory” and“examiner”

IdentifyingBiasusingKLDivergence IdentifyingBiasusingKLDivergence

20 40 60 80 100 120 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

NumberofSplogs

Splogsspoiltheparty Splogsspoiltheparty

Splogsintop100search results.Eachlinerepresentsa query.

&! !

PreliminaryWork:OpinionExtraction

SPLOGS! SPLOGS!

• SPLOGS BY NUMBERS
• 75%ofupdatepings(eBiquity2006)
• 20%ofindexedBlogosphere(Umbria2006)
• 56%ofupdatepings(eBiquity2007)

• SPLOG-2005
• SampledSummer2005atTechnorati
• Asearchengine,somanysplogsalreadyremoved
• Labeledsamplesof700blogsand700splogs
• OnlyBlog-homepages
• SPLOG-2006
• SampledOct2006atWeblogs.com
• Labeledsamplesof750blogsand750splogs
• Blog-homepages+feeds

DATASETS DATASETS

EXPERIMENTAL SETUP EXPERIMENTAL SETUP

• Binaryfeatureencoding
• Top50Kselectedusingfrequencycount
• SVMs

– Defaultparameters – LinearKernel

• Nostemmingorstopwordelimination
• NaïveBayes
• Tenfoldcross-validation

URL

2005 2006

URL

• 3,4,5charactergramsfromURL
• Capturesprofitablecontexts
• Highlyeffectiveatpingstreams
• Supportsanextremelylowcostclassifier

2005 2006

WORDS

2005 2006

WORDS

2005 2006

• Words(Text)onaBlog
• Previouslyeffectiveintopicclassification
• Capturesprofitableadvertisingcontexts
• InterestingAuthenticGenreObserved

OUTLINKS

2005 2006

OUTLINKS

2005 2006

• Out-linkstokenizedbynon-alphabets
• SimilartoURLn-grams,likelymorerobust
• Novelfeaturespace

ANCHORS

2005 2006

ANCHORS

2005 2006

• Anchortexttokenizedintowords
• Subsumedbywords,butobfuscationdifficult
• Capturepersonalizationofpublishingtemplate
• Novelfeaturespace

• !"#$%&&'()

"*+"*', #,,, $-. ,#$/0" $,",(1$ #$2$ 3$/"$'. 42/#!!! */&/ ",*/5!.

Splog software ?! Splog software ?!

!

"#$%&'()#*

HTMLTAGS

2005 2006

HTMLTAGS

2005 2006

• UseHTMLTags– stylisticinformation
• Capturesignaturesofsplogsoftware
• Fullylanguageindependent
• Novelfeaturespace

BLOG IDENTIFIER BLOG IDENTIFIER LANGUAGE IDENTIFIER LANGUAGE IDENTIFIER

• REGULAR

EXPRESSIONS REGULAR EXPRESSIONS BLACKLISTS WHITELISTS BLACKLISTS WHITELISTS URL FILTERS URL FILTERS HOMEPAGE FILTERS HOMEPAGE FILTERS FEED FILTERS FEED FILTERS

• Ping Stream

Ping Stream Ping Stream

META-PING SYSTEM

Increasing Cost

THEGAMETHEORETICWEB THEGAMETHEORETICWEB

QouthPeterNorvig QouthPeterNorvig

• “TheotherthingthatIhadn'treallythoughtaboutwhenwe

startedthisallishowkindofgametheoreticthewholething

• is. Atfirstwethoughtofourselvesasthisobserverofthe
• Web. ThattheWebwasoutthereandwemadeacopyofit

andindexeditandifpeoplewantedtheycouldcomeand accessthatindex.ButitwasjustareflectionoftheWebout

• there. Andnowweunderstandthatwe'reco-evolvingwith

theWebandthatwhenwemakeamoveitchangestheWeb andwhentheWebchangeswechangeandgoingbackand forth.Andsoallthesearchengineoptimizersandsoonare watchingandwhatwedoandwewatchwhattheydoandthe

• Webistheinteractionbetweenus. AndthatissomethingI

hadn'tevenconsideredbeforewesawithappening.”

• %+-.//3

%+-.//3

ThisistrueofSocialMediaaswell ThisistrueofSocialMediaaswell

• IfIknowthatyouareoutthere,tryingtoinfermy
• pinions(orpreventmefromspamming)thenIwill

activelyworktodefeatthat.Sincethecontentisuser generated,Icandothatfairlyquickly.

• Spamadaptationisaclassicexample.

• Changeindistributioninfeaturespace
• ConceptDrift– Seasonal,seeninbothsplogsand

blogs

• AdversarialScenario– seeninsplogs
• ConceptDescriptionneedstobeupdated

ADAPTIVE CONTEXT ADAPTIVE CONTEXT

f1,f2,f3 ..fm

P(O(x)/splog(x)) P(splog(x)/O(x))

ENSEMBLE INTUITION ENSEMBLE INTUITION

• Streamofunlabeledinstances(drifting)
• Baseclassifierswithpotentially

independentfeaturespaces

• Isanensemble(probabilisticcommittee)
• fthecataloguemorerobusttodrift?
• Areinstancesclassifiedbytheensemble

effectivetoretrainbaseclassifiers(semi- supervisedlearning)?

• Motivatedbyco-training

ENSEMBLE INTUITION ENSEMBLE INTUITION

baseclassifiers updatedclassifiers ensemblecommittee (probabilistic)

classify classify classify retrain

unlabeled instances

POTENTIAL TO ADAPT POTENTIAL TO ADAPT

URL Anchor Chargram Outlink Tag Wordgrams Words

EXPERIMENTAL SETUP EXPERIMENTAL SETUP

• Acatalogofsevenclassifiers
• SPLOG-2005asbaselabeleddataset
• SPLOG-2006asevaluationstream
• 10KTopFeatures
• SVMbasedlearning
• SPLOG-2006separatedoutintounlabeled

streamandtestset(3-fold)

• F-1performancemetricevaluation

RESULTS – WORD DRIFT RESULTS – WORD DRIFT

RESULTS – ALL CLASSIFIERS RESULTS – ALL CLASSIFIERS

Conclusion Conclusion

• Using

wecandevelopanaccuratemodelfor influence,biasandtrustontheBlogosphere.

• Weapplythisframeworkonreal-worlddataand

describetechniquesforidentifyinginfluenceonthe Blogosphere.

• Splogsareabigissue– wehavedevelopedefficient

techniquestodetecttheminnearrealtime.

• DoestheGameTheoreticNatureofthissystem

raisefundamentalnewchallengesforDataMining.

BackupSlides

Graphsareeverywhere..andsoarePowerlaws!!

InternetMappingProject [lumeta.com]

FriendshipNetwork[Moody‘01]

Insimplewords,powerlawcanbeexplainedby“richget richerphenomenon” OR“20%ofthepopulationholds 80%ofthewealth” Consideringwebasagraph: Scale%freenetwork: Structureandproperties independentofnetworksize Fewhighconnectivity node(hubs)

///!&

Propertiesofinterest(graphtheory) Averagedegreeofnode,degreedistribution,degreecorrelation, distributionof strongly/weaklyconnectedcomponents,clusteringcoefficientand reciprocity

GenerativeModelsforBlogosphere GenerativeModelsforBlogosphere

• Reducetimetogeneratedata
• crawlingtheblogosphereoverafewweeks
• samplingtherightblogstogetarepresentativesample
• Reducetimeinpreprocessinganddatacleaning
• removinglinkspointingoutsidethedataset,outsidethetimeframe
• splogremoval[1]
• Generategraphsofdifferentproperties\sizes
• averagedegreeofnode,degreedistributions
• Testingofnewalgorithmsforbloggraphs
• e.g.spreadofinfluenceinblogosphere[2],communitydetection[3]
• Extrapolation
• howwillfastgrowthaffecttheblogosphereproperties?
• howdoesthisaffecttheconnectedcomponents?

[1]Kolarietal“Svmsfortheblogosphere:Blogidentificationandsplogdetection,” inAAAISpringSymposiumonComputational ApproachestoAnalyzingWeblogs,2006. [2]Javaetal“Modelingthespreadofinfluenceontheblogosphere,” tech.rep.,UniversityofMaryland,BaltimoreCounty,March2006. [3]Linetal“DiscoveryofBlogCommunitiesbasedonMutualAwareness

GenerativeModelsforBlogosphere GenerativeModelsforBlogosphere

+,- ..

• *.

/.

PreferentialAttachment:Thelikelihoodoflinkingtoapopularwebsiteishigher

0/&'#1!&'''2'/3!# "%% 4#",,,,65.!!#"%% 678!39#:'#;<&!!#0:&'#&'=!#;&!&'&9'&&!#( 7"&#"%%6 ">?#1!'#&'8&#83'&! ''!! ( 7"&#"%%6

• Twolevelnetwork:blogandpostlevel
• Inlinksandoutlinkstoandfromposts
• NEEDtomodelbloggerinteractions

ExistingApproaches ExistingApproaches

1. Probabilityofrandomreads(rR) 2. Probabilityofrandomlyselectingwriter(rW) 3. Probabilitythatnewnodedoesnotlinktotheexisting network(pD) 4. Growthexponent(g)

– howmanylinksshouldbeaddedeverystep?

ModelParameters ModelParameters

?@'3!'A @ !/A BB5'& &!' ''

1.Addnewblognode 2.Selectwriter 3.Writersreadblogposts,write posts

C!' &' AD4 BB5'& &!'A ?@& E &' A E '& A @/''3 &' 'F 4&'/ 4&' !'&' Reciprocallinks Stronglyconnectedcomponents Subsetofnodeshavingdirected pathfromeverynodetoeveryother node Weaklyconnectedcomponents Informationflow

(0 (0

&3' & 3!

ProposedModel ProposedModel

PropertiesofSimulatedBlogGraphs PropertiesofSimulatedBlogGraphs

Effectoftextwindowsize Effectoftextwindowsize

• Optimalwindowsizeis750charactersforourexperiments
• Smallwindowsize– Non-opinionatedphrases
• LargeWindowsize– Analysisofnon-relatedtext
• Specifictoourexperiments,numbersmaynotbegeneralized

Effectofatomicpropagationparameters Effectofatomicpropagationparameters

• X-axisBitset={directtrust,co–citation,transposetrustand

trustcoupling}={0001- 1111}

• Eachparametersettoeither0oritsoptimalvalue
• Collectiveinfluenceofallparametershelps!

AtomicPropagation AtomicPropagation

• DirectPropagation
• Given:AtrustsBandBtrustsC
• Implies:AtrustsC
• Operator:M
• Co-citation
• Given:AtrustsBandC,DtrustC
• Implies:DtrustsB
• Operator:MT *M
• ;
• ;
• .

AtomicPropagationContd… AtomicPropagationContd…

• TransposeTrust
• Given:AtrustsBandCtrustsB
• Implies:CtrustsA,AtrustsC
• Operator:MT
• TrustCoupling
• Given:DtrustsA,AtrustsC

andBtrustsC

• Implies:DtrustsB
• Operator:M*MT
• ;
• .

;

AtomicPropagationcontd… AtomicPropagationcontd…

• CombinedOperator
• Ci=a1 *M+a2 *MT*M+a3 *MT +a4 *M*MT
• ai {0.4,0.4,0.1,0.1}representsweighingfactor
• BeliefMatrixafterith atomicpropagation
• Mi+1 =Mi *Ci
• Weperformpropagationstill“convergence” (tillthe

newiterationdoesnotchangevaluesinMabove “threshold”)

SeparatingBlogWheatfromBlogChaff SeparatingBlogWheatfromBlogChaff Datacleaningfor

• Splogremoval
• Postcontentidentification

Non English Blog removal Non English Blog removal

• Collection Parsing

Collection Parsing

• Splog Detection

Splog Detection

• Title and

Content Extraction Title and Content Extraction

• Non English

Blog removal Non English Blog removal

• Collection Parsing

Collection Parsing

• Splog Detection

Splog Detection

• Title and

Content Extraction Title and Content Extraction

• BlogVox:SeparatingBlogWheatfromBlogChaff",IJCAI2007Analyticsof

NoisyandUnstructuredText

DataCleaning:Splogs DataCleaning:Splogs

HostAds Indexaffiliates, Promote pageRank Plagiarized content PreliminaryWork:OpinionExtraction

20 40 60 80 100 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 NumberofSplogs

EffectofSplogs EffectofSplogs

Distributionofsplogsfor ‘spamterms’ inTRECcorpus pregnancy insurance discount "BlogTrackOpenTask:SpamBlog Classification",4(.//56 !',

DataCleaning:ContentIdentification DataCleaning:ContentIdentification

• BaselineHeuristic
• SVMMethod

?&'4??

1

0/!<!

D'?

14! <4

'#)

D'?"DC8 .3&G 2/ + +. (.3%

D'?D' *DC8,

14

@' 1!

• 4

D' H @'!&'/! 12?& 4.I8I ?/@'2

• "

$ ) 6

• J
• K

%

• "
• K

$

4?? &

BlogVoxOpinionExtractionSystem BlogVoxOpinionExtractionSystem

• TREC06:Finding

posts,eitherpositiveornegative, aboutaquery

• 2006TRECBlogcorpus:
• 80Kblogs
• 300Kposts
• 50testqueries
• BlogVox opinionextraction

system

• Documentandsentencelevel

scorers

• Combinedscoresusingan

SVMmeta-learner

• Datacleaning:splogsand

postidentification

BlogVox

BrandMonitoring/BusinessAnalytics BrandMonitoring/BusinessAnalytics

Limitations

• Proprietary
• Somecompaniesconductextensivemanualresearch

BlogAnalytics/MarketIntelligence Buzz Opinions Influence Reputation Competition FinancialAnalyst

TopCitedMediaSources TopCitedMediaSources

TopMSMSourcesontheBlogosphere

PropagatingInfluence PropagatingInfluence

• Trust-only
• Ignoredistrust(negativepolarities)completely
• FinalBeliefMatrix=Mk ,M0 =T
• (K:Numberofatomicpropagationstillconvergence)
• One-stepDistrust
• Distrustpropagatessinglestepwhiletrustpropagatesrepeatedly
• FinalBeliefMatrix=Mk *(T-D),M0 =T
• (K:Numberofatomicpropagationstillconvergence)
• PropagatedDistrust
• Treatdistrustandtrustequivalent
• FinalBeliefMatrix=Mk ,M0 =T- D
• (K:Numberofatomicpropagationstillconvergence)

SPAMDEXING SPAMDEXING

?&&!# ?&!# ?&;'!# :!3?& C3?& ?G45 ?&G''! &5&! ;'2! &!&&'3! && 'E'3!

.5

7&9&? 4 &5& ! ?&&!# ?&! 6177-389 ?& /' &'!

(i) (ii) (iii)

WORDGRAMS

2005 2006

WORDGRAMS

2005 2006

• Word-2-grams,2adjacentwords
• ShallowNLPtechniquetotacklewordsalad
• Wordsaladlesscommoninwebspam(TFIDF)
• Word-x-gramfeatures,exponentialwithx

CHARACTERGRAMS

2005 2006

CHARACTERGRAMS

2005 2006

• 3,4,5charactergramsfromblogcontent
• Cancapturecharactersalad(e.g.p1lls)
• Featureselectionimportant