Ca Can mo model els le learned fr from a a da datase set re - - PDF document

4/20/18 1

Ca Can mo model els le learned fr from a a da datase set re refl flect ac acquisi3o 3on of proce

• cedural kno

knowl wledg dge?

An An ex experiment wi with au automa3c me measu sureme ment of

• f on
• nline

e re review qua quality

Mar$na Megasari, Pandu Wicaksono, Chiao Yun Li, Clément Chaussade, Shibo Cheng, Nicolas Labroche, Patrick Marcel, Verónika Peralta DOLAP 2018

Co Contribu)ons

q(Yet another) model of reviews helpfulness

A first

q

assessment of the skill of wri9ng helpful reviews Showing

q

that skill assessment makes sense even for models learned automa9cally

2

4/20/18 2

Review 1 Review n …

Helpfulness Helpfulness Reviewer

Pr Principle

3

Review 1 Review n …

Helpfulness Helpfulness Reviewer

Pr Principle

4

Skill acquisi6on model Probability the skill is acquired

4/20/18 3

Review 1 Review n …

Helpfulness Helpfulness Helpfulness model Feature extrac9on Reviewer Probability the skill is acquired

Pr Principle

Skill acquisition model Helpfulness model

5

Skill acquisi9on model Probability the skill is acquired

Tracin cing h help lpfuln lness f for a

• r a r

revie iewer

0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Helpfulness score

Posi:on in the sequence of reviews helpfulness (votes) 6

4/20/18 4

Tracin cing s skill of ill of t the r revie iewer r

0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Helpfulness score

Posi:on in the sequence of reviews helpfulness (votes) KT helpfulness 7

Tr Tracing helpfulness of the model learned

0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Helpfulness score

Position in the sequence of reviews helpfulness (votes) model of helpfulness KT helpfulness 8

4/20/18 5

Tr Tracing skill of the model learned

0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Helpfulness score

Posi:on in the sequence of reviews helpfulness (votes) model of helpfulness KT helpfulness KT model 9

Wh What do we need?

Skill acquisi+on model

q

Bayesian Knowledge Tracing §

Data

q

Amazon.com § book reviews

Model

q

Linear combina:on of features that par:cipate in helpfulness §

10

4/20/18 6

Sk Skill ill ac acquisi*on

• n: Ba

Bayes esia ian Kn Knowledge Tr Tracing (K (KT)

User-centric paradigm for evalua5ng procedural knowledge [Corbe' & Anderson, UMAI 1995] OUTPUT: P(Ln)

Probability that skill S is mastered aCer exercice n

Skill S

11

Hypo Hypotheses heses behi behind nd KT

It targets procedural knowledge

q

Knowledge § about how to do something Applica5on of § procedural knowledge may not be easily explained Different § from declara5ve knowledge, that is o?en verbalized

Problem resolu5on is binary

q

Pass/fail scheme §

No forge:ng

q

4 parameters usually set empirically

q 12

4/20/18 7

The The 4 pa parameters of

• f KT

KT

qP(L0): initial knowledge

§ Probability the skill is already mastered before the first problem

qP(T): transition from not mastered to mastered

§ Probability the skill will be learned at each new opportunity

qP(g): Guess

§ Probability the learner will guess correctly while the skill is not mastered

qP(s): Slip

§ Probability the learner will make a mistake while the skill is mastered

13

De Definition

Probability the skill is mastered at step n

q

P(Ln|Xn =xn) = P(Ln−1|Xn =xn) + (1 − P(Ln−1|Xn =xn))P(T) § § Intui3on : probability the skill is learned at step n-1 or not learned at step n-1 but learned at this step n

qWith

Xn = 1 means problem n resolved sucessfully, Xn=0 means not resolved § P(Ln−1|Xn =1) = P(Ln−1)(1 − P(s)) / (P (Ln−1)(1 − P(s)) + (1 − P (Ln−1))P(g)) §

intui3on: skill has been learned and used correctly / all cases of correct resolu3on

q

P(Ln−1|Xn =0) = P(Ln−1)P(s) / (P (Ln−1)P(s) + (1 − P (Ln−1))(1 − P(g)) ) §

14

4/20/18 8

KT KT extension ions we im imple lemented to

• fi

fit t ou

• ur

co conte text

qNon-binary problem resolution

§ KT with partial credits [Wang & Heffernan, AIED 2013]

qParameter learning avoiding local minimum, degenerate parameters

and computational costs

§ Estimating the most likely opportunity at which each individual learned the skill [Hawkins & al., ITS 2014]

qGithub link

§ https://github.com/Cubiccl/Continuous-Knowledge-Tracing/releases/tag/1.0

15

Da Data: a: Amaz mazon book re reviews [H [He & & Mc McAuley, , WWW 2016]

16

In

q

• ur context

the § skill is that of wri.ng helpful reviews each § wri5en review is treated like an

• pportunity to exercise the skill

Actual § helpfulness is the ra.o of helpful votes received by the review

Preprocessing

q

details in the paper

4/20/18 9

Fe Features & metrics for the model of help lpfuln lness

q16 features grouped in 3 categories

§ Conformity

q Rating, polarity, deviation to average rating

§ Understandability

q Spelling error ratio, 5 classical readability measures

§ Extensiveness

q Text and summary length

qConsistent with other models in the literature [Korfiatis & al., ECRA

2012]

§ More sophisticated models exist, but our point was to test a “simple” one

17

The The model del

Linear combina,on of feature scores

q

Learned with linear regression, perceptron,

q

SVM

Regression was the best compromise between 2me § and effec2veness Feature selec2on had no significant impact §

18

4/20/18 10

Te Tests

qImplementation

§ Java 8 § Weka 3.8 for model learning § SentiWordNet for polarity extraction § Stanford POS tagging library for part-of-speech tagging

q2 preprocessed datasets

§ minVotes = 12: 41,681 reviews § minVotes = 23: 11,083 reviews § In each dataset, reviewers have between 30 to 50 reviews

19

Te Tests

Helpfulness model accuracy is similar to the recent proposals of the

q

state-of-the-art

RMSE: the § error between the helpfulness model scores and the actual helpfulness scores

20

4/20/18 11

Te Tests

qUsing KTs

§ a-mKRMSE: error between the KT of the actual helpfulness scores and the KT of the helpfulness as computed with the model § a-AggKRMSE: error between the KT of the actual helpfulness scores and the aggregation of the KT scores of each feature taken independently (sub- skill)

21

Le Lesson

• ns le

learn rned & perspect ctiv ives

KT is op(mis(c and has an intrinsic smoothing behavior

q

Finer skills works be9er than coarser ones

q

Perspec(ves

q

Short term §

Tes+ng with more helpfulness models and skill acquisi+on models

q

Understanding

q

be;er the rela+onship between the linear coefficient learned for the helpfulness model and the KT parameters of the corresponding sub-skills

Longer § term

Applica+on to

q

• ther datasets, contexts and skills

Eg § , how to assess data explora+on, or how to assess deep learning’s produc+ons

22

4/20/18 12

Re References

q [Corbe' & Anderson, UMAI 1995]

Albert §

• T. Corbe,, John R. Anderson: Knowledge Tracing: Modelling the Acquisi>on of Procedural
• Knowledge. User Model. User-Adapt. Interact. 4(4): 253-278 (1995)

[Wang & Heffernan, AIED 2013]

q

Yutao § Wang, Neil T. Heffernan: Extending Knowledge Tracing to Allow Par>al Credit: Using Con>nuous versus Binary Nodes. AIED 2013: 181-188

[Hawkins & al., ITS 2014]

q

William J. Hawkins, § Neil T. Heffernan, Ryan Shaun Joazeiro de Baker: Learning Bayesian Knowledge Tracing Parameters with a Knowledge Heuris>c and Empirical Probabili>es. Intelligent Tutoring Systems2014: 150-155

[He &

q

McAuley, WWW 2016]

Ruining § He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolu>on of fashion trends with

• ne-class collabora>ve filtering. In WWW. 507–517.

q [KorfiaNs & al., ECRA 2012]

Nikolaos § Korfia>s, Elena García-Bariocanal, and Salvador Sánchez-Alonso. 2012. Evalua>ng content quality and helpfulness of online product reviews: The interplay of review helpfulness vs. review content. Electronic Commerce Research and Applica>ons 11, 3 (2012), 205–217. 23

Q& Q&A

24