ET-805 Cohens Kappa Ramkumar.Rajendran@iitb.ac.in From Last Class - - PowerPoint PPT Presentation

ET-805 Cohen’s Kappa

Ramkumar.Rajendran@iitb.ac.in

From Last Class

• Modeling Learner’s affective state
• Emotionally Intelligent Tutoring Agents

Muddy Points

• What is Kappa?
• Multi-class classification problem
• Predicting user behavior and interest using one pattern!

2

Cohen’s Kappa

• Compares Observed and Expected outcome
• Used to measure inter-rater reliability
• More robust than simple Accuracy to compare the

performance

• Important for unbalanced data classification

3

Activity - Solve a problem

We develop a system to detect students’ frustration using log data from a learning

• environment. We used human observation as a label and created a classifier. Below is the

contingency table. Calculate Accuracy (3 mins)

4

Frustrated Not Frustrated Frustrated 56 TP 15 FP Not Frustrated 10 FN 44 TN

Observed (Classifier Output) Expected(Human Obs.)

Activity - Solve a problem

We develop a system to detect students’ frustration using log data from a learning

• environment. We used human observation as a label and created a classifier. Below is the

contingency table. Calculate Precision and Recall? Precision = TP / (TP+ FP) Recall = TP / (TP+ FN)

5

Frustrated Not Frustrated Frustrated 56 TP 15 FP Not Frustrated 10 FN 44 TN

Observed (Classifier Output) Expected(Human Obs.)

Activity - Results

Accuracy = 100 / 125 = 0.8 Precision = 56 / (56 + 15) = Recall = 56 / (56 + 10) =

6

Activity - Problem

For the below table Calculate Accuracy? = 0.8

7

Frustrated Not Frustrated Frustrated 30 TP 68 FP Not Frustrated 20 FN 322 TN

Observed (Classifier Output) Expected(Human Obs.)

Activity - TPS

Think Based on the accuracy you have calculated, which classifier you will choose for your study! Why? Write 2 points to justify your choice (3 mins) Share

• Precision of classifier 1 is higher than classifier 2
• Recall is also higher in Classifier1

8

Cohen’s Kappa

Kappa = (Observed accuracy - Expected accuracy) / (1 - Expected accuracy) Observed Accuracy = (TP + TN ) / Total Expected Accuracy = Measures number of instances in each class agree with the ground truth

9

Cohen, Jacob (1960). A coefficient of agreement for nominal scales". Educational and Psychological Measurement. 20 (1): 37–46

Expected Accuracy Calculation

• Calculate marginal freq of each rater for each class

Frustrated = (10+5 )15 * 20 (5 + 15)/ 55 = 15 * 20 /55 = 5.45 !Frustrated = 35 * 40 / 55 = 25.45 Expected Accuracy = (5 .45 + 25.45)/55 = 0.56 Frus = 0 !Frus = 55*35/55 = 35 Expexted Acc = 35+0/55 = 0.63

10

Frustrated Not Frustrated Frustrated 5 TP 10 FP Not Frustrated 15 FN 25 TN Observed (Classifier Output) Expected(Human Obs.) Frustrated Not Frustrated Frustrated 0 TP 0 FP Not Frustrated 20 FN 35 TN

Kappa

Kappa = (Observed accuracy - Expected accuracy) / (1 - Expected accuracy)

Cohen’s Kappa = (0.54 - 0.56) / (1- 0.56) = -0.0 = 0.64 -

11

Activity - Discussion

• Example problems to show importance of Kappa for

unbalanced dataset

• 100 instance - 10 F + 90 NF
• Compute Kappa for classifier if all instance are classified

as NF.

• Compute Kappa if Classifier predicts 2 instance of F

correctly and rest all as NF

• Meaning of Kappa value

12

Multiclass Classification

• Supervised and Unsupervised classifiers
• Binary and Multiclass Classification

Binary - Yes or No, Frustration or No frustration, etc Multiclass - Bored, Frustrated, and Neutral

13

Multiclass Classifier - One vs All

14

F B N F 10 5 5 B 5 15 10 N 15 10 90 Accuracy = 115 / 165 = 0.7 F NF F 10 10 NF 20 125 Accuracy = 135 / 165 = 0.82

Activity - Discussion

• What is your inference from One vs All Multiclass classification
• One vs Rest
• 15

Reading Work

• F score
• AUC
• ROC

16

Second Assignment

• Only 4 submission
• Modeling part is ok, SPM, comparing high vs low performing

student’s actions, patterns that provide different style of learning.

• Ped part is weak. Write couple of examples or what ped logic

you will develop for the student model you developed

17

Course Project

• GIFT
• Topic: Math for class 6 NCERT book

18

Last Activity - Muddy Points

List down

• two important and
• two least clear

(muddy) points from today’s class

• https://tinyurl.com/et8

05mp

19