ET-805 Cohens Kappa Ramkumar.Rajendran@iitb.ac.in From Last Class - PowerPoint PPT Presentation

ET-805 Cohen’s Kappa Ramkumar.Rajendran@iitb.ac.in

From Last Class - Modeling Learner’s affective state - Emotionally Intelligent Tutoring Agents Muddy Points - What is Kappa? - Multi-class classification problem - Predicting user behavior and interest using one pattern! 2

Cohen’s Kappa - Compares Observed and Expected outcome - Used to measure inter-rater reliability - More robust than simple Accuracy to compare the performance - Important for unbalanced data classification 3

Activity - Solve a problem We develop a system to detect students’ frustration using log data from a learning environment. We used human observation as a label and created a classifier. Below is the contingency table. Expected(Human Obs.) Frustrated Not Observed Frustrated (Classifier Output) Frustrated 56 TP 15 FP Calculate Accuracy (3 mins) Not 10 FN 44 TN Frustrated 4

Activity - Solve a problem We develop a system to detect students’ frustration using log data from a learning environment. We used human observation as a label and created a classifier. Below is the contingency table. Expected(Human Obs.) Frustrated Not Observed Frustrated (Classifier Output) Frustrated 56 TP 15 FP Calculate Precision and Recall? Not 10 FN 44 TN Precision = TP / (TP+ FP) Frustrated Recall = TP / (TP+ FN) 5

Activity - Results Accuracy = 100 / 125 = 0.8 Precision = 56 / (56 + 15) = Recall = 56 / (56 + 10) = 6

Activity - Problem For the below table Expected(Human Obs.) Frustrated Not Observed Frustrated (Classifier Output) Frustrated 30 TP 68 FP Calculate Accuracy? Not 20 FN 322 TN = 0.8 Frustrated 7

Activity - TPS Think Based on the accuracy you have calculated, which classifier you will choose for your study! Why? Write 2 points to justify your choice (3 mins) Share - Precision of classifier 1 is higher than classifier 2 - Recall is also higher in Classifier1 8

Cohen’s Kappa Kappa = (Observed accuracy - Expected accuracy) / (1 - Expected accuracy) Observed Accuracy = (TP + TN ) / Total Expected Accuracy = Measures number of instances in each class agree with the ground truth Cohen, Jacob (1960). A coefficient of agreement for nominal scales". Educational and Psychological Measurement . 20 (1): 37–46 9

Expected Accuracy Calculation Expected(Human Obs.) - Calculate marginal freq of each rater for each class Frustrated Not Frustrated Observed Frustrated = (10+5 )15 * 20 (5 + 15)/ 55 (Classifier Frustrated 5 TP 10 FP Output) = 15 * 20 /55 = 5.45 Not 15 FN 25 TN !Frustrated = 35 * 40 / 55 = 25.45 Frustrated Expected Accuracy = (5 .45 + 25.45)/55 = 0.56 Frustrated Not Frustrated Frus = 0 Frustrated 0 TP 0 FP !Frus = 55*35/55 = 35 Not 20 FN 35 TN Expexted Acc = 35+0/55 = 0.63 Frustrated 10

Kappa Kappa = (Observed accuracy - Expected accuracy) / (1 - Expected accuracy) Cohen’s Kappa = (0.54 - 0.56) / (1- 0.56) = -0.0 = 0.64 - 11

Activity - Discussion - Example problems to show importance of Kappa for unbalanced dataset - 100 instance - 10 F + 90 NF - Compute Kappa for classifier if all instance are classified as NF. - Compute Kappa if Classifier predicts 2 instance of F correctly and rest all as NF - Meaning of Kappa value 12

Multiclass Classification - Supervised and Unsupervised classifiers - Binary and Multiclass Classification Binary - Yes or No, Frustration or No frustration, etc Multiclass - Bored, Frustrated, and Neutral 13

Multiclass Classifier - One vs All F B N F NF F 10 5 5 F 10 10 B 5 15 10 NF 20 125 N 15 10 90 Accuracy = 135 / 165 = 0.82 Accuracy = 115 / 165 = 0.7 14

Activity - Discussion - What is your inference from One vs All Multiclass classification - One vs Rest - 15

Reading Work F score ● AUC ● ROC ● 16

Second Assignment - Only 4 submission - Modeling part is ok, SPM, comparing high vs low performing student’s actions, patterns that provide different style of learning. - Ped part is weak. Write couple of examples or what ped logic you will develop for the student model you developed 17

Course Project - GIFT - Topic: Math for class 6 NCERT book 18

Last Activity - Muddy Points List down - two important and - two least clear (muddy) points from today’s class - https://tinyurl.com/et8 05mp 19