Group Communication Analysis A computational linguistics approach - PowerPoint PPT Presentation

Group Communication Analysis A computational linguistics approach for detecting sociocognitive roles in multi-party interactions Dr. Nia Dowell Postdoctoral Research Fellow School of Information Digital Innovation Greenhouse University of Michigan ndowell@umich.edu http://niadowell.com

Social Roles Research Business Psychology Sociology Social Education Management Computing Social Collaborative Collaborative Work Place Group Team Media Problems Learning Meetings Cognition Effectiveness Solving

Prominent Perspectives on Roles A position to which a person is assigned and Assigned Roles then performs the behavior associated with that position Role Concept Concerns Dysfunctional group roles • What is actually captured in role assignment research? • A single role inhibits role and group flexibility, and the potential advantages of this • • Disregards the dynamic and interactive way in which roles are created, negotiated, and evolve among group members during social interaction

Prominent Perspectives on Roles A position to which a person is assigned and Assigned Roles then performs the behavior associated with that position Role Concept Develop naturally out of the interpersonal interaction without any prior instruction or Emergent Roles assignment, and are characterized by their behavioral proximity (similarities and differences) to other interactional partners

High Effort High Effort Over-rider Captain Group “I” “We” Orientation Ghost Free-rider Low Effort Low Effort Strijbos & De Laat (2010) Marcos-Garcia et al., 2015

Can we automatically identify the roles students take on during collaborative interactions? Sociocognitive Conversational Social Patterns Processes Roles Language Online Multi- and party Discourse Interaction

Discourse Speaker Time How do we go from this semi-unstructured data to something meaningful, something that allows us to capture the important sociocognitive processes taking place within the interaction. In Infer semanti tic relati tionship among stu tudents ts’ co contributions

Discourse Cohesion Latent Semantic Analysis This similarity measure represents the semantic and conceptual meanings of individual words, utterances, texts, and larger stretches of discourse based on the statistical regularities between words in a large corpus of naturalistic text

Overall Responsivity Participation Internal Cohesion Responsivity Dynamics Measures Social Impact Group Communication Analysis Newness Discourse Cohesion Analyses Communication Density

Jamie Pennebaker + Team

Participants: 840 undergraduates in an introductory-level psychology course Groups: 184 randomly assigned groups Talking Questions

Measuring Performance Group Proportion of on-topic Level discussion Student Pre-test Post-test Level [% Posttest - % Pretest] / [1 - % Pretest]

Detecting Emergent Roles Pre Clustering Testing Training Multicollinearity Cluster Tendency Hopkins statistic = .15

Majority rule Optimal Number of Clusters Op Optimal number of clusters using PAM 10 Frequency among all indices 8 WSS 6 4 Total within sum of squares 2 1000 0 800 2 3 4 5 6 9 10 Number of Clusters k Optimal number of clusters using K-means 600 10 400 Frequency among all indices 8 200 1 2 3 4 5 6 7 8 9 10 6 Number of Cluster k 4 The disadvantage of elbow and similar methods is 2 that, they measure a global clustering characteristic only 0 2 3 4 5 6 10 Number of Clusters k

Cluster Evaluation and Validation Internal Validation 4 Cluster Model Stability Validation & Theoretical Justification 6 Cluster Model External Validation

From Model to Meaning Socially- Soci Task-Le Ta Leader Ov Over-rid rider Driver Dr er Fo Follower Lurker Lu De Detached ed 0.8 0.6 0.4 0.2 CENTROIDS 0 -0.2 -0.4 -0.6 -0.8 Participation Social Impact Overall Responsivity Internal Cohesion Newness Communication Density

Student Roles and Learning

Linear Mixed Effect Dependent Independent Random Models Variables Variables Variables Proportional learning Identified roles Learner and Group Individual Learner gains Proportion of topic- Group Proportional Group relevant discussion occurrence of each identified role

Linear Mixed Effect Dependent Random Models Variables Variables Proportional learning Learner and Group Individual Learner gains Proportion of topic- Group Group relevant discussion Null Models

Li Linear r Mixed Effect Mo Models Ev Evaluation Akaike Information Criterion (AIC) Log Likelihood (LL) Likelihood ratio test Marginal ( R 2 m ) Conditional ( R 2 c )

Ho How do do le learne ners’ s’ ro roles in influe luenc nce indiv individua idual l le learne rners’ s’ pe perf rform rmanc nce? 0.4 Driver Task-Leader Over-rider Lurker Follower Socially Detached 0.3 ** 0.2 0.1 LME COEFFICIENTS (B) 0 -0.1 -0.2 -0.3 * -0.4 ** ** -0.5 * p < .05, ** p < .01, *** p < .001; N = 704 χ2(7) = 14.93, p = .001, R 2m = .02, R 2c = .95.

Ho How do do le learne ners’ s’ ro roles in influe luenc nce overall ll group up pe perf rform rmanc nce? Unproductive roles model Productive roles model Drivers Task-Leaders Socially-Detached Over-riders Lurkers Followers Drivers Task-Leaders Socially-Detached Over-riders Lurkers Followers 2 ** ** 2 1.5 ** ** 1.5 1 1 LME COEFFICIENTS (B) 0.5 0.5 0 0 -0.5 -0.5 -1 -1 -1.5 -1.5 -2 -2 * * * * * * -2.5 -2.5 χ2(3) = 23.62, p < .001, R 2m = .15, R 2c = .90 χ2(3) = 20.92 p < .001, R 2m = .13, R 2c = .89

Take Home Driver • Roles influence student and group outcomes Task-leader • Drivers > Lurkers Socially- • Drivers = Task leaders and Socially-detached detached Over-rider learners Follower • Difference in learning is not a result of the Lurker students simply being more prolific • Optimal group composition ≠ simply high participating learners • Optimal group composition = high and low participators aware of and invested in the social climate of the group interaction • Effect size differences

How well the identified clusters generalize to held out and completely different computer-mediated collaborative learning contexts?

SMOC: Synchronous Massive Online Class • Intro psychology course • Similar to the Traditional CSCL dataset, but • Students randomly assigned to groups larger and more distributed in terms of • 200-300 groups of 4-5 students per day people and topics • • Students were in 9 chats groups learner N = 1,713, group N = 3,380 • Interactions last 3-9 minutes, averaging 5 throughout the semester minutes • Over 26 different chat topics

Land Science: A Virtual Internship • Land Science is an interactive urban-planning simulation with collaborative problem-solving in an simulation environment • Interns receive instructions and coaching from Mentors • Interns participate in collaborative problem solving chat sessions to achieve collective goals • learner N = 38, group N = 630

Traditional CSCL SMOC Land Science Traditional CSCL SMOC Land Science Training Data Training Data Training Data Land Science Traditional CSCL SMOC Testing Data Testing Data Testing Data

Traditional CSCL Traditional CSCL Training Testing Predict SMOC cluster Land Science

Pr Prediction Evaluat ation Cross-tabulation assessment Adjusted Rand Index (ARI) • computes the proportion of agreement between 2 cluster partitions & penalizes for any randomness in the overlap • Steinley (2004) considers ARI values greater than 0.90 - excellent, values greater than 0.80 - good, values greater than 0.65 - moderate, and values less than 0.65 - poor Cramer V Effect size for the strength of the relationship between 2 cluster partitions •

Predict Traditional CSCL Traditional CSCL Training Testing Training Testing

Predict Traditional CSCL Traditional CSCL Training Testing ARI = .83; Cramer V = .92 Cross-tabulation of the predicted and actual cluster assignments Testing Training Predicted Clusters Clusters Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6 32 0 0 0 0 0 Cluster 1 2 29 0 0 0 0 Cluster 2 0 0 15 2 1 0 Cluster 3 0 0 0 18 0 0 Cluster 4 4 0 0 1 13 0 Cluster 5 0 0 0 0 0 19 Cluster 6

Traditional CSCL SMOC Land Science Traditional CSCL SMOC Land Science Training Data Training Data Training Data Land Science Traditional CSCL SMOC Testing Data Testing Data Testing Data

Internal & External Generalization Six-Cluster Model 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Predictor Traditional Predictor SMOC Predictor Land Science CSCL Traditional CSCL SMOC Land Science

Internal & External Generalization Six-Cluster Model 1 0.9 0.8 Adjusted Rand Index 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Predictor Predictor Predictor Land Traditional SMOC Science CSCL Traditional CSCL SMOC Land Science

Internal & External Generalization Six-Cluster Model 1 0.9 0.8 Adjusted Rand Index 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Predictor Predictor Predictor Land Traditional SMOC Science CSCL Traditional CSCL SMOC Land Science