Investigating Collaboration Dynamics in Different Ontology - PowerPoint PPT Presentation

Investigating Collaboration Dynamics in Different Ontology Development Environments Marco Rospocher Tania Tudorache and Mark Musen DKM, Fondazione Bruno Kessler BMIR, Stanford University rospocher@fbk.eu tudorache@stanford.edu, musen@stanford.edu The 7th International Conference on Knowledge Science, Engineering and Management - KSEM 2014 16–18 October 2014, Sibiu, Romania

Introduction • Understanding processes and dynamics behind the collaborative development of ontologies is important • for Ontology tool engineers • to understand how to optimize their tools to make the work of the users more straightforward and effective • for Ontology project managers • to obtain tools and metrics to assess and monitor the development status and the quality of the ontology under their responsibility Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Our Contribution • We conducted some exploratory investigations on • the way people edit an ontology in collaborative settings • the role of discussion activities in collaborative ontology development • Novelty: • two different ontology development frameworks • discussion activities • Key Aspects: • analysis based on truly objective data • five real ontology development projects Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Investigations I1. Is the editing process localized? I2. Is the formalization of an ontology entity truly collaborative? I3. Are discussed ontology entities actually discussed by two or more users? I4. Are highly discussed ontology entities also highly edited? I5. Do users tend to edit more than to discuss? Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Tools: • A collaborative ontology authoring tool for the Web • Form-based mechanism • Extensive collaboration support • tracking of all changes that users perform in a structured log • notes and (threaded) discussions • Ontology entity/branch watch mechanism (with email notifications) • highly configurable access policies • In this study we used iCAT • a custom configuration of WebProtégé used by medical experts Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Tools: Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Tools: • A collaborative wiki-based tool for modeling ontological and procedural knowledge • Form-based editing • Different ways to navigate the ontology under development • class hierarchy tree / list in a tabular form all entities defined in the ontology / search for a specific entity • Support for user collaboration • discussions, by means of talk pages • watchlists and notifications • recent activity awareness features Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Tools: Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

and • Main similarities web-based • • functionalities for supporting both editing and discussion user editing mainly via forms • similar collaboration features (e.g., notification, watchlist, history) • • Main differences • granularity and the modality of editing and discussion activities • navigation and hierarchy awareness • discussion awareness • Motivations for using WebProtègè and MoKi in our study • provide detailed change and discussion logs • used in several real-world projects Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Ontologies • WebProtégé Ontology Development projects considered: The 11th Revision of the International Classification of Diseases (ICD) • led by the World Health Organization (WHO) • a taxonomy and descriptions of diseases used in United Nations countries • The International Classification of Traditional Medicine (ICTM) • • led by the World Health Organization (WHO) • standard terminology and classification for diagnoses and interventions in Traditional Medicine • 4 languages covered: English, Chinese, Japanese and Korean Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Ontologies • MoKi Ontology Development projects considered: • Organic Agriculture (OA) • classify educational material in a multilingual web-portal containing organic agriculture and agro-ecology resources • 15 languages • Viticulture (Vit) • concepts related to the science, production, and study of grapes • no discussion functionalities • Motivation and Emotion (ME) • motivational and emotional aspects of the learning process in pedagogy • educational material and the interventions to be used for facing motivational or emotional difficulties Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Ontologies ICD ICTM OA Vit ME Developed with No. of ontology: classes 50,609 1,511 284 481 72 ● ● ● ● ● ● individuals 228,629 18,364 81 0 13 ● ● ● ● ● ● properties 228 219 31 0 13 ● ● ● ● ● ● No. of active users 109 23 10 3 3 No. of edits 331,147 40,840 2,915 2,227 407 No. of discussions 71,371 1,726 452 0 52 Status ongoing ongoing ongoing ongoing completed Development period (ca) 42 months 30 months 5 months 3 months 1 week Used for investigations All All All I1, I2 All Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Results I1: Is the editing process localized? • We tested whether users, after editing a class A, tend to edit another class B closely or semantically related to the previous one • Six cases considered: siblings child parent A B A B B A none descendant ancestor A B B B A A • We counted the number of these occurrences, normalizing over the total number of cases Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Results I1: Is the editing process localized? • Most of the times (60% to 73%), next edited entity is a sibling, a child or a parent Exception: OA • strong multilingual focus • editing following the • alphabetically-sorted list of concepts • Outcome: users tend to work locally on the ontology may be due to class navigation functionalities, similar yet different • in the tools • (to be further investigated) Do tool functionalities impact the way people perform their editing activities? Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Results I2: Is the editing truly collaborative? • We examined how many distinct users usually edit an ontology entity, whether a class, individual, or property • We classified ontology entities in three categories edited by only one user • edited by two distinct users • edited by three or more distinct users • Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Results I2: Is the editing truly collaborative? • Most of the ontology entities (75% to 96%) edited by at most 2 users • Exception: OA • 65% of the entities edited by at least five distinct users strong multilingual focus • rather low entities / user ratio • ( ∼ 40), multiple users editing activities more likely to occur • This kind of analysis may provide useful insights also to ontology project managers: • to detect entities having a very few number of editors (may require some intervention) Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Results I3: Are discussions truly collaborative? • We examined how many distinct users usually discuss an ontology entity, whether a class, individual, or property • Similarly to I2, we classified ontology entities in three categories discussed by only one user • discussed by two distinct users • discussed by three or more distinct users • Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen

Results I3: Are discussions truly collaborative? • ICD and ICTM: most of the ontology entities ( ∼ 91%) are discussed by a single users • OA and ME: most of the ontology entities (resp., 97% and 75%) are discussed by at least 2 users • Possible explanations: • difference in size of the ontology and the number of users • entity / user ratio (ICD: ∼ 382 and ICTM: ∼ 171; OA: ∼ 40 and ME: ∼ 33) • different discussion-awareness support provided by the tools • ICD and ICTM have used the notes mechanism not only for discussion, but mostly for notes or additional documentation Investigating Collaboration Dynamics in Different Ontology Development Environments - Rospocher, Tudorache, Musen