Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions An Ontology Selection and Ranking System Based on the Analytic Hierarchy Process Adrian Groza 1 , Irina Dragoste 1 , Iulia Sincai 1 , Ioana Jimborean 1 , Vasile Moraru 2 1 Department of Computer Science, Technical University of Cluj-Napoca, Romania Adrian.Groza@cs.utcluj.ro 2 Department of Applied Informatics, Technical University of Moldova moraru@mail.utm.md September 24, 2014
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Outline Project Domain 1 Ontology Evaluation Analytic Hierarchy Process AHP adaptation for Ontology Evaluation 2 Criteria Tree Metrics for Atomic Criteria Including Negative Criteria Alternative Weight Elicitation Domain Coverage 3 System Design 4 Experiments 5 Conclusions 6
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Ontology Evaluation Ontology evaluation and selection MCDM problem (Multiple-Criteria-Decision-Making): domain coverage , size , consistency etc. both qualitative ( language expressivity ) and quantitative ( number of classes ) criteria both positive ( domain coverage ) and negative ( inconsistencies, unsatisfiable classes ) criteria depends on evaluation context (wide knowledge representation, efficiency, re-usability)
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Analytic Hierarchy Process Analytic Hierarchy Process MCDM solution developed by Thomas Saaty in early 1970s; Figure : Hierarchy of problem goal, criteria and alternatives
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Analytic Hierarchy Process Criteria Preference - Pairwise Comparisons criteria weights ⇐ derived from pairwise comparisons between brother nodes → positive reciprocal matrix a i j = a i / a j the PC (Pairwise Comparisons) matrix can contain inconsistent judgments
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Analytic Hierarchy Process PC matrix Consistency Definition A reciprocal matrix A is said to be (cardinally) consistent if a i j = a i k a kj ∀ i,j,k where a i j is called a direct judgment, given by the Decision Maker, and a i k a kj is an indirect judgment. Definition A reciprocal matrix A is said to be ordinally transitive (ordinally consistent) if ∀ i ∃ j , k s.t. a i j ≥ a i k ⇒ a j k ≤ 1 .
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Analytic Hierarchy Process Cardinal Consistency Metrics Consistency Ratio (CR) : λ max − n n − 1 / RI Consistency Measure (CM) : max ( CM i , j , k ) , i � = j � = k CM i , j , k = min ( a ij − a ik a kj , a ij − a ik a kj ) a ij a ik a kj n 1 Congruence ( Θ ) : Θ ij = � δ ( a ij , a ik a kj ) , i � = j � = k n − 2 k =1 δ ( a ij , a ik a kj ) = | log ( a ij ) − log ( ik a kj ) | n − 1 n 2 � � Θ = Θ ij 2( n − 1) i =1 j = i +1
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Analytic Hierarchy Process Ordinal Consistency Metrics The Number of Three-way Cycles (L) : E i → E j → E k → E i log ( a ij ) log ( a ik ) ≤ and log ( a ik ) log ( a jk ) < 0 OR log ( a ij ) = 0 and log ( a ik ) = 0 and log ( ajk ) � = 0 Dissonance( Ψ ) : 1 � Ψ ij = step ( − log a ij log a ik a kj ) , i � = j � = k n − 2 k � 1 , if x > 0 step ( x ) = 0 , otherwise n − 1 n 2 � � Ψ = Ψ ij n ( n − 1) i =1 j = i +1
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Analytic Hierarchy Process Eigenvalue Method elicit weights right eigenvector w = ( w 1 , ..., w n ) is calculated from its PC matrix A : Aw = λ max w (1) where λ max is largest eigenvalue of A
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Analytic Hierarchy Process Weight Elicitation Accuracy Metrics TD → Total Direct Deviation from Direct Judgments : n n ( a ij − w i w j ) 2 TD ( w ) = � � i =1 j =1 TD2 → Indirect Total Deviation from Indirect Judgments : n n n ( a ik a kj − w i w j ) 2 � � � TD 2( w ) = i =1 j =1 k =1 n − 1 n � � NV → Number of Priority Violations : NV ( w ) = v ij i =1 j = i +1 1 , if ( w i < w j ) and ( a ij > 1) 0 . 5 , if ( w i � = w j ) and ( a ij = 1) v ij = 0 . 5 , if ( w i = w j ) and ( a ij � = 1) 0 , otherwise
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Analytic Hierarchy Process Alternatives evaluation - Weighted Sum Method assess and normalize alternative i for each atomic criterion k ⇒ V i leaf k moving up trough the tree, for each node alternative values are defined as a weighted sum of the values computed below for each tree level. V i k = V i 1 ∗ w 1 k + V i 2 ∗ w 2 k + ... (2) where ( w 1 k , w 2 k , ... ) = w k is the eigenvector of non-leaf criterion k and V i k represents the value of alternative i evaluated against criterion k . V i goal = global value of alternative i
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Criteria Tree Ontology Criteria
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Metrics for Atomic Criteria Qualitative Criteria proposed solution for defining metrics for qualitative criteria ( language expressivity, inconsistency ) Algorithm 1 Define Qualitative Criterion metric (ontology) IF (Qualitative Criterion) is atomic property THEN IF ontology has property Qualitative Criterion metric THEN Qualitative Criterion metric(ontology) := 1 ELSE Qualitative Criterion metric(ontology) := 0 ELSE DECOMPOSE Qualitative Criterion
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Metrics for Atomic Criteria Language Expressivity 24 language features to asses Language Expressivity
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Including Negative Criteria Negative (Cost) Criteria original AHP: use different trees for benefit and cost criteria proposed solution: include negative criteria in the same tree leaf level negative criteria: inconsistency, unsatisfiable classes leaf i = 1 − leaf i , if criterion leaf is negative (3)
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Alternative Weight Elicitation Assessing alternatives existing solutions: human manual evaluation, using PC matrices ( PriEst ) and fuzzy intervals ( ONTOMETRIC ) proposed solution: automatically , from ontology measurements
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Alternative Weight Elicitation Alternatives Measurements Normalization sum Method steps to 1 step 1: leaf i = leaf i / � j leaf j Weighted step 2 : � leaf i , √ Arithmetic leaf - positive V i leaf = Mean 1 − leaf i , leaf - negative step 3: V i leaf = V i leaf / � j V j leaf , leaf - negative step 1: leaf i = leaf i / Max ( leaf j ) Max step 2 : X � leaf i , Normalization leaf - positive V i leaf = 1 − leaf i , leaf - negative
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Search Using Synonyms Knowledge Domain: terms to be searched in ontology concepts lexical and semantic search: WordNet synonyms polysemy disambiguation T = {� t i , Syn ( t i ) � | i � 1 }
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Domain Coverage Metric The coverage of a given domain T for an ontology O is the ratio of terms matched by classes of the ontology: DomainCoverage ( T , O ) = matched ( T , O ) , | T | where —T— counts the � t i , Syn ( t i ) � pairs; matched ( T , O ) = the number of pairs � t i , Syn ( t i ) � for which ∃ a class c ∈ O s.t. c = t i or c ∈ Syn ( t i )
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions System Architecture
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Functionality
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Domain Definition
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Functionality
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Domain Coverage Pre-selection
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions Functionality
Project Domain AHP adaptation for Ontology Evaluation Domain Coverage System Design Experiments Conclusions AHP using PriEsT Components
Recommend
More recommend
