Q 2 1 1 TDT4250 - Modeling of Information Systems, Autumn 2006 - - PDF document

1

1

1

TDT4250 - Modeling of Information Systems, Autumn 2006

Quality of models

John Krogstie Professor, IDI, NTNU Senior Researcher, SINTEF ICT

2

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Overview of presentation

What is quality ? Detailed presentation of semiotic model quality

framework (SEQUAL)

• Quality of models
• Quality of modelling languages (briefly, more later)

Based on chapter 3.1 in book manuscript

2

2

3

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

What is ’quality’ ?

Is ’The Lord of the Rings’ a good (set of) movie ?

4

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Different views on quality exist

Product vs. Process quality (e.g. CMM) According to requirements (ISO 9000) The user is satisfied (Denning) Properties of the product (-ilities) Properties of a requirements specification or

model (Davis/Pohl)

Quality related to different semiotic levels

(Lindland, Stamper)

”The quality that can be defined is not true

quality” - Robert Pirsig

3

3

5

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Properties of the product (design oriented)

How well does it function? Efficiency, Integrity, Reliability, Survivability, Usability How good is the design? Correctness, Maintainability, Verifiability How easy is it to evolve the product? Expandability, Flexibility, Interoperability, Portability,

Reusability

6

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Properties of a requirements specification (Davis)

Complete Correct Understandable Modifiable Consistent Unambigious Reusable Verifiable etc.

4

4

7

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Dimensions related to requirements specification process (Pohl)

8

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Core framework on quality of models (1994)

5

5

9

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

The Method Evaluation Model for evaluating the early version of the framework

P erceived E as e of Us e P erceived Us efulnes s Intention to Us e Actual E fficiency Actual E ffective- nes s PERFORMANCE PERCEPTIONS BEHAVIOUR Actual Us ag e INTENTIONS

10

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Research Design

Case #1 Case #1

Dependent Variables:

D1 = S yntactic Quality D2 = S emantic Quality D3 = Pragmatic Quality D4 = Overall Quality D5 = Perceived E ase of Use D6 = Perceived Usefulnes s D7 = Intention to Use

Experimental Tasks

Case #1

• 1. Modelling Task
• 2. E

valuation Tas k

• 3. Post-task S

urvey 20 different cas es in textual form

Experimental Treatment

Performance Bas ed Variables Perception Bas ed Variables Participants trained how to us e the quality framework

Materials

proces s models quality reviews survey res pons es

Participants

194 third year IS students Web-based Evaluation S ys tem random assignment

6

6

11

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Main results

• Reliability Analysis: Not consistent use of the framework by all
• Validity Analysis:
• Completeness: The three level found to be complete (for this kind of

use, i.e. with an objectively given domain)

• Necessity: All three levels necessary
• Independence: The three levels independent
• Interaction

S yntactic Quality S emantic Quality Pragmatic Quality

+ + +

12

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Main results continue

Many errors in the models made Many errors not reported Most errors reported where errors of the type

indicated

Participants found the framework Easy to use Useful Intention to use Need for specialisation of the framework

7

7

13

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Some weaknesses with earlier work on quality

• f models

Only for some types of models (Requirements

specification/design/completed product)

Cover only models in some languages Uncoordinated list of wanted properties Mix quality of process, tool, language, and model Mix quality goals and means to achieve these

goals

Is better suited for limited, constructed cases than

real modelling problems in practice.

14

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

SEQUAL – A framework for understanding and assessing quality of models

For models as a knowledge representation in

general

Can be extended and specialised towards specific

types of model and modelling languages

Differentiate between goals of modelling and

means to achieve these goals

Set-oriented definition to enable a formal

discussion of the different quality levels

Differentiate between quality of different levels

based on semiotic theory

Takes into account that models are socially

constructed

8

8

15

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Main elements of a modelling activity

Example of goal: Create a requirements specification for a travel agency on the net .

Persons Means for representation Area of interest Tools

Modeling task Goal of Modelling

16

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Sets in the quality framework

A: Actors that develops or has to relate to (parts

• f) the model. Can be persons or tools.

L: What can be expressed in the modelling

language

M: What is expressed in the model D: What can be expressed about the domain (area

• f interest)

K: The explicit knowlegde of the participating

persons

I: What the persons in the audience interpret the

model to say

T: What relevant tools interpret the model to say G: The goals of the modelling

All of these sets evolves as part of modeling

9

9

17

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Different types of goals possible - example

A new work processes should be documented through

the models.

The models developed should help sharing best

practice between different units.

The models developed should be helpful in the process

• f refining the processes.

The models developed should teach the software

developers about the domain.

The models developed should define the scope of the

software application.

The models developed should help analyze and

harmonize the current work processes.

The models developed should be used as a procedural

tool in everyday work.

The models developed should support the use of the

software application.

18

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Who are the human actors (A)?

A subgroup of those that are influenced by the

modelling (stakeholders).

Different types of stakeholders. E.g. in the context

• f system development:

System analysts, designers and programmers Users Managers

10

10

19

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

What is expressed in a model?

This is based on the model and the semantics of

the language used for modelling.

Can differentiate between what is explicitly and

implicitly expressed.

Language: Propositional logic Explicit model: A, A -> B Implicit model: B Overall model: A, B, A -> B

20

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Different domains relevant for IS-development

EIS: Existing Information System ECIS: Existing Computer-based Information

System

FIS: Future Information System FCIS: Future Computer-based Information System Requirements specification: Looking at the externally

• bservable behaviour of the IS

Design: Looking at the internal fabric of the technical

system

CIS: The actual computerised information system,

including the data within the system.

11

11

21

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Different domains for IS-development according to MDA-thinking in OMG

CIM (vs. FIS): Computational Independant Model PIM (vs. FCIS - RE/Design): Platform Independant

Model

PSM (vs. FCIS – Detailed design): Platform

Specific Model

22

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

SEQUAL

Goals of modelling Language extension

Modeling domain

Social actor interpretation Technical actor interpretation Social actor explicit knowledge

Organizational

Model externalization

Physical Perceived semantic Social Syntactic Pragmatic (human understanding) Pragmatic (tool understanding) Pragmatic (action) Empirical Semantic Pragmatic (learning)

12

12

23

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Overall structure of framework

Quality type (physical, emprical…) One or more quality goals per quality type

Means to achieve this goal

• Beneficial existing quality
• Model properties
• Language properties
• Modeling activities
• Tool-support

24

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Model example to illustrate the different quality levels

PERSON PAPER WRITES LANGUAGE TITLE N M

Language: ER Domain: Conference organizing Goal: Design of database solution to support conference

• rganizing

13

13

25

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Physical Quality

Internalizability Model persistence Model availability

• > Database functionality (model repository)

26

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Empirical quality

Look on aspects related to Ergonomics Graph and document layout Readability Information theory The model must be externalised Language properties Expressive economy Use of emphasis Modelling and tool activities (Automatic) graph-layout, readability index, grammar

checking, evaluation of use of colour.

14

14

27

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Example of poor graph-layout

28

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Syntactic quality

Syntactic correctness : M\L = Ø Two types of errors Syntactic invalidity Syntactic incompleteness The model must be externalised Language properties Formal syntax Activities Error prevention Error detection Error correction (automatically or by suggestion

(”spellcheck” ) )

15

15

29

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Example of syntactic invalidity

30

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Example of syntactic incompleteness

16

16

31

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Semantic quality

Goal Validity: M\D =Ø Completeness: D\M = Ø Necessary/useful that the model is externalised

and is syntactically correct

Language properties: Formal semantics Activities: Model testing (consistency checking),

reuse of models, ’driving questions’

32

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Example of semantic invalidity

17

17

33

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Pragmatic quality

Goals:

• Comprehension, do the audience understand what the model

express ? (I=M)

• Learning
• Action (change)

Useful that the model have high physical, empirical,

and syntactic quality before evaluating pragmatic quality.

Language properties:

• Operational semantics
• Executability
• Explicit modelling of intention

Activities: Inspection, visualization, filtering/views,

explanation generation, simulation, animation, reporting, execution/prototyping, model-generated solutions

34

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Example on language-oriented view

18

18

35

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Example of model-oriented view

36

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Example of model translation

19

19

37

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Perceived semantic quality

Goal Perceived validity I\K = Ø Perceived completeness: K\I = Ø Useful that the model has high physical,

empirical, syntactic, and pragmatic quality before investigating perceived semantic quality

Same means and activities as for semantic

quality.

38

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Social quality

Goal: Agreement Agreement in knowledge/interpretation/model Relative vs. absolute agreement Important first to address physical, pragmatic and

perceived semantic quality

Language properties: Possibility to explicitly

express inconsistencies based on disagreement.

Activities: Model integration and conflict

resolution

20

20

39

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Organizational quality

• The organizational quality of the model relates to
• that all statements in the model contribute to fulfilling

the goals of modelling (organizational goal validity)

• that all the goals of modelling are addressed through

the model (organizational goal completeness)

• Organizational quality introduce a context that

relax wanted quality for a model on the other levels (e.g. trade-of between completeness of the model relative to cost).

• Expressed with the notion of feasible quality

(particularly on the levels of semantic, pragmatic, perceived semantic and social quality)

40

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Example: Exam question 2a, Spring 2004

Case description about a company Assume that case description is correct about

domain

Assess empirical, syntactic and semantic quality Find defects Classify defects For semantic quality, further classification:

Invalidity, incompleteness

21

21

41

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

SEQUAL – language quality

Goals of modelling Language extension

Modeling domain

Social actor interpretation Technical actor interpretation Social actor explicit knowledge Model externalization

Comprehensibility appropriateness Organizational appropriateness Participant appropriateness Domain appropriateness Tool appropriateness Modeller appropriateness

42

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Aspects of language quality

Is the language appropriate to the domain? Is the language appropriate for the participants

knowledge of modelling languages?

Is the language appropriate to express the knowledge

• f the modeller?

Is the language appropriate for making

comprehensable model?

Is the language appropriate for tools? Is the language appropriate for the

chosen/standardised tools and modelling languages within the organization

Differentiate criteria for the conceptual basis of the

language (language-model/meta-model), and the external representation of the language (notation)

22

22

43

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Overview of total framework

Which of these models are best ?

Goals of modelling Language extension

Modeling domain

Social actor interpretation Technical actor interpretation Social actor explicit knowledge

Organizational

Model externalization

Physical Perceived semantic Social Syntactic Pragmatic (human understanding) Pragmatic (tool understanding) Pragmatic (action) Empirical Semantic Pragmatic (learning) 44

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

SEQUAL specializations

SEQUAL-Gen SEQUAL-EM SEQUAL-A1 SEQUAL-BPM SEQUAL-ONT SEQUAL-SOA SEQUAL-MDA SEQUAL-RE SEQUAL- DESIGN SEQUAL-DQ SEQUAL-UML SEQUAL-IM SEQUAL-DM SEQUAL-Met

23

23

45

TDT4250 - Modeling of Information Systems, Autumn 2006

Q

Usage of the framework

E.g. in ATHENA

• Evaluation of a modeling language under development
• Evaluation of the model of the modeling language (meta-

model)

• Evaluation of a modeling tool/environment
• Evaluation of a modeling methodology
• The methodology as a model
• The way the methodology support development of models of high

quality

Evaluation and choice of modeling languages (UML,

BPMN, EEML, others)

Evaluation of models (e.g. Assignments in this course) Methodology guidelines for developing good models Guidelines for developing new modeling languages

46

TDT4250 - Modeling of Information Systems, Autumn 2006

Quality of models

John Krogstie Professor, IDI, NTNU Senior Researcher, SINTEF ICT