Introduction to design science methodology Roel Wieringa Slides - - PowerPoint PPT Presentation

▶
introduction to design science methodology
SMART_READER_LITE
LIVE PREVIEW

Introduction to design science methodology Roel Wieringa Slides - - PowerPoint PPT Presentation

Dec 05, 2022 226 likes •622 views

Introduction to design science methodology Roel Wieringa Slides based on Wieringa, R.J. (2014) Design science methodology for information systems and software engineering. Springer Verlag 30th May 2019 RCIS Brussels 1 Outline Design

slide-1
SLIDE 1

Introduction to design science methodology

Roel Wieringa Slides based on Wieringa, R.J. (2014) Design science methodology for information systems and software

  • engineering. Springer Verlag

30th May 2019 RCIS Brussels 1

slide-2
SLIDE 2

Outline

  • Design problems and knowledge questions
  • The design cycle
  • The empirical cycle

30th May 2019 RCIS Brussels 2

slide-3
SLIDE 3

Design science

  • Design science is the design and investigation of

artifacts in context

  • Examples

– Design and investigation of a method for user interface design for genomic databases – Design and investigation of an ontology‐based extension of OO‐method – Design and investigation of deep learning system to classify pathologies in X‐rays of the lower back – ….

30th May 2019 RCIS Brussels 3

slide-4
SLIDE 4

DESIGN PROBLEMS AND KNOWLEDGE QUESTIONS

30th May 2019 RCIS Brussels 4

slide-5
SLIDE 5

Your examples

30th May 2019 RCIS Brussels 5

slide-6
SLIDE 6

Two kinds of research problems in design science

30th May 2019 RCIS Brussels 6

To design an artifact to improve a problem context To answer knowledge questions about the artifact in context Problems & Artifacts to investigate Knowledge, New design problems Change your knowledge Change the real world Design software to estimate Direction

  • f Arrival of plane waves, to be used

in satelite TV receivers in cars

  • Is the DoA estimation accurate

enough in this context?

  • Is it fast enough?

Design a Multi‐Agent Route Planning system to be used for aircraft taxi route planning

  • Is this routing algorithm deadlock‐

free on airports?

  • How much delay does it produce?

Design a data location regulation auditing method

  • Is the method usable and useful for

consultants? DESIGN INVESTIGATION

slide-7
SLIDE 7

7

Heuristics

Design problems

√ To change the world √ Solution is design √ Many solutions √ Evaluated by usefulness √ What is useful depends on stakeholder goals √ Degrees of utility

Knowledge questions

√ To change your knowledge √ Answer is a proposition √ One answer √ Evaluated by truth √ What is true depends on the real world √ Degrees of certainty

Doing Thinking

30th May 2019 RCIS Brussels

slide-8
SLIDE 8

Nesting of problems

  • To solve a design problem:

– Study the problem – Test the proposed solution

  • To answer a knowledge question:

– Design your research

30th May 2019 RCIS Brussels 8

slide-9
SLIDE 9

Your examples revisited

30th May 2019 RCIS Brussels 9

slide-10
SLIDE 10

Design science

30th May 2019 RCIS Brussels 10

Framework for design science

Improvement design Answering knowledge questions Social context: Location of stakeholders: E.g. project sponsors, manufacturers, customers, users, maintenance, interfacing systems, negative stakeholders, attackers, government, labor, ... Knowledge context: Mathematics, social science, natural science, design science, design specifications, useful facts, practical knowledge, common sense, other beliefs

Goals, budgets Designs

  • Source of relevance.
  • Relevance, and money, comes and goes
  • Source and destination of theories
  • Theories are forever
  • Stakeholders may not know they are stakeholders
slide-11
SLIDE 11

30th May 2019 RCIS Brussels 11

Stakeholders

  • A stakeholder of a problem is a biological or legal person

affected by treating a problem.

– People, organizations, job roles, contractual roles, etc.

  • Stakeholders may not know that they are stakeholders

– They may accept the problem as normal – There may not be a problem at all … but you think/hope that there is an improvement opportunity

slide-12
SLIDE 12

Checklist by role (Ian Alexander http://www.scenarioplus.org.uk/papers/papers.htm > A taxonomy of stakeholders)

System under Development

  • Normal operator (end user)
  • Operational support
  • Maintenance operator

Immediate context

  • Functional beneficiary (client)
  • Roles responsible for interfacing

systems Wider context

  • Political beneficiary (who gains

status)

  • Financial beneficiary
  • Negative stakeholder (who

is/perceives to be hurt by the product)

  • Threat agent (who wants to hurt

the product)

  • Regulator

Involved in development

  • Champion/Sponsor
  • Developer
  • Consultant
  • Purchaser (customer)
  • Suppliers of components

30th May 2019 RCIS Brussels 12

These are just examples

slide-13
SLIDE 13

Examples

  • Design and investigation of a method for user interface design

for genomic databases

– Stakeholders: …….

  • Design and investigation of an ontology‐based extension of

OO‐method

– Stakeholders: …..

  • Design and investigation of deep learning system to classify

pathologies in X‐rays of the lower back

– Stakeholders: …..

30th May 2019 RCIS Brussels 13

slide-14
SLIDE 14

More examples

30th May 2019 RCIS Brussels 14

slide-15
SLIDE 15

THE DESIGN CYCLE

30th May 2019 RCIS Brussels 15

slide-16
SLIDE 16

The engineering cycle

  • Problem investigation
  • Treatment design
  • Design validation
  • Treatment implementation
  • Implementation evaluation

16

Stakeholders, goals, phenomena, evaluation, diagnosis

30th May 2019 RCIS Brussels

slide-17
SLIDE 17

The engineering cycle

  • Problem investigation
  • Treatment design
  • Design validation
  • Treatment implementation
  • Implementation evaluation

17

Treatment = interaction between artifact and context

  • Interaction between pill and patient
  • Interaction between Software and its Context
  • Interaction between method and its context of use
  • You design the artifact in order to create a treatment

30th May 2019 RCIS Brussels

slide-18
SLIDE 18

The engineering cycle

  • Problem investigation
  • Treatment design
  • Design validation
  • Treatment implementation
  • Implementation evaluation

18

Artifact & Context → Effects? Effects satisfy Criteria? Trade-off: Changes in artifact Sensitivity: Changes in context

30th May 2019 RCIS Brussels

slide-19
SLIDE 19

The engineering cycle

  • Problem investigation
  • Treatment design
  • Design validation
  • Treatment implementation
  • Implementation evaluation

19

Transfer to practice! Commercialization, sale

30th May 2019 RCIS Brussels

slide-20
SLIDE 20

The engineering cycle

  • Problem investigation
  • Treatment design
  • Design validation
  • Treatment implementation
  • Implementation evaluation

20

Phenomena: Artifact & Context → Effects? Evaluation: Effects satisfy Criteria?

30th May 2019 RCIS Brussels

slide-21
SLIDE 21

30th May 2019 RCIS Brussels 21

Where are we

Implementation evaluation = Problem investigation Treatment design Treatment validation Treatment Implementation (transfer to the real world)

  • Stakeholders? Goals?
  • Phenomena? Causes? Effects?
  • Effects contribute to Goals?
  • Specify requirements!
  • Contribution to goals?
  • Available treatments?
  • Design new ones!
  • Context & Artifact → Effects?
  • Effects satisfy Requirements?
  • Trade‐offs for different artifacts?
  • Sensitivity for different Contexts?

Engineering cycle

Legend: ? Knowledge questions ! Tasks

slide-22
SLIDE 22

30th May 2019 RCIS Brussels 22

Where are we

Implementation evaluation = Problem investigation Treatment design Treatment validation Treatment Implementation (transfer to the real world)

  • Stakeholders? Goals?
  • Phenomena? Causes? Effects?
  • Effects contribute to Goals?
  • Specify requirements!
  • Contribution to goals?
  • Available treatments?
  • Design new ones!
  • Context & Artifact → Effects?
  • Effects satisfy Requirements?
  • Trade‐offs for different artifacts?
  • Sensitivity for different Contexts?

Design cycle

Legend: ? Knowledge questions ! Tasks

slide-23
SLIDE 23

30th May 2019 RCIS Brussels 23

Knowledge questions that need empirical study

Implementation evaluation = Problem investigation Treatment design Treatment validation Treatment Implementation (transfer to the real world)

  • Stakeholders? Goals?
  • Phenomena? Causes? Effects?
  • Effects contribute to Goals?
  • Specify requirements!
  • Contribution to goals?
  • Available treatments?
  • Design new ones!
  • Context & Artifact → Effects?
  • Effects satisfy Requirements?
  • Trade‐offs for different artifacts?
  • Sensitivity for different Contexts?

Design cycle

slide-24
SLIDE 24

30th May 2019 RCIS Brussels 24

Outline of a thesis

Implementation evaluation = Problem investigation Treatment design Treatment validation

  • Stakeholders? Goals?
  • Phenomena? Causes? Effects?
  • Effects contribute to Goals?
  • Specify requirements!
  • Contribution to goals?
  • Available treatments?
  • Design new ones!
  • Context & Artifact → Effects?
  • Effects satisfy Requirements?
  • Trade‐offs for different artifacts?
  • Sensitivity for different Contexts?

Design cycle

slide-25
SLIDE 25

30th May 2019 RCIS Brussels 25

Outline of a thesis

Implementation evaluation = Problem investigation Treatment design Treatment validation

  • Stakeholders? Goals?
  • Phenomena? Causes? Effects?
  • Effects contribute to Goals?
  • Specify requirements!
  • Contribution to goals?
  • Available treatments?
  • Design new ones!
  • Context & Artifact → Effects?
  • Effects satisfy Requirements?
  • Trade‐offs for different artifacts?
  • Sensitivity for different Contexts?

Chapter 1: Motivation

  • Stakeholders, goal, phenomena, how bad

Chapter 2: Methodology

  • Research questions
  • Method(s) used to answer them

Chapter 3: Problem investigation

  • Empirical research
  • Literature survey about the problem

Chapter 4: Requirements for a solution

  • Sources of the requirements
  • Motivation in terms of stakeholder goals

Chapter 5: Survey of current solutions

  • Literature survey (state of the art)

Chapter 6: My solution proposal Chapter 7: Test 1

  • Experiment

Chapter 8 : Test 2

  • Experiment

Chapter 9: Summary, answers to research questions, discussion, future work

slide-26
SLIDE 26

30th May 2019 RCIS Brussels 26

Outline of a thesis

Implementation evaluation = Problem investigation Treatment design Treatment validation

  • Stakeholders? Goals?
  • Phenomena? Causes? Effects?
  • Effects contribute to Goals?
  • Specify requirements!
  • Contribution to goals?
  • Available treatments?
  • Design new ones!
  • Context & Artifact → Effects?
  • Effects satisfy Requirements?
  • Trade‐offs for different artifacts?
  • Sensitivity for different Contexts?

Chapter 2: Methodology

  • Research goal: To improve some problem context

by some artefact in order to contribute to some stakeholder goals

  • Research questions
  • What are the problematic phenomena?
  • Specify & motivate the requirements for a

solution (“What are the requirements?”)

  • What solutions exist?
  • How good/bad are they?
  • Design a new solution (“what is a good

solution?”)

  • Does my solution in the intended problem

context satisfy the requirements?

  • What happens if I change the design?
  • What happens if I change the context?
  • Method(s) used to answer them
slide-27
SLIDE 27

What are your research questions?

30th May 2019 RCIS Brussels 27

slide-28
SLIDE 28

THE EMPIRICAL CYCLE

30th May 2019 RCIS Brussels 28

slide-29
SLIDE 29

Nesting of cycles

Problem investigation Treatment design Treatment validation Knowledge problem investigation (How to do the validation?) Experiment design & validation (design and validate a prototype & test environment) Implementation (construction of prototype & test environment, lab or field) Evaluation (analyze results) Implementation (tech transfer) Implementation evaluation (in the field)

30th May 2019 RCIS Brussels 29

Research project: design cycle This is a very special engineering cycle, called the empirical cycle.

slide-30
SLIDE 30

Nesting of cycles

Problem investigation Knowledge problem investigation (How to investigate the design problem?) Experiment design & validation (design and validate a prototype & test environment) Implementation (construction of prototype & test environment, lab or field) Evaluation (analyze results) Treatment design Treatment validation Implementation (tech transfer) Implementation evaluation (in the field)

30th May 2019 RCIS Brussels 30

Research project: design cycle This is a very special engineering cycle, called the empirical cycle.

slide-31
SLIDE 31

The empirical research cycle

  • This is the rational decision cycle applied to answer

knowledge questions (empirical research questions)

– Knowledge problem investigation – Research design – Design validation – Research execution – Results evaluation

31 30th May 2019 RCIS Brussels

slide-32
SLIDE 32
  • Knowledge problem investigation
  • Research design
  • Design validation
  • Research execution
  • Results evaluation

32

Research questions, Unit of study

30th May 2019 RCIS Brussels

slide-33
SLIDE 33
  • Knowledge problem investigation
  • Research design
  • Design validation
  • Research execution
  • Results evaluation

33

Survey, observational case, Experiment, Action case, Simulation, ...

30th May 2019 RCIS Brussels

slide-34
SLIDE 34
  • Knowledge problem investigation
  • Research design
  • Design validation
  • Research execution
  • Results evaluation

34

Would this really answer our questions? Risk assessment of doing the wrong thing to answer the questions

30th May 2019 RCIS Brussels

slide-35
SLIDE 35
  • Knowledge problem investigation
  • Research design
  • Design validation
  • Research execution
  • Results evaluation

35

Did this really answer our questions? Risk assessment of answering the questions incorrectly

30th May 2019 RCIS Brussels

slide-36
SLIDE 36

30th May 2019 RCIS Brussels 36

Research problem analysis 4. Conceptual framework? 5. Research questions? 6. Population? Research execution

  • 11. What happened?

Research design 7. Object of study? 8. Treatment specification? 9. Measurement specification?

  • 10. Inference?

Analysis of results

  • 12. Data?
  • 13. Observations?
  • 14. Explanations?
  • 15. Generalizations?
  • 16. Answers?

Empirical cycle

New research problem Research design validation 7. Object of study justification? 8. Treatment specification justification? 9. Measurement specification justification?

  • 10. Inference justification?
  • Very detailed
  • This integrates

all checklists!

slide-37
SLIDE 37

Research designs and inferences

30th May 2019 RCIS Brussels 37

Observational study (no treatment) Experimental study (treatment) Case‐based: investigate single cases, look at architecture and mechanisms. Inference: Architectural explanation, generalization by analogy Observational case study

  • Expert opinion (mental

simulation by experts),

  • Case‐based experiments

(simulations, prototyping),

  • Technical action research

(experimental use of the artifact in the real world) Sample‐based: investigate samples drawn from a population, look at averages and variation. Inference: Statistical inference, causal explanation, possible architectural explanation and analogy Survey

  • Sample‐based experiments

(e.g. treatment group – control group experiments)

Validation methods (depends on time and budget) Problem investigation methods

slide-38
SLIDE 38

Research strategy

  • Scaling up:

– Expert opinion – Lab experiment (test experimental prototype in lab context) – Field experiment (test experimental prototype in field context) – TAR (apply your experimental solution in a real‐world project)

30th May 2019 RCIS Brussels 38

Small samples Large samples Population More realistic conditions of practice More robust generalizations Idealized Practical

Sample‐based experiments Case‐based experiments Technical action research Expert opinion

Labor Laboratory tory credibility credibility

Street credibility

slide-39
SLIDE 39
  • Questions
  • Remarks
  • Examples
  • Discussion
  • More information
  • Extra slides
  • Any other business
  • Lunch

30th May 2019 RCIS Brussels 39