Technical action research 3. Generalizing from TAR 4. Summary Roel - - PDF document

17‐5‐2012 1

Technical action research

Roel Wieringa University of twente

1 16th May, 2012 RCIS 2012, Valencia

• 1. What is TAR?
• 2. Logical structure of TAR
• 3. Generalizing from TAR
• 4. Summary

2 16th May, 2012 RCIS 2012, Valencia

• 1. Wat is Technical Action

Research?

3 16th May, 2012 RCIS 2012, Valencia

What is Technical Action Research?

• Example

– Researcher develops a technique to assess confidentiality risks in an IT architecture – She applies it to a problem that a company has ... – producing an advice to the company ... – and drawing lessons learned about the method

• She served two goals:

– The company’s goal is to assess confidentiality risks – The researcher’s goal is to learn something about her method

4 16th May, 2012 RCIS 2012, Valencia

What is Technical Action Research?

• The researcher plays three roles:

– Designer: Designing a technique – Helper: Using the technique to help others – Researcher: Drawing lessons learned about technique

• The key to a proper methodology for TAR is keeping

these roles separate

5 16th May, 2012 RCIS 2012, Valencia

Contrast with observational study (observational case study or survey)

• Example:

– Researcher observes one or more agile projects to investigate how requirements are prioritized – Avoids influencing the projects – Observes, analyzes, concludes lessons learned

• No change goal: The company is not influenced
• Researcher’s goal is to learn about prioritization in agile projects

as it is currently happening

• (the resulting knowledge may be useful to the companies)

6 16th May, 2012 RCIS 2012, Valencia

17‐5‐2012 2

Contrast with consulting

• Consulting

– Consultant is paid by client – Consultant applies known techniques rather than experimental technique – Reuse of techniques rather than critical evaluation – Aims at helping the client and acquiring repeat business, rather than testing a technique – Knowledge dissemination (if any) is internal

7 16th May, 2012 RCIS 2012, Valencia

Contrast with “classical” action research

• In classsical AR, researcher helps client to identify

and solve a problem

– Emancipation of the powerless – Learning about their situation

• In TAR, the researcher wants to learn something

about a technique by using it to solve a client’s problem

8 16th May, 2012 RCIS 2012, Valencia 9

Action Research cycle (Susman & Evered 1978)

16th May, 2012 RCIS 2012, Valencia

AR in information systems research

• AR in information systems

– Identify problem in an organization – Jointly search for a solution – Implement it – Evaluate – Specify learning

• TAR is technology‐driven, not problem driven

– The technology may be motivated by a desire to solve a class of problems – Not a singlular problem

10 16th May, 2012 RCIS 2012, Valencia

Why TAR for the researcher

• Researcher developed a technique behind her desk
• Applied it to first to small and then to realistic

examples

• Compared with other proposals
• Then what?

– Students will do as teacher tells: no realistic validation – Best way to learn about the technique is to apply it yourself

• Important to scale up from desk to practice

11 16th May, 2012 RCIS 2012, Valencia

TAR and design science

• Design science is designing and investigating artifacts
• Characteristic for design science is scaling up to

practice

– Start at the desk, – continue in the lab, – end up in the field – In the field you do TAR and/or statistical field experiments – Similar to scaling up in pharmaceutical research

12 16th May, 2012 RCIS 2012, Valencia

17‐5‐2012 3

Why TAR for the client

• Risky project with large chance of non‐result
• What is in it for the client?

– Free consult – Potentially useful result – Advance knowledge of and experience with new techniques – Good relationships with university (PR, HRM)

13 16th May, 2012 RCIS 2012, Valencia

• 2. Logical structure of TAR

14 16th May, 2012 RCIS 2012, Valencia 15

Action Research cycle (Susman & Evered 1978)

16th May, 2012 RCIS 2012, Valencia

• This conflates two action cycles:

– Action cycle of client – Action cycle of researcher

• Each has a different goal and justification

16 16th May, 2012 RCIS 2012, Valencia

The engineering cycle

• Rational action cycle

– Problem investigation – Treatment ( = action) design – Design validation – Treatment ( = action) implementation – Implementation evaluation

• Medical terminology

17 16th May, 2012 RCIS 2012, Valencia

• Problem investigation
• Treatment design with new or existing artifacts
• Design validation before applying the treatment
• Treatment implementation by applying to the problem
• Implementation evaluation

16th May, 2012 RCIS 2012, Valencia 18

Artifact Problem context: Stakeholders, Software, Hardware, Organizations, Processes, … Treatment

17‐5‐2012 4

• Problem investigation
• Treatment design
• Design validation
• Treatment implementation
• Implementation evaluation

19

Stakeholders, goals Phenomena, diagnosis, evaluation

16th May, 2012 RCIS 2012, Valencia

• Problem investigation
• Treatment design
• Design validation
• Treatment implementation
• Implementation evaluation

20

Treatment = interaction between artifact and context

• Interaction between Pill and Patient
• Interaction between Software and its Context
• Interaction between Method and its Context of use

16th May, 2012 RCIS 2012, Valencia

• Problem investigation
• Treatment design
• Design validation
• Treatment implementation
• Implementation evaluation

21

Artifact & Context → Effects? Trade-off: Changes in artifact? Sensitivity: Changes in context? Effects serve Stakeholder goals?

16th May, 2012 RCIS 2012, Valencia

• Problem investigation
• Treatment design
• Design validation
• Treatment implementation
• Implementation evaluation

22

Transfer to practice!

16th May, 2012 RCIS 2012, Valencia

• Problem investigation
• Treatment design
• Design validation
• Treatment implementation
• Implementation evaluation

23

Stakeholders, goals? Phenomena: Artifact & Context → Effects? Evaluation: Effects serve Stakeholder goals?

16th May, 2012 RCIS 2012, Valencia

• Example: Extending an enterprise architecture

(EA) method with goal‐oriented requirements engineering (GORE) to manage links to business goals

24 16th May, 2012 RCIS 2012, Valencia

17‐5‐2012 5

25

Problem investigation: Relation between EA and business objectives not known Treatment design: Extend EA method with GORE techniques (ARMOR) Artifact validation: Usable? Useful? Trade-offs? Sensitivity? Implementation: Transfer to practice Evaluation: Monitor usage Researcher’s cycle Client cycle Problem investigation: Goal of EA project? Treatment design: Plan the project Design validation: Validate the plan Execute EA evaluation EA satisfies client’s goals?

16th May, 2012 RCIS 2012, Valencia

• Two goals

– The client evaluates its redesigned EA against its goals – The researcher validates ARMOR against his goal

• Three roles for the researcher

– Designing a technique – Using it to help a client – Learning from it

• How do we use the client cycle to answer these validation

questions?

26 16th May, 2012 RCIS 2012, Valencia

The empirical research cycle

• Rational choice of an answer to a knowledge question

– Knowledge problem investigation – Research design – Design validation – Research execution – Results evaluation

27 16th May, 2012 RCIS 2012, Valencia

• Knowledge problem investigation
• Research design
• Design validation
• Research execution
• Results evaluation

28

Conceptual framework, Research questions, Population

16th May, 2012 RCIS 2012, Valencia

• Knowledge problem investigation
• Research design
• Design validation
• Research execution
• Results evaluation

29

Survey, observational case, Experiment, Action case, Simulation, ...

16th May, 2012 RCIS 2012, Valencia

• Knowledge problem investigation
• Research design
• Design validation
• Research execution
• Results evaluation

30

Would this really answer our questions? Risk assessment of doing the wrong thing to answer the questions

16th May, 2012 RCIS 2012, Valencia

17‐5‐2012 6

• Knowledge problem investigation
• Research design
• Design validation
• Research execution
• Results evaluation

31

Did this really answer our questions? Risk assessment of answering the questions incorrectly

16th May, 2012 RCIS 2012, Valencia 32

Corresponds to the three roles of the researcher: Designer, Researcher, Helper

Problem investigation: Relation between EA and business objectives not known Treatment design: Extend EA method with GORE techniques (ARMOR) Artifact validation: Is ARMOR usable? Useful? Trade-offs? Sensitivity? Research problem: Research question, Unit of study Research design: Select AR, acquire client & plan cycle Design validation: Would this answer our questions? Researcher’s Engineering cycle Researcher’s Empirical research cycle Problem investigation: Goal of EA project? Treatment design: Plan the project Design validation: Validate the plan Client’s engineering cycle Research execution Results evaluation: Answer the questions; Assess risk Execute EA evaluation EA satisfies client’s goals?

16th May, 2012 RCIS2012, Valencia 33

Action Research cycle (Susman & Evered 1978)

16th May, 2012 RCIS 2012, Valencia

Problem investigation: Relation between EA and business objectives not known Treatment design: Extend EA method with GORE techniques (ARMOR) Artifact validation: Is ARMOR usable? Useful? Trade-offs? Sensitivity? Research problem: Research question, Unit of study Research design: Select AR, acquire client & plan cycle Design validation: Would this answer our questions? Researcher’s Engineering cycle Researcher’s Empirical research cycle Problem investigation: Goal of EA project? Treatment design: Plan the project Design validation: Validate the plan Client’s engineering cycle Research execution Results evaluation: Answer the questions; Assess risk Execute EA evaluation EA satisfies client’s goals?

34

Specifying learning Evaluating Action planning Diagnosing Action taking Now we can see what is ignored in classical AR

16th May, 2012 RCIS 2012, Valencia

• Classical AR is problem‐driven

– Formulate a problem with the client – Jointly decide on action – Execute – Evaluate the results – Leassons learned to apply in this case or other cases

16th May, 2012 RCIS 2012, Valencia 35

Principles of canonical action research (Davison, Martinsons, Kock. Information Systems Journal 2004) apply to TAR

1. For each client cycle, TAR requires a client‐researcher agreement 2. TAR is iterative (in each of the three cycles) 3. TAR should be theory‐based (basis for expectation of treatment effect, and of generalization) 4. TAR implements change through action 5. TAR contains learning by reflection on action

16th May, 2012 RCIS 2012, Valencia 36

17‐5‐2012 7

37 16th May, 2012 RCIS 2012, Valencia

• 3. Generalizing from TAR

Discussion

38 16th May, 2012 RCIS 2012, Valencia 16th May, 2012 39

Instruments to influence the OoS in a particular way (and no other way) Instruments to observe the OoS (and avoid influence on OoS)

RCIS 2012, Valencia

General model of empirical scientific research

Entity actually studied by a researcher: A set of one or more population elements or surrogates for population elements. Sample, case,

• r model

16th May, 2012 40

Instruments used by researcher to help the organization, e,.g. teaching materials, software, etc. Instruments to observe what happened, e.g. a diary, logs, etc.

RCIS 2012, Valencia

Model of action research

An individual organization deemed to be representative for a population of unobserved similar organizations

Kinds of generalization

• Statistical inference is reasoning about samples

– Make an assumption about population distribution and parameters – Predict sampe statistic – Observations confirm or disconfirm the assumption

• Case‐based inference is reasoning about cases

– Observe phenomena in a case – Explain in terms of architecture – Predict that cases with similar architecture will exhibit similar phenomena

41 16th May, 2012 RCIS 2012, Valencia

Case‐based reasoning

• Reasoning from an observed case to an unobserved case
• Is based on similarity between cases.
• Source in legal reasoning

– When are two cases “similar”? – What follows from this “similarity”?

• Also well‐known in engineering

– Test an airfoil in a wind tunnel. – Infer how a real airplane with similar shape behaves in the air.

• If cases A and B are “similar” then some observations of A can

also be expected to occur in B

– Must be justified by a theory of similarity.

16th May, 2012 RCIS 2012, Valencia 42

17‐5‐2012 8

• Researcher is not representative of intended users
• Client company is representative of similar companies

– service organization, experienced architects, mature EA process are relevant features that impact the effectiveness of ARMOR 43 ARMOR method Researcher Method use Client company EA design

• Company with EA organization

represents

• Future companies where

ARMOR will be applied

• Designer‐researcher

represents

• Architects who will

eventually use method

• Limited description

represents

• Eventual method

description

Artefact prototype Simulated context Simulated context

16th May, 2012 RCIS 2012, Valencia

Architectural similarity

• Architecture of a system

– Entities with capabilities – Relations of influence

• Mechanism of an architecture

– The way entities interact when a system stimulus occurs

• Relevant similarities of cases are architectural

– The case is a sociotechnical system with an architecture – Components are people, software, hardware, organizations – Components have capabilities and influence relations – E.g. people have competencies, devices and software, have specifications, organizations have capabilities, matter has potential to respond

44 16th May, 2012 RCIS 2012, Valencia

Architectural inference

• How can this inference be valid?

– Because it is plausible that the mechanisms observed in the

• bserved case will also occur in the unobserved case

– How plausible? That is for you to show

• Architectural inference

– Identify the case architecture – Identify the mechanisms by which the case responds to stimuli – Explain the observations in terms of these mechanisms – Conclude that in cases with similar architecture, similar mechanisms will produce similar responses – Provided there are no countervailing mechanisms

45 16th May, 2012 RCIS 2012, Valencia

Inference from the EA case

• The same method will have the same effects in other
• rganizations like this:
• The company is a financial service organization,
• It has an architecture department
• Containing experienced architects
• With a mature EA process

16th May, 2012 RCIS 2012, Valencia 46

• But:
• This is a large semi‐government organization
• The architects are willing to try something new
• Have good relationships with the consultancy company
• Are experience in the EA language (ArchiMate)
• The consultant is the researcher who developed the GORE extension
• f ArchiMate

16th May, 2012 RCIS 2012, Valencia 47

Threats to validity

• Suppose that a treatment has been applied in case A with

effects E,

• And that this can be explained by means of case A architecture.
• Case B has the same architecture and we conclude that the

same treatment will have the same effect

• What could be wrong?

– The actors in case B have different capabilities – The architecture of A does not really explain the effects – The mechanism is nondeterministic – Architecture A contains other mechanisms too – Case B has other architectures too, with different mechanisms

16th May, 2012 RCIS 2012, Valencia 48

17‐5‐2012 9

Theoretical support

• Your arguments will be stronger if you can find

support in logic or previously established theory

– For example theory about components and their capabilities

16th May, 2012 RCIS 2012, Valencia 49

The rational argument game

• You propose a generalization, and give an

architectural argument

• The opponent (you, in the validity discussion)

points out weaknesses in your argument

• You publish
• Your other opponents (peer group) try to

repeat it or point out even more flaws in your argument

16th May, 2012 RCIS 2012, Valencia 50

Discussion assignment

• Give examples of each of these reasons in the enterprise

architecture case (or any other case you are familiar with)

16th May, 2012 RCIS 2012, Valencia 51

• 4. Summary

52 16th May, 2012 RCIS 2012, Valencia

• Technical action research is the validation of an

artifact by applying it in a realistic case

• The technical researcher is

– a designer – a helper – a researcher of knowledge questions

• Generalize by identifying architecture and

mechanisms

– Find support in logic or previously established theory

53 16th May, 2012 RCIS 2012, Valencia

R.J. Wieringa and A. Morali . “Technical Action Research as a Validation Method in Information Systems Design Science.” In K. Peffers and M. Rothenberger and B. Kuechler (eds.) Seventh International Conference on Design Science Research in Information Systems and Technology (DESRIST). LNCS 7286. Pages 220‐238. Springer, 2012.

16th May, 2012 RCIS 2012, Valencia 54