Interview Review: an empirical study on detecting ambiguities in - - PowerPoint PPT Presentation

▶
interview review
SMART_READER_LITE
LIVE PREVIEW

Interview Review: an empirical study on detecting ambiguities in - - PowerPoint PPT Presentation

Feb 06, 2023 320 likes •1.02k views

Interview Review: an empirical study on detecting ambiguities in requirements elicitation interviews Paola Spoletini Kennesaw State University, USA Alessio Ferrari ISTI-CNR, Italy Muneera Bano SUT Melbourne, Australia Didar Zowghi UTS

slide-1
SLIDE 1

Interview Review:

an empirical study on detecting ambiguities in requirements elicitation interviews

Paola Spoletini

Kennesaw State University, USA

Alessio Ferrari

ISTI-CNR, Italy

Muneera Bano

SUT Melbourne, Australia

Didar Zowghi

UTS Sydney, Australia

Stefania Gnesi

ISTI-CNR, Italy

3/22/18 Interview Reviews - REFSQ'18

slide-2
SLIDE 2

Interview Review:

an empirical study on detecting ambiguities in requirements elicitation interviews

Paola Spoletini

Kennesaw State University, USA

Alessio Ferrari

ISTI-CNR, Italy

Muneera Bano

SUT Melbourne, Australia

Didar Zowghi

UTS Sydney, Australia

Stefania Gnesi

ISTI-CNR, Italy

3/22/18 Interview Reviews - REFSQ'18

slide-3
SLIDE 3

Ambiguities

3/22/18 Interview Reviews - REFSQ'18 SPEECH FRAGMENT k k CONCEPT ACCEPTANCE k Requirements Goals Domain Specification INTERPRETATION k

slide-4
SLIDE 4

Ambiguities

3/22/18 Interview Reviews - REFSQ'18 SPEECH FRAGMENT k k CONCEPT ACCEPTANCE k Requirements Goals Domain Specification INTERPRETATION k

An ambiguity occurs when a customer articulates a unit of information, and the meaning assigned by the requirements analyst to such articulation differs from the meaning intended by the customer

slide-5
SLIDE 5

Ambiguities

3/22/18 Interview Reviews - REFSQ'18 SPEECH FRAGMENT k k CONCEPT ACCEPTANCE k Requirements Goals Domain Specification INTERPRETATION k

slide-6
SLIDE 6

Ambiguities

3/22/18 Interview Reviews - REFSQ'18 SPEECH FRAGMENT k k CONCEPT ACCEPTANCE k Requirements Goals Domain Specification INTERPRETATION k

Interp rpretation tation Unc Unclar arity ity

slide-7
SLIDE 7

Ambiguities

3/22/18 Interview Reviews - REFSQ'18 SPEECH FRAGMENT k k CONCEPT ACCEPTANCE k Requirements Goals Domain Specification INTERPRETATION k

Acceptan ptance Unc Unclarit arity

slide-8
SLIDE 8

Ambiguities

3/22/18 Interview Reviews - REFSQ'18 SPEECH FRAGMENT k k CONCEPT ACCEPTANCE k Requirements Goals Domain Specification INTERPRETATION k

Multiple iple Un Understandin ding

slide-9
SLIDE 9

Ambiguities

3/22/18 Interview Reviews - REFSQ'18 SPEECH FRAGMENT k k CONCEPT ACCEPTANCE k Requirements Goals Domain Specification INTERPRETATION k

Interp rpretation tation Unc Unclar arity ity

slide-10
SLIDE 10

Ambiguities

3/22/18 Interview Reviews - REFSQ'18 SPEECH FRAGMENT k k CONCEPT ACCEPTANCE k Requirements Goals Domain Specification INTERPRETATION k

Interp rpretation tation Unc Unclar arity ity Detected d Incorrect di disamb mbigu iguatio ation

slide-11
SLIDE 11

Ambiguities

3/22/18 Interview Reviews - REFSQ'18 SPEECH FRAGMENT k k CONCEPT ACCEPTANCE k Requirements Goals Domain Specification INTERPRETATION k

Und Undetected d Incorrect t di disamb mbigu iguatio ation

slide-12
SLIDE 12

Ambiguities

3/22/18 Interview Reviews - REFSQ'18 SPEECH FRAGMENT k k CONCEPT ACCEPTANCE k Requirements Goals Domain Specification INTERPRETATION k

Und Undetected d Incorrect t di disamb mbigu iguatio ation

Misunderstanding, conflicting situations…

slide-13
SLIDE 13

Interview Review:

an empirical study on detecting ambiguities in requirements elicitation interviews

Paola Spoletini

Kennesaw State University, USA

Alessio Ferrari

ISTI-CNR, Italy

Muneera Bano

SUT Melbourne, Australia

Didar Zowghi

UTS Sydney, Australia

Stefania Gnesi

ISTI-CNR, Italy

3/22/18 Interview Reviews - REFSQ'18

slide-14
SLIDE 14

Reviews

3/22/18 Interview Reviews - REFSQ'18

slide-15
SLIDE 15

Reviews

3/22/18 Interview Reviews - REFSQ'18

slide-16
SLIDE 16

How are reviews effective?

3/22/18 Interview Reviews - REFSQ'18

slide-17
SLIDE 17

How are reviews effective?

3/22/18 Interview Reviews - REFSQ'18

■ Often effective in identification of defects in requirements specifications ■ Widely used in the industry

slide-18
SLIDE 18

How are reviews effective?

“Software requirements are based on flawed ‘upstream’ requirements and reviews on requirements specifications are thus in vain”

F.Salger,“Requirementsreviewsrevisited:Residualchallengesandopen research questions,” in RE’13. IEEE, 2013, pp. 250–255

3/22/18 Interview Reviews - REFSQ'18

■ Often effective in identification of defects in requirements specifications ■ Widely used in the industry

slide-19
SLIDE 19

3/22/18 Interview Reviews - REFSQ'18

Interview Review

“Software requirements are based on flawed ‘upstream’ requirements and reviews on requirements specifications are thus in vain”

F.Salger,“Requirementsreviewsrevisited:Residualchallengesandopen research questions,” in RE’13. IEEE, 2013,

  • pp. 250–255
slide-20
SLIDE 20

3/22/18 Interview Reviews - REFSQ'18

Interview Review

“Software requirements are based on flawed ‘upstream’ requirements and reviews on requirements specifications are thus in vain”

F.Salger,“Requirementsreviewsrevisited:Residualchallengesandopen research questions,” in RE’13. IEEE, 2013,

  • pp. 250–255
slide-21
SLIDE 21

3/22/18 Interview Reviews - REFSQ'18

Interview Review

“Software requirements are based on flawed ‘upstream’ requirements and reviews on requirements specifications are thus in vain”

F.Salger,“Requirementsreviewsrevisited:Residualchallengesandopen research questions,” in RE’13. IEEE, 2013,

  • pp. 250–255
slide-22
SLIDE 22

3/22/18 Interview Reviews - REFSQ'18

Interview Review

“Software requirements are based on flawed ‘upstream’ requirements and reviews on requirements specifications are thus in vain”

F.Salger,“Requirementsreviewsrevisited:Residualchallengesandopen research questions,” in RE’13. IEEE, 2013,

  • pp. 250–255
slide-23
SLIDE 23

3/22/18 Interview Reviews - REFSQ'18

Interview Review

“Software requirements are based on flawed ‘upstream’ requirements and reviews on requirements specifications are thus in vain”

F.Salger,“Requirementsreviewsrevisited:Residualchallengesandopen research questions,” in RE’13. IEEE, 2013,

  • pp. 250–255

Intuition: tuition: Review of requirements elicitation interviews allows identifying ambiguities that can be leveraged to ask useful follow- up questions in future interviews.

slide-24
SLIDE 24

Research questions?

3/22/18 Interview Reviews - REFSQ'18

slide-25
SLIDE 25

Research questions?

RQ1: Is there a difference between ambiguities explicitly revealed by an analyst during an interview, and ambiguities identified by a reviewer who listens to the interview recording?

3/22/18 Interview Reviews - REFSQ'18

slide-26
SLIDE 26

Research questions?

RQ1: Is there a difference between ambiguities explicitly revealed by an analyst during an interview, and ambiguities identified by a reviewer who listens to the interview recording? RQ2: Is there a difference between ambiguities identified by the analyst when s/he listens to the interview recording, and ambiguities identified by a reviewer who listens to the interview recording?

3/22/18 Interview Reviews - REFSQ'18

slide-27
SLIDE 27

Research questions?

RQ1: Is there a difference between ambiguities explicitly revealed by an analyst during an interview, and ambiguities identified by a reviewer who listens to the interview recording? RQ2: Is there a difference between ambiguities identified by the analyst when s/he listens to the interview recording, and ambiguities identified by a reviewer who listens to the interview recording? RQ3: Can the ambiguities identified during interview review be used to ask useful questions in future interviews?

3/22/18 Interview Reviews - REFSQ'18

slide-28
SLIDE 28

Exploratory study

■ 38 students from KSU, 19 interviews – Software intensive system – 20 minutes per interview – 2 hour lecture on elicitation ■ 2 reviewers, 10 interviews – Researcher in requirements elicitation – Professional analyst

3/22/18 Interview Reviews - REFSQ'18

slide-29
SLIDE 29

Interview Review:

an empirical study on detecting ambiguities in requirements elicitation interviews

Paola Spoletini

Kennesaw State University, USA

Alessio Ferrari

ISTI-CNR, Italy

Muneera Bano

SUT Melbourne, Australia

Didar Zowghi

UTS Sydney, Australia

Stefania Gnesi

ISTI-CNR, Italy

3/22/18 Interview Reviews - REFSQ'18

slide-30
SLIDE 30

Research questions?

RQ1: Is there a difference between ambiguities explicitly revealed by an analyst during an interview, and ambiguities identified by a reviewer who listens to the interview recording? RQ2: Is there a difference between ambiguities identified by the analyst when s/he listens to the interview recording, and ambiguities identified by a reviewer who listens to the interview recording? RQ3: Can the ambiguities identified during interview review be used to ask useful questions in future interviews?

3/22/18 Interview Reviews - REFSQ'18

slide-31
SLIDE 31

Research questions?

RQ1: Is there a difference between ambiguities explicitly revealed by an analyst during an interview, and ambiguities identified by a reviewer who listens to the interview recording? RQ2: Is there a difference between ambiguities identified by the analyst when s/he listens to the interview recording, and ambiguities identified by a reviewer who listens to the interview recording?

3/22/18 Interview Reviews - REFSQ'18

slide-32
SLIDE 32

Variables

3/22/18 Interview Reviews - REFSQ'18

slide-33
SLIDE 33

Variables

■ Independent variable: Perspec pecti tive e – Role – Moment

3/22/18 Interview Reviews - REFSQ'18

slide-34
SLIDE 34

Variables

■ Independent variable: Perspec pecti tive e – Role – Moment

3/22/18 Interview Reviews - REFSQ'18

AI

slide-35
SLIDE 35

Variables

■ Independent variable: Perspec pecti tive e – Role – Moment

3/22/18 Interview Reviews - REFSQ'18

AI AR

slide-36
SLIDE 36

Variables

■ Independent variable: Perspec pecti tive e – Role – Moment

3/22/18 Interview Reviews - REFSQ'18

AI AR RR

slide-37
SLIDE 37

Variables

■ Independent variable: Perspec pecti tive e – Role – Moment ■ Dependent variable: Perform

  • rmanc

nce in detecting ambiguities – Set of found ambiguities – Total number

3/22/18 Interview Reviews - REFSQ'18

AI AR RR

slide-38
SLIDE 38

Hypotheses

3/22/18 Interview Reviews - REFSQ'18

slide-39
SLIDE 39

Hypotheses

3/22/18 Interview Reviews - REFSQ'18

RQ1: Is there a difference between ambiguities explicitly revealed by an analyst during an interview, and ambiguities identified by a reviewer who listens to the interview recording?

slide-40
SLIDE 40

Hypotheses

■ H1.10: The reviewer’s performance during the review is irrelevant with respect to the analyst’s performance during the interview;

3/22/18 Interview Reviews - REFSQ'18

RQ1: Is there a difference between ambiguities explicitly revealed by an analyst during an interview, and ambiguities identified by a reviewer who listens to the interview recording?

slide-41
SLIDE 41

Hypotheses

■ H1.10: The reviewer’s performance during the review is irrelevant with respect to the analyst’s performance during the interview; ■ H1.20: The analyst’s performance during the review is irrelevant with respect to the analyst’s performance during the interview.

3/22/18 Interview Reviews - REFSQ'18

RQ1: Is there a difference between ambiguities explicitly revealed by an analyst during an interview, and ambiguities identified by a reviewer who listens to the interview recording?

slide-42
SLIDE 42

Hypotheses

3/22/18 Interview Reviews - REFSQ'18

slide-43
SLIDE 43

Hypotheses

3/22/18 Interview Reviews - REFSQ'18

RQ2: Is there a difference between ambiguities

identified by the analyst when s/he listens to the interview recording, and ambiguities identified by a reviewer who listens to the interview recording?

slide-44
SLIDE 44

Hypotheses

■ H2.10:The analyst’s performance during the review is irrelevant with respect to the reviewer’s performance during the review;

3/22/18 Interview Reviews - REFSQ'18

RQ2: Is there a difference between ambiguities

identified by the analyst when s/he listens to the interview recording, and ambiguities identified by a reviewer who listens to the interview recording?

slide-45
SLIDE 45

Hypotheses

■ H2.10:The analyst’s performance during the review is irrelevant with respect to the reviewer’s performance during the review; ■ H2.20:The reviewer’s performance during the review is irrelevant with respect to the analyst’s performance during the review.

3/22/18 Interview Reviews - REFSQ'18

RQ2: Is there a difference between ambiguities

identified by the analyst when s/he listens to the interview recording, and ambiguities identified by a reviewer who listens to the interview recording?

slide-46
SLIDE 46

Experiment settings

■ Students from KSU and UTS – Software intensive system – Limited time per interview – Lecture on elicitation ■ Reviewers are students – A customer in another interview – The student analyst

3/22/18 Interview Reviews - REFSQ'18

slide-47
SLIDE 47

Reviews

■ Guidelines to identify ambiguities – You have not understood the meaning of what you heard – You have not understood the purpose of what you heard – What you heard is too general – … ■ Content – Time: when the fragment happened – Fragment: the fragment that triggered the ambiguity – Question: the question that you would ask to the customer to clarify

3/22/18 Interview Reviews - REFSQ'18

slide-48
SLIDE 48

Overall evaluation

3/22/18 Interview Reviews - REFSQ'18

slide-49
SLIDE 49

Overall evaluation

3/22/18 Interview Reviews - REFSQ'18

slide-50
SLIDE 50

Overall evaluation

3/22/18 Interview Reviews - REFSQ'18

slide-51
SLIDE 51

Overall evaluation

3/22/18 Interview Reviews - REFSQ'18

slide-52
SLIDE 52

RQ1: Contribution of the review

3/22/18 Interview Reviews - REFSQ'18

slide-53
SLIDE 53

RQ1: Contribution of the review

3/22/18 Interview Reviews - REFSQ'18

slide-54
SLIDE 54

RQ2: Contribution of different reviews

3/22/18 Interview Reviews - REFSQ'18

slide-55
SLIDE 55

RQ2: Contribution of different reviews

3/22/18 Interview Reviews - REFSQ'18

slide-56
SLIDE 56

Threats to validity

slide-57
SLIDE 57

Threats to validity

■ Graduate students vs undergraduate students

slide-58
SLIDE 58

Threats to validity

■ Graduate students vs undergraduate students ■ Previous experience

slide-59
SLIDE 59

Threats to validity

■ Graduate students vs undergraduate students ■ Previous experience ■ Experiment run with students

slide-60
SLIDE 60

Conclusion

3/22/18 Interview Reviews - REFSQ'18

slide-61
SLIDE 61

Conclusion

3/22/18 Interview Reviews - REFSQ'18

slide-62
SLIDE 62

Conclusion

3/22/18 Interview Reviews - REFSQ'18

slide-63
SLIDE 63

Conclusion

3/22/18 Interview Reviews - REFSQ'18

slide-64
SLIDE 64

Conclusion

3/22/18 Interview Reviews - REFSQ'18

slide-65
SLIDE 65

Future work

RQ3: 3: Can the ambiguities identified during interview review be used to ask useful questions in future interviews?

3/22/18 Interview Reviews - REFSQ'18

slide-66
SLIDE 66

Future work

RQ3: 3: Can the ambiguities identified during interview review be used to ask useful questions in future interviews? ■ The protocol is applied in real world

3/22/18 Interview Reviews - REFSQ'18

slide-67
SLIDE 67

Future work

RQ3: 3: Can the ambiguities identified during interview review be used to ask useful questions in future interviews? ■ The protocol is applied in real world ■ The useful of the questions generated by the protocol will be measured – Perceived usefulness – Actual usefulness

3/22/18 Interview Reviews - REFSQ'18

slide-68
SLIDE 68

3/22/18 Interview Reviews - REFSQ'18

slide-69
SLIDE 69

Results

■ Identified ambiguities: – Analyst: 23 – Both: 21 – Reviewers:38 ■ Time – Recordings: 2 hours and 37 minutes, – Reviewer 1: 5 hours – Reviewer 2: 8 hours and 33 minutes

3/22/18 Interview Reviews - REFSQ'18