[PPT] - Pitfalls and Countermeasures in Software Quality Measurements and PowerPoint Presentation

SLIDE 1

Pitfalls and Countermeasures in Software Quality Measurements and Evaluations

Hironori Washizaki washizaki@waseda.jp

Prof., Global Software Engineering Laboratory, Waseda University Visiting Prof., National Institute of Informatics Director, SYSTEM INFORMATION CO., LTD. Vice-Chair, IEEE CS Japan Chapter Chair, SEMAT Japan Chapter Convenor, ISO/IEC/JTC1/SC7/WG20 PC Co-Chair, APSEC 2018 Nagoya!

5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ) Nanjing, Dec 4, 2017

H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017

SLIDE 2

Metric and Measurement

Mapping attributes to values/names
n scales

– Quality control by single metric – Estimating metric values by other metrics

You cannot control what you cannot

measure!

Req. Def. Design Impl. Testing

Func. Spec.

Function point Module design Coupling Source code Complexity LOC Defect report Defect density Test case

N. tests

passed

Effort, time

Cyclomatic complexity

0.2 0.4 0.6 0.8 1 0.05 0.1 0.15 0.2 0.25 0.3

Reuse rate #Header files included

2

SLIDE 3

Pitfalls and Countermeasures

3

Pitfall Countermeasure Negative Hawthorne effects Goal-orientation Multidimensional measurements Organization misalignment Visualization

f

relationships among

rganizational

goals, strategies, and measurements Exhaustive identification of rationales Uncertain future Prediction incorporating uncertainty Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation

H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017

SLIDE 4

Pitfalls and Countermeasures

4

Pitfall Countermeasure Negative Hawthorne effects Goal-orientation Multidimensional measurements Organization misalignment Visualization

f

relationships among

rganizational

goals, strategies, and measurements Exhaustive identification of rationales Uncertain future Prediction incorporating uncertainty Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation

H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017

SLIDE 5

Hawthorne Effect

5

nicolasdsampson.com, Observe And Learn: The Magic Of Paying Attention http://nicolasdsampson.com/wp-content/uploads/2012/10/2010_12_06_observe-learn-magic-paying-attention.jpg

SLIDE 6

Goal-Question-Metric (GQM) Paradigm

Goal-oriented framework for identifying goals and necessary

corresponding metrics

Goal: measurement goals
Question: questions for evaluating goal achievement
Metric: objective or subjective metrics for obtaining

necessary quantitative data to answer questions

Identifying metrics Interpreting measurement results Metric Goal Question Collected data Goal achievement evaluation Answer Measured value

R. van Solingen, E. Berghout, “The Goal/Question/Metric Method” McGraw-Hill Education, 1999

SLIDE 7

7

GQM-based Multidimensional Measurements

H. Washizaki, R. Namiki, T. Fukuoka, Y. Harada, H. Watanabe, "A Framework for Measuring and Evaluating Program

Source Code Quality", 8th Int’l Conference on Product Focused Software Development and Process Improvement (Profes2007)

Reliability Maintain

ability

Portability Efficiency Reusability

Analysability Changeability Testability

Goal Metric

Depth of call graph T Cyclomatic Number min Estimated static path length min Measured by some tools Measured by some tools Are functions not complicated?

Question

Sub- Question Is call- nesting not too deep? Is logic not too complex? Independent Independent Specific Specific Are entities named properly? (Scale type: T threshold, min smaller is better, max larger is better)

SLIDE 8

SEMAT-based Multidimensional measurements

8

Software System Opportunity Stakeholders Requirements Work Team Way of Working

Customer Solution Endeavour

Ivar Jacobson, Pan-Wei Ng, Paul E. McMahon, Ian Spence, Svante Lidman, “The Essence of Software Engineering: The SEMAT Kernel,” Communications of the ACM, vol.55, no.12, pp.42-49, December 2012.

SLIDE 9

alpha as Project Measurement

9

 State 1

 XXXXXXXXXXXXXXXXXX  XXXXXXXXXXX  XXXXXXXXXXXX

 State 2

 XXXXXXXXXXXXXXXXXX  XXXXXXXXXXX  XXXXXXXXXXXX

 State 3

 XXXXXXXXXXXXXXXXXX  XXXXXXXXXXX  XXXXXXXXXXXX  ……..  …….

Checklist

Adopted/modified from I. Jacobson, et al.: Tutorial: Essence - Kernel and Language for Software Engineering Practices, ICSE'13

alpha Indicator

1 2 3 ４５

SLIDE 10

Example from ITA WG on project failuers

10

An employee in charge of Bank office

inquires "registered customer information cannot be browsed from the terminal“.

It was because a batch processing for the

previous day has ended abnormally due to wrong data input.

It took about two hours to recover the

data, and employees at each office had to handle customer inquiry manually.

For that reason, we had received plenty of

complaints from customers who had been waiting for a long time!

H. Washizaki, “Analyzing and refining project failure cases from wider viewpoints by using SEMAT Essence,” Essence

Conference in Seoul, 2017.

SLIDE 11

11

Delay in service deployment Delay in service deployment Huge data inputs manually Huge data inputs manually Wrong data input Wrong data input Defects in

peration manual

Defects in

peration manual

Misunderstanding

f abnormal state

as normal state Misunderstanding

f abnormal state

as normal state Problem Cause Root cause Requirement s: NG “Coherent” Software System: NG “Demonstrable” Automate data input Automate data input Countermeasure Frequent rule- violations Frequent rule- violations Work: NG “Under Control” Way of Working: NG “In Use” Review and modify rules Review and modify rules Monitor and enforce compliance with rules Monitor and enforce compliance with rules Opportunity: NG “Benefit Accrued” Team: NG “Collaborating” Stakeholders: NG “Satisfied in Use”

SLIDE 12

Pitfalls and Countermeasures

12

Pitfall Countermeasure Negative Hawthorne effects Goal-orientation Multidimensional measurements Organization misalignment Visualization

f

relationships among

rganizational

goals, strategies, and measurements Exhaustive identification of rationales Uncertain future Prediction incorporating uncertainty Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation

H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017

SLIDE 13

Organization misalignment

13

We increase customer satisfaction by quality and usability improvement! Top management We reduce customer- reported defects by improving testing process and product maintainability. Development team We track defect data and code quality metrics. Quality assurance team

SLIDE 14

GQM+Strategies

M1: Customer satisfaction survey M1: Customer satisfaction survey Measurement Data Goals and Strategies Business Level Software Level Goal: Increase customer satisfaction by 10% Goal: Increase customer satisfaction by 10% Strategy: Improve product quality Strategy: Improve product quality Strategy: Improve usability of product Strategy: Improve usability of product Goal: Reduce customer- reported defects by 20% Goal: Reduce customer- reported defects by 20% No sub-level goal defined Strategy: Improve efficiency of system testing Strategy: Improve efficiency of system testing Strategy: Improve maintainability of software Strategy: Improve maintainability of software Isolated strategy M2: Field defect data M2: Field defect data M3: Code quality metrics (McCabe, coupling, cohesion) M3: Code quality metrics (McCabe, coupling, cohesion) Isolated data

14

Alignment and tracing among goal, strategy

and data

Context & Assumption Context & Assumption

SLIDE 15

Context-Assumption-Matrix [IEICE’16]

15 Takanobu Kobori, Hironori Washizaki, et al., “Exhaustive and efficient identification of rationales using GQM+Strategies with stakeholder relationship analysis,” IEICE Transactions on Information and Systems, Vol.E99-D, No.9, pp.2219-2228, 2016.

Target View View

Hospital al Insuran ance compan any Health senso sor maker ・・・ Hospital al Insuran ance compan any Health sensor sor maker ・・・

Insurance company can

ffer personalized products

by having medical info… Context / Assumption

Insurance company Medical info provider Personal medical info. Customized product Resource Goal depends

SLIDE 16

16

Insurance company

G S A S

Health sensor maker Medical info. provider

Expectation from the view of insurance company

Expectation from the view of health sensor maker Receive medical info. Personalized products developed Strategy Insurance product and fee can be personalized based on personal medical info. Goal Context / Assumption Sales of new products Personal fitness Confirm personal info. Management system of receiver of info. Secure management

f medical info.

Strategy Must ensure personal info. guideline Goal Context / Assumption

Insurance company Medical info. provider Confirming management systems Pati ent Getting consent

Conflict, redundancy, complement Conflict, redundancy, complement

SLIDE 17

17

Towards clear dependency and alignment

G S A S

Insurance company Medical info. provider Personal medical info. Realizing personalized insurance Objective of medical

info. use

Managing personal info.

Clarify objective

f use of medical

info. Strategy Establish personal

info. management

system Strategy

Medical info. provider

Secure management

f medical info.

Strategy Must ensure personal info. guideline Goal Context / Assumption

Health sensor maker Insurance company

Receive medical info. Personalized products developed Strategy Goal Context / Assumption Sales of new products Personal fitness Insurance product and fee can be personalized based on personal medical info.

SLIDE 18

S1 S2 S3 S5 S6 S4 S7 S1 S2 S3 S4 S6 S5 S7

Interpretive Structural Modeling (ISM)

18

Impact

S1 S2 S3 S4 S5 S6 S7 S1 1 S2 1 S3 1 1* 1 1 S4 1 1* 1 1 S5 1* 1 1 1 S6 1* 1 1 1 1* S7 1* 1 1 1

Power. Hierarchical structuring

可到達行列

S1 S2 S3 S4 S5 S6 S7 S1 1 S2 1 1 S3 1 1 1 S4 1 1 1 S5 1 1 1 S6 1 1 1 S7 1 1 1

Relation matrix Reachability matrix

SLIDE 19

ISM-based Alignment [HICSS’16]

S1 S2 S3 S4 ・・ S1 S2 S3 S4 ・・ 1 1

(1) Decompose

se into elements

ts (2)Restr tru cturing (3)Analys lysis& is& alignment

S5 S1 S2 S3 S4 S6 S7

Hierarc rchical structure re S5 S1 S2 S3 S4 S6 S7

ISM

1

Elements (especially strategies)

1 1 1 1

S5 S1 S2 S3 S4 S6 S7

GQM+Str trate ategie gies

G G G G

Reachability matrix

Conflict specified

Alignment for single GQM+Strategies model
Future: alignment over areas and stakeholders

Yohei Aoki, Takanobu Kobori, Hironori Washizaki, et al., “Identifying Misalignment of Goals and Strategies across Organizational Units by Interpretive Structural Modeling,” 49th Hawaii International Conference on System Sciences (HICSS), 2016

SLIDE 20

Pitfalls and Countermeasures

20

Pitfall Countermeasure Negative Hawthorne effects Goal-orientation Multidimensional measurements Organization misalignment Visualization

f

relationships among

rganizational

goals, strategies, and measurements Exhaustive identification of rationales Uncertain future Prediction incorporating uncertainty Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation

H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017

SLIDE 21

Uncertain Future

21

SLIDE 22

Identifying parts that are hard to maintain… How large? ELOC

N. functions

Measurement System Improvement by ML

N. Tsuda, et al. Iterative Process to Improve GQM Models with Metrics Thresholds to Detect High-risk Files, SANER 2015

Review Quality measurement Machine learning

Goal Question Measurement 22 10 25 300 150

N. functions

ELOC OK OK NG

71

N. functions

ELOC OK OK NG Improvement

X

SLIDE 23

]]

#Issues Actual Predicted

Day

Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.

Logistic Gompertz

Non-homogeneous Poisson process(NHPP)

Software reliability model (SRM) as actionable metric

23

SLIDE 24

Prediction with uncertainty

10 20 30 40 50 60 70 80 90 2 4 6 8 10 12 14

#Issues

Time

ActualData Our model

24

Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.

SLIDE 25

Prediction with uncertainty

10 20 30 40 50 60 70 80 90 2 4 6 8 10 12 14

#Issues

Time

ActualData Our model

+

25

Lower (worst case) Upper (best case)

T1 T2

Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.

SLIDE 26

Uncertainty patterns and prediction

26

0.5 1

Constant Increase Decrease

Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.

SLIDE 27

Pitfalls and Countermeasures

27

Pitfall Countermeasure Negative Hawthorne effects Goal-orientation Multidimensional measurements Organization misalignment Visualization

f

relationships among

rganizational

goals, strategies, and measurements Exhaustive identification of rationales Uncertain future Prediction incorporating uncertainty Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation

H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017

SLIDE 28

Self-Certified Quality

28

Image: How to Spot a Fake: Avoiding Degree Mill Scams https://wenr.wes.org/2015/06/spot-fake-avoiding-degree-mill-scams

SLIDE 29

ISO/IEC 25000 SQuaRE-based Quality Measurement and Benchmark

29

Q1. Any path going through internal

servers only?

Q2. Any path going through outside

servers?

Q3. Any P2P communications?

…

G. The events or actions cannot

be repudiated later through communication channels (paths).

M. Signed communication path ratio

= #Signed_paths / #Total_paths E.g. Non-repudiat ation Waseda U. Team Vendor

1 Concretize SQuaRE measurements by GQM 2

Prepare measurement methods: data forms, static analysis, questionnaire, user- testing

3 Conduct code static analysis, user-testing Fill data forms, questionnaire 4 Measure and evaluate quality

Scores by using percentile

E.g., Top 30% = 0.7 High Low #Products Measured value

H. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on

ISO/IEC 25022 and ISO/IEC 25023, IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)

SLIDE 30

21 Japanese Products Measurement

30

Functional suitability Performance efficiency Compatibility Usability Reliability Security Maintainability Portability Effectiveness Efficiency Satisfaction Freedom from risk Context coverage

Low security and compatibility in some products
Necessary to address these in IoT era
H. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on

ISO/IEC 25022 and ISO/IEC 25023, IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)

SLIDE 31

Relationships among characteristics

31

Internal/External Quality Quality in Use

p-value < 0.1 p-value < 0.1

性能効率性互換性使用性信頼性

セキュリティ

保守性移植性有効性効率性満足性

リスク回避性利用状況網羅性

機能適合性

0. 31
0. 19
0. 72
0. 37
0. 05
0. 50
0. 31
0. 14
0. 52
1. 00
1. 00
1. 00

性能効率性

0. 44
0. 24
0. 36
0. 17
0. 37
0. 32
0. 32
0. 10 -0. 50 -0. 50 -0. 50

互換性

0. 04
0. 17
0. 06
0. 36
0. 04 -0. 14
0. 05
0. 50 -0. 50 -0. 50

使用性

0. 17
0. 21
0. 11
0. 44
0. 09 -0. 20 -1. 00 -1. 00 -1. 00

信頼性

0. 30
0. 41
0. 45
0. 08
0. 11
1. 00
1. 00
1. 00

セキュリティ

0. 06
0. 19
0. 64
0. 34
0. 50
0. 50
0. 50

保守性

0. 26
0. 29
0. 01
1. 00
1. 00
1. 00

移植性

0. 21
0. 67
0. 50
0. 50
0. 50

有効性

0. 03
1. 00 -1. 00 -1. 00

効率性

1. 00
1. 00
1. 00

満足性

1. 00
1. 00

リスク回避性

1. 00

Func. Perf. Comp. Usa. Relia. Sec. Main. Port. Effe. Effic. Sati. Free.

Perf. Comp. Usa. Relia. Sec. Main. Port.
Effe. Effic. Sati. Free. Cont.
Negative correlation between usability

and functionality.

Need to adopt user-centered

development

H. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on

ISO/IEC 25022 and ISO/IEC 25023, IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)

SLIDE 32

Appropriate design Inappropriate design

Security Patterns and Testing

32

Role-based access control (RBAC) pattern

SLIDE 33

TESEM: Test Driven Secure Modeling Tool

[ARES’13][ARES’13][IJSSE’14][ICST’15][Information’16]

33

Security Design Pattern

Problem Solution Context

Test design as requirement

! create Actor ! create UI : ! create Subject..

Test Script Test case testing

[ARES’13] Validating Security Design Pattern Applications Using Model Testing, Int’l Conf. Availability, Reliability and Security [ARES’14] Verification of Implementing Security Design Patterns Using a Test Template, Conf. Availability, Reliability and Security [IJSSE’14] Validating Security Design Pattern Applications by Testing Design Models, Int’l J. Secure Software Engineering 5(4) [ICST’15] TESEM: A Tool for Verifying Security Design Pattern Applications by Model Testing, IEEE ICST’15 Tools Track [Information’16] Implementation Support of Security Design Patterns Using Test Templates, Information 7(2)

testing

SLIDE 34

34

nSuccess: succeeded });

}; function succeeded() { disableAll(); jumpToConfirm(); }; window.onload = setEventHandler; function setEventHandler() { $(“reg_type”).onchange = calcPrice; ・・・ $(“reg_addcart”).onclick = addCart; }; function calcPrice() { ・・・ }; function addCart() { if(isValidInput()) { reqRunTrans(); } else { alert(“Invalid user inputs”); } }; function reqRunTrans() { new Ajax.Request(“runTrans.php”, { method: “GET”, parameters: getParams(),

nSuccess: succeeded });

}; function succeeded() { disableAll(); jumpToConfirm(); };

SLIDE 35

35

Finite State Machine Extraction

…………………………………………… ……………………………………………

function setEventHandler() { $(“addcart”).onclick=addCart;

…………………………………………… …………………………………………… …………………………………………… ……………………………………………

function setEventHandler() { $(“addcart”).onclick=addCart;

…………………………………………… ……………………………………………

nclick

setEventHandler addCart

E.g. User Action [Mahamoff’06] “Prevent multiple calls of specific user event handler”

Y. Maezawa, K. Nishiura, H. Washizaki, S. Honiden, Validating Ajax Applications Using a Delay-Based Mutation Technique”,

29th IEEE/ACM International Conference on Automated Software Engineering (ASE 2014)

Y. Maezawa, H. Washizaki, Y. Tanabe and S. Honiden , “Automated Verification of Pattern-based Interaction Invariants in Ajax

Applications, 28th IEEE/ACM International Conference on Automated Software Engineering (ASE2013)

Duplicate

rder

Duplicate

rder

[Mahamoff’06] M. Mahamoff, “Ajax Design Patterns”, O’Reilly Media Inc., 2006.

SLIDE 36

36

nSuccess: succeeded }); };

nSuccess: succeeded }); };

function succeeded() { disableAll(); jumpToConfirm(); };

SLIDE 37

Pitfalls and Countermeasures

37

Pitfall Countermeasure Negative Hawthorne effects Goal-orientation Multidimensional measurements Organization misalignment Visualization

f

relationships among

rganizational

goals, strategies, and measurements Exhaustive identification of rationales Uncertain future Prediction incorporating uncertainty Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation

SLIDE 38