Evaluating Usefulness of Software Metrics An Industrial Experience - - PowerPoint PPT Presentation

Evaluating Usefulness of Software Metrics

An Industrial Experience Report

@EricBouwers @avandeursen @jstvssr

Identify attributes Define metrics Validate metrics

Validity versus Usefulness

Does it measure what we want to measure? What can we do with the metric?

When is a metric useful?

A metric is considered useful if the metric:

… corresponds to the intuition

• f the measurer

(Fenton et. al, Software Metrics: A Rigorous and Practical Approach, 1998)

… is actively used in a decision making process

(Gopal et. al, The impact of institutional forces on software metrics programs, 2005)

A four step evaluation process Embed Analyze

Observe Interview Data Gathering

Context:

Evaluating Implemented Architectures

Subject metrics Component Balance

• E. Bouwers, et.al. Quantifying the Analyzability of Software Architectures. WICSA 2011

0.0 0.2 0.3 0.8

Subject metrics Component Independence

B C A 1 2 3 4

• E. Bouwers, et.al. Dependency profiles for software architecture evaluations. ICSM 2011

Less = better

• 1. Internal code
• 2. Incoming code
• 3. Outbound code
• 4. Transit code

Embedding The software measurement model

V

• l

u m e D u p l i c a t i

• n

U n i t s i z e U n i t c

• m

p l e x i t y U n i t i n t e r f a c i n g M

• d

u l e c

• u

p l i n g C

• m

p

• n

e n t b a l a n c e C

• m

p

• n

e n t i n d e p e n d e n c e Analysability X X X X Modifiability X X X Testability X X X Modularity X X X Reusability X X

Embedding A bechmark based approach 0.14 0.96 0.09 0.84

Threshold Score 0.9 HHHHH 0.8 HHHHI 0.5 HHHII 0.3 HHIII 0.1 HIIII

0.09 0.14 0.84 0.96

…. ….

0.34 HHIII

Note: example thresholds

sort

Embedding Who, what, how?

http://www.freeiconsweb.com/

Data gathering Memos and interviews

17 different consultants 49 memo’s 11 different customers/suppliers 30 minutes interviews with 11 consultants Open discussion: ‘How do you use ….’ Closed questions (1-5 scale):

• How useful do you find the metric?
• Does it make your job easier?

Overall results from observations

Decision making Intuition Component Definition Application Implementation Targeted improvements Start of discussions Communication device Small systems Older technologies Influence of nr of components File-system versus mental model System Scope Subjectivity Model introduction Definition of actions Effect prediction Steering Context Effort prediction Component Balance Component Independence Linear Equation Volume metric Dependency types False positives Cross language dependencies

Specific results Decision making

Specific results Communication device

File-system Mental model Measurement

Chromatix Pulse pack - http://iconsforlife.com - Alessandro Rei Dark Glass

Specific results

Interviews Results of talking to 11 experts

0! 1! 2! 3! 4! 5! 1! 2! 3! 4! 5!

Frequency! Scores!

Not easy to use But useful

Analyze

0! 1! 2! 3! 4! 5! 1! 2! 3! 4! 5!

Frequency! Scores!

So are the metrics useful?

B C A 1 2 3 4

In the context of evaluating implemented architectures? YES!

Is this evaluation useful?

Embed Observe Interview Analyze For industry? YES! For academia? YES!

To summarize

Volume Duplication Unit size Unit complexity Unit interfacing Module coupling Component balance Component independence Analysability X X X X Modifiability X X X Testability X X X Modularity X X X Reusability X X

Decision making Intuition Component Definition Application Implementation Targeted improvements Start of discussions Communication device Small systems Older technologies Influence of nr of components File-system versus mental model System Scope Subjectivity Model introduction Definition of actions Effect prediction Steering Context Effort prediction Component Balance Component Independence Linear Equation Volume metric Dependency types False positives Cross language dependencies

0! 1! 2! 3! 4! 5! 1! 2! 3! 4! 5!

Frequency! Scores!

Usefulness