Evaluating Usefulness of Software Metrics An Industrial Experience - PowerPoint PPT Presentation

Evaluating Usefulness of Software Metrics An Industrial Experience Report @EricBouwers @avandeursen @jstvssr

Identify Define Validate attributes metrics metrics

Validity versus Usefulness Does it measure What can we do what we want to with the metric? measure?

When is a metric useful?

A metric is considered useful if the metric: … corresponds to the intuition of the measurer (Fenton et. al, Software Metrics: A Rigorous and Practical Approach, 1998) … is actively used in a decision making process (Gopal et. al, The impact of institutional forces on software metrics programs, 2005)

A four step evaluation process Embed Data Gathering Observe Interview Analyze

Context: Evaluating Implemented Architectures

Subject metrics Component Balance 0.0 0.2 0.3 0.8 E. Bouwers, et.al. Quantifying the Analyzability of Software Architectures . WICSA 2011

Subject metrics Component Independence A 1. Internal code 2 4 2. Incoming code 3. Outbound code 1 3 4. Transit code C B Less = better E. Bouwers, et.al. Dependency profiles for software architecture evaluations . ICSM 2011

Embedding The software measurement model C o m C M p o U U o m o n n n D d i p i e t u u U t V o n c p i l o n n n e t o l l i e i t m i u c t c e n n m a s o p r d t f t i u e e z l a i b e o p e p c a x n l e i l i i n n a t n y g g n d c e e n c e Analysability X X X X Modifiability X X X Testability X X X Modularity X X X Reusability X X

Embedding A bechmark based approach 0.14 0.96 Threshold Score 0.9 HHHHH 0.96 0.84 0.8 HHHHI 0.5 HHHII …. …. sort 0.3 HHIII 0.1 HIIII 0.09 0.14 0.84 0.09 0.34 HHIII Note: example thresholds

Embedding Who, what, how? http://www.freeiconsweb.com/

Data gathering Memos and interviews 49 memo’s 17 different consultants 11 different customers/suppliers 30 minutes interviews with 11 consultants Open discussion: ‘How do you use ….’ Closed questions (1-5 scale): • How useful do you find the metric? • Does it make your job easier?

Overall results from observations Model introduction Targeted improvements Definition of actions Effect prediction Decision making Start of discussions Application Steering Communication device Context Small systems Effort prediction Intuition Older technologies Linear Equation Influence of nr of components Component Balance Volume metric Dependency types Implementation File-system versus mental model Component False positives Component Definition System Scope Independence Cross language Subjectivity dependencies

Specific results Decision making

Specific results Communication device File-system Measurement Mental model Chromatix Pulse pack - http://iconsforlife.com - Alessandro Rei Dark Glass

Specific results

Interviews Results of talking to 11 experts Not easy to use But useful 5 ! 4 ! Frequency ! 3 ! 2 ! 1 ! 0 ! 1 ! 2 ! 3 ! 4 ! 5 ! Scores !

Analyze 5 ! 4 ! Frequency ! 3 ! 2 ! 1 ! 0 ! 1 ! 2 ! 3 ! 4 ! 5 ! Scores !

So are the metrics useful? A 2 4 1 3 C B In the context of evaluating implemented architectures? YES!

Is this evaluation useful? Embed Observe Interview Analyze For industry? For academia? YES! YES!

To summarize Component independence Component balance Module coupling Unit complexity Unit interfacing Duplication Volume Unit size Analysability X X X X Modifiability X X X Testability X X X Modularity X X X Reusability X X 5 ! Model introduction Targeted improvements Definition of actions 4 ! Effect prediction Decision making Start of discussions Application Steering Communication device Frequency ! 3 ! Context Small systems Effort prediction Intuition Older technologies 2 ! Linear Equation Influence of nr of components Component Balance Volume metric 1 ! Implementation Dependency types File-system versus mental model Component False positives Component Definition System Scope Independence 0 ! Cross language Subjectivity 1 ! 2 ! 3 ! 4 ! 5 ! dependencies Scores ! Usefulness