W W HAT IS T EST D RIVEN D EVELOPMENT ? T D D ? TDD is a - PowerPoint PPT Presentation

A C OMPARATIVE C ASE S TUDY ON THE I MPACT OF T EST -D RIVEN D EVELOPMENT ON P ROGRAM D ESIGN AND T EST C OVERAGE M ARIA S INIAALTO AND P EKKA A BRAHAMSSON ESEM, 2007 Irena Pletikosa Cvijikj Softvare Engineering Seminar Softvare Engineering Seminar ETH Zurich, 30.03.2010

C ON  Introduction NTE  Empirical Body of Evidence ENT  Empirical Results from a  Empirical Results from a Comparative Case Study T  Conclusions 2

I NTRODUCTION “...TDD is one of the most fundamental practices 3 enabling the development of software in an agile and iterative manner…”

W W HAT IS T EST D RIVEN D EVELOPMENT ? T D D ? TDD is a software development technique based on the repetition of a short development cycle: •Write a test •Write the code to pass the test •Refactor the code for better quality. No Write a test Pass? Yes No No Yes Refactor Pass? Write some code 4

W W HY T EST D RIVEN D EVELOPMENT ? T D D ?  Improves test coverage.  Leads to modularized flexible extensible code  Leads to modularized, flexible, extensible code.  Enhances developers job satisfaction/confidence.  Enables simultaneous work on same code.  Limits number of defects.  Increases productivity. 5

M M OTIVATION FOR T HIS P APER T P Confirm existing claims by: C fi i i l i b •Organizing existing body of evidence O i i i ti b d f id •Conducting comparative case study H1: TDD leads to increased test coverage H2: TDD improves design quality 6

E MPIRICAL B ODY OF E VIDENCE “…there are a few studies addressing the impact of 7 TDD on program design and are currently very scarce…”

C C LASIFICATION OF TDD S TUDIES TDD S Type Subjects Context #Studies Industry y Industrial Real project p j 4 developers Semi–industry Industrial Experimental task 4 developers d l Student Real project 1 developers p Academic Student Experimental task 7 developers Total 16 8

S S UMMARY OF R ESULTS R Positive Results Negative Results  Increased code quality  Reduced code quality  High test coverage  Low test coverage  Increased productivity  Reduced productivity  Good acceptance  Strong reluctance to adopt  Reduced late integration R d d l i i  Greater incidence of failures G i id f f il problems at the acceptance level 9

C C ONCLUSION ? ? Productivity Code Quality Code Quality Acceptance Test coverage 10 No meaningful conclusion! N No meaningful conclusion! N i i f l f l l l i i ! !

E MPIRICAL R ESULTS FROM A C C OMPARATIVE C ASE S TUDY C S “…the main goal of this comparative empirical 11 evaluation of TDD is to explore the impact of TDD on program design and test coverage…”

S S TUDY D ESIGN D  Controlled case study  3 projects real products for real customer  3 projects - real products for real customer  Project duration: 9 weeks (not simultaneous)  Participants: senior undergraduate students  Programming language: Java  Agile development method: Mobile-D 12 [Ihme, T. and Abrahamsson, P. 2004]

C C ASE P ROJECTS S UMMARY P S Project 1 Project 2 Project 3 # of developers p 4* 5* 4** Dev. technique Test-last Test-last TDD Application type pp yp Web Mobile Web Product concept Research data Stock market PM tool management browser Product size (LOC) 7700 7000 5800 * All * All participant had some industrial experience ti i t h d i d t i l i ** Only one participant worked in industry before 13

R R ESULTS Coupling Between Object Classes Lack of Cohesion in Methods Test Coverage 14

OO M OO M ETRICS D EFINITIONS D Coupling Between Objects is… p g j • …the count of the classes to which this class is coupled. Two classes are coupled when methods declared in one class use methods or instance variables of the other class Lack of Cohesion in Methods is… • …the number of different methods within a class that reference a given instance variable. 15 [Chidamber and Kemerer 1994, Henderson-Sellers 1996]

CBO R CBO R ESULTS 16

LCOM R LCOM R ESULTS 17

T T EST C OVERAGE C 18

S S UMMARY OF R ESULTS R  WMC, DIT, NOC, RFC: no significant difference  CBO: no conclusion can be made  CBO: no conclusion can be made  LCOM: experience in TDD usage is required  Test coverage: significant increase  Productivity: no effect 19

T T HREATS TO V ALIDITY V Differences in: •programming experience, •level of complexity for projects, •size of the project and •size of the project, and •distribution of the work. Non-TDD developers were… •encouraged to write tests but encouraged to write tests, but… •…not informed about the test coverage measurement. Use of students as study subjects. 20

C ONCLUSIONS “…the results presented in this paper are 21 important as they contribute to the gradual build up of empirical evidence on software engineering innovations…” innovations…

C ONCLUSIONS Claims that TDD leads to more loosely coupled objects can not be confirmed coupled objects can not be confirmed. TDD does not automatically result in highly cohesive code. TDD leads to significantly increased test coverage. 22

A PPENDIX 23

C C RITICISMS FOR TDD TDD  Does not compensate the lack of up-front design.  Not suitable for security and multithreaded  Not suitable for security and multithreaded applications.  Rapid changes cause expensive breakage in tests.  R id h i b k i t t  Lack of skills produce inadequate test coverage.  Tests become part of the maintenance overhead.  May bring a false sense of security. y g y 24

E E XISTING B ODY OF K NOWLEDGE B K Details on Previous Studies 25

TDD TDD IN I NDUSTRIAL S ETTINGS I S Reference # of subjects Study focus/Comparison Bhat and Nagappan 2 projects, Non-TDT 11-14 developers Lui and Chan 2 teams Traditional test-last Williams et al and Willi t l d 9 d 9 developers l Ad h Ad-hoc unit testing it t ti Maximillien and Williams Damm et al. 2 projects Specialized testing tool 26

R R ESULTS IN I NDUSTRIAL S ETTINGS I S Positive Results Negative Results  Increased code quality  Increased development time/reduced productivity  Increased test coverage g  Tests used as auto documentation  Improved task estimation  Improved process tracking  Defect rate reduction  Decreased defect repair time  Decreased project lead time D d j l d i  Daily integration reduces 27 late integration problems late integration problems

TDD TDD IN S EMI -I NDUSTRIAL S ETTINGS S I S Reference # of subjects Study focus/Comparison Canfora et al. 28 developers Traditional test-last М üller 5 TDD and 3 Traditional projects conventional projects G George and Williams d Willi 24 d 24 developers l T Traditional test-last diti l t t l t Geras at al. 14 developers Traditional test-last Abrahamsson et al. Ab h t l 4 d 4 developers l E Exploratory data l t d t 28

R R ESULTS IN S EMI -I NDUSTRIAL S ETTINGS S I S Positive Results Negative Results  Better performance  Increased development predictability time/reduced productivity  Improved code quality  Greater incidence of failures at the acceptance  Increased test coverage l level l  Increased number of tests  Strong reluctance to adopt  More frequent test TDD TDD running  Developers didn’t see the benefits 29

TDD TDD IN A CADEMIC S ETTINGS A S Reference # of subjects Study focus/Comparison Janzen and Saiedian 3 teams Iterative test-last, no tests Kaufmann and Janzen 8 developers Test-last Müller and Hagner 19 developers Traditional test-last Pancur et al. 38 developers Iterative test-last Erdogmus et al. 24 developers Iterative test-last Steinberg / Exploratory data Edwards 59 developers Automated tool, no tests 30

R R ESULTS IN A CADEMIC S ETTINGS A S Positive Results Negative Results  Increased productivity  Low test coverage  TDD accepted after trying  Concerns regarding complexity and coupling  Improved code quality  Lower final reliability on  Increased confidence acceptance test acceptance test  Better program/requirements  Reduced code quality understanding  Adopting TDD was difficult  Adopting TDD was difficult  Fast and correct use of methods  Fast and correct use of methods and it was found as not very  Reduced debugging and effective refactoring effort  Less faults and easier correction  Increased cohesion / looser 31 coupling

O O BJECT O RIENTED M ETRICS O M 32

O O BJECT -O RIENTED M ETRICS O M WMC – Weighted Methods per Class DIT – Depth of Inheritance Tree NOC – Number Of Children CBO – Coupling Between Objects RFC – Response for a Class LCOM – Lack of Cohesion in Methods 33