To Mock Or Not To Mock? An Empirical Study on Mocking Practices - PowerPoint PPT Presentation

To Mock Or Not To Mock? An Empirical Study on Mocking Practices Davide Spadini , Mauricio Aniche, 
 Magiel Bruntink, Alberto Bacchelli @DavideSpadini

How do I test this?

With dependencies Without dependencies

Mock Unit under test Unit under test

With dependencies

With dependencies

With dependencies

Without dependencies • Faster • Easier

Without dependencies You lose in reality, after all, it is just a simulation!

So, what should we do? Not to Mock To Mock

Research Questions RQ1: What test dependencies do developers mock? RQ2: Why do developers decide to (not) mock specific dependencies? RQ3: Which are the main challenges experienced with testing using mocks?

Methodology Opensource Industrial

Methodology Sonarqube T A P 1 Alura T A P 1 VRaptor T A P 1 Spring Framework T A P 1

Methodology Sonarqube T A T A P 1 Mock P 3 P 2 P 1 13,547 dependencies Alura T A T A P 1 Mock P 3 P 2 P 1 1,665 dependencies VRaptor T A T A P 1 Mock P 3 P 2 P 1 1,333 dependencies Spring Framework T A T A P 1 Mock P 3 P 2 P 1 21,768 dependencies MockExtractor

Methodology Data collection Sonarqube T A T A P 1 T A P 1 … … Mock P 3 P 2 T n P n P 1 13,547 dependencies 674 dependencies Alura T A T A P 1 T A P 1 … … Mock P 3 P 2 T n P n P 1 1,665 dependencies 445 dependencies VRaptor T A T A P 1 T A P 1 … … Mock P 3 P 2 T n P n P 1 1,333 dependencies 438 dependencies Spring Framework T A T A P 1 T A P 1 … … Mock P 3 P 2 T n P n P 1 21,768 dependencies 621 dependencies MockExtractor Sampling

Methodology Data collection Data analysis Manual analysis Sonarqube T A T A P 1 T A P 1 … … Mock P 3 P 2 T n P n P 1 13,547 dependencies 674 dependencies Alura Sample 1 T A Manual analysis T A P 1 (2nd author) T A P 1 … … Mock P 3 P 2 T n P n P 1 1,665 dependencies 445 dependencies Validation VRaptor T A T A P 1 T A P 1 … … Mock P 3 P 2 T n P n P 1 1,333 dependencies 438 dependencies Sample 2 Spring Framework Manual analysis (1st author) T A T A P 1 T A P 1 … … Mock P 3 P 2 T n P n P 1 21,768 dependencies 621 dependencies MockExtractor Sampling

Methodology Data collection Data analysis Manual analysis Categorisation Sonarqube T A Discussion T A P 1 T A P 1 … … Commit Commit 132 Commit Mock Comments Comments Comments P 3 P 2 T n P n categories P 1 13,547 dependencies 674 dependencies Alura Sample 1 Commit 7 Comments T A Manual analysis T A P 1 (2nd author) T A P 1 … … Categories Mock P 3 P 2 T n P n P 1 1,665 dependencies 445 dependencies Validation VRaptor T A T A P 1 T A P 1 … … Mock P 3 P 2 T n P n P 1 1,333 dependencies 438 dependencies Sample 2 Spring Framework Manual analysis (1st author) T A T A P 1 T A P 1 … … Mock P 3 P 2 T n P n P 1 21,768 dependencies 621 dependencies MockExtractor Sampling

RQ1: What test dependencies do developers mock? RQ1

RQ1: What test dependencies do developers mock? Mocked Not mocked 100% 28% 31% 32% 64% 93% 94% RQ1 75% 72% 69% 68% 50% 36% 25% 7% 6% 0% Database Web service External 
 Domain 
 Java libraries Test support Dependencies object

Methodology Data collection Data analysis Categorisation Sonarqube Manual analysis T A Discussion T A P 1 T A P 1 … … Commit Commit 132 Commit Mock Comments Comments Comments P 3 P 2 T n P n categories P 1 13,547 dependencies 674 dependencies Alura Sample 1 Commit 7 Comments T A Manual analysis T A P 1 (2nd author) T A P 1 … … Categories Mock P 3 P 2 T n P n P 1 Interviews & Validation 1,665 dependencies 445 dependencies Validation VRaptor Interview T A Guideline T A P 1 T A P 1 … … Mock 3 developers P 3 P 2 T n P n P 1 1,333 dependencies 438 dependencies Sample 2 Spring Framework Manual analysis (1st author) T A T A P 1 T A P 1 … … Mock P 3 P 2 T n P n P 1 21,768 dependencies 621 dependencies MockExtractor Sampling

RQ2: Why do developers decide to (not) mock a dependency? RQ2

RQ2: Why do developers decide to (not) mock a dependency? RQ2 Vehicle Car Plane Boat Not when the Mock interfaces Concrete focus of the rather than a implementation test is the specific is not easy integration implementation

Methodology Data collection Data analysis Sonarqube Manual analysis Categorisation T A Discussion T A P 1 T A P 1 … … Commit Commit 132 Commit Mock Comments Comments Comments P 3 P 2 T n P n categories P 1 13,547 dependencies 674 dependencies Alura Sample 1 Commit 7 Comments T A Manual analysis T A P 1 (2nd author) T A P 1 … … Categories Mock P 3 P 2 T n P n P 1 Interviews & Validation 1,665 dependencies 445 dependencies Validation VRaptor Interview T A Guideline T A P 1 T A P 1 … … Mock 3 developers P 3 P 2 T n P n P 1 Commit Review Commit Comments Interview Comment 1,333 dependencies Comments 438 dependencies Sample 2 Transcript Spring Framework Manual analysis (1st author) T A Commit Commit T A P 1 Comments T A P 1 Comments … … Mock P 3 P 2 T n P n P 1 Survey 21,768 dependencies 621 dependencies Affinity Diagram (105 respondents) MockExtractor Sampling

RQ3: Which are the main challenges experienced with testing using mocks? RQ3

RQ3: Which are the main challenges experienced with testing using mocks? RQ3 Production quality Dealing with Legacy systems code vs coupling mocks

Discussion Open questions Mockito developer • Software quality vs mocks • He agreed on all the findings • Are slow tests more mocked? • How Mockito can help? • How faster are tests with • What Mockito is supposed to mocks? do more?