SLIDE 1
To Mock Or Not To Mock?
An Empirical Study on Mocking Practices
Davide Spadini, Mauricio Aniche, Magiel Bruntink, Alberto Bacchelli @DavideSpadini
To Mock Or Not To Mock? An Empirical Study on Mocking Practices - - PowerPoint PPT Presentation
To Mock Or Not To Mock? An Empirical Study on Mocking Practices - - PowerPoint PPT Presentation
To Mock Or Not To Mock? An Empirical Study on Mocking Practices Davide Spadini , Mauricio Aniche, Magiel Bruntink, Alberto Bacchelli @DavideSpadini How do I test this? With dependencies Without dependencies Mock Unit under test Unit
SLIDE 2
SLIDE 3
With dependencies Without dependencies
SLIDE 4
Mock
Unit under test Unit under test
SLIDE 5
SLIDE 6
With dependencies
SLIDE 7
With dependencies
SLIDE 8
With dependencies
SLIDE 9
- Faster
- Easier
Without dependencies
SLIDE 10
Without dependencies
You lose in reality, after all, it is just a simulation!
SLIDE 11
To Mock Not to Mock
So, what should we do?
SLIDE 12
RQ1: What test dependencies do developers mock? RQ2: Why do developers decide to (not) mock specific dependencies? RQ3: Which are the main challenges experienced with testing using mocks?
Research Questions
SLIDE 13
Opensource Industrial
Methodology
SLIDE 14
TA P1 Sonarqube Alura VRaptor Spring Framework TA P1 TA P1 TA P1
Methodology
SLIDE 15
TA MockExtractor TA P3 Mock P1 P2
13,547 dependencies
P1 Sonarqube TA P3 Mock P1 P2
1,665 dependencies
Alura TA P3 Mock P1 P2
1,333 dependencies
VRaptor TA P3 Mock P1 P2
21,768 dependencies
Spring Framework TA P1 TA P1 TA P1
Methodology
SLIDE 16
TA MockExtractor TA P3 Mock P1 P2
13,547 dependencies 674 dependencies
TA P1 Tn Pn … … Sampling P1 Sonarqube TA P3 Mock P1 P2
1,665 dependencies 445 dependencies
TA P1 Tn Pn … … Alura TA P3 Mock P1 P2
1,333 dependencies 438 dependencies
TA P1 Tn Pn … … VRaptor TA P3 Mock P1 P2
21,768 dependencies 621 dependencies
TA P1 Tn Pn … … Spring Framework
Data collection
TA P1 TA P1 TA P1
Methodology
SLIDE 17
TA MockExtractor TA P3 Mock P1 P2
13,547 dependencies 674 dependencies
TA P1 Tn Pn … … Sampling
Sample 1
P1 Sonarqube TA P3 Mock P1 P2
1,665 dependencies 445 dependencies
TA P1 Tn Pn … … Alura TA P3 Mock P1 P2
1,333 dependencies 438 dependencies
TA P1 Tn Pn … … VRaptor TA P3 Mock P1 P2
21,768 dependencies 621 dependencies
TA P1 Tn Pn … … Spring Framework
Sample 2
Validation Manual analysis (2nd author) Manual analysis (1st author)
Data analysis Data collection Manual analysis
TA P1 TA P1 TA P1
Methodology
SLIDE 18
TA MockExtractor TA P3 Mock P1 P2
13,547 dependencies 674 dependencies
TA P1 Tn Pn … … Sampling
Commit Comments Commit Comments Commit Comments
132 categories
Commit Comments
7
Categories
Sample 1
P1 Sonarqube TA P3 Mock P1 P2
1,665 dependencies 445 dependencies
TA P1 Tn Pn … … Alura TA P3 Mock P1 P2
1,333 dependencies 438 dependencies
TA P1 Tn Pn … … VRaptor TA P3 Mock P1 P2
21,768 dependencies 621 dependencies
TA P1 Tn Pn … … Spring Framework
Sample 2
Validation Manual analysis (2nd author) Manual analysis (1st author) Discussion
Data analysis Data collection Manual analysis Categorisation
TA P1 TA P1 TA P1
Methodology
SLIDE 19
RQ1
RQ1: What test dependencies do developers mock?
SLIDE 20
RQ1: What test dependencies do developers mock?
RQ1
0% 25% 50% 75% 100% Database Web service External Dependencies Domain
- bject
Java libraries Test support
94% 93% 64% 32% 31% 28%
6% 7% 36% 68% 69% 72%
Mocked Not mocked
SLIDE 21
TA MockExtractor TA P3 Mock P1 P2
13,547 dependencies 674 dependencies
TA P1 Tn Pn … … Sampling
Commit Comments Commit Comments Commit Comments
132 categories
Commit Comments
7
Categories
Interview Guideline 3 developers
Sample 1
P1 Sonarqube TA P3 Mock P1 P2
1,665 dependencies 445 dependencies
TA P1 Tn Pn … … Alura TA P3 Mock P1 P2
1,333 dependencies 438 dependencies
TA P1 Tn Pn … … VRaptor TA P3 Mock P1 P2
21,768 dependencies 621 dependencies
TA P1 Tn Pn … … Spring Framework
Sample 2
Validation Manual analysis (2nd author) Manual analysis (1st author) Discussion
Data analysis Interviews & Validation Data collection Manual analysis Categorisation
TA P1 TA P1 TA P1
Methodology
SLIDE 22
RQ2
RQ2: Why do developers decide to (not) mock a dependency?
SLIDE 23
RQ2
RQ2: Why do developers decide to (not) mock a dependency?
Not when the focus of the test is the integration
Vehicle Car Plane Boat
Mock interfaces rather than a specific implementation Concrete implementation is not easy
SLIDE 24
TA MockExtractor TA P3 Mock P1 P2
13,547 dependencies 674 dependencies
TA P1 Tn Pn … … Sampling
Commit Comments Commit Comments Commit Comments
132 categories
Commit Comments
7
Categories
Interview Guideline 3 developers
Commit Comments Review Comment Commit Comments Interview Transcript Commit Comments Commit Comments
Survey (105 respondents)
Sample 1
P1 Sonarqube TA P3 Mock P1 P2
1,665 dependencies 445 dependencies
TA P1 Tn Pn … … Alura TA P3 Mock P1 P2
1,333 dependencies 438 dependencies
TA P1 Tn Pn … … VRaptor TA P3 Mock P1 P2
21,768 dependencies 621 dependencies
TA P1 Tn Pn … … Spring Framework
Sample 2
Validation Manual analysis (2nd author) Manual analysis (1st author) Discussion
Affinity Diagram
Data analysis Interviews & Validation Data collection Manual analysis Categorisation
TA P1 TA P1 TA P1
Methodology
SLIDE 25
RQ3
RQ3: Which are the main challenges experienced with testing using mocks?
SLIDE 26
RQ3
RQ3: Which are the main challenges experienced with testing using mocks?
Legacy systems Dealing with coupling Production quality code vs mocks
SLIDE 27
Discussion
Mockito developer
- He agreed on all the findings
- How Mockito can help?
- What Mockito is supposed to
do more?
Open questions
- Software quality vs mocks
- Are slow tests more mocked?
- How faster are tests with
mocks?
SLIDE 28