SLIDE 1
www.agilegurgaon.com Meetu Arora Sr V.P. Quality Assurance - - PowerPoint PPT Presentation
www.agilegurgaon.com Meetu Arora Sr V.P. Quality Assurance - - PowerPoint PPT Presentation
www.agilegurgaon.com Meetu Arora Sr V.P. Quality Assurance Naukri.com www.agilegurgaon.com Testing Team Journey @Naukri.com Automation WATIR 2007 2007 2006 2006 2008 2008 Agile testing team 2010 2010 Inception of 2012 2012
SLIDE 2
SLIDE 3
www.agilegurgaon.com
Testing Team Journey @Naukri.com
2006 2006
2012 2012
2007 2007
2008 2008 2010 2010
2015
Automation WATIR
Agile testing team
Aut Automatio ion Sel Seleniu ium En Entir ire team eam: Ma Manual l & & Aut Automatio ion (Sele (Seleniu ium) Automation QTP Inception of Testing Team
SLIDE 4
www.agilegurgaon.com
- Identify Need
- Identify Metrics/What
- Identify Path/How
- Prerequisites
- Implementation
- Our Measurements
- Results
SLIDE 5
www.agilegurgaon.com
SLIDE 6
www.agilegurgaon.com
Identify Metrics/What
SLIDE 7
www.agilegurgaon.com
Identify Path/How
SLIDE 8
www.agilegurgaon.com
Prerequisites
- Testing Team capable of doing automation
- Test case consolidation & management
- Dedicated scrum teams v/s shared resources
- Metric Baselines
Post live defect seepage
Test case coverage Automation coverage – UT, IT, FT Automation test flakiness Automation Execution time
Velocity/Delivery
Planning Efficiency Build Quality
SLIDE 9
www.agilegurgaon.com
Implementation Implementation
SLIDE 10
www.agilegurgaon.com
Transition to Automation
35 Manual testers & 3 Automation testers TO 38 Manual + Automation = POWER testers
SLIDE 11
www.agilegurgaon.com
Transition to Automation - Challenges
- Skill mismatch
- Team dynamics
- Lack of inclination to move towards automation
- High investment in terms of time and effort
SLIDE 12
www.agilegurgaon.com
Transition to Automation - Path
- Perseverance
- Don’t fall back
- Very Small Steps
- Low hanging fruit first - ROI
- Tester empowerment through automation
- Focus on frameworks
- We created Selenium POM code generator, which has been open
sourced: https://github.com/naukri- engineering/SeleniumCodeGenerator
- TestNG and XSLT for reporting
- Generic Function library
- Coding Guidelines and Code Review Process
- Contests for motivation
- Provide migration channels
SLIDE 13
www.agilegurgaon.com
Test Case Consolidation and Management
To know more about how to integrate automation scripts with Jenkins visit our blog: http://engineering.naukri.com/2015/05/integrate-your-automation-with-jenkins/
SLIDE 14
www.agilegurgaon.com
Testing Throughout
- Testing individual stories and Integration testing as
you go along
- Progressive/Parallel automation testing
- Minimizing Automated Tests Flakiness
- Reducing our automated tests execution time
- Run automated regression suites periodically using Jenkins
To know more about Continuous Testing @ Naukri read our blog: http://engineering.naukri.com/2016/03/continuous-testing-naukri/
SLIDE 15
www.agilegurgaon.com
Automation Coverage
596 554 611 876 929 1228 1354 1508 1859 2249 2636 3047 7746 8153 8318 8679 8925 9313 10062 10283 10491 10846 10542 10641 68.76% 74.32% 75.67% 81.56% 84.55% 86.29% 90.82% 92.10% 92.72% 95.96% 96.65% 96.99% 0% 20% 40% 60% 80% 100% 2000 4000 6000 8000 10000 12000 14000 16000 April May June July August Septemebr October November December January February March Automation Coverage
- No. of Test cases
Naukri India
Not Automatable Automatable %Automation Coverage out of Automatable 124 309 347 317 336 339 422 417 402 410 415 417 1818 3072 3077 3859 4436 4576 5145 5658 5784 5861 5885 6242 25.00% 27.80% 29.29% 46.30% 58.00% 71.50% 76.80% 84.60% 88.10% 92.90% 93.00% 93.20% 0% 20% 40% 60% 80% 100% 1000 2000 3000 4000 5000 6000 7000 April May June July August Septemebr October November December January February March Automation Coverage
- No. of Test cases
Naukri Gulf
Not Automatable Automatable %Automation Coverage out of Automatable
SLIDE 16
www.agilegurgaon.com
Automation Flakiness Reduction
- 15% to 5% in Naukri
- 45% to 7% in Mobile Apps
- 17% to 7% in NaukriGulf
To know more about how we were able to optimize our tests and reduce flakiness visit our blog: http://engineering.naukri.com/2016/03/reduce- test-automation-flakiness/
SLIDE 17
www.agilegurgaon.com
Automation Execution Time Reduction
By selenium grid implementation we were able to reduce our execution time to 1/5th – From 25Hrs to 5Hrs
To know more about Selenium Grid implementation visit our blog: http://engineering.naukri.com/2015/10/parallel-testing-at-naukri/
SLIDE 18
www.agilegurgaon.com
Preventing Bugs
- Tester, developer, product owner, architect are all part
- f the backlog grooming
- Entire team focuses on defining what and how
- Product backlog grooming is done one iteration in
advance
- Testers contribute test cases upfront during this
period and add them to the user stories in form of acceptance criteria or alternate paths
SLIDE 19
www.agilegurgaon.com
Preventing Bugs
- Peer testing at developer end
- Automated Build verification tests have been created
and are run before providing builds to testers
- Progressive Automation testing approach is used.
SLIDE 20
www.agilegurgaon.com
Testing Understanding
- Put yourself in the customer shoes
- We encourage our scrum teams to have direct interaction with actual customers
– proactive & reactive
- Effective feedback loops
- Our tech support team regularly shares reports on issue patterns which are used
by scrum teams as inputs to design/test/improve systems
- Measure the customer usage pattern and use it to design test cases
- We regularly analyse user data patterns to come up with and refine our test
strategy
SLIDE 21
www.agilegurgaon.com
Building the best system
- Build Implicit Requirements
- Focus more on building positive product scenarios
- Focus on Bug Causal Analysis
- Peripheral testing: Focus v/s Defocus
To know more about peripheral testing please visit our blog http://engineering.naukri.com/2016/03/peripheral-testing/
SLIDE 22
www.agilegurgaon.com
Team responsibility for quality
- Everybody tests as and when needed
- Measure Quality at various levels
- Build Quality Meter
- Post Production Issue Seepage
SLIDE 23
www.agilegurgaon.com
Build Quality Meter
Bug Severity Bug Type: Functional, UI, product design, Implicit, Validation, Incomplete requirement, Insufficient impact analysis, Integration Environment, Configuration, Inadequate testing,DOA
SLIDE 24
www.agilegurgaon.com
Our Measurements
Sprint15-28Dec Sprint29Dec-11Jan Sprint12-25Jan Sprint26Jan-8Feb Sprint9-22Feb Sprint23Feb-7March Sprint8-21Mar Commitment 39 40 44 42 44 42 46 Completed Unplanned P1/P2 2 3 2 5 3 4 2 Completed Unplanned 6 5 1 2 4 2 5 Planned Live/Staging 6 1 2 10 5 3 4 Completed Planned 19 27 39 25 30 32 34 Target 41 41 41 41 41 41 41 5 10 15 20 25 30 35 40 45 50
Velocity Report
Indicative Data has been used for illustration purpose
SLIDE 25
www.agilegurgaon.com
Our Measurements
4 4 11 1 2 9 1 3 5 7 5 1 1 93 80.86 70.68 60.76
10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100 26th Jan'16-08th Feb'16 9th Feb'16-22nd Feb'16 23rd Feb'16-7th March'16 8th March'16-21st March'16 BUILD QUALITY % BUGS
ITERATION
Build Quality Trend
Functional Bugs UI Bugs Validation Live Implicit Integration Build Quality Bad [Less than 70] Average [70-80] Good [80-90] Excellent [90 +]
Excellent Good Average Bad
Indicative Data has been used for illustration purpose
SLIDE 26
www.agilegurgaon.com
Our Measurements
2 14 9 17 1 11 4 13 1 11 4 13 2 4 6 8 10 12 14 16 18 26th Jan'16-08th Feb'16 9th Feb'16-22nd Feb'16 23rd Feb'16-7th March'16 8th March'16-21st March'16
- NO. OF STORIES
Functional Automation Coverage
Total Stories Automatable Automated
Indicative Data has been used for illustration purpose
SLIDE 27
www.agilegurgaon.com
Our Measurements
26 Jan - 8 Feb 9 -22 Feb 23 Feb-7 Mar 8-21 March'16 Client Reported Issues (Data Fixes) 2 4 3 1 Client Reported Issues (Code Fixes) 1 2 1 Live issues (Except client issues) 2 1 3 3
2 1 3 3 1 2 1 2 4 3 1 1 2 3 4 5 6 7 8 9 10
BUG COUNT
Bugs Reported on Live
Indicative Data has been used for illustration purpose
SLIDE 28
www.agilegurgaon.com
Results
- Post live defect seepage: 50% reduction
- Test Cases: 10K increase
- Automation Coverage: 44% increase
- Automation Scripts Execution time: 20% reduction
- Automation flakiness: 20% reduction
- Velocity: 25% increase
- 15% Build quality improvement
- 30 % Planning Efficiency improvement
SLIDE 29
www.agilegurgaon.com
Key Take Away’s
- Defining the “Right Metric”
- Measure, Review, Improve ….
- For Agile testing through out focus should be on progressive/parallel
automation testing along with creating reliable tests that take minimal time to execute.
- Defect Prevention is the Key
- Team v/s Individual mind-set
SLIDE 30
www.agilegurgaon.com
SLIDE 31