"Performance and Security Testing in Agile Development" - - PDF document
AW4 Class 6/9/2010 12:45:00 PM "Performance and Security Testing in Agile Development" Presented by: Tracy DeDore Hew lett-Packard Brought to you by: 330 Corporate Way, Suite 300, Orange Park, FL 32073 888 268 8770 904
Class 6/9/2010 12:45:00 PM
Presented by: Tracy DeDore Hew lett-Packard
Brought to you by:
330 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
Hewlett-Packard Tracy DeDore is the Agile Solution Marketing Manager for HP’s BTO Software Application organization. She is a twenty-four year veteran of HP and has held a variety
and software product marketing, as well as ten years managing network and mobility solution development and marketing.
June 9, 2010 Tracy DeDore
Incorporating Performance and Security Testing in Agile Development
Agenda
and constraints
without sacrificing quality
The promise of Agile
To realize the promise of Agile, it must move from a development practice to a delivery practice
– Developers – Project Managers – Business Analysts – QA – Performance Engineers – Security Specialists/Penetration Testers
Agile delivery
stakeholders
processes
assets and artifacts to accelerate work
speed issue resolution and improve decisions
Key Agile management challenges
QA Dir. VP Apps
Rqmts. Lead
QA Dir.
lead “Will the business benefit from Agile?” “Why is Agile not working for us?” “Developers go one way, QA another. How do we better collaborate?” “No more point solutions. I need a unified vision.” “We’ll go Agile, even if no
“Just get rid of the process overhead and let us work!” “Requirements, QA – these guys don’t get it.” “It’s not clear what Agile means for us.” “Some think Agile means ‘No more QA’!” “We’re building in sprints, but still testing at the end.” “Where do we fit?” “We need a simpler way to capture reqts.” “I know change is part
know the right features made it in?” “We’re building so fast we have no time to performance test.” “I don’t know if performance is improving or getting worse with each sprint.” “How can we test earlier, when the environment isn’t even in place?”
The challenges of Agile delivery
Bottom line: Moving to Agile is a big change
QA practices
paramount
every iteration
“You can never be agile without automated testing solutions.”
Gartner AADI Conference, Dec. 09, Matt Hotle, VP Distinguished Analyst
“You can never be agile without automated testing solutions.”
Gartner AADI Conference, Dec. 09, Matt Hotle, VP Distinguished Analyst
“As companies move to Agile, and as portfolios shift from version 1 into maintenance, the coding costs may drop, but the cost to test the application continues to rise. Thus, it is key to find ways to reduce the costs to test software and drive automation.”
SOA Testing: Confronting the Nightmare of Testing Shared Services”, Application Architecture, Development and Integration Conference, December 7‐9, 2009 Thomas Murphy, Gartner Research Director
“As companies move to Agile, and as portfolios shift from version 1 into maintenance, the coding costs may drop, but the cost to test the application continues to rise. Thus, it is key to find ways to reduce the costs to test software and drive automation.”
SOA Testing: Confronting the Nightmare of Testing Shared Services”, Application Architecture, Development and Integration Conference, December 7‐9, 2009 Thomas Murphy, Gartner Research Director
Why the QA Director (and team) are key to Agile success
Jan Feb Mar Apr May Jun
RQMT RQMT DESGN DESGN TEST TEST CODE / UT CODE / UT
Jan Feb Mar Apr May Jun
Agile delivery – as it should be
Time-boxed for focus Surfaces issues sooner Hands-on w ith stakeholders Designed for change Rigorous, cumulative testing True measure
Jan Feb Mar Apr May Jun
RQMT RQMT RQMT RQMT DESIGN DESIGN CODE/UT CODE/UT RQMT RQMT DESIGN DESIGN CODE/UT CODE/UT DESIGN DESIGN CODE/UT CODE/UT
CODE/UT CODE/UT
Agile delivery – as it too often is
Time-boxed for focus Surfaces issues sooner Hands-on w ith stakeholders Designed for change Rigorous, cumulative testing True measure
Jan Feb Mar Apr May Jun
RQMT RQMT RQMT RQMT DESGN DESGN CODE/UT CODE/UT RQMT RQMT DESGN DESGN CODE/UT CODE/UT DESGN DESGN CODE/UT CODE/UT
CODE/UT CODE/UT
Agile delivery – as it too often is
Time-boxed for focus Surfaces issues sooner Hands-on w ith stakeholders Designed for change Rigorous, cumulative testing True measure
Scrum and Waterfall together in a single project so as to ensure failure at a faster rate than with Waterfall alone.
Scrum and Waterfall together in a single project so as to ensure failure at a faster rate than with Waterfall alone.
Waterfall
“sprint-time”
project teams
Some reasons the Agile vision goes unrealized
Why?
– Unit testing is mistaken for system test – Regular regression testing not seen as viable in “sprint‐time” – Change impact analysis is haphazard – “Scrummerfall” – The organizational impacts are underestimated
Reasons the Agile vision goes unrealized
“Agile is still a relatively new topic in the realm of software testing… the role of the traditional software test and QA
Don't Let Short‐Term Agile Create Long‐Term Pain, Gartner Apr. 09
“Agile is still a relatively new topic in the realm of software testing… the role of the traditional software test and QA
Don't Let Short‐Term Agile Create Long‐Term Pain, Gartner Apr. 09
caused by application failures, costing an average of $100k per hour for mission-critical apps
Gartner, 2008
Gartner, From Concept to Production, Software Changes and Configuration Management, April 2008Gartner, From Concept to Production, Software Changes and Configuration Management, April 2008
Typical cost of testing in a development project
Number of defects introduced at the requirements phase
Amount of effort required to fix poor requirements
The no1 leading cause of IT w aste is poor defect mngt and rew ork
Cost to repair a defect in production vs. requirements
The cost of poor quality
Req Dev Test Prod Where Defects Are Detected Req Dev Test Prod Where Defects Are Introduced Req Dev Test Prod Relative Cost to Fix Req Dev Test Prod Actual Cost Cost
Defects Ideal Cost
Source: NIST 2002 RTI Project 7007.011
X X
Potential Value of Earlier Compliance and Testing
=
The value of effective quality management
automation
business process testing in one platform
that delay testing and increase environment costs
and modern technologies
Agile testing challenges
Eliminate rework and reduce cost across the enterprise
Customers need a Quality Management solution that enables their business objectives
Improve time to market Mitigate risk
Agile Project Management
Agile project management
Release planning through the hardening sprint
Functional, performance, and security requirements
RELEASE BACKLOG
Product backlog SPRINT 1 SPRINT 2 SPRINT 3
…N
HARDENING SPRINT
SPRINT BACKLOG
Functional, performance, and security should be addressed starting with the release planning phase AND the hardening sprint Part of every iteration
RELEASE
PLANNING
SPRINT
PLANNING
2-4 weeks 2-4 weeks 2-4 weeks 2-4 weeks
GUI LAYER GUI LAYER
GUI 1 (.Net) GUI 1 (.Net) GUI 2 (Ajax) GUI 2 (Ajax) GUI 3 (Java) GUI 3 (Java)
PROCESS LAYER PROCESS LAYER
Business Process1 Business Process1 Business Process 2 Business Process 2
BUSINESS LAYER (Services, Components, API) BUSINESS LAYER (Services, Components, API)
Component 1
(JAVA / .Net API)
Component 1
(JAVA / .Net API)
Component 2
(Web Service)
Component 2
(Web Service)
Component 3
(JMS Service)
Component 3
(JMS Service)
Multiple components, multiple sprints, multiple moving parts
Modern Applications + Agile = Challenge
SPRINT TESTING SPRINT TESTING HARDENING SPRINT HARDENING SPRINT C1 C1 C2 C1 C2 C3 G2 C1 C2 G1 G1 C3 C1 C2 G1 G3 G2 C3 C1 C2 G1
… …
BP1 BP2 G3 G2 C3 C1 C2 G1 C (GUI-less) component G2 GUI element BP Business Process Stable QA should be able to manually test faster within the iteration
Manual testing must be organized and immediate
C1 C1 C2 C1 C2 C3 G2 C1 C2 G1 G1 C3 C1 C2 G1 G3 G2 C3 C1 C2 G1
… …
BP1 BP2 G3 G2 C3 C1 C2 G1
Regression testing at the end of each iteration (or part of continuous build)
QA should be able to automate on the available application layer as early as possible C (GUI-less) component G2 GUI element BP Business Process Stable
Automate as much and as earlier as possible
Functional testing today
Create order test
Create Order View Order Login Logout
Traditional serial testing
Update order test
Login Logout View Order Update Order Search Order
Delete order test
Login Logout View Order Search Order Delete Order
25
Create once, reuse components in multiple tests
Build a change‐friendly component testing framework
Create Order View Order Login Logout
Delete order Update order Create order
Test Data Search Order Delete Order Update Order
Assess performance and security risks upfront
Effective user stories facilitate performance and security testing
review to identify areas of high risk For example:
based on risk
the system from a single point of control
Web server
Database Internet/ WAN Simulated users Controller
Automated load testing
Performance Testing automation accelerates the testing cycle
Other solutions Automated Performance Testing
Project timeline
Triage and diagnose Script Development Fix Run Test Run Test Triage And Diagnose Script Dev.
Go live Go live
Repeat Repeat Fix
}
Less development time
Less time chasing problems + Resource efficiency =
Incorporate Performance Testing into every sprint
during system architecture and release planning
performance user story
Performance Testing in Agile – HOW?
awareness
per iteration
Pervasive dependencies and constraints can kill agility
Swing UI Order Mgmt CORBA App App Server Inventory Payments 3rd Party Outsourced Database Web 2.0 UI Database Legacy Partner App Mainframe ESB
System availability constraints delay testing efforts and increase test environment costs
Cost per transaction Scarce production resources Time consuming to replicate data for test
How to get agility in constrained environments?
Virtualize Data Virtualize Behavior Virtualize Performance
Before
Unavailable/ inaccessible Wrong Test Data Poor Performing
App Under Test App Under Test
Testing Solutions
App Under Test App Under Test
After
System Dependency Constraints Virtual Test Environments Testing Solutions
Combine functional and performance testing with service virtualization to simulate the realistic and dynamic behavior of dependent systems
Effective user stories facilitate performance and security testing
review to identify areas of high risk For example:
based on risk
Wide variety of regulations by industry and geography
Internal measures Variety of mitigation methods under development (including compliance) Regulations/compliance begin to come into force New ones under development Maturity Security Programs
Security risks have never been greater
Time
Attacks The approach
Individual fame Individual gain Loose collaboration among groups
Industrialized identity theft, and illicit information markets
The costs of a security attack are huge
$202 $202
Total average cost of a data breach per compromised record*
* Ponemon Institute, 2008 Annual Study: $U.S. Cost of a Data Breach
30,000 30,000
Average # of compromised records per breach^
X
$6.65 M
Average Total Cost per breach*
=
^Source: The Open Security FoundationCompany: Records Affected*: Cost: Organization Cost: $2.53 Billion $19 Billion $5.05 Billion Financial Services 12,500,000 $202 X = Major Retailer 94,000,000 $202 X = Government 25,000,000 $202 X =
No industry is exempt
Applications are the target
Network: Secured by firewall Servers: Protected by intrusion prevention air Applications: Unprotected and ignored
sites, over 85% showed a vulnerability that could give hackers the ability to read, modify and transmit sensitive data.
‐‐ Web Application Security Consortium
Web security risks are complex
Organizations must learn to bridge the gap
Security professionals are
applications Developers/QA are overwhelmed by security
Application Security is everyone's responsibility
QA & Security Teams
Build security in
Enterprise Security Platform
People Process Technology
Ingredients for application security success Security Testing in Agile‐ HOW?
process testing
QA practices
Incorporating Performance and Security Testing in Agile Development
delivery practice
testing into every iteration
http://www.hp.com/go/agile HP Agile Development Blog: http://www.communities.hp.com/online/blogs/agile‐development/default.aspx Future of Testing Blog: http://www.communities.hp.com/online/blogs/functionaltesting/default.aspx Requirements Management: Blog http://www.communities.hp.com/online/blogs/requirementsmanagement/defa ult.aspx LoadRunner/Performance Center Blog: http://www.communities.hp.com/online/blogs/loadrunner/default.aspx Application Security Blog: http://www.communities.hp.com/securitysoftware/blogs/