W13 Wednesday, May 17, 2006 3:00PM P ROGRESSIVE P ERFORMANCE T ESTING - PDF document

BIO PRESENTATION SUPPLEMENTAL MATERIALS W13 Wednesday, May 17, 2006 3:00PM P ROGRESSIVE P ERFORMANCE T ESTING : A DAPTING TO C HANGING C ONDITIONS Jeff Jew ell ProtoTest LLC International Conference On Software Testing Analysis and Review May 15-19, 2006 Orlando, Florida USA

Jeff Jew ell Jeff Jewell is a senior software testing consultant with ProtoTest in Denver, Colorado, and is a Certified Software Test Engineer. Jeff has more than ten years of experience in software development and quality assurance. He has held lead and management positions in software testing at a variety of companies, from small web startups to developers of ERP systems. He has experience with automated regression testing of web and Windows client applications, and performance testing of web applications. Jeff’s tool experience includes implementing and using both commercial and open source test tools for test management, automated testing, performance testing, and defect tracking. As a ProtoTest consultant Jeff has been involved in a number of different performance testing projects for clients. He has also conducted QA process and test tool assessments and implementations for a number of clients. Jeff also helps to develop and deliver ProtoTest’s software testing training classes.

Progressive Performance Progressive Performance Testing Testing Adapting to Changing Conditions Jeff Jew ell ProtoTest, LLC w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Overview Overview • Two projects covered – Online Banking – E-commerce Web Site • For each project will show – Initial Approach – Initial Results and Issues – Test and System Adaptations – Intermediate Results and Issues – Test and System Adaptations – Final Results – Lessons Learned ProtoTest, LLC Slide 2 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Online Banking Project Online Banking Project • Rewrite of existing banking system – New ASP.NET front end – Third party legacy middleware and back-end • Performance Test Goals – Match performance of previous system – Test peak number of concurrent users – Page loads usually under 5 seconds ProtoTest, LLC Slide 3 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Tool Selection Tool Selection • Semi-formal tool selection process – Based on requirements – Budget a big concern • Selected OpenSTA – Open Source tool – Active user & developer communities – Supported recording HTTPS ProtoTest, LLC Slide 4 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Initial Approach Initial Approach • Script typical user tasks – Login – Account Detail – Account History – Balance Transfer • System administrators to monitor servers – Web, Application, and Database Servers – Specific processes identified to monitor • Establish baselines with initial test runs • Initial pool of 200 test user accounts ProtoTest, LLC Slide 5 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Initial Results Initial Results • Poor login performance with light load ProtoTest, LLC Slide 6 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Initial Troubleshooting Initial Troubleshooting • Only login process slowed down – Login includes authentication and account info – Web server still responded to other requests – No significant processor usage on web and application servers • No monitoring of database servers – Some servers shared with production – Not concerned about database performance – Harder to monitor ProtoTest, LLC Slide 7 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Initial Issues and Resolutions Initial Issues and Resolutions • Page size for accounts page > 200KB – ASP.NET session state variables large – Reduced initial size to < 30KB • Login database server maxed out – Only used for authentication – Old hardware – Shared with production – Replaced server hardware ProtoTest, LLC Slide 8 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Test Adaptations Test Adaptations • Better server monitoring • Focusing only on login script – Login seemed to be the biggest problem – Retrieved all current account info • Better handling of session state • Better logging of test info ProtoTest, LLC Slide 9 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Intermediate Results Intermediate Results • Login performance much improved • Still higher than expected ProtoTest, LLC Slide 1 0 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Intermediate Issues & Intermediate Issues & Resolutions Resolutions • Accounts info now a bottleneck • Application server doing the most work – Java processes – Third party applications • Bypassed some middleware processes • Optimized SQL Queries • First run of each day was slow due to server caching ProtoTest, LLC Slide 1 1 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Final Results Final Results Login Timer Comparison 30.00 25.00 Seconds 20.00 15.00 10.00 5.00 0.00 2 5 8 1 4 7 0 3 6 9 2 5 8 1 4 7 0 1 1 1 2 2 2 2 3 3 3 4 4 4 5 Active Users April 19 Run May 4 Run ProtoTest, LLC Slide 1 2 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Lessons Learned Lessons Learned • Faulty assumptions waste time – Did need to worry about database servers – Hardware was an issue • Didn’t need all the scripts we originally planned • Didn’t need as many VUs as we thought • Had to rerecord scripts after code changes • Account variations created test anomalies • Understand server caching mechanisms ProtoTest, LLC Slide 1 3 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

E- - Commerce Project Commerce Project E • Rewrite of web-based order entry system • Company located in Dallas • Java back-end, Oracle DB, HPUX servers • Third party payment processing • Performance Test Goals – Want to own test scripts when finished – Initial goal of 50 concurrent users – Transactions under 10 seconds – Average 60 items on each order ProtoTest, LLC Slide 1 4 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Initial Approach Initial Approach • Lots of test scripts – Main scripts submitted orders • 3 different payment methods • Random # of items on orders (30-100) • Random items selected for each order – Secondary scripts somewhat useful • 30 test user accounts for each payment type • Lots of logging of script info • Testing remotely from Denver office ProtoTest, LLC Slide 1 5 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Initial Results – – Timers Timers Initial Results • Poor timer values across the board ProtoTest, LLC Slide 1 6 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Initial Results – – Portal Page Portal Page Initial Results HTTP Response Time vs. Response Size Portal HTTP Get vs. Timer 40,000 150 35,000 Bytes or Milliseconds 30,000 100 Seconds 25,000 20,000 50 15,000 10,000 0 5,000 Elapsed Time 0 HPPT Get Portal T03_PORTAL Timer Time Size • Primary HTTP request didn’t take much time • Other page elements contributed to timer length • Response size directly proportional to time ProtoTest, LLC Slide 1 7 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Issues & Resolutions Issues & Resolutions • No noticeable load on server processors • Bandwidth constraints on load generator – Only T1 connection from Denver office – Wasted time for all involved in test run – Planned future test runs on-site • Load balancing wasn’t working – Originally configured based on IP Address – Changed to use session cookie ProtoTest, LLC Slide 1 8 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Test Adaptations Test Adaptations • Numbers of VUs increased – Peak times on web site have 1500 users • Orders were standardized to 60 items – Numbers of items impacted performance – Variableness made comparison difficult • 1000 accounts for each payment type • Changed the user ramp-up mechanisms ProtoTest, LLC Slide 1 9 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Intermediate Results Intermediate Results – – Slow Timers Slow Timers Tim er Averages by Active Users 160 140 120 Seconds 100 80 60 40 20 0 11 -20 21 -30 31 -40 41 -50 51 -60 61 -70 71 -80 81 -90 10 1-1 1 0 11 1-1 2 0 12 1-1 3 0 13 1-1 4 0 14 1-1 5 0 1 5 1-1 6 0 16 1-1 7 0 17 1-1 8 0 91 -10 0 Virtual Users T0 6 _ PRI CE_ ORDER T0 7 _ PRI CE_ ORDER T1 2 _ CONFI RM_ SUBMI T • Pricing and Submitting orders very slow ProtoTest, LLC Slide 2 0 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0

Intermediate Results Intermediate Results – – Response Times Response Times URL Average (ms) GET http://URL/secure/login 789 POST http://URL/secure/login 1,072 GET http://URL/portal 1,047 POST http://URL/oe/buildOrder.jsp 9,192 GET http://URL/oe/orderList.jsp?showHistory=false 1,763 POST http://URL/oe/orderList.jsp 2,312 GET http://URL/oe/priceOrder.jsp 54,329 GET http://URL/account/accountPayment.jsp?fromOrder=true 2,089 POST http://URL/account/accountPayment.jsp 2,467 POST http://URL/account/accountPaymentCreditCard.jsp 2,449 POST http://URL/account/accountPaymentElectronicCheck.jsp 4,360 GET http://URL/oe/submitOrder.jsp 12,637 POST http://URL/oe/submitOrder.jsp 85,822 GET http://URL/secure/logout 958 ProtoTest, LLC Slide 2 1 w w w .prototest.com 3 0 3 .7 0 3 .1 5 1 0