NF Risk Assessment Framework Increasing Predictability of - - PowerPoint PPT Presentation

NF Risk Assessment Framework – Increasing Predictability of Non-Functional Defects

2014

Outline

2

 Conventional approach to NFT  What is not NFT?  Why Risk Assessment?  NF Risk Assessment Framework  Framework Explained  Applicability & Benefits

Conventional Approach to Non-Functional Testing (NFT)

3

Also Release Management wants it  Application should not crash in production !! Will users get the response within 5 seconds? I have got a good 3 weeks before go-live Do we have enough CPU and Memory? Somebody’s gotta do it

OK. I know what to do. Let’s do a Load Test first and then increase the number

• f users to Stress

the system Let’s gate crash the Quality Gateway 

Non- Functional Testing ??

Pitfalls of the Conventional Approach to NFT

4

Pre-production Production

• Am I focusing too much focus
• n response time SLA only?
• Do I know what I am looking

for ?

• Am I ensuring adequate

coverage besides the happy scenarios?

• Should I test integrated with
• ther applications or defer it to

the next release?

• May be I have taken a one

size fits all approach!

• Day 1 – so far so good!
• Day 2 – all was well until 3 pm

and then seemed kind of sluggish

• Day 3 – Nothing serious but

‘system was unavailable’ for 15 min this morning (suspected due to a Gateway outage the previous night)

• Day 5 – 4 EoDs were great. But

EoW is a different story 

• Day 10 – Not as quick as it was

last week (already aging?)

Well, what constitutes Non-functional Testing?

5

Non-functional Testing Performance Testing

Wikipedia

• Baseline testing
• Compliance testing
• Documentation testing
• Endurance testing
• Load testing
• Localization testing and

Internationalization testing

• Performance testing
• Recovery testing
• Resilience testing
• Security testing
• Scalability testing
• Stress testing
• Usability testing
• Volume testing

Software Quality characteristics as per ISO 9126 Standard

• Functionality
• Interoperability
• Security
• Reliability
• Fault Tolerance
• Recoverability
• Usability
• Efficiency
• Performance
• Maintainability
• Stability
• Portability
• Adaptability
• Instability

Source: http://en.wikipedia.org/wiki/Non-functional_testing Source: http://www.sqa.net/iso9126.html

NFT – Where to start and where to end?

6

? How do I know what non-functional quality attributes to test for? ? How do I know whether my test approach is good enough? ? How do I know how much testing is enough testing? ? How do I know what can possibly go wrong in production? ? How do I know my application is ready for production?

Non-Functional Risk Assessment Therefore Assess

7

System Appreciation & Technical Assessment Develop Risk Matrix Create Risk Catalogue

Non – Functional Risk Assessment Framework Risks – Tests Traceability Start Risk Assessment

Step 1: System Appreciation and Technical Assessment

Study Product Architecture and Design Understand Future Deployment and Workload Characteristics

• Architecture,

Protocols

• Transaction Model
• Interfaces
• QoS
• Data retention
• Data replication
• Overlapping

processes

• Infra deployment view
• Capacity, Network
• HA and DR
• Workload pattern
• Volumetrics
• Customer behavior
• Business growth
• Co-existing components

8

Historical Non-Functional Incidents Analysis Incident Description Technical Analysis Non-Functional? (Y/N) NF Domain Class of Issue Sub-Class Potential way of detecting the issue Applicable NF Test

9

Step 2: Develop NF Risk Matrix

Risk Matrix = Threats Vs Focus Areas

“Threat” – technical attribute or event that can impact the non-functional quality of the SUT “Focus Area” – a component or set of functionalities in the SUT that is critical to the non-functional quality of the SUT

Dedicated center of excellence providing full- fledged performan ce testing solutions

10

Risk Matrix – Indicative Threats

• Processing Overlaps
• Concurrency
• Integration Complexity
• Network Latency
• VM Sharing
• JVM Sharing
• Database Sharing
• Large Volume Workloads
• Co-existence with Maintenance
• Vertical Scalability (lack of)
• Horizontal Scalability (lack of)
• Stress Conditions
• Prolonged Usage
• Large Volume Workloads
• Database Size
• Multi Geo Access
• Offline/Shutdown
• Incorrect Error Handling
• Zone App Crash
• VM Crash
• JVM Crash
• Database Crash
• Shared Resources
• Unconstrained Resource Usage

Dedicated center of excellence providing full- fledged performan ce testing solutions

11

Risk Matrix – Threats vs Focus Areas

Online Transaction Processing (Transactional) EOD Processing Adhoc Reports Interfaces Processing Global App Behaviour Zone Behaviour Infrastructure Utilization

NF Domain Threat Infrastructure Processing Overlaps r p a r r p

a

Concurrency p r g a p r

a

Integration Complexity a a w p w r

g

Network Latency r g a w P a

r

Processing Overlaps a a g g a p

w

Stress Conditions a p a r r p

a

Concurrency p a g r a a

w

Prolonged Usage r a a a p p

a

Concurrency a a r a r p

a

Large Volume Workloads p a r a g r

a

Database Size r a r w w a

w

Multi Geo Access p w a w p a

r

Offline / Shutdown r r a r r p

w

Incorrect Error Handling a p g a a a

w

Zone App Crash r r r r w r

w

VM Crash a r g a a r

w

JVM Crash a p g a a r

w

Platform (Appserver/DB) Crash r r r r r r

w Non-Functional Focus Areas >>

Scalability

NFT Risk Matrix

Resilience & Recoverability Reliability Performance TI Processing TI Systems

Dedicated center of excellence providing full- fledged performan ce testing solutions

12

Risk Matrix – Risk Ranking

Domains of Concern 1 2 3 4 5 Total

Scalability 8 25 28 14 9 84 Reliability 6 4 13 3 2 28 Performance 4 7 11 1 5 28 Resilience and Recoverability 3 18 11 3 7 42 Capacity 4 7 3 14 Interoperability 1 1 3 2 7 Compatibility 14 14

Total Count of Risks 22 59 87 24 25 217 Take-aways NF Tests will be designed targeting each of the Rank1 and Rank2 Risks These tests will also include Test Scenarios covering Rank3 Risks

Non Functional and Technical Risks Summary

Risk Ranking

Dedicated center of excellence providing full- fledged performan ce testing solutions

13

Step3: Create Risk Catalog

# 1 2 NF Domain Scalability Performance Threat Processing Overlap Multi Geo Access Sub- Threats Intra Zone Processing

• Risk

Two or more Multi Bank Entities (MBE) within a Zone could be performing different operations at the same time, leveraging the same application/OS/database resources and processing the same data set or accessing from the same data source (table/schema/database) User sites are spread across the globe, however all user access have to pass through the Global Single Sign On (SSO). There will be only one primary instance of Global App in one location and all users will be routed through this single Global app. Impact There will be intermittent delays in

• nline transaction processing (OLTP)
• r delays in Message transmission

into the Transport Client User accesses from multiple geographies to the global App and the response therefore will potentially be slow, influenced by the bandwidth congestion over the wide area network (WAN) between the user sites and the global app Parameters to Measure OLTP Response Time Global Dashboard Response Time Focus Area Zone Behavior Global App (SSO, Dashboard)

Dedicated center of excellence providing full- fledged performan ce testing solutions

14

Risks – Tests Traceability

15

Study Product Architecture and Design Historical Non-Functional Incidents Analysis Understand Future Deployment and Workload Characteristics

Non-Functional Risk Matrix

Non-Functional Focus/Impact Areas >> Online Transaction Processing (Transactional) EOD Processing Adhoc Reports Interfaces Processing Global App Behaviour Zone Behaviour Infrastructure Utilization

NF Domain NF Threat Processing Systems Infrastructure Scalability Processing Overlaps r p a r r p

a

Horizontal Scalability (lack of) r a w a a r

w

Reliability Processing Overlaps a a g g a p

w

Prolonged Usage r a a a p p

a

NF Risk Catalogue Risks – Tests Traceability

Non-Functional Risk Assessment Framework

Applicability

16

 COTS providers/product development initiatives  Large IT initiatives (e.g. platform revamps)  Large infrastructure consolidations/transformations

X Stable/matured systems X Periodic releases X Minor enhancements

Benefits & Advantages

17

 Gain a precise understanding of the technically vulnerable areas of the SUT  Develop an exhaustive repository of non-functional test scenarios  Ability to design tests to simulate the specific technical risks  Ensure maximum possible coverage and traceability of the NF risks in the SUT  Predictability into all probable outcomes in production in the event of a technical failure or an unexpected workload situation or projected business growth

Benefits:

Thank you! Q ? A

Vijayanand Chelliahdhas

vijayanand.c@hcl.com vijay.c.anand@gmail.com

18

Benefits: