Testing 221 238 197 223 171 213 Manoj Nambiar, Tata Blue 50% - - PowerPoint PPT Presentation

109 207 246 Dark 1 255 255 255 Light 1 131 56 155 Dark 2 99 190 Light 2 85 165 28 Accent 1 214 73 42 Accent 2 185 175 164 Accent 3 151 75 7 Accent 4 193 187 Accent 5 255 221 62 Accent 6 255 255 255 Hyperlink 236 137 29 Followed Hyperlink 127 175 221 Tata Blue 50% 203 215 238 Tata Blue 25% 179 149 197 Purple 50 % 212 195 223 Purple 25 % 255 242 171 Yellow 50 % 255 249 213 Yellow 25 % 229 205 186 Brown 50 % 248 241 235 Brown 25 % 180 213 154 Green 50 % 214 231 200 Green 25 % 241 240 202 Light Green 50% 251 251 241 Light Green 25%

Title and Content

Manoj Nambiar, Principal Scientist, TCS Innovations Lab – Performance Engineering

Upcoming Challenges in Large Scale Performance Testing

Agenda

• Large Systems
• Application Topologies – IoT/streaming
• Nature of Data
• Measurement Considerations
• Devops & Performance Testing
• Summary
• 1 -

Large Scale Systems

• 2 -

Lots of hardware Lots of things Lots of data Lots of users / stakeholders Lots of code Lots of connections With growing variety in the lots

Characteristics – Problems with scale

• Developed and used by many stakeholders
• Possibly conflicting purposes and needs
• Made of heterogeneous systems
• Complex dependencies
• Continuously evolving
• Hardware, software and human failures more the norm than exception
• The whole is more than the sum of its parts
• We will assume the following for our discussion
• Owners of a system control its development
• Decisions are made rationally and are driven by technical criteria
• There is a definable problem and clear system boundaries
• 3 -

Traditional examples

• Bank
• 50 million + of account holders
• 15000 branches
• 45000 ATMs
• Stock Exchange
• 5000 brokers
• 3000 algo trading engines
• Millions of orders a second
• Department of Income Tax
• 35 million tax payers, one income declaration deadline a year.
• 4000 assessment officers
• Large marketing data and analytics system
• Billion + transactions a day
• Large securities exchange regulatory body
• Analytics over 5 petabytes of data
• 4 -

Contemporary systems – what is the system?

• 5 -

Analytics Message Routing & Event Processing Sensor Data Management

Apps Device Management

Device Agents

Things with Embedded Sensors Mobile Devices Gateway Devices Cloud Services Apps, Clients & Portals End users http(s), tcp, udp, mqtt LWM2M OPC-UA, Modbus, Continua

The bigger it gets there is more network to it!

Typical gaps with Production

• 6 -

6

Desktop Web & App Dev DB Desktop Web & App DB

Large Storage WAN Millions

• f rows

in DB Dev/Test Prod Projects run in to trouble because of the significant gap between development and production environments

LAN

Network Emulators

• 7 -

Bandwidth Latency Packet Drop Disconnect Network Emulator Developer Development/ Test Server Load Generator Real customer experience over the Internet, during development and testing on a LAN environment

Server capable of manipulating packets

Emulating multiple networks

• 8 -

WAN Emulator server Application Server

London 2 Server WAN Tokyo to Server WAN 3

Load Injectors

Achieved by

• Sets of virtual users using separate IP addresses
• Modifying routing tables/gateways on load injectors
• Use of traffic filtering, QoS, emulation, firewall & NAT on WAN emulator server

Virtual users representing London

• ffice workload

Virtual subnet configured to represent end to end enterprise network from Tokyo to server

Sidney to Server WAN

Application Network Topology in IoT

• 9 -

Analytics Message Routing & Event Processing Sensor Data Management

Apps Device Management

Device Agents

Things with Embedded Sensors Mobile Devices Gateway Devices Centralized Services Apps, Clients & Portals End users http(s), tcp, udp, mqtt LWM2M OPC-UA, Modbus, Continua

Multi-hop System WAN Topology Hop 1 Hop 2

Multi-hop set up using WAN emulator

• 10 -

WAN Emulator Server

Sensor Location 1 Sensor Location 2 Sensor Location 3 Device Agent Location 1 Device Agent Location 2 Device Agent Location 3

Centralized Services Device Emulation Servers Device Agent Servers

Some Network Emulators available today

• End to End network emulators
• Software based

Coming with load testing software – LoadRunner, Neotys Available with load testing services – StormRunner, Appvance …. Standalone (open source) software based – WANem (netem), Nistnet, ns2, Dummynet (FreeBSD) …. Hardware based – PacketStorm 6XG, Apposite Netropy ….

• Topology based network emulators (software)
• Imunes – based on Docker & Open vSwitch, Gui based
• Mininet - based on namespaces, OpenFLow
• 11 -

Network topology emulation - Imunes example

• 12 -

Mobility behavior

• 13 -

Number of innovative applications in IoT based on location System performance can be affected by a specific movement pattern

• of sensors or sensations

Strategy for capturing sensor/sensation movement

• 14 -

WAN Emulator PC

Sensor Location 1 Sensor Location 2 Sensor Location 3

Device Agents Device Agents Device Agents

Device Agent Location 1 Device Agent Location 2 Device Agent Location 3

Centralized Services

Central Virtual user /device controller

1. Maintain locations of users 2. Simulate user and/or sensation motion based on predefined crtical scenarios 3. Control number of virtual users in each location based on user/device mobility

No performance testing solution available today that takes device mobility into account Sensor/ Sensation movement model

Another challenge – Big Data

• True reproduction of Volume, Variety and Velocity aspects essential
• Velocity of data can be controlled by the virtual users and devices
• Volumes - Generating large data volumes is a time-consuming effort
• Use of parallelism must in data generation and loading
• Variety – it is not just about structured or unstructured data
• In what ways the data will grow is important
• 15 -

Aiding correct data generation

• Large volume data generation should be a vector representation
• How is my data going to grow?
• Applies to structured and unstructured data
• Rather than just
• How many sales am I expecting in the next 5 years?
• Also consider (as an example)
• How many new types of products are expected to be introduced
• What is the distribution of sales amongst products?
• How will the sales volumes distributed across the week?
• Number of sales offers in a year
• Reactions to end of season sales
• 16 -

A Simple Example

• Consider a table with 100 collumns
• T = [.….c10……c20………………………………….]
• Consider query
• Select sum(c10) from T where T.c20 = 1000
• Consider 2 cases

1. Data is uniformly distributed on c20 2. At least 80% of the rows have c20 == 1000

• The SQL query in case 2 will take much more time than in case 1
• Higher the data volumes higher the difference
• 17 -

Essential to get the distribution of data right

Tools available today

• Grid Tools
• Supports exponential and normal distribution for data values
• UpScene Advanced Data Generator
• GS Data Generator
• Redgate SQL data generator
• EMS data generator
• Datanamic data generator for multi-db
• Most tools support generation of uniformly distributed data
• between a min and max value
• No tool supports skewed data generation
• 18 -

Need for a Mature data generation tool

• 19 -

Business growth questions

Test Engineer 1. Test engineer feeds the schema 2. Test engineer creates questions for business user 3. Maps the schema with the questions

• 4. Business user answers the questions
• 5. Data generation tool

generates and loads the data on the cluster (SUT)

SUT cluster schema

Newer Systems - Data Sources and sinks

• 20 -

Analytics Message Routing & Event Processing Sensor Data Management

Apps Device Management

Device Agents

Things with Embedded Sensors Mobile Devices Gateway Devices Centralized Services Apps, Clients & Portals End users http(s), tcp, udp, mqtt LWM2M OPC-UA, Modbus, Continua

Examples

• Sensor to record in a database
• Sensor information transformed to recommendation in real time
• Sensor data transforming into an control
• Measurement technique not as common as response time

Measurement Challenges

• It may not be just about application response times any more.
• Need for one way latency measurement tools –
• may be specific to the application
• A basic framework that can be easily configured will help.
• Correlate a unit transaction across logs - all the way from source to sink
• Application support may be required
• Technologies promising “eventual consistency” could complicate real-time needs
• Percentage of passed transactions will be important to report across incoming

traffic rates

• Need for a way for the business to specify what constitutes acceptable performance
• There has to be a way to evaluate cost impacts

 based on various latency distributions

• 21 -

No known tool support at present

Devops & Performance Testing

• Problem
• Every release has to be performance assured

Even one line of code can wreak havoc

• Time to delivery constraints make Performance Testing impractical
• Opportunity
• Devops enables feedback from production back to development
• Essential to make use of this opportunity to enhance relevance of

performance tests

• 22 -

Large Scale Test Components

• 23 -

System

• Hardware
• Software
• Network
• External Services
• Data

Workload Measurement

Deployment Challenges

• Manual setup can be very time consuming
• Easy to get something wrong due to manual errors
• As systems get complex manual set-up may not be an option
• A higher level (meta) middleware is required to automate test setups
• SUT
• Load generation framework
• WAN emulators
• No common platform to orchestrate such a test setup
• 24 -

Higher level Middleware framework

• 25 -

Meta Middleware Test Setup description Software Product repository Code & Scripts repository Deployment Cloud Makes testing multiple deployment configurations feasible pull pull pull deploy

Devops – Continuous QUALITY Delivery Pipeline

• 26 -

Workload profile Data distributions

Analyze gaps

Updated virtual user scripts, scenarios Modified data generation

Development Performance Test Production

Summary

• Innovative applications and technologies force us to rethink/reconsider
• Application network topologies
• Nature of data growth
• Performance Measurement
• Increasing heterogeneity of systems make automation is all the more essential
• A truly integrated Devops setup
• Helps getting performance tests closer to production relevance
• There is a need for tools that can make this possible
• 27 -

Need for Software define performance testing!

Disclaimer: images used in this presentation

• btained via Google Images.
• Images used in this lecture courtesy to many anonymous

contributors to Google Images, and to Google Image Search.

• Many thanks!

Thank You Thank You