Testing 221 238 197 223 171 213 Manoj Nambiar, Tata Blue 50% - PowerPoint PPT Presentation

Title and Content 109 255 131 0 85 214 207 255 56 99 165 73 246 255 155 190 28 42 Dark 1 Light 1 Dark 2 Light 2 Accent 1 Accent 2 185 151 193 255 255 236 175 75 187 221 255 137 164 7 0 62 255 29 Accent 3 Accent 4 Accent 5 Accent 6 Hyperlink Followed Hyperlink Upcoming Challenges in Large Scale Performance 127 203 179 212 255 255 175 215 149 195 242 249 Testing 221 238 197 223 171 213 Manoj Nambiar, Tata Blue 50% Tata Blue 25% Purple 50 % Purple 25 % Yellow 50 % Yellow 25 % Principal Scientist, TCS Innovations Lab – Performance Engineering 229 248 180 214 241 251 205 241 213 231 240 251 186 235 154 200 202 241 Brown 50 % Brown 25 % Green 50 % Green 25 % Light Green 50% Light Green 25%

Agenda  Large Systems  Application Topologies – IoT/streaming  Nature of Data  Measurement Considerations  Devops & Performance Testing  Summary - 1 -

Large Scale Systems Lots of connections Lots of code Lots of hardware Lots of things Lots of users / stakeholders Lots of data With growing variety in the lots - 2 -

Characteristics – Problems with scale  Developed and used by many stakeholders  Possibly conflicting purposes and needs  Made of heterogeneous systems  Complex dependencies  Continuously evolving  Hardware, software and human failures more the norm than exception The whole is more than the sum of its parts   We will assume the following for our discussion  Owners of a system control its development  Decisions are made rationally and are driven by technical criteria  There is a definable problem and clear system boundaries - 3 -

Traditional examples  Bank  50 million + of account holders  15000 branches  45000 ATMs  Stock Exchange  5000 brokers  3000 algo trading engines Millions of orders a second  Department of Income Tax   35 million tax payers, one income declaration deadline a year.  4000 assessment officers  Large marketing data and analytics system  Billion + transactions a day  Large securities exchange regulatory body  Analytics over 5 petabytes of data - 4 -

Contemporary systems – what is the system? Sensor Data Management End users Device Apps Management LWM2M Message Routing Device & Event Processing Agents http(s), tcp, udp, mqtt Analytics OPC-UA, Modbus, Continua Apps, Clients & Portals Things with Gateway Devices Cloud Services Embedded Sensors Mobile Devices The bigger it gets there is more network to it! - 5 -

Typical gaps with Production Web & App Desktop Dev DB LAN Dev/Test DB Prod Web & App Large Desktop Storage Millions WAN of rows in DB Projects run in to trouble because of the significant gap between development and production environments - 6 - 6

Network Emulators Real customer experience over the Internet, during development and testing on a LAN environment Developer Bandwidth Development/ Test Server Latency Packet Drop Load Generator Disconnect Network Emulator Server capable of manipulating packets - 7 -

Emulating multiple networks Load Injectors WAN Emulator server Virtual users London 2 Server WAN Application representing London Server office workload Sidney to Server WAN Tokyo to Server WAN 3 Virtual subnet configured to represent end to end enterprise network from Tokyo to Achieved by server • Sets of virtual users using separate IP addresses • Modifying routing tables/gateways on load injectors • Use of traffic filtering, QoS, emulation, firewall & NAT on WAN emulator server - 8 -

Application Network Topology in IoT Hop 1 Hop 2 Sensor Data Management End users Device Apps Management LWM2M Message Routing Device & Event Processing Agents http(s), tcp, udp, mqtt Analytics OPC-UA, Modbus, Continua Apps, Clients & Portals Things with Gateway Devices Centralized Services Embedded Sensors Mobile Devices Multi-hop System WAN Topology - 9 -

Multi-hop set up using WAN emulator WAN Emulator Server Sensor Location 1 Device Agent Location 1 Device Emulation Servers Sensor Location 2 Device Agent Location 2 Sensor Location 3 Device Agent Location 3 Centralized Services Device Agent Servers - 10 -

Some Network Emulators available today  End to End network emulators  Software based  Coming with load testing software – LoadRunner, Neotys  Available with load testing services – StormRunner, Appvance ….  Standalone (open source) software based – WANem (netem), Nistnet, ns2, Dummynet (FreeBSD) ….  Hardware based – PacketStorm 6XG, Apposite Netropy ….  Topology based network emulators (software)  Imunes – based on Docker & Open vSwitch, Gui based  Mininet - based on namespaces, OpenFLow - 11 -

Network topology emulation - Imunes example - 12 -

Mobility behavior Number of innovative applications in IoT based on location System performance can be affected by a specific movement pattern - of sensors or sensations - 13 -

Strategy for capturing sensor/sensation movement WAN Emulator Sensor/ PC Sensation movement Sensor Location 1 Device Agent Location 1 model Sensor Location 2 Device Agent Location 2 Central Virtual user Sensor Location 3 Device Agent Location 3 /device controller Centralized Services No performance testing solution available today that takes device mobility into account 1. Maintain locations of users Device Device Device Agents Agents 2. Simulate user and/or sensation motion based on predefined crtical scenarios Agents 3. Control number of virtual users in each location based on user/device mobility - 14 -

Another challenge – Big Data  True reproduction of Volume, Variety and Velocity aspects essential Velocity of data can be controlled by the virtual users and devices   Volumes - Generating large data volumes is a time-consuming effort  Use of parallelism must in data generation and loading Variety – it is not just about structured or unstructured data   In what ways the data will grow is important - 15 -

Aiding correct data generation  Large volume data generation should be a vector representation  How is my data going to grow?  Applies to structured and unstructured data  Rather than just  How many sales am I expecting in the next 5 years?  Also consider (as an example)  How many new types of products are expected to be introduced  What is the distribution of sales amongst products?  How will the sales volumes distributed across the week?  Number of sales offers in a year  Reactions to end of season sales - 16 -

A Simple Example  Consider a table with 100 collumns  T = [.….c10……c20………………………………….]  Consider query  Select sum(c10) from T where T.c20 = 1000  Consider 2 cases 1. Data is uniformly distributed on c20 2. At least 80% of the rows have c20 == 1000  The SQL query in case 2 will take much more time than in case 1  Higher the data volumes higher the difference Essential to get the distribution of data right - 17 -

Tools available today  Grid Tools  Supports exponential and normal distribution for data values  UpScene Advanced Data Generator GS Data Generator   Redgate SQL data generator  EMS data generator  Datanamic data generator for multi-db Most tools support generation of uniformly distributed data   between a min and max value  No tool supports skewed data generation - 18 -

Need for a Mature data generation tool 4. Business user answers the questions Business growth questions Test Engineer 1. Test engineer feeds the schema 2. Test engineer creates questions for business user 3. Maps the schema with the questions 5. Data generation tool generates and loads the data on the cluster (SUT) SUT cluster schema - 19 -

Newer Systems - Data Sources and sinks Sensor Data Management End users Device Apps Management LWM2M Message Routing Device & Event Processing Agents http(s), tcp, udp, mqtt Analytics OPC-UA, Modbus, Continua Apps, Clients & Portals Things with Gateway Devices Centralized Services Embedded Sensors Mobile Devices Examples • Sensor to record in a database • Sensor information transformed to recommendation in real time • Sensor data transforming into an control • Measurement technique not as common as response time - 20 -

Measurement Challenges  It may not be just about application response times any more.  Need for one way latency measurement tools –  may be specific to the application  A basic framework that can be easily configured will help.  Correlate a unit transaction across logs - all the way from source to sink  Application support may be required Technologies promising “ eventual consistency ” could complicate real-time needs   Percentage of passed transactions will be important to report across incoming traffic rates  Need for a way for the business to specify what constitutes acceptable performance  There has to be a way to evaluate cost impacts  based on various latency distributions No known tool support at present - 21 -