What's Our Job When the Machines Do Testing? Presented - PDF document

¡ ¡ W2 ¡ AI ¡and ¡Data ¡Analytics ¡ Wednesday, ¡October ¡17th, ¡2018 ¡10:15 ¡AM ¡ ¡ ¡ ¡ ¡ ¡ ¡ What's ¡Our ¡Job ¡When ¡the ¡Machines ¡Do ¡ Testing? ¡ ¡ Presented ¡by: ¡ ¡ ¡ Geoff ¡Meyer ¡ ¡ ¡ ¡ ¡ Brought ¡to ¡you ¡by: ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ 350 ¡Corporate ¡Way, ¡Suite ¡400, ¡Orange ¡Park, ¡FL ¡32073 ¡ ¡ 888 -­‑-­‑-­‑ 268 -­‑-­‑-­‑ 8770 ¡ ·√·√ ¡904 -­‑-­‑-­‑ 278 -­‑-­‑-­‑ 0524 ¡-­‑ ¡info@techwell.com ¡-­‑ ¡http://www.starwest.techwell.com/ ¡ ¡ ¡ ¡ ¡ ¡ ¡

¡ ¡ ¡ ¡ Geoff ¡Meyer ¡ ¡ A ¡test ¡architect ¡in ¡the ¡Dell ¡EMC ¡infrastructure ¡solutions ¡group, ¡Geoff ¡Meyer ¡has ¡ more ¡than ¡thirty ¡years ¡of ¡industry ¡experience ¡as ¡a ¡software ¡developer, ¡manager, ¡ program ¡manager, ¡and ¡director. ¡Geoff ¡oversees ¡the ¡test ¡strategy ¡and ¡architecture ¡for ¡ more ¡than ¡four ¡hundred ¡software ¡and ¡hardware ¡testers ¡across ¡India, ¡Taiwan, ¡and ¡ the ¡United ¡States. ¡He ¡leads ¡initiatives ¡in ¡continuous ¡testing, ¡predictive ¡analytics, ¡and ¡ infrastructure ¡as ¡a ¡service ¡(IaaS). ¡Outside ¡of ¡work, ¡Geoff ¡is ¡a ¡member ¡of ¡the ¡Agile ¡ Austin ¡community, ¡contributor ¡to ¡the ¡Agile ¡and ¡STAR ¡conferences, ¡and ¡an ¡active ¡ mentor ¡to ¡veterans ¡participating ¡in ¡the ¡Vets4Quality.org ¡program, ¡which ¡provides ¡ an ¡on-­‑ramp ¡to ¡a ¡career ¡in ¡software ¡quality ¡assurance. ¡ ¡

What’s Our Job When The 17 Oct 2018 Geoff Meyer, Test Architect Machines Do Testing? geoff_meyer@dell.com

Navigating the Age of the Machine Machine Partnerships in Test A Journey in the age of Smart Assistants What’s our job?

Beware!

Abundance Number of People living in Extreme Poverty

Budding Effect

Convenience

Industrial Revolutions are not new 1 st Industrial Revolution 2 nd IR 3 rd IR 4 th IR 2000 2018 1800 1900

Analytics and AI in Business Human Resources Legal Discovery Global Audit Global Services & Support Financial Services Sales & Pricing Marketing Analytics Operations …Identify what doesn’t work well in a process, service or product and make it go away ~ Malcolm Frank, Cognizant Future of Work

But it’s OK Almost 90%! Enhanced Invented Replaced 75% 12% 13% "Think about a job as the sum of it’s tasks.” ~ What to do when Machines do everything Cognizant Center for the Future of Work

Context at DellEMC Servers 465 Trillion Test Configurations!! Server Configuration Elements Chassis Processor Memory DIMM Memory Configuration Hard Disk Drive (HDD) Non-Volatile Memory (NVM) Embedded Systems Management Power Management BIOS Power Supply Bezel Network Daughter Card RAID Controller Network Interface Card (NIC) Host Bus Adapter (HBA) Additional PCIe Cards Cooling

DellEMC Server: Evolution of Testing Waterfall Agile Scripted Automation Shift-left testing DevOps Automated UI Testing Axon Workflow Automation Cognitive Automation Continuous 1997 Deployment Integration Non-Stop Test Ops Configuration Portability Continuous Test Assistant (Zeno) Reactor Reuse Marketplace 2010 Test Suite Assistant Continuous HW-Aware Testing AI-Assisted UI Deployment Priority-driven testing Automation 2013 SDETs Continuous Monitoring Lab-as-a-Service TMMi 2015 2017 2019

The Analytics Continuum How can we Self-Learning make it What will Happen? Happen? Artificial Intelligence Why did it Prescriptive Business Value Happen? Predictive What Diagnostic Happened? Descriptive Report Correlate Predict Recommend Autonomous Source: Gartner Data Analytics Sophistication

Data Analytics Modeling Data Data Sources Cleanse Insights, Predictions, Recommendations Domain Analytics Knowledge Engine (i.e. Rules) Feedback

What goes into the Model? “Risk -based Testing” Rules and heuristics “Prior Failed “Only deploy Test Cases” BVT- verified builds” Best- practices Rules of Thumb Rules Positive/Negative Patterns Tribal Knowledge Common “Related historical Sense “Test the defect test failures” that got fixed”

It’s all about the data “Nobody really goes out of their way to point out the importance of data…” ~ Brian Sletten, Bosatsu Consulting

What if we had a Smart Assistant?

What is the The Smart Assistant release risk given the testing that’s been completed? What are the high-value SUT What test scripts configurations? should be retired rather than be re-factored? What’s the optimal What tests can detect coverage for this the maximum number of defects build/test cycle? What automated test given the changes in the failures appear to be current build duplicates?

Data Sources - Product Engineering

Selecting our Technology Partners

Operationalizing • Standardize your Data Sources • Develop Connectors to extract data • Design/Develop your data lake • Workflow Automation • Visualization Dashboards

SUT Configuration Model Team “Q” - System Under Test Current Testing Scenario Challenges ➢ 465 Trillion possible server configs!!! ➢ Which are the High Value Configs? ➢ How to ensure Optimal Configs Coverage? Data Sources Objectives ➢ Quickly predict “best - available” SUT configurations during planning and test execution ➢ Ensure Optimal Configs Coverage ➢ Prioritizes High-Value Configs Q Technology Partner: Dell Performance Analytics Group

Test Planning Model “JARVIS” Objectives • Use historical test data and defects as predictors and to expose patterns • Automate deep-think testing tasks • Codify Subject-Matter Expertise • Real-time access to active Increased Test repositories Capacity Reduced test Which manual tests are most effective, and should be automated? cycle time How can I accelerate discovery of break/fix? Re-allocate to Which of my test cases appear to be obsolete? Exploratory Testing Am I over-testing or under-testing? Fast Find of Automation Break/Fix candidates Technology Partner: De-prioritization candidates

Testbots are here AI-assisted UI Automation • Increases UI test coverage • At substantially less cost of creation and maintenance

Duplicate Defect Prediction DellEMC XtremIO

What’s our job?

What can Humans do Better? • Creative, Collaborative, Problem Solving • Contextualized Intelligence • Empathy • Storytelling

Key Skills/Attributes of a Tester Janet Gregory

Testing ~ Value Creation Testing vs. Checking • Establishes Expected Behavior • Collaborative • Curious Sprints • New Feature Exploratory 1 2 3 4 5 6 Development/Test • Cognitive • Analyze Potential Risks Legacy test • Requires Thinking Legacy Regression Checking automation maintenance Checking ~ Value Sustainment 1 1 1 1 1 • Confirms Expected Behavior New feature test 2 2 2 2 • Robotic automation development New Feature Regression Checking • Tedious & maintenance 3 3 3 • Scripted 4 4 • Vigilance for deviations 5 • Monitor Known Risks “No one confuses spellcheckers with editing, so why confuse automated checks as testing?” • Requires Processing ~ Michael Bolton

Envisioning the Future of Testing Dependencies Requirements Product roadmap Technologies Technical Debt Value Creation Collaborative Value Resources Sustainment

Envisioning the Future of Testing Automation Dependencies Requirements • Unit tests Product roadmap Technologies • Code Complexity Technical Debt • Build Verification Testing • Regression Testing • Non-Functional Testing • Value Simulation/Emulation • DevOps Creation • Continuous Integration • Environment Provisioning • Continuous Deployment • Continuous Testing • Cognitive Tasks (AI/ML) Collaborative • Process Orchestration Value Resources • Non-Step Test Operations Sustainment • Continuous Monitoring

What about Data Science skills? Creative Commons

What are our roles? Test Engineer Test Strategy Strategy Predictive Modeling Requirements Elaboration Test Case Analysis Planning Test Case Development Test Suite Planning AI Trainer Test Environment Mgmt Test Configuration Planning Provisioning SW/FW Provisioning(DevOps) Test Data Mgmt Test Case Execution Test Automation Development Exploratory Testing Dev/Testing AI Maker Data Connectors Data Scientist SDET Test Failure Triage Tracking Defect Tracking