Smart Testing with AI Using Data Mining Presented by: - - PDF document

smart testing with ai using data mining
SMART_READER_LITE
LIVE PREVIEW

Smart Testing with AI Using Data Mining Presented by: - - PDF document

T16 Analytics in Testing Thursday, October 3rd, 2019 1:30 PM Smart Testing with AI Using Data Mining Presented by: Lorna


slide-1
SLIDE 1

¡ ¡ T16 ¡

Analytics ¡in ¡Testing ¡ Thursday, ¡October ¡3rd, ¡2019 ¡1:30 ¡PM ¡ ¡ ¡ ¡ ¡

Smart ¡Testing ¡with ¡AI ¡Using ¡Data ¡ Mining ¡ ¡

Presented ¡by: ¡ ¡ ¡

¡ Lorna ¡Smyth ¡

¡ Smartbear ¡ ¡

Brought ¡to ¡you ¡by: ¡ ¡ ¡ ¡

¡

¡

¡ ¡

888-­‑-­‑-­‑268-­‑-­‑-­‑8770 ¡·√·√ ¡904-­‑-­‑-­‑278-­‑-­‑-­‑0524 ¡-­‑ ¡info@techwell.com ¡-­‑ ¡http://www.starwest.techwell.com/ ¡ ¡ ¡

¡

¡ ¡ ¡

slide-2
SLIDE 2

¡

Lorna ¡Smyth ¡

¡ Lorna ¡Smyth ¡is ¡from ¡Galway ¡on ¡the ¡west ¡coast ¡of ¡Ireland. ¡She ¡is ¡currently ¡a ¡technical ¡ sales ¡engineer ¡for ¡SmartBear's ¡test ¡automation ¡tools. ¡She ¡has ¡worked ¡in ¡various ¡IT ¡ roles ¡over ¡the ¡past ¡eight ¡years, ¡including ¡technical ¡analyst, ¡data ¡migration ¡test ¡lead, ¡ and ¡software ¡licensing ¡specialist. ¡She ¡specializes ¡in ¡test ¡automation ¡and ¡delivers ¡ webinars, ¡technical ¡demonstrations, ¡and ¡deep-­‑dive ¡troubleshooting ¡calls ¡on ¡test ¡ automation ¡for ¡SmartBear’s ¡customers. ¡Lorna ¡is ¡an ¡advocate ¡for ¡women ¡in ¡IT ¡and ¡is ¡ heavily ¡involved ¡in ¡hosting ¡local ¡networking ¡events. ¡She ¡also ¡participates ¡in ¡ community ¡outreach ¡to ¡drive ¡awareness ¡of ¡IT ¡roles ¡in ¡the ¡West ¡of ¡Ireland. ¡Lorna ¡is ¡ an ¡adventure ¡and ¡sports ¡enthusiast, ¡which ¡has ¡led ¡her ¡to ¡living ¡in ¡Canada, ¡Norway, ¡ and ¡Austria, ¡following ¡her ¡love ¡of ¡the ¡mountains. ¡Lorna ¡has ¡recently ¡returned ¡to ¡ Ireland ¡and ¡is ¡loving ¡life ¡working ¡in ¡test ¡automation. ¡ ¡

slide-3
SLIDE 3

8/19/2019 1

Lorna Smyth Solutions Engineer for Test Automation tools

slide-4
SLIDE 4

8/19/2019 2 ✓ What is Artificial Intelligence(AI)? ✓ How is AI transforming the world? ✓ What is Data Mining? ✓ How can AI address Software Testing Challenges? ✓ Benefits and Challenges of AI in Testing ✓ How can you Embrace AI?

What is AI?

“Your biggest fear?” “ Intelligence demonstrated by machines, in contrast to the natural intelligence displayed by humans.” “Computers that mimic cognitive functions that humans associate with the human mind, such as learning and problem solving” “Emphasizes the creation of intelligent machines that work and react like humans” “a conglomeration of concepts and technologies that mean different things to different people” “Something that is going to change the world…”

slide-5
SLIDE 5

8/19/2019 3

What is AI?

Neural networks Machine learning Deep learning

the networks of hardware and software that approximate the web of neurons in the human brain. a technique using algorithms to teach machines to learn. helps machines learn to go deeper into data to recognize patterns.

AI is the broad category of methodologies that teach a computer to perform tasks as an “intelligent” person would.

Data Mining

How is AI transforming the world?

slide-6
SLIDE 6

8/19/2019 4

Impact of AI

  • AI will impact global GDP by $15.7 trillion by 2030. (Source: PwC)
  • 72% of Business leaders termed AI as a business advantage. (Source :PwC)
  • 77% of devices we use feature one form of AI or another. (Source: PEGA)
  • Machine learning engineers usually earn up to $115k per annum.
  • By 2020, it’s predicted robots will be able to flirt and make jokes.

(Source: Google)

  • Automated bots could automate up to 38% of the jobs in the US, 30% of

the jobs in the UK, 21% in Japan, and 35% in Germany. (Source: PwC)

AI is transforming all industries, not just software testing…

slide-7
SLIDE 7

8/19/2019 5

…personalizing content, making our lives easier through email responses.. …chat bots and targeted ads….

slide-8
SLIDE 8

8/19/2019 6

Ever wondered how the Wimbledon editorial team focus their content?

IBM Watson’s AI

slide-9
SLIDE 9

8/19/2019 7

HEALTHCARE

Virtual Nursing Assistants Robot Assisted Surgery Administrative Workflow Fraud Detection Dosage Error Reduction Connected Machines Preliminary Diagnosis Image Based Diagnosis Clinical Trial Participation Patient Engagement Bumrungrad International Hospital, Thailand: 83% concordance

83%

211 patients with breast, colorectal, gastric and lung cancer

Manipal Comprehensive Hospital, India: 73% concordance

638 patients with breast cancer

Gachon University, Gil Medical Center, South Korea: 49% concordance

656 patients with colon cancer

73% 49%

slide-10
SLIDE 10

8/19/2019 8

FINTECH

  • Digital Financial Coach
  • Client Risk Profile – automate categorization of clients High to

Low; recommend relevant financial products.

  • Fraud Detection – using algorithms to identify fraudlent claims.
  • Contract Analyser – Contract analysis is a repetitive internal task

which can be delegated to a machine.

  • JP Morgan saved 360,000 hours over a year from its employees’ load in only a few

seconds.

Impact of AI in Software Testing?

  • Test Case Design
  • Test Management
  • Test Execution
  • Test Result Analysis
slide-11
SLIDE 11

8/19/2019 9

What is Data Mining?

Image source: AIAnalyticsHub

slide-12
SLIDE 12

8/19/2019 10

Data Mining Techniques:

Classification Clustering Regression Outer Sequential Patterns Association Rules Prediction

Classification: Retrieve information and classify into different classes. Clustering: Identify data that is like each other. Regression: Identify and analyze relationship between variables.

Data Mining Techniques:

  • E.g. Analyse loan applicant data, class as ‘risky’ or ‘safe’.
  • E.g. Analyse test data/conditions, If ‘test’ run on Windows X, then

class as ‘likely to fail’.

  • E.g. Segregate applicants with similar incomes.
  • E.g. Segregate defects with similar causes of failure (e.g. Performance,

Functional, System).

  • E.g. Credit Policy, Loan Purpose, Income, Credit Score, Spend History.
  • E.g. Operating System, Browser, Environment, Resolution.
slide-13
SLIDE 13

8/19/2019 11

Outer Detection: Identify items that do not match an expected pattern/behavior.

Data Mining Techniques:

Association Rules: Identify associations between two or more items – discovers hidden patterns..

  • E.g. A single, female aged between 35-45 with account of over $5000 is

likely to need a loan between $10,000-$15,000.

  • E.g. Performance tests that are ran on Mac OS X are 20% likely to fail
  • E.g. Unexpected Spend/Transaction on a bank statement.
  • E.g. Defect/Error types that are unexpected or haven’t been found

before. Prediction: combination of data mining techniques, analyzes past events or instances in a sequence for predicting a future event. Sequential Patterns: Identify patterns in transaction data for a certain time period.

Data Mining Techniques:

  • E.g. Identifying a pattern of spending at certain times of the year.
  • E.g. Identifying patterns of performance issues during holiday season.
  • E.g. Predict an individuals overall creditworthiness.
  • E.g. Predictive defect finding based on past defects found.
slide-14
SLIDE 14

8/19/2019 12

Understand Business Problem

Data Mining Process

Verify & Deploy Evaluate, Train & Test Model Model Data Understand Data Prepare & Process Data

What do you mean by modelling the data?

slide-15
SLIDE 15

8/19/2019 13

Modeling Data: Neural Network example Let’s train an AI….

slide-16
SLIDE 16

8/19/2019 14

AI and Software Testing

We Want Faster Release Cycles

Automation Capability High Smart Bulky Automation Tools More time for Business Goals Speed Quality Cost Culture Innovation SDLC Time Taken Weeks Hours Low Robust Automation Tools Continuous Testing Autonomous Testing Manual Testing Waterfall Early Agile Agile Devops CI/CD

slide-17
SLIDE 17

8/19/2019 15

Why now?

  • Many useful machine learning systems are relatively easy to build.
  • The availability of, and ability to store and process large amounts of data,

make training machine learning systems easier and more accurate.

  • Software testing industry can now see the potential of adopting AI in

testing. ➢ Testing is a good use case as it’s built on a logical foundation ➢ Companies are struggling to keep up with user expectations.

Current Automated UI testing challenges

  • Unreliable Object Recognition
  • Test Framework Design
  • Inadequate Prioritization
  • Test refactoring
  • Test scalability
  • Inadequate Documentation

TEST DESIGN TEST MAINTENANCE AND EXECUTION

slide-18
SLIDE 18

8/19/2019 16

INTELLIGENT TEST DESIGN

Advanced Object Recognition Framework Generation Responsive Web Design Risk Profiling

What does Intelligent Test Design Look Like?

Risk Profiling Prioritize tests based on business risk by environment, configuration, and different aspects of your application Object Recognition Automatically recognize new objects and updates to add them to the DOM and structure without manual effort Framework Generation Automatically scan your application to recommend a test framework

slide-19
SLIDE 19

8/19/2019 17

Neural Network for Testing

Each application update and a new test case can act as inputs to help maintain an application.

Example Outputs:

  • Recommendation

systems

  • Reinforcement

learning

  • Device selection

Example Inputs:

  • Application

controls

  • Properties of

controls

  • New test cases
  • New test results

INTELLIGENT TEST MAINTENANCE & EXECUTION

Predictive Self- Healing Intelligent Bug Hunting Application Resilience Process Automation

slide-20
SLIDE 20

8/19/2019 18 Predictive Self-healing Dynamically update your test suite when your application changes or evolves Application Resilience Netflix Simian Army (Chaos Monkeys) Achieving resilience by applying predictive auto-scaling and continuous fitness functions Intelligent Bug Hunting Discover bugs in the application through AI- powered exploratory testing Process Automation Automate business workflows for end-to-end testing

  • Analyse and categorize log files to save time and enhance quality

quicker than before.

  • Automatically identify if a change in code is new functionality or a

defect of a new release.

  • Forecast client requirements.
  • Reduce the probability of ignored bugs.

Source: Oleksii Kharkovyna

slide-21
SLIDE 21

8/19/2019 19

HOW COMPANIES ARE USING AI IN SOFTWARE TESTING TODAY

Meaningful tests require varied and large test data sets.

GOOGLE

Google DeepMind created an AI program that utilizes deep reinforcement learning to play video games by itself, thus, producing a huge amount of test data that they can learn from.

slide-22
SLIDE 22

8/19/2019 20 Testim uses AI to speed up the authoring, execution, and maintenance of automated tests.

  • It emphasizes on functional, end to end testing and user interface testing.
  • The tool becomes smarter with more runs and increases the stability of test suites.

TESTIM

  • Upload your app and Appdiff’s

bots will start testing the application. ➢Finds bugs, crashes and error dialogs ➢Builds reports on this data.

  • If an element or object changes,

the tool will adjust and identify it without any user intervention.

APPDIFF

slide-23
SLIDE 23

8/19/2019 21

EGGPLANT AI

  • Bug hunting: looks for common patterns that typically lead to bugs(e.g.

going backwards and forwards in between screens).

  • Coverage analysis: graphical analysis of coverage in terms of states,

actions, and data.

  • Real user journeys: prioritizes paths within the application that have

been executed by actual users.

  • User weights: actions can be extended with weights, which increase
  • r decrease the probability of a particular of being chosen.

Uses Computer Vision algorithms to perform automated visual testing.

  • Detect Layout issues, report issues perceptible to users

APPLITOOLS

slide-24
SLIDE 24

8/19/2019 22

APPIUM

Appium Classifier Plugin: Allows you to automatically find an icon that’s visible in the application under test. Pre-requisite is that the icon must look like one of the pre-trained labels like “cart”, “edit”, and “twitter”. Appvance makes use of AI to generate test cases based on user behavior.

APPVANCE

slide-25
SLIDE 25

8/19/2019 23

REPORT PORTAL

AnalyseTest Results Automatically

  • Triage your automation test results
  • Create fancy graphs to share with

everyone Algorithms are using the historical data in the dashboard database

  • Automatically sort defects into categories based
  • n previous results
  • Report based on those categories
  • Predict possible failures
  • Expanded test coverage with object recognition with AI: Test complex and

hard-to-find application components such as charts, mainframes, PDFs and SAP applications.

SMARTBEAR

In SmartBear’s automated functional testing tool, they’re leveraging AI to improve

  • bject recognition and test maintenance.
slide-26
SLIDE 26

8/19/2019 24

  • Smarter test maintenance with an intelligent recommendation system:

Notify you of any unused objects and changes made to your application.

  • Captcha Recognition

SMARTBEAR

Object Recognition Use Case

slide-27
SLIDE 27

8/19/2019 25

Benefits & Challenges

✓ Quicker releases

Benefits of adopting AI in your Software Testing

✓ Improved quality – less human error, increased test coverage ✓ Eliminates flaky tests – can automatically update ✓ Find issues quickly – less impact to customers ✓ Accelerate results analysis – make smart decisions quicker ✓ Increased traceability – defects and test case coverage

slide-28
SLIDE 28

8/19/2019 26

Challenges of adopting AI in your Software Testing

  • Resource expertise
  • Cost of resources and time
  • Choosing the right problem to solve
  • Data: having access, having meaningful and clean data,

having the right data infrastructure

  • Integrating with your current testing process
  • Measuring the success

How to get started?

  • Start small.
  • Involve the right resources – e.g. you may need to consider a data

scientist or mathematician.

  • Decide which tests are most most suitable to address with AI.
  • Source, organize and cleanse your data.
  • Define measures of success – if you want to get buy-in based on the

success of your first project, that success needs to be measurable.

  • Decide whether you want to develop your own algorithms, leverage

existing solutions and tools, or a mix?

slide-29
SLIDE 29

8/19/2019 27

Without AI: a lot of time spent on…

Failures Refactor Tests Object Recognition Defects Found Repetition Stability Bottle Necks Decision Time Resource Allocation

With AI: gives us more time to focus on…

Creativity Scalability Customer Satisfaction Test Coverage Business Impact Results

slide-30
SLIDE 30

8/19/2019 28

It’s here, embrace it – AI is changing our world, are you willing to change with it? Resources:

  • Online papers, blogs, articles
  • Podcasts
  • Meet-ups
  • Education Courses
  • Online communities
  • Books/E-Books

AI may be executing our tests in the future, but testers and test automation engineers will still be needed as the subject matter experts to teach the AI.

slide-31
SLIDE 31

8/19/2019 29

slide-32
SLIDE 32

8/19/2019 30

Thank You!

@DigiLorna