S V V .lu software verification & validation Achieving - PowerPoint PPT Presentation

S V V .lu software verification & validation Achieving Scalability in Software Testing with Machine Learning and Metaheuristic Search Lionel Briand

Definition of Software Testing • ISTQB: “Software testing is a process of executing a program or application with the intent of finding the software bugs. It can also be stated as the process of validating and verifying that a software program or application or product meets the business and technical requirements that guided its design and development.” 2

Scope • The main challenge in testing software systems is scalability • Addressing scalability entails effective automation • Lessons learned from industrial research collaborations: satellite, automotive, finance, energy … • Experiences from combining metaheuristic search, machine learning, and other AI techniques, in addressing testing scalability 3

Scalability • The extent to which a technique can be applied on large or complex artifacts (e.g., input spaces, code, models) and still provide useful, automated support with acceptable effort, CPU, and memory? 4

Collaborative Research @ SnT • Research in context • Addresses actual needs • Well-defined problem • Long-term collaborations • Our lab is the industry 5

SVV Dept. • Established in 2012, part of the SnT centre • Requirements Engineering, Security Analysis, Design Verification, Automated Testing, Runtime Monitoring • ~ 25 lab members • Partnerships with industry • ERC Advanced grant 6

Outline • Overview, problem definition • Example research projects with industry partners: • Vulnerability testing (Banking) • Testing advanced driver assistance systems • Testing controllers (automotive) • Stress testing critical task deadlines (Energy) • Reflections and lessons learned 7

Introduction 8

Software Testing SW Representation (e.g., specifications) Derive Test cases Expected Results or properties SW Code Execute Test cases Get Test Results Test Oracle Compare [Test Result!=Oracle] [Test Result==Oracle] Automation! 9

Search-Based Software Testing portion of • Express test generation problem input domain denoting required test data as a search or optimization problem randomly-generated • Search for test input data with inputs certain properties, i.e., constraints Input domain Random search may fail to fulfil low-probability • Non-linearity of software (if, loops, …): complex, discontinuous, non- linear search spaces (Baresel) Genetic Algorithm Fitness • Many search algorithms (metaheuristics), from local search to global search, e.g., Hill Climbing, Simulated Annealing and Genetic Algorithms Input domain Genetic Algorithms are global searches, sampling man 10 “Search-Based Software Testing: Past, Present and Future” Phil McMinn

Vulnerability Testing 11

X-Force Threat Intelligence Index 2017 42% Code Injection 32% Manipulated data structures 9% Collect and analyze information 4% Indicator 3% Employ probabilistic techniques More than 40% of all 3% Manipulate system resources attacks were injection 3% Subvert access control attacks (e.g., SQLi) 2% Abuse existing functionality 2% Engage in deceptive … https://www.ibm.com/security/xforce/ 12

Web Applications Client Server SQL Database 13

Web Applications Client Server SQL Database Username str1 SELECT * Name Surname … Password str2 FROM Users WHERE John Smith … (usr = ‘str1’ AND psw = ‘str2’) OK Web form SQL query Result 14

Injection Attacks Client Server SQL Database SELECT * Username Name Surname … FROM Users Aria Stark … Password ‘) OR 1=1 -- WHERE (usr = ‘’ AND John Snow … psw = ‘’) OR 1=1 -- OK … … … Web form Query result SQL query 15

Protection Layers Client Database Firewall Server SQL Data input Database Validation and Sanitization Web Application Firewall 16

Web Application Firewalls (WAFs) WAF malicious Server legitimate malicious malicious 17

WAF Rule Set Rule set of Apache ModSecurity https://github.com/SpiderLabs/ModSecurity 18

Misconfigured WAFs False Positive False Negative ALLOWED BLOCKED 19

Grammar-based Attack Generation • BNF grammar for SQLi attacks • Random strategy: randomly selected production rules are applied recursively until only terminals are left • Random strategy not efficient for bypassing attacks that are difficult to find • Machine learning? Search? • How to guide the search? How can ML help?

Anatomy of SQLi attacks Bypassing Attack Derivation Tree <START> ‘ OR“a”=“a”# <sQuoteContext> <sq> <wsp> <sqliAttack> <cmt> Attack Slices _ ‘ # <boolAttack> S = { ‘ <opOR> <boolTrueExpr> _ OR <bynaryTrue> OR”a”=“a” <dq> <ch> <dq> <opEq> <dq> <ch> <dq> # “ a ” = “ a ” 21

Learning Attack Patterns Training Set Decision Tree S 1 S 2 S 3 S 4 … S n Outcome S 2 A 1 1 1 0 0 … 0 Passed No Yes A 2 0 1 0 0 … 0 Blocked S 3 S n … … … … … … … … No Yes A m 1 1 1 1 … 1 Blocked … S 4 S 1 • Random trees No Yes • Random forest Blocked Passed 22

Learning Attack Patterns Training Set Decision Tree S 1 S 2 S 3 S 4 … S n Outcome S 2 A 1 1 1 0 0 … 0 Passed Yes No A 2 0 1 0 0 … 0 Blocked S 3 S n … … … … … … … … No Yes A m 1 1 1 1 … 1 Blocked … S 4 S 1 Attack Pattern No Yes S 2 ∧ ¬ S n ∧ S 1 Blocked Passed 23

Generating Attacks via ML and EAs Evolutionary Algorithm (EA) Machine Learning S 2 Yes No S 3 S n No Yes … S 4 S 1 No Yes Iteratively refine successful attack conditions Blocked Passed 24

Some Results Industrial WAFs Apache ModSecurity Distinct Attacks Distinct Attacks Machine Learning-driven attack generation led to more distinct, successful attacks being discovered faster 25

Related Work • Automated repair of WAFs • Automated testing targeting XML and SQL injections in web applications 26

Testing Advanced Driving Assistance Systems 27

Cyber-Physical Systems • A system of collaborating computational elements controlling physical entities 28

Advanced Driver Assistance Systems (ADAS) Automated Emergency Braking (AEB) Lane Departure Warning (LDW) Pedestrian Protection (PP) Traffic Sign Recognition (TSR) 29

Automotive Environment • Highly varied environments, e.g., road topology, weather, building and pedestrians … • Huge number of possible scenarios, e.g., determined by trajectories of pedestrians and cars • ADAS play an increasingly critical role • A challenge for testing 30

Advanced Driver Assistance Systems (ADAS) Decisions are made over time based on sensor data Environment Sensors Sensors /Camera Actuators Decision ADAS Controller 31

A General and Fundamental Shift • Increasingly so, it is easier to learn behavior from data using machine learning, rather than specify and code • Deep learning, reinforcement learning … • Example: Neural networks (deep learning) • Millions of weights learned • No explicit code, no specifications • Verification, testing? 32

CPS Development Process Software-in-the-Loop Hardware-in-the-Loop Model-in-the-Loop Stage Stage Stage Architecture modelling Functional modeling: Deployed executables on • Structure • Controllers target platform • Behavior • Plant • Traceability • Decision System engineering modeling Hardware (Sensors ...) (SysML) Analog simulators Continuous and discrete Analysis: Testing (expensive) Simulink models • Model execution and testing Model simulation and • Model-based testing testing • Traceability and change impact analysis • ... (partial) Code generation 33

Automotive Environment • Highly varied environments, e.g., road topology, weather, building and pedestrians … • Huge number of possible scenarios, e.g., determined by trajectories of pedestrians and cars • ADAS play an increasingly critical role • A challenge for testing 34

Our Goal • Developing an automated testing technique for ADAS • To help engineers efficiently and effectively explore the complex test input space of ADAS • To identify critical (failure-revealing) test scenarios • Characterization of input conditions that lead to most critical situations, e.g., safety violations 35

Automated Emergency Braking System (AEB) Decision making Vision Brake (Camera) Controller Objects’ position/speed “Brake-request” Sensor when braking is needed to avoid collisions 36 36

Example Critical Situation • “AEB properly detects a pedestrian in front of the car with a high degree of certainty and applies braking, but an accident still happens where the car hits the pedestrian with a relatively high speed” 37

Testing ADAS On-road testing Simulation-based (model) testing A simulator based on Physical/Mathematical models 38