Practical Techniques for Verification and Validation of Robots - PowerPoint PPT Presentation

Practical Techniques for Verification and Validation of Robots Kerstin Eder with a demo by Dejanira Araiza Illan University of Bristol and Bristol Robotics Laboratory

Would you swallow a robot?

The Safety Challenge § Autonomous Systems § Engineering Challenge – Advances in control science – Focus on “making things work” 3

4 Pictures from www.wikipedia.org

The Safety Challenge § Autonomous Systems § Engineering Challenge – Advances in control science – Focus on “making things work” § Fundamental concern: – Can such systems be trusted? 5

6

Dependability § A system is dependable (or trustworthy) only if it can be shown to be safe and useful. – Safety is the property of avoiding harmful conditions. – Liveness requires that the system achieves its goals a.k.a. usefulness. § Demonstrable safety and liveness are required. 7

Safety Assurance Assurance is the essential concept: § A system may never cause harm throughout its entire operating life, § but if we cannot be assured of that before we start to use it, then the system can not be trusted. 8

Designing Dependable Systems § Create flawless designs. AND § Design the system in such a way that the flawlessness can be demonstrated. "Waterfall" by M.C Escher.

EPSRC “Principles of Robotics” “Robots are products. They should be designed using processes which assure their safety and security.” http://www.epsrc.ac.uk/ourportfolio/themes/engineering/activities/Pages/principlesofrobotics.aspx 10

Verification and Validation for Safety in Robots To develop techniques and methodologies that can be used to design autonomous intelligent systems that are demonstrably trustworthy. 11

Correctness from specification to implementation User Requirements High-level Specification Translate Optimizer Design and Analysis (Simulink) Implement Controller (SW/HW) e.g. C, C++, RTL (VHDL/Verilog) 12

What can be done at the code level? P. Trojanek and K. Eder. Verification and testing of mobile robot navigation algorithms: A case study in SPARK. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 1489-1494. Sep 2014. http://dx.doi.org/10.1109/IROS.2014.6942753 13

What can go wrong in robot navigation software? Generic bugs: § Array and vector out-of-bounds accesses § Null pointer dereferencing § Accesses to uninitialized data Domain-specific bugs: § Integer and floating-point arithmetic errors § Mathematic functions domain errors § Dynamic memory allocation errors § Concurrency bugs § blocking inter-thread communication (non real-time) 14

State of the art verification approaches § Model checking § infeasible for real (off-the-shelf) code § Static analysis of C++ § not possible § Static analysis of C § requires verbose and difficult to maintain annotations 15

HW Design Reconvergence Model Fabrication Silicon HW Design Specification Chip Netlist 16

HW Design Reconvergence Model Verification Test Fabrication Silicon HW Design Specification Chip Netlist Tape out § Design Verification is the process used to gain confidence in the correctness of a design w.r.t. the requirements and specification. § Test: Manufacturing Test vs Functional Test 18

Design-for-Test(ability) 19 http://anysilicon.com/overview-and-dynamics-of-scan-testing/

Design-for-Verification Approach § SPARK, a verifiable subset of Ada § software reliability a primary goal § no memory allocation, pointers, concurrency § side-effect free functions § SPARK specification and tools freely available for academic use § Required code modifications: § Pre- and post-conditions, loop (in)variants § Numeric subtypes (e.g. positive) § Formal data containers 20

Navigation in SPARK § Three open-source implementations of navigation algorithms translated from C/C++ (2.7 kSLOC) to SPARK (3.5 kSLOC) - Vector Field Histogram - Nearness Diagram - Smooth Nearness-Diagram § Explicit annotations are less than 5% of the code SPARK code is on average 30% longer than C/C++ § 21

Verification Conditions 22

Formal Verification Results Number of discharged verification conditions and the running time of static analysis 23

Results § Several bugs discovered by run-time checks injected by the Ada compiler - Fixed code proved to be run-time safe - except floating-point over- and underflows - These require the use of complementary techniques, e.g. abstract interpretation. Up to 97% of the verification conditions discharged § automatically by SMT solvers in less than 10 minutes Performance of the SPARK and C/C++ code similar § Moral: If you want to make runtime errors an issue of the past, then select your tools (programming language and dev env) wisely! 24

Moral If you want to make runtime errors an issue of the past, then you must select your tools (programming language and dev env) wisely! 25

http://github.com/riveras/spark-navigation P. Trojanek and K. Eder. Verification and testing of mobile robot navigation algorithms: A case study in SPARK. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 1489-1494. Sep 2014. http://dx.doi.org/10.1109/IROS.2014.6942753 26

Correctness from Specification to Implementation User Requirements Verification High-level Specification Translate Optimizer Design and Analysis (Simulink) Implement Controller (SW/HW) e.g. C, C++, RTL (VHDL/Verilog) 27

What can be done at the design level? D. Araiza Illan, K. Eder, A. Richards. Formal Verification of Control Systems’ Properties with Theorem Proving. International Conference on Control (CONTROL), pp. 244 – 249. IEEE, Jul 2014. http://dx.doi.org/10.1109/CONTROL.2014.6915147 D. Araiza Illan, K. Eder, A. Richards. Verification of Control Systems Implemented in Simulink with Assertion Checks and Theorem Proving: A Case Study . European Control Conference (ECC), pp. tbc. Jul 2015. http://arxiv.org/abs/1505.05699 28

What is an assertion? § An assertion is a statement that a particular property is required to be true. – A property is a Boolean-valued expression § Assertions can be checked either during simulation or using a formal property checker. § Assertions have been used in SW design for a long time. – assert() function is part of C #include <assert.h> – Used to detect NULL pointers, out-of-range data, ensure loop invariants, etc. § Revolution through Foster & Bening’s OVL for Verilog . – Clever way of encoding re-usable assertion library in Verilog. J – > 30 checker types (assertion templates) – http://accellera.org/activities/working-groups/ovl

30

31