Verification and Validation of Knowledge-Based Systems Prepared by - PowerPoint PPT Presentation

Verification and Validation of Knowledge-Based Systems Prepared by Dr Ahmed Rafea

Survey on KBS Development

V&V Definition

WHY IS V&V OF KS DIFFICULT? • Knowledge Representation Formalisms • Evolving and Large KBs • Characteristics of KBs • Domain Experts in V&V • KS Anomalies – For instance, a rule based KB may have potential inconsistencies, incompleteness, circularity or redundancies among the rules in the KB. • Testing of KS – Testing Criteria – Difficulties in Generation of Test Case Inputs – Difficulties in Generation of Test Case Outputs – Input and Output Spaces for Selection of TestCases Can Be Huge – High Costs of Testing

V&V TECHNIQUES • Broadly speaking, those techniques can be categorized into two groups: – static methods (analysis ) – dynamic methods (testing )

Static Methods • Static V&V methods can have different objectives such as detecting completeness and consistency faults and proving the correctness of the programs. Techniques in this category are: – informal (reading/reviews, inspections, and walkthroughs) – semiformal checks such as type-checking performed by compilers – formal techniques (axiomatic mathematical proofs). • Some recent attempts include the informal analysis such as the use of checklist approach, the formal analysis such as the assertional approach and the object-oriented specification approach.

Dynamic Methods • Dynamic V&V methods require the execution of a system through the use of test suites. • Test cases can be derived either from a functional or structural viewpoint • In the functional testing, also known as the black-box testing , a program is treated as a black box. • The structural testing, also known as the white- box testing , constructs the test cases based on the implementation details.

A Structured Testing Methodology • Verification – Domain Knowledge Verification – Inference Layer Verification • Validation – Test Case Generation Methods – Judging system acceptability – Regression Testing

Domain Knowledge Verification(1) • The domain knowledge verification process detects most of the coded KB errors. The verification process can be divided into three phases: – Consistency checking phase. – Completeness checking phase. – Path checking phase.

Domain Knowledge Verification(2)

Consistency checking • The consistency checker works on the relations between expressions of the domain layer , one relation at a time. • Consistency of the KB appears as: – undefined object , – undefined attribute, – undefined attribute values, – duplicate rule pairs, – conflict rule pairs, and – subsumed rule pairs.

Creating the relation between expressions table • The relation between expressions table contains the needed information about all the relations between expressions in the KB. • The basic idea behind constructing this table is to accelerate searching for any defined at tribute in the KB which is heavily used in subsequent phases • This table consists of the following fields: • Relation name : The name of the relation between expressions as defined in the KB. • Input attribute : The names of object-attribute pairs given in the rules antecedences. • Output attribute : The names of object-attribute pairs given in the rules consequences.

Completeness checking • As the number of rules grows large, i t becomes impossible to check every possible path through the system. There are four indicative situations of gaps in the knowledge base: – unused attribute values, – missing rules, – Unfireable rules, and – unused consequence.

Path checking • The Path Checking consists of: – Detecting circular • Circular paths are detected when an attribute appears as an input attribute of one relation and as an output attribute of another relation and a path between the other edges of these relations can be reached. – Redundant paths. • A redundant path is found when it is possible to reach the same conclusion from the same inputs through different paths • These paths will be detected from a graph data structure. This graph links the input attributes to the output attributes using the relation between expressions

Inference Layer Verification

Step checker phase • The main functions of the step checker is: – detecting inference step consistency error • Inconsistency arises when the input- or output-role refers to data element that is not defined in any relation between expressions of this inference step.

Inference checker phase • The inference checker works on the input/output roles of the inference layer . • Since each inference has a defined input- role and output-role, each output role should either be an input-role to the following inference step or the last output.

Task Layer Verification • The task applies the inference steps defined in the inference layer • The task dynamic input role must be consistent with the input role of the first inference step or sub-task applied. • The task dynamic output role must be consistent with the last inference step or sub-task applied.

Validation • There are three main concerns in the proposed validation process. – First, determine the appropriate method to automatically generate test cases for each KB component. – Second, human experts should judge the KB’s solution. – Third, when a KB component is modified it should be re-tested (regression testing).

Test Case Generation Methods • We have to define two types of tests for each KB component, automatic test, developer test. • Automatic test : Test cases are automatically generated in the form of input concept-attributes pair and their suggested values. – The tester provides both the knowledge engineer and the domain expert by a list of test cases. This list of test cases serves two functions. • First the knowledge engineer compares them with the requirements specification to check consistency between the test cases and the specification (They serve the verification activity). • Second, the domain expert finds out whether they are valid or not • Developer test : By this test the developer will be able to randomly test his system. He can run different KB components independently. – A screen, holding the input concept-attributes pairs used in the KB components, is automatically generated and displayed. – This screen contains the possible legal values (in the case of nominal attribute) or its boundary (in the case of numerical attribute) for each attribute. – Thus, the developer can supply any combination of values, run the KB component, and observer the result. – The existence of domain expert will enrich this test since he could apply different combination according to his expertise.

Judging system acceptability(1) • It is always difficult to define a standard against which to judge the acceptability of the system. there is no such standard; instead, a so-called agreement method must be employed, where the performance of the system is compared with that of other performers • Agreement should be measured using some principled approach. The most well-known agreement method is based on Turing's famous test for intelligence. • Test cases are given to human experts who are asked to determine the outcome. If expert agree on the outcome (85% for example) of test cases the tested component is accepted. This will continued until a complete system is provided.

Judging system acceptability(2) • A second step is to let a group of human experts examine anonymously the outputs obtained from the system and the outputs obtained by other humans for the same cases. If the third-party experts cannot distinguish between the problem-solving abilities of the system and the other humans, then the system is deemed to be acceptable. • Third, the system should be monitored in the real environment; for 3 months and feedback on system effectiveness and user interface are to be assessed. • The system is then evaluated in a classroom setting consists of the domain expert and end users. The expert assesses the correctness of the KB, the quality of the explanation, and the quality of the answers. The user assesses his/her ability to interface with the system, the timelines of the response, the reasonableness of the output and explanation, and how the system fits in with the operating environment.

Regression testing • Regression testing in its most basic form is simply testing done to determine whether a product has regressed to a less functional state than in the previous build • It is a common practice to evaluate KBS acceptance every six or twelve months to keep track of whether its competence and accuracy remain high • In the use of restrict regression, that all previous test cases be reapplied. • This is of course an expensive overhead to impose on a testing scheme. We adopt this by having a database that contains old test cases that have been applied to each KB components. • When a change applied to any component it is re-tested and comparison between the old and the new cases is taken place. • The result is to be fed back to the developer to record the effect of modification.