Collective Intelligence-Based Quality Assurance: Combining - - PowerPoint PPT Presentation

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Institute of Software Technology and Interactive Systems

Collective Intelligence-Based Quality Assurance: Combining Inspection and Risk Assessment to Support Process Improvement in Multi-Disciplinary Engineering

Dietmar Winkler1,2, Jürgen Musil2, Angelika Musil2, Stefan Biffl2

1SBA Research gGmbH, Vienna, Austria 2Vienna University of Technology, Christian Doppler Laboratory “Software

Engineering Integration for Flexible Automation Systems (CDL-Flex)” cdl.ifs.tuwien.ac.at <firstname.lastname>@tuwien.ac.at

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Institute of Software Technology and Interactive Systems

Motivation & Goals

Context: § Multi-Disciplinary Engineering, such as steel mills or manufacturing facilities. § Isolated Quality Assurance Method applications (e.g., Inspection, FMEA). § Isolated and limited explicit and reusable engineering knowledge across methods and engineering phases (implicit knowledge is embodied by human experts). Goals: § Process for making implicit available knowledge usable for reuse. § Concept for combining inspection and the FMEA with collective intelligence support.

2

System Design Implementation Test / Commissioning Operation

Electrical Engineer Process Engineer

System Construction

Variable List Software Model PLC Code Process Design Electrical Plan Plant Topology Mechanical Design Electrical I/O Plant Planner Electrical Engineer Mechanical Engineer Software Engineer Electrical Engineer Process Engineer Software Engineer Plant Operation Documents Operator Test Specification Test Plan Plant PlannerMechanical Engineer QA QA QA QA QA QA QA QA

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Institute of Software Technology and Interactive Systems

Related Work

Software Reviews / Inspection § Focus on Software Engineering. § Formal and structured process approach to identify defect early and efficient in Software Engineering artefacts. § Guidelines and reading techniques, e.g., perspectives or scenarios. § Implicit Engineering Knowledge is available (human experts). § Limited tool support for inspection, e.g., Paper-Based, Gerrit Code Review or DefectRadar, but no tool support for

• rganizing / reusing engineering knowledge.

3

• 1. Inspection

Planning

• 2. (Optional)

Overview

• 4. Inspection

Meeting

• 5. Rework
• 6. Follow -Up
• 3. Individual

Preparation Moderator Moderator Moderator Inspection Team Individual Inspectors Inspection Team Author re-inspection (if required)

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Institute of Software Technology and Interactive Systems

Related Work

Failure Mode and Effect Analysis (FMEA) § Focus mainly on Systems Engineering. § Early Risk Assessment and Risk Avoidance (risk-priority-numbers and corrective actions). § Implicit Engineering Knowledge is available (human experts). § Individual Tools are available that follow the FMEA process in a specific domain. § However, very limited support of engineering knowledge across tools and disciplines.

4

• 1. FM EA

Planning

• 2. Component

and Feature identificat ion

• 3. Determine

candidat e defects

• 4. Effect Assessment:

Estimat e P, S, and D and calculate RPN

• 6. Recommend

correct ive actions

• 8. FM EA

Report

• 7. Changes

No Yes Re-Estimation based

• n modifications

5. Corrective Act ions Required?

Challenge: How to capture and reuse engineering knowledge based on available implicit expert knowledge or isolated tool data?

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Institute of Software Technology and Interactive Systems

Research Issues

Goal § Process approach to bridge the gap between methods and tools to reuse engineering knowledge: Knowledge collection, aggregation, and reuse. Research Issues § RI-1. How can a collective intelligence-based quality assurance (CI-based QA) approach support process improvement in MDE from a process perspective? § RI-2. What capabilities are required to enable CI-based QA for Inspection and the FMEA? – Process and tool capabilities. – Requirements for tool development and evaluation.

5

• 1. Inspection

Planning

• 2. (Optional)

Overview

• 4. Inspection

Meeting

• 5. Rework
• 6. Follow -Up
• 3. Individual

Preparation Moderator Moderator Moderator Inspection Team Individual Inspectors Inspection Team Author re-inspection (if required)

• 1. FM EA

Planning

• 2. Component

and Feature identificat ion

• 3. Determine

candidat e defects

• 4. Effect Assessment:

Estimat e P, S, and D and calculate RPN

• 6. Recommend

correct ive actions

• 8. FM EA

Report

• 7. Changes

No Yes Re-Estimation based

• n modifications

5. Corrective Act ions Required?

CI-Based QA Inspection Process FMEA Process

Collection Collection Reuse Reuse Aggregation

Examples for prominent Collective Intelligence Systems:

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Institute of Software Technology and Interactive Systems

Needs and required Tool Capabilities

Core goal of Isolated Methods under Investigation: § Inspection: early, effective, and efficient defect detection. § FMEA: early risk assessment. Needs and required tool capabilities for CI-based QA (derived from research & industry experts) § Defect Detection Performance – Support for effective and efficient defect detection early in the engineering process. – Immediate improvements of artifacts and engineering plans already after application

• f isolated methods.

§ Risk Assessment – Systematic and traceable risk assessment and quality assurance processes. – Defined responsibilities and roles for method application. – Guidance of less-experienced team members during method application. § Reuse of Experience and Tool Support – Reuse of experiences and knowledge from method application for engineering process improvement. – Inspection/FMEA tool support.

6

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Institute of Software Technology and Interactive Systems

Combining Inspection & FMEA from Process Perspective

§ Combined Inspection and FMEA process bridged with a Collective Intelligence System (CI-based QA) with input/output artefacts. (A) Inspection Process. (B) FMEA Process. (C) Engineering Knowledge.

7 (Software) Inspection Process FMEA Process Inspection Object, e.g., Systems Design Requirements

• A. Inspection Process

FMEA Team Reading Techniques, e.g., PBR Risk Analysis Methods, eg, FTA Team Defect List Engineering Knowledge RPN, Risks & corrective actions Engineering Knowledge Inspection Team

• B. FMEA Process

Engineering Knowledge

A2 A1 A3 B1 B2 B3

• C. Collective Intelligence System

A4 B4 C1 C2 A*, B*, and C* represent inputs and outputs to / from individual and important process steps

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Institute of Software Technology and Interactive Systems

Improvement Capabilities

Isolated Artefact Improvements (as it is) § Defect Detection with traditional inspection process steps (Ax). § Risk assessment with the traditional FMEA approach (Bx). Artefact Improvement of Method Combinations § Team defect lists (output of inspection) can improve the risk assessment based on real-world defects (Ax and Bx). Improvement of Inspection / FMEA Methodology § Inspection improvement based on lessons learned from Inspection Application (Ax, C1). § FMEA improvement based on lessons learned from FMEA Application (Bx, C2). § Combined and cross-method benefits arise from reusing and generalizing engineering knowledge as a foundation for method improvement (A4/B4 C1/C2)

8

(Software) Inspection Process FMEA Process Inspection Object, e.g., Systems Design Requirements

• A. Inspection Process

FMEA Team Reading Techniques, e.g., PBR Risk Analysis Methods, eg, FTA Team Defect List Engineering Knowledge RPN, Risks & corrective actions Engineering Knowledge Inspection Team

• B. FMEA Process

Engineering Knowledge A2 A1 A3 B1 B2 B3

• C. Collective Intelligence System

A4 B4 C1 C2

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Institute of Software Technology and Interactive Systems

Conceptual Evaluation

§ Derived from discussions with research and industry experts. § Traditional inspection and FMEA processes and the CI-based QA process approach.

9

Needs and Capabilities Inspection FMEA CI-Based QA

+ Effective and efficient defect detection ++

• ++

+ Effective and efficient risk assessment

• ++

++ + Systematic quality assurance

• ++

+ Traceable results

• ++

+ Defined roles and responsibilities

• + Guidelines for method application (methodological support)
• ++

+ Reuse of Experiences and Knowledge

• ++

+ Immediate artifact improvements

• ++

+ Tool support

• + Implementation/Application Effort
• Reuse of Experience and Tool Support for Engineering Process Improvement

Risk Assessment

Defect Detection Performance

Legend: ++ strong support, o neutral support, weak support * Tool support & prototype solution of the CI-based QA approach is currently under development. *

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Institute of Software Technology and Interactive Systems

Summary and Future Work

Summary § On process level, the goal was to bridge the gap between isolated and limited explicit engineering knowledge that hinder efficient reuse. § A CI-based QA process approach is reasonable to bundle benefits of different (isolated) QA methods, such as Inspection and the FMEA (RI-1). § Based on expert discussion we elicited key capabilities and fundamental requirements for CI-based QA (RI-2) and initially evaluated these key capabilities with the conceptual process prototype. Limitations § Initial conceptual evaluation on process level for two selected QA approaches. § Collective intelligence systems have been considered as black-box with expected key capabilities. § Currently no fully-featured CI System available (currently under development). Future Work § Definition of a collective intelligence system that is capable of supporting key capabilities. § Implementation and evaluation of the CI-based QA approach. § Empirical evaluations also in larger industry context.

10

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Institute of Software Technology and Interactive Systems

Thank you ...

Collective Intelligence-Based Quality Assurance: Combining Inspection and Risk Assessment to Support Process Improvement in Multi-Disciplinary Engineering

Dietmar Winkler1,2, Jürgen Musil2, Angelika Musil2, Stefan Biffl2

1SBA Research gGmbH, Vienna, Austria 2Vienna University of Technology, Christian Doppler Laboratory “Software Engineering

Integration for Flexible Automation Systems” (CDL-Flex) cdl.ifs.tuwien.ac.at dietmar.winkler@tuwien.ac.at