Traditional Systems . . . Systems of Systems Specific Example Need for Generating . . . How to Generate Worst-Case Crisp Case Scenarios When Testing Component-Wise . . . Applying the . . . Already Deployed Systems Algorithm: Next . . . From Crisp Case to a . . . Against New Situations Home Page Title Page Francisco Zapata 1 , Ricardo Pineda 1 , and Martine Ceberio 2 ◭◭ ◮◮ ◭ ◮ 1 Research Institute for Manufacturing & Engineering Systems (RIMES) Page 1 of 21 2 Department of Computer Science University of Texas at El Paso Go Back 500 W. University, El Paso, TX 79968, USA Full Screen 1 fazapatagonzalez@utep.edu, rlpineda@utep.edu, 2 mceberio@utep.edu Close Quit
Traditional Systems . . . Systems of Systems 1. Traditional Systems Engineering Approach Specific Example • Traditionally, a system of interest (SOI) is developed Need for Generating . . . by eliciting requirements from the stakeholders. Crisp Case Component-Wise . . . • These requirements are analyzed to build an architec- Applying the . . . tural design that will drive the system development. Algorithm: Next . . . • Through an iterative process the system is constantly From Crisp Case to a . . . refined via: Home Page – elicitation and update of requirements, Title Page – design, ◭◭ ◮◮ – development, and ◭ ◮ – testing. Page 2 of 21 • Eventually, a final product is obtained. Go Back • In this approach, the development of the SOI is limited to the requirements specified by the stakeholders. Full Screen • Here, emergent behavior is not welcomed. Close Quit
Traditional Systems . . . Systems of Systems 2. Systems of Systems Specific Example • Since the 1990s: Need for Generating . . . Crisp Case – advances in Information and Communication Tech- Component-Wise . . . nologies (ICT) Applying the . . . – have enabled greater capabilities to exchange infor- Algorithm: Next . . . mation between systems in near real-time. From Crisp Case to a . . . • The integration of these independently developed sys- Home Page tems required: Title Page – communication interface standards, ◭◭ ◮◮ – information models, and ◭ ◮ – inter-operatibility standards. Page 3 of 21 • This integration need has given birth to a new kind of Go Back meta-systems called Systems of Systems (SoS). Full Screen • Example: an airplane contains navigation, propulsion, GPS, communication, and other systems. Close Quit
Traditional Systems . . . Systems of Systems 3. Systems of Systems (cont-d) Specific Example • A SoS is a system of interest which is: Need for Generating . . . Crisp Case – a collection of large-scale, heterogenous systems, Component-Wise . . . – that inter-operate to achieve a greater common ob- Applying the . . . jective. Algorithm: Next . . . • A SoS is characterized by the following attributes: From Crisp Case to a . . . Home Page – operational independence, Title Page – managerial independence, ◭◭ ◮◮ – SoS evolutionary development, – SoS incremental functionality (knowledge domains), ◭ ◮ – geographical distribution. Page 4 of 21 • For constituent systems, new behavior is not welcomed. Go Back • But for the meta-system, some new emerging behavior Full Screen may be welcomed. Close Quit
Traditional Systems . . . Systems of Systems 4. Formulation of the Problem Specific Example • Before a complex system is deployed: Need for Generating . . . Crisp Case – Integration, Verification, Validation, Test and Eval- Component-Wise . . . uation (IVVT&E) methodologies Applying the . . . – are applied to known well-defined operational sce- Algorithm: Next . . . narios. From Crisp Case to a . . . • Once the system is deployed, new possible scenarios Home Page may emerge. Title Page • It is desirable to develop methodologies to test a system ◭◭ ◮◮ against such emergent scenarios: ◭ ◮ – an unmanned Aircraft System (UAS) encounters Page 5 of 21 new scenarios that were not predicted; Go Back – a health care monitoring system may encounter a new illness that was not known before. Full Screen Close Quit
Traditional Systems . . . Systems of Systems 5. Specific Example Specific Example • In this paper, we start the analysis by considering the Need for Generating . . . simplest example. Crisp Case Component-Wise . . . • As such an example, we take an automatic system that Applying the . . . helps prevent a car from getting too close to the walls Algorithm: Next . . . of a freeway. From Crisp Case to a . . . • At first glance, all we need for this is a sensoring system Home Page that measures a distance x from a car to an obstacle. Title Page • There are usually several distance sensors, and the sys- ◭◭ ◮◮ tem is set up to work well in the expected situations. ◭ ◮ • The problem starts when we encounter a new unex- Page 6 of 21 pected situation, e.g., a hole in the nearby wall. Go Back Full Screen Close Quit
Traditional Systems . . . Systems of Systems 6. Specific Example (cont-d) Specific Example • In the case of a hole in the wall: Need for Generating . . . Crisp Case – some sensors measure the distance to a wall, while Component-Wise . . . – other sensors measure the distance to a next far- Applying the . . . away wall (located very far from the road). Algorithm: Next . . . • As a result, the existing algorithms may under-estimate From Crisp Case to a . . . the distance to the obstacle. Home Page • So, even when the car is very close to the wall, the sys- Title Page tem may operate under the false impression of safety. ◭◭ ◮◮ ◭ ◮ Page 7 of 21 Go Back Full Screen Close Quit
Traditional Systems . . . Systems of Systems 7. Need for Generating Worst-Case Scenarios Specific Example • Once the system designers realize that novel situations Need for Generating . . . are possible: Crisp Case Component-Wise . . . – they can come up with methods to improve the Applying the . . . system’s performance on non-standard situations; Algorithm: Next . . . – then, they need to test these methods. From Crisp Case to a . . . • A usual way of testing a system is to test it on worst- Home Page case scenarios. Title Page • So, we face a question of generating such worst-case ◭◭ ◮◮ scenarios. ◭ ◮ • In this talk, we explore: Page 8 of 21 – the ways of generating worst-case scenarios to val- Go Back idate system behavior under unexpected scenarios Full Screen – on the example of the above car problem. Close Quit
Traditional Systems . . . Systems of Systems 8. How the Distance-Measuring System Is Set Up Specific Example Now: A Simplified Description Need for Generating . . . • The distance-measuring system usually involve several Crisp Case sensors to account for robustness (redundancy). Component-Wise . . . Applying the . . . • Each of the sensors produces a measurement result x i . Algorithm: Next . . . • So, we need to estimate the actual distance d based on From Crisp Case to a . . . these measurement results x 1 , . . . , x n . Home Page • Because of the measurement noise, for each distance d , Title Page we get slightly different values x i ≈ d ◭◭ ◮◮ • In many cases, the measurement error is normally dis- ◭ ◮ tributed, with a standard deviation σ . Page 9 of 21 • In other words, for each result x i , we have a probability distribution with the probability density Go Back − ( x i − d ) 2 1 � � Full Screen ρ d,i ( x i ) = 2 π · σ · exp √ . 2 σ 2 Close Quit
Traditional Systems . . . Systems of Systems 9. How the Distance-Measuring System Is Set Up Specific Example Now (cont-d) Need for Generating . . . • Measurement errors corresponding to different mea- Crisp Case surements are usually independent. Component-Wise . . . Applying the . . . • So, the probability density ρ d ( x ) for the vector x = ( x 1 , . . . , x n ) of measurement results is a product: Algorithm: Next . . . From Crisp Case to a . . . n − ( x i − d ) 2 � � 1 � ρ d ( x 1 , . . . , x n ) = 2 π · σ · exp Home Page √ . 2 σ 2 i =1 Title Page • As a desired estimate d for the distance, it is reasonable ◭◭ ◮◮ to select the most probable value d , ◭ ◮ • In other words, we select the value d for which the Page 10 of 21 probability ρ d ( x 1 , . . . , x n ) is the largest possible. Go Back • Equating the derivative to 0, we get an estimate x = x 1 + . . . + x n Full Screen . n Close Quit
Traditional Systems . . . Systems of Systems 10. Criterion for Selecting a Worst-Case Scenario Specific Example • Reminder: a reasonable way to estimate the distance d Need for Generating . . . is to take the average x of measured values x 1 , . . . , x n . Crisp Case Component-Wise . . . • This average works well in standard situations. Applying the . . . • In non-standard situations, an alert is needed when the Algorithm: Next . . . smallest m of the distances is dangerously small: From Crisp Case to a . . . def = min( x 1 , . . . , x n ) ≪ d min . m Home Page • When the minimum m is close to the average x , the Title Page situation is not so bad. ◭◭ ◮◮ • Situation is bad when there is a drastic difference be- ◭ ◮ tween x and m Page 11 of 21 • The worst-case scenario is when the difference x − m Go Back is the largest: Full Screen x − m = x 1 + . . . + x n − min( x 1 , . . . , x n ) → max . n Close Quit
Recommend
More recommend
