Adaptive Fault Tolerant Systems: Adaptive Fault Tolerant Systems: - PowerPoint PPT Presentation

1 Adaptive Fault Tolerant Systems: Adaptive Fault Tolerant Systems: Reflective Design and Validation Reflective Design and Validation Marc-Olivier Killijian Dependable Computing and Fault Tolerance Research Group – Toulouse - France

2 Motivations Motivations Motivations • Provide a framework for FT developers – Open – Flexible – Dependability of both embedded and large scale distributed systems – Adaptation of fault tolerance strategies to environmental conditions and evolutions • Validate this framework – Test – Fault-injection

3 History History History • Reflection for Dependability 1. Friends v1 - off-the-shelf MOP – Limits: static MO, inheritance, etc. 2. Friends v2 - ad-hoc MOP / CT reflection 3. Multi-Level Reflection Validation of the platform • Test of MOP based architectures • Fault-injection and failure modes analysis •

4 Outline Outline Outline • Reflection for Dependability 1. Friends v1 - off-the-shelf MOP – Limits: static MO, inheritance, etc. 2. Friends v2 - ad-hoc MOP / CT reflection 3. Multi-Level Reflection Validation of the platform • Test of MOP based architectures • Fault-injection and failure modes analysis •

5 Why Reflection? ? Why Reflection? Why Reflection • Separation of concerns – Non functional requirements – Applications • Adaptation – Selection of mechanisms w.r.t. needs – Changing strategies dynamically • Portability/Reuse – Reflective platform (relates to adaptation) – Meta-level software (mechanisms)

6 Overall Philosophy Overall Philosophy Overall Philosophy Metalevel Comm. protocols (metaobjects and policies) OS / kernel Middleware Hardware platform Wrappers One-to-one One-to-many API Dynamic association Baselevel (application objects) Mechanisms Failure Testing SWIFI Development Modes analysis strategies Faults Assumptions Fault model

7 Friends v2 : A MOP on on Corba Corba Friends v2 : A MOP • MOP design Identify information to be reified and controlled • MOP implementation Compile-time reflection Using CORBA facilities • Prototype for illustration Architecture and basic services Fault tolerance mechanisms Preliminary performance analysis

8 Necessary information : Necessary information : integrated mechanism example integrated mechanism example 4-send checkpoint 1-invocation Client Replica Server 6-return 7-reply 2-process invocation 3-obtain checkpoint 5-apply checkpoint

9 Necessary information : Necessary information : metaobjects example example metaobjects 4-send checkpoint Meta MO MO Stub Server Replica 7-reply 6-return Client Stub Server Replica 5-apply checkpoint 1-invocation 2-process invocation 3-obtain checkpoint Observation Control invocations method calls state capture state restoration

10 Which protocol? Which protocol? Meta Object Observation Control – Interception – Activation • Creation • Creation • Destruction • Destruction • Invocations • Invocations (In and out) – State capture – State restoration – Links control – Links control • Object/metaobject • Object/metaobject • Clients/servers • Clients/servers Reification Intercession Object

11 Protocol definition Protocol definition Meta 1 Meta 2 stub object Metastub Metaobject Stub Object Client Stub Server 1 2 Protocol and interfaces specific to a mechanism

12 Using Open Compiler Using Open Compiler Meta Class Compile-time MOP Class Open + Class Compiler MOP interInfo TranslateMethodCall ( ReifiedInfo) { ….. o.foo(); NewCode return NewCode; }

13 Architecture Architecture Services OF MOF GS OF MOF GS OF MOF GS OF MOF GS OF MOF GS OF MOF GS MS1 MO1 MO2 MS1 MO1 MO2 MOP MOP MOP C1 S1 S2 C1 S1 S2 ORB ORB Node 1 Node 2 Node 3 Node 1 Node 2 Node 3

14 Results Results • A method for designing a MOP – Analysis of mechanisms’ needs ! MOP features • Metaobject protocol for fault tolerance – Transparent and dynamic association – Automatic handling of internal state (full/partial) • Portable serialization [OOPSLA’02] – Smart stubs delegate adaptation to meta-stubs – CORBA compliant (black-box) – Some programming conventions

15 Lessons Learnt Lessons Learnt FT • Generic MOP – No assumption on low layers Language – Based on CORBA features ! With a platform « black-box» ORB • Language dependent • Limitations • external state Runtime • determinism • “Open” platform (ORB , OS and language) OS ! Additions of new features to the MOP ! Optimization of reflective mechanisms ! Language level reflection still necessary

16 Limits to to be addressed be addressed Limits to be addressed Limits • Behavioral issues – Concurrency models: Middleware level – Threading and synchronization: Middleware/OS level – Communication in progress: Middleware/OS level • Structural/State issue – Site-independent internal state : Open Languages – Site-dependent internal state: • Problems: Identification, handling • Available means: Syscall interception, Journals and replay monitors – External state • Middleware level • OS level " Concept of multilevel reflection

17 Which protocol? Which protocol? Meta Observation Control Object – Interception – Activation/control • Creation • Creation • Destruction • Destruction • Invocations • Invocations (In and out) • Threading • Threading • Synchronization • Synchronization • Communication • Communication – State restoration – State capture • Internal objects • Internal objects • Site-dependent objects • Site-dependent objects • External objects (MW+OS) • External objects (MW+OS) – Links control – Links control • Object/metaobject • Object/metaobject • Clients/servers • Clients/servers Object Reification Intercession

18 Which Platform ? ? Which Platform ? Which Platform C S S Fault-Tolerance Universal VM for Distributed Objects Middleware Middleware Middleware Language Support Language Support OS OS OS Hardware Hardware Hardware

19 Which Platform ? ? Which Platform ? Which Platform This one ? But difference between OS/MW … LS/MW? C S S Fault-Tolerance Universal VM for Distributed Objects Middleware Middleware Middleware Language Support Language Support OS OS OS Hardware Hardware Hardware

20 Which Platform ? ? Which Platform ? Which Platform Or this one ? C S S Fault-Tolerance Universal VM for Distributed Objects Middleware Middleware Middleware Language Support Language Support OS OS OS Hardware Hardware Hardware

21 Which Middleware ? Middleware ? Which Middleware ? Which FT needs to be aware of everything (potentially) C S S Fault-Tolerance Universal VM for Distributed Objects Under-ware Under-ware Hardware Hardware Hardware

22 Which Middleware ? Middleware ? Which Middleware ? Which FT needs to be aware of everything (potentially) but how ? Reflective languages … Fault-Tolerance Reflective middleware … Reflective OS … A lot of different concepts to manipulate

23 Multi-level Reflection level Reflection Multi-level Reflection Multi- mono-level multi-level aggregation meta-models meta-model Fault-Tolerance Self-contained, integrated, meta-interface

24 Multilevel Reflection Multilevel Reflection Multilevel Reflection • Apply reflection to a complete platform – Application, Middleware, Operating System • Consistent view of the internal entities/concepts – Transactions, stable storage, assumptions – Memory, data, code – Objects, methods, invocations, servers, proxies – Threads, pipes, files – Context switches, interrupts • Define metainterfaces and navigation tools – Which metainterface (one per level? Generic?) – Consistency # metamodel

25 Different Aspects Different Aspects Different Aspects • Intra-level information – Necessary for FT – Efficiency (lowest possible? Same concepts at ≠ levels?) • Inter-level information – ML management (inter-level coupling) – Adaptation – Concepts/levels navigation mono-level multi-level aggregation meta-models meta-model Fault-Tolerance Self-contained, integrated, meta-interface

26 Requirements of FT- Requirements of FT- Requirements of FT- Mechanisms? Mechanisms? Mechanisms? • Non determinism of scheduling/execution time ! ! Interlevel interactions mostly asynchronous ! ! Trend: Leverage know-how on FT asynch. distributed sys. ! ! Causality tracking/ monitoring of non-determinism is ! ! needed. ! ! State capture/ recovery at appropriate granularity is ! ! needed. ! ! ! … (?) ! p o 1 p t 1 t 2 q o 2 o 2 r t 2 q t 1 o 1 r P P [Kasbekar99, Basile02]

27 (I) Inter-Level Coupling Inter-Level Coupling Inter-Level Coupling • A Level = 1..n COTS = A set of interfaces = – Concepts – Primitives / base entities (keywords, syscalls, data types, …) – Rules on how to use them • (concepts, base entities, rules) = programming model – Very broad notion (includes programming languages) – Self contained • Base entities “a-tomic” within that programming model – Can’t be split in smaller entities within the programming model. – Implemented by more elementary entities within the component. – Implementation is internal ⇒ hidden to component user.