EOS: E Exactly xactly- -O Once E nce E- -S Service Middleware - - PowerPoint PPT Presentation

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

EOS: EOS: E Exactly xactly-

• O

Once E nce E-

• S

Service Middleware ervice Middleware

German Shegalov, Gerhard Weikum Roger Barga, David Lomet

(University of Saarland, Germany) (Microsoft Research, U.S.A.)

PRESS THE SUBMIT BUTTON ONCE AND ONLY ONCE. PLEASE BE PATIENT, SUBMISSION CAN TAKE UP TO 30 SECONDS

R E G I S T E R

Image by Bruce W. Grant

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Banal Examples

• The same item is ordered twice in an online-store. In some

cases the order has to be cancelled by calling a toll hotline.

• Home banking (PIN/TAN procedure). Each TAN may be used
• nly once.

– The first try to place a money transfer order results in a browser timeout. – After the subsequent browser refresh the server says: "A TAN was used twice. All other TANs have been frozen. To reactivate, please, contact the nearest branch office".

• 8 health insurance cards for a family of 3 persons.

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

ACK

Why does it happen?

Client Web Application Server Database Server

Purchase Request Order Confirmation Start Transaction

SQL Request SQL Response SQL Request SQL Response

Commit Transaction

Timeline

ACK

Transaction Restart

Purchase Request

Resubmission

Non-idempotent execution!

Exactly Once Guarantee is needed for correct execution

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Sample n-Tier System

Database Server Database Server Database Server Expedia Application Server Sabre Application Server Database Server Database Server Amadeus Application Server Database Server Expedia Application Server Sabre Application Server Database Server Database Server Amadeus Application Server Client Expedia Application Web Server Expedia Application Web Server Passport.com authentification

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Computational Model

§ Set of interacting components components: clients, servers etc.

piecewise deterministic (PWD PWD) deterministic replay deterministic replay results in the same end state end state as originally asynchronous events asynchronous events are logged in the proper order (timers, interrupts, incoming messages etc.) relevant resources resources are mapped to logical ids

§ Failures are soft (no damage to stable storage) § Fail-stop so that only correct data is logged

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Components Guarantees

§ § Testable State: Testable State: the component can determine if its state include some effects (e.g. receive or send of particular message). § § Testable Message: Testable Message: the component can determine if it sent or received a particular message. § § Persistent State: Persistent State: the component‘s state will be available as of some specific time (e.g. when some message was sent). § § Persistent Message: Persistent Message: content of a particular message will be available either by periodic resend or explicitly requested.

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Component Types

• eXternal Component (XCom

XCom) modelling human users

– cannot provide any of the guarantees

• Persistent Component (PCom

PCom)

– guarantees persistence and testability

• Transactional Component (TCom

TCom)

– guarantees persistence and testability

• nly at transaction boundaries

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Interaction Contracts

An IC IC between two components provides a mutually committed state transition committed state transition. Both components guarantee that the state transition is persistent. An IC can be set up a priori or dynamically negotiated.

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Committed IC (CIC CIC)/Sender

Sender Obligation 1 (S1): Sender Obligation 1 (S1): Persistent Sender State § The sender state is as of the time of the message or more recent. Sender Obligation 2 (S2): Sender Obligation 2 (S2): Persistent Message § § S2a: S2a: periodically re-send the message until (perhaps implicitly) released. § § S2b: S2b: re-send upon request until (explicitly) released. Sender Obligation 3 (S3): Sender Obligation 3 (S3): Unique Message § Unique message content (headers, timestamps etc.)

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Committed IC (CIC CIC)/Receiver

Receiver Obligation 1 (R1): Receiver Obligation 1 (R1): Message Dupl. Elimination § Dupl. messages detected & eliminated. Receiver Obligation 2 (R2): Receiver Obligation 2 (R2): Persistent Receiver State § § R2a: R2a: the receiver makes its state persistent before releasing S2a (stable interaction, message has been processed). § § R2b: R2b: the receiver makes its state persistent before releasing the sender from S2b (installed interaction)

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1 1. makePersistentState(); 2. msn++; 3. makePersistentMessage(m); 4. periodicResend(m); 5. stopResend(m); 6. forgetReceiver(msn);

Committed IC (CIC CIC)

Sender Receiver

1. if ( notDuplicated(m)) { 2. process(m); 3. makePersistentState(); 4. } 5. stableNotification(); 6. doSomething(); 7. ... 8. if(notInstalled(msn) 9. forceLog();

• 10. installedNotification();
• 11. forgetSender(msn);

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Immediately CIC (ICIC ICIC)

§ The receiver makes message and state persistent immediately immediately § The sender is released from S2a and S2b immediately by notification about installed interaction skipping stability notification.

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

eXternal IC (XIC XIC)

Output Message Send (X1): Output Message Send (X1): § A Pcom (client) displays an output message to the external user after having logged it. Input Message Receive (X2): Input Message Receive (X2): § The Xcom sends a message via keyboard, mouse etc. to a Pcom (client) and the message is logged immediately.

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Transactional IC (TIC TIC)/TCom

Atomic state transition(T1): Atomic state transition(T1): § either commit or abort of the transaction and persistence Faithful reply message(T2): Faithful reply message(T2): § faithful final reply message regarding the outcome of TC1 Persistent commit reply message(T3): Persistent commit reply message(T3): § persistence of the commit reply message

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Transactional IC (TIC TIC)/PCom

Persistent state and commit request message (P1): Persistent state and commit request message (P1): § persistence of commit request message and the corresponding state Unique messages (P2) Unique messages (P2) No periodic message re No periodic message re-

• send like in CIC

send like in CIC

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

If ICs are applied then all all failures can be masked failures can be masked with exception of failures during last external interaction.

Proof Proof

Induction on messages and state commits.

General Systems

XCom PCom1 1 PCom2 TCom 2 3 4 5 6 1) a is sent before b => a < b 2) b is sent after a received => a < b 3) ordering of asynchronous events is obtained from the log 4) a < b and b < c => a < c

Theorem Theorem

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

CIC CIC

Prototype System

Internet Explorer Database Server Database Server E-Service 1 Database Server Database Server E-Service 2 ...

XIC XIC CIC CIC TIC TIC (I)CIC (I)CIC CIC CIC TIC TIC

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Implementation Overview

Client (web browser) Recovery: Client (web browser) Recovery: § DHTML event handling § client persistence with XML store § JScript Application Server Recovery: Application Server Recovery: § enhanced PHP session management § enhanced PHP ODBC support Database Server Recovery: Database Server Recovery: § enhanced ODBC driver manager § stored procedures Phoenix/ODBC style

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Web Browser Recovery (I)

1. 1. < <sdk:logger sdk:logger style= style=' 'behavior:url(#default#userData behavior:url(#default#userData);' );' 2. 2. id= id=' 'pagestate pagestate' ' 3. 3. type='hidden'> type='hidden'> 4. 4. </ </sdk:logger sdk:logger> > 5. 5. 6. 6. < <script script language="JavaScript language="JavaScript"> "> 7. 7. pagestate pagestate.load(installationPoint .load(installationPoint); ); 8. 8. pagestate pagestate.save(installationPoint .save(installationPoint); ); 9. 9. </ </script script> >

How to log? How to log?

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Web Browser Recovery (II)

What to log? What to log?

1. 1. var var inputFields inputFields = = document.all.tags('input document.all.tags('input'); '); 2. 2. var var htmlForms htmlForms = = document.forms document.forms; ; 3. 3. 4. 4. for(var for(var i=0; i < i=0; i < inputFields.length inputFields.length; i++) ; i++) 5. 5. inputFields[i]. inputFields[i].attachEvent attachEvent( ('onPropertyChange 'onPropertyChange' ', , 6. 6. updatePageState updatePageState); ); 7. 7. 8. 8. for ( for (var var i=0; i < i=0; i < htmlForms.length htmlForms.length; i++) { ; i++) { 9. 9. htmlForms[i]. htmlForms[i].attachEvent attachEvent( ('onSubmit 'onSubmit' ', , logFormSubmission logFormSubmission); ); 10. 10. htmlForms[i]. htmlForms[i].attachEvent attachEvent( ('onSubmit 'onSubmit' ', , periodicResend periodicResend); ); 11. 11. } }

XIC XIC CIC CIC Compromise 1: Compromise 1: no way to install interaction on the browser side, i.e., browser recovery always depends on PHP.

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Web AppServer Recovery

1. if( http_reply = lookUpReplyLog(sess_id, msn + 1) ) { 2. sendToBrowser(http_reply); 3. exit(0); // nothing to do anymore 4. } //end if 5. //there is no reply log entry yet 6. loadSessionData(session_id, msn); 17. //execute servlet, 18. //add new msn and JavaScript code for XIC 19. http_reply = 20. executeScript(http_request, sess_id, ++msn, JSCode); 21. // create new installation point 22. saveSessionData(session_id, msn); 23. saveLastUrl(sess_id, http_request->url); 24. addToReplyLog(http_reply, msn); 25. sendToBrowser(http_reply);

Installed Installed CIC CIC

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Web AppServer Recovery (II)

Compromise 4: Compromise 4: some users will never return after a failure. We need to define heuristic expiration period for garbage-collection Compromise 2: Compromise 2: no way to request the original request message from the client after a crash on the web server Compromise 3: Compromise 3: no sense to periodically re-send the http reply. The PHP engine will have to wait for the browser to start recovery

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Stress Test Results

1 client machine test 0.1 0.2 1 5 10 session steps sec 5 client machines test 0.2 0.4 0.6 0.8 1 5 10 session steps sec

• riginal Zend

engine modified Zend engine with CIC

O v e r h e a d l e s s t h a n . 1 s e c e v e n f

• r

1 s t e p s s e s s i

• n

Accepted

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Open Issues

• Web application considerations:

– identifying and killing user-cancelled requests within PHP Engine

• Many application-specific optimization opportunities for

application-independent recovery.

• G. Shegalov et. al

EOS: Exactly-Once E-Service Middleware VLDB’2002 Demo Session 1

Summary

EOS EOS provides a framework for application-independent application recovery § (almost) perfect failure masking § provably correct handling of failures − exactly-once semantics § faster software development by designing new applications as if there were no failures if there were no failures at all. § almost no changes no changes in existing legacy software § acceptable overhead