Web-enabling Legacy Systems via Presentation Access: From Webulation - - PDF document

SLIDE 1

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/267236046

Web-enabling Legacy Systems via Presentation Access: From Webulation to Automation

Article

CITATIONS READS

74

2 authors, including: Some of the authors of this publication are also working on these related projects: IBM Centers for Advanced Studies View project WikiDev 2.0: A web platform for collaborative software development View project Mohammad El-Ramly Cairo University

43 PUBLICATIONS 625 CITATIONS

SEE PROFILE

All content following this page was uploaded by Mohammad El-Ramly on 18 September 2015.

The user has requested enhancement of the downloaded file.

SLIDE 2

Web-enabling Legacy Systems via Presentation Access: From Webulation to Automation

Mohammad El-Ramly Eleni Stroulia Department of Computer Science University of Leicester Leicester, LE1 7RH, UK mer14@cs.le.ac.uk Computing Science Department University of Alberta Edmonton, T6G 2H1, Canada stroulia@cs.ualberta.ca Abstract

After years of development and billions of invested dollars, legacy mainframe systems have become the lifeblood of many corporations. And today, given the dramatic changes that the Internet revolution has brought to today’s business processes, it has become necessary to make these systems accessible to partners, employees and

• customer. In principle, there are three different

approaches to that end: one can enable access to the legacy system via its data, its logic or its presentation

• layers. In this paper, we review the state-of-the-art in the

last approach, namely Web-enabling via presentation layer access. We discuss both academic research methods and industrial practices, we describe their evolution and their potential, and we comparatively evaluate their pros and cons. Finally, we present CelLEST, a lightweight method we developed for semi-automated reengineering and Web-enabling of legacy user interfaces. We present how this method uses artificial-intelligence algorithms to leverage and advance current manual technology.

• 1. Introduction

Over years of development and investment, business software systems, such as bank finance systems, customer-relationship management (CRM) systems and airline-reservation systems, grew in size and value. They constitute one of the most important assets for many

• companies. [

19] Corporations have invested substantially in developing these mainframe-based legacy systems and making them Y2K and Euro compliant. [ 25] In return, mainframe-based legacy systems have proven reliability and scalability in providing business-critical processing needs, especially for applications involving huge numbers of transactions and simultaneous users like banking and airline-reservation applications. Moreover, many of the business processes and policies of companies are encapsulated in the logic of legacy systems. For many corporations, legacy systems will remain their Information Technology (IT) backbone for years to come. [ 3] When the Internet emerged as a medium for conducting business, it created new opportunities for growth and cost

• reduction. This gave birth to a new research area: “Web-

enabling legacy systems” to develop methods for opening existing legacy software for access through the Internet, an extranet and/or an intranet to the public and/or to a selected user-base of employees, customers, partners, etc. Separately, industry developed its own solutions as well. A legacy software application in general, consists of three layers: the presentation layer, the program logic layer and the data and data access layer. A Web application can access a legacy system via one or more of these layers, depending on the available legacy and Web

• technologies. This requires reproducing the legacy

business processes and data in new formats and presentations for the old and new users [ 18], and possibly re-architecting the legacy application. The challenge is that many legacy systems were developed using the technology of the 1970s to mid-

• 1980s. They have been modified many times by different

programmers and frequently are poorly structured and

• documented. Separation between data, logic, and

presentation layers rarely exists. They were not designed with integration in mind. As a result, they have become complex and difficult to understand, maintain, renovate, reengineer, and/or integrate with other systems. They aged! [ 20, 27] A main weakness of mainframe legacy applications is their user interfaces (UIs). They fall short in three areas: user access, usability and navigation: [ 4] User Access. Most legacy systems are proprietary monolithic systems that were not designed for integration with the Web or other technologies. Usually they do not have clear separation between their presentation, logic and data layers. This makes opening a legacy system for access via a new platform or for front-end integration with other systems a hard task.

• Usability. Legacy text-based UIs have limited display

capabilities, and are non-intuitive and hard to learn. They were adequate for their time but dissatisfy today’s users, who are used to graphical user interfaces (GUIs) and Web

• interfaces. Training their new users is slow and costly.
• Navigation. The limited presentation capabilities of

legacy text-based UIs impose tedious navigation patterns to accomplish user tasks. E.g., flipping a multi-page report may require using function keys or issuing some

SLIDE 3

commands to move between the many screens of the

• report. While in a GUI environment, a scroll bar gives

instance access to any report page with a mouse click. In many cases, it is desired to migrate the UI of legacy system to a GUI platform or to the WWW or to integrate it with other legacy front-ends. But the performance of the legacy system and its hosting platform is satisfactory and the owner organization is unwilling to alter its code or structure in any way due to the cost and risk involved. Because invasive solutions are generally undesirable and often unnecessary, there is a need for lightweight non-invasive UI reengineering methods to support the Web-enabling of legacy systems. Web-enabling via presentation access falls in this category. It is long- practiced by industry but almost ignored by academia. In this paper, we aim to “bridge the gap” between industry and academia by giving a detailed discussion of this class of Web-enabling solutions. We compare this approach with the alternatives; we describe its evolution, state of the art and its future directions. We describe our work in the CelLEST UI reengineering project [ 22, 24] in the Software Engineering Research Lab. at University of Alberta to leverage the current industry practices, minimize the labor needed to implement them, and develop the next generation of this technology. We employed a combination of document analysis, feature extraction, clustering, user action modeling, visualization, data mining, task model inference, XML wrapping and automated GUI layout to help develop an intelligent semi- automated lightweight method for legacy system Web- enabling and UI reengineering. Furthermore, we have implemented this method in a prototype environment and evaluated it through a set of real-world case studies. This paper is organized as follows: Sections 2 and 3 brief other existing Web-enabling technology, i.e., Web- enabling via data and logic access. Section 4 discusses Web-enabling via presentation access, its evolution, state

• f the art, pros and cons. Section 5 is an overview of the

CelLEST semi-automated method for UI reengineering. Section 6 is a brief on the CelLEST method evaluation and limitations along with some conclusions.

• 2. Web-enabling via Data Access

In this class of solutions, a Web application directly accesses the legacy database, processes the legacy data and presents it to the Web browser clients. The Web application includes all the data-processing logic required. This logic may duplicate the original legacy-application logic, if the intended Web users are the same users of the legacy system, or it may be completely different, targeting a new user group. The legacy data must be wrapped in

• rder to be accessed using a different interface or protocol

than that for which the data was designed initially. This requires using data access middleware, i.e., database gateways and bridges. A database gateway translates between two data-access

• protocols. A number of de facto industry standard

database gateways exist. For example, the Open Database Connectivity (ODBC) from Microsoft can be invoked inside Active Server Pages (ASP) or programs in C/C++, Perl, VB, etc. The Java Database Connectivity (JDBC) by Sun is designed for database-independent connectivity between Java applications and a wide range of SQL

• databases. The JDBC API can be invoked inside a JAVA

program, applet or servlet or a Java Server Page (JSP). Finally, the ODMG is the standard of the Object Data Management Group for persistent object storage. A bridge translates a standard protocol to another, such as the JDBC-ODBC bridge for example. [ 9] Web-enabling via data access is a straightforward solution, appropriate for multi-source legacy data integration to new applications. At the same time, it has considerable drawbacks. First, any important logic, e.g., data validation and business rules, is bypassed and not utilized and needs to be duplicated in the Web

• application. This incurs high cost in development and
• maintenance. Second, it increases the data coupling

between the legacy and Web applications. It is most suitable when the Web application logic is simple, e.g., Web-enabling the services of a library system. [ 2, 21]

• 3. Web-enabling via Logic Access

Web-enabling via application logic access relies on the availability of a mechanism to access the business logic independent from the user interface related code. This can be accomplished in different ways: Code Access. If the legacy business logic is implemented separately from the presentation logic, which is rarely the case, then theoretically, it is possible to insert a thin control layer that accepts the data extracted from the client browser’s HTTP request and invokes the appropriate legacy module. The business logic subroutine processes the request and gives back the results, which are then forwarded to the client. [ 25] API Access. Packages like SAP, PeopleSoft, etc. offer APIs that can be accessed via Java Native Interface (JNI)

• r Common Gateway Interface (CGI) code. But, in-house

developed applications rarely have defined APIs. [ 2] Distributed Object Technology (DOT) extends

• bject technology to the net-centric information systems
• f modern enterprises by using object middleware, e.g.,

OMG's CORBA and Microsoft COM+. The idea is to

• bjectify (or objectize) the legacy system by creating an

OO interface to individual applications, common services and business data, which makes the legacy software components “look” like objects. Then it can be accessed by other applications across a network through the OO

• interface. [

9, 21, 29] This is a quite invasive reengineering solution. The main challenge is objectifying the legacy system, i.e.,

SLIDE 4

analyzing, decomposing, and then translating the monolithic plain semantics of the procedural – in most cases - legacy system to the richly hierarchic structured semantics of an object-oriented (OO) system. [ 9] Several methods have been developed for this task. [ 6] The effort needed depends on the language, style and architecture used to develop the legacy system. Component wrapping is a natural extension to DOT. In contrast with objects, components must conform to a component model. This constraint enables the component framework to provide the components with quality

• services. An example server-side component architecture

for writing reusable business logic and portable enterprise applications is Enterprise JavaBeans (EJB) from Sun

• Microsystems. EJB is the basis of Sun's Java 2 Platform,

Enterprise Edition (J2EE). EJB components are written entirely in Java and run on any EJB compliant server. Each bean encapsulates a piece of business logic. EJB servers provide system-level services such as transactions, security, threading, state management, resource pooling, distributed naming, and remote invocation. [ 9, 14] EJB can wrap exiting legacy system functions and offer them as operating system and platform independent

• components. Like DOT, the legacy system needs to be

componentified by separating its interface into modules consisting of logical units or functions. [ 9] DOT and component wrapping approaches can be integrated by generating CORBA wrappers for the legacy components and then using EJB to develop an application server to integrate them and offer their services in the context of Web applications. [ 29]

• 4. Web-enabling via Presentation Access

In contrast to the data and logic access approaches, Web-enabling via presentation access is non-invasive and almost risk-free. This approach, in its simple form of Web emulation (webulation), emerged quite early. Gradually, webulation evolved into the more advanced screen- scraping technology that takes advantage of the GUI capabilities of Web browsers. The next generation was screen mapping, which can reengineer the legacy UI using complex manipulations of the legacy data streams used for communication between the legacy host and the legacy terminals. This allows remodeling the legacy UI into the form-based style natural to Web-applications’ UI, and possibly integrating it with other legacy UIs or Web-

• applications. The method that we have developed in the

CelLEST project advances this screen-mapping approach further: it essentially automates it using intelligent algorithms to learn the legacy-system behavior and then build the models needed for screen mapping. These models are used to automatically build a working Web- based GUI or abstract GUI specifications. More often than not, the legacy application is accessed via its presentation layer, represented by the legacy

• utbound and inbound data streams. In fewer cases, the

legacy UI is accessed via its presentation description if exists, e.g., CICS maps for IBM mainframe systems or AS/400 Data Description Specification (DDS) screen

• definitions. In either cases, access is limited to the data

and operations offered via the application presentation.

4.1 Web Emulation (Webulation)

Web Emulation, or Webulation [ 4], emerged as the natural extension of the long-practiced legacy host emulation to the WWW. The basic innovation is that the emulator runs in a Web browser or a Web server. Browser displays have the native look and feel of the host legacy

• screens. Transactions work exactly the same as on a

legacy host “green screen” terminal, e.g., IBM 3270, by returning one screen display for one input request. Full support of legacy function and user customization of colors and fonts are normally available. Additionally, icons for function keys, copy/paste, macro recording, file transfer and other basic operations are provided. [ 1, 5] Webulation is relatively quick and inexpensive and does not need any Web application development. It offers instant access to the legacy application to intranet and extranet users who are already familiar with the legacy

• system. But, it does not make the legacy system any

easier to use for new users. [ 5] Additionally, it does not allow tailored Web-access to target different user groups with different sets of UI functionality. It solves accessibility issue, but it does not improve the usability or navigability of the legacy system. A typical implementation of webulation uses Java applets downloaded into the client Web browser. The applet runs in the Web browser Java runtime environment and establishes a connection with a Telnet server that manages access to the host application. [ 5, 26]

4.2 Screen Scraping (Refacing)

Screen scraping takes webulation a step further by

• ffering enhanced one-for-one browser presentation of the

legacy UI. [ 3] It reads the data stream intended for, e.g., the mainframe terminal, either via a client based terminal emulator (Java applet) or a server based program, and turns it into a Web-based GUI presentation. A legacy screen can be translated to a Web-based GUI either on the fly or using a user-defined customization. See Figure 1. In the first case, a middleware is interleaved between the Web front-end and the legacy software to act as a presentation translator by intercepting outbound legacy displays and converting them on the fly into a Web front- end using whatever available information. For IBM 3270, this is done by converting the unprotected/input fields into edit control objects and turning the other fields into labels. Some tools convert predefined strings like F1,… F24 into

• buttons. The resulting user interface is a slight

improvement over webulation.

SLIDE 5

In the second case, more substantial improvement can be achieved. In this case, an individual customization is created for every screen to radically change its appearance and takes advantage of the presentation potential of the target platform. So, legacy screens appear “dressed up” in Web-based GUIs, with widgets, lists, buttons, images, Web links, check and choice boxes, colors, fonts, etc. Also, one can reorder data fields, change tab sequence and hide unnecessary data. [ 1, 4, 5] Some screen-scraping tools can create context-sensitive field help and lists of valid values for data fields. [ 5] To reface a number of legacy screens fast, some tools use customization templates, which contain common elements to all the Web-based screens replacing the legacy screens, e.g., a logo, a Web link, a customized tool bar, etc. [ 15] The operation of the applications is still one-for-one, i.e.,

• ne browser request equals one legacy screen display. [

1] To know which customization to apply to a screen instance (snapshot), the screen scraping application should recognize the identity of the instance using a predicate or signature for every screen. Typically, this signature is based on some unique keyword(s) that appear on the screen at some location(s). Some tools offer rich pattern definition languages for the application builder to define a

• signature. [

5, 7] Such languages would allow specifying that a specific text must exist or not exist at a certain location or within an area on the screen and/or that it must be of a certain case or color, or compared to a hard-coded value (<, <=, =, >=, >). They may allow combining multiple pattern expressions with logical operators. In many cases, this technique is sufficient to recognize the screens’ identities. But in other cases, like unstructured and multi-mode screens for example, it is quite challenging to discover an appropriate signature. E.g., some host applications have more than one mode for the same screen, e.g., Create, Review, or Update modes, with the same structure and appearance but with slight differences in the status of some fields. Each mode needs separate signature and customization. In the state of the art practices, a screen signature is manually defined and hard coded for every screen by an expert analyst, who is very familiar with the legacy system, the pattern language available and its supporting tools. Screen scraping takes advantage of the Web presentation capabilities; however, it does not benefit from its enhanced navigability. So it enhances accessibility and usability but not navigability of the legacy system. Another screen-scraping approach is to access the legacy presentation at a level lower than the UI, or the data streams used to construct it. This is the level of screen maps or descriptions if existing, e.g., CICS maps for S/390 systems or Data Description Specifications (DDS) source files for AS/400 systems. Some products extract the data necessary to build an HTML or XML form from CICS maps instead of IBM 3270 data streams.

4.3 Screen Mapping (Remodeling)

Screen mapping (remodeling), is a natural extension to screen scraping. It enables substantial reengineering of the legacy UI. It enables fairly extensive modifications to the sequence of information presented to the user by combining several screens into a single GUI presentation. It offers many-for-one browser presentation of the legacy

• UI. [

3] So, multiple host screens related to a certain user task are combined in one Web form that represents the user task more naturally in the Web world. This greatly enhances the usability and navigability of the system, while maintaining the back-end host navigational flow. It is possible to partially apply screen mapping to a legacy system by reengineering the frequent or cumbersome user tasks of choice, while using webulation and/or screen scraping for the rest of the system. The steps of a screen mapping solution are:

• 1. Build a behavioral model for the portion of the legacy

system UI to be reengineered,

• 2. Build a plan for each user task to be reengineered,
• 3. Build/buy the middleware needed to mediate between

the legacy back-end and the Web front-end, and

• 4. Build a Web-based GUI, which executes the user task

plans via the host access middleware. The behavior model is a state-transition graph whose nodes (states) represent the screens of the legacy system. Each screen is distinguished by a predicate or a signature that uniquely identifies its instances (snapshots). Each arc

• n the graph is a permissible user action for this screen

that causes a transition to another screen. A task plan is a description of how to open sessions with the host legacy host, gather data, complete transactions, and close sessions to accomplish a user task. Typically, this includes what user actions are needed to navigate the legacy UI in service of the user task, what inputs need to be passed to the legacy application on which screens on which locations and what outputs will be retrieved from which locations on which legacy screens. A host access middleware executes the task plans by driving the necessary navigation via the legacy host, passing the user inputs received from the Web-based GUI to the legacy application, and collecting the required

• utputs to feed the Web-based front-end. This middleware

uses terminal access protocols such as VT100, 3270 or 5250 to communicate with the legacy system via a “virtual terminal”. Data are moved in and out of the legacy host via its UI as if data entry personnel were flawlessly entering it. [ 10] Such middleware can be built with EJB beans, Java servlets, or similar approaches. The Web-based GUI, e.g. HTML or XHTML, presents the reengineered UI to the user, collects his input, triggers the middleware to execute the right task plan, and reproduces the task output in HTML or XHTML. CGI scripts, Java Server Pages, or similar technologies are used to collect the user input via the client Web browser.

SLIDE 6

Figure 1: A Legacy Screen (left), Refaced On-the-fly (middle) and Refaced Using Screen Customization (right) Screen mapping has been an industrial practice exclusively and its steps are implemented all manually. An analyst goes through each screen of the legacy system, to find a unique signature for it and to model its behavior in order to build the behavior (state-transition) model. Then s/he interviews the users about every task to be migrated to the Web and tries to understand all its possible paths and exceptions. Then, s/he manually builds a plan for this task, describing all the user actions needed to perform it, all the inputs to be entered and where they

• ccur and all the outputs to be collected and from where

they are obtained. Next, a developer builds the new Web front-end that executes the user tasks. For each task, s/he designs the necessary GUI and its layout and implements

• it. If it is required to migrate to a different platform too,

e.g., WAP devices [ 10], the behavioral model and task plans can be reused but the new UI implementation should be carried out from scratch for the new platform. Typically, screen mapping allows some flexibility in legacy UI and task modeling to accommodate uncertainty and unexpected events, e.g., it may allow defining [ 5, 16]:

• Global Screens, i.e., screens that are intermittently or

randomly displayed and always require the same action, e.g., error screens, system messages, etc.

• Data Looping, which allows repeating an action, e.g.,

displaying the next page of a query result, until a condition is met.

• Landmarks, i.e., specific labels on a legacy screen

that are used for relative identification of information elements (inputs or outputs) on the screen, instead of relying solely on the coordinates of the elements.

• Alternative Paths, i.e., auxiliary navigational paths

that may exist for a user task. A screen mapping solution can consolidate multiple existing applications, without modification, into one single integrated view. [ 5] Some tools can combine multiple back-end resources, e.g., 3270, 5250, VT, JDBC, etc., under one Web front-end. Thus, new user tasks can be created/automated to eliminate the need to manually transfer data between various back-ends to accomplish a

• task. This is useful for integrating similar legacy systems

at the front-end level due to business merger for example,

• r complementary systems in the same organizations.

Additionally, screen mapping can be used to slightly extend the legacy system functionality via repurposing or via addition of new business logic to the Web application. Repurposing is using the legacy system for tasks that it was not intended for. E.g., an insurance company offered

+------------------------------------------------------------------------------- |02 OCT 00 COUNTER SERVICE | +------------------------------------------------------------------------------- +---------------------------------- CLIENT ------------------------------------+ | Teller : DEW DANIEL WESTFALL Drawer : 91A CASH DRAWER A | | | | 1. Club : 272 Balance : | | 2. Last N. : Status : | | First N. : Rate Type: | | | | 5. Address1 : 17. Expiry : | | 6. Address2 : 18. Join Yr : | | 19. Mem Since: | | 7. City : 20. Drvs Lic : | | 9. Prov/Sta : 21. Pricing : | |12. H. Phone : | |13. B. Phone : | | |15. Contact | |16. Notes | | | +------------------------------------------------------------------------------+ Confirm New Member (Y/N) <<N>> . . . +-------------------------------------------------------------------------- |02 OCT 00 COUNTER SERVICE | +-------------------------------------------------------------------------- +---------------------------------- CLIENT -------------------------------+ | Teller : DEW DANIEL WESTFALL Drawer : 91A CASH DRAWER A | | | | 1. Club : 272 Balance : | | 2. Last N. : 0987654 Status : | | First N. : Rate Type: | | | | 5. Address1 : 17. Expiry : | | 6. Address2 : 18. Join Yr : | | 19. Mem Since: | | 7. City : 20. Drvs Lic : | | 9. Prov/Sta : 10. 21. Pricing : | |12. H. Phone : | |13. B. Phone : | | |15. Contact | |16. Notes | | | +-------------------------------------------------------------------------+ Confirm New Member (Y/N) <<N>> . . .

Yes No

Counter Services: New Client Form

Auto Moto Association

* Required Fields * Club * Last Name * First Name * Address * City

AB

* Province Regular ($60) * Rate Type

780 -

* Home Phone

780 -

• Buss. Phone

Ext.

Clear Form Next >> More Options

* Driver Lic. # Plus ($75)

SLIDE 7

access to its legacy claims system to a new user group; lawyers who are neither familiar with the system, nor the system was built to answer their questions. But, the bits and pieces of information they need can be retrieved through the legacy UI. Using screen mapping, a Web front-end was designed to allow querying the system, with lawyers’ queries. Then, an application server middleware executes the relevant task plan, collect the required data and present it to the lawyer in a suitable way.

4.4 Using Presentation Access to Create an API for a Legacy System

Besides Web-enabling, it might be required to package some of the legacy services in a callable API so that new Web-based applications can be built on top of the legacy system as explained in Section 3. Screen mapping was used for this purpose; the task plans essentially constitute the distinct callable services and this new API is accessed by other applications directly, in an EJB architecture or through a distributed object environment. A Web application layer can then be developed to access this new API of the legacy application. [ 8, 9] To do so, one needs to implement the first three steps of screen mapping only, i.e., develop a behavioral model of the legacy UI, a plan for each task to be wrapped, and a host access middleware. The task plans are then executed from inside the new API.

4.5 Advantages, Limitations and Risks

A market survey [ 3] showed that 60% of the IT staff administrating, maintaining or accessing information from legacy systems use screen scraping or screen mapping technology to integrate legacy systems with other

• systems. 45% of them batch data to a server for access

through a client application, and a similar number modify the host application to suit client access. Out of those who use presentation access technology, 60% use it to avoid changing the host application, 30% cannot change the host application and 44% use it because of its lower cost. These results summarize the advantages of Web- enabling via presentation access. It is a risk-free, non- invasive, less expensive solution. It needs no change to the legacy application. It is almost the only choice when changing the legacy application is not an option, e.g., due to lack of ownership or unavailability of the source code. It can be applied gradually using a mixture of methods. For example, webulation can give instant Web-access of the legacy application to the users familiar with it, while the UI of the complex user tasks is reengineered via screen mapping to provide an easy to use HTML front- end to the external new users, e.g., students registering for classes or customers placing orders. Finally, presentation access can be used for lightweight integration with other legacy systems and Web applications or for repurposing and limited functionality extension of the legacy system. On the other hand, screen scraping and screen mapping are labor-intensive non-automated processes. The available supporting tools are mostly limited to aiding the manual practices. They do not automate the subtasks involved, which may require a lot of effort and intuition. Additionally, these technologies are appropriate to mature stable applications. But for dynamic applications that need frequent changes as business processes change, keeping the Web application layer up to date with the latest changes incurs high maintenance overhead. Presentation access of legacy systems is criticized for being slow as it adds a remote extra layer on top of the legacy system. This depends on the implementation model

• used. Modern server side and host side implementations
• vercome this deficiency. In a host side implementation,

the host-access and the screen mapping middleware reside

• n the legacy host, e.g. S/390. A task plan is executed fully
• n the host and HTML pages are generated as needed and

sent to the user with the task results or to collect inputs. Another disadvantage of presentation access solutions is their vulnerability to unexpected events related to the host connection behavior, like keyboard lockups, session disconnections, broadcast messages from the host and error messages from the legacy application. [ 28] Careful analysis and modeling of the legacy text-base UI and the tasks of interest reduces this risk by anticipating as many

• f such events as possible and including a recovery

mechanism in the Web front-end application, but would require more investment and effort. Finally, presentation access Web-enabling has its

• limitations. It cannot extend the legacy functionality

beyond what is already achievable, directly or indirectly, through the legacy presentation. It only gives access to the data exposed through the legacy presentation In practice, a variety of Web-enabling technologies may be used within the same organization due to the wide variety of legacy and Web technologies available and the unique requirements of every Web-enabling project.

• 5. Automating Screen Mapping in CelLEST

In the CelLEST project [ 22, 24], we used artificial intelligence, software engineering, and other methods to automate the process of “learning” and reengineering legacy UIs as much as possible. Our overall objective has been to develop an intelligent semi-automated lightweight method for legacy system UI reengineering, Web- enabling, and front-end integration. We implemented and evaluated our method in a prototype environment. The premise of our automated approach is that monitoring the legacy system users while working with the legacy application and recording traces of their interaction (dialog) with the legacy UI can provide the basis for learning how the legacy UI behaves. The recorded traces are used to build, semi-automatically, the models and artifacts required for screen mapping. This includes a behavioral model of the legacy UI (state- transition model), models of the frequent user tasks of interest and an automatically generated Web-based GUI for these tasks. User feedback is used to ensure the correctness and completeness of the models generated.

SLIDE 8

The most important innovative feature of the CelLEST method is the automation of all aspects of the UI mapping and migration process, including "learning", modeling and reengineering of the legacy UI and the user tasks of

• interest. This substantial degree of automation reduces the

time and effort required for such migration tasks and, consequently, it alleviates the problem of dealing with frequently changing legacy systems, which is quite costly in the traditional manual practices. The second novel and unique feature of the CelLEST method is that it reengineers user tasks into abstract GUI specifications, represented in XML-based syntax. These specifications are then translated to XHTML for Web access or WML (Wireless Markup Language) for WAP (Wireless Application Protocol) access, using the appropriate CelLEST interpreter. Hence, the CelLEST approach can accomplish simultaneous reengineering of the same legacy UI to different platforms, using the platform- independent abstract GUI specifications. In the following, we give an overview of the CelLEST process for legacy system Web-enabling and UI reengineering, and show how it surpasses the current

• practices. Figure 2 shows the overall CelLEST process.

Tasks T1, T2 and T3 are the reverse engineering phase of the process. T4 and T5 are the forward engineering phase.

5.1 T1: Legacy Interface Behavior Modeling.

The purpose of this task is building a behavioral model for the legacy text-based UI, in the form of a state- transition model. [ 11, 22, 23] Each node (state) of the model corresponds to a screen of the legacy system UI, identified by a unique predicate. The predicates are inferred semi-automatically, using automatically extracted features for every screen snapshot recorded in the traces, and a clustering algorithm that groups similar snapshots together and then infers such predicates. Using a pattern- inference algorithm, the arcs of the state-transition model are inferred automatically. Each arc models a permissible user action. A prototype tool, LeNDI (The LEgacy Navigation Domain Identifier), was developed to test the methods and algorithms used in this task. LeNDI deals with data transfer protocols that are native block-mode protocols or can be emulated in block-mode, e.g., IBM 3270 and VT100, although the concepts implemented in it are applicable to scroll-mode data transfer protocols as well, e.g., IBM 5250. In the sequel, we brief the subtasks performed by LeNDI. Trace Collection. While the legacy system users are performing their regular tasks, their interaction (dialog) with the legacy UI is recorded in the form of traces or sequences, using a specially instrumented emulator. For block-mode data transfer protocols like IBM 3270, a trace is a series of screen snapshots forwarded by the legacy application to the user’s terminal, interleaved with user actions in the form of sequences of keystrokes performed in response to receiving screen snapshots. Additional information is captured as well. For IBM 3270 data streams, these include the total number of fields, the number of unprotected fields and the initial cursor

• position. We call these traces “interaction traces”.

Feature Extraction. [ 23] In order to automatically extract legacy screen signatures or predicates one needs a rich set of features to build such predicates. LeNDI employs a variety of document analysis techniques to extract visual and other features for every snapshot. The

• utput of this subtask is a feature vector for every
• snapshot. These features include:
• 1. The existence of special system keywords, sentences or

information at the top or bottom of the snapshot, e.g., title, code, date, time or page number.

• 2. The information received with the outbound legacy data

streams, e.g., the location and type (protected or unprotected) of 3270 data fields and the cursor position.

• 3. Snapshot layout features like the classification of a

snapshot to “general”, “table” or “list” with some attributes for the last two categories, e.g., the number of columns, rows, etc. Another layout feature is vertical and horizontal histograms built for the entire snapshot content or for partial content of interest, e.g., numbers. Snapshot Clustering and Screen Recognition. Snapshot clustering is the process of grouping similar snapshots together for the purpose of inferring their common identity represented by a signature or predicate that uniquely distinguishes them from other snapshots. Then, given the snapshot clusters, a classifier can be induced that can correctly classify the individual snapshots into their corresponding clusters. This classifier can then be used at runtime to recognize new, previously unseen snapshots as instances of the UI states or screens, Figure 2: The CelLEST UI Reengineering Process.

• task patterns

and examples

• T2
• Task

Discovery

• T4
• GUI

Specification

• T3
• Task

Modeling

• T1
• Interface

Modeling

• Traces
• interface state-

transition model

• Task Model
• Legacy application
• Emulator

Web-based GUI

SLIDE 9

and hence, to infer which actions are possible on each screen and to which screens they lead. Also, knowing the snapshot identity allows the new reengineered GUI to apply whatever relevant input or output steps of a task plan to the snapshot, via the host-access middleware. LeNDI employs two clustering techniques. The first is an incremental clustering algorithm. [ 23] The Second is a two-phase top-down clustering algorithm. [ 11] The user can choose which one to use depending on the system under analysis. The accuracy of the classification depends mostly on the quality of the input traces, i.e., how well it covers the user interface behaviors. It is important to note here that our clustering process is interactive and can be guided by a CelLEST user familiar with the legacy UI. User feedback on misclustered snapshots is used to correct and fix the classifier generated. User Action Modeling. Action modeling [ 11] is the process of inferring a model for the user action needed to transfer the legacy system UI from a screen to another, i.e., from a state on the state-transition model to another. Different styles of user-interaction with legacy systems exist, e.g., function-keys, menu-driven, command-driven, and form-filling. An action can have several formats; e.g., a command keyword may have multiple synonyms or it may have an equivalent function key. Currently, LeNDI can model command-driven and function-keys styles. LeNDI infers each action model by comparing the instances of this action recorded in the interaction traces and applying a set of rules for command language design. For each action, LeNDI infers its syntax in terms of the function or control key(s) used and the command keyword(s) and its arguments. For the arguments, it infers their number, their syntax if any and whether they are

• ptional or mandatory. The availability of many instances
• f an action results in an accurate action model. LeNDI’s

user can override, rewrite or fix an inferred action model.

5.2 T2 and T3: Frequent User Task Discovery and Modeling

The purpose of these two tasks is to automate the process of modeling the frequent user tasks of interest as much as possible. Hence, T2 and T3 save the intensive labor work needed to define all possible navigation paths for every task that needs to be reengineered and every piece of data exchange that takes place during the task. This is done by automatic learning from the interaction traces about the frequent user tasks in terms of what navigational path is traversed and what type of input is entered on (and output collected from) which location on which screen for every task. T2: Task Discovery. [ 12,

13] To automate the discovery

• f the user's frequent interaction-patterns with the legacy

system, which correspond to the frequent user tasks, two algorithms for sequential pattern mining were developed, called Interaction Pattern Miner (IPM) [ 12] and Interaction Pattern Miner 2 (IPM2). [

13] Both algorithms

can discover similar segments of interaction with the legacy system, in the recorded traces, even with some noise in the form of spurious irrelevant screens. IPM is a depth-first algorithm, while IPM2 is a breadth-first

• algorithm. They require defining a criterion for interesting

patterns in order to use it for deciding if a pattern is worthy of reporting or not. The criterion includes the pattern's minimum length, minimum number

• f
• ccurrences (support), the maximum number of insertion

errors allowed in any instance of this pattern and minimum score. Given a pattern p, the scoring function is: score (p) = log2 |p| * log2 support(p) * density(p) where |p| is the length of p, support(p) is the support of p and density(p) is the ratio of |p| to the average length of the instances of p. Since these instances may include noise, they can be longer than the pattern. For example, the segments {2,4,3,4}, {2,4,3,2,4} and {2,3,4} are instances of the pattern {2,3,4} with at most 2 insertions, where 2, 3 and 4 are the IDs given to the legacy screens by LeNDI. The average density of this pattern is 0.75. IPM and IPM2 discover all maximal patterns in the recorded interaction traces that meet the user’s interestingness criterion. A maximal pattern is not a sub-pattern of another one with the same support. After reviewing the discovered patterns, the user decides which

• nes correspond fully or partially to real frequent user

tasks and are not spurious repeated navigations. The instances of each user task are used to build its task model. T3: Task Modeling. Mathaino [ 16, 17] is another prototype tool of the CelLEST environment. It accomplishes a reverse engineering task, T3, and two forward engineering tasks, T4 and T5. Mathaino uses intermediate platform-independent GUI representations to support simultaneous legacy UI migration to multiple

• platforms. In T3, the instances of each user task

discovered during T2 are analyzed comparatively to construct an abstract model of:

• 1. the navigational sequence through the legacy system UI

to perform the user task,

• 2. the types of the data entered by the user through the

navigation (and of the data displayed to him/her), and the locations where they occur on the legacy screens,

• 3. the domain of values of the inputs, and
• 4. the interdependencies among these values.

To analyze and model the information exchange during the user task, examples of the user inputs and outputs are

• needed. User inputs are already recorded in the interaction
• traces. But the trace does not record any evidence of the
• utput the user retrieves from each snapshot in the task.

So, an expert user needs to manually highlight on the snapshots of the task instances the areas that contain the

• utputs extracted to successfully complete the task.

Given the annotated task instances, Mathaino analyzes the flow of information to and from the legacy system to identify the user inputs required to accomplish his task, by studying all the recorded instances of this task. It compares the values used for each input field across all the task instances, and the values of all input and output

SLIDE 10

fields in the same task instance. Then, each data input field is labeled as

• 1. Constant, whose value is the same in all task instances;
• 2. Derived, whose value is obtained earlier in the task

from an output field and is used as input later;

• 3. Redundant, Whose value is inputed multiple times in

the same task;

• 4. Range, whose value is always one of a limited set of

values; and

• 5. Unpredictable, whose value is independently supplied

by the user. Categorizing input fields leads to a significant reduction in the user input required by the reengineered UI of the task, e.g., the user will not need to input a derived input as it will be supplied automatically. Also, it helps choosing the proper abstract widget type for each input field in task

• T4. The CelLEST user may inspect the identified pieces
• f information and name them with meaningful names

Additionally, based on a comparative analysis of all instances of the same data field, Mathaino infers the coordinates of this field on the legacy screen it belongs to, in case they are fixed. In dynamic screens, such as free- forms, Mathaino attempts to discover starting and/or terminating landmarks to use for locating the data field. Finally, if alternative paths exist for a user task or subtask, the branching screens are identified and each alternative path is analyzed as described above. At runtime, the signature of the snapshot received as a result

• f performing an action on an instance of the branching

screen decides which path to follow. The task model produced in T3 specifies the path on the interface state-transition model through which the user navigates, the flow of information between the legacy application and the user, and the syntax of the interactions through which the information is exchanged. Effectively, it constitutes a declarative and executable specification of the modeled task of the legacy application. Given values for all the “unpredictable” pieces of information identified, the model can be used to drive the legacy application and execute the modeled task.

5.3 T4: Generation of Abstract GUI Specification

Mathaino uses model-based UI design heuristics to automatically produce abstract specifications for the new reengineered UI of each task, using its model. Thus, it eliminates the need for the current manual practice of piece-by-piece mapping of the task model to a GUI

• design. The specification is described in terms of a set of

abstract forms; each corresponds to a set of screens of the legacy system. Mathaino ensures that all the output fields identified in the task model are displayed on one of the

• forms. For each abstract form, a related plan for navigating

through the legacy screens at runtime is produced. Using model-based user-interface design heuristics, Mathaino proposes an abstract widget for each input or

• utput field. For example, a field with an enumerated

range of values is represented by a combo box or a set of radio buttons depending on the number of values and an "unpredictable" variable is represented by a textbox. Then, the widgets are laid out on the form in a tabular

• manner. The user can override the default choices of

widgets and layout settings. For example, s/he can change the widget type issued for a field or the number of layout

• columns. After applying user feedback, an XML

representation of the abstract specifications is produced.

5.4 T5: Runtime GUI Generation

The CelLEST runtime environment consists of two

• components. The front-end component is the runtime
• interpreter. It is responsible for interpreting the XML

abstract GUI forms on a specific platform. It supplies widgets in the target platform that most closely match the abstract input widgets. Currently, an XHTML interpreter (for Web-enabling) and a WML interpreter (for WAP- enabling) are available. [ 17] The back-end component is the host navigator, built

• ver a host-access middleware. The host navigator

executes the navigation plans of the abstract forms and passes the inputs to the legacy system and collects back the outputs. But first, the XHTML or WML interpreter passes the plan details to the host navigator in a platform- independent format. The XHTML Interpreter. For Web-enabling, the XHTML interpreter dynamically parses the XML abstract GUI forms at runtime and translates them to XHTML CGI forms. It maps the abstract GUI widgets to the appropriate CGI widgets. It uses XHTML tables to layout the produced Web page in the closest format to that chosen by the user for the abstract GUI. Also, it parses the CGI response produced by the client Web browser into the platform-independent form needed by the back-end host navigator. It is a server-side component that runs as a Java servlet on the Web server. The WML Interpreter. WML is markup language for rendering Web pages on WAP-enabled mobile Internet devices like Cellular phones and Personal Digital

• Assistants. A Web page in WML (also called WML deck)

is limited to a maximum of 1200 bytes. To overcome the device display limitation, a deck can be divided into a number of cards. The device can display only one card at a time. The only input widgets supported by WMP are simple text boxes. WML does not support CGI but provides some features that can simulate CGI. The WML interpreter is adjusted to deal with these

• constraints. E.g., it implements an abstract GUI form

using several WML decks if 1200 bytes are not enough. It uses a numerical menu to represent "range" input fields. It internally caches the user responses to the multiple decks corresponding to a single abstract GUI form, before submitting them to the host navigator.

• 6. Reflections and Conclusions

The CelLEST method automates the screen-mapping Web-enabling approach, which has been widely used in

SLIDE 11

the software industry. Like all methods for Web-enabling via presentation access, its applicability is threatened by that fact that it cannot substantially extend the functionality delivered by the legacy application. But, an examination of the actual Web-enabling problems, reported in [3], shows that this is not a limitation in practice, since presentation- based integrations constitute the preferred approach in the majority of these cases. Therefore, the automation that CelLEST brings to this widespread practice is a substantial innovation, with a potential impact. On the other hand, the effectiveness of the method depends on the “quality” of the recorded traces and can potentially suffer when the traces are not sufficiently representative of the whole legacy UI behavior or the user tasks of interest. This potential shortcoming can be addressed with a methodological argument: given that the recording middleware is unintrusive to the legacy users, it can remain installed and recording through long periods

• f time and on many legacy terminals. Thus, the danger of

“missing” services of interest can become minimal. We have evaluated the CelLEST environment with several case studies. Our experience shows that the method is indeed lightweight in terms of the skills it requires – CelLEST users were able to effectively model a legacy system after a short tutorial. In addition, for small- to medium-size legacy applications, with approximately 30 distinct legacy screens, the automatically constructed model is 92 to 98% accurate, requiring a few corrections by the user. [ 11, 16, 22, 24] In conclusion, we are confident that the CelLEST method constitutes an advancement of the state-of-the-art in Web-enabling legacy systems and that more artificial- intelligence results will be applied to other reengineering and integration challenges to provide novel automated solutions to them.

Acknowledgments

We acknowledge the effort of CelLEST project team: Bruce Matichuk, Roland Penner, Paul Iglinski and Rohit Kapoor. This work was supported by a collaborative research and development grant, NSERC 215451-98 and a kind contribution from Celcorp.

References

• 1. Akers, L. Web-enabling legacy applications – an overview for

VSE users. VSE/ESA Software Newsletter, IBM, Third/Fourth Quarter, 2000.

• 2. Ambler,

S. Legacy Integration Techniques for Java Applications: How to Reuse Your Legacy Investments within Java applications. IBM developerWorks, Nov. 2000.

• 3. Attachmate, Repurposing Legacy Applications for the Web:

Screen-Based Access in Perspective. A White Paper, Attachmate Corporation, Oct. 2000

• 4. Berman, D. and Bregar, K. Don't Replace -- Extend: Why

Leveraging Your Legacy Systems Is the Way to Go. Enterprise Systems, June 2001.

• 5. Braswell, B., Forshay, G. and Martinez, J. IBM Web-to-Host

Integration Solutions. Redbooks Series, IBM, Jan. 2002.

• 6. Canfora, G., Cimitile, A., De Lucia, A. and, Di Lucca, G.

Decomposing Legacy Systems into Objects: An Eclectic

• Approach. Information & Software Technology 43(6), 401-412,

2001.

• 7. Celcorp. CelEngineer User’s Guide – Evaluation Version 2.0.

Celcorp, 1999.

• 8. Chadha, R. Integration of Web with Legacy Systems Through

Java Applets and Distributed Objects, Workshop

• n

Compositional Software Architectures, 1998.

• 9. Comella-Dorda, S., Wallnau, K., Seacord, R. and Robert, J. A

Survey of Legacy System Modernization Approaches. Technical Note CMU/SEI-2000-TN-003, SEI, 2000.

• 10. Crigler, R. Use Screen Mapping For Wireless Access to Legacy

Enterprise Data. Enterprise Application Integration (EAI) Journal, Aug. 2001

• 11. El-Ramly, M., Iglinski, P., Stroulia, E., Sorenson, P. and

Matichuk, B. Modeling the System-User Dialog Using Interaction Traces. In Proc. 8th Working Conf. on Reverse Engineering (WCRE’8), 208-217, 2001.

• 12. El-Ramly, M., Stroulia E. and Sorenson, P., Recovering

Software Requirements from System-user Interaction Traces, In

• Proc. 14th Int. Conf. on Software Engineering and Knowledge

Engineering (SEKE’02), 2002.

• 13. El-Ramly, M., Stroulia E. and Sorenson, P. Interaction-Pattern

Mining: Extracting Usage Scenarios from Run-time Behavior

• Traces. In Proc. 8th Int. Conf. on Knowledge Discovery and

Data Mining (KDD 2002), 2002.

• 14. Howe, D. (Editor). The Free On-line Dictionary of Computing.

www.foldoc.org

• 15. IBM. Screen Customizer Version 2.0.60: Getting Started. IBM,

1999.

• 16. Kapoor, R. Device-Retargetable User Interface Reengineering

Using XML. Dept. of Computing Science, University of Alberta,

• Tech. Report TR01-11, Aug. 2001.
• 17. Kapoor, R. and Stroulia, E. Simultaneous Legacy Interface

Migration to Multiple Platforms. In Proc. 9th Int. Conf. on Human-Computer Interaction, Lawrence Erlbaum Associates,

• vol. 1, 51-55, Aug. 2001.
• 18. Langan, G. From Legacy to the Web. Enterprise Application

Integration (EAI) Journal, Jan. 2000.

• 19. Liu, Zheng-Yang, Mike Ballantyne, and Lee Seward, An

Assistant for Re-Engineering Legacy systems, In Proc. 6th Innovative Applications of Artificial Intelligence Conf., 95-102, 1994.

• 20. Parnas, D. Software Aging. in Proc. 16th Int. Conf. on Software

Engineering (ICSE'94), 279-287, 1994.

• 21. Ruh, W. A., Maginnis, F. X. and Brown, W. J. Types of
• Integration. In “Enterprise Application Integration: A Wiley

Tech Brief”, John Wiley & Sons, Oct. 2000.

• 22. Stroulia, E., El-Ramly, M., Iglinski P. and Sorenson, P. User

Interface Reverse Engineering in Support of Interface Migration to the Web, Automated Software Engineering, 3(10), 271-301, 2003.

• 23. Stroulia, E., El-Ramly, M., Kong, L., Sorenson, P. and

Matichuk, B. Reverse Engineering Legacy Interfaces: An Interaction-Driven Approach. In Proc. 6th Working Conf. on Reverse Engineering, 292-302, 1999.

• 24. Stroulia, E., El-Ramly M. and Sorenson, P. From Legacy to

Web through Interaction Modeling, In Proc. Int. Conf. on Software Maintenance (ICSM), 2002.

• 25. Sneed, H. M. Accessing Legacy Mainframe Applications via the
• Internet. In Proc. 2nd Int. Workshop on Web Site Evolution

(WSE’2000), 2000.

• 26. Tan, Y., Lindquist, D., Rowe, T. and Hind, J. IBM eNetwork

Host On-Demand: The Beginning of a New Era for Accessing Host Information in a Web Environment. IBM System J., 37(1), 133-151, 1998.

• 27. Visaggio, G., Ageing of a Data Intensive Legacy System:

Symptoms and Remedies, Journal of Software Maintenance and Evolution, 15(3), 281-308, 2001.

• 28. Yample, T. Web-based Technologies for User Interface
• Rejuvenation. In “Web-to-Host Connectivity”, Editors: Guruge,
• A. and Lindgren, L., CRC Press, 185-197, 2000.
• 29. Zou, Y. and Kontogiannis, K. Enabling Technologies for Web-

Based Legacy System Integration, In Proc. of 1st Int. Workshop

• n Web Site Evolution (WSE’99), 1999.

View publication stats View publication stats