SLIDE 1 P R E S E N T A T I O N International Conference On Software Testing Analysis & Review October 29-November 2, 2001 San Jose, CA USA Friday, November 2, 2001 11:15 AM
EVOLUTION OF AUTOMATED TESTING FOR ENTERPRISE SYSTEMS
Cherie Coles
BNSF Railroad
F8
Presentation Bios
SLIDE 2
Evolution of Automated Testing for Enterprise Systems
BNSF BNSF
SLIDE 3 Automation Time Line
Productive Flexible Repeatable Robust Reliable Consistent
Time
Automated Solutions Methodology
SLIDE 4
About BNSF
$8 billion annual revenues $24 million capital 33,500 miles of track, 29 states & 2 Canadian provinces 45,000 employees across the western United States and Canada 5000 locomotives 200,000 freight cars
SLIDE 5
About BNSF - IT Architecture
Largest sysplexed IBM 9X2 mainframe network Over 2,700 MIPS (millions of instructions per second) Over 300 AIX and NT servers 10,000 terminals 15,000 PC devices Telecom Network equivalent to largest phone companies Over 100 Line-of-Business applications
SLIDE 6 About BNSF
Security Analysts
“...the most advanced and sophisticated railroad
technology…stock price up 20% from current levels.”
“In short, the technology demonstrations increased
- ur confidence in the company’s ability to continue to
reduce costs and improve productivity…we believe BNI will continue to increase productivity…through leveraging it’s sophisticated technology.”
“…very impressed with the company’s use of
technology, which allows it to do more with less…the company is now in a position to reap the benefits of the investment.”
SLIDE 7 Application Interface
- Maintenance
- Crews
- Equipment
Core Business
- Logistics
- Safety
- Trains & Track
- Government Reporting
- Human Resources
- Financial
Business
- Labor Relations
- E-commerce
- Movement
- Equipment
Customer
QA
SLIDE 8
The QA Team
16 Team Members Over 300 years business experience combined One member with development background All have business knowledge, technical aptitude, and motivation
Good Tools + Great People = Efficient and Effective Product Evaluation
SLIDE 9
What We Test
Main Frame Core business systems Web / JAVA - e-Commerce, Customer Interface Client / Server - Customer, Logistics, Yard Activity Infrastructure - Load, Performance
SLIDE 10
Testing Infrastructure
NT Integrated Test Lab (isolated domain) - 1998 Mainframe-Controlled Test Environment - 2001 Mid-tier (AIX) Server (4th quarter 2001)
SLIDE 11
QA STATS
Manual Testing
Monthly Average YTD:
350 Tests Completed 200 Packages 1100 Modules
SLIDE 12
QA STATS
Automated Testing
300 timekeeping test scripts run each night
(equivalent to 8 staff days manually)
25,000 voice files converted in one week
(equivalent to 3 staff months manually)
Database performance test takes 4 hours set up
and run (equivalent to “an army” of people banging on their keyboards manually)
Automated test data set up takes 1 hour
(equivalent to 1 or more days manually)
SLIDE 13
Benefits
Increased productivity for TS and Customers Better resource utilization - unattended after
hours of testing
Early involvement, smoother implementation,
decreased cost
Greater knowledge transfer
SLIDE 14
The Journey
Fully automated and unattended Application-driven - Object Oriented Data-driven - Dynamic SQL Data-driven - Spreadsheets Hard-coded scripts for specific use Record-and-playback Manual Leveraging automation
SLIDE 15 Record-and-Playback: Consistent “Macros”
Y2K Script 3
Type Car Init1 Type Car No1 Type Track1 Type Station1 Type Waybill1 Type Waybill1a Type Waybill1b
Repeatable Application Tests Y2K Script 2
Type Car Init2 Type Car No2 Type Track2 Type Station2 Type Waybill2 Type Waybill2a Type Waybill2b Type Car Init3 Type Car No3 Type Track3 Type Station3 Type Waybill3 Type Waybill3a Type Waybill3b
Y2K Script 1
SLIDE 16 Hard-Coded Scripts
Test Script 3
Type Car Init1 Type Car No1 Type Track1 Type Station1 Type Waybill1 Type Waybill1a Type Waybill1b
Repeatable Scenarios Test Script 2
Type Car Init2 Type Car No2 Type Track2 Type Station2 Type Waybill2 Type Waybill2a Type Waybill2b Type Car Init3 Type Car No3 Type Track3 Type Station3 Type Waybill3 Type Waybill3a Type Waybill3b
Test Script 1
SLIDE 17 Automated Solutions Data-Driven
100 Test Cases with Required Data Elements
20 40 60 80 100 120 Cars Tracks Stations Waybills
Application-specific Test Script Works with all Test Cases
Enter TestData Type {“RtCtrl”} Type CarInit Type CarNo Type Track Type Station Type Waybill1 Type Waybill2 Type Waybill3
Application
SLIDE 18 Automated Solutions OO Testing Methodology
Test Case Scenarios Application A
20 40 60 80 100 120 Cars Tracks Stations Waybills 20 40 60 80 100 120 Cars Tracks Stations Waybills
Test Case Scenarios Application B Test Case Scenarios Application C
20 40 60 80 100 120 Cars Tracks Stations Waybills 20 40 60 80 100 120 Cars Tracks Stations Waybills
Test Case Scenarios Application D
Enter TestData Type {“RtCtrl”} Type CarInit Type CarNo Type Track Type Station Type Waybill1 Type Waybill2 Type Waybill3
Robust Test Script works with all Test Case Scenarios across applications Applications A, B, C, D….
SLIDE 19 Automated Solutions Data Base (On the Horizon)
Test Script Database: Applications A, B, C, D... Applications A, B, C, D... Application Database
SQL
Test Case Scenario Database: Applications A, B, C, D...
SQL
SLIDE 20 Keys to Automation Success - Have a Vision!
- Find a business driver
- Develop team partnership
- Stabilize application design
- Access to business knowledge
- Recognize and plan for learning curve
- Think out-of-the-box
- Match business knowledge with technical skill
- Continue skill development
- Build from the diversity of the team
SLIDE 21
Evolution of Automated Testing for Enterprise Systems To be continued...
BNSF BNSF
SLIDE 22 Mike Amos Mike Amos is a QA Engineer for BNSF. Mike has held many positions on the Canadian National and BNSF Railroads. Currently Mike is Technology Services, Project Manager
- f Quality Assurance for Major Application Development Projects at BNSF.
mike.amos@bnsf.com Pete Sidoti Pete Sidoti is a QA Engineer for BNSF. Pete has had many roles in 20 years with the BNSF from civil engineering to information technology. The last 5 years have been in the QA field. Pete received his MBA from Baker University in 1986. Pete is Project Manager of Technology Services, QA Infrastructure for BNSF. pete.sidoti@bnsf.com Cherie Coles Cherie Coles is manager for BNSF Technology Services, Quality Assurance. She has ten years experience in IT working in Infrastructure Engineering and Application
- Development. Prior to her career with BNSF she graduated from University of Texas at
Arlington, with a degree in Business Administration. She earned her CPA while at ARCO Oil and Gas Company in the Systems Management Organization. cherie.coles@bnsf.com
