The Ten Most Important Automation The Ten Most Important Automation - - PDF document

T3

Concurrent Session

Thursday 10/25/2007 9:45 AM JUMP TO: Biographical Information The Presentation

The Ten Most Important Automation The Ten Most Important Automation Questions—and Answers Questions—and Answers

Presented by: Mukesh Mulchandani and Krishna Iyer, ZenTEST Labs

Presented at: The International Conference on Software Testing Analysis and Review October 22-26, 2007; Anaheim, CA, USA 330 Corporate Way, Suite 300 , Orange Park, FL 32043 888-268-8770 904-278-0524 sqeinfo@sqe.com www.sqe.com

Mukesh Mulchandani Mukesh Mulchandani

Mukesh Mulchandani (co-author), is the CTO of ZenTEST Labs, an independent software testing solutions company based out of India. Mukesh has 7 years of work experience in Software Testing and automation consulting. Mukesh has experience in servicing outsourced testing having worked on test automaton projects with clients such as S1 Corporation, Kanbay, Capgemini Consulting and some of the Fortune 500 clients like Morgan Stanley and Merrill Lynch. He has played a major role in designing functional automation processes at company level. Mukesh has also provided testing consulting services to 2 of the top 5 IT consulting companies in the world. Mukesh is a certified Mercury Product consultant for WinRunner and QTP and also a Certified Software Test Engineer from QAI USA. Mukesh has co- authored whitepaper for various conference including STAREAST and STARWEST

Krishna Iyer Krishna Iyer

Krishna Iyer is the CEO of ZenTEST Labs, an independent software testing solutions company based out of India. Krishna has 9 years of Quality management and Process consulting experience. Krishna has experience in test outsourcing of large multi-site projects having worked with clients such as GE, CitiFinancial, IBM, HSBC, etc both in India and the United States. He also has participated in CMM assessments and has been one of the key players in taking a large software integrator to CMMi Level 5. Krishna is a qualified accountant by education and a certified software quality analyst as

• well. He is a prolific speaker and writer and has presented in various conferences

including STARWEST and STAREAST. Krishna has made various presentations on automation frameworks.

The Ten Most Important Automation Questions and Answers

Krishna Iyer and Mukesh Mulchandani

StarWEST 2007

2 StartWest 2007

Objective of this presentations

To present choices every test automation

project must exercise (the first 5 questions)

To project future trends in automation

(the last 5 questions)

To discuss unique concepts such as

vertical vs. horizontal automation

To demonstrate advanced automation

concepts

To share ideas for increasing ROI of your

automation project based on ZenTEST Labs’ experience

3 StartWest 2007

The ten most important automation questions

Five Choices

1. To automate now or automate later? (the time choice) 2. To automate this or automate that? (the test case choice) 3. Vertical Automation or Horizontal Automation? (the flow choice) 4. Test data hard coded or Test data kept reusable? (the obj ect

• riented choice)

5. How To handle dynamic obj ects? (the obj ect map choice)

Five Trends

1. Keyword or functional decomposition? (The combined trend) 2. Building extensibility or not? (The user trend) 3. Whether to introduce automated test driven development (The ‘ time to market’ trend) 4. Whether to automate and execute early (The ‘ unit test’ trend) 5. Whether to offer automation scripts to the client (the selling trend)

4 StartWest 2007

Q#1: To automate now or automate later…

▲Key to successful test automation is knowing

WHEN to automate.

▲Often overlooked and sometimes under

valued.

▲Even best automation approach fails to give

higher ROI because of wrong timing of test

• automation. As it is

Pointless if it is at the fag end of a product

lifecycle

Painful if the product is unstable

5 StartWest 2007

Q#1: To automate now or automate later… contd.

Most important factors to be considered

before starting automation.

Is the application/module that is planned for automation stable enough for automation. Is manual testing process in place Detailed test conditions and pre conditions. Accurate Test data and expected result.

When your sure of ROI paying off

6 StartWest 2007

Q#1: To automate now or automate later… contd. What defines “Stable Enough”

Core functionality & navigation flow is

approved and accepted by end client.

No critical defects to be fixed around the

functionality which will change the application from its current state.

No planned major enhancements in the

functionality for min next 3 regression rounds.

7 StartWest 2007

Q#2: To automate this or automate that

After knowing when to automate, its

critical to know what to automate and what not to automate.

Remember: Its not feasible to automate

all the test cases.

Clear test cases selection criterion

should be available with test automation team to improve effectiveness of automation suite.

8 StartWest 2007

Q#2: Test Case Selection Criterion

▲Is the test case repeatable? ▲Does this test case require manual intervention? ▲Has the test case passed manual verification? ▲Are all the preconditions for the test case taken care-of? ▲Are the execution steps very clear? ▲Do we have test data for this test case? ▲Is the expected result clear enough to decide the test case

status (PASS & FAIL)?

▲Will the test case survive the functional changes around it? ▲Is the test case straightforward for automation? ▲Can I trust this script to really test this part of the feature?

9 StartWest 2007

Q#3: Vertical OR Horizontal Automation.

Automating End to End Flow where each test cases is independent of each other such as

Create payment2 Create payment1 Reject payment2 Approve payment1

TC1&2: Login(user1)

Logout->Login(user2)

Horizontal Automation

Vertical Automation

Create payment1 Approve payment1

Logout->Login(user2) TC1: Login(user1)

Create payment2 Reject payment1

Logout->Login(user2) TC2: Login(user1)

Vertical automation is executing test cases feature wise to save execution time. e.g.

10 StartWest 2007

Q#3: Vertical OR Horizontal Automation.

Advantages of Horizontal Automation

▲ Flexibility in running any test case as each test case is

independent of other.

▲ Easy to organize and automate. ▲ Test log is created the movement each test case

execution is over Advantages of Vertical Automation

▲ Faster test execution as navigation is minimized, (such

as login -logout)

▲ Effective when same functional flow is tested with

different data sets

11 StartWest 2007

Q#4: Test data hard coded or test data kept reusable

▲An good automation architect ensures that

scripts are reusable.

▲And a great automation architect ensures

that scripts and test data, both, are reusable.

12 StartWest 2007

▲ Generally , there is more emphasis on reusing test script

than reusing test data.

▲ Generally one test set is used per test cases. For e.g.

approve payment test case and reject payment test case may use same user (i.e. user1) but the test data sheet is stored within the test case and gets repeated again and again for each test case.

▲ On the other hand to keep the test data reusable: Store data screen wise and not test case wise Assign reference id to each test data. Pass reference id to each test case for accessing the

test data.

Q#4: Test data hard coded or test data kept reusable

13 StartWest 2007

Q#5: How to handle dynamic objects; object map

• r advance scripting?

▲ Tip1. Store different unique properties than what is normally stored.

e.g. if a button label is dynamically changing again and again, then store button class and msw_id instead of storing class and label.

▲ Tip2. Don’t store the object in object map/ repository. Create the

• bject dynamically through scripting and use it.

▲ When to use advance scripting for dynamic objects. When objects are dynamically changing. When similar types of objects are available with minor logical

• differences. e.g. pagination, where list of records are displayed
• n diff pages, links to these pages are similar objects. The page

number is different and gets incremented by 1. Instead of storing all the 100 page links, here the engineer needs to store one

• bject and change the page number property dynamically, this

reduces the object count in the repository, which saves lot of RAM.

14 StartWest 2007

Q#6: Keyword driven or Functional decomposition approach Keyword driven (KD)

In the Keyword driven approach, each business process is mapped into actions and further each operation is mapped as a keyword. It is easy for non technical users to create test scenarios without knowing much of the testing

• tool. Scripts are not modular

and major advantages of functional decomposition are lost.

Functional decomposition (FD)

In the functional decomposition approach, business processes are created first and while creating the test scripts, each business process is called in a sequence. This approach is modular but for every test case, a test script is

• required. In this approach

scripts are maintainable to the extent that implementation of the business process is not changed.

15 StartWest 2007

Q#6: Keyword driven or Functional decomposition approach Keyword driven

Keyword calling

EnterText-

username, mukesh

EnterText- password, hello Click

OK

ClickLink

create payment

EnterText account_name, 1209892 EnterText amount 122$ EnterText date 04-Aug-2007

. . . so on

Functional decomposition

Library Login() { Enter username Enter password Click ok }

• Test Script for approval

login () create_payment () logout () login () approve_payment()

16 StartWest 2007

Q#6: Keyword driven or Functional decomposition approach

Important decision criteria to decide when to use functional decomposition and when to use keyword driven framework.

1.

Who has to maintain the scripts: If Technical team is available for maintaining then use FD. If Non technical available for maintaining then use KD.

2.

When to automate: IF script creation has to happen before the app development is in progress then us FD.

3.

Maintenance effort of test data & expected results: If the application is very big and effort is high in maintaining test data and expected results then use KD else FD

4.

Availability of good architecture: For KD, without a good architecture in place these scripts have a much higher risk of

• failure. Use FD if good architecture is not available. For KD,

building the core utility scripts requires a very degree of proficiency in the testing tool being used.

17 StartWest 2007

Q#6: Keyword driven or Functional decomposition approach

Moving beyond FD and KD. The trend shall be to

combine both

Where keywords are set of business processes,

which are packaged as user defined functions. These keywords/user defined functions are called in a sequence in excel/database to test the business rules.

The modularity of functional decomposition and

the usability of keyword driven

Various frameworks including ZenTEST’s

ZenFRAME support the same.

18 StartWest 2007

Q#7: Building extensibility or not

• To further make automation framework more effective,

attempt should be to made to separate technical and business usage. Business users should be able to add test scenarios using the framework without worrying about the technical scripting part.

• Minimum, framework should be able to easily handle

functional changes in the application such as:

• Addition of new fields/objects
• Removal of new fields/objects
• Changes in business validations, etc

19 StartWest 2007

Q#7: Building extensibility or not

Basic theme behind building an extensible

automation framework should be to

“Keep everything, that changes or has chances of changing,

separate from the script.”

• 1. Object properties changes so keep that separate, this is default

feature in most of the tools.

• 2. Test data changes, so keep test data separate from script
• 3. Sequence/ flow of application changes so separate that in excel or

database (Most of us follow the first 2 points as more or less they are default feature in the automation tools, now lets take advantage of the third approach)

20 StartWest 2007

Q#8 : Whether to introduce automated test driven development

Automated Test Driven Development

• Practices in the adj acent

table, but applied to automated test driven development

• In other words, proj ect

Managers could decide to make the application or product being built automation friendly.

Test Driven Development

• The process of writing software from

the outside in

• A method for breaking a problem

into bite-sized pieces - “ test, code, design, repeat”

• Tests drive or dictate the code that

is developed

• “ Before you write code, think about

what it will do. Think about a block of code from the perspective of the user

• f that code.”
• Write a test that shows what you

want the code to do before you write the code

21 StartWest 2007

Q#9 : Whether to automate and execute automation scripts in the early stages of development

Agile Test Automation (Automation during development) Principles by James Bach

Consider thinking of test automation as …Any use of tools to

support testing (James Bach)

Test automation means tool support for all aspects of a test

project, not just test execution.

Test automation progresses when supported by dedicated

programmers (toolsmiths).

Toolsmiths are directed by testers. Test toolsmiths gather and apply a wide variety of tools to

support testing.

Test toolsmiths advocate for testability features and produce

tools that exploit those features.

Test automation is organized to fulfill short term goals. Long term test automation tasks are avoided in the absence

• f specific approval based on a compelling business case.

22 StartWest 2007

Q#10: Whether to offer your automation scripts to your end client.

Ideally automation should start once the application is

stable but with good technical QA engineers one can start much before the application is even ready for manual testing. This can be achieved by

Using abstract automation approach for building the automation

flow and important components.

Manually scripting application objects and user actions.

There is high dependence on good requirements and

screen layout.

Traditional automation is mainly used by QA engineers

for regression testing, but latest trends show that the Automation suite can yield more returns when developers can use it for their unit testing.

23 StartWest 2007

Choice # 10: Whether to offer your automation scripts to your end client.

Recent Trends indicate that soon Test scripts will also be

shipped along with the application.

Every patch release will have the modified test scripts

with it.

Clients have started demanding automation script for

doing their UAT and testing application during bug fixes.

It has also become a new source of income for product

companies, wherein additional cost can be billed for automation scripts which are anyways available with QA team.

This also brings in additional budget for future

automation effort and automation gets buy-in from senior management.

24 StartWest 2007

Bibliography

When should a test be automated?

Article by Brian Marick, 1998.

When to automate testing

David Weiss Blog

Agile Test Automation

White paper by James Bach

krishna@zentestlabs.com mukesh@zentestlabs.com

Thank You

Krishna Iyer Mukesh Mulchandani blog:http://www.zentest.typepad.com