Today's software testing world is being revolutionized by the emergence of very powerful automated testing tools. These tools, when used properly, can deliver very astonishing results. they are tireless, executing suites of test procedures hour after hour with remarkable accuracy. These tools promise engineers and managers alike more accurate testing, more effective defect detection and an overall increase in productivity. Too good to be true? Not necessarily.

There is a certain amount of misunderstanding of the capabilities of test automation tools and how to implement them. On the surface, these tools are best known for their ability to record and play back keystrokes and mouse movements. These creates the impression in many peoples mind that all they have to do to automate a test process is to take an existing manual test procedure, execute it while recording it with a tool which then will allow for its successful playback time and time again. While the basic principle of the previous statement is true and simple, the reality of performing the task is far from it. Many companies will purchase the tools, spend resources trying to implement them, get highly frustrated and finally abandon their use. Why would they do that? Lets take a look at the most common reasons why the implementation of automated software testing tools fail and how to avoid the pitfalls:

As with most other undertakings, lack of planning is the number one reason why the successful deployment of test automation fails. What makes the planning of software testing more difficult than other types of planning is the poor understanding that many project managers and developers have about testing tools. The time allotted for testing manually is not accurately reflected in many schedules. Most testers will agree that, unfortunately, the time allowed for testing an application is too short. But when testing tools enter the picture, specially for the first time, and the schedule is not adjusted for their proper implementation, the results are certain to be disastrous; bringing about an unfair, bad reputation to the test tool being employed.

So, how much time is adequate for the deployment of test tools? Unfortunately , there is no simple answer to this question. There are many factors that enter into the picture of successfully deploying test automation. Among these factor are: How GUI intensive is the application? Are there many timing issues involved? Is there a core group of people assigned to the automation process? Are these people properly trained? and the list continues.

The best place to start successful planning is by involving both the software developers and the testers in the planning. It is imperative that the people involved in the test effort have a very good understanding of the general principles of testing. There should also be at least one person that has been trained in the use of the test tool and processes a good grasp of its implementation. It is duly to note that if this last element is missing there is bound to be problems because there is a big difference between simply understanding test principles and knowing how a test tool will behave when handling different situations.

The lack of resources issue addresses hardware and personnel concerns which include; quantity and quality of human resources, dedicated test machines, appropriate hardware, etc.

These tools shift the focus of the human aspect from that of repeating a series of tedious tasks over and over to the more creative effort of innovative test procedure development and maintenance methods. These shift of focus brings about a shift in skill level as well. Testing software has often been thought of as a repetitive but simple task that can be performed by just about anybody. People in the business understand that this statement is far from being true. Furthermore, the effective deployment of automation in a client-server, windows environment proves to require a degree of skill equal to that needed by software developers.

LACK OF UNDERSTANDING OF THE TOOL

Anybody that has seen a demonstration of the record and playback ability of a test automation tool probably get the same ides: "my problems are over, I have found an easy way to test my application". After all, it appears to be as simple as pressing the keys you would normally press during the normal course of testing, record them, play them back and presto...you are done. Well, while the appearance definitely points in that direction the reality is not quite as simple. This fact is normally learned after the first few attempts at playing back a set of newly recorded procedures.

There are many factors that affect the proper play-back of a test procedure, among them we have ; timing issues, did the server respond as quick as it did when the procedure was initially recorded? Recording environment, Is the procedure beginning and ending in the right area? Security issues, should passwords be hardcoded or not? These are just an example of some of the issues that arise during the test automation phase demonstrating the complexity of the effort. In addition, these automation tools often have an array of special functions that may not be easy to understand or to be aware of their existence. It is highly recommended that users of the tool get the proper training necessary to employ the tool to its full potential. Keep in mind that developing test automation scripts is very much like developing software, hence, the principles that would apply to software development generally apply to test script development.

UNREALISTIC SCHEDULES

Have you ever had to stay at work long hours to complete a task? Ever had to wait after hours for the final build so that it can go through formal verification? Unrealistic schedules are a factor that have plagued the software industry since its inception and, unfortunately, the bad practices that create these factor still with us today. The problem gets compounded when test automation enters the picture because it is difficult to assess the time it will take to automate if metrics are not at the disposal of the scheduler. Companies still fail to understand that initially it will take longer to test an application using an automated tool than it does manually. Even if they understand the principle, the schedules often do not reflect the facts. If the proper amount of time and resources is not made available the automation effort will definitely fail and imperil the manual testing as well. It is necessary in many cases to have a manual test effort and an automation test effort going simultaneously.

POOR CODING STANDARDS

Developing automated test procedures is not much different than developing code. Many of the test tools come with visual-basic-like scripting languages which make the tools highly versatile and complex. Just as in the case of software development, it is necessary to define how values will be passed, how subroutines will be named, which functions will be employed and which ones are off limits, etc.

A great deal of testing tasks can be accomplished by creating and employing extended scripts. However, if a strict set of coding standards is not developed prior to implementation, the results can lead to the loss of control of the automation effort. For example; a certain application-under-test may be tested by using 100 test procedures developed in NT 3.51. Shortly after the procedures are completed, NT 4.0 is released to the market and the application-under-test is now required to support the new version of the operating system. Even if the test tool supports the new version and it is backwards compatible with the previous version, problems will arise with test procedures that require the use of the file manager/windows explorer application because they are very different from one version of operating system to the other.

A strong set of coding guidelines will greatly minimize the amount of rework to the test procedures caused by any changes in the application-under-test, wheter the change involves the use of a different operating system or simply the release of the next build/iteration of the application in question.

There are many challenges to be faced while implementing a test automation tool. The key to achieving success is to remain flexible and open minded. Test automation tools can greatly simplify the verification effort by taking over the execution of repetitive tasks, allowing test engineers to concentrate on more important tasks.