Is Test Automation a Silver Bullet…?

How can we make test teams more productive?

My first exposure to automated testing, was a text book example of how NOT to implement test automation.

I was working for a company where there were five us on the software testing team. We all worked on a minimum of 3 projects each, simultaneously, and because we were dealing with ongoing software and data updates (automotive repair information) we were all way over allocated.

Upper management was trying to figure out how to increase throughput and a PM came up with a genius idea.

Test Automation.

They believed this approach would dramatically reduce testing time, and allow the test team to take on more projects at the same time.

A sales team showed up and performed a quick demo against one of the desktop applications I was testing; while I watched they had the tool recording the mouse clicks and keystrokes, while they navigated and interacted with the application under test.

Once completed with the normal user workflow, they stopped recording, then played it back.

We watched as the previous navigation was repeated virtually, and the magic of test automation was revealed.

All we needed to do was record all our existing test cases, then hit playback and the tool would execute all our test cases automatically.

We saw this is a Christmas Miracle, because we believed this would dramatically decrease the time it takes to validate a software application, and that no additional work was required.

The Problem.

The problem was, best practices for implementation of test automation were not utilized – we didn’t know how to plan for test automation, or determine which functionality can (and should) be validated with a test automation tool.

We dove in head first and started trying to automate every feature & back end functionality we could think of, in order to reduce our work load.  What we didn’t realize, was the amount of overhead included with Maintenance of all the automated test cases after we recorded them.

It didn’t take long for us to discover that we were spending far more time investigating issues with automated test script issues, than we ever did performing manual black box testing.

One day the automated test would run all the way through successfully.

The next day (without any known changes) the automated test would stop, without a clear indication of the problem.

Which left us wondering, is the problem with the software we are testing – or with the test tool itself?

The Justification for Test Automation.

So does this mean, we should’ve never implemented test automation to support our software testing efforts?

A valid question – but we believe the answer is Not Necessarily.

When you are fixing a leaky pipe, you don’t take the pipe wrench and beat the pipe as if you’re using a hammer.

There’s nothing wrong with the pipe wrench, it just needs to be used correctly.

Same for Test Automation Tools.

Benefits of Test Automation.

You enjoy repeatability, reusability, extensibility, increased test velocity, faster defect discovery, shorter build turnaround, and increased confidence at time of release.

Let’s look at repeatability. You can run automated tests over and over again, during periods of flux, when new builds arrive and regression results need to be determined repeatedly.

Then there’s reusability, We can reuse automated tests for features or functionality that are similar, and only require minor changes for them to run.

The same holds true for extensibility. Instead of creating a whole new test to cover a new feature – you can copy then extend the properties of an existing test case.

Reasonable expectations for successful test automation projects.

First you’ll deal with the architecture of integrating the framework with your product.

Then you’ll have the initial creation of test suites.

Followed by the maintenance of your test suites – which is ongoing.

Just because you’ve finished creating your test suites, doesn’t mean you’re done with them.

For example, in many systems, when core features/functionality are changed…the test scripts must be updated to reflect those changes.

Otherwise they’ll fail.

And because you cannot automate everything, not all manual test cases are good candidates – so manual testing doesn’t go away entirely.

Another example; when a new platform is added. Instead of supporting only a web interface, you decide to include mobile devices.

This will generate the need for a new set of automated test scripts so they’ll all run properly.

How do I get started?

With our teams’ decades of test automation experience, we’re able to help clients determine viability as well as the correct application of this technology…for their software products.

There are numerous additional considerations for ‘best practice’ approach towards test automation (such as starting small with automating smoke tests to build a modular suite) and we’re happy to provide a free consultation to determine whether test automation is right for you.