Test Automation Webinar Q&A Series – Automation ROI

Posted by MindTree Blog Archives in

 Testing

May 14, 2009 | Total views: 76

1

Link

MindTree Blog Archives

More articles by the author

Question: In your experience, is automating an ecommerce site produce a high ROI?

Response: Yes, automation of some forms of testing of an ecommerce site can produce a high ROI. BVTs are the low-hanging fruit. Build automation should be as fully automated as possible, and wherever possible include automated build verification tests (BVTs). The ideal BVT is of the canary-in-a-coal mine variety; its UI path is stable; and its failure unambiguously indicates that the build is not ready to deploy in a test environment. Even if only half of your BVTs meet these criteria, your ROI for that small investment will be high.

If you attempt UI-based automation for areas that are frequently changed (whether because of new or removed SKUs, or new formatting, or new advertising, etc.), you won’t even get back the money you put into automation, and may even see negative ROI. That negative ROI, however, can be offset by long-term gains if you anticipate stabilizing UI in the areas you are automating, because although you’re changing scripts to keep up with UI changes, you’re also seeing where your automation framework needs simplification or tuning, you’re seeing where you need to develop common library routines for repair-once-benefit-many, and – as I mentioned in the webinar – you’re encouraging your team to look at their testing systematically and to find bugs early.

In some situations the cost of maintenance is compensated by the effort saved by early discovery of bugs.

Automation of testing of the backend – SOA integration of multiple data feeds, for example-will pay for itself over time, especially if you are consuming external libraries and services which, when they are upgraded or swapped out for better or cheaper sources, will require some intense V&V before going live again. If you have automation to fall back on, you can fulfill this requirement even if your marketing team has decided at the same time to push whatchamacallits instead of widgets.

Whether any of this applies to your particular situation, only you and your team members would know.

Question: Is ROI a justification of testing? Seems like ROI should be much more than that. On small compact projects is there a simpler view of ROI without considering a CFO?

Response: Well… I hope my presentation answered your question with a resounding “yes” to your belief, that ROI should be much more than a justification of testing. If you’ll look at the list of test automation benefits I provided, not as arguments to marshall in defense of a proposed automation project, but as a menu of methods for achieving quality objectives (that in turn support business objectives), ROI analysis becomes a planning tool.
I was going to say “a powerful planning tool”, but that all depends on how rigorous you are in your construction of formulas; how thoroughly you understand the interdependencies of schedule, available resources, and market (or IT org) conditions and expectations; and whether you update the ROI analysis periodically throughout production.

ROI as a planning tool is most effective when coupled with cost/benefit analysis and the business case. The work probably won’t be executed exactly as planned, but you’ll have a better understanding of implications for any proposed changes along the way.

Question: Is there any difference between calculating ROI for testing and ROI for a typical Application development?

Response: Yes, there are many differences. What those differences are depends on what investments you propose to make in changing (hopefully improving) how you develop software. But the questions you would ask to determine ROI in application development are quite similar: Are those improvements measurable? How? Are they expensive to implement? Do the benefits of those changes return value greater than what you invested? And so on.

Question: Does reduction in headcount or avoidance of headcount come into play on ROI? How?

Response: If you’ll take a look at slides 17 & 18 in my presentation – the discussion of the quantifiable of test automation benefits – you’ll see that many of the benefits could result in reducing the need for headcount. The question ROI analysis asks, of course, is whether the cost of automation development (including maintenance) is lesser, equal to , or greater than the savings derived from reducing headcount. One way to increase ROI, in any case, is to revector those “heads” for higher order testing processes.

Question: How do we convince management that automation will help with the software testing when they think that it’s better to just add more people to do manual testing?

Response: If they are not persuaded by reasoning along the lines of what I said in the webinar, you may be able to persuade them with a Proof of Concept project, in which – on your own time if you cannot get time on the job – you define scope for testing a component or dialog or screen or set of APIs, and measure productivity and bugs found by two teams (or two people), one testing manually and the other using automation, both testing the same code or product, but in isolation from each other. How long to write the test cases, to write and automate the test cases, to execute manually, to execute automation, etc. However I have run into managers who do not want to change, period. If it’s not broke why fix it. The same people complain about the lazy manual testing team who take twice as long as the developers, causing delays in delivery!

Question: Can the presence (or lack) of defects after executing an automated test be part of an ROI?

Response: You’re probably condensing your question so much that it doesn’t fully reflect your thoughts – but I must point out that running automation doesn’t reduce defects; at best it finds defects that have been introduced into the code, usually through re-factoring, bug-fixing, and new features. If automation developers test their scripts against working builds, they will find bugs before the script is executed. Every bug found by scripts after that is a regression, in that sense.

You may be asking, however, whether defects that “leak” through testing and are discovered after release can be considered as a negative factor in ROI. Yes, certainly they can, provided that one can demonstrate that without the factor whose ROI you are measuring. those defects would have been caught, i.e. would not have leaked.

Question: When formulating a ROI, how would your calculate for on-shore and off-shore testing?

Response: The most important factor in formulating ROI for sourcing is, what do you need to achieve?

Be sure to use fully loaded costs for both in order to get a reliable comparison of actual costs. An offshore test services vendor may or may not use higher maturity testing processes than your in-house team, and your costing should reflect that. The offshore vendor may have better (or worse) training than your in-house team. What you’re testing has a lot to do with ROI as well – what provides high ROI in testing one system may not be sufficient in testing another. The offshore vendor who wins on price may disappoint expectations of rigorous testing; the in-house team that claims that domain familiarity will always win the day may not understand boundary cases, or the importance of resolving every bug discovered in testing.

Question: I am nervous about using Open source tools for automation …what are your thoughts of Open Source vs Vendor Tools we pay for (a la Mercury)?

Response: We’ve used Selenium for automation on many projects, building re-usable frameworks that enable reduction of cost as well as time to market. We’ve used other open source tools as well. You want open source tools that have lively community support. If you run across an open source tool whose developer community chatter has died off, that usually means that something better has come along and they’ve all gone there. The advantage of COTS tools is that their commercial interest requires that they respond to customer inputs, that they upgrade when new platforms or protocols render their last release obsolete, and that they provide or support the development of adapters, filters, and plug-ins. One disadvantage is that you’re dependent on the quality of their developers and their criteria for support. If you don’t have a platinum grade account, a vendor may not get to your query for months. Some tool vendors are much better than others at customer technical support. I’ve seen some great support from open source tools, people responding to queries on user forums within minutes of their posting, giving very practical workarounds or solutions. Then there’s the advantage of open source, that you can tweak your implementation at will. If you have the know-how, that’s a huge advantage.