Test Automation Webinar Q&A Series – Software Testing and Test Automation Tools

Posted by MindTree Blog Archives in

 Testing

May 20, 2009 | Total views: 100

4

MindTree Blog Archives

More articles by the author

Question: Slightly off topic question, from your experience, what tester to developer ratio do you recommend?

Response: You need enough testers to fully test (whatever that means in the context of the application or system under test) all code written by the developers on your team, including interoperability testing that will check interfaces to external systems, and any non-functional testing such as security, compliance, performance, accessibility, etc. For operating systems the ratio might be 1:1 or even 1.5:1 (testers to developers). I’ve seen IT and product teams with ratios of 1:12 and even 1:20, but they’re pulling in marketing and anyone else in the building during crunch time. I look for understaffing problems in anything lower than 1:5, but that’s just attitude, not something I’d bank on.

Question: As QA, what is the best way to prevent defect, whether or not it is design defect or actual code defect from happening?

Response: Set criteria for specific quality attributes and test for those criteria as early in the SDLC as possible, beginning with design. Review requirements and use cases (stories) and be sure the requirements are achievable and testable. In design, push development to write code that is easy (or easier) to test. Design hardened code. Profile functions for efficiency. Push for instrumentation. Drive developers as close to TDD as they can stand it. Insist on expert and peer review of developers’ code. Demand unit tests. Demand API automation. Demand using unit tests and API tests to achieve code coverage as close to 80% as you can get. Push for continuous integration, prefer workable code to meeting scheduled release dates.

Question: I am nervous about using Open source tools for automation. What are your thoughts of Open Source vs Vendor Tools we pay for (a la Mercury)?

Response: We’ve used Selenium for automation on many projects, building re-usable frameworks that enable reduction of cost as well as time to market. We’ve used other open source tools as well. You want open source tools that have lively community support. If you run across an open source tool whose developer community chatter has died off, that usually means that something better has come along and they’ve all gone there. The advantage of COTS tools is that their commercial interest requires that they respond to customer inputs, that they upgrade when new platforms or protocols render their last release obsolete, and that they provide or support the development of adapters, filters, and plug-ins. One disadvantage is that you’re dependent on the quality of their developers and their criteria for support. If you don’t have a platinum grade account, a vendor may not get to your query for months. Some tool vendors are much better than others at customer technical support. I’ve seen some great support from open source tools, people responding to queries on user forums within minutes of their posting, giving very practical workarounds or solutions. Then there’s the advantage of open source, that you can tweak your implementation at will. If you have the know-how, that’s a huge advantage.

Question: Any suggestion on which tool is the best know from your experience? I have only been into automation for few months.

Response: I couldn’t answer that without knowing what kind of testing you’re automating, the scale of your project, and what else you might want your test automation tool to do besides providing a scripting and execution environment.

The major tool vendors all have tools that work very well, and they all have features that appear to have received less attention during development than they should have. If you look at a review of test automation tools, or put one together yourself, you’ll find lots of positioning – this tool supports Linux but that one doesn’t, this one supports Windows 3.1 thru Windows Vista but this one only supports Win2K-Vista, this one handles distributed execution and remote operation while this one requires all machines to be on the same network, etc.

You’ll probably find out that the more careful you are in your shopping for tools, the more you’ll learn about your own requirements, and what opportunities are available through tooling that you didn’t know about.