Data and Knowledge in Testing

Posted by MindTree Blog Archives in

 Testing

Mar 18, 2009 | Total views: 2

0

MindTree Blog Archives

More articles by the author

A central concept of software testing – perhaps this is the rare case of a principle in software testing that is at once necessary and sufficient – is that its ultimate purpose is to gather the data needed to verify that all release criteria for the software under test (SUT) have been met.

All of the test engineers I know or work with though understand that data by itself is not knowledge. We establish quality objectives that support business objectives for the SUT. How will the pencil be used? Have other solutions been tried? We design tests that will affirm or deny predictive or descriptive propositions such as the “API GetLastRegName functions as designed”; or we compare our test data with target, baseline, or benchmark data, and conclude that the SUT has exceeded or fallen short of what we sought or expected.

As we accumulate more relevant data (where relevance is in terms of the selected quality objectives), we believe that we have increased our knowledge about the SUT. Oddly, we also tend to think that as our knowledge increases, our ignorance decreases. I don’t think so. Ignorance persists; or as a software designer friend of mine might say, we persist ignorance across multiple domains.

The fault lies in thinking that knowledge is a zero sum game. I certainly don’t remember thinking that, and I very much doubt that you do either. But it’s is in our minds, perhaps a fossil surviving from when such thinking was necessary for survival. If knowledge of our SUT did have an empirical limit, then the complement of our knowledge would be ignorance – our ignorance would decrease as our knowledge increased. We could one day conceivably eliminate our ignorance of the SUT – and with it, eliminate all bugs and all risk. But there are always other environments in which to test, different processors, different chipsets.

Unless we limit knowledge’s domain, it cannot push back the frontiers of ignorance. We structure our knowledge by the form of our inquiry. Certain inquiries have very limited domains, offering hope (and therefore the illusion) of mastery. But without curiosity and appreciation, which are both forms of inquiry, we learn nothing.