Software Testing and Bias

I was reading a brief article over on about Confirmation bias. Essentially it boils down to the fact that people are more likely to seek out confirmation that what they have done is correct. In doing so they may overlook tests that don’t fit the solution that they have in mind.

The article gives the example of sets of 3 numbers that satisfy an unknown set of conditions. When people are presented with one known correct answer of [2,4,6] they assume that the pattern is [2x, 2(x+1), 2(x+2)] and don’t bother to supply sets of numbers that fall outside this assumption. Through doing this they may miss a key aspect of the unknown set of conditions and assume that what they have derived is correct. If the pattern that gives a correct solution is any even number, all of the tests that are derived around the initial incorrect assumption will pass, but the true nature of the conditions that yield a correct set is still not properly explored. In this situation only a small subset of correct responses are assumed to be correct.

In software testing this type of bias can lead to test cases that only test the obvious output and input. Most bugs appear when input outside the norm is passed in and A good tester will not only test for a correct result from well formed data, but a correct result for garbage data as well.

Another form of bias that the article talks about is clustering bias. When a person is presented with a number and then asked to answer a numerical related question such as how many countries are there that end is stan, they will cluster their responses around the number they were given previously.

In software testing this type of bias can lead to test cases that are similar enough that they don’t test anything else of real value. All of the test cases need a very specific aspect of the code that they should test. If your tests for different functions are cut and paste jobs from other functions chances are you’re going to miss something.

The testing mindset is one of assume the code is wrong, what would prove with reasonable certainty (we’re not writing proofs for shuttle navigation software here) that your own assumption is incorrect. Falling into one of these biases when writing test code will render your tests next to useless as the likelihood that something important gets missed will increase dramatically.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>