Bias is everywhere. That is my unbiased opinion.
Prejudice? Deviation? Error? Those are strong words.
We live in a world that desperately wants there to be a middle ground where everyone can meet, free of any bias, and find common ground. That’s a utopian view that has little to do with reality. And looking at the above definition, it’s not just a philosophical or political issue; it’s also a statistical one.
Bias exists whether we choose to acknowledge it or not. It is the choice to acknowledge its existence that alters how we view the world, experience our lives and, in the case of software, can impact the effectiveness of testing.
This existence of bias is a hot topic in the world of software testing. Debate rages on about the types of bias there are, how they are introduced and, most importantly, how they can be prevented so results are as clean as possible.
Here’s an admittedly non-exhaustive overview of various types of testing bias. The reason I am quick to label it non-exhaustive is because this is a blog post and there are volumes upon volumes being produced on each type of bias as we speak. Also, since bias can be subjective, there are new takes on the subject being developed all the time - let us know if we should delve into each one more extensively in the future.
Observational Bias: This can occur early in a product life cycle. When examining a text list of the requirements of the product, the existence of wireframes can bias or skew results because it could cause testers to miss elements of the text requirements that don’t get a fair shake in the wireframe. In other words, results are biased because a picture may not be worth a thousand words in a software tester’s world.
“For example what if the issue is: that a website has several serious issues when viewed in a particular web browser, but not in a more 'mainstream' browser. When this issue is presented to the decision maker - How could it be presented?
His point is that while the second option seems to give more information it doesn’t address the real problem. It only addresses the result of the problem. If not explored further the problem itself could be missed completely.
Survivorship Bias - This is an interesting concept that Houghton explores as well. He uses the idea of how advertising messages are biased by using the example of a company saying, “99% of our customers are satisfied.” Sure that sounds nice and reads well, but the story behind the 99% could be that it is 99% of those that fought through the myriad issues with a product and survived the process to get what they can from the product or service. This is interesting in that we are wired to hear a 99% rating as a positive but when peeling back the layers it’s not at all.
In testing this kind of mindset could skew results easily and allow problems to persist even though, on the outside everything appears well.
Confirmation basis is far reaching and can include almost any definition of bias you would like. As one writer put it “this is the big daddy of all biases since it has so many variations”. Those variations as laid out in a presentation by Michael Bolton include assimilation bias, belief perseverance, availability bias, anchoring bias, recency bias, congruency bias and experimenter bias. And that’s only a partial list!
As pointed out earlier it is impossible to cover the full spectrum of potential bias that could impact software testing in a single blog post but there is one more to look at that is likely to fall on the side of the ledger that is less definable because it really addresses the human side of software testing. It’s almost more of a marketing term but it’s hard not to imagine that it can’t exist.
Emotional or Experimenter Bias- What if the tester has a particular bias toward the subject of what is being tested? What if there are personal issues that limit a tester’s ability to be neutral. Maybe there is a violent edge to a piece of software and the tester is against that kind of thing. What if the software being tested was around adult entertainment? What if it had to do with something as current and hot as the NSA scandal?
It’s this last form of bias that could be the most insidious to the success of a software testing project. It’s also the one that may not be identifiable, or even if it was, could it be measured?
So what have we accomplished here? We have defined bias and identified / defined only a few of countless number of variations of bias that could impact the results of software tests. In some ways, it may feel as if there is no hope for completely removing bias from software testing.
Many talk about the removal of bias in testing. One area we didn’t even examine earlier could be an organizational bias about testing, which simply means you work in an environment where testing is undervalued or not respected. If that doesn’t skew a tester’s point of view nothing will and it lead to so many other forms of bias even those discussed earlier.
“If people are ignoring the information being produced by the testing team, in my opinion – that’s the test team’s fault. Testing produces some of the most vital information to make business decisions about risk, release dates, and coverage – how can that information be ignored! Speak the language of your project to understand what 'value' means to your business. When you align your testing strategy and reporting methods to those, I guarantee you will not be ignored. In our organization, the responsibility of ensuring testing gets the focus it deserves lies with the test team, and no one else."
To sum it up, Klain is saying that testers can simply be more vocal and stand their ground to gain respect that might limit bias even toward their own roles.
As with anything having any number of opinions and options as answers we are going to need to ultimately decide for ourselves.
One way to approach this is truly from the other side of the computer screen. While end users don’t like buggy software, they are also more forgiving than those in the testing industry may think. In other words, they realize that perfect doesn’t exist, especially in the world of technology. Even those with cult-like followings like Apple products users see flaws in their favorite things but being a survivor (meaning they fought through their issues and become one of the 99% that are ‘happy’) they accept the imperfections.
That’s not to say that every effort shouldn’t be made to remove any form of bias from a software testing environment. It should. The expectation, however, is that there should be an acceptable amount of bias that makes its way through. The job of every tester is to mitigate its impact and ensure that the end product is performing at its optimal level with the least amount of hiccups possible. Will the process be perfect? Since human beings are involved I would say no and maybe that’s good enough.
Software Testing Certifications: Helpful or Hurtful?