Spending time on quality or bug fixing?
A while ago, I was in conversation with someone about automated software testing. We talked about the cost of writing tests. Then he made a statement that went more or less like: “Software testing is valuable, but it must not cost more than the time saved by bugs found, otherwise it would be pointless.” At first glance, this sounded sensible. But still a feeling kept nagging me that there was a fallacy in this reasoning, yet I could not point out exactly what was wrong with the statement. Until holiday came, where I experienced the consequences of this approach in a completely different setting.
Holiday home software style
When we made the reservation for a holiday home, we reviewed the owner’s website to find out what we could expect. It looked promising for a family with small children: dishwasher, all windows with insect screens, bikes with 2 baby seats, 2 go-karts for the children and a separate room for the baby. The owner proudly stated that they replaced the fridge with a bigger one, renovated the shed, and recently improved the insulation. Sounds good, doesn’t it?
It turned out that the owner of the home was a friendly guy, but, sadly, also rather sloppy. He practices exactly the same reasoning into practice: put as little energy as possible into the house, the guests will eventually complain when something is wrong. And as long as they don’t complain, everything is fine.
When we arrived, one of the first things to do is to put our food into the freezer to keep it from further thawing. Except that there was no freezer anywhere to be seen. When we called the owner. He apologized: “Oops, I was thinking about ordering one, but I forgot about that. Please give me the dimensions of the available space so that I can order one.” So we did, and at the end of the first week a freezer arrived. The owner trusted us to install it ourselves. The first week, it was warm weather, but we had no ice cream.
Second thing to you normally do is to inspect the vacation home. It turns out that the parent bedroom was merged with the baby room into a single room. So this was a mismatch with what was advertised. Everyone who has raised toddlers knows that this makes sleeping in impossible.
Insect screens were also not available for all windows, and the insect screens on other windows were fastened with sticky tape which has come off. We had to repair this first before we could open windows. Luckily we brought our own single use insect screens so that we at least could have some fresh air.
It was just reasonable to ask from the owner to inflate the tires of the go-karts and bikes. Except that it turned out that he only had provided a (too) heavy duty air compressor. And one of the go-karts had a flat tire. And there were no instructions prepared on how to use it and where to find the parts needed. And when we inflated the tires of the other go-kart, a tire burst because it was old. So we called again. The owner apologized again, but had no clue what to do. Repairing was not possible on short term. So we decided to swap tires so that we had a broken go-kart and a good one. Needless to say that the kids were disappointed.
When we wanted to go cycling, we noticed that there was only a single baby seat, which also was not mounted yet on the bike (which required special tools (which were not available)), instead of two seats. So we had to call the owner again. He immediately went to work on a solution, and we could go the the nearest bike shop to have the baby seat mounted, and to pick up another baby seat.
All in all, we had to solve a number of problems. These problems were actually the owner’s problems, but he made them our problems. It has cost us a lot of time to have them sorted out, to get things fixed and repaired. We shouldn’t have had these problems in the first place! We had to contact the owner quite a few times, so, from a spent time perspective, he would also have been better off when he made things in order before we came.
What is wrong with the cost approach
Software quality works exactly the same as the holiday home: if the software does not conform to its specifications (does not work as advertised), then there is a quality issue. The program does not work as specified. In my opinion, this is also the case when it works most of the time, but not in all cases.
Back to the original quote: “Software testing is valuable, but it must not cost more than the time saved by bugs found, otherwise it would be pointless.” This statement expresses a criterion with a cost function. Or put differently: this is a rule about the amount of test automation, based on the cost of fixing bugs. If you think you will find bugs that cost a lot of time to fix, then you may add more tests. But if you expect to find only a few bugs that are easy to fix, then adding automated tests is a waste of time.
Although the statement relies on an estimate of the future cost of fixing bugs, such an estimate is impossible to make, especially when you didn’t do testing in the first place. You simply don’t know how well it works, so you also cannot tell how much time it will cost to repair the issues found. I think it is even safe to say that the cost of fixing bugs will always exceed the cost of testing, even if you estimated otherwise, because you just gave away the way to find remaining unknowns in your program. Moreover, the later a bug is being fixed, the more it does cost, and this increase can be orders of magnitude. You just cannot trade quality for time.
Building a software product involves a producer but also clients. The cost part is only what the producer pays, usually expressed hours of work. So the statement is also selfish: it leaves the impact of possible bugs out of scope. If the client suffers badly from a bug that is easy to fix, he will have high cost like lost revenue or expensive workarounds. But the statement does not included these costs at all. This has the result that the producer saves some costs, at excessive costs for the client.
When saving on testing, actually a lot of the actual testing effort is delegated to the user: he finds the issues and needs to spend time to report the bugs and find workarounds. But the client is not a software tester, it is a software user. And he may not even be able to use the product as he intends because of these bugs; this reduces the value of the software for the user. This has never made a client happy, no matter how fast you can fix the bugs he finds. Moreover, since there are only a few automated tests, there are no safeguards to prevent existing functionality to break. You will experience that bug fixes introduce new bugs. The overall quality may not improve at all.
But my main point: in my opinion, it is unethical to work like that. A healthy business makes profit by actually providing value for its customers. This means that profit is a consequence of providing value and not the other way around. I think that that is the only way to have a sustainable business. Having a business therefore requires some kind of empathy for the client: to understand what he wants to accomplish (and why), and to develop a solution that works for him.
Software quality is no different from the quality of any other product. Whether I buy a car or furniture, rent a vacation home or call in a repairman: if the product or service is not 100% as expected, I complain. There really is no reason that this should be different for software.