Programmers Should Not Test Their Own Code

I should just leave the title as the whole post, as it's a mantra I've had since I started programming as a job, but what would be the fun in that?

Now I am not talking about unit testing, or small testing, or whatever you consider the day to day making sure every line of code or method does the right thing. I mean testing the application from the user or system level (if it's not user facing).

Testing an application, particularly if it is user facing as most of my 3 decades of work has been, is a discipline all by itself. It is not programming. Likewise programming is nothing like testing either. To think that one person can professionally do both is to denigrate each.

As a programmer I live in the application every day, running things constantly as I ensure that what I just wrote does what I intend. I might only run a small portion of the application or even just a single test structure. I might run a basic scenario. But there is no way any of this can be construed as testing the application itself.

Now our CTO refuses to hire any more people who do testing and emphasizes that developers should develop and test their code as well, and I can't understand this at all. It's a bad idea that will only lead to more and more broken software. It's like expecting the architect to also construct the building, or the race car driver to build the tires. Like the biblical quote where you can't serve both God and money, to be a good developer precludes having the insight and discipline to properly test an application.

I learned early the value of a good tester during the development of Deltagraph in 1988-1993. We had someone who was patient enough to repeat the same things day after day, to try the non-obvious things, to push the application in directions it was not intended to go, and never to accept our assurances that we had fixed something. The publisher also had someone very similar, and between the two of them the application was very stable and fairly light on issues despite the enormous number of features and code size.

On the other hand I have seen programmers whose concept of testing was try one and done. I've learned a lot through the years on the importance of stepping back and trying to use the application like a user sometimes, but it's not easy to look at it not as a collection of code but as a black box. In addition you run in a debugger and with all sorts of pre-existing files and never in a pristine startup condition. You might not even notice what the other programmers contributed.

When we first showed Trapeze 1.0 at the 1987 Macworld I discovered while doing demos that clicking a certain menu would result in a 1/3 chance of a crash, not a good thing to have in public. After the show I found that running with the debugger moved the app in memory enough that a random pointer dereferenced failed to cause a crash. Remove the debugger and the pointer now pointed to a value that could cause the crash. As a programmer at the time I never thought that testing under a different environment could matter.

Over the years I've seen a lot of programmer tested applications fail miserably because the programmer thought so highly of their code that failure was not an option, and thus testing was unnecessary. Other programmers of lesser ability simply didn't understand what the word "thorough" meant.

The value of a good tester cannot be underestimated. Generally they are disciplined, they like repetition, they refuse to accept that finding no bugs is proof that there aren't any. They will argue with the programmer that something is a bug and should be fixed. Testers should be a bit contentious and even irritating in the face of a lackadaisical view towards problems. They should pound apps with a sledge hammer. Testing is not for the meek. It's not for the unimaginative. It's a skill and a professional arena that needs to be just as important as the programmers, product people, UX designers and others.

As the game company I worked at for a couple years proved, no matter how great your features are, if your app crashes and fails to function, no one will ever see them, and they will stop giving you money. They didn't have a real tester and failed to comprehend that not hiring one or more for that matter even fixing the obvious issues might stem the loss of customers. I fixed what I could but it by the time I got there it was too late.

A (possibly apocryphal) story about Netscape was that they hired 100 engineers before they hired a tester. Seen them around lately?

No matter how careful or artful a programmer might be about small testing the need for big testing is not diminished. Quality applications are built by a combination of skilled people and one of those needed skills is testing.

Go ahead and fire all those testers and see what you wind up with. I just hope you don't work at the bank I have money (not much) at!