The Absolute Need To Understand

One sign of a good programmer is their absolute need to understand. Whether the environment they are coding in, the language or frameworks they use, or even the theory behind what they are trying to develop, the need to understand is a driving force in people who make good programmers.

Sometimes it's hard to explain to people who are not programmers what we do. Programming is part creative, part technical, part detective and sometimes part lunacy. We work in an abstract medium that then becomes a tangible thing when people are invited to interact with it. Our job is basically telling a computer what we want it to do. The challenge comes in how really stupid the computer is, and how difficult it is to explain our needs precisely enough to get it to work correctly. So we try to abstract the problem, building on top of layers of systems that hopefully are dependable, and hoping the bits we add are also dependable. Since we can't hope to understand everything we try as much as possible to understand as little as possible to get the job done.

That's where the trouble comes in. In our haste to make things easier to work with, we begin to lack the understanding of what is going on "under the hood". Programmers that focus only on "knowing the recipes" of a particular set of systems, languages and frameworks shy away from anything new, since it would involve learning a new set of recipes (possibly unknown). With no actual understanding of how their original technologies worked, the knowledge necessary to easily move to something new is beyond them. Think of a cook who can follow a familiar recipe, and a chef who understands how food is prepared; one can do the same recipe every time but is lost when faced with a set of new ingredients, whereas the chef sees a whole set of possible dishes. Of course there are times when it's enough to follow a recipe but in our industry change happens a lot and the ability to understand how to use a new technology becomes a necessary skill.

So what do I mean by "understand"?

You want to call a method on an object supplied by some framework, so you look up the parameters and description in some document (in Java, usually javadoc). Then maybe you find an example with Google and apply it in your code. That's the recipe at work, and usually it's enough to get the job done. Later on someone reports a bug in your program, and you have no idea what's going on, as the bug report is vague, it only happens sporadically, and your clues are few and far between. Pressure mounts from management to find a solution, customers are angry, and you begin to wish for a new job. Maybe talk begins of hiring consultants or contractors to help solve the problem, or managers begin promising upper management that they (you) will work around the clock until its fixed.

Then someone looks at the code and realizes you used a hashed data structure (in Java, let say HashMap) but failed to supply a hash function, just using the default supplied by the framework. It turns out the keys your were using in the hash were not consistent with equality, and failed the assumptions of the hash structure, leading to occasionally losing objects when they were mapped to the "same" key.

I have seen this happen. Of course anyone can write buggy code, unless you are NASA and can afford to spend tens of millions on a rigid programming methodology. The difference lies in wanting to understand not just the recipe, but the reason why something works. Understanding the concept is more important than just knowing the recipe. Even if a new system is radically different than the old, the concepts remain applicable, and the need to understand how the new system works will make it easier to master.

The best programmers I have worked with understand the nature of understanding. It's why good programmers are always looking for open source alternatives, even if they never need to change the source, being able to understand what is going on inside the system is the most powerful aspect of having the source. If you need to you can even fix it. I remember dealing with BEA Weblogic 5.X and getting patches to problems I reported, which in turn generated new problems, leading to new patches which lead to new problems, etc. For my own web projects I always use Jetty. If something really stupid appears to be happening in my web application, I can even stick the jetty source into my application, and follow the path in the debugger until I do understand. If I really need help I can even communicate with the people who wrote it and have an even deeper understanding.

Yet I have also worked with a lot of programmers who are content to know one set of things, to know nothing of why they work, and who refuse to see beyond their limited skillset. A lot of companies have staffs which value a standard set of technologies, buy only commercial closed-source packages (to have someone to blame when things go wrong), and suppress any desire to learn new things. In such an environment understanding is reduced to compliance with a set of recipes. One place I interviewed a long time ago prided itself on having architects who never coded, project leads who only wrote API's and programmers who only filled in method bodies. No time for understanding, no ability to learn, no opportunity to make things better.

The concept of Tao (from the Chinese philosophy of that name) is based upon the understanding that the only constant in the universe is change. Dealing with change requires understanding far deeper than just doing your job. If you want to become a better programmer, read, play, experiment, dig into source, demand time to learn, even at your own expense. Don't just learn the basic recipes but dig into why they work.

Understand?