Behind the Curtain: The Smell You Smell May Be Your Company Software

During my many years as a consultant, contractor and employee I've seen a lot of really bad code, especially in larger companies. So who's to blame?

The funny thing is that is isn't usually the programming staff that's to blame (though it does happen). Good programmers (you do have good ones, right?) when given enough options can usually find a way to make things work. Often I've found that the environment they work in doesn't give them enough authority to pick their tools, frameworks, or often times even the process of development. Often managed from the top by people with no technological clue the teams suffer the shame of building abysmal systems that they know are crap but are powerless to stop it.

Two jobs ago the company was run by an authoritative but otherwise clueless CIO, and most of the IT middle management was not any better. The company's primary database had always been an AS/400 and that team was considered more important than the java team, which suffered basically as the lowest caste. All the web applications had to use as their data source a relational DB2 shell which ran on top of the AS/400 OS. The underlying tables were all accessed by batch RPG programs, which did not respect the relational constraints, so there weren't any. This led to years of non-unique primary keys, faulty queries, tables changing on the fly, etc: the java apps were constantly failing left and right due to this and thus the java team was considered incompetent. Often indexes and table changes were not correctly propagated from DEV to QA to Production so testing was often impossible until you had customers complaining. So the software stunk but like a sudden smell in the elevator, the leadership just ignored it or blamed someone convenient.

Of course this isn't a single example. One place I did consulating at (Cheap Tickets in 1999) had a refund process from hell, which took 3 months on average to refund customer's money for unused tickets. The process I found involved several terribly overworked (and underpaid) clerks working through mounds of paper, various pieces of software never intended to be used for this purpose (including using the Sabre Reservation System as a database), post-it notes, and lots of manual labor. During the process you couldn't even determine whether a customer even had a refund or what the status was. After losing customers left and right they finally admitted that the process was broken, and my company was hired to build something. After a few weeks I had a system that cut the time to a single week and anyone in company could check the status at any time. None of this was rocket science but no one in the company for the longest time even gave any thought as to why all these customers were complaining.

One of my friends from a past job recently started a new one and discovered that the new company's software was even more disorganized that the one we worked at together (hard to imagine).

At Sabre I studied the air pricing intake department to offer some advice on how to fix their productivity problems (this was 2000 or so) and found they used 17 different applications to do their jobs; it took an average of 6 months before any employee was able to understand everything enough to be effective. After three such studies (including ours) they still didn't want to deal with the problems. Maybe they fixed it by now, I have no idea.

I've seen places that refused to use source code control (too much effort), or moved application changes directly from the programmer to production. One manager I knew insisted that a programmer put into a production a complex SQL query before he even had time to run it once. I look at the mess that Wells Fargo's online banking is (I don't know the software, but just looking at the URLs and all the application switching going on) and shudder what those poor coders must be up against.

I've seen smelly coding environments in way too many places over the years; someday it would be nice to see a functional place where the applications work correctly, the customers are happy, and the code is maintainable.

Nice dream, anyway. Anyone out there work in a place without smelly code?