Tales From The Dark Side

Sep 17, 2014

I've worked as a consultant and for consulting firms, and have had to deal with consultants, consulting firms and the aftermath.

It's often not a pretty story. Sometimes you really need outside expertise, but the quality of what you get is often unrelated to the cost, popularity or track record of who you hire. You might get a great result, or a new more colorful vocabulary.

These are a few of my stories, occasionally altered to protect the innocent or more commonly I've forgotten the details. I'm sure most programmers have experience with either side of the table.

The point of view of a consulting firm is to make money. Well, duh. The key is to find a victim, often known as a customer, that has a huge need and large sums to pay. Like a former salesman told me, always look for dumb customers with money. The most important point is to ensure that no matter what happens in the project you always come out with more money than you spend. Even if you think you might deliver a disaster make sure that even if there is a lawsuit or the customer gets angry or whatever happens there is enough money in the project to cover a pile of problems.

Consulting firms love to sell based on their amazing list of former customers and supposed expertise in some area. What often happens is that there is one smart person whom you talk with, and then they hire $10/hr people somewhere in the third world to actually do most of the work. The key is that the customer never gets to know or pick those folks. You might get some undiscovered genius programmers. You might get an Easter Island statue. You really don't know. It's a complete game of chance.

When I worked for a consulting firm during Dotcom era three of us went to a local customer to make a proposal. EDS sent one person. After our presentation the EDS guy stood up and said something to the effect of "I can have 100 people working here tomorrow". They were so enthused they practically gave the project to EDS before we could even open our mouths. Naturally these 100 people would cost a fortune, the project only needed 3 people, and many of the 100 likely were sleeping in a doorway the day before, but it was surely impressive.

Another time we made a proposal to a state government to build a new permitting site. Another big name firm came in after us and offered to take over the entire state's IT system and throw in the permitting site for free. Not sure how that turned out, the current site was clearly built by someone else a few years later.

In fact the code I am currently stuck working on was written by a many letter consulting firm and their Elbonian coders. It's way too complex for what it does and I can tell a whole host of people worked on it many of whom barely understood what they were doing. I don't really want to know how much it cost to build. I would like to ride in the salesperson's Rolls.

At a financial services company I once worked for, before my time, they hired a consulting firm to write what seemed like a simple project of some fill-in forms. After a year of the firm running a dozen or more people in and out the company killed the project. There was a lawsuit and I think some deal was struck. Apparently the consulting firm had used the project to train new people at my employer's expense to push out to other jobs instead of actually doing useful work. I bet they still made a nice profit but my employer wound up with some horrible PDF based monstrosity.

In one consulting project I was stuck on during Dotcom I was handed a programmer who did not know how to program and was expected to have him bill. He didn't understand the concept of a for loop; I wound up sending him to training continuously and wrote code for both me and him. I felt dirty. Thankfully the customer pivoted and no longer needed the project completed.

My most favorite terrible project was my 3 month Mexican travel. A sister construction company to the consulting firm I worked for (I worked for their US office) had been sold by IBM on Websphere Commerce Server as an ecommerce site in a box. This was in early years of WCS. He had asked if it could be customized and IBM told him, sure, you can change colors and fonts and things like that. He then assumed everything in WCS was completely customizable and proceeded to hire some US IBM folks to help. Then he hired IBM Mexico folks, people from the US office I worked for, the Argentinian office, and I think random people from the street. Eventually we had 40 people stuck in a single room in Mexico slaving away hacking every single feature of WCS including the database. It was insane, but IBM was fine with whatever they asked for, even though all of them (and us) knew this was beyond stupid. After like $3 million and 3 months they ran out of money. It never resulted in anything. That's the key thing, you should get paid even if the customer is a mindless idiot.

Funny thing about that last project I estimated it as a custom Java web app it was no more than 3 months with 3 people. But what's the fun in that?

Governments and consulting firms go hand in hand. Usually the Government does the handing out, and the consulting firms get the handfuls. You can't read the news without seeing these stories; they're almost commodity news.

I vaguely remember the FAA once tried to replace the Air Traffic Control System (this happens a lot) and resulted in two big name consulting firms building two ginormous proposals costing two billion total. When they finally showed them to an actual air traffic controller, they found out the whole thing couldn't possibly work. Then again this story might be an amalgam of a hundred others, but it makes sense.

I worked with a couple of guys at a consulting firm who had spent time on a Government healthcare system for the military; I think they were 4 levels of subcontractors down from the primary, and had basically no idea what they were supposed to do. This is another great way to make money, every level gets a nice cut, and expect the people below to do the work, but with so many levels no one actually does any because communication is impossible. But as long as money is available it will be consumed.

My favorite piece of code ever came about after a national magazine wanted to build a consumer product website where people could search for products particularly every automobile and truck in the world, even by searching for vehicles that potentially could only be ordered custom built. They announced it in their magazine a year ahead of time. They hired a consulting firm to build it. After six months of work they told the magazine what they wanted to do was impossible to build. So our consulting firm president offered that we could do it no problem. Three months later they hit the same wall, searches took 60+ seconds on production hardware with all the data and there was only a month left and no one could make it work. I volunteered to take a look, realized the problem couldn't be modeled normally in a relational database (essentially it was a meta problem, the data was a whole host of rules that determined what the data actually was that you wanted to search) and found a solution, replaced the existing code, and we made the deadline. Sadly they kept not paying us until the Dotcom crash happened, then their online group and my employer went belly up. The first firm however cleverly got paid just fine. Oh well, at least I had a blast working on something really hard.

The cool thing about working for a consulting firm is getting a wide variety of things to work on, though it's been a while and maybe today people specialize so much it gets horribly repetitive. During the Dotcom era we had all sorts of crazy customers; sometimes the work required a lot of imagination. I especially enjoyed fixing broken systems and people; when you make someone's terrible life so much easier or save them tons of money that they thank you profusely it's hugely rewarding. If I could just work on code that made people happy I'd be happy myself!

Working for or hiring a consulting firm is always a battle between how to pay (fixed bid or time+materials) and how to manage (heavy control or highly interactive) and who does the work (local, remote) and how are the requirements determined (fixed up front or flexible). There isn't any formula that everyone likes. Often choosing the wrong formula and people results in huge battles and no success making you wonder why the customer didn't just hire some people and do it themselves. You can get greatness from a consulting firm but often you get mediocrity or even complete disaster and you may not know which until it's too late. It's like betting on horses; no matter how much you study them they could race to win or die on the first turn.