Software Project Disaster Types: #2 The Death March

December 02, 2006

This is the second in a series on project disaster types.

Description Unlike a train wreck, where most or all involved think things are going well, in a death march the lower level workers (such as developers) know a disaster is coming but are powerless to stop it. Usually the managers beyond some level are confident that progress is being made and all will be well. In this case the train is going backwards with the leaders in the front, who see nothing but open space, and the developers are all in the back looking out at the disaster ahead.

Example I spent three months in Mexico on a project (called it ConsNet) that was a classic death march. A sister company to the consulting firm I worked for was in the business of supplying construction materials and tools all over Mexico, and wanted to have a web storefront where customers could order and track deliveries. Simple enough you think, there are thousands of these all over the web.

The first mistake was a high level VP letting IBM come in and sell him their eCommerce server, which has a lot of built-in features, has some flexibility and options but was basically a store-in-a-box. The VP asked if it was possible to do "a few customizations" and the sales guy of course said yes, assuming that the guy understood what a canned software package could do. Well it turns out that they wanted to customize everything and use very little of the built in feature set. Of course this is very hard so they asked IBM for help and they sent an architect and a team of developers, both from the US and Mexico. The local company set some developers as well, as did the sister firm (that I worked for) from its US offices and Argentina.

Soon there were 40 of us crammed into a large room, working shoulder to shoulder in two languages. Most of the features we were being asked to do were not supported by IBM at all (after all this was not a custom package) so even their developers were forced to hack the software. With so many people involved in working on the project, the costs were astronomical, so the pressure to get completed quickly mounted. Naturally new features crept in every day (one of the managers loved to walk around the tables making "suggestions" which then became part of the requirements. They even hired a goon to visit each developer every morning, carefully note what you were going to do that day, then return 10 hours later and carefully note what you completed. Often he would be told things like "I'm going to rebuild the flumsters and then retune the database tuner" since he didn't know a computer from a hole in the wall. Needless to say it was clear to everyone that this beast would never work. All of us griped every day for three months on the stupidity of making a custom application out of a canned one.

The local management of course told higher management that things were going well, but internally they blamed everyone and everything but the project itself. Eventually nothing ever worked so upper management discovered things were not so rosy, but by that time the parent company of all of us decided that the whole project was too expensive, filed suit against IBM, kicked all the developers out except for a few local folks and eventually let the construction supply division simply die out. The local coders worked for two more years before finally giving up.

The project managers both got a promotion. To this day I don't know how they did it, but I bet it involves a great story.

Solution Like the train wreck, death marches often involve a lack of communication. Usually one level of management takes on a project that is impossible or difficult, but cannot explain to higher management the difficulties and risks, and basically hopes things will work out anyway. This sometimes works as often higher level management is uninterested in the details and only becomes concerned when deadlines are missed or business plans are foiled; so as long as you get it done in the end, however dreadful it might be, you still "succeed".

The only way to avoid death marches is again follow an iterative project plan, keep all interested parties informed of both successes and problems. Often a disastrous project can be fixed or even canceled before it gets to far down the road if everyone is able to properly judge the progress.

In the case of ConsNet, there were no actual requirements every written down, however limited. Once it became obvious that the IBM software was incapable of being heavily customized, no one was willing to say "whoa" since so much money had already been spent. Then the failure to properly analyze the business needs and determine what type of software was needed (in this case write from scratch) led to an ever increasing number of programmers. In the end a $300,000 custom online store cost $3,000,000 and ruined the entire division.