Before I poke waterfalls with a stick, a couple of stories.

Story, Waterfall One

Project Manager: What are your requirements?

Customer: I want to float on the Niagara river.

Project Manager: What would you like to float with?

Customer: A boat?

Project Manager: What type of boat do you require?

Customer: Maybe a canoe?

Project Manager: OK, sign here and we will build you a fine canoe!

[Later]

Project Manager: Time for the customer acceptance test, please try it out.

Customer: Hey it works great.

Project Manager: OK, project complete, good luck!

[Customer floats down the river, falls over the edge of the waterfall]

Customer: Wait, this isn't what I need!

Project Manager: Sorry, you agreed to it. Request a change order...

[Customer dies horrible death]

Story, Waterfall Two

Project Manager: What are your requirements?

Customer: I want to float on the Niagara river.

Project Manager: What would you like to float with?

Customer: A boat?

Project Manager: What type of boat do you think you might need?

Customer: Maybe a canoe?

Project Manager: OK, we'll try that out first.

[Team builds a canoe, floats it on the river and it crashes over the falls and breaks]

Project Manager: Hmm, that wasn't the right thing. Maybe an enclosed boat?

Customer: Like a barrel, maybe?

Project Manager: OK, for our next iteration, a barrel.

[Team builds a barrel, puts dummy inside, pushes it over the falls, barrel survives but dummy is cracked]

Project Manager: Hmm, that wasn't quite the right thing. Maybe we pad it heavily?

Customer: OK

[Team builds a padded barrel, puts dummy inside, pushes it over the falls, barrel survives and the dummy is OK]

Customer: That's the thing, I'll take it!

[Customer floats down the river, falls over the edge of the waterfall and survives]

Customer: That's exactly what I needed, thanks!

Discussion

Exaggerated stories aside, the fundamental difference between the venerable (and sadly popular) Waterfall methodology (with its pretty pig-name, SDLC) and the more modern iterative approaches boils down to one main point:

"Customers don't know what they want, much less what they need, until they see it."

In all my almost 27 years in this business, I've learned this over and over again. It's very difficult for customers (i.e. the consumers of your software project) to precisely describe their needs sufficiently to provide an exact list of what they need. Most people have a tough time imagining a software solution sufficiently without a concrete example. Generally they must be guided in an iterative manner to come to a decision point; usually they need to actually get hands on with at least a minimal representation of the software in order to "see" whether it will meet their needs.

The Waterfall/SLDC methodology requires that each step in the process be complete before the following step can reasonably start. You can't start development before all the requirements are known and documented. In theory this should be possible; in practice I've rarely seen it happen except in limited circumstances.

I once knew a guy in the Bay Area who specialized in writing SCSI drivers. He was an expert, and basically did the exact same thing, with minor changes, over and over (plus he made enough to spend 6 months every year in Alaska on vacation, but that's another story!). This type of situation, where everything can be specified up front as there are no real unknowns, makes such a rigid process work. If I write a general ledger for the seventh time in my life, I bet there is little different than the sixth time. If all I am doing is some maintenance on an existing application, I would expect that changes could be documented ahead of time.

If I am writing a brand new system (like I am now at work) there are way too many unknowns to do any kind of up front specification and even design. I don't even know how I will accomplish even the most basic requirements until I have spent some time trying approaches. My "customer" the company is not sure what new products and services that might be possible if the new system can be flexible or scaleable enough. We are replacing an existing system, but it has so many problems and irritations that no one wants to simply replicate it. In this environment all we can do it start with a basic set of needs, build enough to see if the basic architecture works (it has to scale 10x at a minimum of our existing one), then examine how it can be expanded as people see what it can do. There is only one way to work with something unknown and that is iteratively.

This is a hard thing to explain to people for whom waterfall is an embedded part of their psyche. Of course it is also hard to tell people why most of our projects are failing as well. Generally the response to failed waterfall projects is more requirements, more documentation, more analysis. When I was at First Command, we had a 15 step (meetings, documents, signoffs) process before any code was actually written. You would think it would keep projects from happening, and it did (on purpose I always thought).

Story, Sabre Corporate Air Pricing

So how does an iterative, customer involved approach work? As an example, take my experience building a customer facing and internal workflow application for the Sabre Corporate Air Pricing group. Back in 2000, this group (who's job was taking information about flight discounts from travel agencies serving Corporations, and programming them into the Sabre Reservation System). They had about 100 or so people who did the work of "coding". They used faxes and emailed documents from the agencies, and managed the workload by the "drop files on chairs" method. Agencies never knew who long it would take, errors were common and corrections took a long time. Meanwhile errors in discounts were caught by the airlines, and Sabre had to eat the penalties (millions of dollars a year worth). Yet Sabre IT had no interest in helping this group automate; they did write a minimal spec for a web app for the agencies but nothing more. So the group finally had enough and hired the consulting firm I worked for to build the agency facing web app.

Since I was the only person who knew JSP (which Sabre had started using) I worked on that app, completed it per the spec, and then asked a simple question: "OK, now you have data coming from the agencies, what are you going to do with it?". IT had no interest in the question, and the CAP group didn't know anything about web applications at all.

At this point in a normal waterfall you would gather requirements and have them sign off on them. Fortunately I was allowed to do my own thing, and together with another engineer, Dave, we proceeded to learn their business sufficiently to see what was needed. I spent a lot of time explaining what a web application could be made to do, and built numerous HTML prototypes to show them what was possible, and as they began to understand, we jointly decided that they needed the public web app connected to a backend workflow system to manage the workload, and connect the work in progress with status visible to the originating agency, and keep a full audit system also visible to all parties. Over sixth months the two of us (with a bit of help from another) build an application with about 70 jsp pages and a complex Oracle database to back it up.

Then IT finally woke up and decided to get involved (by this time the app was working and in production use by a few customers as a beta). They then spent the next 6 months following their own methodology to add the login page. So we spent the extra time adding many more features and getting more and more beta testers using the system (the agencies loved it).

Sabre basically paid for this system immediately as now they could distinguish internal errors from customer errors and no longer had to pay many penalties at all. In the following two years the system was eventually extended to completely automate the process and (sadly) eliminated the entire department to an enormous savings.

All from a simple question and an iterative and interactive approach to the discovery of what the customer actually needed. Sometimes the only way to find out what a customer needs is to build it over and over again until the app and the needs converge. Agile, scrum or iterative; whatever it's called it's the best way to build something new.

Then no one needs to die pointlessly in a waterfall.

My Tags: methodology

One thing I have learned in my career is that what a customer needs in a software solution is often very different from what winds up in a requirements document. The software that is delivered then becomes a source of contention between the customer and IT:

"It's not what I need."

"It's what you asked for."

That's the crux of the problem. Requirements to an IT team, especially in a Waterfall environment (or one of its Lipstick-On-Pig alternate names), are elements which lead to architecture and design. They need to be reasonably complete and precise. A customer's needs are likely to be far more imprecise and incomplete. They are likely unaware of how software is actually developed, and may even be unfamiliar as to what is possible and are thus unable to know what to ask for.

The difficulty in most corporate environments is to find someone able to comfortably communicate with both groups. Typically you find a "Business Analyst" who's job it is to extract requirements from the customer and present them in some kind of document to the IT staff responsible for building the solution. Most larger companies do not typically allow programmers to interact directly with customers.

IT leadership usually hear the customers complaints and react by trying to "nail down" the requirements even harder by more rigorous and complicated methodologies changes. More meetings, more documents and especially "signoff" contracts to force the user to agree on what will be done are an attempt to deflect any finger-pointing later on. All of these tactics are to enforce the "It's what you ask for" point of view.

In my experience, putting more emphasis on the "what I need" point is view is more difficult but ultimately the right thing to focus on. Customers often don't know precisely what they need, or if they do they are unable to articulate how a solution might look. Often they are unaware of the range of possible ways to derive a solution and thus ask for the wrong things. Sometimes customers may read of solutions or imagine themselves as knowledgeable in software and demand inappropriate things.

All of these things require a different kind of "analyst": one who is equally as comfortable talking with customers as they are of building software solutions. I call this type of person a "project architect". Part teacher, part analyst, part architect: they need to be able to understand the customer's business, listen and interpret the customer's comments, teach the customer what is possible, and then derive a potential solution.

Ideally these potential solutions need to be demonstrated in prototypes or mockups so that the customer can give feedback; the advantage of having a programmer/architect in this role is that they can easily build things to show the customer instead of relying on explaining to others to get something to show. People react much better to something they can see and interact with than long lists of requirements or complex documents or (gasp) UML diagrams.

Despite the apparent obviousness of all this most larger companies and government agencies continue to follow the age-old waterfall methodology and the traditional business analyst role. I have seen and read of so many project disasters where the inability to adequately determine the customer's needs torpedoed the project before it any code was even written, even if the project continued thereafter.

Example

In the Corporate Air Pricing project I did at Sabre in 2000-2001, the customer group took contract information from travel agencies who served corporate customers (who had made contractual agreements with the airlines for discounts), and programmed the details into the reservation system. It was all managed with data in a Word template file, emailed or faxed from the travel agency and then coded by the 100+ person team at Sabre. Any errors caught by the airlines later after tickets were issued and used were charged back and usually Sabre had to eat them (to the tune of millions of dollars per year) since it could not prove if they or the travel agencies were to blame.

The CAP group was unable to interest the IT staff to help, other than write up a simplistic web application spec for the TA facing website, so they contacted the consulting group I worked for help on implementing the spec. After building it (it was more of a design document than anything else) I asked a simple question, "OK, now what are you going to do with the data?".

Since the IT "help" had been so minimal they had no idea. What followed was a lot of learning their business, teaching them about what one could do in a web application, both for the TA customers and internal to the CAP group. The customers needed to know the status of a contract, make corrections and add updates, track who made the changes and when and know when they could start pricing tickets. Sabre needed to be able to demonstrate when and what changes were made, manage the workload of the CAP team, track the employees workflow and other management functions. The process needed to become more efficient (even though the programming into the reservation system itself was ugly and slow).

During the process I continued to build and demonstrate HTML mockups so that we had something to look at and discuss. The people I worked with were eager to learn, and once they saw something they could respond with more pointed questions. Even after development began constant access to the work in progress allowed us to gain feedback (I had a team of 2.5 including me) and guide the project to correctly match the needs of the CAP group.

In six months the project was essentially complete. At this point Sabre IT got involved (and then EDS) and proceeded to hold up the project for an additional six month to implement a login screen so we added more functionality and managed an extensive live beta during that time.

Eventually my partner in the project continued when our employee went out of business for another consulting firm and ultimately the project became completely automated, sadly eliminating the CAP group entirely, but saving Sabre a ton of money.

Lessons

• Find people comfortable and able to communicate with customers and programmers and have experience actually building solutions.
• Use iterative and interactive processes to discover the customer's needs.
• Use mockups, prototypes and pictures to converse with the customer, not words.
• Learn the customer's business and ask lots of questions before, during and after development.
• Don't spend worthless time building systems to defend bad software solutions.

Sadly it is difficult to convince large corporate or governmental entities how a real customer-focused methodology works. The bottom line remains that IT built software solutions exist to serve the customer's actual needs, not necessarily what they ask for.

Ask any child lemonade stand vendor, if you serve lemonade to your customers without the sugar they need, they get mighty sour.

My Tags: it methodology project management

Since returning to Apple ten years ago, almost everything that has come out of Apple has been amazing. Not everything has been successful (the Cube) but everything has grabbed people's attention in ways no other technology company has. The primary reason is that Steve refuses to put out (or put up with) crap.

And neither should you or I.

Sadly most software you use, see or interact with every day is crap. If you imagine the oft-repeated quote, that 70% of all software projects fail or are cancelled before completion, sometimes you wonder if 90% of the rest should have been.

Working for Steve can be a nightmare for the unknowing. Like a perfectionist chef dropping dishes in the kitchen that don't meet his requirements Steve will tell you your life's work is crap. Of course by now he has surrounded himself with a staff that understands that it's not personal, Steve really cares about the whole project being the best it can be, and you better care as well. In many places having the CEO personally look at everything would be a disaster, but in this case Steve not only cares but has an eye for what is not crap and a willingness to push, prod, poke and even kick you into doing it the right way.

If everyone had such a vision (though I would hope a bit more tact, but maybe that's necessary) then the software and other technology we interact with everyday would work so much better.

For myself, I also hate crappy software, crappy tools and crappy methodologies; not being the CEO though means having to carefully find some way to encourage better development, better decision-making, and just better thinking in the place I am working. It's not easy.

In some places I have worked the enviroment was so poisoned that many developers just give up, produce crap, take their paycheck and go home. When managers get promoted after mis-managing projects that end in disaster (see my disaster posts) it's hard not to give in. Sometimes (been there twice in the last year) you just have to go elsewhere.

I worked at Apple in the year before Steve returned, and I know that accepting mediocrity was commonplace. My two favorite Steve quotes from around that era:

"The products suck! There's no sex in them anymore!"

"The cure for Apple is not cost-cutting. The cure for Apple is to innovate its way out of its current predicament."

At that point Apple was in danger of dying completely. During my tenure there Apple lost $750,000,000 in a single quarter. Since Steve returned the stock is up something like 1,000%. By refusing to release crap, he has turned Apple into a completely dominant company.

So what's the lesson for us mere mortal, non-CEO, possibly peon-ish programmers? Fight crap, fight mediocrity, fight the easy way out. Use innovative technologies (if they aren't crap). Fight stupid management if you have to. Don't let the broken down machine in your company break down your desire to produce excellent software.

If all else fails, go work at Apple and see what Steve thinks of your work.

Just don't produce crap.

My Tags: apple programming it methodology

Looking over some java ads (which can be entertaining unless you are looking for a job), I kept seeing references to the SDLC process. I'm familiar with many different software methodologies but that acronym seemed new. Once I looked it up (what did we ever do before Google?) I realized it was just the old familiar long-discredited-but-still-popular Waterfall methodology. Instead of doing something that works, let's just rename it something cool!

Two jobs ago we had a similar monstrosity which actually had 15 steps, yet it was never followed by anyone unless ordered from on high (which rarely happened). The methodology only existed to kill projects that IT management didn't want to do (bury it with meetings a paperwork) and apparently to torment developers (at least it seemed that way to us). Successful projects tended to follow a more agile plan, even though such a term was never used or mentioned. In my entire career I can't remember a single successful project that followed a strict waterfall system, most projects I have had leadership into have been iterative in nature since my earliest days as a developer.

So why is the waterfall still practiced, especially in larger corporate entities? The illusion of control. What makes more sense than to plan each step one after another, in an orderly fashion, with predictable results!

"The SDLC process is an eleven phase project life cycle providing a consistent, effective and repeatable practice that serves as the framework for delivering high quality, on time and on budget IT enabled business solutions."

I found this quote and the above picture on a university IT page. It's a perfect example of the fantasy world many upper IT management people live in where if you simply have the perfect plan, everything will work perfectly. Sadly this fantasy is all too common and usually leads to terrible project failures, as it simply doesn't intersect with the reality in most companies. Determining what a customer (be they internal or external) needs and being able to deliver it in a reasonable budget and timeline can never be predetermined perfectly. Better to follow a methodology that assumes imperfection, adjusts to change, and lives in the messy world than people work in.

What type of iterative approach is not the important thing, there are many different ways to follow a cyclical determination-development-delivery process. The important thing is by breaking down the project into smaller pieces, getting input from the customer on a regular basis in response to the smaller deliveries, you are far more likely to deliver something that not only meets the customer's requests, but actually delivers the customer's needs. Customers rarely know what they need and may actually request the wrong thing, as they may not even be aware of what they actually need at the start of a project. This is one of the base fallacies of the waterfall methodology, that people are able to properly and completely (to the point of "signing off") say what they really need before any work is done. Such perfect customers only exist in fairy tales. People can recognize what they need when it's demonstrated to them (either in real code or some kind of functional prototype). Customers always respond to what they can see, rarely to what they hear.

So why do people keep following this fairy-tale process, knowing it will fail? If you have a good explanation let me know. Pigs may be tasty (unless you are a vegan) but they shouldn't be your software development methodology.

My Tags: project management methodology it