A Tale Of Two Waterfalls
Before I poke waterfalls with a stick, a couple of stories.
Story, Waterfall One
Project Manager: What are your requirements?
Customer: I want to float on the Niagara river.
Project Manager: What would you like to float with?
Customer: A boat?
Project Manager: What type of boat do you require?
Customer: Maybe a canoe?
Project Manager: OK, sign here and we will build you a fine canoe!
[Later]
Project Manager: Time for the customer acceptance test, please try it out.
Customer: Hey it works great.
Project Manager: OK, project complete, good luck!
[Customer floats down the river, falls over the edge of the waterfall]
Customer: Wait, this isn't what I need!
Project Manager: Sorry, you agreed to it. Request a change order...
[Customer dies horrible death]
Story, Waterfall Two
Project Manager: What are your requirements?
Customer: I want to float on the Niagara river.
Project Manager: What would you like to float with?
Customer: A boat?
Project Manager: What type of boat do you think you might need?
Customer: Maybe a canoe?
Project Manager: OK, we'll try that out first.
[Team builds a canoe, floats it on the river and it crashes over the falls and breaks]
Project Manager: Hmm, that wasn't the right thing. Maybe an enclosed boat?
Customer: Like a barrel, maybe?
Project Manager: OK, for our next iteration, a barrel.
[Team builds a barrel, puts dummy inside, pushes it over the falls, barrel survives but dummy is cracked]
Project Manager: Hmm, that wasn't quite the right thing. Maybe we pad it heavily?
Customer: OK
[Team builds a padded barrel, puts dummy inside, pushes it over the falls, barrel survives and the dummy is OK]
Customer: That's the thing, I'll take it!
[Customer floats down the river, falls over the edge of the waterfall and survives]
Customer: That's exactly what I needed, thanks!
Discussion
Exaggerated stories aside, the fundamental difference between the venerable (and sadly popular) Waterfall methodology (with its pretty pig-name, SDLC) and the more modern iterative approaches boils down to one main point:
"Customers don't know what they want, much less what they need, until they see it."
In all my almost 27 years in this business, I've learned this over and over again. It's very difficult for customers (i.e. the consumers of your software project) to precisely describe their needs sufficiently to provide an exact list of what they need. Most people have a tough time imagining a software solution sufficiently without a concrete example. Generally they must be guided in an iterative manner to come to a decision point; usually they need to actually get hands on with at least a minimal representation of the software in order to "see" whether it will meet their needs.
The Waterfall/SLDC methodology requires that each step in the process be complete before the following step can reasonably start. You can't start development before all the requirements are known and documented. In theory this should be possible; in practice I've rarely seen it happen except in limited circumstances.
I once knew a guy in the Bay Area who specialized in writing SCSI drivers. He was an expert, and basically did the exact same thing, with minor changes, over and over (plus he made enough to spend 6 months every year in Alaska on vacation, but that's another story!). This type of situation, where everything can be specified up front as there are no real unknowns, makes such a rigid process work. If I write a general ledger for the seventh time in my life, I bet there is little different than the sixth time. If all I am doing is some maintenance on an existing application, I would expect that changes could be documented ahead of time.
If I am writing a brand new system (like I am now at work) there are way too many unknowns to do any kind of up front specification and even design. I don't even know how I will accomplish even the most basic requirements until I have spent some time trying approaches. My "customer" the company is not sure what new products and services that might be possible if the new system can be flexible or scaleable enough. We are replacing an existing system, but it has so many problems and irritations that no one wants to simply replicate it. In this environment all we can do it start with a basic set of needs, build enough to see if the basic architecture works (it has to scale 10x at a minimum of our existing one), then examine how it can be expanded as people see what it can do. There is only one way to work with something unknown and that is iteratively.
This is a hard thing to explain to people for whom waterfall is an embedded part of their psyche. Of course it is also hard to tell people why most of our projects are failing as well. Generally the response to failed waterfall projects is more requirements, more documentation, more analysis. When I was at First Command, we had a 15 step (meetings, documents, signoffs) process before any code was actually written. You would think it would keep projects from happening, and it did (on purpose I always thought).
Story, Sabre Corporate Air Pricing
So how does an iterative, customer involved approach work? As an example, take my experience building a customer facing and internal workflow application for the Sabre Corporate Air Pricing group. Back in 2000, this group (who's job was taking information about flight discounts from travel agencies serving Corporations, and programming them into the Sabre Reservation System). They had about 100 or so people who did the work of "coding". They used faxes and emailed documents from the agencies, and managed the workload by the "drop files on chairs" method. Agencies never knew who long it would take, errors were common and corrections took a long time. Meanwhile errors in discounts were caught by the airlines, and Sabre had to eat the penalties (millions of dollars a year worth). Yet Sabre IT had no interest in helping this group automate; they did write a minimal spec for a web app for the agencies but nothing more. So the group finally had enough and hired the consulting firm I worked for to build the agency facing web app.
Since I was the only person who knew JSP (which Sabre had started using) I worked on that app, completed it per the spec, and then asked a simple question: "OK, now you have data coming from the agencies, what are you going to do with it?". IT had no interest in the question, and the CAP group didn't know anything about web applications at all.
At this point in a normal waterfall you would gather requirements and have them sign off on them. Fortunately I was allowed to do my own thing, and together with another engineer, Dave, we proceeded to learn their business sufficiently to see what was needed. I spent a lot of time explaining what a web application could be made to do, and built numerous HTML prototypes to show them what was possible, and as they began to understand, we jointly decided that they needed the public web app connected to a backend workflow system to manage the workload, and connect the work in progress with status visible to the originating agency, and keep a full audit system also visible to all parties. Over sixth months the two of us (with a bit of help from another) build an application with about 70 jsp pages and a complex Oracle database to back it up.
Then IT finally woke up and decided to get involved (by this time the app was working and in production use by a few customers as a beta). They then spent the next 6 months following their own methodology to add the login page. So we spent the extra time adding many more features and getting more and more beta testers using the system (the agencies loved it).
Sabre basically paid for this system immediately as now they could distinguish internal errors from customer errors and no longer had to pay many penalties at all. In the following two years the system was eventually extended to completely automate the process and (sadly) eliminated the entire department to an enormous savings.
All from a simple question and an iterative and interactive approach to the discovery of what the customer actually needed. Sometimes the only way to find out what a customer needs is to build it over and over again until the app and the needs converge. Agile, scrum or iterative; whatever it's called it's the best way to build something new.
Then no one needs to die pointlessly in a waterfall.