Before I poke waterfalls with a stick, a couple of stories.

Story, Waterfall One

Project Manager: What are your requirements?

Customer: I want to float on the Niagara river.

Project Manager: What would you like to float with?

Customer: A boat?

Project Manager: What type of boat do you require?

Customer: Maybe a canoe?

Project Manager: OK, sign here and we will build you a fine canoe!

[Later]

Project Manager: Time for the customer acceptance test, please try it out.

Customer: Hey it works great.

Project Manager: OK, project complete, good luck!

[Customer floats down the river, falls over the edge of the waterfall]

Customer: Wait, this isn't what I need!

Project Manager: Sorry, you agreed to it. Request a change order...

[Customer dies horrible death]

Story, Waterfall Two

Project Manager: What are your requirements?

Customer: I want to float on the Niagara river.

Project Manager: What would you like to float with?

Customer: A boat?

Project Manager: What type of boat do you think you might need?

Customer: Maybe a canoe?

Project Manager: OK, we'll try that out first.

[Team builds a canoe, floats it on the river and it crashes over the falls and breaks]

Project Manager: Hmm, that wasn't the right thing. Maybe an enclosed boat?

Customer: Like a barrel, maybe?

Project Manager: OK, for our next iteration, a barrel.

[Team builds a barrel, puts dummy inside, pushes it over the falls, barrel survives but dummy is cracked]

Project Manager: Hmm, that wasn't quite the right thing. Maybe we pad it heavily?

Customer: OK

[Team builds a padded barrel, puts dummy inside, pushes it over the falls, barrel survives and the dummy is OK]

Customer: That's the thing, I'll take it!

[Customer floats down the river, falls over the edge of the waterfall and survives]

Customer: That's exactly what I needed, thanks!

Discussion

Exaggerated stories aside, the fundamental difference between the venerable (and sadly popular) Waterfall methodology (with its pretty pig-name, SDLC) and the more modern iterative approaches boils down to one main point:

"Customers don't know what they want, much less what they need, until they see it."

In all my almost 27 years in this business, I've learned this over and over again. It's very difficult for customers (i.e. the consumers of your software project) to precisely describe their needs sufficiently to provide an exact list of what they need. Most people have a tough time imagining a software solution sufficiently without a concrete example. Generally they must be guided in an iterative manner to come to a decision point; usually they need to actually get hands on with at least a minimal representation of the software in order to "see" whether it will meet their needs.

The Waterfall/SLDC methodology requires that each step in the process be complete before the following step can reasonably start. You can't start development before all the requirements are known and documented. In theory this should be possible; in practice I've rarely seen it happen except in limited circumstances.

I once knew a guy in the Bay Area who specialized in writing SCSI drivers. He was an expert, and basically did the exact same thing, with minor changes, over and over (plus he made enough to spend 6 months every year in Alaska on vacation, but that's another story!). This type of situation, where everything can be specified up front as there are no real unknowns, makes such a rigid process work. If I write a general ledger for the seventh time in my life, I bet there is little different than the sixth time. If all I am doing is some maintenance on an existing application, I would expect that changes could be documented ahead of time.

If I am writing a brand new system (like I am now at work) there are way too many unknowns to do any kind of up front specification and even design. I don't even know how I will accomplish even the most basic requirements until I have spent some time trying approaches. My "customer" the company is not sure what new products and services that might be possible if the new system can be flexible or scaleable enough. We are replacing an existing system, but it has so many problems and irritations that no one wants to simply replicate it. In this environment all we can do it start with a basic set of needs, build enough to see if the basic architecture works (it has to scale 10x at a minimum of our existing one), then examine how it can be expanded as people see what it can do. There is only one way to work with something unknown and that is iteratively.

This is a hard thing to explain to people for whom waterfall is an embedded part of their psyche. Of course it is also hard to tell people why most of our projects are failing as well. Generally the response to failed waterfall projects is more requirements, more documentation, more analysis. When I was at First Command, we had a 15 step (meetings, documents, signoffs) process before any code was actually written. You would think it would keep projects from happening, and it did (on purpose I always thought).

Story, Sabre Corporate Air Pricing

So how does an iterative, customer involved approach work? As an example, take my experience building a customer facing and internal workflow application for the Sabre Corporate Air Pricing group. Back in 2000, this group (who's job was taking information about flight discounts from travel agencies serving Corporations, and programming them into the Sabre Reservation System). They had about 100 or so people who did the work of "coding". They used faxes and emailed documents from the agencies, and managed the workload by the "drop files on chairs" method. Agencies never knew who long it would take, errors were common and corrections took a long time. Meanwhile errors in discounts were caught by the airlines, and Sabre had to eat the penalties (millions of dollars a year worth). Yet Sabre IT had no interest in helping this group automate; they did write a minimal spec for a web app for the agencies but nothing more. So the group finally had enough and hired the consulting firm I worked for to build the agency facing web app.

Since I was the only person who knew JSP (which Sabre had started using) I worked on that app, completed it per the spec, and then asked a simple question: "OK, now you have data coming from the agencies, what are you going to do with it?". IT had no interest in the question, and the CAP group didn't know anything about web applications at all.

At this point in a normal waterfall you would gather requirements and have them sign off on them. Fortunately I was allowed to do my own thing, and together with another engineer, Dave, we proceeded to learn their business sufficiently to see what was needed. I spent a lot of time explaining what a web application could be made to do, and built numerous HTML prototypes to show them what was possible, and as they began to understand, we jointly decided that they needed the public web app connected to a backend workflow system to manage the workload, and connect the work in progress with status visible to the originating agency, and keep a full audit system also visible to all parties. Over sixth months the two of us (with a bit of help from another) build an application with about 70 jsp pages and a complex Oracle database to back it up.

Then IT finally woke up and decided to get involved (by this time the app was working and in production use by a few customers as a beta). They then spent the next 6 months following their own methodology to add the login page. So we spent the extra time adding many more features and getting more and more beta testers using the system (the agencies loved it).

Sabre basically paid for this system immediately as now they could distinguish internal errors from customer errors and no longer had to pay many penalties at all. In the following two years the system was eventually extended to completely automate the process and (sadly) eliminated the entire department to an enormous savings.

All from a simple question and an iterative and interactive approach to the discovery of what the customer actually needed. Sometimes the only way to find out what a customer needs is to build it over and over again until the app and the needs converge. Agile, scrum or iterative; whatever it's called it's the best way to build something new.

Then no one needs to die pointlessly in a waterfall.

My Tags: methodology

Writing multithreaded code is like juggling chainsaws; amazing when it works and truly sucky when it doesn't.

Right now at my job I am writing the foundation for a transaction processing cluster in Java, so I'm immersed in lots and lots of threads and interacting applications. When you are processing 8000 of something per second, any problems in your approach or in your choice of frameworks is magnified.

In job interviews, a popular question is "what is the major problem you have to solve in writing multithreaded code?" Generally, if they have read a little about it, they often say "avoiding deadlocks". If they have done a bit of thread coding, maybe in Swing, they might say "protected shared data". Only the truly experienced in complex threaded coding will say "avoiding doing nothing".

What's so bad about nothing? Assuming you can avoid deadlocks (generally not hard if you're disciplined) and understand when to protect data, when working in a complex high speed system you want to accomplish both of those basic requirements without slowing down the real work your threads are doing. In a Swing app, so a couple threads block for a while, or even less of a problem is a few waiting around for something to happen. In a large cluster of servers, having threads sitting around doing nothing is a waste of money. If Google is processing millions of searches daily, and half of their cpu's capacity is basically blocked waiting for some resource, it costs a lot of money and power: instead of 50,000 servers you need 100,000. That a hefty price to pay for poor thread coding.

Sure, my company will probably only use 100, and they could afford 200 servers, but why waste resources just to save a few brain cells.

For example, one of my trials used Jetty on one application, and Apache HttpClient on another. With two worker threads on the HttpClient I was able to process my test transactions. When I increased the worker threads to 7, they fell behind; at 10 it was almost dead in the water. WTF? Doing some debugging (with Yourkit Profiler, very nice) the Jetty side was mostly sitting around waiting, but the worker threads in the other app were mostly blocking (as much of 95% of the cpu time). Ah the joys of threaded coding.

The issue turned out to be a synchronized method in HttpClient that deals with Http Parameters, for each parameter for every call on every thread it would pass through this method, creating a common sync point for all the threads. Something that might seem OK for a couple threads was fatal with 10 threads running full tilt at 1000 somethings per second. The chainsaws can be really brutal.

I finally settled on a couple of technologies that work pretty well, on my developer PC (4 core Core Duo something) I was handling 8000 somethings per second at about 85% CPU.

That's another sign of properly avoided nothing, if you slam your creation with unfettered requests and the CPU rises close to 80-95% then you are seeing proper behavior. In my HTTPClient test the CPU rarely got above 25%; mostly a lot of nothing was going on. This was clear with Yourkit. Nothing is not your friend. Don't stop coding when nothing is broken; that's only step one.

Multithreaded coding in a complex application requires a lot of discipline (never assume anything, test everything), experimentation (there is no one way to make it work) and patience (sometimes lots of negative problems lead you to see the solution). It's not easy and it's not quick (never mind the project manager).

Then again neither is juggling chainsaws; plus the downside is pretty nasty.

My Tags: java development performance

I keep reading blog posts where people equate Java with Cobol, that it's a dying language not suited for work today, and that people who use Java are generally less intelligent about programming in general. Given that dinosaurs ruled the earth another 50 million years or so after the Triassic period, I'd say Java still has a long useful lifespan.

Not that it's perfect at all, but I've been using Java for almost 10 years now, and compared to my years of C and C++ I am fairly glad I moved on. I've worked for years starting with Basic, then Fortran, Pascal, Assembly, C, C++, Objective-C and finally to Java; I've seen a steady improvement in my ability to be productive with my language and its entourage (frameworks, libraries and communities). Like the dinosaur, every language eventually falls out of favor as new "mammals" appear but rarely does a language die out completely. Like most animals and plants, they survive and even thrive in different environments way beyond the general evolutionary landscape that might otherwise kill them off. How else can you explain Object Cobol?

For me I find I can do almost anything in Java, provided a reasonable set of choices for the environment I am writing code in. I dislike most of J2EE but there are so many better ways that have sprung up to write Java applications unless you are forced to use the worst of breed "standards". It's not the language that kills so much as the accompanying baggage you have to drag around with you. Java may not be a perfect language design (if such as thing can even be agreed on) but it gets my job done.

Not that I intend to sit on my butt and become dinosaur fodder. Scala, Erlang and Haskell are languages I find fascinating to study, although Scala being JVM based makes it more likely to be useful to me. I think of Groovy as a more laid-back Java, and relatively easy to pick up as well for us Triassic Java coders.

Give me any decent language, and I can write any decent program. The difference is always in how long will it take, how reliable will it be, and how well can it be maintained. That is the mark of a good programmer no matter what they use to write with. Of course the choice of language does make a difference, but choosing the right tools for the job is also a mark of a good programmer. Often you have no choice (other than seek alternative employment or become an advocate for change) and you have to work with what you have. There is real skill in writing good code, and real skill in understanding the strengths and weaknesses of your chosen (or forced upon) toolset.

Often a language which requires a great deal of experience and understanding to master leads the average programmer to fail miserably. I'm sorry, writing complex C++ programs is not for a beginner. Even Java can be difficult (such as at my newest job where a whole set of mainframe programmers face the difficulty of learning a new language or having to leave) if you are not familiar with the concepts of OO programming. Is there a market for people who need a simpler language? Of course there is; but never assume mastery of a simple language or toolset means automatically that you can move on to more complex projects, or even assume your basic language choice can itself handle tougher problems. The world is littered with way too many VB/Access apps on an enterprise scale, for example.

So is there something wrong with learning Java, or using Java, or even improving Java? No there isn't, unless you take the approach that you will stop using Java when they pry it from your cold dead hands. It isn't the final stop on the programming roadway but it's a powerful, useful language with an enormous open source community, great tools, and works for a variety of uses.

Face it, there isn't any one solution to writing code, any more than there was when I started in 1981 writing Fortran. No matter what your favorite new language is, someday it will suck and people will laugh at you. So the trick is to work with the best choices available to you, and keep your eyes open for the new mammals on the block.

If I'm still coding in 50 million years I'll probably not be using Java.

My Tags: java essay languages

In the mid-90's I spent 5 years working exclusively in C++ after years of C. I learned all the ins and out of STL, templates, overloading and the like. Now after 10 years of Java I've spent a little time working with C++ again - working on some game code for WWIIOnline, rewriting my sound code again to support Core Audio in Leopard (and eliminating OpenAL use).

Argh, C++ is so painful to me now, and XCode as an IDE is beyond lame compared to my beloved IntelliJ and even the recently learned Eclipse. Poke me in the eye with a C# stick, it would hurt less.

XCode is the latest generation from Apple, going back to the early days of MPW in the 80's, and then passing through the tools from NEXT. Some days I really feel that the folks at Apple have never worked with a real Java IDE. Other days I feel sorry for them having to carry the baggage of a generation of make files and command line tools around all day. I mean IntelliJ and Eclipse are constantly compiling the Java source while I am typing it in, making it easy for the IDE to spot errors, inconsistencies, provide tips and hints and refactorings on the fly. C++ (and even Objective-C) despite on paper being faster in execution, apparently cannot be compiled fast enough to provide the same level of feedback. With IntelliJ I rarely ever see a compiler error. With Eclipse I rarely even see the compiler at all.

So working in C++ and XCode is really painful for a Java guy like me. Yeah if I was working in Cocoa and Objective-C, it might be more pleasant, but that's not the option here. Even then, in XCode I miss all the help I get from the IDE's that I'm used to. I know people love vi and emacs and swear by command lines but I don't miss the command line these days anymore than I miss the IBM PC XT.

Of course C++ all by itself is a painful language anyway. Sure, I can create the most complicated meta-templated code I'll never understand 5 minutes later but how is that productive? I remember when I first started working on this it took me ages to remember all the requirements for storing objects in a map. Maybe my brain has been too comfortable in Java where putting an object in a HashMap is just that. Maybe the occasional overriding of hashCode() and equals() but nothing much to remember.

Don't get me thinking about memory management. In Java I don't. In C++ you always have to remember when to get rid of stuff. At least in the latest Objective-C they finally added automatic garbage collection. This is the 21st century after all. As Homer Simpson once sang in an episode where he becomes the garbage commissioner "let someone else do it".

I wrote a commercial C++ memory manager in the 90's so I'm well aware of what it takes, and even wrote the whole thing in heavily templated C++, but that was a century ago it seems. Today I'd rather work on the problem at hand instead of futzing with a language and tools that get in the way. Of course for some folks even Java gets in the way (and I understand that as well but it's not a choice for my job yet) but at least it's not C++.

When I read about the newer features being added to C++ I can only think of all the profits the Tylenol folks are going to get. Effective use of C++'s features requires a real dedication to understanding their strengths and weaknesses. It's not a language for occasional use. When I used it everyday it was painful but familiar; today having worked with Java I have a hard time keeping C++ features in my head.

The bottom line for me is I want the computer to do all the heavy lifting, and let me focus on the actual work to be done with as little annoyance and loss of focus as possible. No matter what language or tools you choose they better meet that requirement (for you) or try something else.

Or break out the pain killers.

My Tags: xcode java c

Amazing, Apple's stock value just exceeded IBM's for the first time in history.

I would have never thought this possible back in 1984.

My Tags: apple