Shipping

In every project there comes an important time every one of us gets to, assuming it's not cancelled:

Shipping.

Somehow all the work you've slaved over has to be deployed to production, rolled out to customers, pushed to an App Store, or like I did 30 years ago, baked into a master floppy. It's time for your baby to be born, or graduate, or whatever metaphor works for you.

I've done this more times than I can count, and I've never experienced a disaster, even a little one, other than the first one.

First Shipping Day

In January 1987 my startup released its first (and only) product at MacWorld in San Francisco. We had rental Macs set up in our booth. I did a lot of the demos and found … I had a statistical chance of crashing the app every time I chose our central feature: a hierarchical popup menu.

This was the big draw, as we were the first Mac app to feature either a popup menu (one not in the menu bar) or a hierarchical menu. Apple didn't release these in MacOS until a few months later that I recall.

But we had tested things at the office always with a debugger; at the show the Macs didn't have a debugger. In those days all the memory was physical, and the debugger moved things around just enough that the menu worked. Without the debugger a single memory location had a random value which sometimes was just wrong enough to crash the app.

It was pretty embarrassing, but thankfully we had only built a few packages for the show and I fixed it easily afterwards.

Do It Better

This taught me a valuable lesson, that shipping has to be the last thing in a process designed to wind up with a predictable result. All the things you did before that day affect what happens on that day. If you've done everything right, that day should be an easy, no fear step. It just happens to be the last one but it's not a big deal.

If you can make it available and go home without a worry, you've done all the right things. That's what I am used to. At the Travel company (two jobs ago) I let other people push the button to publish our stuff to the App Store, like a little ceremony. We never worried because we were all on board with how we got there, and knew it would function.

We were asked by Apple to be there on iOS 7 launch day, and we only got the final GM the Friday before. But the process of how we built the app, how we tested the app, how we decided on what to do, still worked like a charm. The app went live on that day and other than an Apple crash in MapKit, our code started around 0.14% crash rate and eventually went down to 0.07%. No worries, mate!

This Week!

This week at my present very big employer, 16 months of work is finally going live. For the first time in my life I am worried about what will happen. It's not just an app (or in this case a big chunk of the app), it a massive product spanning hundreds of people's work in every kind of computing system you can think of, from hardware to many many services, databases, mobile apps both external and internal, and a massive army of people. My part is the public iOS app in which the feature will appear one day this week.

It's not just the app by itself though, everything has to work together to produce the "product". While there are contingency plans to deal with all sorts of issues, we can't fix most things on the fly, and of course the public apps least of all. Once we turn it on we can't turn it off again.

This is basically terrifying. If it goes well our customers will be happy; if it blows chunks it might make some seriously bad press.

I have never worked on something this long with no release. But this product has changed so many times it was crazy, when asked what do you want in the first release no one could make up their mind, so they said just build everything and we'll flip some switches at the last minute to turn on what we want! Then they came up with more.

Unlike my previous experiences testing was not done the right way, for some reason we hire cheap contractors in foreign countries to test the public stuff, to save money. I have my doubts when people test for months and report virtually no bugs, I might be pretty good at this, but no one writes perfect code! Over the previous 30 years most of the time I had constant testing during the entire project by people in the same group, and I knew them, and they knew the product backwards and forwards. Having confidence on shipping day is directly related to having continuous testing. Most of what I have written in my life has been apps (native or web-based) used by people. I've always believed that apps used by people should be (mostly) tested by people, and code used by computers tested by computers.

When your testing is holistic (test everything all the time), by the time shipping comes along, its just another day, you know it works, you aren't guessing or hoping. If your product decisions were precise and immediate, you know you have the right product. If your development was pleasant to do, and feedback was constant, you know the code is fine.

If instead you skimp on testing, you have way too many people constantly changing their minds and a process that doesn't support that, plus way too many teams building way too complex code and too many managers and process people and constant demands to meet random deadlines and tons of last second changes—well the end result is unpredictable.

Unpredictable is not what you ever want. Yet I am living in a world I don't control, or even recognize. I can only do the best I can with what I have to work with. This is not my cup of tea.

I do not want to hope everything works perfectly on shipping day.

I want to know, ship it, and go home.