Programming Without A Net

I must admit that the thing I enjoy most in programming is doing something no one knows how to do, something that no query on stackoverflow.com finds an answer, something that's completely befuddling everyone around.

Today that's harder to find, everything is public, everyone posts descriptions of even the most trivial ideas and Google finds them. In the 80's almost everything you did was new, and unknown, and there wasn't anywhere to get help or anyone to ask. No Google, no open source, nothing.

Yet working on something that seems impossible, at least to anyone you know, is a real rush. It's your brain, your creativity, your imagination against an immovable problem. If you fail it was inevitable; if you succeed an awesome feeling.

When the three of us sat around after I started Data Tailor in 1985, to build an odd spreadsheet like application eventually called Trapeze, all we had was my initial idea, that you could calculate things not with cell addressing, but with real names attached to blocks of cells. No one had ever tried to do something so odd, and even regular spreadsheets were still fairly new and immature. So we invented it, named blocks that sized themselves to their inputs, so that you could do profit = sales - expenses and no matter what you did to the data everything was always correct. It was pretty cool, it made a cool app, but we knew nothing of marketing (talk about impossible) and competing with Excel with something different was, basically, dumb.

But the joy of creating something unique was still great, I've still gotten emails from former customers over the years about how much they loved the product.

Of course today you can still build something with no road map, no one to guide you, going off into the final frontier, but it's not quite as amazing since there are so many apps, so many blogs, so much information it's much more refining and less raw invention.

When I built a Postscript library for Deltagraph in 1988, there were only a handful of apps that handled Postscript natively, namely Pagemaker, Illustrator and Freehand. I had a book, a laser printer and a ton of paper. I had no development environment or debugger, just paper. I screwed around with the libraries in those apps for weeks trying to uncover the secrets. Eventually I got it built, and also was able to write Illustrator files which became Deltagraph's biggest feature. It was a lot of fun, programming in the dark (or light, when the paper remained white and nothing came out!).

Programming where you have no idea what to do, or any clue on what direction to go, can be pretty intimidating, and I know a lot of programmers don't like it much. Failure isn't always fun and when it looks impossible it often is, and it's a sucky feeling to waste a bunch of time getting nowhere. But like people who climb mountains or base jump or swim with sharks, there is a joy in tackling something crazy or at least really hard even if it eventually crashes and burns. I have no desire to do those kinds of things, programming isn't usually able to kill you. But the focus, the intense imagination you have to bring to some impossible problem is probably a similar rush.

Of course succeeding is even better, like getting to the top of a mountain. But I think the test of your ability is the real thing.

Back in 1998 a consumer magazine (not the good one) said they would have a fabulous search engine for products and especially vehicles in one year. They hired a consulting firm to work on it. After six months the firm said it was impossible and gave up. Our president said we could do it. Three months or so later and a lot of code and database work and a search engine specialist and it took 60 seconds to do even a single query on production hardware. There were 6 weeks remaining before that 1 year deadline and everyone was in a panic.

The reason it was difficult was that at the time (things are a bit simpler today) automobiles had base models, and optional features were grouped into packages. These packages had rules where you couldn't combine some, other were required and many combinations were exchangeable. For example if you had a package with an engine you couldn't have any other package with an engine. The search engine allowed you to specify features and basic model choices and a cost level, and it would determine what vehicles could be legally configured to satisfy your requests. But the combination of the database model and the code was incredibly complicated, slow, and gave the wrong answers. 60 seconds for a search was unusably slow.

I wasn't involved, I was working on another project. But it looked interesting, and everyone's inability to fix the problem was intriguing so I volunteered to "take a look". I did something different, I actually analyzed the raw data which apparently no one had done in their hurry to build something quick.

I saw that the data was a kind of tree, where each leaf was a logical operation starting from each feature (I wish I had saved the source code, I barely remember it now). So I extracted the data into text files, processed it into a partial tree, and then in the application loaded the data into memory where I could search it quickly by walking the trees. I wrote a new search engine (in C, the app was Objective-C/WebObjects) which exactly replaced the existing one and otherwise made no changes. I even had to sub-allocate memory since the HP/UX memory allocator was dreadful. Having written a commercial allocator I had plenty of pieces to build on.

In four weeks I got it done, we plugged it in and it worked perfectly for the next 18 months as the site went live on time. It was probably the most fun I ever had programming despite the pressure; no one ever found a bug in the code. Eventually the dot-com collapse happened, the magazine failed to pay us (they had a separate entity so the magazine wasn't hurt) and we went belly up like so many.

I still remember how hard the problem was (this discussion leaves out a lot of detail) and how unlikely everyone thought it would ever ship. It's not like I'm some inhuman genius; it's just I wasn't afraid of the problem or failing, loved the challenge, and got lucky that it made sense.

So the next time impossible stares you in the face, stare back. You can build just about anything if you let your imagination and creativity loose. Sometimes today we can be overwhelmed with too much helpful information since it's everywhere. Maybe it's the absence of help that greases the brain into coming up with good solutions; I don't know if it is, but if you want to have fun as a programmer, beating an impossible type problem is the best feeling you can get.