What You Don't Know Will Hurt You

For three different jobs in a row I've had to convince the team that tracking crashes in their apps was not only a good idea but necessary. Why this isn't obvious to everyone is not obvious to me.

Even on StackOverflow, answering questions on iOS development, I am constantly amazed at people who don't even know there are services that collect and symbolicate your crashes.

I learned a long time ago that most people will not complain and just assume app crashes are common and to be tolerated, although these days a small vocal percentage will take the time to leave a bad review in an app store.

If you don't know crashes are occurring you are creating a terrible impression of your competence, creating potentially crippling support issues, and even worse you may continue to add new functionality to a code base that becomes less and less stable. Ignorance of quality problems is not bliss it's the road to perdition.

Twenty-five year ago there was no way to track and report on the crashes in your application; the only way you could know what customers were experiencing was if they called in by phone, and then you likely got crash reports with almost no information (I remember one in Deltagraph: "All of a sudden I launch the program and it crashes." I was able to fix it even with only that single report, but that's rare).

At my current job and in the last I convinced them to use Crashlytics, which neatly reports and tracks every crash you get, complete with symbols and app data, leaving you with almost everything you need to fix the problem. There are a number of these services. Even if they cost money (which Crashlytics oddly doesn't), they would be worth every penny. Combine crash reporting with enough analytics to know what the customer was doing when the crash happens, and fixing the problem is almost automatic.

Even web applications, where collecting log information is so easy, are not always tracked very well. My last employer's main web application (from which 90% of the revenue came) had an exception for years where if you entered a name with a '.' in it, like Dr.,  the order form would throw an exception; this was not discovered until someone added Tealeaf (IBM product to record how people use an application) showing the problem occurring. Every single time a name with a period in it would make the order form crash, leading to an angry customer and a loss of revenue. The exception was logged, yet no one ever looked at the logs or investigated further.

Before we added Crashlytics to our iPad app, which was not written by us at that previous employer, the 1.0 version we shipped had no crash reporting, and the iTunes Connect feature which supposedly recorded crashes if you had nothing else installed, would not display any reports; I assumed because the volume was too great! Once we were able to ship the next version with Crashlytics, I found a lot of the same crash happened all the time and it was an easy fix.

The original main iOS app we had (that dated all the way back to the first iOS SDK) had 12,000 one-star reviews, which when I was hired was a big concern but no one knew what was going wrong. Eventually we had a 3 month bug fix project for that app which helped a lot (between the Analyzer and fixing compiler warnings there were nearly 800 issues) and we also added Crashlytics and we could see the app crashes which were much more reasonable and these could be fixed quickly. Yet for 3 years the crashiness of the app was a huge problem that they couldn't fix for lack of information.

Of course you can't just track crashes, you have to pay attention to them, and then try to fix them. Following crashes on a regular basis is simply common sense, which sometimes isn't all that common. Fixing the problems is great, but you can't simply fix them, you have to watch when you release new versions to ensure they don't happen again, and to see if anything new happens. I don't know who does this in most companies, but as a programmer I've always watch crashed reports, reviews, analytics and anything else I can get my hands on to see the state of anything I was involved in, even if it's not my direct job. To me this is simply part of being a programmer.

One great thing we eventually did at that previous job, where all of our apps were e-commerce travel related, was to add parameters to every API call in the apps, which was logged in our mobile API, so that any issues could be tracked from device through the path of calls to eventually a booking. The company finally spent some money on Splunk (a log tracker system) and all logs were fed into it. This way we could follow an issue all the way to the back end. Having so much information made fixing things so much easier.

The whole point of all of this is to make it simple to track and report on anything negative that the customer might be experiencing, and have all the information you need to fix it. If you don't do all these things you may as well be coding blind with no hands.

At my current job we have three apps with iOS and Android versions, all used by our field staff every day. Before I came there was of course no crash reporting or even analytics. Once I added Crashlytics to the iOS apps (and the Android programmer did the same to his) we found a few crashes in the iOS apps (which were written externally long before I came) which I fixed immediately. The Android apps had a host of issues, most of which we had no idea of. When people work for you they are even less likely to report crashes. It was amazing how patient these folks were when their job was made painful.

Now I can see that the iOS apps rarely crash at all, and the Android programmer is working his way through the problems (he didn't write the original code either). This makes management people much happier; they can see for themselves that things are working well.

Of course you should strive to produce things that don't crash at all but that's a whole other discussion. No matter how well you do that, you still need to know what the customer is experiencing even if the answer is "everything's perfect." There is no comfort in ignorance.