Quality Requires QA

Today I read an analysis of why Facebook's iOS app has such dismal ratings.. Surprise, surprise, they don't believe in QA.  I've spent more than 3 decades writing apps, and I've seen this happen a lot and it always winds up terrible.

Reminds me of a post I wrote 4 years ago QA! QA? We don't need no stinking QA!.

"As you may know, Facebook does not have big QA teams... we believe that developers are responsible for their own code, and they're supposed to write the tests to do that." is what an engineering manager from Facebook said. This makes me laugh, or groan, or cry.

Programmers should test the code that they write, but real testing should be done by professional tester/QA people. This is an engineering discipline just like programming. It's not something you do on the side. Applications are often large complex beasts, and expecting the programmer to do double duty is simply insane. I've written a long post on this exact topic as well: Programmers Should Not Test Their Own Code.

How an organization as large and wealthy as Facebook can be so dumb is hard to imagine. Apple I can see having QA issues: they maintain two operating systems, myriads of separate apps, hardware, clouds, and stuff we don't even get to see. How they even manage to not break everything all the time is a miracle. Of course they should do better, but at least they have a volume argument.

Facebook is clearly a huge infrastructure with enormous data to corral (350,000,000 photos a day or something crazy) but the iOS app is just an app. I write iOS apps, a few were fairly complex e-commerce travel apps at my previous employer. We had an amazing QA team of 3 for our mobile group with about a dozen programmers. They were perfect for the role, professional, ornery, stubborn and even perverse in digging up the most complex bugs. They refused to let anything pass.

Previously, before I started there and during the first year, the CTO refused to hire QA and said much the same thing as the quote above, no QA, engineers test their own code. It was miserable, the original iOS app (written during the first iOS beta by two people with no experience with Objective-C) had a then record 12,000 one star reviews, mostly for crashes. The only actual QA was someone who mostly did web deployments and just tested a few bookings.

After I was there for six months, I taught a group of our Java developers to do iOS to add to the existing two and then lead the team in fixing most of the glaring issues that had been ignored for 3 years (mostly due to a lack of real QA) including a ton of Analyzer errors. We had to test ourselves since we still didn't have anyone. It came out reasonably well, the reviews improved and the crashes were much more minimal at least.

Eventually the CTO left, and we immediately hired 3 real QA folks. A little after this we started on a brand new iOS app to replace the old one, with and slightly behind a fully native Android app as well. As I've always requested, QA was involved from the earliest time the apps did anything useful at all.

We had a hard deadline of the iOS 7 launch day (Apple wanted us for the keynote), but we got it done. Two of the QA folks were dedicated to this, and they tested the entire app every day, not just whatever was new. This is key, as repeated testing will find things you thought you hadn't broken in addition to whatever was new. Importantly you also get a look at anything klunky or confusing since they spent a lot of time in the app. It's boring as hell but people who really care about quality know how important it is. We also addressed any issues immediately instead of waiting for some future date. Letting problems fester too long may make fixing the bug much harder.

We made our deadline with one of our three LOB's and then finished the other two a few weeks later. The app had a fantasticly low crash rate and got high ratings. I think the hard case QA team made it so. The Android app shipped a little later with similar quality.

Of course as I've said before, our parent company essentially sold our brand to our biggest competitor, and all of this vanished. Oh well at least we went out in style.

Although we tried Scrum for a couple months we junked it pretty fast and went back to a Lean process, which meant no fixed iterations and left time for keeping our other apps up to date. The hard core two week sprint schedule is a killer for QA which often has little or no time to do any. Personally I think this is a serious failing of a too rigid process. Maybe that's one reason Facebook does what it does.

Likewise I am not a fan of automated testing being the only thing you do. People are far more imaginative, nasty, perverse and the like and are more likely to do what users do, especially when they really understand how to approach testing. Us programmers are more likely to just look at whatever we wrote and move quickly to get back to coding. An app should be tested as a holistic item, it isn't a collection of disconnected code. All parts contribute to the user's perception of quality, or lack thereof. It should be tested from top to bottom all the time.

Review comments are not a perfect science, but in my experience when people are mad they are likely to comment and leave a bad rating, and when they are happy they don't bother. Our old iOS app had the 12,000 1 star ratings. The new one (for the 8 months it lived) had only a hundred or so ratings period. Virtually all were good ratings at least.

Quality is Not Job 0xff: There is No Good Enough. Maybe it doesn't matter much to Facebook since they so big and there is no real alternative for most people, so a crappy app is good enough. Plus you can always use the web.

As for me, the only good quality is the best quality, and that takes dedicated QA people who demand it.