The Codist

I have been trying to hire a landscaping company since the late spring to work on my house, and so far I haven't had any luck in getting people to call me back, even when they say they will call. I paid one company for a design and I had to chase them down nearly 6 weeks later to even deliver what I paid for. Then they promised to give me an estimate but never called back.

Two others said they would call me and I never heard from them again either. Yet all three show up as solid A companies on angieslist.com on a search for recent grades. If you actually dig into the details of their ratings you often see patterns where there are a whole slew of C, D and F ratings of these A companies.

So how are these ratings meaningful? They aren't. I have found some really good people that gave me awesome service and I would use them again in a heartbeat, and I do give A ratings. But I intend to rate the company in the first paragraph with a F as soon as I can find someone to actually do the work since I am not sure how easily companies can tell how many F's you give (note I haven't given any yet).

In any rating system you expect to find some kind of spread of values, hopefully centered around the most likely. Any time people communicate you will find a few times where things didn't work out or something got in the way or the rater was a troll. Given enough data you expect to see a curve that gives you an idea of the mean and deviation.

Personally I think angieslist.com has a selective filter on the data to provide the "recent grades/reviews" so that a few bad reviews don't colorize the result. But in trying to be fair (if that is even their intention) they make it difficult to actually find a good vendor. To really know what the vendor is most likely to be like (mean) and how consistent they are (deviation) you have to manually read through a whole list of reviews one at a time.

The first company has something like 250 reviews. In the list you see A - 250 with 20 near me. Then you click on that and see the overall data which shows A's across the board. Yet look below and I see currently C, D, F, C and A for work in the last month. This makes no sense, it looks like a company in trouble. Can you trust Angieslist.com given this type of "math"?

Rating systems are always hard to design. The story behind angiestlist.com is that you could "trust" the ratings and reviews since it only comes from customers not vendors.

How ratings are presented can make a huge difference in how a reader might perceive them. Ratings from long ago might not be relevant anymore or they might be more important. Ratings from non-customers might slew the results. Trolls might leave bad ratings on purpose or competitors might game the numbers.

The App Store rating system is fairly often bashed. Generally you only see the rating of the last release unless you look deeper so people with frequent releases can hide a world of sins. Great apps that have one bad release might make it look like the app is terrible. People pay to have their apps rated highly and to be downloaded more skewing the results.

Before I came to work at my current (soon to be former, marketing company only now using parter technology) job the existing iPhone app had a record 10,000+ one star reviews due to a lot of crashes and problems over several years, not all with the app (the backend systems were pretty awful). I took the job because I knew I could help and did, after a lot of work that app didn't get many more 1 stars and the new from scratch replacement app got almost none. Now that app was replaced by the partner's existing app rebranded as ours. Yet it's the same bundle ID so if you look at the ratings you see 6 years of them for 3 completely different apps. Does the deep rating make sense at all? Is the 2.5 star rating of their recent release fair at all since people upgrading (all the prior apps no longer work, our servers are all gone) are venting that this app is missing features they had previously?

Yet you have to have some way to identify things you want to buy or use other than wading through pages of text reviews (which people hate to read and even more write). A "grade" or "star rating" makes it simpler but at the cost of truth and applicability.

Hotels and especially Tripadvisor.com have this issue as well, how do you police the ratings to eliminate irrelevancy and trolls while keeping the rating honest? How do you display ratings in a fair way yet still keep to the idea of mean and deviation without misleading the customer? Of course for some businesses it might be good to design a system to deliberately mislead the customer, however deviant that might be to the general public.

Rating systems are sometimes like marketing: give people an idea but make sure you control the reaction even if it means fudging the truth.

If angieslist.com spends too much effort in providing a clear picture of both the negative and positive sides of a vendor (which is their marketing talk) then anything negative might make the vendor leave or not participate in whatever brings in money. Yet if they completely cover anything bad up, they risk customers giving up on using it. I bet this pressure generally leads to less transparency like showing a page with all A's yet the recent detail is more like D. When you are a public company any negative financial incentive is a potential problem.

In the App Store case ratings are not as important, as Apple is getting a cut of everyone's sales, and the App Store ecosystem has so many ancillary benefits like selling more iPhones. Ratings themselves do little to keep people from trying an app. Despite all the 1 star reviews of that old app our mobile team still doubled sales every year for more than 5 years. We didn't make money from the app but what it allowed people to buy, and not all competitors are fungible; strangely people will use a terrible app if they really do want the service. I sometimes see games in the top hundred grossing list with low ratings yet people keep downloading and giving them money anyway.

So I am still trying to get a landscaper. Like the story of the infinite monkeys writing Shakespeare eventually I will find one who actually wants my business. Either that or I will hire the monkeys. Who knows, I might give them an A.