Stars, Stars, Everywhere are Stars

A couple months ago, David Antrobus had a post titled, My God, It’s Full of Stars. His point in this post was that an author (or reader for that matter) shouldn’t get too hung up on the numerical rating of a review, instead focusing on what the review says. He was right for many reasons, not least of which is that one person’s garbage is another’s treasure. At least that’s the only explanation I’ve found for the yard sale signs that spring up every weekend.

David’s post generated a lot of good discussion. One subject that came up among those commenting without reaching a consensus had to do with the expected distribution of reviews among the different ratings. In this post, I’m going to give my theory, which ultimately leads to the same conclusion as David’s post, that no one, authors or potential readers, should give much weight to the number of stars, instead concentrating on review content.

Most people, when presented with a rating system, picture a distribution that resembles a bell curve. That a typical five star reviewing scheme cleanly maps to the A, B, C, D, or F grades many of us received in our school days, only reinforces that thought. Other people (the easy graders?) picture a flatter distribution, with the most extreme allocating a fifth, twenty percent, to each possible rating or grade. As a starting point, either of these distributions or anywhere in between seems logical.

What picturing these distributions fails to take into account is that they apply to the entire universe of items being graded, the millions of books available. If our purchases were random, our reviews and the reviews of others would logically look like one of these distributions, but our purchases aren’t random. Instead, as readers, we use all kinds of techniques to exclude those books we are less likely to enjoy and pick those more likely to be our thing. We read the book description and, if it doesn’t appeal to us, move on. We choose the new book from an author if we liked her last one and skip the next from the author whose last book we didn’t like. We exchange recommendations with fellow readers, read reviews, and use many other ever-changing techniques to identify those books most suited to our reading requirements and tastes. Even book bloggers such as me have choices and are more likely to pick the book we’ll like than the one we won’t to read and review. Anyone who reviews all the books they read and has a rating distribution that resembles a bell curve is showing they’re either being too tough in their reviews or are exposing themselves as poor shoppers. However, it also makes the average of most customer reviews look inflated to anyone thinking of a bell curve distribution.

Amazon’s rating system exacerbates this. Here is how they define each of the star ratings.

1 star – I Hate It
2 star – I Didn’t Like It
3 star – It’s Ok
4 star – I Like It
5 star – I Love It

Among customers who are good shoppers, one and two star reviews should be rare. Unless they do a bad job of picking books to read, if they follow Amazon’s rating instructions, the majority of reviews should be four stars with their best choices receiving five stars and the books that didn’t quite make the grade as three stars. I tested my theory, randomly selecting a significant number of people rated “top reviewers” on Amazon. I threw out those who never gave negative reviews, which was almost half (yet another reason average review ratings seem high). Among those remaining, the average review rating was 4.2. The design of Amazon’s rating system gives most products a relatively high average rating. Between this, a subset of customers who only write reviews for products they like (which I’m willing to bet is higher than the other subset, who only write reviews to complain), it isn’t surprising that the average rating feels high to many people and that review ratings are of little value for evaluating a potential book purchase?

Author: Big Al

Big Al (who insists he only has one name, like Cher, Sting, and Madonna) spends his days writing computer programs that are full of typos, homonym errors, and incorrect verb usage. During his evenings, he writes reviews of indie books for BigAl’s Books and Pals and has recently taken over The IndieView, a website founded by indie author Simon Royle as a resource for indie authors, indie reviewers, and those who read either.

16 thoughts on “Stars, Stars, Everywhere are Stars”

  1. Reviews are so subjective and just a person’s opinion. What one person will like, another won’t. It is funny to place a star rating on reviews because I received a wonderful review where upon the reviewer recommended my book for USA Today’s Happy Ever Blog as a recommended read, but then they placed a 3 star rating on it. I had not seen the review listed on their website/blog after their recommendation in Dec 2011, so recently asked about it. They posted the exact review but when I went to look was surprised at the 3 star rating. So it made me wonder, since this was not an Amazon review specifically, what their star rating system is, but I didn’t ask or see it posted anywhere on their blog. Anyway, it seems to me, almost everyone has a different opinion on what their star ratings mean, and how vastly different they are to someone else’s’; just as reviews for one book can be vastly different.

    I just have to say I received a review the other day and out of 18 so far, this reviewer got exactly what my book was about, what I was trying to convey and she understood where I went with it. It is wonderful to read a review from a reviewer who ‘gets you’.

    Thanks Al.

    1. Thanks for the comments, Jacqueline. You’re right about a review being one person’s opinion. Something we all need to keep in mind.

  2. Great post, Al! I completely agree with you. I don’t love the star system, but I don’t see it going away any time soon with major retailers, if ever. I do want to point out that I believe another major source of 1-3 stars occurs when authors have built up a decent catalog. The reviews for their books go from being about the single book in question to a comparison of it to the author’s other works. Whether it should be done that way or not, it seems to happen a lot.

    1. I agree, Brian. It isn’t going away anytime soon. When I was setting up my blog I gave serious consideration to not using stars, but realized if I was going to post the reviews anywhere else I’d have to give them a star rating. My solution was to use them, but to put the stars at the very end of the review in an attempt to remove emphasis from them.

    1. Thanks, Lynne. It gave me an excuse to number crunch. Much easier than trying to string words to make sense. 🙂

  5. I enjoyed this post Al. And yes, the stats are skewed. If such subjective data were being used in a proper statistical analysis then all the other variables would be taken into account. As it is, the ratings are virtually meaningless.

    I refuse to post ratings unless I’m trying to make a special point. As I only review books I haven’t disliked, my numbers would only represent /how much/ I’d liked a particular book.

    For my money bad stats are worse than no stats at all.

  6. Another factor is that different reviewers view the system and apply it each according to their interpretation of it, and the reader also does that, so that the match between how the readerr responds to the rating and what the reviewer intended may be in different universes entirely. Bottom line – the system doesn’t work.

  8. Hey, I know I replied to this! Can’t remember what I said, now, but thanks for the post, Big Al, and for acknowledging my slightly annoying take on all this. 😉

Comments are closed.