An update on review scoring - The Verge

Every once in a while, we need to evaluate our review scoring parameters and make adjustments to how we do this. This year, we’ve moved to a 10-point scale and reset what each of those points means.

The most contentious part of any review is the numeric score that’s given to the product, which reduces our evaluation of how a product performs in what is often a very competitive space to a value on a 10-point scale. We have often talked about getting rid of our scores, but readers tell us they often look to the score first when reading a review and rely on scores for their purchasing decisions.

As a result, we take scoring very seriously here at The Verge. We know how much weight each score has on the perception of a product and how much conversation there is about Verge scores. Reviewers pitch scores to their editors and those scores are often hotly debated before we publish a review. Our aim is to make sure that the score provides valuable information to our audience that they can use to better understand how a product performs.

As time goes on, our average score for a category of product tends to gets higher, something we call score inflation. That’s partially because of positive progress in the industry — today’s devices are better than yesterday’s — but score inflation can also change the meaning of a score. A rating of six on our scale is actually a good score — it means a product is better than average but still might have a few issues — but if readers only ever see sevens and eights, they will assume that a six is a much harsher score than it actually is. Every once in a while, we need to revisit our scoring procedures and see if we’re being critical enough across the board.

And so, that’s what we’ve done. To tamp down score inflation, we’ve started using more of the scoring scale as we’ve rated devices. That translates to seeing lower scores on reviews more frequently than you might have in recent years and devices that are better than others getting lower scores than they might have in the past. In fact, we’ve already been doing this for the past few months; if you’re a regular reader you might have noticed the change.

We’ve also moved to a 10-point, whole number scale, which we feel is simpler to understand and encourages using more of the scale than when we used a 20-point or 100-point system. Now, when we’re debating between two scores, we often will give the tie to the lower score as opposed to using a half-point compromise.

https://www.theverge.com/23286429/review-score-update-processes

---