Consumers Driven By Online Reviews Influenced More By Number of Raters Than Ratings

Technologically informed, if not sophisticated, consumers are not necessarily making decisions in ways consistent with marketers understanding of traditional motivations. JL

Katharine Schwab reports in Fast Company:

People tend to choose products that have a greater number of reviews–even if those same products have lower ratings. Since the researchers also found that there isn’t any relationship between a product’s ratings and the number of reviews it gets, the study means that people are making decisions based on popularity, rather than quality. This points to a flaw in ratings from e-commerce: if people aren’t making informed decisions based on ratings because of cognitive bias, those interfaces need to offer ratings in a way that leads people to make better decisions.

How do you decide which salt-and-pepper shakers to buy? Do you choose the set with 2,500 reviews and 4.5 stars, or the set with 500 reviews and 4.9 stars?

According to a new study published in the journal Psychological Science, people tend to choose products that have a greater number of reviews–even if those same products have lower ratings. And since the researchers also found that there isn’t any relationship between a product’s ratings and the number of reviews it gets, the study means that people are making decisions based on popularity, rather than quality.
For Derek Powell, a post-doctoral scholar at Stanford who was the first author on the paper, this is indicative of the changing state of “social learning”–our collective ability to learn from other people’s experiences–in the age of the internet. With the vast quantity of information online, it turns out that people might not be equipped with the decision-making skills to process it. This insight points to a potentially deep flaw in any ratings-based interface, from e-commerce sites like Amazon to food sites like Yelp: if people aren’t making informed decisions based on star-rating systems because of cognitive bias, then perhaps those interfaces need to compensate, offering ratings information in a way that’s more clear and leads people to make better decisions.

[Photo: Walter Zerla/Getty Images]

“Greater connectivity has the potential to supercharge one of the most powerful learning mechanisms afforded by evolution and culture,” Powell writes in the paper. “Yet it also places new demands on people’s abilities to translate numbers into meaningful social cues.”He calls that process of making decisions based on numbers–like the number of reviews or star rating–“intuitive statistics.” To test whether people’s intuitive statistics matched up to actual statistical models, Powell and his colleagues ran an experiment where they asked 132 participants to choose between two similarly priced phone cases that had different ratings and numbers of reviews. The researchers found that people consistently chose the phone case with

greater number of reviews, even if the overall rating wasn’t as good.
“It is persuasive when a product has 1,000 reviews,” Powell says, even if it’s not as highly rated. Rationally, that would suggest that more people thought less of the product, but the results show that people tend to think the popularity of a product is a better reason to buy it than the average rating. There’s definitely truth to it from my perspective–I hadn’t realized it, but I do take the number of reviews into account when choosing something to buy online. Of course, I also look at more than just the rating and the popularity, something the study doesn’t take into account.
While the study only examines people’s decisions based on two numbers and doesn’t look into the impact of other factors like the written content of reviews, it does have implications for both consumers and retailers. As a customer, it’s helpful to know about an internal cognitive bias toward more popular items.

And for retailers and designers, it raises a question about how product pages on giant e-commerce websites like Amazon and Walmart are designed. Powell says that he thinks it’s in these companies’ best interest to make sure their customers buy the best product so they have a good experience and continue using the service, and the study points out that customers may not be making informed decisions on what products they buy because of cognitive bias. The study’s results hint that perhaps companies should look into more ways of displaying review statistics that don’t tap into this psychologically unintentional behavior.
But what are the alternatives? Powell says he and his colleagues are interested in continuing their research into different ways communicating reviews.
What if, he says, reviews were laid out in a histogram, showing the distribution of reviews from one star to five stars instead of a single average? That could potentially help customers make more informed decisions. For instance, a product with a lot of one-star and a lot of five-star reviews for an average of three stars is different than a product with mostly three-star reviews. Powell points to research showing that people tend to respond better to visuals than number-based statistics; more graphics-focused ways of displaying reviews could help people overcome their biases and weigh their options in a more objective way.
For designers, the study serves as a reminder: people don’t always behave rationally or interpret information logically, especially when they’re online shopping.