Online reviews are really important for small businesses. Good reviews mean more customers.
Sometimes, people try to harm a business by giving it tons of bad reviews because they don't like the owner's political views or something the business did. This is called "review bombing."
Websites like Yelp try to stop this by deleting these bad reviews. But a new study found that this can be unfair.
Here's why:
Real people can be silenced: Sometimes people leave good reviews to support a business they agree with. But these reviews might get deleted because the website thinks they are part of a "review bomb."
Different situations: Websites might delete bad reviews from people far away who don't even know the business. But they might also delete good reviews from local people who really support the business.
The study says websites need to be more careful about deleting reviews. They shouldn't just get rid of them because they are about politics. They need to make sure they are being fair to businesses and customers.
Sanitizing can be constraining
The findings come from a new open-access study by a Rutgers researcher.
“Simply put, everything you think you know about review bombing is wrong,” said Will B. Payne, assistant professor of geographic information science at Rutgers and author of the study, published in the journal Big Data & Society.
Online reviews can have a significant impact on an independent business’s revenue, particularly those on Yelp, the leading local review platform in the United States. One study found that a one-star increase in the average Yelp rating causes a 5% to 9% increase in revenue for nonchain restaurants.
To understand the geographic reach of review bombing incidents and how platforms define acceptable speech, Payne assessed Yelp’s moderation of comments on U.S. businesses embroiled in political controversies between 2004 and 2021.
He selected two businesses with large numbers of Yelp reviews for in-depth analysis: Washington, D.C.-based pizzeria Comet Ping Pong (subject of the Pizzagate conspiracy theory in 2016) and St. Louis-based Pi Pizzeria, whose owner, Chris Sommers, became the target of online and offline harassment by pro-police supporters after he publicly backed the Black Lives Matter movement in 2017.
In Comet Ping Pong’s case, Payne found that review bombing resulted in primarily negative comments by reviewers mostly on the West Coast – thousands of miles away from the restaurant – while Pi Pizzeria experienced a much more local pattern (largely from the St. Louis area), with an even split of supporters and detractors.
Same response, different results
Payne found that Yelp’s automated and human review filtering systems largely responded the same way to each incident, but with considerably different effects.
For Comet Ping Pong, of the 283 reviews flagged and removed by Yelp, 229 were negative one-star reviews. By contrast, of the 588 Pi Pizzeria reviews that Yelp removed, most were in support of Sommers’ actions, positive reviews that averaged close to the restaurant’s four-star rating of Yelp-approved reviews.
“Local customers were censored for simply thanking Chris Sommers for standing with them as they marched against police violence,” Payne said. “They weren’t fake reviews about a conspiracy theory; they were legitimate statements by people supporting a business, in this case for the support its owner gave to the neighborhood.”
Payne also looked at Google’s approach to content moderation and found that unlike Yelp, Google rarely removes politically themed reviews. This, too, can be a double-edged sword; Comet Ping Pong still has dozens of public Google reviews referencing the false Pizzagate conspiracy.
The data does have several limitations, Payne said. First is the possibility that the self-reported location of Yelp users was inaccurate, or that some users could have moved between the time they set up their Yelp profile and when they wrote a review.
Additionally, reviews on Google Maps – a popular Yelp competitor – don’t contain user location information and can be removed by Google without leaving the public metadata traces that Yelp provides for transparency.
Moderation gone too far?
As review bombing continues to test review platforms’ approaches to political discourse – the most recent example surfaced this month, when Yelp halted reviews of a McDonald’s franchise in Feasterville, Penn., where former President Donald J. Trump had held a campaign event – Payne said it’s worth considering whether content moderation has gone too far.
The question is particularly relevant for Yelp, which has used corporate communications and review search filters to support Black-owned, women-owned, and LGBTQ-inclusive businesses – speech that isn’t permitted by reviewers themselves unless accompanying a customer experience review.
“Having a one-size-fits-all, review bombing or political speech policy can lead to the suppression of legitimate expressions of support for the role a small business plays in the community, as in the case of Pi Pizzeria,” Payne said. “Some might disagree that the political positions of a business owner should guide consumer behavior, but on Yelp, it’s a choice that users can’t even make for themselves.”
About the study
To conduct the study, Payne created a database of businesses affected by national and local politics.
Using news sources to identify specific cases and date ranges, he built a dataset of tens of thousands of political-themed reviews. Topics included the 2016 and 2020 U.S. elections, the Black Lives Matter and #MeToo movements and the COVID-19 pandemic.
Next, he analyzed Yelp’s publicly available metadata for reviews of affected businesses, including review date, username, star rating and user location.
The study is published in Big Data & Society.