Can You Trust an Amazon Review?

Reviewers are gaming the system at Amazon and elsewhere for mischief, politics, and profit

Loading the podcast player...

Steven Cherry: Hi, this is Steven Cherry for IEEE Spectrum’s “Techwise Conversations.”

How do you know if something—a book, movie, restaurant—is any good? A hundred thousand new books were published last year alone. We used to only have professional reviews, but now user reviews and user feedback are everywhere. But how trustworthy is it?

We had on the show recently an electronic publishing analyst who noted that people have been gaming the system when it comes to book reviews on Amazon, arranging for favorable reviews, or even writing them themselves, then arranging for masses of people to up-vote the reviews. The reverse has also been happening—people attacking a book and down-voting it. The New York Times recently wrote about one such case, a biography of Michael Jackson. That article was based, in part, on a research paper entitled “Six Degrees of Reputation: The Use and Abuse of Online Review and Recommendation Systems,” by two Cornell-affiliated researchers, Shay David and my guest today, Trevor Pinch.

Trevor Pinch is a professor of science and technology studies in the Cornell University sociology department. His own Ph.D. is from the University of Bath, and he’s written on a wide variety of interests, including quantum mechanics, medicine, and Moog synthesizers. He joins us by phone.

Trevor, welcome to the podcast.

Trevor Pinch: Hi, Steven. Glad to be with you.

Steven Cherry: So at sites like Amazon and Yelp, there’s the reviewers and their reviews. You write that it’s a closed loop of sorts. What do you mean?

Trevor Pinch: By that we mean that there are more and more ways that the actual reviews themselves are assessed by other readers and reviewers, so Amazon—this is where we get the title of our article, the “Six Degrees of Reputation”—Amazon has this kind of nested tiers of different ways of assessing the quality of a review. It does this by little things, of course, by placing the reviews…so the reviews…they’ve changed this system, but the most positive and now the most negative review, which are the most helpful, are now placed at the top.

But also, these reviews can be ranked by other readers as to whether the reviews were helpful or not. So these reviews can be ranked as to whether they were helpful, and then they can be further organized by Amazon into a ranking system of the reviewers themselves. And I think most people know this by now, but you can get badges on Amazon for how successful you are as a reviewer, and they start at a thousand. So the top 1000 can have a badge, or top 500, or 50, or 10, or 5, or if you’re the No. 1 reviewer on Amazon, you’ll have a badge for that.

Steven Cherry: You started down this road as an author yourself who discovered plagiarism in the Amazon reviews of one of your own books. How big a problem is plagiarism, including self-plagiarism?

Trevor Pinch: Yeah, this is a funny story because this is how my coauthor, Shay David, got involved. I just happened to…he’s a computer science, information science Ph.D. student at Cornell. We were working on another project. I just happened to mention to him that I discovered, of all things, that one of my books had the readers’ reviews on Amazon, it seemed, copied under false e-mails to another book on a similar topic on the history of synthesizers.

And I mentioned that I’d complained to the publishers, Harvard University Press, about this, and they’d complained to Amazon, and Amazon hadn’t taken the reviews down. And they’re still there to this day, by the way, these plagiarized reviews. And we thought, “Well, that’s an interesting thing. I wonder how widespread it is.” And Shay, with his computer science expertise, was able to write basically a plagiarism algorithm to search for this more systematically. And we looked through 50 000 reviews of CDs and books, and we found 1 percent of copied text.

There’s been another study at Cornell by the college of information tech on a much bigger dataset, several million. They’ve found consistently this figure of about 1 percent of copied text amongst these reviews.

Steven Cherry: Now, the theory of mass-reviewing on sites like Amazon is that there’s a sort of wisdom of the crowds, right? The prejudices, favorable and unfavorable, tend to balance out, and the crowd as a whole is smarter than any one member of it, and in fact smarter than the experts. That’s the theory, and I guess that can happen, but not when people are gaming the system. So how are people gaming the system? What are they doing?

Trevor Pinch: Yeah, before I say more about the gaming, I’d just like to say that in regards to my own books, the reason I was so outraged by this abuse of the system, this copying of text plagiarism, is that I actually loved the reviews. So I’d just like to say that there’s something very positive about what’s going on, which Amazon has stumbled into with their customer reviews.

But the system can be gamed. One of the ways it can be gamed is if you become, as I said already, a top 1000 reviewer and you get identified with a badge. Of course, you not only develop a fan club, but, of course, authors and publishers and agents can identify you as a top reviewer and start to send you stuff. And this seems to be a fairly routine process. It might involve…I spoke to an author at Cornell; he was surprised that I didn’t do this. He said, “Well, you find someone who’s high up in the review rankings and send them your book that’s about to come out, and maybe they’ll review it for you. And since they’re a good reviewer, you know, a top-ranked reviewer, they’ll probably write you a nice review.”

And so what’s happened is, we did this survey of these top reviewers, and we found many of them are plied with free stuff, not only books. It’s moved on to products; they’re given free products. So they’re not genuine customers in the sense that they’ve purchased the item. And that’s what I object to. It won’t necessarily lead them to be biased in their account of the object or the book, if it was a book review, but they’re not genuine customers. They’ve been given the stuff for free, and in fact, Amazon has its own program for this now that encourages this: It’s called Amazon Vine.

Now all the reviews, if you’ve been sent these items, you can choose them. It’s like a sort of newsletter that’s sent around. When you get in the top 1000, you start to be sent Amazon Vine—you’re asked to join the Amazon Vine system—and you’re sent products. You can choose, you know, what you want to review. It can be fancy Belgian chocolates. It could be something like a pretty expensive printer. And you’re given that stuff for free. But at least at Amazon Vine, there’s a badge which says on Amazon, “This was an Amazon Vine review.” I don’t think most people only casually on Amazon know anything about it, what that would mean, that it’s a free product.

So that’s one way it can be gamed. So it can be gamed positively, and as the New York Times reported, of course it can be gamed negatively with one-star reviews with so-called haters or sock puppets—people who set up fake e-mails to attack a book, or they can use their own e-mail if they’re not worried about it. And they may not be genuinely disliking the book in the sense that they purchased the book and they’ve given it an objective read. And the New York Times reported on its website on a book about Michael Jackson, where there was a Michael Jackson rapid media response team, who were clearly Michael Jackson fans, who didn’t like any book that had to say anything negative about Michael Jackson.

Steven Cherry: Now, did you find any intimations…I know it’s one thing to ply a potential reviewer with some free, a free copy of the product, but are people hiring other people to praise or damn a book or product?

Trevor Pinch: There’s evidence of this, yeah. The Guardian did a recent little investigation. The Guardian newspaper in Britain did a little investigation to this, and, yeah, I think on these websites, like, where you can hire, basically outsource cheap labor, they found evidence that there were, people were being offered to write negative or positive reviews of books and other products.

And so it seems, you know, it’s probably a cheap investment for a publisher or a company. Of course, there’s a downside if they get caught doing this, and of course these websites are advertising for labor, kind of public websites, and people can, as the Guardian did, go there and find evidence. And then you’re going to be in real trouble, and if this happens for some companies, like Yelp for instance, they have the badge of shame now, if they catch you involved in this sort of activity.

There is a downside to doing it, a negative incentive to doing it, but I’m certain plenty of people have done it. A more likely way of doing it, by the way, is just for authors to tell their friends, or authors themselves, as various cases have been found. Famously, the Canadian side of Amazon, there was a glitch on it revealing the real e-mail addresses, this is a few years ago, of the reviewers, and they found out a large number of book reviewers panning their rivals’ books and praising to the hills their own books. So it probably goes on a lot like that, you know, family members, close friends. So that’s hard to spot, that sort of gaming of the system.

Steven Cherry: There was another author manipulation that you described in the paper. I thought it was a great story. It was the author who created what he called the “Amazon Hour.” What was that?

Trevor Pinch: Oh, yes, that’s a lovely story, yes. Yes, so basically you work out which is the best time for people all to go on Amazon, and this is related to the sales of your books. You know, it’s one of the great things about Amazon, I have to say, as an author. When they first introduced that, this sales ranking, you could see, you know, what rank you were in terms of sales of your book, and it was kind of addictive, until you realized that, you know, if you’re an academic, your book is probably going to stay, you know, it might get as high as 50 000. It’s probably down in the millions a lot of the time, and that can be a bit depressing.

But also, you also notice that sometimes you get these spikes, and that’s clearly people suddenly buying your book if there’s a review. So this person worked out when was the best time of the day, when there’s a 24-hour cycle. If everybody could buy the book at that time, this is before Twitter, he sent out e-mail messages telling everyone to buy the book at that time, and it could get such a big spike that he got to be the No.1 book sale, selling book on Amazon for a very short time. I think it was just literally for like half an hour. But then he could always say he had been the No. 1 seller on Amazon.

Steven Cherry: Now, you conclude your paper by noting that it’s still early days for these systems. One thing that’s for sure, though, is that professional reviewing is waning. There’s still only a handful of newspapers that still have a Sunday book review section, for example, and user review systems are waxing. So do you think that the problems you’ve found will eventually ruin the commons for everybody, or will they get sorted out in time?

Trevor Pinch: Well, who knows which way it’s going to go. By the way, I lament also the less mainstream organizations, newspapers publishing reviews, because I just love reading those book reviews anyway. And I would hope there’d be room for both. But I think in the long term, companies like Amazon are going to have to address this issue, because I think initially people would buy books just based on reading the reviews. They may read one or two, and that would be enough, and I think that people got more skeptical as this problem of fake reviews and abuse has surfaced from our research and other people’s research.

By the way, the area of hotel reviews, that’s a notoriously…TripAdvisor and places like that, there are many, many fake reviews. But everyone recognizes that it’s an incredibly useful service, that there are genuine customers, and people’s opinions can really sway, and they give, for books, a very detailed reading.

So it would be nice if we had some way of eliminating some of the difficulties. I think one way this will happen is, some of the travel sites have done this—I think and Expedia do this. You can only write a hotel review on Expedia or if you’ve actually stayed in the hotel and paid for the hotel room. It’s rather like the Genuine Purchase badge that Amazon has introduced, so it shows that you’ve actually bought that item that you’re writing your review of.

And I think having the real name of the person and some evidence they’ve actually bought the product or stayed in the hotel must be the way to go. It makes the system more transparent and will stop a lot of the abuse. Of course, the reason the companies aren’t doing that, like Amazon, is that would drastically reduce the number of reviews that were there, and the number of reviews there is a really important thing. It’s just the sheer quantity of them that makes the site so attractive to people. They can find just anything. So I think, you know, these companies should start to use some of the smarts and computer science and information scientists and social scientists to try and work on the problem as well, to try and tackle it. [In] which areas might there be problems? Where is the plagiarism, and so on?

Steven Cherry: Well, if they do that, I’m sure it’ll be because they’re spurred on by the vigilance of researchers like yourself. So thanks for looking into the problems, and thanks for your time today.

Trevor Pinch: Thank you very much indeed.

Steven Cherry: We’ve been speaking with Cornell University sociologist Trevor Pinch about people gaming the review system at Amazon and elsewhere for mischief, politics, and profit.

For IEEE Spectrum’s “Techwise Conversations,” I’m Steven Cherry.

This interview was recorded 31 January 2013.
Segment producer: Barbara Finkelstein; audio engineer: Francesco Ferorelli

Read more “Techwise Conversations” or follow us on Twitter.

NOTE: Transcripts are created for the convenience of our readers and listeners and may not perfectly match their associated interviews and narratives. The authoritative record of IEEE Spectrum’s audio programming is the audio version.