Why You Shouldn't Trust Internet Comments 180
sciencehabit writes "A new study suggests that all the reviews you read on Yelp and Amazon are easily manipulated. It's not that companies are stacking the deck, necessarily, it's that a few positive comments early on can influence future commenters. In fact, when researchers gamed the system on a real news aggregation site, the items received fake positive votes from the researchers were 32% more likely to receive more positive votes compared with a control (abstract). And those comments were no more likely than the control to be down-voted by the next viewer to see them. By the end of the study, positively manipulated comments got an overall boost of about 25%. However, the same did not hold true for negative manipulation. The ratings of comments that got a fake down vote were usually negated by an up vote by the next user to see them."
Excuse me?! (Score:5, Funny)
This comment is IMMENSELY trustworthy!
Re:Excuse me?! (Score:5, Informative)
Absolutely! Probably the truthiest thing I ever read.
Re:Excuse me?! (Score:4, Informative)
Re: (Score:2)
Can I quote you on that?
Re:Excuse me?! (Score:5, Funny)
I heartily endorse that statement.
By the way, Coca-cola is delicious.
Re:Excuse me?! (Score:5, Funny)
Re:Excuse me?! (Score:5, Funny)
This comment is IMMENSELY trustworthy!
I have found that 1 + 1 = 3 for very large values of 1.
no citation required, comment speaks for its own veracity
Re: (Score:3)
I have found that 1 + 1 = 3 for very large values of 1.
You might want to upgrade your old Pentium PC at some point.
Slashdot. best damn site on the Internet. (Score:2)
informative and good coupons. best cat videos.
Survivor bias (Score:3, Insightful)
Nothing new here, move on...
Re: (Score:3)
After a while you learn which comments to trust and which comments that are there to troll or spam.
Of course - subtle trollings are harder to detect, but they may still contain a grain of truth too.
Re:Shill posts (Score:2)
A subtle astroturfing is one thing, but many of the fake postings are pretty hilariously transparent. Good help is hard to find.
Re: (Score:2)
I distrust them all. Honestly, I am baffled why so many people go to Yelp when it should be obvious to everyone how useless it is. No one posts if they get the appropriate level of service; so posts tend to come from extremes instead. If service is not up to part the negative reviews come out in force, because no one wastes their time online just to say "meh" the exaggerate how bad it was. And posts from the local A&W hamburger chain describing it in superlative terms can be discounted similarly. Y
Re: (Score:2)
That's a shame. I feel most compelled to write reviews on Amazon when I'm either really unhappy or really pleased with a product, so many of them are either 5's or 1's. Perhaps I should just make them all 2's and 4's so they're more believable?
Re:Survivor bias (Score:4, Insightful)
I am the same way.
I'm also far more likely to review something if I disagree with the collective opinion on the site. I figure the 47th glowing review is useless but the 3rd bad one adds clarity.
Re: (Score:2)
Generally agreed, although I also find that many 1 star reviews also fall in one or both of two additional categories:
1. Someone bought something completely inappropriate for their situation, for example, a CF card instead of an SD card to use with their camera.
2. Someone didn't read or understand the specs/requirements/instructions.
True story about #2: when I was researching my gas stove, the only 1-star comment was from someone replacing an electric stove. They were upset that they couldn't plug the new s
Re:Survivor bias (Score:5, Insightful)
This is very true....and that is before you even get to the other question... even if they are not lieing, how do you know they are really a good commentor? Now, at this point, as we have had several blender's burn out, my wife is familiar with several of the models on the market. I honestly doubt there are too many other devices that we are as familiar with across different brands/models.
So what if you think this is the greatest cordless drill ever and the battery just goes forever. How do I know you are not basing that on a comparison with some cheap crap drill you bought in the mid 90s with a battery that shit the bed after a handful of recharges?
Not only that but, its rare that someone goes back and re-evalutes the product later. I have seen it...I have totally seen amazon reviews that said one thing, then had an edit explaining "Now that I have been using it for 6 months some issues have cropped up..."
Even less likely is that they get the product and use it before they up or downvote the comment. So really the only question on comment ratings is not really about accuracy but about whether it helped you decide to buy or not.
I generally look for posts of substance, that seem to actually have knowledge of what a product does and how it should work... but that are not so in depth as to look professionally writen, or by commenters that comment on 6 items a day (seriously, you really think I think you could possibly buy and evaluate that much crap?)
Also I find looking for the mid range star ratings is best. I generally skip right over the 5 star ratings for some 3 and 4 stars first, to get a feel for whats wrong with it. Often the features are less important than the defects, and its more about picking which defects I can live with than which features are best.
Re: (Score:2)
That's one reason why star ratings need to be replaced with a ranked voting system. In order for your vote to have meaning, you should have to rank it in comparison to competing products. Maybe you rank this new drill higher than the old one. Th
Re: (Score:3)
Actually there is something new: in addition to trolls, shills, and gadflies, some of the comments on the internet are placed there by people experimenting on internet comments.
Amazing (Score:5, Funny)
What an excellent report. I read this to my Son and he loved it, would recommend to anyone and definitely read again. 5*
Re:Amazing (Score:5, Funny)
I agree, and my daughter makes $1500 AT HOME. ****>>>CHECK IT OUT!!!****
Re:Amazing (Score:5, Funny)
I know she does. Perhaps you should take away her video cam.
Re: (Score:2)
Comment removed (Score:5, Funny)
Re: (Score:2)
I haven't even seen a Beowulf cluster, you insensitive clod!
Re:Tell me about it (Score:4, Insightful)
Re: (Score:2, Funny)
Better yet: A Beowulf cluster of insensitive sharks using their lasers to melt away the grits from Natalie Portman's back and yelling, "In Soviet Engrish, all our grit are belong to you!"
Re: (Score:3)
And reciting Emily Dickinson poems!
Re:Tell me about it (Score:5, Funny)
just imagine a beowulf cluster of insensitive clods!
I think this is the perfect description of Slashdot.
Re: (Score:2)
Yo dawg, I hear you like insensitive clods ...
This slashdot thread (Score:5, Funny)
is the best slashdot thread we've seen all week! I love it!
Re:This slashdot thread (Score:5, Interesting)
Re: (Score:2)
So true - you can't trust reviews in aggregate. You have to look one-by-one and try to detect the biases.
Great article! (Score:2, Insightful)
This is by far the most insightful treatment I've ever read on this important issue. Everyone who does business on the Internet must read this valuable study.
I don't think so... (Score:2)
If someone has a bad experience, they will be pissed off and want to leave a bad comment. Often, that is the only recourse for a shitty service offered, or if a refund/exchange is no offered.
If someone has a genuinely good experience, they may be happy and want the product/business to do well, so they will leave a comment to aid in that.
I would think the only influence comes from choosing based on comments, not on leaving comments.
Re:I don't think so... (Score:5, Insightful)
I'd say it's far more complicated then that.
If you have a bad experience and go to the product review and the other reviews are bad, you are apt to write a review confirming what you are reading.
On the other hand if you have a bad experience and all the other product reviews are good you may have a moment of self doubt (did I mess up with the product) which makes you less willing to post a negative review.
Re:I don't think so... (Score:4, Insightful)
I hate to say this, but I find I am more likely to take the time to write a bad review than a good one. (Anger is a great motivator.) I assume others are like this as well so I read the negative reviews in that light.
Also, any review in all caps, good or bad, I automatically discard.
Re:I don't think so... (Score:4, Insightful)
I hate to say this, but I find I am more likely to take the time to write a bad review than a good one. (Anger is a great motivator.) I assume others are like this as well so I read the negative reviews in that light.
Also, any review in all caps, good or bad, I automatically discard.
That's been my experience, too. Anger motivates you to want to do something, so people lash out on the comment board. People who are satisfied, by definition, aren't really motivated to take any additional steps.
Manipulation (Score:5, Informative)
What's interesting about the study is:
1) The manipulation was a single positive or negative vote applied at random immediately when the article went live.
2) People would tend to correct false negatives, but amplify false positives.
Re: (Score:2)
In other words, people tend to post review where there already are reviews. It's as though we were social animals.
Re: (Score:2)
It's not about a review site.
Re: (Score:2)
Ok, in this instance it was votes. I would say this is only a minor difference, the herd mentality still shows.
Re:Manipulation (Score:5, Informative)
My mistake, it's not even articles, it's just Reddit comments.
Re: (Score:2)
The issue is partially do the emotional attachment we have towards our stuff.
We have a Mac We love our Mac, We have Linux we love Linux....
We like reinforcement by others that we had made the right purchasing decision.
The stuff we tend to hate is the stuff we didn't have a decision or a personal investment in (Such as your work Computer)
It is very rare that we hate the products that we own, unless it is time to replace them, or we bought a lemon.
Otherwise we have an emotional attachment to them and will ra
Re: (Score:2)
People would rather risk encountering low quality information, than risk losing high-quality information.
Yeah maybe subtle differences but not important. (Score:5, Interesting)
Re: (Score:2)
It's not even comments. The paper refers to "upvotes" which makes it pretty clear that the study took place on Reddit and involved incrementing/decrementing the score when an article went live, by a single point.
Well sure (Score:5, Insightful)
My hypothesis:
1) Products with positive comments are more likely to be purchased.
2) People identify themselves by their choices, and no one wants to make a bad choice. Ergo, almost by definition, any choice people make is "the right one". At the very least, people are predisposed to liking what they spent money on.
Remember; When discussing all things retail, it's not how good the product is, but how well it satisfied the need. At least half the time, that need is largely imaginary.
Sure, I could RTFA, but this is more fun.
Re: (Score:3)
I agree with point 1, though it is really just stating the obvious, so isn't saying much.
I'd have to disagree with point 2. It could, perhaps, apply to products whose enjoyment is primarily a matter of taste or the response they elicit from other people, such as fashionable clothing. I can imagine situations where a poorly researched impulse purchase of an expensive item like an automobile could lead to self-deluding rationalization to justify the purchase. I'll grant you that. For more functional items
Re: (Score:2)
I'd suggest it's a behavior that's related to cost. The more something cost, perhaps even the more of a luxury item it is, the more likely someone is to be biased.
Re:Well sure (Score:5, Funny)
people are predisposed to liking what they spent money on.
I don't know about that. I spent a bunch of money on an ex-wife and I don't like her much at all.
ethics problems (Score:4, Insightful)
Re: (Score:2)
Perception vs actual rating (Score:5, Insightful)
When I check reviews, one of the first things I do is check the negative reviews. Why? Because half the one-stars are often jackasses with no clue what product and/or service they were buying. Other times, knowledgable and otherwise reasonable people have found the service or product being rated to be inadequate in some significant way.
And then I look for high ratings to see if they are reviewing the product in a reasonable manner. From there, I make my own decisions regarding the validity of both sides.
Anyone who decides just based on the stars/review-based numbers is a fool.
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Agreed - In similar vein, I have seen one-star reviews of restaurants stating that 'the line/wait was too long", meaning they never even *tried* the place;
I am reminded of the Yogi Berra quote: "Nobody goes there anymore. It's too crowded."
Re: (Score:2)
I do that exact same thing. Most of the "HUGE ISSUES ZOMG" are either some preference that didn't meet personal expectations or just plain 'ol PEBKAC. I also take note of the volume of reviews too. If 5000 people rank a product 4 stars, that's a lot of use cases to check out. If a bunch of people report the same failure, it points out commonality and such.
I am kind of OCD about making important purchases though :P
Re: (Score:3)
Re: (Score:2)
I once saw a review where the buyer gave it one star because the item he ordered wasn't really what he wanted.
Yes, he ordered an item, they delivered it promptly, but he decided he really should have ordered something else and so he gave it one star. You just can't win with some people.
Re: (Score:3)
I wrote a iOS game that was out for a few years. We got a 1 star review because we "weren't Tetris". Except the game wasn't Tetris, didn't claim to be Tetris, and had absolutely nothing to do with Tetris except for being vaguely in a similar genre of "real time puzzle game".
FANTASTIC (Score:2)
Pressured by vendors (Score:5, Interesting)
This elicited an immediate email from customer service offering various deals to bribe/entice her to change or withdraw the review. Companies are free to do what they wish on their website but that still struck me as disingenuous.
Re: (Score:2)
Re:Pressured by vendors (Score:4, Interesting)
I actually have no problem with that - if done right. If the response was "I'll give you free stuff, but only if you remove your review", then yes, that is super sketchy. But a lot of times it's more just "I'm sorry that happened. Would you like to give us another shot on us? It was probably a fluke." And that is exactly what customer service *should* be like. If you go back and it was a fluke, then you change your review, and everyone's happy. If you go back and it happens again, then that company clearly needs to pay their other departments as much as their customer service department, but I'd still rather that than a response of "tough luck, go away."
Re:Pressured by vendors (Score:5, Interesting)
This is the way companies work these days.
Have a problem with a product? Don't navigate through dozens of useless pages on a support site, don't wait for an hour trying to get through to their helpdesk; post a complaint on twitter, wait a few minutes and they will contact you.
I wish this was a joke, but this is honestly how I deal with some companies nowadays.
Re: (Score:2)
This is why reviews are more valuable when managed by a third party that is integrated into the retail website, while keeping their own name and reputation on the line for being independent reviews "provider".
Because of sheer volume of sales, Amazon reviews are an obvious starting point, but I normally have to exclude the 5 stars and 1 starts in order to understand if the product has serious problems or if it's the user that failed to RTFM. Going direct to the website of the reviews company is a very useful
You all joke... (Score:5, Insightful)
typo (Score:2)
A+++ Would do business with again (Score:2)
Isn't this *everyone's* rating on eBay, and yet eBay is 99% populated by crooks at this point?
You reading this APK? (Score:3)
Your hosts file comments are not trustworthy.
The following comment is a lie. (Score:2)
The subject of this comment is true.
I only read negative comments (Score:3)
There is no value in positive comments.
If you're considering to buy something, you want to know where it fails, not where it succeeds.
Moreover I personally would never leave a positive comment. If it works as advertised, life can go on as normal. If it doesn't, then I leave a comment.
Re:I only read negative comments (Score:5, Insightful)
Re: (Score:2)
There is no value in positive comments
Of course there is. Thoughtful positive comments that describe the hands-on experience of using a product and how well the advertised features are implemented can be extremely useful. There are plenty of product categories where all the top products function well, but have different strengths/weaknesses/usage_patterns/quirks. Detailed positive reviews are quite valuable because they provide details not offered by the advertising/product description - these details help the purchaser choose the product be
Re: (Score:2)
We're talking about Amazon-type reviews here, no fully-fledged detailed reviews.
Re: (Score:2)
Same still holds. If you were correct and everyone followed what you claim, every item on Amazon would be rated one star. Positive ratings and reviews are necessary to offset the inevitable negative ones, at the very least.
Re: I only read negative comments (Score:2)
98% of everything is crud. Amazon reviews just confirm the rule.
Re: (Score:2)
Of course, 100% of your comments are crud. You just confirmed it.
Re: (Score:2)
I didn't explicitly say so, but I was recollecting helpful Amazon reviews when I wrote my comment. They certainly vary in detail - most reviewers don't give a fully detailed review, but they provide something more than "I love it". Often they tell a story about whether a product solved the problem they bought it for (did the pet hair attachment for the vacuum actually work for them and their dog? Did the crockpot burn the food after running all day?). If there are more than a few reviews, different peop
Not surprising (Score:3)
To use an example from /., it's not at all uncommon to see a comment go completely unnoticed from moderators for an hour, and then get a +1 from somebody, and within 30 minutes have gone from Score:1 or Score:2 to Score:5.
Bandwagon effects are quite well-known. After all, all your friends are paying attention to them! It seems to be a useful psychological reaction: If all your friends and family are jumping off the bridge, chances are you will too on the theory that they probably have a good reason to do so.
Re:Not surprising (Score:5, Insightful)
Or maybe they just weren't visible to the moderators until a few upvotes brought them above the viewing threshold. Once visible, comments that happen to be genuinely insightful, informative etc, would get more upvotes because they deserve it. You don't have to invoke bandwagon effects to describe what you observe. Also, when I have moderator points, I tend to upvote good comments that *don't* have a high score because they are worth drawing attention to. Wasting moderator points on a "me too" upvote of a +5 comment is a poor use of the privilege. In my view, the purpose of moderation isn't to "skew" the discussion to reinforce the echo chamber., Rather moderation should improve the overall signal/noise ratio so threshold settings are actually meaningful.
Re: (Score:2)
Oblig: http://xkcd.com/1170/ [xkcd.com]
+1, an excellent article. (Score:2)
Companies have whole fake blogging departments (Score:2)
Ethics (Score:2)
Assuming the researched posted fake reviews real businesses and products, isn't there an ethic issue with this research.
I wouldn't like to be the owner of an otherwise good restaurant that ended up getting lots of bad reviews just because some researchers' coin flip decided my restaurant should get bad reviews for their research.
It just seems a tad unethical to try and randomly ruin businesses for a research paper.
Perhaps we should do some research on what happens when you randomly bankrupt researchers.
setting the tone works on /. (Score:4, Informative)
leasthelpful (Score:2)
I only trust Amazon product comments if they are vetted and posted to http://leasthelpful.com/ [leasthelpful.com]
Awesome (Score:2)
A++++ wuld read again
And probably will when it gets duped in three days time.
I had to retract one + because of a missing word in the summary:
In fact, when researchers gamed the system on a real news aggregation site, the items that received fake positive votes from the researchers were 32% more likely to receive more positive votes compared with a control
This has been around a long time (Score:2)
"Accentuate the positive, eliminate the negative" says the song. Same thing with upvotes.
-1 irrelevant comment (Score:2)
Look at the negative reviews (Score:4, Insightful)
Everytime I look to buy something I am not not an expert on or are on the fence about quality, I look up reviews and sort by rating. And I look at negative ratings first because a vendor or retailer won't pad a product with negative reviews. And even though there will always be negative reviews from people who dont have a clue or give little to no info, there are som rea gems out there that give you a clear picture of what you are getting into.
Obligatory anecdotes:
When looking on newegg I sort by lowest score first and read the reviews. You always have some dummies who obviously have no idea what they are doing and rate 1 star because of a mistake they made. But you also run across some genuinely informative negative reviews which are more influential to me than positive reviews. For example, I was looking to buy a uATX board from ECS that had the AMD bobcat CPU onboard. It was perfect, had extra PCI shots for SATA cards for a low power Linux server box. It turns out in a few of the negative reviews there was an IRQ bug that severely impacted performance. I was close to buying it but then scratched it off my list.
Recently I was also looking to purchase a generator from a coworker who bought it after hurricane sandy but never used it, the box is unopened. The brand name was Generac and I have owned two other Generac products, a power washer and a 4kw generator. Both of those machines went south after little use, the generators exhaust valve stuck open when the valve seal went bad and allowed oil to seep down the valve and seize it (I fixed that but it never ran quite right, stalled and was a bitch to start). The power washer engine needed its carb rebuilt and then the water pump blew a shaft seal. But that was 6+ years ago and I figured Generac got their shit together by now. After reading negative reviews on Amazon I came to understand that Generac will try to weasel out of warranty repairs and "authorized" repair shops frequently change as they get shafted after Generac refuses to reimburse them for warranty repairs already performed. There were also negative reviews that warned of blown stator coils after a few hours use and lemons that wouldn't start out of the box. Since the generator could not be returned to the original vendor I took a pass even though I would get a sweet deal ($200 off retail as he wanted to dump it). Maybe it would have worked fine but I didn't want to risk losing 800 bucks and damage my business reputation.
Re: (Score:2)
And I look at negative ratings first because a vendor or retailer won't pad a product with negative reviews.
Right. Because retailers are too stupid to think of the idea of paying shills to leave negative reviews of their competitors. There are also fake negative reviews.
This post is untrustworthy (Score:2)
Why is this a surprise? (Score:2)
You can see the phenomenon on Slashdot itself.
Rating of reviews (Score:2)
The article only talks about how reviews get rated after an initial rating, not about how subsequent reviews are good/bad if the initial review is good/bad. That would be the 'xxx people found this review helpful' - giving it a bunch of helpful ratings initially make more people likely to vote it helpful.
On a slightly related note, I wish Amazon would allow viewers to only view rating histograms of 'Amazon verified purchasers'. I remember when Amazon hadn't even released the Fire (except to Vine members -
Re: (Score:2)
They all mod
They love me
On Slashdot.
(apologies to Weird Al)
Re: (Score:2)
That's true, any untruth is but an illusion.
I now find drinking Monster and Rock Star have given me godly powers
Re: (Score:2)
Bad reviews can also be fake, but yes by skipping the 5 star reviews you will skip over the highest percentage of shill reviews. That still leaves a lot of shill reviews however, some highly skilled and subtle and others more obvious. The shills that were clever enough not to go for a full 5 stars are probably also clever enough to write a realistic sounding review that mentions one or two mildly bad aspects of a product along with the good ones.
Re: (Score:2)
This doesn't work with the non-obvious shills on sites like Amazon. There is no way for you to know whether a detailed story that sounds convincing is true or not. As has already been pointed out humans nearly always overestimate their ablity to detect a lie. Not all shills are obvious about it.