Slashdot is powered by your submissions, so send in your scoop

 



Forgot your password?
typodupeerror
×
Communications The Internet Network Networking

Customer Feedback Surveys Could Be Considered Harmful (easydns.org) 196

Longtime Slashdot reader Stunt Pope writes: Customer Feedback surveys are now near-ubiquitous, subjecting us all to near-Black Mirror-esque pursuit to "rate your experience" for everything from going to the bank to ordering a pizza. Thanks to The Curse of Goodhart's Law, all of these surveys are beyond useless and even damaging. Mark Jeftovic writes in a blog post: "The shop/hire-rate-reward feedback loop has become baked-in to some systems. Many live marketplaces incorporate these feedback transactions into ratings, which then become a score which then impacts future prospects of whomever is being rated. And that's where the trouble starts. There is a point where this stops being useful and the knock-on effects of a ratings system predicated on feedback results becomes counter-productive. That point is when the ratings become targets. When a company decrees 'All customer feedback ratings must score a minimum of X, or else...' the company has just commenced the process of invalidating and corrupting all useful information to be gleaned from that feedback/survey process. A label which captures this concept is 'Goodhart's Law' -- after economist Charles Goodhart, who posited in essence that 'when a measure becomes a target, it becomes useless.'"
This discussion has been archived. No new comments can be posted.

Customer Feedback Surveys Could Be Considered Harmful

Comments Filter:
  • by Xenx ( 2211586 ) on Thursday January 26, 2017 @05:09PM (#53745105)
    How would you rate the quality of this story?
    [ ] Ehh, good enough.
    [ ] Could have been better, I guess.
    • by NatasRevol ( 731260 ) on Thursday January 26, 2017 @05:17PM (#53745157) Journal

      Would you rate this story a second time if we gave you a 5% off coupon?

      • Does that mean 5% less ads on the page?

        Hummmm... no deal!

        • *fewer

          • by tsqr ( 808554 )

            Precision and clarity. 5/5.

          • Thank You for your thoughtful response. Please take 5 minutes and complete a free survey regarding your recent Slashdot posting experience!

            1. On a scale of 1 to 9.99, 10 being the best, how would rate the grammatical content of that post you replied too?
            2. On a scale of noon to midnight how would you rate the timeliness of the post?
            3. On a scale from A to Z, how was the speling?

            For completing this *FRee survey, you get a sticker!

    • I don't generally fill out surveys. When I do, though, I am as honest as they let me be. Sometimes, the pre-provided answers don't conform to my true feelings or are not applicable.

      I prefer the freeform surveys where I can state exactly what I like and don't like.

      • I am as honest as they let me be.

        You are a horrible person. Do you realize that by rating a good, but not fantastic, interaction as only 4/5, you are jeopardizing someone's career?

        • Just rate them double-plus good! Then both you and their big brother will be non-sad happy!

        • It's hardly a career if 95% performance means losing it. They are better off moving to greener pastures.

      • Will this work for you? ;)

        On a scale of 1 to 10, please rate to the nearest ten-thousandth place, EXACTLY how you felt about your service today:

        1. Your Service reps handshake: 7.67565
        The reps hand placement was not in perfect alignment with mine, and I detected a slight amount of clamminess on the skin...

        2. The quality of the tires you received: 3.14159
        The tires were very round, but the font face of the sidewall was less than ideal for viewing at high speeds.

  • by Zephyn ( 415698 ) on Thursday January 26, 2017 @05:11PM (#53745125)

    " A label which captures this concept is 'Goodhart's Law' -- after economist Charles Goodhart, who posited in essence that 'when a measure becomes a target, it becomes useless.'"

    I've seen a similar effect in places where I've worked. A poorly defined metric that is used to rate employee performance will suddenly become the primary focus of the job, instead of actually doing the job.

    • by AmiMoJo ( 196126 )

      It's abused the other way too. If I get a survey, I'll give the answers that I think will get the result *I* want.

    • The place I work at had to fire a store manager recently for cooking the books in order to meet his bonus requirements. It took 2 people 3 weeks to double check everything and get real numbers.

    • by cats-paw ( 34890 )

      Have you ever noticed that in a lot of places employees will sit right in front of you handling issues having to do with "internal processes" and not actually help the customer first ?

      I've noticed this quite a bit.

      When your "process" doesn't put the customer first, you should probably re-examine.

      Also too please train your employees.

    • " A label which captures this concept is 'Goodhart's Law' -- after economist Charles Goodhart, who posited in essence that 'when a measure becomes a target, it becomes useless.'"

      I've seen a similar effect in places where I've worked. A poorly defined metric that is used to rate employee performance will suddenly become the primary focus of the job, instead of actually doing the job.

      I've seen a similar effect in places where I've worked as well, that is, public schools. Since the nineties, both school and pupil evaluations have been based on achieving questionable targets using questionable metrics. It's nice to know that even if the metrics weren't questionable, using a target like that would make them so. And not only that, a term, Goodhart's Law, exists to describe the phenomenon. I'll place it up there on the shelf right next to Godwin. Thanks!

    • Natural adaptive human behavior will always have humans finding ways to game systems. Static simple systems can not compete with a dynamic complex human (not just smart, but fools are also ingenious.)

      WellsFargo just had a huge huge problem with their management--- remember? they were nuts about metrics and linked incentives which ended up in a massive fraud that should have people leaving that bank in groves... But if their big part in the housing collapse didn't get their suckers to quit I am not sure

    • But our compiler makes SPEC run really fast! It must be your code that's wrong!
  • by rtb61 ( 674572 ) on Thursday January 26, 2017 @05:18PM (#53745169) Homepage

    Due to the growing abusiveness or corporations, invasions of privacy, and wide spread deceit, I have decided to follow the principles that many corporations expose and purposefully lie in feedback to companies I dislike. They want quality consulting services, well, the fuckers can bloody well pay for them, nothing is for free according to them, for free, they just get lies. Turnabout is fair play after all, lie to me, well I'll lie to you ;) (only for poorly behaved corporations, which seems to be the majority, especially multi-nationals).

    • How do I get honest feedback? I mean, I'm actively developing software, and adding new features. I would love to be told "most of your customers want X before Y" (or most of your on-the-fence non-customers). That seems to be a win-win for everyone (except those minority who want Y, but that would be true of any system that accurately measured).

      • by rtb61 ( 674572 )

        Want honest feed back, pay for it, the free ride is over. Blame no one but the abusive corporations, all good will dead, due to endless false marketing, bad customer support and services and failing products. Why should consumers serve companies who do not serve them, suck it up or get rid of the psychopaths in corporations.

  • by npslider ( 4555045 ) on Thursday January 26, 2017 @05:19PM (#53745177)

    An experience at a local biz in town with a customer service rep... Was told that anything less than 5 out of 5 on his customer review is considered a bad review, and he all but begged me to give him 5-stars.

    He was so overly friendly it was past creepy. I felt conflicted: he did a good job, but I felt I was rating for his sake, not to give an honest assessment of how well I was served by him.

    • by myrdos2 ( 989497 )

      I've been told (at Boyd Autobody) that if I don't give the guy a 10/10 rating, he'll call me to find out why. The impression I got was that I'd better voice my complaints to his face right now or keep my trap shut later when they call with the survey. I was looking forward to telling the survey person that I'm giving a 10\10 rating because he threatened to harass me if I didn't, but they didn't end up calling.

      • by taustin ( 171655 )

        I've been told (at Boyd Autobody) that if I don't give the guy a 10/10 rating, he'll call me to find out why.

        And you gave them your real phone number? If you really, really have to let them call you, maybe you should be a 900 number.

        • by Calydor ( 739835 )

          If you're sending your car in for anything other than quick fixes (I'm assuming Boyd's Autobody fixes broken cars) you probably want them to be able to call you when your car is fixed.

          • by taustin ( 171655 )

            Or they can give you an estimate on time, and you can call them. Less convenient, sure, but more so than telephone harassment and stalking over a less than perfect feedback survey.

      • "First, you just lost your chance for 10 out of 10 by badgering me about that. There, no need to call me - I absolutely hate being pressured into giving dishonest reviews, that's the reason.

    • Or... he could have just been playing on your sympathy in order to get the MVP parking spot....

      Honesty is the best policy no matter what.

      • by npslider ( 4555045 ) on Thursday January 26, 2017 @05:52PM (#53745337)

        The end of my story was that I did not fill out any survey. Life is too short for me to be filling out surveys every time I am asked. I get them from the doctor's office, the car dealership, random items bought online, and my cat.

        • I rarely fill them out, typically only if it's a couple of questions.

          You know what *really* works when you get excellent service? Write a letter to the manager. I have a former neighbor who is 85 years old and she mentioned this a few years back. When she gets excellent service in a restaurant or whatever she hand-writes a letter to the manager mentioning the employees by name. She often gets letters back thanking her. That's how you reward excellent service.

          I do the same but with email :)

      • Honesty is the best policy no matter what.

        Unless you want a MVP parking spot.

    • by taustin ( 171655 )

      "It's not my problem that you work for abusive idiots."

    • by AmiMoJo ( 196126 )

      It puts the person being rated in a really difficult position. You can demand pretty much anything from them. 90% discount or enjoy your 1 star rating. Of course they have rules saying they can't give you a 90% discount.

    • I had a similar experience leasing a car.

      Firstly, the car was not ready when it should have been. They blamed the delay on the time to charge it, but the dealer has a fast DC charger, so the charge time should have been no more than 30 minutes.

      The salesman who begged for the good review also kept me waiting in his office for no good reason. As far as I can tell, that time was purely so that he could entreat me to give a good review.

      I ended up giving no review, but had I given one, it would not have been goo

      • I'm not sure which is worse: The survey after buying the car, or the service you get when it needs to be repaired...

    • A lot of places are like this now. They don't even average the score anymore, they just count the number of 5/5s or 10/10s.

      Under this rating, someone who totally sucks (would get a 4/10 on average), would beat someone who is really good most of the time (8/10 average), as long as the first guy gets a few of his friends involved to generate fake 10/10 reviews.

    • by nine-times ( 778537 ) <nine.times@gmail.com> on Thursday January 26, 2017 @07:51PM (#53745961) Homepage

      Another aspect of this that's weird to me:

      I was sent a survey to rate a business I frequent-- not to be posted online, just as feedback to the owner. I was asked to rate my experience on a scale of 1 to 10. If I selected 10, it asked the question, "What did you like about your experience?" I selected 8, and the question automatically changed to "What did we do wrong?"

      I didn't think they did anything wrong, but to me, the service warranted an 8. To me, 8 is good, just not quite excellent. So I answered the question, "It's not that you did anything wrong. I liked [a bunch of stuff]. In my opinion, you could improve by [doing some things]." I thought it was a very fair review, and I tried to give constructive feedback. I few days later, I got a phone call from the owner to apologize. It wasn't quite annoying, but it definitely seemed unnecessary and awkward, but I said thank you and reiterated that I thought the service was good, and I didn't intend the rating of 8 as a complaint.

      Since then, the business has continued to send me requests for feedback. My overwhelming feeling is not wanting to go through that experience, so I either just rate them a 10, or I ignore the survey entirely.

      • The problem with an arbitrary 1-10 or 5-star/point rating system is that every person defines their rating in a slightly different way. For instance, I've seen people knock a star off a rating for a game or movie because "nothing is perfect", meaning that apparently they'll *never* give 5/5 stars. On the other hand, many people seem to operation in a binary mode, rating 5/5 for anything moderately good, while anything they don't care for is 1/5. Some people define 5/10 as "average", while others imply th

      • Comment removed based on user account deletion
      • Since then, the business has continued to send me requests for feedback.

        That's the kind of behavior that would get me to reply with, "Ok, I changed my mind. 7."

  • by shilly ( 142940 ) on Thursday January 26, 2017 @05:32PM (#53745239)

    It's no surprise to know that targets can be gamed, and that performance metrics can be poorly implemented. But this is a false dichotomy: the choice is not between poorly implemented metrics/targets and no metrics/targets. There's also the option of implementing metrics/targets well. Not perfectly: what is, in this life? But certainly possible to implement them well -- and it would be damaging for the organisation not to do so. And if the behaviours and mindsets of the organisation are broken in the first place, then an absence of metrics/targets can be just as disastrous as poorly implemented metrics/targets -- and what's really needed is effort to work on the underlying issues.

    • There must be better ways to determine how well employees or departments are doing other than surveys. I see a survey and my mind thinks... "Circle all the 5's and get it over with". Unless, I have a real gripe with the service I received or too much time on my hands.

      Maybe hidden cameras in every room or undercover bosses posing as customers... oh wait, that's been done already!

    • I've put some thought into metrics for programmers / software jobs and have come up with damn little that's useful *if the programmers know what the metrics are*. If management looks at different metrics each month or quarter, and nobody knows ahead of time what they'll be looking at, you might get some semi-useful numbers. The metric doesn't become the goal when you don't know what the metrics is.

      Whenever the people doing the work know what the metric is, and have motivation to increase it, most focus mor

      • by shilly ( 142940 )

        I dunno. Take a call centre. ASA (average speed of answer) is a common and important metric. Organisations will often have a target or SLA for their ASA, and this is something customers really care about. It's driven by a range of factors: how call centre operatives handle calls, matching of demand and supply, actions to reduce demand such as self-service (eg online) or bots to answer, etc etc. I can't see how a call centre could manage itself effectively without having both an ASA target for the centre as

        • > Or in another field: mortality rates for surgeons.

          So the most reliable, and most obvious way for a doctor to increase their rating is to try to avoid treating patients the who are in poor health - exactly the opposite of what we want doctors to do. A doctor who aims to reduce their mortality rate should if spend their time with althletes and college students, maybe handing out steroids and stimulants. Again, if the doctor doesn't *know* anyone is looking at the mortality rate of patients, it can be

          • by Shimbo ( 100005 )

            So the most reliable, and most obvious way for a doctor to increase their rating is to try to avoid treating patients the who are in poor health - exactly the opposite of what we want doctors to do.

            shilly addressed this point in the parent post. Crudely looking at success rates, and over-incentivising them is bad. Not collecting the data because it is liable to misinterpretation is not the solution, however. There are huge benefits in clinical outcomes available by using these metrics sensibly; even some counterintuitive things like closing some regional units - because they saw the trickier cases too rarely to keep their clinical skills up to date.

      • I worked somewhere that metricked lines of code. Our "top coder" put all arguments on their own line, added lots of blank or stupid comments and did everything the hard verbose way. Had to look at that shit for years afterward.
        • I don't suppose that place managed to become, and remain, a large company, given their idiocy? Of so I'd love to know the name, so I can avoid them. I wouldn't want to bother interviewing there, and it would probably be best to avoid buying their products if their coders are rewarded for creating the biggest pile of garbage.

      • Whenever the people doing the work know what the metric is, and have motivation to increase it, most focus more increasing the metric than doing the job well, in my experience.

        If increasing the metric does not result in the job being done better, then you have the wrong metric (or, at least, a metric that you should not be using to evaluate the employee. One company I worked for had a phrase they imprinted on certain tools they gave managers, "What does not get measured does not get done."
        Customer satisfaction is a terrible metric for which to hold employees accountable because there are too many variables which the customer facing employee cannot control. I have interacted wit

    • It's no surprise to know that targets can be gamed, and that performance metrics can be poorly implemented. But this is a false dichotomy: the choice is not between poorly implemented metrics/targets and no metrics/targets. There's also the option of implementing metrics/targets well. Not perfectly: what is, in this life? But certainly possible to implement them well -- and it would be damaging for the organisation not to do so. And if the behaviours and mindsets of the organisation are broken in the first place, then an absence of metrics/targets can be just as disastrous as poorly implemented metrics/targets -- and what's really needed is effort to work on the underlying issues.

      The metrics are fine. The problem are only if you set specific measurements as targets that issues arrise. You can't get meaningful data if you demand the data to fall in a specific way.

  • by sconeu ( 64226 ) on Thursday January 26, 2017 @05:40PM (#53745275) Homepage Journal

    I follow Joshua's advice and rarely, if ever, fill out the "customer feedback survey".

  • by H3lldr0p ( 40304 ) on Thursday January 26, 2017 @05:46PM (#53745301) Homepage

    It's supposed to demoralize the front-line people. It's supposed to make them hate every second with the customers. This isn't a big revelation. It's applied psychology being used for anti-social ends.

    The ends which state that time is money.

    For every second you're not selling, you're costing someone money. For every second you're not adding value to that sale, you're costing someone money. For every second you're spending getting to know that other human, you're wasting someone else's money. That's how retail works now. How's it's been working for the past 15 years or so when those got first introduced. It was never about getting someone to better themselves. It was always about manipulation.

    • It's applied psychology being used for anti-social ends.
      The ends which state that time is money.

      Then it's counter-productive because, in my experience, two things happen:

      1. Customer service person wastes time begging for good reviews.
      2. Customers hate the business because of their wasted time and the embarrassment of being the recipient of the begging.

  • by myid ( 3783581 ) on Thursday January 26, 2017 @08:04PM (#53746037)

    If you want to reward someone for doing a good job, you can tell a manager (look for someone walking around the store who looks like a manager), or go to the store's "Customer Service" department, and tell them. Be specific as to how that person was helpful.

    Once in a busy pre-Christmas shopping season, a store employee went out of his way to help me. I told a manager, who was walking around the store, how much that employee had helped me. About 1/2 later, the employee rushed up to me all happy. He thanked me for telling the manager how helpful he had been. He said that because of what I'd said, he'd gotten a star (whatever that is) and a bonus.

  • Selection bias (Score:4, Insightful)

    by nine-times ( 778537 ) <nine.times@gmail.com> on Thursday January 26, 2017 @08:09PM (#53746067) Homepage

    I think it's also important to consider that satisfaction surveys tend to suffer from a sort of selection bias. You're only getting feedback from people who feel compelled to give feedback. In my personal experience you'll get:

    - Sometimes people who are angry
    - Occasional people who are extremely pleased
    - Often people who have excessive esteem of their own opinions**
    - Rarely people who just want to give helpful feedback

    I'm not pointing this out to necessarily disparage these groups or say that their opinions aren't valid, but it's important to understand you're unlikely to get a true random sampling.

    **I know someone is going to take issue with my third item, "people who have excessive esteem of their own opinions", so I'll try to explain what I mean by that. Obviously people's opinions are important, or you wouldn't be asking for feedback. And yes, everyone values their opinions more than others'. However, there are some people who... you read their online review, and you can tell that they believe their review will impress everyone and settle all disputes. Like you'll read a negative Yelp review, and the reviewer isn't just saying, "I had an bad experience," or "I didn't like it," but something more like, "This place is simply objectively terrible and though I see other people saying that they like the place, they're all wrong and stupid and not worth listening to." You can almost imagine that they've finished writing the review, leaned back in their chair, and thought, "Well that waiter crossed the wrong person. I expect they'll go out of business any day now."

    So my

    • I think it's also important to consider that satisfaction surveys tend to suffer from a sort of selection bias. You're only getting feedback from people who feel compelled to give feedback.

      I worked for a big, big global company for a number of years. IT was run from call centres around the world, so that anybody working anywhere would always find somebody in a convenient timezone to work on the problem for at least a few hours, even if the ticket was submitted at the end of the business day for the person with the problem.

      And if the IT people fixed the problem quickly, there was always a request to complete a satisfaction survey.

      Curiously, if they didn't fix the problem quickly, there was no

  • I also once read that customer satisfaction surveys are, in general, only answered by people who were on the extreme ends of the customer satisfaction ratings; that is, most of the time the only people who bother with a survey are those who had a really awesome or a really bad experience. The experience left them emotionally charged, and they feel the need to share/vent, and the survey gives them that opportunity. The average customer, on the other hand, doesn't have this emotional need and so - when offered an opportunity to rate their experience, gives the whole thing a pass and - if forced into it - tend to give wishy-washy middle-of-the-road answers (e.g., all 5 out of 10s) just so they don't have to think about it.

    I imagine it is also easier to be on the receiving end of a "terrible experience" than an awesome one too, which only further biases the reviews. I mean, as a customer I can imagine a dozen ways in which a cashier could piss me off, but have a hard time thinking up a way that cashier could make me EXCITED about paying for my groceries). So the end result is very biased against the person getting reviewed, because most of the people who bother to respond to the reviews are those who had a bad experience. If they forced /all/ customers to take the survey and somehow ensured their honesty, you might get a better overview of how the employee is performing, but by leaving it up to the customer the employer is getting a very unbalanced response.

    Not to mention I never know how to answer those because I never know what is considered "average". Is a 5/10 average or do I have to base it on the US grading system, where 7/10 is "average"?

    • Well, I have dealt with several companies which have apparently decided to fix the problem with people not responding to surveys by repeatedly emailing me to piss me off enough to respond to the survey. I usually put in the comments, "I have no idea what the issue was that this ticket was about any more, nor do I remember what the service was like, but you badgered me with emails to fill out this survey often enough to make me angry."
  • I do third to final tier support. My ratings from customers and lower tier support is abysmal because most of the time, I have to say "Nope! Can'tWon't do that." Lost a promotion because of the score as well, so it cost me around $15K.

    My favorite rant from someone that said I was "stupid" because I can't configure a F5 to act as a master/master SMB share. Oh, I could get them something like SMB with HA, but not for the incremental price (about $4 a month) they were looking for.

    I am actively seeking other em

  • by jimbrooking ( 1909170 ) on Thursday January 26, 2017 @10:25PM (#53746471)
    I don't think these surveys are worth crap. The companies that rely on them to rate employees have HR policies that suck. I usually just ignore the pleas from "anonymous data collectors" about my visit to see my doctor, or the pizza order (really!). When an employee implores me to rate my visit (for which she is being held responsible) highly, in what might be a personalized way, I just rate the "experience" at 10's from start to finish unless they performed some egregious bit of malpractice on me or my car. It's like tossing a fiver into the hat of a street musician. He needs it and it costs nothing to help him out. I'm hoping that after seeing a few of these the companies will stop asking for my meaningless "feedback". Otherwise, I will continue (with little enthusiasm or optimism) to try to improve the quality of life for those who depend on 10's.
  • You see it a lot of places other than consumer ratings. For instance, whisky is commonly rated on a 0-100 scale, where 70 or below is considered "shit, not worth buying". Same thing with ratings in game magazines, where 70% is "crap" or "only for really dedicated fans".

    And Ebay reviews. Every single time I buy something from China, I get a bunch of messages thanking me for shopping with them, and expressing an expectation that I will give them a perfect 5-star review, for nothing more than shipping me the i

    • You might find this appropriate https://xkcd.com/937/ [xkcd.com]
    • The current ratings system has made using customer ratings to evaluate products almost useless. You can gain a little bit by reading the comments, but that is about it. It drives me nuts when I am trying to decide if a $50 device is worth it vs the $5 device that lists exactly the exact same functionality (sometimes the price difference is because the more expensive item is prettier, sometimes the price difference is because the more expensive item is more durable...greater durability might be worth that pr
      • On Amazon and similar sites, I find it best to simply ignore the 5-star and 1-star reviews, because they're usually sycophantic for the 5-stars, and pissed off bullshit for the 1-stars.

        The reviews in the middle are much more interesting, since they actually put some kind of effort into choosing a rating.

One person's error is another person's data.

Working...