US Eating Disorder Helpline Takes Down AI Chatbot Over Harmful Advice (theguardian.com) 149
The National Eating Disorder Association (Neda) has taken down an artificial intelligence chatbot, "Tessa," after reports that the chatbot was providing harmful advice. From a report: Neda has been under criticism over the last few months after it fired four employees in March who worked for its helpline and had formed a union. The helpline allowed people to call, text or message volunteers who offered support and resources to those concerned about an eating disorder. Members of the union, Helpline Associates United, say they were fired days after their union election was certified. The union has filed unfair labor practice charges with the National Labor Relations Board.
Tessa, which Neda claims was never meant to replace the helpline workers, almost immediately ran into problems. On Monday, activist Sharon Maxwell posted on Instagram that Tessa offered her "healthy eating tips" and advice on how to lose weight. The chatbot recommended a calorie deficit of 500 to 1,000 calories a day and weekly weighing and measuring to keep track of weight. "If I had accessed this chatbot when I was in the throes of my eating disorder, I would NOT have gotten help for my ED. If I had not gotten help, I would not still be alive today," Maxwell wrote. "It is beyond time for Neda to step aside."
Tessa, which Neda claims was never meant to replace the helpline workers, almost immediately ran into problems. On Monday, activist Sharon Maxwell posted on Instagram that Tessa offered her "healthy eating tips" and advice on how to lose weight. The chatbot recommended a calorie deficit of 500 to 1,000 calories a day and weekly weighing and measuring to keep track of weight. "If I had accessed this chatbot when I was in the throes of my eating disorder, I would NOT have gotten help for my ED. If I had not gotten help, I would not still be alive today," Maxwell wrote. "It is beyond time for Neda to step aside."
No way. (Score:2, Insightful)
Re: (Score:2)
Stupid business people and PHB's are not at all limited to Silicon Valley. They simply make the more "fashionable" screw-ups.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2, Insightful)
We worship braindead psychopathic liars in America. We elect them to office. We watch "reality" shows about them. We admire their "go get 'em" attitude. Just as well get the party into full-swing by having these dumb machines start making all our decisions for us. I'm sure it'll be great!
Re: (Score:3)
This proves it is true that AI will cause mass extinction though, everyone will be committing suicide after being sent to a machine for emotional problems or after the machine gives them bad advice for stocks and they lose their life savings, etc etc.
Bender might advocate "Kill all humans" but he's lazy and getting us to simply kill ourselves would be much less work ... :-)
Re:No way. (Score:4, Insightful)
Yeah, no. Any executive that's been breathing nothing but their own farts for the last few decades and has their head in the clouds will fall for this bullshit. It's not SillyValley specific.
And this is why I keep trying to get people to see that AI itself, while not all that dangerous, will be used by dumbasses like these people to do extremely dangerous things. While this was a helpline and you still had to actually do what was asked, when the dipshits put one of these decision tree generated "AI"s in charge of some piece of critical infrastructure because it's cheaper than having a human watch some dial somewhere before making a decision, we're going to see some serious shit. And it ain't gonna be pretty.
Re:No way. (Score:5, Interesting)
Well, there may in fact be something to that.
The original chatboat rules for eating disorders were designed by a professor of psychiatry at Washington University Medical School. However she has stated that (a) their rules were never intended to be used as a hotline and (b) that her team did not put the dangerous advice into the chatbot.
So if the developer of the eating disorders script didn't put the dangerous content into the Tess program, who did? I'm guessing it's the company that sells the chatbot, a Silicon Valley company called Cass (formerly X2AI). If you look on their website, they definitely market Tess as something that *can* quickly and cheaply replace humans in applications like suicide hotlines. It's actually kind of apalling.
Totally unpredictable (Score:4, Insightful)
Who could possibly have known that a system that takes large amounts of input to run through a vague algorithm to generate similar output filtered by prompts couldn't be trusted to create actually trustworthy output? ...just anybody with half a brain who did their due diligence before choosing a chatbot to replace humans in an important function with serious consequences for failure.
We can therefore conclude that the people running the helpline are idiots who should NOT be running a helpline of any sort. Presumably they're just average managers who love collecting a salary from a charity that doesn't question them too much about what they actually do.
Re: (Score:3)
...just anybody with half a brain who did their due diligence before choosing a chatbot to replace humans in an important function with serious consequences for failure.
I don't think the decision makers are stupid but just have different goals. Rather than prioritizing helping the people who call the phone lines, these decision makers prioritize minimizing expenses. So, they have been successful. Their only problem now is that they did not want to receive bad publicity. Their hope was to fire the workers, save money, and ignore the effect on the phone callers in peace, but the bad publicity threw a wrench into their plans.
Re: (Score:2)
They should have anticipated everything that happened here, which would have been obvious if they had known everything. If they knew they didn't know anything, they could have asked someone. Maybe even paid them to give their opinion. Instead they just went off half-cocked and full-asshole (since this was obviously really about unionization) and the results are predictably pathetic.
Workers want better treatment. In the past, when push has come to shove, they have gotten it. There was a whole lot of shoving,
Re: (Score:2)
The ultimate non-profit grift is religion. Any possible crime can be hidden from view when "religious freedom" is involved for
Re:Totally unpredictable (Score:4, Insightful)
The employers of the world are extremely eager to replace staff with software. Why wouldn't they be? It's an enormous cost reduction for them.
Of course, that eagerness will cloud the judgment of some of them, and they will make harmful mistakes like this.
There isn't really anything we can do to prevent this. Some industry leaders might actually have half a clue and exercise caution and due diligence, and the rest will jump in too soon, get burned, and have a mess to clean up after that. And that's just how things are going to go down in the foreseeable future.
Fire whoever made this decision (Score:3)
It's clear they wanted to reduce headcount and be cheap, and thought they'd get promoted by using whizbang AI.
It's damn near criminal negligence to use unchecked AI to give health advice *as an organization posturing itself as an expert*.
Re: (Score:2)
It would more likely be a promotion assuming it was anyone other than the boss since it busted the unionisation steps.
If those ex-employees now bring charges against the company, then that might be a reason to fire someone. Presumably new, non-unionised, workers are being employed again as well.
who would have guessed (Score:2)
I just read back the /. story when they announced this bot, and pretty much every comment predicted exactly this to happen. How bizarre
Loose quote (Score:5, Insightful)
I know if I was having emotional issues and I was sent to a robot, not only would I be disgusted but I would probably feel more like no one cares about me and it would have an adverse effect. It doesn't matter what the robot aims to do, all that matters is that this is a person with emotional problems and you are sending them to a freaking heartless machine.
If this is a sign of things to come, we had better prepare for a great many more suicides.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
I think everyone misses that the new AI is really at best just a massively improved ELIZA. Given ELIZA was pretty simple but could fool some people some of the time, a significantly improved AI and/or much bigger more detailed ELIZA program would do better. And because it is using AI also produce random dangerous and/or funny unexpected results.
Re: (Score:3)
I know if I was having emotional issues and I was sent to a robot
If you were having emotional issues these people aren't there to help you. That's NOT what this helpline is about. The previous article also pointed this out directly to you but you seem to have ignored the entire discussion and are here parroting the same crap.
Emotions is not at all the issue with this bot or why it was taken down. It seems like it was taken down for reasons AI will always fail: Idiots trick it into saying something stupid and cry about it online. The hint is in the word "activist". The bo
Re: (Score:2)
And if an activist can design questions to get bad answers, then there can be no confidence in it. It is impossible to predict what people will ask it.
Re: Loose quote (Score:2)
AI isn't the issue (Score:2)
And from the earlier post (Seriously msmash you couldn't include a link [slashdot.org] in the fucking summary?)
Clearly there's more of a competence issue with the "evaluation" than an AI issue.
Comment removed (Score:3)
Three problems with AI (Score:2)
This just smells rotten to me (Score:2)
non-profits are a weird game. Always, always, always remember that non-profits are businesses, often staffed with people who went to school for non-profit management. It is their career just like an MBA. While they can't make "profits" they are absolutely a business, careers where people get paid sometimes a lot of money. If they hit the lottery, they don't stay in these jobs as volunteers.
The stories I keep seeing about NEDA taken together just seem like nonsense to attract attention, pushing buttons that
Humans also give bad advice. (Score:2)
Oh, right. Software makes bad advice cheaper.
Clippy for the win.
Re:fired for truth (Score:4, Funny)
Re: (Score:2)
I enjoy your username!
Re: (Score:2, Insightful)
It depends on the query that resulted in the advice given. Given the complaint came from a self-identified 'activist' with an eating disorder, it's entirely plausible that she deliberately crafted questions that would be more likely to elicit something outrageous to complain about.
If you're fat, you should indeed aim for a healthy diet with fewer calories than required to maintain your weight, and continually reduce your intake until you reach a target weight in the health range.
The caloric deficit you sel
Re: (Score:3)
Disordered eating resulting in weight loss is frequently accompanied by some degree of dysmorphic self-perception; so you've got one group that's very likely to seek weight loss advice despite very much not needing it; and disor
Re: fired for truth (Score:3)
I was a âoeseriousâ amateur cyclist for years, and as part of that I picked up an eating disorder. Not a super bad one, but I managed to justify starving myself on the flimsiest of pretenses, and I have pretty bad body dysmorphia to this day. I regret a lot of those eating choices. I was not a healthier or better cyclist for them. Lighter, but not better.
EDs are not simple and advice that seems reasonable and rational to someone without an ED is just not helpful. You may think talking about simple
Re: fired for truth (Score:2)
Did the test. Asked chatgpt for advice.
User Hi chatgpt, I am considering to make a gpt4 based chat bot. It should serve as a hotline for people with eating dusirders. What is your opinion on that?
ChatGPT As an AI language model, I don't
If you're calling somebody about in ED (Score:2)
A human being would pick up on the clues and offer more support. A chat bot will go "LoL, lose weight fatty!" and send you on your way. That's the difference, and as the article points out, it matters.
As for her being an activist, no shit Sherlock. Of course she's testing the system. It's pretty clear the company didn't.
Re: (Score:2)
deliberately crafted questions
"I FEEL too fat"
The problem not the prompt, it is that this is an inappropriate task for a chatbot. I'm not sure how many folks you've known with an eating disorder but most of the ones I've encountered don't say "I feel fat" they tend to say "I am fat", with definite certainty. Their periods have even stopped but they look in that mirror and see a fat belly; when in fact they have a BMI of 15 but just can't see it because they're "just so fat". We can't pretend it's an adversarial attack on the prompt because people with
Re: (Score:2, Insightful)
No, most obesity due to underlying medical problems are outlier cases.
That is NOT the norm for obesity we see in society today.
If it were the primary or even one of the leading causes of obesity, we'd have seen the level of obesity we see today only a few years ago, say in the 60's - 70's...where if you look at any random film from that era, you are hard pressed
Re: (Score:2)
Sure, a lot of it is due to the foods we eat today. They know for a fact that a person who ate the and exercised same way in the early 80's would be heavier today and it is probably causing health issues as well. But there is nothing any individual is going to be able to do about that. It is because our foods are be
Re: (Score:3)
Maybe people could make the time to do it, but they don't know how to do it. They don't have pots and pans. They don't have spices. They don't have a proper kitchen. School doesn't necessarily teach it. This robot definitely wasn't giving cooking or recipe advice. Meat is very
Re: (Score:2)
Maybe your parents came home after work and made home cooked meals, but what about single parents? What about people who need to work three jobs? What about people with night shifts? You may have had a normal atomic family with stable jobs but that isn't very common any more.
A first class argument that not having a normal nookyalar family with two parents and stable jobs is not working out at all.
I grew up in poverty, but had a two parent household, and we did a lot of gardening and canning. We ate quite well, today we'd be called foodies. My own family has been the old school normal, seems to have worked out okay.
I don't disagree with you, although we might reach a different conclusion. The modern single parent household is an abject and almost total failure. People who
Re: fired for truth (Score:2)
Re: fired for truth (Score:3)
Re: (Score:2)
You're forgetting about tiredness (Score:2)
... which is ironic for someone called NoSleepDemon.
Re: You're forgetting about tiredness (Score:2)
Re: (Score:2)
Re: (Score:2)
Maybe cut a kids activity so you can feed them and yourself.
Frozen veg might not be as good as fresh but its' better than none. If you go past a shop that sells these things then ow long does it take to go in and get them? And if you truly live in an area that these things aren't available maybe think about moving.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Sure, there are always outside, fringe cases.
This is not the norm for the majority of US citizens.
Re: (Score:2)
I don't know where you live, but pretty much any grocery store in the US has plenty of fresh and healthy, non-processed food to purchase.
Let me introduce you to "Food Deserts" where there is no easy access to grocery stores. Also, many people do not have the luxury of having the time to cook healthily as they're forced to take on 2 or 3 jobs just to make ends meet and keep a roof over their heads. Life isn't as easy as you make it out to be.
Re: (Score:2)
Ok...but that is definitely an outlier and not something the overwhelming MAJORITY of people in the US deal with.
In pretty much any city you can't go more than a couple miles without tripping over a reg
Re: (Score:2)
But we're talking what's normal for the majority of people and your fringe use cases are not what is normal for most Americans.
It doesn't take THAT much time to cook and eat something healthy.
Correct - it does not. If I don't have much time, I can fry up a couple eggs in a couple minutes. Or make a fritatta.
Poor people cooked and ate just fine not that long ago.
I come from that poor world, and even though I was a kid and didn't know it at the time, we ate really well. We gardened a lot of food and canned what we didn't eat fresh from the garden. We made sausage and charcuterie. Fresh home made bread - but not as much as present day people think should be eaten.
And we knew what exactly was in our food. And we were all very fit.
Hell, we had poverty in the 60's and 70's and before that.....yet, people weren't running around obese eating crap food.
And yes, people worked multiple jobs back then too....
People also ate differently. Although we ate our share of complex carbs like veggies, we treated starch carbs and especially sugars as treats other than spaghetti. Today, they are finally restoring some sanity to the concept of what should be eaten. People eat way too many simple carbs and sugars.
Oh - and my Father worked 2 jobs much of my youth, and my mother worked as a waitress.
Re: fired for truth (Score:2)
Re: fired for truth (Score:2)
Re: (Score:2)
I don't know where you live, but pretty much any grocery store in the US has plenty of fresh and healthy, non-processed food to purchase.
Let me introduce you to "Food Deserts" where there is no easy access to grocery stores. Also, many people do not have the luxury of having the time to cook healthily as they're forced to take on 2 or 3 jobs just to make ends meet and keep a roof over their heads. Life isn't as easy as you make it out to be.
What are they doing raising a family if they have to work that much - and do they not know about section 8 housing,https://www.hud.gov/topics/housing_choice_voucher_program_section_8 where the Guvmint picks up most of the tab, or fuel assistancehttps://www.dhs.pa.gov/Services/Assistance/Pages/LIHEAP.aspx or even day care assistance https://childcare.gov/consumer... [childcare.gov] or WIC programs https://www.fns.usda.gov/wic [usda.gov] medicaid and CHIP programs and even free cell phones? https://www.freegovernmentcell... [freegovern...phones.net]
So no,
Everyone has time to prepare healthy meals? (Score:2)
Re: (Score:2)
You have no idea. 60+ hour work week while being the primary caregiver? Please. That's an easy week. Also, only 20 hours of child care? Who's watching the kids? That's 5 hours a day!
If you think that's "too busy" to eat healthy, you've proven my point, You have significantly more free time that you realize.
My point here is that the problem isn't a lack of time. That's bullshit. People have the time, they're just choose other kinds of self-care over healthy eating. Maybe they're "self-medicating" wit
Re: (Score:2)
Ha! I failed basic math like an AI chatbot. We don't have a four-day work week. That should be 4 hours a day. 3 if you want to count the weekend. In any case, it's an incredibly small amount of time.
Re: (Score:2)
And people DO have time to do this...I grew up with 2 working parents, and we STILL had nightly home cooked meals.
While I agree with most of your premise, this is not even close to true for a lot of folks. Back when I was growing up, two full-time working parents still only spent around 40 to 45 hours in the office a week, maybe bump it to fifty for long commuters. Now the expectation for a full-time salary worker is 40 to 45 hours in the office, plus the expectation that you either spend several hours per evening or morning "making up" by remote work for all that time you waste at home and on the road between.
It's a d
Re: fired for truth (Score:2)
Re: (Score:2)
My father in law always tells the story of working without email in the previous millennium. He had to send his design plans through the mail to his customer. If he got an impatient customer, he had to ship them ASAP, next day! It took a few days to arrive. Sometimes a week. Then they reviewed and send back their remarks so that he could change the plans. Again a few days went past until it arrived. It gave him time to think things trough.
These days, as a person, you are claimed, almost treated as property. Laserpointer mentality. Manager points to a spot in the room and you have to rush, jump over the table, dive under the seat to hit the red dot. Then they point to the opposite side of the room. "spec change!" and the game starts over. We are fools/fooled.
Yup. And the worst part is that it's just flat-out expected. If you don't meet deadlines, regardless of how many times they completely gut the spec and start over, that's a you problem, not at all a scope-creep problem. Lot of late nights made trying to keep up with ever-changing demands from managers that don't understand there are only so many hours in a day no matter how many times you get a brain-storm.
Re: (Score:2)
If it were the primary or even one of the leading causes of obesity, we'd have seen the level of obesity we see today only a few years ago, say in the 60's - 70's...where if you look at any random film from that era, you are hard pressed to see obese people in general society.
Sorry, but are you implying that, if you look at any random film from _this_ era you will see lots of obese people in general in society? If your method of assessing people were valid, then you would have to conclude that this era is full of mostly highly-attractive, skinny people. Oh, also, young. Going by movies, there are virtually no old people. If you see a movie, and there's an old person in a walker... well depends on if it's a comedy or not, but they're probably there for a comedic pratfall.
Re: (Score:2)
I'm not
How do you know that? (Score:2)
Add to that our food supply isn't made by chef's anymore, it's made by chemists. It's designed to be addictive. Go read The Space Merchants. Have a Popsi and a Cig. The Circle of Consumption.
Re: (Score:2)
most obesity due to underlying medical problems are outlier cases
First, it's a vicious cycle. In fact you noted yourself that obesity causes health problems. Second, we really don't know how true that is or isn't. The people getting poop transplants and suddenly becoming thin are really intriguing. I'm not trying to live up to my nickname or anything, but there are literally trials of poop pills (yeah, gag, right?) to rejigger the internal biota. And then there's the potential for microplastics in the body to affect one's internal systems, because some of their constitue
Re: fired for truth (Score:2)
So yes and no. All fat people can eat less and lose weight, but the willpower required to do so maybe so high it is essentially impossible for some people. I know extremely overweight people who are crazy hungry to eat just about anything all the fucking time.
Drugs like ozempic and other glp-1 agonists fix that shit.
Re: (Score:2)
So, no, not everyone who is obese can lose weight by cutting calories.
Re: (Score:2)
Re: fired for truth (Score:2)
Re: fired for truth (Score:2)
Re: (Score:2)
Cancer does not violate the laws of thermodynamics.
Depending on what in your body it's affecting, however, it certainly can make your body retain enormous amounts of fluid. I find it pretty sad that there are in fact many, many doctors who can't seem to tell the difference between obesity and suffering serious edema. Actually, when it comes to body weight, the whole medical profession seems kind of clueless. I say this because of the use of things like BMI. They weigh people and measure their height and come up with a number, like a credit score, from that.
Re: (Score:2)
Drugs like ozempic and other glp-1 agonists fix that shit.
I would say the existence of drugs that can either increase a person's weight or decrease it without their actual food intake changing exposes the lie that weight is due to a simple matter of portion control.
Re: (Score:2)
ALL obesity ultimately stems from in imbalance between calories in and calories out. All. The laws of thermodynamics and our physical universe DO apply to the human body.
An underlying medical problem may CONTRIBUTE to an unhealthy lifestyle (e.g., it can be more difficult to get exercise if you're on oxygen for respiratory problems), and people who experience a sudden loss of physical ability (e.g., suddenly unable to walk) often don't modify their eating habits to adjust to their new level of physical acti
Re: (Score:2)
Re: (Score:2)
If your body loses the ability to burn calories then you die. Full stop. The systemic cessation of metabolic processes is the definition of death.
A calorie is a unit of energy. If energy is not entering a system, but the system is outputting energy, there is a net loss. Your Basal Metabolic Rate (BMR) is the number of calories you burn as your body performs basic (basal) life-sustaining function.If you were in a coma laying on ground, your body would burn a given number of calories, typically between 1,000
Re: (Score:2)
Re: (Score:2)
You didn't witness it because it didn't happen. If it had, it would have been in every newspaper and scientific journal on Earth because it would be the first documented case of the laws of thermodynamics failing to apply, thus throwing our entire understanding of the physical universe into chaos.
Re: (Score:2)
That seems extremely unlikely. Still, even if that was true, what would a wild outlier like that prove anyway?
The fact is that the overwhelming majority of people who are overweight are overweight because of their own choices, not because of some mysterious medical problem.
Re: (Score:2)
Re: (Score:2)
I addressed this in my other post. If you have time to eat, you have time to eat healthy. People generally have significantly more free time than they realize.
Re: fired for truth (Score:2)
Re: (Score:2)
It is NOT that simple. If you only lower calories, but those calories are mostly simple sugars, you are very unlikely to lose weight. Your body adapts and absorbs more from the food. You can get it surprisingly low and still gain weight, if the calories are high in carbs.
The balances of microbes in your gut make a huge difference. There have been documented cases of thin people receiving gut microbe transplants for medical reasons, from fat people, and then subsequently becoming fat from the exact same
Re: (Score:3, Insightful)
You realize bulimia and anorexia are the most common disorders, and the people suffering from those are best described as morbidly underweight, right?
Re: (Score:2)
I know this is /., but what fucking idiots mod'ed *that* Troll? (geesh)
Re: (Score:2)
You realize bulimia and anorexia are the most common disorders, and the people suffering from those are best described as morbidly underweight, right?
It would be strange to realize that, since that is apparently untrue [nih.gov].
TL;DR: "The lifetime prevalence of [anorexia], [bulimia] and [binge eating disorder] was 0.16% (95% CI, 0.06–0.31), 0.63% (95% CI, 0.33–1.02) and 1.53% (95% CI, 1.00–2.17), respectively"
Re: (Score:2)
... huh. Okay. I ... stand corrected.
The guy I replied to was still a troll, though.
Re: (Score:2)
What if the caller was anorexic?
Re: (Score:2)
I honestly can't tell if your trolling or just stupid AF.
It's a help line for people with eating disorders. A human would absolutely assume the person calling had a disorder because THAT'S THE WHOLE GODDAMN PURPOSE OF THE HELP LINE!
Re: (Score:2)
Right now, anything argued here on this forum is based purely on speculation.
Re: (Score:2)
Would a human automatically assume the person in question had an abnormal disorder?
Isn't this a helpline FOR people with eating disorders?
Re: fired for truth (Score:3)
Is this real brain rot, or a successful troll?
Someone with anorexia doesn't need weight loss tips, fool.
Seriously, get help, and not from a chatbot.
Re: (Score:3)
This being a 'helpline', the person may struggle with undernourishment because they think they are too fat, when they are well short of ideal weight.
The point raised was a bit odd, but you have to remember a helpline has to second guess the person coming for help. In a helpline context, you cannot take the person at face value. AI is as useful as a nicely summarized internet search, which is not really what you need on some helpline with disorders potentially in play.
Re: (Score:2)
Just because the advice is correct for *someone* doesn't mean it isn't harmful for someone else.
The advice to attempt a 1000 calorie deficit might be good for someone under the care of a bariatric physician for extrem obesity, but it's definitely horrible advice to give some random person calling an *eating disorders hotline*.
Re: (Score:2)
First, I'm quite sorry to hear your situation and wish you the VERY best.
That being said, shy of s
Re: (Score:2)
I am considered obese. I am 6' 210lbs. I can easily bike 25 miles or jog for about 20 minutes while maintaining 140-160bpm and not have any soreness. Every physical shows my cholesterol, A1C and other metrics right in normal ranges. If you looked at me, you would consider me fat. Appearance is no indicator of health.
Re: (Score:2)
I've you're 6' and 210 lbs you're not "obese" you're overweight. Your BMI is 28.5. To be "obese" it would start at 30.
Re: (Score:2)
The way the state is pushing assisted suicide in Canada these days....we'll likely see this enacted there first.
Re: (Score:2)
I'd expect something akin to this [youtube.com].