Researchers Say AI Tool Used in Hospitals Invents Things No One Ever Said 69
AmiMoJo shares a report: Tech behemoth OpenAI has touted its artificial intelligence-powered transcription tool Whisper as having near "human level robustness and accuracy." But Whisper has a major flaw: It is prone to making up chunks of text or even entire sentences, according to interviews with more than a dozen software engineers, developers and academic researchers.
Those experts said some of the invented text -- known in the industry as hallucinations -- can include racial commentary, violent rhetoric and even imagined medical treatments. Experts said that such fabrications are problematic because Whisper is being used in a slew of industries worldwide to translate and transcribe interviews, generate text in popular consumer technologies and create subtitles for videos.
[...] It's impossible to compare Nabla's AI-generated transcript to the original recording because Nabla's tool erases the original audio for "data safety reasons," Nabla's chief technology officer Martin Raison said.
Those experts said some of the invented text -- known in the industry as hallucinations -- can include racial commentary, violent rhetoric and even imagined medical treatments. Experts said that such fabrications are problematic because Whisper is being used in a slew of industries worldwide to translate and transcribe interviews, generate text in popular consumer technologies and create subtitles for videos.
[...] It's impossible to compare Nabla's AI-generated transcript to the original recording because Nabla's tool erases the original audio for "data safety reasons," Nabla's chief technology officer Martin Raison said.
LLMs are LLMs, news at 11 (Score:3)
Re:LLMs are LLMs, news at 11 (Score:5, Insightful)
Traditional speech-to-text is often a bit on the rough side; but it has the 'virtue'(in a sense) of breaking in stupid visible ways if it chokes on a bit of input. You'll get a similar-sounding word that has no business being in that part of a sentence, or a sentence-length or two of total word salad if there's a burst of background noise or a mic level issue or something. It's not a pretty looking failure; but for the same reason it's not all that sneaky. Not as good as a system that gracefully admits that the input is unusable from timestamp A to timestamp B and tells you as much; but a fair way from the exceptionally smooth confabulation you get out of LLMs.
Re: (Score:2)
If MS Teams uses an LLM to generate its transcripts, then no. That is _not_ a safe application. But MS may cause numerous errors in the Teams transcripts using another substandard technology. Does anybody know?
Re: (Score:2)
Teams already produces transcripts with numerous errors using substandard technology. Unless you speak slowly and use small words, expect to read the transcript a few times to make sure you understand what was said.
Re:LLMs are LLMs, news at 11 (Score:4, Informative)
But seriously, transcription is probably relatively safe.
Sounds plausible, but the actual article we're discussing says otherwise.
Re: (Score:3)
Transcription itself is safe. It's just that depending on it for anything significant isn't. (And note that the original data has been"erased for "data safety reasons".) So if the transcript says to use one drug, and the doctor says he ordered another there's no way to check.
This should all be obvious to those who read Slashdot, but folks who are the general public, or even medical professionals, might well not understand the problems.
Re: (Score:2)
Except didn't we just hear these people claim [slashdot.org] that bullsh***ing I mean "hallucinations" was a solved problem?
Re:LLMs are LLMs, news at 11 (Score:4, Interesting)
Except didn't we just hear these people claim [slashdot.org] that bullsh***ing I mean "hallucinations" was a solved problem?
It won't ever be a solved problem of course. At some point, the "truth" will become AI referencing AI, so hallucinations will become whatever AI decides is truth.
Re: (Score:2)
That won't ever be a solved problem. People do it too. But people have a contextual map of the problem that lets them estimate the likelihood of misinterpretation, and this lets them eliminate lots of possible interpretations of the stuff they heard. (And even so, sometimes they eliminate the things actually heard.) This is one basis of humor.
Re: (Score:2)
Re: (Score:2)
That seems to be the problem - it hears some speech and then hallucinates extra sentences that seem to fit the structure.
AI Politician/Political Reporter (Score:1)
Gosh, I can't imagine who would want this. (Score:2)
Re: (Score:2)
Does it save a nickel in operating costs?
Re: (Score:2)
The problem is ... (Score:3)
... no one really knows in detail how these things actually do what they do. They understand the high level feeding in data and guff about N dimentional matrices of semantic relations, they understand the low level side of back propagation setting neural weights, but theres that fuzzy in the middle part to which no one can quite get their hear around whats happening. Frankly given these models have ever increasing billions of artificial neurons I wonder if anyone really will.
Re:The problem is ... (Score:4, Informative)
Or to be more precise, while the actual mechanisms are somewhat understood, the training data is generally not understood at all.
Re: (Score:2)
That's important, alright, but you also don't know which part of the training data the AI was paying attention to. There have been examples where it was attending to a time of day code encoded into the photo, that people didn't even see.
Erase the original for "safety"? Are you insane? (Score:5, Insightful)
Those experts said some of the invented text -- known in the industry as hallucinations -- can include racial commentary, violent rhetoric and even imagined medical treatments.
Okay, that's a problem. A serious problem by any standard.
Nabla's tool erases the original audio for "data safety reasons,"
And that's a much, much bigger and more serious problem. Without the original how would you even know if anything was changed, added, or removed? Obvious things, sure, but what if a dosage was altered or the results of a biopsy (for example) were reported as "clean" when in fact it was not?
Re: (Score:2)
It almost sounds to me like the AI generated text is a dubious legal dodge to avoid being responsible for HIPPA compliance.
Which raises the question of whether they're turning around and selling the (dubiously accurate, hallucinated) medical conversations to advertising partners or something.
Re: (Score:3)
This clearly is to make litigation harder. Avoiding HIPPA compliance may also be a factor. The deletion is in any case clearly malicious.
Re: Erase the original for "safety"? Are you insan (Score:1)
Re: (Score:2)
If patients haven't given permission for their visit to be recorded then why should their visit allowed to be transcribed?
I don't see any common scenario where someone would want to keep something private but allow a less accurate version to be recorded.
Re: Erase the original for "safety"? Are you insa (Score:2)
My doctor asked for consent to record for transcription and I live in a single party audio recording consent state! Not that hard to do...
Re: (Score:3)
Start with assuming someone had lazy logic, which is 9 times out of 10 the fault of something, before you jump to nefarious reasons.
Re: (Score:2)
That doesn't mesh with "date safety" as a stated rationale. I'm all for Hanlon's razor, but this theory doesn't quite match the data.
Re: (Score:2)
Artificial intelligence is not able to overcome natural stupidity. AI, in its current form, is not ready for prime time. But dumbass humans lookin' to save a few more pennies will happily let it play in prime time, while declaring great victory over some nefarious imagined foe.
This has been the only real fear of AI I've had all along. Not that it's going to replace us well. But that it'll be used to replace us poorly. In critical roles. Like hospital administration. Oh well. Not like the uber-rich will use
Re:Erase the original for "safety"? Are you insane (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Nabla's tool erases the original audio for "data safety reasons,"
And that's a much, much bigger and more serious problem. Without the original how would you even know if anything was changed, added, or removed? Obvious things, sure, but what if a dosage was altered or the results of a biopsy (for example) were reported as "clean" when in fact it was not?
Any doctor that uses such a system should lose their medical license, and be criminally prosecuted for every bad thing that happens before they get caught.
Human Mistakes (Score:2)
How does it compare to humans as far as accuracy? Humans and definitely not infallible.
Re:Human Mistakes (Score:5, Informative)
Humans can be held accountable. If you had a medical scribe who wrote "Patient has testicular cancer requires immediate amputation" for no goddamn reason on a transcript, they'd be liable for medical malpractice.
If a computer does it, "oops, bug, no one's fault really, but your balls are in this nice jar here"
This is one of the Big Risks of the current crop of AI horseshit, that a (unjustifiable) decision can be made without anyone being "to blame"
Re: (Score:3)
Maybe we should make companies responsible for the software the put out.
Re: (Score:2)
Humans can be held accountable. If you had a medical scribe who wrote "Patient has testicular cancer requires immediate amputation" for no goddamn reason on a transcript, they'd be liable for medical malpractice.
If a computer does it, "oops, bug, no one's fault really, but your balls are in this nice jar here"
This is one of the Big Risks of the current crop of AI horseshit, that a (unjustifiable) decision can be made without anyone being "to blame"
I expect, if the AI hype remains within the corporate circles, we're going to see a whole lot of, "Oops. The AI did it. Nothing to be done. Next," throughout the world. It's insane how much people have bought into the hype without one moment of critical thought or, you know, experimentation to see whether it's true or not. Full steam ahead and damn the torpedoes. We got pennies to save and people to get off the payroll!
Re: (Score:2)
Re: (Score:3)
I put Alexa through a similar but simpler loop a while ago.
Alexa, where am I?
.... no.
You are in Lancaster.
Alexa, where is Lancaster?
Lancaster is a medium sized city in the UK.
Alexa, am I in the UK?
Re: (Score:3)
Doctors enter hallucinations into records already. (Score:1)
Are the robotic transcriptions marked as such when they are copied into your medical records?
My doctors already enter stuff into the records that they never said. It appears that they paste canned paragraphs into "Notes" of visits.
I've been listening very carefully and taking notes.
(I accompany a disabled friend who uses a different health system and different doctors. Same thing.)
"...the tool has been used to transcribe an estimated 7 million medical visits."
This article would be more helpful (Score:2)
Re: (Score:2)
If it would specify the *exact* hospitals that are using these tools so everyone here can avoid those places. The article doesn't mention the names of these facilities so it's not generally useful to the readership that is trying to make an informed decision.
No fear, citizen! Soon insurance will require AI transcripts from any hospital they provide payment to as a safety measure. You and your doctor need not fear. Neither of you will be in charge of the decisions.
And the hype train rolls down the track... (Score:4, Insightful)
...picking up speed.
LLMs exhibited unexpected emergent behavior. This got the train rolling.
Investors hopped aboard, and the speed increased. Problem is, investors want profits NOW.
Early adopters hopped aboard because they needed to convince their investors that they were using "the next big thing" and it allowed them to reduce costs.
Problem is, AI is a research project that will take years to be really useful and today's offerings suck mightily for real work.
Expect the crapfest to continue as the hype train continues gaining speed
Re: (Score:2)
Maybe we will get lucky and there will be a rather abrupt and terminal stop: An LLM may be involved in somebody getting killed by malpractice and the hospital responsible gets sued into the ground.
LLMs are good for only some things (Score:2)
LLM are good at identifying things (Cancer, cars, etc.)
There are not good at making complex decisions or programming. That needs to be something else on top of LLMs.
Re: (Score:2)
LLM are good at identifying things (Cancer, cars, etc.)
There are not good at making complex decisions or programming. That needs to be something else on top of LLMs.
And yet we are told by the AI bros that software engineers and programmers will be redundant inside of a few years because soon anybody can be a 'prompt engineer' and create sophisticated software easily using simple written instructions. Are you saying that these peerless geniuses are wrong?
Re: LLMs are good for only some things (Score:2)
No they are not.
CNNs combined with other neural net forms are good at those tasks.
LLMs are LANGUAGE models, and are related to structures that are good at transforming language: translation, summarization, etc.
But media hype and LLM bros decided that any AI or algorithmic technique should be replaced by their bullshit generators.
"I have a hammer to sell, so everything should be solved by hitting stuff."
"Safety reasons", huh? (Score:4, Informative)
"It's impossible to compare Nabla's AI-generated transcript to the original recording because Nabla's tool erases the original audio for "data safety reasons," Nabla's chief technology officer Martin Raison said."
"That's safety for us, not for you."
Being used to summarize case notes (Score:4, Insightful)
Yes? LLMs hallucinate? That cannot be prevented? (Score:2)
Who has been living under a rock here?
Re: (Score:2)
It's not an LLM. It a voice-to-text tool.
Then why are you using it? (Score:2)
Re: (Score:2)
For entertainment sure, for anything involving real life? WHY?
Because C-suites, which hospitals now answer to thanks to the profitization of the entire health system, are seeing the same dollar signs every other C-suite sees when the AI prophets start talking about all the savings to be had by cutting staff and replacing them with AI. Who cares if it's accurate? It might save money! And there is no greater moral imperative in our universe than saving money!
Humans hallucinate... (Score:2)
...computer software produces erroneous results.
Let's stop anthropomorphizing these language models please.
They don't think, they don't reason, they don't "make things up", and they don't hallucinate.
Holup (Score:2)
Re: (Score:2)
You know what kind.
Erases the original recording? (Score:2)
That alone is a major lawsuit. And if the transcription is incorrect, and results in very bad followup - like amputation when it wasn't called for, or lack of treatment when it was, that's billions and billions.
Copy a bad 3 as a perfect 8 or vice versa (Score:2)
We've had this for about 20 years already with the copy machines. Instead of letting a copy degrade from something nearly unreadable to something even worse they'll make it better. This whole LLM thing is different just in scale. Sure, one might say that's the same difference between a squid and Einstein but I don't think see it yet.
ChatGPT does this too (Score:2)
I have not had it happen with the "advanced mode" version, but as recent as 2 weeks ago, the the ChatGPT voice chat tool would sometimes interpret background noise as either far-east Asian language or short phrases like "Thanks." I used the tool in my car and there is a fair bit of road noise. so if I paused during a conversation, it would think I said "Thanks" and keep replying with things like "You are welcome, I am happy to help!" At least it assumed I said good things?
Private equity went big on health care (Score:2)
From a practical standpoint they're going to try to squeeze every penny they can out of the system and that's going to mean worse outcomes for you. And that's going to mean corner cutting like this whether it works or not.
Uh huh (Score:2)
Nabla's tool erases the original audio for "data safety reasons,"
It's in the manual, indexed under CYA.
W1A predicted this years ago (Score:2)
No surprise there (Score:1)
Generative solutions do not belong in science (Score:2)
I have this conversation with my co-workers frequently. Generative AI solutions have no place in science settings. Science is not about making stuff up. We spend a lot of time carefully documenting empirical evidence and our forward decisions are made from the data, there is no room for fabrication of any kind.
Will multiple tries generate the same errors? (Score:2)
If not, can they do multiple transcriptions of the same audio? The accurate ones will all be alike while the error ones will all be different from each other and the accurate one.
Re: (Score:2)
By their words ye shall know them (Score:2)
"Those experts said some of the invented text -- known in the industry as hallucinations -- can include racial commentary, violent rhetoric and even imagined medical treatments".
Presumably in decreasing order of importance?