Student Handcuffed After School's AI System Mistakes a Bag of Chips for a Gun (theguardian.com) 144
An AI system "apparently mistook a high school student's bag of Doritos for a firearm," reports the Guardian, "and called local police to tell them the pupil was armed."
Taki Allen was sitting with friends on Monday night outside Kenwood high school in Baltimore and eating a snack when police officers with guns approached him. "At first, I didn't know where they were going until they started walking toward me with guns, talking about, 'Get on the ground,' and I was like, 'What?'" Allen told the WBAL-TV 11 News television station.
Allen said they made him get on his knees, handcuffed and searched him — finding nothing. They then showed him a copy of the picture that had triggered the alert. "I was just holding a Doritos bag — it was two hands and one finger out, and they said it looked like a gun," Allen said.
Thanks to Slashdot reader Bruce66423 for sharing the article.
Allen said they made him get on his knees, handcuffed and searched him — finding nothing. They then showed him a copy of the picture that had triggered the alert. "I was just holding a Doritos bag — it was two hands and one finger out, and they said it looked like a gun," Allen said.
Thanks to Slashdot reader Bruce66423 for sharing the article.
That's not AI failure! (Score:5, Insightful)
That's really stupid human oversight.
Re:That's not AI failure! (Score:5, Insightful)
That's how humans use AI.
Re:That's not AI failure! (Score:5, Insightful)
That's how some humans use everything. I used to be shocked by stories where some fuckwit blindly followed Google Maps into rivers or airport runways (long before LLMs) but now I know if a dialog window asks "Should I kill you as painfully as possible?" it'll get a lot of Yes clicks.
If people aren't stupid, then can we at least admit they hate themselves?
Re: (Score:2)
It's how the cops use every new bit of tech. When DNA came in, they were arresting people on very flimsy DNA evidence that later turned out to either be flawed or easily and obviously explained away.
Happened when IP addresses became their new toy. Still happens with fingerprints, which, despite what CSI may tell you, rarely present an exact match.
Re: (Score:2)
No this isn't AI. If police are looking at the picture and telling him he has a gun instead of a Doritos packet then AI isn't "in use here". It really should be though. It's time we go back to employing "Actually Intelligent" people and not just let every dumb mouth breathing cunt join the police force.
Re: (Score:2)
Okay maybe it was AI. TFS is poorly written and TFA is TFA and therefore unread. Great work David. No I refuse to call you "EditorDavid" you don't deserve that title.
Re:That's not AI failure! (Score:5, Insightful)
Actually, it's both.
Re:That's not AI failure! (Score:5, Insightful)
In other words, the AI failed, and the humans also failed.
Re:That's not AI failure! (Score:5, Informative)
No. If you want to avoid false positives, you have to accept false negatives, and conversely. Set the recognition to be "super cautions" and it's going to make mistakes that say "Maybe a gun there". This is literally inevitable.
What's really stupid is that the police looked at the picture of a Doritos bag and a couple of fingers and didn't realize it was a false positive. (Or more likely didn't even bother to look at the evidence before flying off the handle.)
Re:That's not AI failure! (Score:5, Insightful)
What's really stupid is that the police looked at the picture of a Doritos bag and a couple of fingers and didn't realize it was a false positive. (Or more likely didn't even bother to look at the evidence before flying off the handle.)
From TFS, it doesn't sound like the police saw a picture until after the fact. The way it reads, the AI flagged the frame in the video and called the police. The police were responding to a call for an armed person. If you're responding to a call for a possible shooter, it's unlikely you will be grilling the caller, on site, before trying to stop a potential mass shooter.
It seems to me that the company who makes the AI should require a person to review the video once the call goes out. That way they can give the police an all clear before they arrive.
Re: (Score:3)
So it was a Doxxing then. Do the company execs go to jail now for the burden they've put on society?
Re: (Score:3)
So it was a Doxxing then. Do the company execs go to jail now for the burden they've put on society?
No , it's just that people shouldn't eat gaddammed Doritos! This should be obvious, people!
Re: (Score:2)
Re: (Score:3)
Reporters: You mean they were following the orders of the AI?!?
If that’s what lets me off the hook, yes! -CEO
Re: (Score:2)
The way it reads, the AI flagged the frame in the video and called the police. The police were responding to a call for an armed person. If you're responding to a call for a possible shooter, it's unlikely you will be grilling the caller, on site, before trying to stop a potential mass shooter.
Who did it pretend to be? Did it identify itself as a bot? 911 operators usually do have questions of their callers, so they can feed info to the police. I'd be very curious to hear that conversation.
Re: (Score:2)
Nobody called anybody. It's a weapons detection system. They're deliberately running it. The article makes it sound like ChatGPT got bored, started watching webcams, and decided to narc on the Doritos Guy. That's not even close to what happened.
Re: (Score:2)
They're deliberately running it.
Yes I know that. So you are saying nobody actually called the police? How did they know to come, or do they have some sort of setup like a silent alarm that just triggers a light somewhere on a web dashboard? If it is not via 911 you would presumably need some sort of special arrangement with the police. The article only says it "sends an alert" without being more specific.
Re: (Score:2)
I can't see how that would actually work, because there's no way they're going to know which person in a group is the "offender" and these clowns apparently did. While the AI could probably identify the location based on the camera name, I have trouble believing it could even tell them which group of kids much less "the kid with the green shirt, blue jeans and blue backpack." This is almost certainly a moron of a security guard responding to an automatic alarm after only the most casual glance at the video
Re: (Score:3)
Then when did the cops get the picture to show the suspect? On the way to the scene? While approaching with guns drawn? While searching him after handcuffing him? Did the AI forward a description of the suspect instead of the picture?
Re: (Score:3)
From TFS, there's no indication either way of whether they had seen the picture before, and if I had to argue either way from the wording, I'd go with "yes, they had".
Also, when did we switch from calling weapons detections systems "weapons detections systems" to "artificial intelligence systems"? It's still true, but a much less useful choice of wording, and is probably going to make some readers think they were shoving video feeds through ChatGPT or something.
Also, in the picture [metro.co.uk], it was clearly their ce
Re: (Score:2)
Around here the police usually won't come out to surveillance alerts, unless a human has manually reviewed them first.
Re: (Score:2)
What's really stupid is that the police looked at the picture of a Doritos bag and a couple of fingers and didn't realize it was a false positive. (Or more likely didn't even bother to look at the evidence before flying off the handle.)
From TFS, it doesn't sound like the police saw a picture until after the fact. The way it reads, the AI flagged the frame in the video and called the police. The police were responding to a call for an armed person. If you're responding to a call for a possible shooter, it's unlikely you will be grilling the caller, on site, before trying to stop a potential mass shooter.
It seems to me that the company who makes the AI should require a person to review the video once the call goes out. That way they can give the police an all clear before they arrive.
TFA says the cops had the picture, they showed it to the student while they were detaining him at gunpoint. Here's how the student described the photo when the cops showed it to him:
“I was just holding a Doritos bag – it was two hands and one finger out, and they said it looked like a gun,” Allen said.
Either they did a shitty job of looking at the photo or they didn't look at all.
Re: (Score:2)
Agreed! Chips are junk food, that poor kid isn't being looked after properly!
That's not stupid human oversight (Score:5, Insightful)
Cops are dumb (Score:2)
AI is dumber. Though if they were the Flamin' Hot Doritos the mistake may be understandable.
They have to be (Score:3)
We don't pay well enough to counter that so there you go. Cops are dumb by design
Re: (Score:2)
Re: (Score:3)
Sure, but that's beyond dumb. Any moron knows the difference between a hand with a finger extended holding a bag or Doritos and a gun.
As punishment, they should be required to report to the front of the cafeteria and be paddled by the student in front of the entire student body.
The local news can make a day of it.
Re: They have to be (Score:3)
You clearly aren't American...
Re: (Score:2)
There was a single specific police department right next to the FBI HQ that decided they didn't want anybody over 135(?) IQ because the FBI would sniper them the moment they got a year's experience.
Re: (Score:2)
The starting pay kinda sucks and the job is dangerous which is why they usually don't get the best & brightest.
It's is a dangerous job, often thankless, and subject to the whims of local and national politicians. They really need to be paid well from the start. Some places (Seattle for one [publicola.com]) are starting to recognize that... probably because it's getting harder to recruit.
Re: (Score:3)
Bad pay, no college education required, but granted the ability to exert power over others... exactly what sort of person do you think you're going to attract with that sort of job description?
Re: They have to be (Score:5, Informative)
The state cops in my state make up about 33 percent of the top 1000 people on the payroll. They make more than presidents of the state colleges. But not the state university. They make more than the vast majority of professors either at college or university level.
It is not uncommon to have a decent sized group of state cops making 300k or more.
âoeWe work wicked hard and do a lot of overtime.â
Sure you do.
Re: (Score:2)
I think the idea is to pay new cops more, not the old already highest paid cops even more.
Something like double the hiring salary, then cut pay raises from seniority in half.
Re: (Score:3)
Happened 25 years ago. https://abcnews.go.com/amp/US/... [go.com]
Re:They have to be (Score:5, Informative)
https://abcnews.go.com/US/cour... [go.com]
Jordan, a 49-year-old college graduate, took the exam in 1996 and scored 33 points, the equivalent of an IQ of 125. But New London police interviewed only candidates who scored 20 to 27, on the theory that those who scored too high could get bored with police work and leave soon after undergoing costly training.
For that matter it was covered here on SlashDot
https://yro.slashdot.org/story... [slashdot.org]
Re: (Score:2)
Sorry about the double post.
It's not even that dangerous, only number 22 and if you remove traffic accidents (mostly caused because they drive like assholes) it's even lower. I've done four of the jobs higher in this list, and none of them payed what a starting cop gets.
https://www.ishn.com/articles/... [ishn.com]
Re: (Score:2)
The New London, NY police fought in court and won, even after appeal, that they can deny someone employment in the police force based on too high of an IQ:
https://abcnews.go.com/US/cour... [go.com]
Re: They have to be (Score:5, Interesting)
You have some weird conceptions about big cities. Homicide rates aren't an urban vs. rural thing, they're a north vs. south thing [wikimedia.org] . It's the south that has the high per-capita murder rate. Which is in turn because said areas are the poorest places in the US [caliper.com]. The trend holds true even in areas that are relatively culturally homogenous - for example, there's not much of a difference in culture between northwest Texas rural counties and northeast Texas rural counties, but northwest Texas is much wealthier per-capita, and also has a much lower homicide rate.
The TL/DR: crime correlates with despair, and places like the Mississippi Delta are characterized by chronic high unemployment, low wages, and limited access to quality education and resources. This combines with a legacy of racial violence/mistrust and lax firearm laws, and the result is exactly what one would expect.
One could make the argument that, well, okay, it may be the rural south that has a high murder rate per capita, okay, but there's lots of people in big cities, so it's a multiplier. Yes, that's true, but there's also lots of cops in big cities, so it doesn't change their odds of being the one responding to a situation where shots are fired, to the degree that police departments are equally well staffed per-capita.
It's also worth mentioning that the rural crime rate trends in the US are much worse than the urban crime rate trends. I hate to risk derailing this by the meremention of Trump, but he tapped into a very legitimate wellspring of anger; the economic growth in the US over the past several decades has been very uneven, and a lot of people, esp. in rural areas, the rust belt, and the south have felt left behind, with insufficient care from politicians as to their plight. While the ragebait media landscape has tended to try to focus their anger on cities and minorities, as "evil outsiders catered to by elites", US cities are, frankly, doing quite well on average, and have thrived in the US's growing service economy. But people in the rural south, the Mississippi Delta, the rust belt, etc (outside of the "energy belts", like in west Texas, that produce oil, gas, wind power, etc)... their lived experiences of a lack of opportunity and declining communities are very much real. They're just projecting them (wrongly) onto big cities outside of their region.
Human on the loop required (Score:5, Insightful)
Re:Human on the loop required (Score:5, Insightful)
If you have humans double-checking your AI, then you don't save money. Never gonna happen.
Re: (Score:2)
Do you then suppose that double-checking an potential false positive is somehow more expensive than sending a squad to a school?
Re: (Score:3)
Sending the squad to a school is already paid for. It doesn't hit the budget. Assigning review duties to AI alerts? Gotta pay somebody for that shit.
Re:Human on the loop required (Score:5, Insightful)
Gotta pay somebody for that shit.
Police are paid for by taxes, and public school is paid for by taxes. Everyone's already paid. It is pure criminal negligence that allows machine vision to automatically call police. The student's parents should sue the school, and the school administrators should be prosecuted for filing a false police report.
Comment removed (Score:4, Insightful)
Re: (Score:2)
Or just have the leader of the squad that's about to go threaten a kid over a bag of Doritos give the photo a look first. It pays for itself by leaving the squad free to pursue something more relevant and worthwhile. The 30 seconds it would take to look at the picture isn't even enough time to get to their cars.
Re: (Score:3)
Re: (Score:2)
They had the picture with them when they went to arrest the kid. There was no active threat. Even a glance at the photo would have revealed the truth. Then there's the bonus of not having yet another law abiding citizen one day instructing his children "Don't EVER trust the cops".
Apparently the best adjective to describe current police operations is "Keystone".
Re: (Score:3)
Re: (Score:2)
The police department is going to hire and train on-demand, 24-hour AI inspectors, but they'll have other responsibilities? I doubt that.
The lawsuits may eventually cost more than hiring a checker, but that's a problem for somebody several years from now. They already have lawyers on retainer, but not AI checkers.
Re:Human on the loop required (Score:5, Insightful)
If you have humans double-checking your AI, then you don't save money. Never gonna happen.
Of course you do, and literally every other AI enforcement system uses this approach already. AI reviews everything, a human only confirms the positives to ensure they aren't false.
For example Australia is now using AI enforcement systems to detect people using mobile phones behind the steering wheel. The AI will flag a picture, send it to a human to review, and the human issues the fine. Employing humans to look at every picture of every vehicle would be many MANY orders of magnitude more expensive.
Re: (Score:2)
How did you calculate this? Let's think of a purely human and a human-checks-AI system:
1) Four persons watching 40 security cameras, each keeping their attention at ten cameras (sounds like a good estimate for human attention).
2) One person checking an AI generated alert of a camera every few hours to every few days, depending on the number of (false) positives the AI system generates.
Re: (Score:2)
That run on AI and have kill-decisions controlled by AI.
Re: (Score:2)
well it may take an case that makes to the court room where the source code is put under question.
Re: (Score:2)
AI source code isn't the issue. The models are the issue, and they're just a bunch of math.
Re: (Score:2)
Re: Human on the loop required (Score:2)
Cops might need a human double doing the checking but most people could do it themselvesâ¦
Re: (Score:2)
You must - MUST - have a human double checking this. Dispatching any kind of response without human review invites catastrophe.
According to TFS, the cops on the scene had a copy of the photo, so this should already be the case.
They then showed him a copy of the picture that had triggered the alert. "I was just holding a Doritos bag — it was two hands and one finger out, and they said it looked like a gun," Allen said.
Hopefully, they had already surmised this was a false alarm, but guessing they had to cover their asses and check it out and follow through as though it was a real situation, since it was flagged by the system. Not defending this, but can see why it happened. Had the video simply been originally viewed by a person, who would have realized it was a finger, it would have ended there. On the downside, peopl
Re: (Score:3)
Enter the vibe checkers.
Re: (Score:2)
That's fine. Except that if it turns out its really a gun, when they're going over all the missteps involved, the time spent waiting for somebody to go over the result manually will absolutely come up.
Re: Human on the loop required (Score:2)
The term you're looking for is "human in the loop" aka HITL.
Just like traffic cams (Score:5, Funny)
Re: (Score:2)
Reviewing footage costs a lot of money. There's no room in the budget for it.
Re: (Score:2)
They looked at the picture afterward...just not before bothering to go on site.
If they had looked at it afterward BEFORE bothering to go on site, they could have saved everyone a lot of effort.
Re: (Score:3)
The AI already told them what they saw. When you remove the cognitive load of recognizing what something is and then you prime people with a set answer, they'll believe it much if not most of the time. If I told you that the lyrics to "Kiss the Sky" were "Kiss This Guy" and you'd never heard of the song before, you'd hear "Kiss This Guy" when you listened. I told my kids that "Smack My Bitch Up" was "Smack My Pitch Up" and they believed it for years.
Re: (Score:3)
I dunno about all that....but humoursly enough...in the Billy Joel song, "Pressure", for DECADES, I thought the lyric was:
But you will come to a place
Where the only thing you feel
Her lonely cunt in your face
And you'll have to deal with
Pressure
And I always thought, "wow!! That's really some strong imagery for a Billy Joel song".
Only in the last few years did I figure out the lyric is actually:
But you will come to a place
Where the only thing you feel
Are loaded guns in your face
And you'll have to deal with
Pre
Re:Just like traffic cams (Score:5, Informative)
Anecdote: I received a red-light camera ticket in the mail a few years back. Pre-AI, they were using some sort of automated license plate reading software, which confused an 'O' with a 'Q'. They dutifully attached a copy of the picture of a late model GMC pickup truck with my vehicle plate and description. A 45 year old Toyota Landcruiser. Not even close.
My mistake was in calling the municipal court and having the error fixed clerically. Which saved me time but ended up not having me challenge the ticket in court. Which would have made the error a public record and a data point useful for others to challenge the systems accuracy.
Re: (Score:2)
The system's accuracy wouldn't be in question and precisely nothing would change. You literally demonstrated just now you had a method to correct the error, that would be accepted form of accuracy.
Clerical errors happen all the time. I remember a story where someone was clocked doing 407km/h (impossible) on the highway. He was driving a Peugeot 407. Nothing was automated here, a human was doing the work. That doesn't mean the system was inacurate, and the ticket also got thrown out following the normal proc
Re: (Score:2)
Clerical errors happen all the time.
There was no clerical review. The method offered for a ticket challenge is to request a court date. Which costs drivers time and money, a kind of penalty by itself. Some people just pay the ticket, since (as they advertise) the infraction doesn't go on your record. I just happen to know cops and how broken the court systems are. In a proper and just system, nobody should be able to get a charge dropped through unofficial channels.
Had there been a cursory human review, this would never have passed muster. B
Re: (Score:2)
You don't need to review all footage, just the flagged positive. Given how this is currently news, and how the school isn't having a shooter every day, it doesn't sound like it would cost even a cent extra to have someone at the school review the identified picture and click a button *before* calling in the guns.
Does your company have a dedicated budget for you typing in a 2FA code?? Of course not, that would be absurd.
Re: Just like traffic cams (Score:2)
Funnier than the other joke but I was looking for the obligatory joke about the dangers of potato chips to some folks...
Re: (Score:2)
Its "almost" like you need an officer to review the footage before issuing the ticket so we arnt humiliating people and wasting anybody's time with nonsense.
This is a system that called the police. The police are out of the loop beyond being told there was a shooter flagged. You need a human reviewing potential positives *before* calling the police.
Where is the picture? (Score:5, Interesting)
Re: (Score:2)
If it could possibly be interpreted as a gun and the AI recognition primes, the viewers, they will usually see it as a gun.
Re:Where is the picture? (Score:5, Informative)
The police were acting on the assumption that "what they heard" was correct, which they always do, and always did, before AI.
"What they heard" was some kind of school administrator placing an emergency call about a student with a gun.
That administrator hasn't said anything publicly yet that I've seen, but the AI company is claiming their own human staff had reviewed the images, flagged it as a false positive, and communicated that false positive status to the school admins.
The question is who knew what and when.
Re: (Score:3)
Bungie had an oversized Master Chief manikin in its lobby when they were located in Kirkland, WA. They were in two buildings half a block from each other, one of them sharing a parking lot with the Kirkland Senior Center. One day Master Chief's five foot long gun was needed in the other building for something, so after the photos were done one of the Asian employees was walking down the hill to take it back to the lobby.
The police got a panicked call from an old lady screaming that there was "an Arab with
Re: (Score:2)
and police, who should KNOW the difference
Police saw the picture after the fact.
new rule no outside food but you can buy from vend (Score:2, Offtopic)
new rule no outside food but you can buy from vending with X2 mark up
Re: (Score:2)
With chip bags carefully designed to not look like guns.
Re: new rule no outside food but you can buy from (Score:2)
Fingers are kind of a wildcard though
Re: (Score:2)
Outlaw fingers from schools.
Re: (Score:3)
Yeah, after the redesign the bags will look like knives.
Re: (Score:3)
You joke, but a few years ago a second grader was **expelled** for making a DRAWING of a knife.
Re: (Score:3)
Doritos is a subsidiary of Pepsi-Cola. The school probably had an exclusive marketing contract with Coca-Cola.
Good to know (Score:2)
Kids can now sneak guns into school inside Doritos bags.
Police: We're not falling for that one again!
Re: Good to know (Score:2)
The boy who cried Doritos
Not an accident (Score:2)
Summary is bad, it was even stupider than that (Score:5, Insightful)
From https://www.wbaltv.com/article... [wbaltv.com]
So the school resource officer (SRO) called the cops after the school had already determined it was a false positive and cancelled the alert. So the cops showing up, and this kid having to get handcuffed and searched, is all thanks to the SRO - who really deserves to be fired and quite possibly sued.
Re: (Score:3)
An additional level of failure is the over-reaction (IMO) of the police. A school security officer misidentifying a gun by mistake is always possible. But as there was no indication a threatening attitude in the initial report nor visibly when the police approached, the officers should not send people to the ground in handcuffs as their first reaction.
As I understand, the officers were following their protocols for such situations; which would mean there is a systemic issue in that police protocols are inad
Re: (Score:3)
The half-wit security guard did something, the trigger-happy police did something and terrorism didn't happen: The system works. The friendly-fire and the gun-shot, never-armed suspect, when it happens, is the unfortunate price of 'saving' other people. The policy is, assume (not prepare) for the worst and damn the (avoidable) corpses.
Like so much of the USA, cruelty is built-in. Until voters demand equ
Re: (Score:2)
An additional level of failure is the over-reaction (IMO) of the police.
This did not seem like an overreaction to me. Maybe if it were in Europe, but not in the USA where stories of random armed people are actually quite common. If you're called to a scene for someone who has a gun, it stands to reason that you would go in guns drawn and command the person to get on the ground, and cuff them.
You don't know anything, maybe that kid sitting there had a gun in his pocket ready to draw.
An overreaction is stories of shoot first and asking questions later, like the police who shot an
Re: (Score:2)
Unfortunately students with weapons (kitchen knives) are not uncommon, and murders have occurred, to the point since March 2025 the French police is conducting random searches at schools. But this does not change the expected behaviour from educators and police. In short, whatever happens with school students inside or immediately next to schools is the primary responsibility of the school principal, and police does not touch students until there is a certain threat. Here some illustrative examples from Fra
Re: (Score:2)
I wanted to conclude that while these sad stories triggered a debate on how dangerous it is to be a school employee nowadays, or how low paid it is for the level of commitment it requires, this did absolutely not lead to any discussion that police should touch the kids at first suspicion.
that was bad. (Score:2)
this is a future we do not need to see. that student deserves a pay day for that fuckup.
Re: (Score:2)
Precisely. If you make mistakes like this expensive enough for the police station then the problem solves itself. The real problem is that someone promised the police and the school a magic new technology that would make their schoolyard safer. So far the system probably has zero wins, and one spectacular failure. If the political and economic fallout for the failure is high enough then the school turns off the crappy system, and it encourages other schools to do the same. Potential new buyers for the
Sue the shit out of them (Score:4, Informative)
Every time something like this happens, the victim should sue the hell out of the operator. Fighting back is the only way to make this stop. Without a legal and financial detriment, those cowboy software unicorns will do what they want without a drop of common sense.
Re: (Score:3)
those cowboy software unicorns
This has nothing to do with software. It was the school resource officer who called the police even after it was reported as a false positive.
This is why vigilante justice sucks, you're so ready to put the wrong person's head on a stake. The software unicorn developed software with flags, with human review, and the operator of the software correctly flagged the report as false.
The school... (Score:3)
The school and SRO will play the "Wag the Dog" movie scene where a Tostitos sack of chips is changed into a cat.
https://youtu.be/Bp79QTKqKyg?t... [youtu.be]
Now they change the bag of Doritos into a gun, and blame the AI for hallucinating, and blame the AI. D'oh! Then the school will issue the press release "We take our students protection and civil rights very seriously..."
JoshK.
Please put down your weapon (Score:2)
Please put down your weapon. You have twenty seconds to comply [youtu.be]
Re: Wearing camo from head to toe... (Score:2)