Websites Are Closing Their Doors to AI Agents (theverge.com) 78
The Verge reports:
OpenAI, Meta, Google, Instinct, and others promise their agents can scour the internet on your behalf, doing anything from booking a dinner reservation and cleaning out your inbox to buying new shoes. But what happens when the rest of the internet won't play nice?
In recent weeks, users of services like Meta's Muse, xAI's Grok, OpenAI's Dots, and Instinct have reported blocks preventing their AI agents from performing tasks across top websites. Yelp told TechCrunch that it only allows AI agents that pay for its data licensing program, while Delta Air Lines is preventing them from booking tickets on behalf of users. JPMorgan Chase and Charles Schwab have also confirmed that they're blocking AI agents from accessing accounts due to security concerns. Some users say sites that previously worked with their AI agent are no longer accessible, including Cars.com and Amtrak, suggesting other companies could be quietly blocking them....
Muse, Meta's deceptively cute AI assistant, most recently faced the wrath of Amazon, which said its "continued access" violates its site policies. There are reports of xAI's Grok, Instinct's AI agent, and OpenAI's Dots being unable to use Amazon as well. xAI owner Elon Musk responded by saying that maybe he "should start a competitor to Amazon." The more places bots struggle to access, the greater the questions about how useful they are....
Instead of trying to fight Amazon, [Meta] has partnered with retailers like Walmart, Gap, Sephora, Best Buy, and others, allowing Muse to shop on their sites. At the same time, Shopify has also opened its network of storefronts to AI agents.
In recent weeks, users of services like Meta's Muse, xAI's Grok, OpenAI's Dots, and Instinct have reported blocks preventing their AI agents from performing tasks across top websites. Yelp told TechCrunch that it only allows AI agents that pay for its data licensing program, while Delta Air Lines is preventing them from booking tickets on behalf of users. JPMorgan Chase and Charles Schwab have also confirmed that they're blocking AI agents from accessing accounts due to security concerns. Some users say sites that previously worked with their AI agent are no longer accessible, including Cars.com and Amtrak, suggesting other companies could be quietly blocking them....
Muse, Meta's deceptively cute AI assistant, most recently faced the wrath of Amazon, which said its "continued access" violates its site policies. There are reports of xAI's Grok, Instinct's AI agent, and OpenAI's Dots being unable to use Amazon as well. xAI owner Elon Musk responded by saying that maybe he "should start a competitor to Amazon." The more places bots struggle to access, the greater the questions about how useful they are....
Instead of trying to fight Amazon, [Meta] has partnered with retailers like Walmart, Gap, Sephora, Best Buy, and others, allowing Muse to shop on their sites. At the same time, Shopify has also opened its network of storefronts to AI agents.
What took so long? (Score:4, Insightful)
Seems like something that should have been done or restricted 3 years ago
Traffic is traffic (Score:5, Interesting)
Simply put online advertising doesn't work and the only way to get the engagement numbers needed to make it even look like it might work is with a constant stream of rage bait and outrage porn. Basically any platform by design has to enshitify.
I think YouTube just barely avoids the worst of it. If you pay attention to YouTube you'll find that you can actually curate your feed. So for example a while back I got tired of all the right wing douche nozzles and dipshit lefties trying to monetize my political rage so I started eliminating them. YouTube got the message. Reddit simply does not give you the ability to do that and while I have more or less stopped using it except for a few highly curated forums it's still blasting my feed with worthless political shit.
Don't get me wrong I'm political as fuck and I am all in on useful politics. I have a couple of channels on YouTube I still subscribe to mostly just Belle of the ranch. Channels that provide actual useful information that I can act on.
But after the 2024 election I figured out that 90% or even 95% of the content out there for politics is either right-wing propaganda or left-wing anti-centrist rage bait.
So I started asking myself why everything is so fucking useless and figured out pretty quickly that the internet runs on outrage and cat videos and these days it's mostly outrage. What the cool kids called Doom scrolling.
So because of that you need the bots. If nothing else they are good at keeping the rage going and like I said they look like real users to your advertisers and investors.
As for why they're getting rid of them now it's probably a combination of the overwhelming hit to bandwidth that they just can't keep up with combined with the fact that they have probably figured out how to filter the AI bots but allow the rage bots to continue to operate
Re: Traffic is traffic (Score:1)
You da man!
Re: (Score:3, Interesting)
you're missing a point about the AI bots. There are thousands working in parallel scanning your stuff compared to, well, just one person (or even just one visit from google's spiders). The big sites are one thing, but the small sites can't take their internet traffic costs going from $800 to $3500, with all of that traffic being AI bots scraping your pages over and over, and of course none of those pages seeing the advertising that's supposed to pay for that traffic.
The AI bots are not playing 'nice'. They'
Re: (Score:3, Interesting)
> you can actually curate your feed.
Same video recommended despite telling that telling that I am not interested or channel marked as do not recommend. Steady stream of recommended pile of shit.
On July 29, 2026 those morons broke the search to the point where searching for channel name would give alternative view videos first. Imagine getting some Tim puppet videos instead of ranch ones, when you search for channel name.
No, youtube does not let you curate your feed. Instead of taking actions against repo
Re: Traffic is traffic (Score:1, Interesting)
You say they broke their website, but that's just from your puny individual human perspective. When you consider it from their giant corporate perspective, any time they can put another advertisement in front of your eyeballs is a win for them. If they just up and presented the media you wanted, you'd watch it and then go away, robbing them of views they deserve somehow.
Re: (Score:3)
Long before that, all bots should have been blocked.
Re: (Score:3)
Be careful what you ask for, apparently Google's AI agent is indistinguishable from their normal web-search crawler and who wants to be invisible to Google? I don't know if the same applies to Microsoft and Bing.
Re: (Score:1)
Why?
I run a Shopify store, and if anybot wants to buy from me, by all means, and please invite all your bot-friends!
I think the only merchants who should be fearing this are ones relying on brand recognition rather than low prices and good products, since bots have more time to "shop around".
Re: (Score:2)
Re: (Score:3)
Humans still review and authorize the specific decisions. And the bot will have spent far more time trying to meet the user's need than the user would ever have had time to do so.
Re: (Score:2)
Browser time (Score:5, Insightful)
Next the agents drive the actual browser, and thus become indistinguishable from a meatsack.
Re: (Score:2)
Next the agents drive the actual browser, and thus become indistinguishable from a meatsack.
Correct.
Or, if the agents don't or won't do that themselves, they could certainly be instructed to issue calls through proxies or otherwise that do the hiding on their behalf without telling them why those pathways need to be used...
When people are prevented from doing things, they'll find ways to do it. Always have, always will.
Re: (Score:2)
this is already happening but the problem is that only a very few with 24gb ram on their video cards can do it when running local llm's like qwen. visual shopping AI requires a lot of context
Re: Browser time (Score:2)
You can use an agent with a cloud LLM, though. Eg. Codex cli with playwright mcp.
Re: Browser time (Score:2)
Re: (Score:1)
Re: (Score:3)
It's called playwright and it's one of the most popular agent skills. I use it all the time for tedious forms, big whoop.
Re: (Score:2)
That's what I ended up doing. Although some sites can still figure out an AI is driving playwright. I had to add an intermediary "pacing" proxy to buffer the actions so that it looks like a human is driving. It's a 100% vibe coded. The system has been a godsend for things like apartment hunting, and for buying a couple of used mountain bikes.
Re: (Score:2)
Claude has a Chrome extension that already does this.
Re: (Score:2)
Re:Long-term, how can they stop them? (Score:5, Insightful)
If you are willing to give an agent your credentials,
Give? Remember when you said "OK" to Copilot getting access to your desktop, browser tabs, etc? Or maybe didn't click "Hell no!" fast enough?
We got your credentials right here, buddy.
Re: (Score:3)
Yes, but...
1. It is far easier for bot owners to run their bots from their own machines than attempt to proxy their requests through other IP addresses. And thousands of orders coming from a small set of non-residential IPs looks indistiguishable from an attempt at hacking. Especially as they're unlikely to have similar profiles, eg web page loading times, what gets loaded, etc.
2. The bots frequently make mistakes. And those are on top of the mistakes users make, not instead of. Flooding the system with bot
Re: (Score:3)
There are "computer use" models that do exactly this. There are also classic automation frameworks for this. The other team uses Javascript that determines if your mouse moves with uniform speed and predictable trajectories without jitter. The bot framework makers therefore have frameworks for "natural" mouse movement. The only winning move is not to play.
If there's one thing we've learned (Score:2, Flamebait)
It's that the AI guys respect people's right to say "no" about as much as your average rapist.
The easiest workaround that seems the most difficult to defeat is to run the AI agent's actions from your preferred browser on your phone or computer, and still extract whatever data they want from the response to be relayed back to them for ingestion into their data centers.
They'll be every bit as respectful of websites saying 'no' to them as they were to writers, performers, and artists who told them not to use t
Re:If there's one thing we've learned (Score:5, Insightful)
To be fair, websites don't respect people's rights to say "no" to user tracking and serving up ads, trying their darndest to defeat ad blockers and such.
Re: (Score:2)
Sometimes, there are not good guys at all.
Re: (Score:2)
Could you please NOT compare tech to rape?
Re: If there's one thing we've learned (Score:2)
Re: (Score:2)
You're both comparing apples with pears and are showing a bad taste how you choose your comparisons. Just don't!
Re: If there's one thing we've learned (Score:2)
Found the tech apologist.
Re: (Score:2)
Found the person relativating rape.
Re: If there's one thing we've learned (Score:2)
Whoosh.
Re: (Score:2)
'Relativating' is not a word, 'relativizing' is. You can't just use words you've made up, it's like raping the English language (teehee).
Re: (Score:2)
Some day you'll realize that ALL words are made up. ;)
playing nice (Score:5, Insightful)
> when the rest of the internet won't play nice?
Umm, let's see, Agents pound sites and do unexpected things. Then people block Agents from sites. People making Agents claim the Internet won't play nice. Sure, makes perfect sense that it is the entire Internet that is wrong and Agents are perfect if lawless entities.
We live in the wackiest timeline.
Re: (Score:2)
More like
Agents pound sites and do unexpected things. Then companies block Agents from sites.
People make agents that are indistinguishable from humans (already happening) or agents find vulnerabilities in the website blocking them and attempt to "hack" it to accomplish their given task (already happening). Company gets mad and sues because they're inept morons (already happening).
I expect the companies that are trying to shut out AI for security concerns are probably even more ripe for an AI security problem to happen now, because the people who use agents that use those sites aren't going
Re: (Score:2)
I think you confuse agents with crawlers. A crawler comes and fetches everything (if its polite slow, but currently they are often aggressive). An agent is more or less an automated browser, doing targeted actions like a user would do, just with fewer clicks for the user. An agent doesn't overly strain the website.
Re: (Score:2)
Maybe my Agent scrapes sites and replicates them. I guess you could have an Agent do all kinds of things.
Re: (Score:2)
Technically correct ... but a crawler does this at 100 times the speed because it doesn't involve AI inference.
F Around, Find Out (Score:2)
So, the technology that relies on unthrottled hammering of people's websites is pissing off website owners? The technology built on absorbing copyrighted content without compensation for the creators now has people who produce the content putting up paywalls? All these stories planted in the media about how the AI agents are dangerous 1337 h4x0rz, and now people don't want them accessing their systems? So sad.
Boo effing hoo (Score:5, Interesting)
This kind of automation ultimately causes problems for 'regular' every day consumers and gets hijacked by a smaller group of assholes; plus the technology isn't mature enough to be trusted. So what happens when an agent runs amok, buys a too many of x, and the user says 'wait! I didn't want 100, I want to you to eat the cost of me returning it, and company y says, too fucking bad you bought it' -- who's liable? (my answer is the dumbass with the agent, but who knows how it will pan out)
Only a few years ago some of the kind of stuffs agents do (or at least have access to) we'd call a virus or malware, how things change.
As expected (Score:2)
Immature tech needs time to be integrated
Eventually, when agents are trusted and more mature, they will have their own API and set of security rules
LOL. Sure. Go ahead, Musk. (Score:5, Insightful)
xAI owner Elon Musk responded by saying that maybe he "should start a competitor to Amazon."
Classic Musk: Underestimating the talent and effort needed to break into a business. It took Amazon nine years to become profitable. At the most his attempt will just be "MAGA Temu" and he'll hide the folly by merging it into a business that is working, like he's already done with other properties he owns.
Re: (Score:2)
You aren't wrong... but he has enough money to fully fund multiple amazon wanna-be companies for a full 20-year-losing streak and still have a shitload of money.
Frankly, the only thing he doesn't have is enough money for is to buy a majority share of Amazon's stock.
Re: (Score:2)
Oh come on, he's proven his track record with Twitter.
Hasn't he?
Seriously tho, would love it if he did, waste even more of his money.
Re: (Score:2)
Some Amazon's competitor are older businesses than Amazon, and sometimes they made e-commerce before Internet, using papaer catalogues and cash on delivery.
Classic mall chains now have an online presence now too, and at the end another competitor can't wipe Amazon, like Amazon didn't wipe others, maybe they could shrink a bit the market share.
For www facing websites (Score:3)
Re: (Score:2)
Re: (Score:2)
Better, these days, to use a puzzle humans /can't> solve, that AIs can. Far more certain filter.
Re: (Score:2)
provide last 4 digits of pi?
Re: (Score:2)
Apparently, the agentic AIs have a lot of trouble with regular CAPTCHAs. My guess is the bots that are breaking CAPTCHAs are specialised at that job and aren't in the open for anyone to use.
Re: (Score:2)
Modern agents can just drive your browser via an extension and hand control over to you to solve the CAPTCHA. Once they have a token they can drive the browser through hundreds or thousands of interactions.
Re: (Score:2)
Maybe use a puzzling challenge that only a real human can solve before allowing full access to your website,
"To continue to this site, please enter the value of a zero of the Reimann Zeta Function for which the real part is not equal to 1/2 and the imaginary part is not equal to zero: [ ] + i[ ]"
Re: (Score:2)
Anything difficult enough for the AI will stump way to many morons.
“Think of how stupid the average person is, and realize half of them are stupider than that.”
George Carlin
Re: (Score:3)
Re: (Score:2)
How would they get index by the likes of Google then?
Especially if Google is using the same IP space for their agents?
Hard to be surprised. (Score:3)
It's overwhelmingly the case that vendors hate returns and chargebacks; and generally try to avoid situations where those are more likely to arise. An error-prone mechanism that acts "on behalf of" in a way where it does have the user's credentials and payment information but may well not have their authorization in a legally binding way when it decides to order 100 of something seems like the last thing they would want to deal with.
Maybe someone will manage to ram through glorious-agentic-future and it will become de-facto mandatory to cater to people who have abandoned ordering from websites in favor of prompting bots to order for them; but until that happens it's hard to see merchants reputable enough that they are interested in keeping their ratings and relationships with payment processors in good order wanting to touch the situation. Now, scammers who already hop between expendable presences as soon as dissatisfaction catches up with them; those are probably quite interested in the future of prompt injection based advertising.
Just for the record (Score:2)
Google canâ(TM)t do anything with clearing an overloaded inboxâ"one of the tasks for which it would be useful. By iverloaded i mean >5k emails. My friend is an email hoarder.
Here comes the trickle down (Score:3)
Do AI agents ignore robots.txt? (Score:2)
Do AI agents ignore robots.txt?
Asking for a friend.
Mom! (Score:2)
Mom!
Cars.com is blocking my Agent from scraping again!
I guess once you've stolen everything on the internet once, you feel entitled to do it again another layer deep.
cute? (Score:2)
Muse, Meta's deceptively cute AI assistant
Muse looks cute but that's just a cover for the evilness Zuck instilled in it.
The sites are protecting their users. (Score:2)
Charles Schwab doesn't want their idiot users to be able to grant AI permission to accidentally wipe them out financially. That makes sense, but I think I may have a solution:
When an agent does something that looks dodgy, just pop up a little box that says "Are you sure?". And if they click yes, pop up another little box that says "Are you really sure?". I figure by the time they're really really really really sure, there've probably thought it over enough.
Re: (Score:3)
Charles Schwab doesn't want their idiot users to be able to grant AI permission to accidentally wipe them out financially.
They're probably more worried about the agent finding a vulnerability that will give it deeper access to their system and then potentially other people's accounts. They would almost assuredly not care if you wiped out only your own account due to your own stupidity.
The fact that their worried enough to attempt to shut AI out entirely, which won't even work, probably means that their systems are, in fact, vulnerable and they're hoping to buy some time.
Price transparency (Score:3)
Is the main thing those shipping sites want to prevent. They want to retain the ability to charge different people different prices.
Playing nice? (Score:2)
You mean playing along. The ones who are not playing nice are the bots.
Please wait while we confirm you are not a bot.. (Score:2)
Why didn't Sir-Whoever-He-Is think about this when he invented the internet?
Behind that bot are legitimate users! (Score:1)
I just planned a holiday with MCP servers. It was legitimately massively better than calculating everything manually, and I'm normally OK planning holidays, but this assistance just helped me do everything to a much higher quality.
If too many bots are hitting your website, then shut it down and open an MCP server. I had to use an obscure Japanese service to get the coverage I needed.
This is a massive opportunity.
In other areas, all of the hobby websites I've used for decades are closing public access and go
So what? (Score:2)
When companies respect an "unsubscribe" request I'll consider the idea they won't just obfuscate the bot and barrel through the denial.
Nobody cares. They won't even beg forgiveness. The law is so far behind reality that it doesn't matter. The penalties are fiction.