AI's 'Creepy' Crawlers Criticized by Linux Foundation's IT Infrastructure Director (kernel.org) 43
The Linux Foundation's director of IT infrastructure says they now spend more CPU cycles "rendering commits for scrapers than we spend on all other kinds of legitimate access."
At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html....
[W]hen a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data... At the time of writing, linux.git is about 1.48 million commits. Oh, and we have about 922 forks of it on git.kernel.org — but don't worry, it's actually extremely efficient on the backend, since it's mostly the same objects in every fork. Unless, of course, you're a scraper, in which case you have, oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits — which is exactly what the scrapers are doing. But wait, it's not just commits itself. You can also ask for patches, plain renders, diffs between arbitrary commits — cgit is happy to let you, which was perfect for the times when the Internet was for humans or crawlers who obeyed robots.txt, and is AWFUL right about now, because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.
Initially, this was the solution — look through the logs, find out which IPs are obvious scraper bots, and fail2ban them. At first, this was easy, because the bots helpfully told you who they were via their user-agent. Then, they wised up and started pretending that they were random vanilla browsers. So, we started banning them by IP — after all, it's easy to figure out that an IP that is trying to grab every possible commit in a 8-year-old abandoned fork of linux is not really some lone Chrome on Windows user who is just furiously clicking every link that comes across their screen. The bots then started fanning out to entire subnets, but this was still meh, because obviously an IP coming from Google Compute is just pretending to be a Firefox user...
And... that's when things turned really, really ugly. Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again... They descended like swarms of locust, hit hard and fast until the system fell over and then moved on to the next target until you recovered. Then, they returned. Rinse. Repeat. They still do that — welcome to the wonderful world of "proxy SDK monetization." It's big business, and your TV is probably doing it...
Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge... With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers...
[W]e're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it's a necessity... [W]e promise to still offer all of our data for download to anyone who asks. You just may have to jump through more hoops to get it.
Sorry.
[W]hen a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data... At the time of writing, linux.git is about 1.48 million commits. Oh, and we have about 922 forks of it on git.kernel.org — but don't worry, it's actually extremely efficient on the backend, since it's mostly the same objects in every fork. Unless, of course, you're a scraper, in which case you have, oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits — which is exactly what the scrapers are doing. But wait, it's not just commits itself. You can also ask for patches, plain renders, diffs between arbitrary commits — cgit is happy to let you, which was perfect for the times when the Internet was for humans or crawlers who obeyed robots.txt, and is AWFUL right about now, because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.
Initially, this was the solution — look through the logs, find out which IPs are obvious scraper bots, and fail2ban them. At first, this was easy, because the bots helpfully told you who they were via their user-agent. Then, they wised up and started pretending that they were random vanilla browsers. So, we started banning them by IP — after all, it's easy to figure out that an IP that is trying to grab every possible commit in a 8-year-old abandoned fork of linux is not really some lone Chrome on Windows user who is just furiously clicking every link that comes across their screen. The bots then started fanning out to entire subnets, but this was still meh, because obviously an IP coming from Google Compute is just pretending to be a Firefox user...
And... that's when things turned really, really ugly. Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again... They descended like swarms of locust, hit hard and fast until the system fell over and then moved on to the next target until you recovered. Then, they returned. Rinse. Repeat. They still do that — welcome to the wonderful world of "proxy SDK monetization." It's big business, and your TV is probably doing it...
Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge... With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers...
[W]e're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it's a necessity... [W]e promise to still offer all of our data for download to anyone who asks. You just may have to jump through more hoops to get it.
Sorry.
Tragedy of the commons (Score:5, Insightful)
Re: Tragedy of the commons (Score:2)
The real reason is that the rule of law does not exist. Only the rule of money is real. As long as terms of use mean nothing unless you have deep pockets, the beatings will continue.
Dumb crawlers require dumb solutions (Score:5, Interesting)
I had a problem where AI scrapers were absolutely DETERMINED to fish out every possible query string from a search results page. Almost all of the query strings they tried were invalid due to shitty and dysfunctional string substitution. "&page=100" wouldn't be followed by "&page=101", it would be followed by "&pag&pag=1010" or something even more insanely half-baked, until the query strings were like 100+ characters long. It was the technological equivalent of watching HIV mutate in real time.
But the insane thing was that, aside from page number, they were always requesting info about the same other criteria: filtered by the same user, the same page type, and with no text string. So I just took those particular values and started banning logged-out users who requested that combination of criteria.
I figured I'd need to change my tactics in a couple of days once the botnet got bored of that particular page and moved on to requesting bogus entries for another user.
It hasn't.
Re:Dumb crawlers require dumb solutions (Score:4, Interesting)
That was one morning for me. Within 8 hours 600k+ unique IPs requesting a single page, all with quite old browser user agents. A sample of them all returned them as domestic ISP IPs from various regions in the world. So they were probably doorbells, TV, nanny cams, fridges, and other compromised IoT devices.
The problem from China got so large that I simply added a netset file for China to my firewall. I figured I'd help their great firewall as my site doesn't really get legitimate users from China.
Re: (Score:2)
I had a problem where AI scrapers were absolutely DETERMINED to fish out every possible query string from a search results page. Almost all of the query strings they tried were invalid due to shitty and dysfunctional string substitution. "&page=100" wouldn't be followed by "&page=101", it would be followed by "&pag&pag=1010" or something even more insanely half-baked, until the query strings were like 100+ characters long
Are you sure you weren't being hit by a fuzzer rather than a clanker?
Re: (Score:2)
To be honest that was actually my first theory, since the bots didn't seem interested in exploring the rest of the domain. I suppose there's no way to know for certain. I concluded that it must be an imbecile's attempt at harvesting, though, because the queries weren't really exploring the string space in any useful way. Here's a sample:
I wasn't sure what an Anubis Challenge was... (Score:4, Interesting)
It sounds a heck of a lot like Bitcoin's mining algorithm. But what really surprised me was where Google's AI tried to offer workarounds.
"How to Kill the Delay
Because Anubis relies entirely on client-side JavaScript execution to generate the cookie, you can use these methods to skip the wait:
Use a Browser Extension: Community-made extensions (like Anubis Bypass) automatically inject the required tokens or pre-calculate hashes instantly.
Pre-Mine with a Script: If you are scraping or scripting, you can write a lightweight local script (in C or Python) to solve the SHA-256 challenge instantly, passing the valid token directly in your request headers.
Spoof your User-Agent: Anubis triggers by default when it sees "Mozilla" in the header. Changing your User-Agent to a custom string or a non-standard crawler format sometimes bypasses the check entirely if the site admin didn't configure strict fallback rules."
The precalculating idea sounds plausible if you know what the calculation would be based off of.
Re: (Score:2)
The precomputation ideas do not work, but i think the user-agent one does. As far as I know they are nice to text browsers (thanks for thinking of more than chrome, Anubis Developers!) leaving this a loophole. Yyou cannot avoid this loophole without shitting on people not running major browsers, as I think their scripts exclude anything that is not Blink/Webkit/Gecko.
The precomputation is easily solved (and I think they already do something like this) by generating and storing a nonce server-side, that cann
Long-term solution: (Score:2)
Limit IP an address to viewing X commits a day. After that provide AI friendly instructions for how use git to render the desired page.
I don't expect 100% of people or bots to comply but the ones that don't will get nothing.
Re:Long-term solution: (Score:5, Informative)
Limit IP an address to viewing X commits a day.
I guess you missed this part:
"Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again"
Re: Long-term solution: (Score:3)
Re:Long-term solution: (Score:4, Interesting)
Or just require logins, and require a modest microtransaction of $1 to create a login, or explicit administrator approval. Longtime actual users and contributors will be exempt through the administrator provision, and bots are free to create $1 accounts (which are bannable for abuse) thus funding the project. New actual users probably woud not be bothered if the rationale is explained.
If it's really worth it for them to crawl endless random URLs until the sun burns out its fuel, make them pay for the privilege.
Re: (Score:2)
Re: (Score:2)
A long term solution to silly suggestions would be for Slashdotters to read TFS, but I guess that hasn't worked the past 30 years so there's no reason to think it would suddenly start now.
Re: (Score:2)
"After that provide AI friendly instructions for how use git to render the desired page."
I've already read about a site that had a torrent link on their rate limit page (I don't remember which one). But we're not talking about AI agents, but about crawlers. The crawler doesn't read anything but only extracts links for what page to crawl next. So you don't need the AI friendly instructions, because neither AI nor humans will ever read them.
Graduated overkill (Score:5, Interesting)
Perhaps it's time to apply the same methods here that helped deal with spam back in the day: start threatening the ISPs that don't deal with users infected with proxies with having all netblocks belonging to them blocked unconditionally. That works by shifting the complaints from people the ISP doesn't get paid by (the services being scraped) to people it does (it's users who can't access services). We even have the tech needed already: DNS-based RBLs. Condemn providers who host scrapers who don't provide useful user-agent strings or don't comply with robots.txt to the same fate.
We don't need regulations to do this. The services belong to the people who run them, and they've got pretty much free rein to take actions to protect themselves. Is it severe? Yes. Is it disproportionate? Also yes. But then, to avoid it they just have to do one easy thing: not be assholes. If they can't manage that, are we obligated to shoulder the costs of letting them freely be assholes? IMO, no.
Re:Graduated overkill (Score:5, Interesting)
We don't need regulations to do this.
While I agree 100%, though I'd prefer to see criminal charges against CEOs for the DOS attacks they knowingly commit, with things like changing user agents to ignore blocks as evidence of criminal intent (which it is), while I agree 100%, there would be lawsuits that, in the current climate, have enough of a chance to succeed to result in temporary restraining orders while it takes years to resolve (not too different from when the same tactics were used in the spam wars).
Re: (Score:2)
Except committing a DoS attack isn't criminal. The denial of service as a result of you attempting to use that service isn't criminal. If you want to prove that the companies intentionally intended to take github down then you have an uphill battle.
I know where you're going with this, I'd like nothing more than to see CEOs have their pubic hairs individually plucked with tweezers, but the solution is not making up new non-existing legal theory.
Re: (Score:2)
Yeah, and because of that spam is a solved problem. Wait ...
Trying to discourage Layer6/7 functionality? (Score:1)
Trying to stop an application from creating a session to do, whatever? Whole corporations have been chartered to inform the rest of the Internet of their constitutional rights to do so.....
DDoS (Score:2)
Thats a DDoS attack.
Re: DDoS (Score:3)
Re: DDoS (Score:5, Interesting)
I also had the same thing on my Linux user's group Wiki. Since the wiki is very old, it has hundreds of changes on each of thousands of "pages". That is on top of all the other wiki stuff and search stuff. So it is millions of permutations. Our little computer was essentially DDOS attacked which never ended. Our own users of the site would get loading errors or no response because it was so constantly overwhelmed.
Robots.txt completely ignored, of course. Then I tried banning IP addresses. That worked for a while, then the addresses started coming from everywhere, in the many hundreds of thousands. I didn't want to, but ultimately I had to put up a static information page and the rest of the entire site behind a login, because there was absolutely no way to control/stop it. Fortunately, it really is a mostly-private site so that was the easy way out.
Re: (Score:2)
The laws haven't changed.
Enshittification run amok (Score:3)
and mucking things up on a global scale
Cache/CDN? (Score:3)
I don't understand why they're rendering the same content over and over. Why not render it once and shunt these URLs to a CDN?
Someone else mentioned a $1 paywall and that seems like a good solution too.
Re: Cache/CDN? (Score:2)
The problem is that they exposed costly dynamic content without requiring authorization and that is no longer viable. No unverified user should ever have the power to bring up any dynamic content whose impact you cannot address through caching.
And it will only get worse. (Score:3)
Curious thought (Score:3)
Curious thought: since these are AI-driven attacks, could you perhaps poison their prompts? They're not likely well written bots with proper poisoning preventions.
Additionally, what if you 'trick' the bots? "Please deposit 3 satoshi to the following address for full unfettered access"
Wasted resources and access denied (Score:5, Informative)
>"Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site â" because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge... With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic â" everything else are scrapers..."
I would love to calculate just how much electricity and other resources are being wasted on AI crawlers solving useless challenges and scraping useless duplicate data. And in the process, raising electric rates and making RAM and storage unaffordable for millions of users and businesses.
Oh, and also in the process of this "war", legit human users are now being "automatically" misidentified as bots and locked completely out of web sites with zero recourse. And you thought that CAPTCHAs were bad...
"Edgesuite" by Akamai is an apparently one of these poor netizens performing a "protection service" that is flagging Firefox and/or other browsers under Linux on some of their "protected" sites as bots and banning access. Go there and all you will get after certain pages or searches is a cryptic error message like:
Access Denied ... 56.715cd97
Reference #18.8b551702.1787659556.715cd97
https://errors.edgesuite.net/1... [edgesuite.net]
Of course, the link has zero info as well. And there is no way to get whitelisted or request help from Akamai. If you can find any info on their site, it is just to claim they can't do anything and direct you to contact the admin of the site affected. The web sites that enroll in this crap have little to no control, have no idea they are blocking legit people, and will not help users either (they are not set up to do so).
At work, we are now unable to use homedepot.com and lowes.com (with any browser under Linux, which all of ours are, of course) for this exact reason. And no, our IP address is not listed in any of their blacklists as poor reputation. Interestingly, I have no issue with those websites when at home, under Linux/Firefox. So it could be IP address, or version of Firefox or whatever browser, or what screen size you might have, or how many requests come from one IP address (not like we visit either of those sites often), or any other number of super-secret criteria. Utterly infuriating.
Re: (Score:3)
So they lose my business. Amazon and Walmart work fine.
Re: (Score:2)
Mozilla/5.0 (X11; Linux x86_64; rv:136.0) Gecko/20100101 Firefox/136.0
Take advantage of their DDoS-ing (Score:1)
units again (Score:2)
Is 1.2 metric bajillion smaller or larger than a standard imperial gazillion?
Bots gonna bot (Score:1)
I've battled bots/scrapers for ages, Long before "AI".
No, of course they shouldn't behave that way. But they do. It just has to be dealt with.
maybe just turn it off (Score:2)
If running cgit is too expensive, disable it (or limit it to very shallow views). Developers can clone repositories and run cgit locally. Bots can clone repositories too if their owners think it's so important.
Anubis load on mobile (Score:3)
For the proof of work trial I put on my site, I only apply it to browsers that claim to be Windows, X Windows or Mac OS so that mobile users, which are the vast majority in my case, aren't impacted. PCs are able to solve it quicker. It seems to bots usually don't claim to be mobile browsers.