Python Removed Wikis for 8 Months to Escape AI Crawlers, Relaunches on GitHub Pages

In February the Python Software Foundation made its wikis read-only "due to excessive automated traffic and security concerns... that traffic turned out to be almost entirely AI bots that were trying to scrape the entire internet."

But their infrastructure was also outdated, and this week they announced new wikis using GitHub Pages "that address the automated traffic and security concerns." The new wikis [for Python, Jython, and the PSF] will have a more modern front-end, as well as a permissions-based content management system for community members to make updates.

With the old wiki structure, we received a large volume of alerts at all hours, each serious enough to page and wake our on-call infrastructure staff, which is very limited (just one person currently, although this will be changing soon!). The quantity of scraping overloaded our servers, causing outages, which happened often enough that these issues required a significant amount of effort for the PSF staff to manage...

Additionally, some portions of the wikis were still built with Python 2. Python 2 reached end of life on January 1, 2020, and no longer receives upstream security fixes. Moving to a static site makes it very fast and reduces security risks, while also having the benefit of a more modern user interface... GitHub was chosen for the home of the new wikis as this is where CPython, python.org, and other properties the PSF hosts currently live. The hope is to continue to streamline the PSF's hosting and stewardship of services to simplify for both PSF staff and the community folks who contribute to them. Previously, the Python and Jython wikis were user-editable by invitation of an existing editor and the PSF wiki was permissions based. Moving forward, all three of the wikis will be permissions based via the GitHub repository they are hosted on, and changes can be requested through pull requests by anyone in the community...

GitHub allows us to distribute granular control, review changes through pull requests, and maintain the uptime this service deserves. Also, since the site is now generated statically, it removes any worry about AI bots overwhelming the hosts that the wikis were previously hosted on.

They write themselves, folks

[drinkypoo]

Additionally, some portions of the wikis were still built with Python 2. Python 2 reached end of life on January 1, 2020, and no longer receives upstream security fixes.

Not even Python could keep up with having to update from Python 2 to Python 3.

Re:

[cruff]

That's irony at its best.

Re: They write themselves, folks

[dbialac]

You gotta keep up with the Nazi holding the gun to your head screaming, "You vil write code dis way!" No other language gives you a set of RFCs telling you exactly how to write code. It doesn't even solve the problems it was intended to address. You can still write 1000 line functions. Easy solution: limit indentations. Within a function, to can indent twice, maybe 2 times in the case of a switch statement. Problem solved, no team of Nazis needed to write RFCs. No supposedly benevolent dictator to write suc

Re:

[phantomfive]

It's the most popular language.

Re:

[_merlin]

Function calls are expensive in Python. This makes it attractive to write huge functions with deeply nested control structures.

Within a function, to can indent twice, maybe 2 times in the case of a switch statement.

Besides the other issues with this sentence, Python doesn't have switch statements. Python 3.10 introduces a "match" statement that's supposed to provide similar functionality, but it isn't the same.

And so the centralization continues

[karmawarrior]

genAI is killing the open Internet. Killing it.

Re:

[taustin]

The ones overloading my web server are, without any doubt, AI bots. Most of them honor the robots.txt file. Amazon's, however, do not, and constitute more traffic than everything else combined (and it get a fair amount of legitimate traffic per day).

Re:

[martin-boundary]

That is probably a good thing. The open internet has been a cesspool of advertising and commercial garbage for a long, long time. The sad part is merely that kids who grew up with that never knew what it was originally.

Re:

[karmawarrior]

No, it's not a good thing. I don't know where you're getting it from that there's somehow less advertising on the corponet, but there not only is just as much, there's an added level of constant personalized manipulation now on a scale we've never seen before.

At least the open web had free, uncensored, content, much of which not funded by advertising.

When it comes to AI organizations

[Waffle Iron]

It looks like if you can't beat 'em, join 'em.

8 Months

[PPH]

That would be sort of like emerging from your fallout shelter after a civilization-ending thermonuclear war. By then, what's the point?

"AI" companies won't have to scrape...

[hazem]

when you just give your content to them. Github is owned by Microsoft. By migrating everything there, you've essentially just given it to them and now they don't have to bother scraping it.

robots.txt

[hadleyburg]

It might be time for an update of the Robots Exclusion Protocol (robots.txt).

It currently lets automated crawlers know what they can access in the site. But it might be appropriate to have a finer tuned protocol which makes this determination based on purpose. e.g. I might be happy with a bot accessing my site for the purpose of search engine indexing, but not for the purpose of AI training... Or vice versa.

It would still be essentially a request rather than a physical restriction, but that's not nothing. A

ORLY

[Suripat]

So... we built AI to read the internet, learn from its collective knowledge, and automate tasks for us. Now we're taking the sources offline because the AI is reading them too aggressively. We wanted machines that could consume the world's knowledge, but apparently forgot to consider what happens when they actually do.