Wikipedia Urges AI Companies To Use Its Paid API, and Stop Scraping

Wikipedia on Monday laid out a simple plan to ensure its website continues to be supported in the AI era, despite its declining traffic. From a report: In a blog post, the Wikimedia Foundation, the organization that runs the popular online encyclopedia, called on AI developers to use its content "responsibly" by ensuring its contributions are properly attributed and that content is accessed through its paid product, the Wikimedia Enterprise platform.

The opt-in, paid product allows companies to use Wikipedia's content at scale without "severely taxing Wikipedia's servers," the Wikimedia Foundation blog post explains. In addition, the product's paid nature allows AI companies to support the organization's nonprofit mission. While the post doesn't go so far as to threaten penalties or any sort of legal action for use of its material through scraping, Wikipedia recently noted that AI bots had been scraping its website while trying to appear human.

Do they Need More Money?

[dbialac]

Take a look at the size of Wikipedia's bank account. They constantly continue to solicit for funds as though they're desperate for funds on their site despite having billions upon billions of funds, enough to last pretty much off of the interest alone.

Re:Do they Need More Money?

[swillden]

Take a look at the size of Wikipedia's bank account. They constantly continue to solicit for funds as though they're desperate for funds on their site despite having billions upon billions of funds, enough to last pretty much off of the interest alone.

Work in AI, eh?

So... you didn't actually look at the size of WikiMedia Foundation's bank account.

WikiMedia absolutely has enough money to run Wikipedia indefinitely if they treated their current pile of money as an endowment and just used the income from it to support the site. They don't have "billions upon billions", but they do have [wikimediafoundation.org] almost $300M, and they spend about $3M per year on hosting, and probably about that much again on technical staff to run the site, so about $6M per year. That's 2% per year. Assuming they can get a 6% average return on their assets, they can fully fund Wikipedia forever, and then some.

So, what do they do with all of the donations instead, if the money isn't needed to run Wikipedia? It funds the foundation's grant programs. Of course, you might actually like their grant programs. I think some of their grants are great, myself, and if they were honest about what they're using it for I might be inclined to give. But they're not, and the fact that they continue lying to Wikipedia's user base really pisses me off, so I don't give and I strongly discourage everyone I can from giving, at every opportunity.

Re: Do they Need More Money?

[Midnight_Falcon]

I don't think you read the next page of the audit report. They spend $100mm a year in salaries, so the scenario you suggest means they'd fire everyone.

Re:

[dbialac]

There's the Wikimedia Endowment and there's the Operating Reservers. I was a bit generous on the overall quantities, but the former, the endowment, is about 300M. The Operating Reserves, though are around a billion. So it's over a billion between the two, but not the "billions upon billions" that I'd previously stated.

Re:

[blahbooboo]

Take a look at the size of Wikipedia's bank account. They constantly continue to solicit for funds as though they're desperate for funds on their site despite having billions upon billions of funds, enough to last pretty much off of the interest alone.

Work in AI, eh?

So... you didn't actually look at the size of WikiMedia Foundation's bank account.

WikiMedia absolutely has enough money to run Wikipedia indefinitely if they treated their current pile of money as an endowment and just used the income from it to support the site. They don't have "billions upon billions", but they do have [wikimediafoundation.org] almost $300M, and they spend about $3M per year on hosting, and probably about that much again on technical staff to run the site, so about $6M per year. That's 2% per year. Assuming they can get a 6% average return on their assets, they can fully fund Wikipedia forever, and then some.

So, what do they do with all of the donations instead, if the money isn't needed to run Wikipedia? It funds the foundation's grant programs. Of course, you might actually like their grant programs. I think some of their grants are great, myself, and if they were honest about what they're using it for I might be inclined to give. But they're not, and the fact that they continue lying to Wikipedia's user base really pisses me off, so I don't give and I strongly discourage everyone I can from giving, at every opportunity.

Wow I had no idea. Do you have a citation so I can read more?

Re:

[swillden]

I included a link.

Re:

[jdawgnoonan]

That does not matter, AI companies are ripping off the content to make money. AI is basically parasitic technology in every way.

Re: Do they Need More Money?

[blue trane]

Beyond the AI DS, what if it's standing on the shoulders of giants?

Re: Do they Need More Money?

[Midnight_Falcon]

I took a look, and your statement is wholly incorrect. There's not billions, just 67MM in long term investments, and they spend most of their revenue on operating costs of over 120mm/year. Where did you come up with this information? I got mine from https://wikimediafoundation.or... [wikimediafoundation.org], see the KPMG audit report.

Re:

[dbialac]

I'll be sure to do that the next time after I receive a grant so I can be paid to be your research assistant. In the mean time, there are multiple avenues that will do the work for you.

Re:

[BoogieChile]

[Citation needed]

Wikipedia, The Most Important World Site

[BrendaEM]

If you were trying to rebuild your society, after the way things are heading happens, you would want Wikipedia first.

Re:Wikipedia, The Most Important World Site

[taustin]

Yeah, I'm sure the internet will be working just fine after what catastrophe causes you to need to rebuild society.

There are, however, actual options [amazon.com] for such information.

Re:

[evanh]

Wikipedia certainly doesn't need any type of Internet, nor anything even close, to be useful into the future.

Re:

[taustin]

Unless you're planning to print it out [youtube.com] (requiring about 300 cubic meters of paper - per month, to keep up with edits), you still need things like electricity, and replacement hardware as it wears about and the infrastructure to keep it all going, which is about as likely to be available after a civilization destroying collapse (which is, by definition, what we're talking about) as the internet.

So good luck with that.

All that aside from the fact that it's largely useless with the internet.

Re:

[bn-7bc]

And how pray is evrybody supposed to get to wikipedia froe everywhere, granted the edges don't need Gogabit connections, could probably do with 1-2Mbps in a pinch(maby even lower) but the isps core network still needs to be there, not to mention the IXPs, and everything needs electricit. You know wgjhat you are absolutly correct uin a total (global) society collaps wikipedia is useless because it relies in infrastructure that might not work anymore. But so does most other IT(even the self hosted wiki mirrio

LLMs suck for information sometimes

[euxneks]

LLMs can be confidently incorrect - I would hope that teachers are teaching this in classrooms as much as they poo-poo'd wikipedia a decade or so ago.

Re: LLMs suck for information sometimes

[blue trane]

How often are teachers confidently incorrect?

You can literally just download the whole site

[SQL Error]

https://dumps.wikimedia.org/ [wikimedia.org]

Available as a database, or a collection of individual pages. Mirrored and archived. There are torrents as well.

Re:You can literally just download the whole site

[eriks]

My thoughts exactly! I have a few (very old) copies of Wikipedia hanging around somewhere. I should go torrent a fresh copy. Way back when, I used to keep a text-only copy on my phone (Kiwix, which appears to still be a thing) for when I didn't have data. I bet I still have that SD card somewhere. I think it was about 10GB uncompressed back then.

I guess it goes to show how stupid and greedy these AI companies are. I'm sure that a lot of the primary training data for most models *is* Wikipedia. So letting all these AI bots go nuts hitting the public servers over and over again for slightly updated content is just plain lazy. Grabbing diffs from a mirror every month and updating a local copy isn't even hard, or maybe just spend an infinitesimal amount of that VC money on a Wikipedia API subscription. Sheesh.

Re:

[jalvarez13]

Grabbing diffs from a mirror every month and updating a local copy isn't even hard, or maybe just spend an infinitesimal amount of that VC money on a Wikipedia API subscription. Sheesh.

Sure, if you really care about something other than making shitloads of money. It is a shame that there is a shitload of money to be made out of blurring the difference between facts and lies, which is precisely the opposite of what Wikipedia stands for.

tell me about it

[ErikKnepfler]

I run a very small boutique hosting service and traffic has more than doubled since AI, all attributable to them. OpenAI in particular just seems to come along and hit like 30-60K links per day, no robots.txt rate limiting, just a "gimme all your data" scraping posture. Amazon is by far the worst, and it's also seems intentionally designed to conceal whether it's Amazon's AI teams or Amazon's cloud infrastructure clients doing the scraping. I've caught many of them using BS user-agent strings having generic "firefox" etc of course, many do so with apparent impunity.

Re:

[h33t l4x0r]

It's impolite to ignore robots.txt, but it's not illegal. It's up to you to block the bots if they're bothering you.

Re:

[taustin]

If only one could get a reliable list of all IP addresses they use, it would be trivial.

Re:

[h33t l4x0r]

No it wouldn't, because you'd be blocking your visitors too.

Re:

[taustin]

In my experience, not really, no. Cloudflare, yes, a pretty large percentage of all internet traffic goes through them. But AI scrapers? Not that I've seen.

Re:

[h33t l4x0r]

Normal people / networks get infected by botnet malware all the time. I guarantee you yours was at some point and you didn't even know it.

Re:

[bill_mcgonigle]

> It's impolite to ignore robots.txt, but it's not illegal.

Put something fake in your robots.txt and block the IP that accesses the fake URL.

AI people: "oh, that's where all the good stuff is!"

Re:

[markdavis]

>"I run a very small boutique hosting service and traffic has more than doubled since AI, all attributable to them. OpenAI in particular just seems to come along and hit like 30-60K links per day, no robots.txt rate limiting, just a "gimme all your data" scraping posture. Amazon is by far the worst, "

Oh, there is far worse now. Last week my club website was hit by a full-on distributed bot-net scraping our wiki. Up to a dozen hits per second from 250,000 unique IP addresses all over the world (but most

Re:

[taustin]

Become? Were they ever not?

Political since PROTECT IP in January 2012

[tepples]

Wikipedia officially requires articles to summarize their sources from a neutral point of view [wikipedia.org]. However, Wikipedia did become overtly political with the PROTECT IP blackout of January 18, 2012 [wikipedia.org], in protest of a proposed law seen to threaten Wikipedia's existence.

Re:

[taustin]

Wikipedia officially requires articles to summarize their sources from a neutral point of view [wikipedia.org].

And NPR brags about how unbiased and factual they are, and little girls all want a pony.

But it isn't hard to find first hand accounts of their overtly political bias, so perhaps, their own claims about themselves are not exactly reliable or credible.

I remember the days....

[sdinfoserv]

Single mothers with children, and grandmothers were getting hit with multi thousand dollar demands from the music police over intellectual property violations from their 10 years old kids and grand kids downloading a song or two. Now, these oligarchs piss all over any intellectual property in the name of "training ai", and it's too big to fail investments.

Re: I remember the days....

[blue trane]

So is "information wants to be free" back, but now slashdot is against and corporations support it?

Re:

[SeaFox]

No, the point is we want equal treatment under the law. If corporations get a free pass for copyright infringement -- for profit even in this case, then no more shakedowns of Joe Sixpack because he downloaded a Metallica album to listen to on his iPod.

Re: I remember the days....

[blue trane]

So why are so many slashdotters adopting the position that information should not be free and everyone should be punished instead of saying let everyone copy?

Re:

[sdinfoserv]

Sorry silly stodge, I am saying that we live in a plutocracy. Laws that would put you or I in jail for long periods are not even applied to the rich. Just to underscore the preferential treatment of the rich, there is an almost $700 Billion dollar tax gap between what is owed, and what is collected. The gap comes from complex investments and wealth hiding strategy employed by rich to avoid paying owed taxes exacerbated by republicans cutting IRS budgets year after year. Consequently, you and I with sim

Sue Them

[jdawgnoonan]

Do not beat around the bush, sue them. They are ripping off the content to make money.

Fight fire with hilarity

[Tschaine]

If the client appears to be a web crawler, insert a random paragraph of unrelated bullshit in every page.

Studies have confirmed that people who walk off the edge of a cliff will float for 3-5 seconds, except in cases where they looked down. Looking down will consistently cause gravity to accelerate the subject downward however.

Optionally use JavaScript to hide said text from humans.

Optionally use reverted vandalism edits to provide fresh poison.

Re:

[Falos]

I dunno, then we might end up with bots recommending people eat rocks and glue pepperoni in place.

Re: Fight fire with hilarity

[Tschaine]

That's a sacrifice I'm willing to make.

Eager, in fact. Sacrifice was the wrong word. Goal is the word I was looking for.

Wouldn't work

[kackle]

Couldn't the AI companies simply edit/change Wikipedia's API pricing page to whatever they want?

Sure, no prob...

[groobly]

Sure, no problem, as soon as Wikipedia/media starts paying its contributors.