AI Hard Drive Shortage Makes Archiving the Internet Harder (404media.co) 73
An anonymous reader quotes a report from 404 Media: Skyrocketing hard drive and storage costs caused by the AI data center boom are making it more expensive and more difficult for digital archivists, academics, Wikipedia, and hobby data hoarders to save data and archive the internet. Specific drives favored by some high profile organizations like the Internet Archive have become far more expensive or are difficult to find at all, archivists said. Over the last several months, prices for both consumer level and enterprise solid state drives, hard drives, and other types of storage have skyrocketed. As an example, a 2TB external Samsung SSD I purchased last fall for $159 now costs $575. PC Part Picker, a website that tracks the average price of different types of drives, shows a universal increase in storage prices starting in about October of last year. Prices of many of the drives it tracks have doubled or increased by more than 150 percent, and at some stores SSDs and hard drives are simply sold out. There is now even a secondary market for some SSDs, with people scalping them on eBay and elsewhere.
Brewster Kahle, founder of the Internet Archive and the Wayback Machine, the most important archiving projects in the history of the internet, told 404 Media that the skyrocketing costs of storage is "a very real issue costing us time and money." "We have found that the preferred 28-30TB drives are just not available or at very high price," Kahle said. "We gather over 100 terabytes of new materials each day, and we have over 210 Petabytes of materials already archived on machines that need continuous upgrades and maintenance, so we need to constantly get new hard drives." "We are fortunate to have an active community that donates to the Archive, and we are also looking for help from hard drive manufacturers in these difficult times. We are always looking for more help," he added. "So far we have ways to work around these shortages, but it is a very real issue causing us time and money."
The Wikimedia Foundation, which runs Wikipedia and various other projects, including Wikimedia Commons, an open repository of royalty free media, told 404 Media that the cost of storage has become a concern for the foundation's projects as well. "With over 65 million articles on Wikipedia alone, access to server and storage capacity is vital to us. We've certainly seen price increases since the end of 2025. These price increases are of concern to us, as with every other player in the industry. We see the primary impact in the purchase of memory and hard drives but also in terms of lead times on server deliveries and our capacity to place future orders," a Wikimedia Foundation spokesperson told us. "The Wikimedia Foundation is a non-profit, and as such how we allocate budget is very carefully considered. We maintain our own data centers to serve our users from all over the world. We're putting workarounds in place where we can, mainly involving being smart with how we prioritize investment in hardware, building in flexibility as well as extending the life of existing hardware where possible."
Western Digital, one of the largest manufacturers of hard drives and other storage systems, said that it has essentially sold out of its 2026 inventory to enterprise clients, many of which run data centers. Micron, which made RAM and SSDs under the brand name Crucial, has exited the consumer market altogether because "AI-driven growth in the data center has led to a surge in demand for memory and storage. Micron has made the difficult decision to exit the Crucial consumer business in order to improve supply and support for our larger, strategic customers in faster-growing segments."
Brewster Kahle, founder of the Internet Archive and the Wayback Machine, the most important archiving projects in the history of the internet, told 404 Media that the skyrocketing costs of storage is "a very real issue costing us time and money." "We have found that the preferred 28-30TB drives are just not available or at very high price," Kahle said. "We gather over 100 terabytes of new materials each day, and we have over 210 Petabytes of materials already archived on machines that need continuous upgrades and maintenance, so we need to constantly get new hard drives." "We are fortunate to have an active community that donates to the Archive, and we are also looking for help from hard drive manufacturers in these difficult times. We are always looking for more help," he added. "So far we have ways to work around these shortages, but it is a very real issue causing us time and money."
The Wikimedia Foundation, which runs Wikipedia and various other projects, including Wikimedia Commons, an open repository of royalty free media, told 404 Media that the cost of storage has become a concern for the foundation's projects as well. "With over 65 million articles on Wikipedia alone, access to server and storage capacity is vital to us. We've certainly seen price increases since the end of 2025. These price increases are of concern to us, as with every other player in the industry. We see the primary impact in the purchase of memory and hard drives but also in terms of lead times on server deliveries and our capacity to place future orders," a Wikimedia Foundation spokesperson told us. "The Wikimedia Foundation is a non-profit, and as such how we allocate budget is very carefully considered. We maintain our own data centers to serve our users from all over the world. We're putting workarounds in place where we can, mainly involving being smart with how we prioritize investment in hardware, building in flexibility as well as extending the life of existing hardware where possible."
Western Digital, one of the largest manufacturers of hard drives and other storage systems, said that it has essentially sold out of its 2026 inventory to enterprise clients, many of which run data centers. Micron, which made RAM and SSDs under the brand name Crucial, has exited the consumer market altogether because "AI-driven growth in the data center has led to a surge in demand for memory and storage. Micron has made the difficult decision to exit the Crucial consumer business in order to improve supply and support for our larger, strategic customers in faster-growing segments."
SSDs as investment (Score:5, Interesting)
> a 2TB external Samsung SSD I purchased last fall for $159 now costs $575
I can confirm - I've got a brand new Samsung 870 EVO 2TB drive sitting on my desk that I bought in 10-2022 for $176, and have never got around to installing.
Same drive is now $676 on Amazon!
Re: (Score:2)
Re: (Score:2)
I remember when I could buy 8TB SSDs for $499 (Canadian). It wasn't that long ago.
Now an 8TB hard drive costs almost that much.
Re: (Score:2)
Yep. And that is because old, out of manufacture hardware sometimes gets expensive. A current Samsung 2TB is more like $300.
What a fail on your part.
Re: (Score:1)
Still, I paid 109 CHF for the 870 EVO (2000 GB, 2.5") on 2023-06-24.
Re: (Score:2)
Sure. So?
Re: (Score:2)
I've still got it for half of what you'll spend if you take your McDonald's pay check to Best Buy.
Re: (Score:1)
And MLC instead of QLC or higher on the fresh-off-the-line offerings so once in use they should last longer and a stray alpha particle won't wipe out as many bits.
Re: (Score:2)
We will see the BRAND NEW "Ultra TB" drives soon. Like the UHD ... it'll "Lower" the cost.
On a completely different different note.....What's the difference between Marketing and Fraud?
We must really struggle with storing data (Score:2)
I hope they triage what they archive (Score:1)
Re: Sure it's not the other way round? (Score:1)
It would be useful if they were required to make any training data they didn't pay for publicly accessible...
But they aren't
Re: (Score:1)
It would be useful if they were required to make any training data they didn't pay for publicly accessible...
Isn't such data already publicly available? That's how the AI companies got it?
Re: Sure it's not the other way round? (Score:1)
It is when they get it, but I'm not sure you understand the purpose of an archive
Re: (Score:1)
It is when they get it, but I'm not sure you understand the purpose of an archive
It seems you do not understand the words you wrote: "make any training data they didn't pay for publicly accessible".
Did you perhaps really mean that AI companies should somehow participate in Internet archiving, which again, is something a little more involved that just making files publicly accessible.
Re: (Score:3)
No, I think he understood the words he wrote. AI companies do NOT make the data publicly available, they insert a slop-generator between the data and the user that means the original data cannot be retrieved.
You think you're being clever, but you're both pretending to misunderstand what's been said, and saying something very stupid and false. You are why Slashdot sucks sometimes. Knock it off.
Re: (Score:1)
You may be correct the original author meant an archive, but that is not what he wrote. I simply pointed that out in a non confrontation manner.
Re: (Score:1)
... asking for a redundant copy is not the same as asking for an archive The latter is more complex than the latter. Ask for the latter is insufficient.
Typo. Should be "The latter is more complex than the former. Ask for the former is insufficient."
Blow the dust off the rust (Score:2)
I have about 20 1TB WD black spinning rust hard drives sitting in a box. I stopped using them years ago when SSD prices finally became reasonable. I can't believe I may have to resort to these again if any of my 1TB or larger SSDs fail in the near future. Ugh. The IT world is a fucking mess.
Re: (Score:2)
Heh. I too have a big stack of spinning drives. I really don't think GPU, RAM or SSD prices are going down soon - even if the AI companies fail there will be a backlog of consumers wanting them. Maybe the AI companies will sell them all for cheap leading to a massive glut, but I suspect data centers will be kept intact instead and a big push will come to do all your computing in the cloud rather than buy your own hardware.
Fuck that and fuck the people pushing that.
Re: (Score:2)
I installed a 28TB hard drive in my gaming PC to back up to. Man is that thing loud when you're used to a quiet PC with SSDs.
I checked prices a few weeks ago and discovered I couldn't even afford to replace that PC if it broke. It would cost as much as a small car.
All so the Internet can create funny cat videos.
Re: (Score:2)
Fine if you have old ones laying around. If you don't, unfortunately prices for spinning rust are skyrocketting too.
A WD 8TB Red Plus with 256MB cache was around 200-240EUR last year. Currently 408EUR
Re: (Score:2)
Luckily when I purchase any NAS units I also purchase a spare drive or two right away as well. My 8TB Synology, which uses 2 of the 8TB WD red drives, is getting old in the tooth. I would hate to have to purchase a replacement drive at today's prices.
This whole AI thing is ridiculous (Score:2)
IMO they are pricing in AGI, if they don't get it or if they aren't predicting inference computing costs correctly, there could be a huge rollback. Then we'll have an oversupply of components instead of a shortage. The amount of spend is ludicrous and unrealistic for future needs
Re:This whole AI thing is ridiculous (Score:5, Insightful)
"Then we'll have an oversupply of components instead of a shortage."
No, there isn't a free market. Corporate interests want ALL of supply so you don't get any. That changes only when corporations say it changes, and specifically when VC funding dictates those changes. You assume brainpower not demonstrated to exist, you compete against privilege and you are guaranteed to lose.
"The amount of spend is ludicrous and unrealistic for future needs"
Because billionaires compete to own the future without knowledge or understanding of what the future is. They're trying to buy up all supply on speculation. Musk rents out capacity he bought on speculation, what matters is that he owns the capacity not that he uses it.
One breakthrough on AI computing requirements could make everything known today obsolete, but we can take for granted that billionaires will know that before anyone else. It's a rigged game. Billionaires will own ALL computing power yet demand that the public pay for all the electrical infrastructure necessary to power it. We share the costs and they make the profits. The Sam Walton way.
Re: (Score:2)
No, there isn't a free market. Corporate interests want ALL of supply so you don't get any.
That is still a free market, the problem is you're seeing it through a lens of someone who hasn't got enough money to play in that market. Their dollar is worth as much as yours, the only problem is they have more of said dollars.
Re: (Score:2)
If the big company cancels the order, the next smaller one is already waiting for their supply.
Re: (Score:2)
When the AI bubble bursts, the government will run a "Cash for Clankers" program where they buy all the old AI hardware and crush it so it can't wreck the market for new parts.
Re: (Score:2)
You would think that they would be preemptive and control the risks, seeing as this could create a problem for them in the future. I guess they'll pass out AI stimulus recession checks when crap hits the fan
Re: (Score:2)
IMO they are pricing in AGI, if they don't get it or if they aren't predicting inference computing costs correctly, there could be a huge rollback. Then we'll have an oversupply of components instead of a shortage. The amount of spend is ludicrous and unrealistic for future needs
We are in an economic mania right now. Governments, corporations, startups, you name it, are all afraid of being left behind. They are buying up memory, disks, computing capacity because, well, if they don't, someone else--one of their competitors--will.
Supply will be expanded and built out while demand remains high.
How long will this take? That's the trillion dollar question. It could be months or it could be years, but at some point, demand and supply will come back into closer to equilibrium. Whether tha
Re: (Score:2)
AI is the GOAT of capital repositioning.
Re: (Score:2)
They are not going to get AGI. We are _very_ far removed from that and we reliably know LLMs cannot do it. The LLM industry will collapse catastrophically, there is no other way this can go. The only question is when.
Tape drives (Score:1)
Our industry has a long history of moving data to slower storage to free up faster storage.
So the Internet Archive should simply move less-used stuff to tape drives, freeing up hard drives for popular stuff.
50TB:
https://newsroom.ibm.com/2023-08-29-Fujifilm-and-IBM-Develop-50TB-Native-Tape-Storage-System,-Featuring-Worlds-Highest-Data-Storage-Tape-Capacity-1
Re: (Score:2)
That is not even the problem. Scarcity is artificial and could disappear tomorrow, that's the problem. The industry aligns with the interests of very few extremely rich people, tape drives don't solve that.
Re: (Score:2)
Also those tape drives appear to cost as much as a car. So not really good solution for regular users.
Re: (Score:3)
How is that supposed to work? I click on a link to an archived site and then wait for the robot in the tape library to grab a particular tape and dump the contents?
Re: (Score:2)
I remember years ago reading a post on some Microsoft techie's blog where he answered a question about why Windows did something in a weird way and it was apparently because it would break otherwise if you tried to do that thing on a system where the file you were trying to access was on actually a tape drive and had to be retrieved very slowly.
In practice, I've had bad tape experiences (Score:1)
So the Internet Archive should simply move less-used stuff to tape drives, freeing up hard drives for popular stuff.
That would be counterproductive. Have you ever worked at a company that did that? I have. It seemed that about 50% of requests for some old files that were moved to tape result in "files not retrievable".
I know tapes are supposed to be ultra reliable. Back in the day I used QIC-80s on my PC, a consumer grade tape system. They worked great. 20 years after I made those tapes I was about to through out the last PC I had that I could plug in the old tape drive. The tapes had been store in my garage which get
Re: (Score:2)
Re: (Score:1)
If keeping drives in your garage performed better than a "well regarded company that specialized in that", I would stop using that company.
Not using that company was my irrational reaction after the first "files not retrievable". :-)
Not using that company was my rational reaction after the second "files not retrievable".
Sadly management did not care.
Fortunately we had an official archivist that would store things. I went to her and she found the ten year old CD-R I had turned in with the source code I was looking for.
Re: (Score:2)
The internet archive is a random access database system. It's not a backup, or a true "archive" in a vault somewhere. If tape is the answer then you have completely failed to understand the problem... or failed to understand tape.
Re: (Score:2)
You have "completely failed to understand the problem".
I won't forget (Score:5, Interesting)
Dear AI companies, I won't forget how you fucked our entire industry for a long time. I won't forget how you forced scarcity and high prices. I won't forget that I need to double-check everything that your half-assed solution pukes out. I won't forget that you set everything on fire so you could try to profit.
Re: (Score:1, Troll)
I'm sure that Sam Altman is cowering in fear knowing that he just offended some random guy on Slashdot!
Re: (Score:2)
You reap what you sow.
Re: (Score:3)
Re: (Score:1)
It's funny. Expressions about sin or karma, some cosmic justice or bounce-back like Gaia Theory are commonly "renounced" even though they are rather abstracted ideas about phenomena we can't completely formalize. You anecdotally dismiss this broad idea by attempting to one false instance to disprove it, as if it is like some scientific theory. Maybe climate change decimating the world is what was reaped? Or corruption in politics like the Epstein case? Maybe our obsession to have instantaneous self-gratifyi
Re: (Score:3)
Totally get it, but assuming the bubble bursts and most of the LLM companies just end being sold for pennies to Google and Microsoft, or go bust, what can you possibly do about it?
It's not even as if you can boycott them NOW while they're functioning entities. You can decide you want to, but then Google ensures you can't do a search without AI, your boss refuses to let you code Java or PHP without AI, you're basically fucked.
It's Big Tech we need to rally against. More self hosting. More ad blocking. etc. S
Re: (Score:3)
And? It's one thing to not forget, but what are you actually goanna do about it? What have you done about it right now? AMD, Intel, NVIDIA, SK Hynix, Samsung, Google, Amazon, everyone is all in on it.
You want to impress us? Pledge never to buy another PC. Just remembering shit is meaningless.
Re: (Score:2)
It might be crazy to think that I make technology purchase decisions, not just for myself but for more than one company.
You're right, it might be hard to avoid AMD, Intel, SK Hynix and some other hardware vendors. But I can absolutely choose which software solutions are going to be purchased.
Re: (Score:2)
It might be crazy to think that I make technology purchase decisions, not just for myself but for more than one company.
And what's your decision? Have you decided your company is going back to typewriters? Your "AI companies" cover the entire industry. If your idea is an industry wide boycott you probably won't be making purchasing decisions for any company in the future.
Re: (Score:2)
The same industry pushed things like Internet Explorer and targeted advertising with real-time location. Just because some part of our industry loves some thing does not make that thing great. Sure, it improves, what we have now really is not very smart but sure can appear to be. My job is sorting out the best solution not handing out shiny toys. I've been providing tech solutions for a long time, and if I provided something that only worked 90% of the time then I might actually be fired. Automation is grea
Re: (Score:2)
All of my hardware costs tripled and there is no budget left for AI. Go figure.
Re: (Score:2)
The true fun hasn't even started yet. Just wait a couple of years until the AI bubble finally pops... destroying the whole world economy in the process. Banks with people's life savings, pension funds, national economies... nothing will be spared.
Stockpiling (Score:3)
I was buying drives to add another stripe when the pricing started to ramp up - I try to buy them over time to get different drives from different lots. Now I wish I'd just bought a bunch.
This time last year they were $369, sometimes cheaper. The most recent one I bought was $500. The cheapest I see them right now is $769.
I think I'll be waiting on that new stripe, but at least I have four spares to keep the existing system running.
Our archive is also struggling (Score:5, Interesting)
The AI companies have created two massive problems for us. The first is their web scraping, which is way beyond abusive: it's an attack. Yes, YES, I know about all the techniques to block it and I've deployed a bunch of them, but every minute spent doing that is a minute not spent doing actual archiving work. And even if I managed to blunt most of these attacks, at least one will get through, and they'll steal everything we've posted (for free) and use it (for profit), against our terms of service and against the express wishes of the people who donated materials to us...which is making it vastly harder to convince donors to help us.
The second is the topic of this discussion: disk drives. We don't need the biggest and the fastest, but we need a lot of them because we're maintaining replicas of the archive in geographically distributed locations. And like everyone else, we either can't find them or we can't afford them. I've been using eBay and Craigslist and I've even been going to estate states to try to pick up used external USB/firewire drives and old desktop PCs so that I can pull the disks and hope they test okay. Again: every minute spent doing that is a minute not doing actual archiving work. (Also: because some of these disks have a lot of hours on them, I have to consider probable remaining lifetimes and account for that.)
This is maddening and heartbreaking at the same time. And the thing is: I've spent a lot of time interacting with other people in this space: GLAM (galleries, libraries, archives, museums). Everybody has this problem. All of these people, who definitely aren't doing their jobs because of the lavish pay and spectacular benefits but because they appreciate and love the cultural area(s) they're in, are all struggling. And none of these institutions have the money to truly address the situation: they're all underfunded because they've always been underfunded.
TL;DR: this is cultural vandalism conducted by billionaires who are willing to burn the entire world down for money and power.
Tape is not dead (Score:2)
Buy a LTO library
Re: (Score:2)
Kind of difficult to use as the storage for a website like the Internet Archive.
AI gate keepers (Score:3)
2025 is the year that personal computing is no longer personal.
You are not going to be permitted to handle your own affairs offline. Everything as a service because AI will be in everything and the equipment must be in the cloud. Average consumers and gamers cannot compete with trillions in hype investment. All you can do is wait for the bubble to pop, and hope your retirement or national economy is not popped with it.
AI made "archiving the Internet" silly (Score:2)
It's harder to archive the Internet.
So what? Nothing of value will be lost.
The clamied price is nonsense (Score:4, Interesting)
A current 2TB Samsung SSD is more around $300.
The extreme observed price for a specific old model comes from old hardware sometimes being more expensive because some people need that specific model as a replacement part. Looks like somebody did not do their homework.
slop (Score:2)
nothing but another price gouge... (Score:2, Insightful)
the drives you and i buy are consumer grade bs that is still produced at same rates as in previous years... there is no real "shortage" or supply crunch of consumer grade drives... not as if they re-fabbed their production lines to build enterprise drives... as for memory costs going up, yeah... it's a small portion of the actual cost of the drives... so even if that doubles... the impact on the cost of the drive is a rounding error.
This is another bullshit price gouge of consumers...
Bring bacl Blu-Ray! (Score:1)
Re: Bring bacl Blu-Ray! (Score:2)
Intentional Attack on Local/Personal AI? (Score:2)
With memory/storage increasingly only accessible to 'enterprise' where does that leave personal computing? If AI is the new dominant tech, then it will soon be out of reach to the average person except through enterprise/government if it can be afforded.
In the unlikely event that Micron reenters the consumer market, boycott them.
More DataCenters Coming (Score:1)