PBS Station Fears Losing 50TB of Data After Being Ghosted By Cloud Provider (arstechnica.com) 86
An anonymous reader quotes a report from Ars Technica: After its cloud storage provider went defunct, a PBS affiliate decided to sue a data center provider to regain access to 50TB of TV shows, videos, and other data dating back 70 years. As reported this week by Current, a trade newspaper covering public broadcasting, St. Louis affiliate Nine PBS filed a lawsuit against Iron Mountain Data Centers on July 28, seeking access to the data. In the litigation filed in Denver District Court, Nine PBS says that its cloud storage provider, Open Source Storage (OSS), used one of Iron Mountain's Denver data centers to store the channel's data. However, OSS is being unresponsive, and Nine PBS says Iron Mountain has refused to release its data.
The data in question includes the station's coverage of the COVID-19 pandemic, East St. Louis' history, The Great Flood of 1993, and over 11,000 files, The Denver Post reported in July. The lawsuit claims that "most" of the data is "unique and irreplaceable," according to the Post. Last month, a judge blocked Iron Mountain from deleting or modifying the data.
In a hearing on Wednesday, a judge ruled that Iron Mountain must hand over any physical devices holding the data, Current reported today. The judge also said that Nine PBS must find a third party, such as a former OSS worker, who can help retrieve the data within 30 days and without sharing or corrupting data belonging to other OSS clients. Nine PBS is already communicating with a former OSS employee "who is willing to help," the report said. If complications arise, such as from the data being encrypted, another hearing will be scheduled. Nine PBS and Iron Mountain must provide updates by September 14. Iron Mountain's spokesperson said the company only provides physical infrastructure, such as the building, network connectivity, power, and environmental controls. "Our customers rent space for their servers and other hardware. These are the client's assets. We don't have access to the data on the hardware/servers because they belong to our customers," the company said.
If it granted "unauthorized access to third-party hardware without a court order," Iron Mountain said the company would violate basic data privacy protocols, breach its contract with OSS, and "potentially [expose] confidential data belonging to other clients of OSS."
The data in question includes the station's coverage of the COVID-19 pandemic, East St. Louis' history, The Great Flood of 1993, and over 11,000 files, The Denver Post reported in July. The lawsuit claims that "most" of the data is "unique and irreplaceable," according to the Post. Last month, a judge blocked Iron Mountain from deleting or modifying the data.
In a hearing on Wednesday, a judge ruled that Iron Mountain must hand over any physical devices holding the data, Current reported today. The judge also said that Nine PBS must find a third party, such as a former OSS worker, who can help retrieve the data within 30 days and without sharing or corrupting data belonging to other OSS clients. Nine PBS is already communicating with a former OSS employee "who is willing to help," the report said. If complications arise, such as from the data being encrypted, another hearing will be scheduled. Nine PBS and Iron Mountain must provide updates by September 14. Iron Mountain's spokesperson said the company only provides physical infrastructure, such as the building, network connectivity, power, and environmental controls. "Our customers rent space for their servers and other hardware. These are the client's assets. We don't have access to the data on the hardware/servers because they belong to our customers," the company said.
If it granted "unauthorized access to third-party hardware without a court order," Iron Mountain said the company would violate basic data privacy protocols, breach its contract with OSS, and "potentially [expose] confidential data belonging to other clients of OSS."
Shame on them! (Score:5, Insightful)
Re: (Score:3)
That's kind of been the bread and butter for cloud providers - "data velocity". They say: keep all your stuff on our cloud, it'll be safe forever, you don't need to worry about it.
A single copy is zero copies and 2 is 1. This is overlooked on all clouds, on the basis that 3 copies on a cloud provider is still 1 copy.
This kind of oversight failure has ended many companies when they've had DR scenarios. I know of 2 that had to close shop because the egress cost of all their data from AWS was larger than insur
Re:Shame on them! (Score:4, Interesting)
Re:Shame on them! (Score:5, Funny)
[verse]
There is no clever coding or cost
That can guarantee no data loss
Nothing can protect it from the likes
Of military action, or asteroid strikes
[verse]
But there is no need to invoke vicious thugs
The biggest losses are from software bugs
And fat-fingered admins who erase
There is no cure when you run out of space
[verse] ... is hell
Single mode of failure's in between
It's not the best, but not the worst I've seen
Any 1 of n up does quite well
Every 1 of n up
[verse]
Triplication loss requires losing three
Erasure coding is much higher degree
Triplication loss is p to the third
Erasure coding's p to the absurd.
[verse]
Erasure Coding stops random leaks
It's more like draining a bathtub, so to speak
If fixes are faster than failures, data's preserved
Otherwise loss is certain and well deserved
[verse]
Erasure coding's clever design
is that two points define a line
On the line a third point can be placed
Then any one of three can be erased
[verse]
A parabola takes not two points, but three,
Or d points, for curves of higher degree.
n=k+d, when k more points are placed,
Then any k of n can be erased.
[verse]
Erasure Coding stops hardware leaks
But not admins or management freaks
So: backup in several unrelated places
Immortality any 1 of n embraces
[verse]
Frequently no matter what you do
You find your worst opponent is you
The safest place your data could have gotten
Is the one that you've forgotten
[verse]
Use Erasure Coding to watch each cloud carefully
Use several independent clouds for redundancy.
Immortal data's impossible, tough,
But perhaps the end of time is good enough
Re: (Score:2)
You trusted a cloud service provider to back up all your data forever???
Hell no.
Re: (Score:3)
No, they trusted a contractual partner to fulfill the requirements of their contract. This is no different than paying someone to build you a house and them not doing it. The complaint is not that they weren't backing up data forever, it's that they are unresponsive and not handing over the data.
Re:Shame on them! (Score:5, Insightful)
I agree but at the end of the day there is no amount of money they contracted partner can pay back that will bring back data that is lost/stolen. Just like having insurance that covers your house burning down; you'll get some money but you won't get your family pictures, heirlooms, etc back.
Something like the 3-2-1 backup strategy should apply here. X copies on Y number of providers and at least 1 is immutable.
Re: (Score:2)
How many houses do you have?
The house is a server and memorabilia is the data. Your "backup plan" for the house is hotels / temporary housing while replacing the failed house. Unfortunately, the "data" in this metaphor is copy-protected by the server admin ($deity). Best you can do is protect the house and use approved migration tools when a failure is either predicted in advance or occurs unexpectedly but is not catastrophic.
Re: (Score:2)
There is no such thing as "immutable" storage, despite being the current buzzword in the storage industry. Even LTO WORM tapes - as close as you can get these days - are just regular tapes with a bit set in the RFID chip that asks the drive to "please don't erase me".
What you really want is at least one copy of the data "offline".
Re: (Score:2)
That is precisely why contractor management is important as well as redress through a legal system. Note the article doesn't say the data is gone, just that their service provider is being unresponsive.
The lowest bidder for the project (Score:2)
No, they trusted a contractual partner to fulfill the requirements of their contract.
Was that contractual partner the lowest bidder for the project? That would sort of contradict the notion that their data was priceless and irreplaceable.
Re: (Score:2)
This should be a big warning to anyone who uses them: this will happen to you.
Re:Shame on them! (Score:4, Insightful)
realy this is what is wrong with modern IT (Score:2)
50tb of storage is at most 5k i have over 300tb in my network at home. Why re we storing that offsite. Hell that is 3 LTO tapes. I have LTO7 in my house.
They don't deserve it back.
Re: (Score:3)
Why re we storing that offsite.
Fires or other natural disasters destroying the building and the data with it?
Re: (Score:3)
I think the question they were trying to ask is, "why are they only storing that data offsite?" Which is a valid question. Offsite copies are a critical part of the strategy for the reasons you describe, but they still stored their shell-encased embryos in a single handwoven wicker vessel.
Re: (Score:2)
Actually i started back in the 80s. Everything I have ever down loaded i still have. Everything I have ever created i still have.
300TB is really not all that much, I still have back ups from a pdp 11 that i ran at UCF in the 80s. It was one of the first email servers.
Every few years or so I build a new bigger one and copy everything to it. This version is a supermicro 36bay chassis with 16 8tb drives and a 4 30tb drives. this was built pre AI BS so not all not cheap but doable
it has a single HP lto7 tape d
Tapes fail too (Score:2)
Hell that is 3 LTO tapes. I have LTO7 in my house.
My company moved some old projects to an off site tape backup service. They wanted to recover some space on our servers.
When I tried to access some of that data years later I got some “data unrecoverable” responses to my requests.
Fortunately one of the original devs for the project in question was still at the company. He had an old CD-R in the back of one of his desk drawers. It worked.
Everybody learns the hard way (Score:5, Insightful)
I learned about backups in 1986, when the floppy disc with my word processor documents got corrupted.
It's just weird that some grownups still don't know you have to keep backups of your data.
Re: (Score:3)
Contracts are a thing. You pay someone for something, they don't do it, you sue them. The fact you think IT is magically different because you know how to do your own backup is irrelevant.
I know how to build my own house, does that mean I am "not a grown up" when I pay someone else to do it?
Re: (Score:3)
I think you missed something here. They had only one copy of the data. Paying someone to take care of it isn't the same as keeping a backup copy.
Re: (Score:2)
No court order in the world can un-burn a dollar.
Re: (Score:2)
Contracts only help if both parties still exist. If one doesn't, it gets complicated, with many lawyers involved.
And only damn fool only has one copy of important data.
Re: Everybody learns the hard way (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
In 1986, you were responsible for picking a physical location for your word processing files, consequently you were empowered to back them up.
Re: (Score:2)
To be fair, every major OS refuses to do backups by default. You have to do everything manually yourself. As much as geeks love to scream about the importance of backups, it's remarkably difficult for ordinary people to do it, especially if you want proper version-controlled backups, and especially if you want catastrophic full-system backups. Most of the time, tech companies don't allow you to make backups, citing "copyright" concerns.
The best anyone offers is cloud storage (ie, a mirror), because that
Re: (Score:2)
Backups are only difficult on Windows because apps store data in random places all over the system instead of in your Documents folder. If Windows was a real operating system you'd just backup your home folder and that would be everything you might need.
I moved all the user folders from C: (a small boot SSD) to d: (an 8TB SSD) and it's insanely hard to stop programs still writing to C: because "obviously that's where the user files are".
Re: (Score:2)
I have a Mac and use Time Machine backups. It is easy to use, you can back up to multiple disks, it stores different versions... it seems about as simple a system as it could be. It may not give you 100% of the features you want, but then it wouldn't be simple. You do have to know to use it and remember to do the backups (it does nag you after a while). It is perhaps a bit more costly than tape (backups to hard disks/ssds)
The one time I needed to restore, it worked perfectly.
But, maybe you don't think MacOS
Fault (Score:5, Informative)
It always sounds like a lot of money to the bean counters until you explain to them the station probably spent tens of millions of dollars producing that content, and all it takes is a corporate meltdown or technical SNAFU and all of it is gone.
Re: (Score:2)
I threw up a little.
Re: (Score:2)
Loosing that data is YOUR fault.
And breaching contractual obligations it the other parties fault. People seem to switch off their brain when it comes to backups. Just because you *can* do something yourself doesn't excuse a contractually obligated partner from fulfilling their part of a contract. No one is talking about a google "you're the product" free tier here. You pay for something you should get it, professionally, and expect to sue if you don't.
Re: (Score:2)
Having one backup of critical data in the hands of a third party is a single point of failure that isn't under your control.
Yeah, the cloud service is at fault for their failure, but whoever at the station who decided on a single point of failure is far, far more at fault for making a stupid, incompetent, lazy decision to rely on something that any competent IT person would (and probably did) tell them isn't, and inherently can't be reliable enough for the task.
Re: (Score:2)
What did you expect on Slashdot? A bunch of us are grizzled IT nerds. We routinely crucify and eviscerate people who ignore sound principles. The importance of keeping backups, duplicates and hard copies has been well demonstrated for centuries.
I think we can all agree both parties fucked up.
Re: (Score:2)
Oh I didn't say the backup strategy isn't important, that was stupid of PBS. But the fact is the *FAULT* lies entirely and exclusively with the company contracted to look after the data. Legally, ethically, and practically.
Re: Fault (Score:2)
That's not what is happening here. One party no longer exists, and rget are suing that company's hosting service who, rightly (imo), is not providing access to the data without a court order. Once they get the court order, they're (presumably) off the hook.
Re: (Score:2)
You put ALL of your data on one cloud provider with no local backup? Loosing that data is YOUR fault. I It always sounds like a lot of money to the bean counters until you explain to them the station probably spent tens of millions of dollars producing that content, and all it takes is a corporate meltdown or technical SNAFU and all of it is gone.
Having a local backup defeats the purpose of having a cloud provider. PBS, because of their financial situation, used a fly by night cloud provider. That's a no no. If a cloud provider is going to manage PBS data (with no local backup) then they should have used a reputable provider. Hopefully PBS management realizes that the solution that looks cheapest at first often turns out to be the most expensive in the long run.
Re: (Score:2)
Having a local backup defeats the purpose of having a cloud provider.
What an idiotic thing to say. No one with any real IT experience could possibly keep a straight face if you said that to them.
Having a local backup is part of the 3-2-1 process that is proven, for decades, to be the right way. If your IT people don't keep multiple backups in at least two separate locations, they're incompetent and shouldn't be trusted with a pencil sharpener.
Re: (Score:2)
lol, do you not know how public broadcasting works?
Re: (Score:2)
We're witnessing how it works, with a single copy of critical backups in the hands of a third party that is now defunct. Which is to say, it works . . . poorly.
The legal fees to get a usable copy of their backups is going to cost them orders of magnitude more than keeping proper backups would have. But apparently, their main plan was to use cinnamon flavored unicorn farts for backups.
Re: (Score:3)
You put ALL of your data on one cloud provider with no local backup? Loosing that data is YOUR fault. It's a few thousand dollars for a half dozen hard drives to store 50TB. Or heck, 50 100GB M-DISC Blu-Rays for less than that. That's what the Disney Imagineering sound department does - M-DISC backup, live copy on file servers, then secondary cloud backup.
While stations vary, most local PBS stations simply do not have the money for that level of IT staffing (and after recent reductions in funding, even those that may have had such staff, have probably had to let them know). Of course, if you want to volunteer your time, and the time of your backup, and the time of your backups backup (all for depth and redundancy) to your local PBS station, perhaps that can change.
Re: (Score:2)
Well, then there's that Toy Story 2 complete data loss and extremely lucky recovery [techspot.com].
Re: (Score:2)
No it isn't the c-suites were sold on “The Cloud” (no doubt over the objections of their own technical people). Now they're going to install AI slop on top of this!
Don't put anything in the cloud... (Score:3)
... that you can't get out in 2 hours or less. If it's mission critical that is.
Re: (Score:2)
that you can't get out in 2 hours or less
That would be a neat trick with 50 TB of storage.
Re: (Score:2)
There is so much wrong with yours statement:
a) 2 hours is arbitrary. Why 2?
b) Mission Critical does not necessarily define a time limit. Mission critical documents could be ones that won't need accessing for weeks or months.
c) Why are you focused on putting things "into" the cloud instead of what is actually important: having a business continuity plan.
Your post is as silly as those people who think "air-gap-and-forget-it" is security. It's a great way of making very dumb business decisions.
they shutdown and did not remove the hardware from (Score:2)
they shutdown and did not remove the hardware from the rack?
Credit to Iron Mountain, and WTF for PBS (Score:3)
That being said, 50TB isn't that much data, and frankly unless the data is in the PB level, why wouldn't PBS have their own local backups? I just checked Amazon at 4:15pm ET on the 14th of August 2026, a 30 TB Iron Wolf Pro HDD is ~$1 600. Assuming a raidz3 (ZFS), and assuming you only make a single pool, 8 disks @ 30TB would give you 150TB of usable storage, which would cost $12 800, and assuming you want to mirror that into a second pool, that would come to $25 600. If you're not willing to spend $26k to protect irreplaceable data, that's really just bad foresight. I understand it's a lot of money, be realistically, if the data is important, and you would like to be certain you don't have to worry about the Cloud provider going bust, it's a sensible precaution.
At the PB level, which would need 37 disks per pool, for a raidz3, would be ~$60k, mirrored, that's ~$120k, which is a TON of money, but, it would assure the data is safe, and again, if the data is irreplaceable, it's a no-brainer. You could manage that with two 45-drives enclosures, for the disks, and still have 16 free disk shelves available.
Re: (Score:2)
So surely companies are going to line up and donate right . . . . right?
Re: (Score:2)
> a 30 TB Iron Wolf Pro HDD is ~$1 600.
And that's at AI Bubble prices, not the prices these people would have been paying. I bought some 20TB drives last year and they were about $300 each... so for $2000 they could have had a local RAID with more than 50TB capacity.
Lose 1KB transaction, shame on Cloud Provider (Score:2)
Now sack your IT Manager and hire someone competent.
Idiots, all of them (Score:1)
Just kidding. No it isn't. You can't order a company to do that, even in the case of a civil action pending.
Re: Idiots, all of them (Score:3)
Lolol of course they can. The data doesn't actually belong to the intermediary and it certainly doesn't belong to iron mountain. It's good that they required a court order, because the party requesting the data is not the party that provided it, but it's still the PBS station's data and they still have a right to it.
Re: (Score:3)
If there was, say, a physical self-storage company, lets call them Tim's Self Storage, and person A paid someone person B to store a bunch of their hard drives in a box in a storage unit that person B rents from Tim's Self Storage - and person B took the box and the hard drives and locked them in the storage unit, and then... disappeared, took the money and ran, refused to talk to person A - it's not entirely unreasonable for person A to contact Tim's Self Storage, maybe even go to court, to get Tim's Self
The cloud is a trap (Score:2)
Run away
But first, back up your data
"Open Source Storage", huh? (Score:2)
"But they were so much *cheaper* than AWS! Two providers each with a copy? Do you think I'm made of money?"
Escrow service for data? (Score:2)
You should have put the data in an escrow for such a situation. We have had an escrow service for our source code and we included that with the purchase of our product where if we disappeared the customer could get the source. Of course 50TB escrow would cost a fortune.
Why no local copy? (Score:2)
This isn't a lot of data. I know someone with half this in their personal Plex library.
So: why no local copy in addition to the cloud copy?
Re: (Score:2)
Actually, the company they used was Open Source Storage to hold the backups. OSS used Iron Mountain for their storage provider (Iron Mountain provides both physical document storage, but they also have a data center in their storage cavern to serve as long term data archival in digital format.)
OSS just up and disappeared on them suddenly (they did have a 30 day agreement to cancel). PBS is suing Iron Mountain to basically get at OSS' stored data so they could recover their archives.
Basically, by filing the
This is why tapers/Tivo-ers/stream recorders exist (Score:1)
50TB? What is their IT department thinking? (Score:1)
My home NAS holds 50TB - that's 8 x 8TB in Raid 5, striped for 7+1. Before the AI nightmare sent prices through the roof, the whole thing cost under two grand, and that's with dual SSDs for read/write buffering. Let's see what they might have been spending for cloud storage:
Google charges $ 0.02 / GB / Month - so 50TB is a grand a month. By the end of a year, they could have had a shelf full of NAS units, a couple in the CEO's garage, whatever. Even if their provider was charging half that, they'd _sti
It's PBS, there's no money for anything (Score:3)
Some of y'all appear to be missing that this is a PBS (Public Broadcasting Service) station, who might not even have the budget for a dedicated IT person. This was probably picked by someone in the video control room when they got told to clear out the space used for the video tape archives. My local PBS station unfortunately didn't have anything like that, so lost their archives when there was a fire at the station about 15 years ago.
If you're criticizing their spending priorities... did you contribute at the last pledge drive?
Re: (Score:2)
So, they can't afford a few grand for proper backups, but they can afford six (or seven) figures in legal fees?
That probably makes sense to them.
Thus the lesson of the cloud! (Score:2)
Don't be a eunuck, have a working backup plan.
Re: (Score:2)
Obligatory Dilbert [programmerhumor.io]
Risk management vs. the CFO (Score:1)
I was personally involved with planning the DR strategy at my previous company. What the bigwigs wanted was some kind of assurance that we wouldn't be screwed if a tornado or fire hit our building where we self-hosted 90% of our corporate IT infrastructure.
What we told them was to build out a DR site at a colo. Disk arrays, DR VMWare cluster, a bunch of Solaris servers, a new fiber uplink, the works basically.
CFO's eyes watered and he screamed that he absolutely wouldn't pay for all of that. Okay buddy, not
CSB time (Score:3)
They laid me off, saying I couldn't meet deadlines reliably. By the same guy that had a habit of giving me sysadmin work when I was busy with "real" work on a tight deadline.
Sure enough, 3-4 months later I get a phone call. Seems their new sysadmin, a person who didn't know squat about Unix nor backups but did it in their spare time, had managed to lose the source code. She knew I'd been taking monthly backups home and asked if I still had the tapes. I looked at my bookcase with a couple months of backups and said "nope, I threw them away".
Re: (Score:2)
I was supposed to do this in my spare time, worked out to 10-20 hours a week.
Are you some kind of charity? That's paid work, unquestionably.
Complexity, specialization, and modernity (Score:2)
We're all, here, aware of the right way to do backup stuff, but we're an unusual group (some of us hear likely that a lot :-).
I've been thinking lately about the complexity I'm faced with day-to-day. Dealing with a 401k, e.g. Totally trusting the advice of experts? Mostly. Diversify, low cost index funds, etc. And then doing some weird shit where I decide to put 10% of it in whatever pundit I heard last suggested. If I went to 'finance slashdot', whatever the hell that might be, they'd be saying the same th
Re: (Score:2)
Not entirely clear on how that little switch works to pop up my toast,
https://www.youtube.com/shorts... [youtube.com]
https://www.youtube.com/shorts... [youtube.com]
Repeat after me... (Score:2)
Harddrive-Crazy / LTO-Tapes (Score:2)
How to prevent this in the future
34 x LTO-5 Tapes
5 x Western Digital Ultrastar DC HC520 12TB, ISE
general hint: dont buy consumer sh*t
30 years of content, 40 years of begging (Score:2)
"We interrupt your regularly scheduled enlightenment to bring you this important message: PBS needs your help—because tote bags don’t grow on trees. If you’ve ever enjoyed a documentary about antique spoons, a soothing nature special narrated by someone with a voice like warm oatmeal, or a pledgedrive host who’s been awake for 19 straight hours, then consider donating today. Your contribution keeps our programming adfree, our volunteers caffeinated, and our Muppets emotionally suppor