NYC Data Center Needs Focus On Fuel 162
Nerval's Lobster writes "Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel? In the wake of Hurricane Sandy, the challenges facing data center operators in the affected zones include pumping water from basements, waiting for utility power to be restored, and managing fuel-truck deliveries. And it's become increasingly clear which companies had the resources and foresight to plan for a disaster like Sandy, and which are simply reacting. Here's the latest on providers around the New York area." And remember, having fuel for machines sometimes only means it's time to start the manual labor.
Idiots (Score:2, Funny)
Data center on the coast - and they're surprised by what happened?
Good God! I saw this coming for years - I guess I AM GOD!
Listen to me my sheep, if you're on the coast, you will be flooded and wiped out by hurricanes!
And more, my sheep, you will be knocked out by tornadoes in the Mid-West!
And yet, my sheep - oh, fuck it! - my morons - you will be taken out by Earthquakes on the West Coast!
And for you in the Middle East... You'll be taken out by terrorists!
And in a few years by the Omicron Persei 7 peoples
Re:Idiots (Score:4, Funny)
And if you are in Britain, you will be taken out by 2mm of snow.
If you read the blogs from hurricane Katrina... (Score:1)
... you would know. The blogs about keeping data centers running in New Orleans during Katrina were incredible.
-1, Obvious (Score:2)
How can you not have a multi-day supply of diesel on hand?
Re: (Score:2, Insightful)
How can you not have a multi-day supply of diesel on hand?
Some places did, but in the city, you can't store it above ground - which means that that week's supply of diesel is now mixed in with the ocean.
Re: (Score:2)
In the Northeast, diesel and #2 are pretty similar. Why not switch the valve and send some furnace oil to the gensets?
Or do they not use oil in NYC?
Re: (Score:2)
#2 is diesel.
Re: (Score:2)
You need a little more compression to light #2 fuel oil, the engine in the duece and a half [wikipedia.org], LDS-427 Turbo Multifuel I-6; LDS-465 Multifuel I-6; 127 to 170 hp (95 to 130 kW) will burn it OK, but they are small for data center usage. I'm sure there are marine engines off the shelf that are both large enough and able to run on both.
Re: (Score:3)
I don't understand why they're using deisel rather than natural gas. With natural gas they could have the generators on the roof. Illinois Secretary of State mainframes have two backup generators, both natural gas. I've lost electricity a whole lot of times, but I've never had a natural gas disruption in my six decades. Not even in 2006 when we were hit with two tornados and electricity was out for a week; I heated my apartment with the oven (the gas furnace needs electricity for blowers and theromostats).
Re: (Score:2)
I've replaced several fuel pumps in cars... I've never had to do anything with the tank. Must have been doing it wrong.
Re: (Score:2)
Maybe "nearly every car" might be an over statement. But every car i have owned has the pump in the fuel tank. My 94 chevy pickup which i have replaced the pump is in the tank. my first car an oldsmobile had the pump in the tank. And my Taurus has the pump in the tank.
Re: (Score:2)
Lets not forget that most modern cars have more than one pump.
Re: (Score:2)
I did some research looks like you only replaced the under hood pump. As i said Nearly every manufacturer uses an in tank main pump.
http://tankonempty.com/viewstory/71 [tankonempty.com]
http://www.ehow.com/how_7838847_install-intank-fuel-pump.html [ehow.com]
But feel free to do your own research or just try to shoot me down again.
Common among data centers? (Score:3)
I was under the impression that a fuel supply was a standard part of the contingency plan for any data center.
I recently visited a new data center opening near me. The operator had contracts with several fuel suppliers that in the event of a power outage, the first one to get a full tank truck through their front gate got paid, and would keep getting paid for each additional truck that was needed. Any latecomers would be turned away, effectively making it an exclusive contract upon arrival.
Re: (Score:2)
You also have to hope that the trucks are allowed to travel. I worked at a DR company that sold contracts on datacenter in a trailer solutions. Many companies that got hit by Katrina declared, and not one single trailer was sent. The contract had provisions that fuel for the generators had to be available. There were going to be problems, so nothing was sent to assist them. Basically free money for the potential use of these trailers.
Re: (Score:3)
Driving tank trucks around any of the Boroughs seems like a pretty tough proposition even today. In a real disaster, plan to not be able to even call the fuel dealers and ask if they are coming, much less getting them through the main gate and down the service road.
Re: (Score:2)
Only works if the fuel truck operators have nowhere else to be.
Otherwise, why woukd you turn up without a guarantee of payment.
Unless a massive premium is offered.
Re: (Score:3)
If the fuel isn't already stored on-site, you're screwed.
Re: (Score:2)
Re: (Score:3)
How do you mean?
Like access to the fuel from private companies? Are they forced to sell to FEMA if they have the fuel vs. someone who they made a contract with?
Diesel does not last forever. (Score:5, Informative)
Once refined, it degrades. Oxidation, bacteria. Algae. Amazing things grow in diesel. You can add preservatives, but these only go so far. Keep it too long, and it's unusable. If you don't use it, you have to dispose of it. There's a large disincentive to keep diesel around.
Re: (Score:2)
Once refined, it degrades. Oxidation, bacteria. Algae. Amazing things grow in diesel. You can add preservatives, but these only go so far
I wonder if it would be economical to manufacture a grade of diesel-compatible fuel that doesn't degrade, for purposes of long-term storage. It would consist purely of highly refined aliphatics, leaving no unsaturated bonds available for oxidative attack. Heavy metal content would need to be near-zero as well, as they could function as catalysts for breakdown, or nutrients for microbes. The storage air-space above it would be purged with inert gas or at least desiccated, again to avoid microbial growth.
E
Re: (Score:2)
If a liquid paraffin similar to medical grade mineral oil (but a lighter fraction) will keep a modern Diesel (yes, with a capital D) happy and survive long-term storage, then the problem is you: Go forth and sell.
Re: (Score:2)
That's not a hard problem to solve. You just fire up the generator and burn the fuel. When you're using the generator, you disconnect from the grid so you're not paying for commercial power.
Or, presuming you live somewhere this is allowed, fire the genny up and let it feed into the grid, thereby offsetting the cost of the liquid fuel (at least a bit).
Re: (Score:2)
That's what mine does, every week there is a power system test including running the generators to burn off some fuel.
Move servers to the cloud (Score:1)
Re:Move servers to the cloud (Score:5, Insightful)
Re: (Score:3)
Re: (Score:2)
Because who would be so dumb to trip over a wire.
Re: (Score:2)
http://xkcd.com/908/ [xkcd.com]
Re: (Score:2)
Re: (Score:2)
No. Anything entirely within a single datacenter or city does not fit any reasonable definition of "cloud." That's the whole point of it; no single point of failure, in case something like Sandy happens.
Have you noticed any outages on google, or youtube, or netflix, or any other geographically dispersed network? I haven't.
Re: (Score:2)
Everyone already knew this (Score:5, Insightful)
Everyone already knew that the on-site fuel supply is the limiting factor of power availability in a disaster. Even fuel delivery contracts mean nothing in a disaster or wide-spread outages - hospitals, EMS and other government services will trump the fuel delivery contract, if a hospital needs fuel, they are going to get the fuel that's been "guaranteed" for your datacenter.
There's no reason to spend big $$ creating a flood proof, earthquake proof, tornado proof, airplane crash proof datacenter in the middle of a city when you can have a disaster recovery site 1000 miles away that's not subject to the same type of disaster. (except maybe an asteroid strike, but there are few datacenters on the moon). No matter how disaster-proof you make your datacenter, mother nature (or man) will always find a way to create a disaster you didn't plan for -- even if that "disaster" is a typo in a router configuration file that takes down the network, or a contractor accidentally shorting out the emergency power cutoff switch wiring when bolting a rack to the wall.
Re: (Score:3)
If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.
Re: (Score:3)
If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York. Now, of course, you generally want a backup site that is more than 100 miles away, in the event of actual disasters, but you definitely want your primary facilities as close to your users as you can get.
Sure, there will always be datacenters in NYC, but that doesn't change the fact that instead of putting all your money into trying to build a datacenter impervious to all hazards, you're better off having a second site far from your primary site. Real-world constraints mean that you can't build that perfectly impervious datacenter when you're subject to real estate prices, building codes, fire codes, and Murphy's Law -- there will always be a disaster that the facility can't handle.
Re: (Score:2)
If you have a need for high speed, low latency, and there is enough demand, you bet your socks you will build and sell space in a datacenter in a city like New York.
I don't think OP was saying that you shouldn't have datacenters in NYC, persay, but rather that you'd be better off spending most of your "disaster-proofing" budget on off-site disaster recovery equipment/services.
Don't put all your eggs in one basket, and such.
Re: (Score:3)
...there are few datacenters on the moon...
The latency is a real bitch.
Re:Everyone already knew this (Score:4, Insightful)
I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.
So as you can see, this isn't a question of them not stocking enough fuel -- the cost of storing that fuel is prohibitively expensive.
And that, people, is why they didn't load up on fuel ahead of the storm. You can't simply pay $83 million a year for your data center to protect against a threat that might only materialize once a decade, and be severe enough to deny fuel deliveries or power restoration for that long of a time frame. Generator backup is a short term solution. There is no long term solution for disaster recovery, at least not one that's cost effective. Not in an urban setting.
The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services. Everyone else plans for a couple day supply and leaves it at that.
Re: (Score:3)
I would have thought it was obvious, but the middle of the city is where the telecommunications infrastructure is. It doesn't exist in a barn a hundred miles from any major city. And fuel has a shelf life. Diesel will slowly oxidize over time, and so the time you can keep it in the tank is about 12 months. You'll burn about 72 gallons an hour per megawatt (as a rough average). So a 2 megawatt data center will need about 3,500 gallons of diesel per day. A gallon takes up 231 cubic inches of space, so a single day's worth of fuel would need a tank with a capacity of 67,375 cubic feet. The average height of a floor in a skyscraper is 12.5 feet. In New York city, the average city block is 264 feet. That means that even if you filled an entire floor of a skyscraper with nothing but diesel fuel, you'd still get less than a week's worth of fuel guys. At a rate of perhaps $100 per month per square foot... you're talking about $83 million a year for a floor of an outlying area just to store that diesel fuel. Mind you, near Wall St., that price is probably going to be double or triple. The price of fuel is peanuts compared to this; 7 days of fuel for a 2MW plant would cost you only $94,000, plus transport costs.
While I agree with your point in general, your math is off. I can see our own 1MW generator and 3-day fuel tank from my office, which is no where near the size of tank you calculated -- it should consume around 1/4 of a city block using your figures.
A week of fuel (24,000 gallons) is 3200 cubic feet [google.com], with a 10 foot high tank, that's only 320 square feet, at $100/month that's still a pricey $384,000/year. But since that 2MW of power is enough to power 1000 servers (assuming 1000 watts/server, half the powe
Re: (Score:2)
I just googled to source the numbers to get a ballpark figure. So yeah, undoubtedly they're off -- people spend months preparing reports for their data center on fuel consumption, storage costs, location, etc. I spent 20 minutes. But putting the numbers together shows that even if you lowball all the numbers by 50%, storage cost is still massively eclipsing fuel cost.
People on slashdot here were being 'armchair CTOs' and saying how if it was their data center, they'd have bought all that extra fuel. "Well
Re: (Score:2)
I just googled to source the numbers to get a ballpark figure. So yeah, undoubtedly they're off -- people spend months preparing reports for their data center on fuel consumption, storage costs, location, etc. I spent 20 minutes. But putting the numbers together shows that even if you lowball all the numbers by 50%, storage cost is still massively eclipsing fuel cost.
Sure, I understand the fuel usage numbers were an estimate, but your calculations weren't off by 50% - you were claiming that it would take an entire 264x264 foot office floor (70,000 square feet) to store a week's worth of fuel. I was pointing out that it only takes about 300 square feet - you were off by a factor of 233, not by 50%. Instead of an unaffordable $83M in rent and an entire office floor to store a week's worth of fuel, it only takes an 18 foot by 18 foot room and $400K/year. That's a huge di
Re: (Score:2)
"The only time you can justify spending that kind of cash is if you're supporting critical infrastructure like phones, hospitals, and emergency services."
Which is why I can't understand why some hospitals in New York City had to evacuate due to power loss. I heard that one hospital evacuated the goddamn NICU.
...
Whoever was responsible for disaster planning for these hospitals ought to be facing criminal charges.
Hospitals, even publicly funded ones, don't have unlimited funds to implement disaster recovery plans. If it comes down to reducing services during the 99.9% of the time when there is no disaster versus spending money to ride out the 0.1% of the time when there's a major disaster, spending the money to help more people now isn't necessarily worse than planning to evacuate during an uncommon disaster.
I'd question the need for a hospital to have 45 days of fuel on-site... keeping the lights on does little goo
Re: (Score:2)
A major disaster is the time when you really need your hospital to function properly. So spending some money on disaster preparedness is prudent.
Yes, we keep large amounts of emergency supplies of consumables on site : medical gases, medications, food, linens, antiseptics, needles, lines, etc.
Telling injured hurricane victims that show up in the E.D. "Sorry, dude, but this storm really screwed us up, man. We can't treat you." isn't acceptable.
Sure, I understand that a hospital needs to function for some time during/after a disaster, but 45 days of supplies seems like expensive overkill - do you have millions of gallons of water on site and your own treatment plant on-site since it's not likely that your municipal plant will withstand a 30+ day power outage.
What hospital is this?
Re: (Score:2)
Exactly. This is why your monday-quarterback "have a disaster recovery site 1000 miles away" is just as totally frigging useless.
You've left straw all over the place. Sweep up and sit down.
Care to elaborate? How is geographically diversity not significantly better than no geographical diversity?
Africa - grid is for backup (Score:4, Interesting)
Cell sites I worked in Africa run a pair of Cummins generators as their main power. In the unlikely event that both of these fail at the same time, there's a chance that the main grid might be working well enough to take over. But the fuel is the biggest priority on these sites.
At the moment, we have tankers on site at all of our NY DCs. 2 of them are on generator only, so we're topping up the tanks every 4 hours. The generators will need full overhauls when we're back on the grid properly, but for now we're keeping our clients serviced which is what matters.
Re: (Score:2)
I've heard this before, but I've never seen the reasoning - why do the generators need full overhauls after running for a few days? Aren't these the same engines that are in trucks/construction equipment and run for thousands of overalls without an overhaul?
Re: (Score:2)
I've heard this before, but I've never seen the reasoning - why do the generators need full overhauls after running for a few days? Aren't these the same engines that are in trucks/construction equipment and run for thousands of overalls without an overhaul?
When i worked at some remote construction sites, they had 1MW+ generators that went for over a year of nearly continuous use between overhauls. As I recall, they did oil changes and other routine maintenance monthly.
An emergency power generator might be run at a higher load than a continuous duty generator, but I can't imagine any large diesel generator needing an overhaul after a few hundred hours of operation.
Re: (Score:3)
In this case, it's because the generators got a little damp due to some rain in the area just before going into service :D
Given the amount of things that have happened, it's a 'better safe than sorry' move to do the overhaul.
They got plenty of practice with Irene (Score:4, Interesting)
Why have a submitter (Score:2)
Ok, so this is a geeknet story submitted by a geeknet employee.
Why make it look like a user submitted the article?
Not that it actually matters.
Should've learned the lessons (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Don't put all your eggs in one basket, that's the best plan.
But there again, budget comes into play. Can be pretty costly to maintain a backup data center in another state though. Infrastructure, hardware, good data lines connecting the two, etc.
Then you'll get what you (are willing to) pay for.
Re: (Score:2)
And colo providers get driven down on cost as customers forget the previous catastrophic event. Every dollar spent enhancing a data center is passed along to the customer in some fashion. When your per-cab or per-foot pricing is higher than the competition--even if you can demonstrate more resiliency--it is a harder sell.
You can talk flood plains, gen sets, fuel locations and delivery schedules, CRAH and chiller redundancy, etc. As long as the SSAE16 or SOC 2 is current and there are no lingering memories o
Re: (Score:2)
There is only one Wall Street.
Re: (Score:2)
This is very easy to plan for. Simply decide how far away your DR location needs to be. If the problem is fuel, then the DR plan was a shelter in place plan, and is going to suffer miserably for many reasons. Long term power outage caused by floods is very obvious, and will bring the exact issues that are being seen here every time. This is likely in a plan for all of these places. A fire would knock these completely offline, so there always has to be consideration of failing to an alternate location. The m
Natural Gas (Score:2, Insightful)
I was visiting 1 summer st once, and I asked the guy who set the place up "why don't you use natural gas? You can just get it from your local utilities, or have it delivered by truck." and he looked at me with the "you poor idiot" look people give me some times. Then he said "Nobody uses natural gas because you can't tell if the supply upstream has been disrupted."
I didn't feel like explaining "no, dude, you pump it into a pressurized tank and monitor the flow, *and* you have a diesel generator as a backu
Re: (Score:2)
Re: (Score:3)
Having large quantities of pressurized natural gas can be an explosion risk. Diesel's just a fire risk. Storing pressurized natural gas likely involves a lot of regulatory issues that diesel doesn't.
Also, natural gas is not very efficient volumetrically: diesel has 37.3 MJ/liter. Natural gas at normal pressure has 0.0364 MJ/liter. Even compressed to 250 bar (3,600 psi), a non-trivial pressure, it only has an energy density of 9 MJ/liter. Liquified natural gas at -160C has an energy density of 22.2 MJ/liter.
Re: (Score:2)
ever tried to service a pump that's inside a tank? real pig to replace them... far easier if they're outside the tank as you can have shut-off valves either side and isolate them while replacing them... you don't have to drain the tank and purge it of fumes first before accessing the pump...
Re: (Score:2)
SOP on gasolineautomotive fuel pump replacements is, if possible, to do so with a full tank to minimize the risk of badness (gas itself doesn't burn, while gasoline vapors both burn and can be explosive).
What is it about a large volume of Diesel fuel that makes this different in any meaningful way?
Re: (Score:2)
Lots of money to be made in this (Score:4, Interesting)
One of my buds is IT director for a company that resupplies generators. The logisitics for it are crazy as you route trucks on available streets, deal with priority of the customers (hospitals front of the line) and optimize resupply into mostly empty tanks before they actually empty, etc... And you have >10 days of this 24/7 after a storm.
After Hurricane Allison here, some companies in this sector went out of business before the power came back on. We were better prepared for Ike, but I think that's because facilities are more willing to sign contracts with the pricey but reputable businesses.
This is the most expensive way to get fuel: you need a massive amount for a short time, and you need it consistently during that time.
Re: (Score:3)
The problem has been getting it from where you fuel up the tanker, to where it's needed. The fuel's cheap comparatively, but the trucks aren't really off-road vehicles. When you've got flooded streets, streets lined with debris, or streets that are completely gone, the trucks really can't make it in to resupply.
So the massive amount of fuel needed is needed right when the ro
Re: (Score:2)
That's not a fuel truck. This [badhaven.com] is a fuel truck.
Re: (Score:2)
I think you mean Tropical Storm Allison. Man that thing flooded the hell out of Houston. Then went on to travel over land all the way to the mid-Atlantic States flooding pretty much everywhere it went. Following this the name Allison was retired from the Hurricane name list. It remains the only name to be retired from the list for a named storm that never reached Hurricane strength.
Natural Gas diesels (Score:5, Interesting)
30 years ago, I retired from the Navy, along with a friend who was a Machinist Mate on Navy submarines. I didn't see him for a couple years. Turns out that he had spent 10 of those years servicing batteries on diesel submarines, and ended up servicing large batteries for the phone company -- about the same size batteries that had been on the boats. He switched to servicing the diesels (because of his prior experience on submarines with their diesel generators), and one of the first to deploy natural gas fired diesels for sustained power, first for the phone company, and then for a local utility.
There's the solution. Don't pipe flammable liquid to the top floor, pipe the natural gas. Contrary to popular belief, it is a safer fuel, and requires less maintenance. Automatic shutoff valves work better, less explosive volume released on a tubing breakage, and it doesn't rot the pipes like liquid diesel fuel, disperses to atmosphere in the case of a leak, and doesn't make everything around it flammable, when it does leak.
Engines run at least 4 times as long between between overhaul cycles, and it doesn't dilute the lube oil.
Natural gas is going to be a HUGE change for this countries infrastructure, both in common usage, as well as emergency failover services.
This guy has made tons of money, by the way. Has a condo and car in San Diego, LA, Seattle, New York, and Atlanta -- cheaper than hotels and taxis, and doesn't know what else to do with his money.
Re: (Score:2)
I'm generally enthusiastic about natural gas, primarily because they can safely run unmanned for extended periods, but let's be honest, there are plenty of problems... Not only do the generators need to be bigger and more expensive, but you've gotta have a back-up plan in case those natural gas lines are either damaged or shut-down.
Propane isn't a direct substitute, but d
There is always something. (Score:2)
At my work we had our generators on the roof above 6th floor. It had a small tank that can run for a couple of hours. In the basement, a larger tank that had fuel for 2 days.
Sadly when the power went away, so did the power to the pump that were supposed to lift it to the root. So the our electrician all of the suddon got busy.
At another workplace we had two huge generators with flywheels for starting them, plenty of fuel and even our own redundant line to the powerplant some miles away. Sadly when the insta
what a stooopid question ;-) (Score:2)
Who knew that the most critical element of operating a data center in New York City was ensuring a steady supply of diesel fuel?
Uhm, anyone that was conscious for Katrina...
Not a problem (Score:2)
We found these heaps of backup tapes down in the basement. We just shoveled them into the boilers and kept the generators running.
Re: (Score:2, Insightful)
Good luck turning a profit.
wrong location (Score:2)
As others have said, the coast is probably a bad place for a data center. Somewhere geologically stable, with reliable power and a good backplane connection would be best.
Re: (Score:2)
Where might be a geologicaly stable location be? If the new madrid fault ruptures [wikipedia.org] half the continent will be ringing at 8 on the rickter scale.
Re:Clusterfuck.... (Score:4, Informative)
This is one of those situations where the invisible hand of the free market fails miserably:
Company A: plans properly for contingencies, but has to charge $10 more per customer.
Company B: thinks they have planned for contingencies as well as A (but accredit the cheaper pricetag to their managerial prowess), makes the same claims about uptime as company A, and undercuts company A.
Company A goes out of business as B steals all of A's customers.
Disaster hits, but due to company A having gone out of business 2 years ago, B can legitimately say "no one could have planned for this!" and likely gets away with it...
Re: (Score:3)
Except that company As are still in business - given a bunch of places did just fine. So the invisible hand seems to have done just fine.
Note these are NYC data centers if you were buying the cheaper option you weren't in one of them anyway.
you can't store 3 days of fuel at high floors (Score:3)
you can't store 3 days of fuel at high floors that can be a big fire hazard.
Re: (Score:1)
You'd think that a generator would be on a lower floor, with a supply area nearby for the extra fuel. I have visions of people carting the generator into the server room and letting rip.
Re:you can't store 3 days of fuel at high floors (Score:5, Insightful)
Which makes me question the wisdom of urban datacenters in the first place.
Re:you can't store 3 days of fuel at high floors (Score:5, Interesting)
Which makes me question the wisdom of urban datacenters in the first place.
It's nothing particularly special to the datacenter being in a city. Get a big enough disaster and you will be knocked off the net, wherever you are and whatever steps you take to prevent it. The best mitigation strategy is to have geographically distributed datacenters that can run replicated services that are structured so that losing one site entirely is not crippling (though it might hurt a lot). This isn't a simple thing to do — for most types of service, you have to design the overall service from the ground up to work that way — but it's not too bad. (And at least with the Cloud you don't have to build your own collection of geographically-separated datacenters or pound your head against the problems you usually get with hosting providers when you're desperate for service in a hurry.)
Mind you, rents in NYC are so high that I'd guess that nobody's going there for the cheapness of the service. Instead, it's probably for the proximity to the Wall St. exchanges. The fun thing with that? Any datacenter which could satisfy the latency requirements for people in that business would have been deeply hit by this particular disaster, and all the places that could easily offer continuous service this time would never be able to normally compete for this particular type of business. To double the fun, the exchanges were closed for two days anyway. That's modern capitalism at work...
Re: (Score:2)
The poor quants. Won't ANYONE think of the poor quants?
Re:you can't store 3 days of fuel at high floors (Score:4, Insightful)
The poor quants. Won't ANYONE think of the poor quants?
The last 15 years have set theoretical physics and mathematics back a generation.
But goddamned if they haven't shrunk the interval of integration on the vig! Truly benefactors of society.
and then the lag will suck big time (Score:2)
and then the lag will suck big time
fine lets put the data center on mars can you take 40min ping times?
Re: (Score:2)
Nah, just need signals that have speeds of c^2.
Re: (Score:2)
Re: (Score:3)
Diesel isn't much of a hazard if stored properly. You need a wick for it to burn or impossible pressures (by atmospherical standards, that is) for it to ignite.
Re:you can't store 3 days of fuel at high floors (Score:5, Insightful)
Think. It is a fire hazard. As in "it starts leaking because of event X happening". And where does stuff go when it leaks? Yes, it goes down.
That is why fuel is stored DOWN in the basement. *Water*, on the other hand, you can store on the roof.
If you don't want to worry about your tank getting flooded, then make it water tight with air inlet well up high out of the water. There are safe and reliable ways of doing this without storing your fuel on the roof!!
Re:you can't store 3 days of fuel at high floors (Score:5, Informative)
You also need to make sure to bolt the damn tank down. Fuel and air are more buoyant than water or mud. More than a few flooded gas stations have suffered damage only because the tanked popped out of the ground.
Re: (Score:2)
A concrete slab, or in the case of an actual basement then you know, the floor/wall.
Re: (Score:2)
Ya, I was surprised they kept data centers up that high. I would have assumed basement level to keep the IT crowd from being exposed to sunlight.
Re: (Score:2)
Diesel generators run fine on Natural Gas and quite a bit cleaner too.
Re: (Score:3)
I agree. I recently has to do a disaster recovery plan for a law firm I administrate. They wanted to be able to weather any event without disruption. I took disruptions that I knew of within the last ten year, and expanding to situations that has happened in other parts of the town. Flooding wasn't really an issue because they are on a hill that would require most of the state to be under water before they were. The biggest concerns was power and internet outages, something happening to the building making
Re: (Score:3)
Yea, That is nice. Maybe later.
I am busy making sure that these servers that transmit data to hundreds of millions of people keep going no matter what kind of pounding it takes or how drenched in fluid it becomes, just like the chicks in the videos that the servers are transmitting.
Re: (Score:2)
Um, some would say the Lord's wrath for the sins of Americans has big ears and can shoot the jumper.
Me? the Lord needs do nothing special to make us miserable. We do that well all by ourselves. Special Wrath not required.
Re: (Score:2)
Re: (Score:2)
Off site data backups may be a good idea. But when HF traders have to be within microseconds of the exchange systems, a cave in the Rocky Mountains just isn't going to cut it.
I'm crying crocodile tears for them.
Re:NY Rookies (Score:4, Insightful)
First, building an electricity-hungry data center in NYC, with its notorious electricity problems and in the path of coastal storms... well... let's just say it's not the brightest star in the cosmos of ideas.
In fact, it kind of reminds me of... what was it? Oh, yeah. A major Southern coastal city that was built... mostly below sea level! Yeah! That's the ticket!
Re: (Score:2)
Compare that to here in the (relatively speaking) West, where electrical outages in any one place happen maybe about once every 5 years, and outages that last more than an hour or so might happen roughly once every 20 years or so.
I am aware that there are storm issues, etc. I'm not trying to sa