Google Pulling Back the Veil On Its Custom-Built Data Centers 47
jfruh writes: In the mid-'00s, as Google scaled up its data centers to meet increasing demand, "we could not buy, for any price, a data-center network that would meet the requirements of our distributed systems," says Amin Vahdat, the company's networking technical lead. So they had to build their own software-defined networks inside what were essentially vast warehouse-sized computers. And now the company is starting to tell the world how they did it.
And you all still don't need it (Score:5, Insightful)
Re: (Score:3)
Re: (Score:3)
Right now there's a massive push for even the smallest web projects to be "cloud scale" to this degree. No, the crazy custom shit that Google does in their datacenter is not something you need in YOUR datacenter.
I thought the current push was for the smallest web projects to run on a public cloud like Google or Amazon, so if you need this kind of scale, you can have it, but you don't have to pay for it until you need it. Is there some other push to drive companies to create their own datacenters to give them the ability to scale?
I know my company is completely in the cloud, there are literally no servers in the "server room", just switches and a firewall. Oh, and two 1 Gig fiber connections from 2 providers.
Re:And you all still don't need it (Score:4, Interesting)
Are those fiber links really from 2 independent providers? Where my company most needs redundancy, the fiber is owned & maintaned by a single provider and every one is a reseller.
Re: (Score:1)
Upload your data to both Google and Amazon! They'll fix you, but good.
Re: (Score:2)
Some providers will offer redundant links, even geographically separated redundant links (fibres enter the building though 2 separate ducts, head off in 2 different directions back to 2 different exchanges via 2 different routes... of course, no-one wants to pay for that so they very rarely actually do it.
If you buy "dark fibre" then you needn't care about a one provider's hardware/network being subject to a particular failure as you're only using a length of glass they happened to provide. Again though, is
Re: (Score:2)
One of the divisions of my organisation acquired quite a bit of dark fibre for almost next to nothing during the dotcom meltdown as providers were going out of business .
They even have 4 manholes in a 2 mile radius that's "theirs" - the covers have the old corporate name stamped on them.
Re: (Score:2)
Are those fiber links really from 2 independent providers? Where my company most needs redundancy, the fiber is owned & maintaned by a single provider and every one is a reseller.
One fiber goes out to the street where it connects to AT&T's fiber ring, a single fiber cut could take out our building, but it would take 2 cuts to take out AT&T's ring.
The second fiber connection goes up to the roof where it's connected to a (microwave?) transmitter that hits an antenna about 5 miles away where the provider has their own backhaul to an IXP.
Re: (Score:3)
Right now there's a massive push for even the smallest web projects to be "cloud scale" to this degree.
No, the "push" is in the opposite direction. Unless you are in the business of running datacenters, you probably shouldn't be running your own datacenter. Google/Amazon/Azure can do it better than you can.
Re: (Score:1)
What could possibly go wrong with that?
Obviously the only requirement for a good data center is scalability.
Re: (Score:2)
Maybe not Azure...
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Sooner or later Dice is going to render this site unusable. What a pack of fucking retards.
Re: (Score:3, Funny)
I think you give them too much credits; they've been trying for years and haven't yet succeeded :)
Re: (Score:1)
For some reason, it doesn't look any different to me than it did a year or a month or week ago. But I bitchslapped Slashdot's configuration so long ago that the last time I looked, I couldn't even find any configuration controls to do whatever I did. My goal at the time was 'simple' and they offered some 'simple' options if I recall. I haven't seen any share buttons.
But I am also hesitant to change ANYTHING in the settings for my account. This site is fragile in strange ways. That's been my experience
Please bring back "Read more" (Score:5, Insightful)
Pulled back veil (Score:2)
To expose a burka.
What a singularly uninformative article. I guess if you've been living in a cave the past 10 years you might not know that Google has a lot of computers and needs a lot of networking and had a project called Firehose (that didn't work) and one called Jupiter which apparently has.
I'm not intrigued by your ideas and I don't want to sign up for your fucking newsletter.
Re: (Score:1)
Re: (Score:3)
Well, if you have a lot of awareness of the landscape, you can sort of fill in the pieces. Basically what they did was reveal enough to show that they may be clever, (the principles described are sound) but not enough to actually change the landscape. What they purport to have done is already an obvious high level strategy to anyone in the world who would have actually been able to pull it off and probably is already in play at some of their competitors. If you ask someone who actually recognizes the spe
my other datacenter's a footlocker (Score:5, Funny)
Seriously. I'm building my new data warehouse inside a wooden footlocker. All I need is an ATX extender and to finish building the drive frames (for 8x3.5" drives, 12x2.5" drives and two DVD burners), and the back arm for the VESA mount for the monitor, then it all gets bolted together and fired up. It looks fuckin' sweet.
Re: (Score:2)
...laugh all you want, it's finished.
F-ing Slashdot (Score:5, Funny)
Re:F-ing Slashdot (Score:5, Insightful)
'share' is one thing I have filtered on ALL sites.
'social' is another. if your url has that string in it, it gets to the dev/null device.
I have zero patience for 'social networking' whores. it sucks I have to spend the first 5 minutes on any new site filtering and adding block entries to ABP and noscript but since its war, this is what we have to do to tame the (already messed up) internet.
and if a site can't show its conent in the first few minutes of my configuring filters, I just leave and never come back.
soon, slash will be that way for me. I've been here for a long time, but I can see it will end, probably this year.
fuck you, dice. just fuck you. sad how it ends like this.
Re: (Score:2)
Re: (Score:1)
Re: (Score:1)
Fucking stupid Share icon on every godddam article. Some MBA thought of this, didn't they?
WTF??? (Score:5, Insightful)
They want more money (Score:2)
Re: (Score:3)
In other words, mindless rubes and hipsters. Soon they'll rename Slashdot iDot or iSlash, of iTwitfuckhamshitprick. It's going to get so bad even APK is going to walk, and then where will I get my hosts file from?
Re: (Score:3)
Re: (Score:1)
Maybe they want to add unicode support?
HAHAHA The joke's on them.
Re: (Score:2)
Because if the news article (and publication) isn't trending on social media, advertisers don't consider it relevant. Advertising-Media industry inbred to the point that likes/re-tweets is universally seen as a measure of relevance.
In
Re: (Score:2)
>where big % of community signed out of social media for various valid reasons.
EXACTLY and it amazes most if not all of us that Dice could fundamentally misunderstand the user base so badly.
I can just imagine the discussions -- hey let's buy Slashdot, it's Facebook for tech nerds!
Custom built data centers (Score:5, Funny)
Re: (Score:2)
Is "pulling back the veil" some sort of sodomy euphemism?
bullshit (Score:1, Offtopic)
"we could not buy, for any price, a data-center network that would meet the requirements of our distributed systems,"
bullshit. if you told a company that specialized in making toothpicks that you would give them a trillion dollars to build a datacenter with XYZ, they would make it happen. did you offer someone a trillion dollars?
Re: (Score:3)
Libraries of Congress (Score:5, Interesting)
So, how many Libraries of Congress is that anyways? ... oh wait ... the Google blog post (ya'know, the actual artist, not the article talking about the article which was linked from the summary) actually states!
"Our current generation — Jupiter fabrics — can deliver more than 1 Petabit/sec of total bisection bandwidth. To put this in perspective, such capacity would be enough for 100,000 servers to exchange information at 10Gb/s each, enough to read the entire scanned contents of the Library of Congress in less than 1/10th of a second." = Source: http://googlecloudplatform.blo... [blogspot.com]
Re: (Score:2)
Except that there is no agreement on how much data the Library of Congress actually has.
"Though some sources have suggested that 10 terabytes represents the total quantity of data stored at the Library of Congress, this is a significant underestimate, given that the Web Archiving program had by itself collected 525 terabytes of data as of July 2014. A slide from a September 2012 presentation by a Library of Congress storage engineer furthermore noted institutional storage capacity in excess of 27 petabytes,