How To Build a 100,000-Port Ethernet Switch

BobB-nw writes "University of California at San Diego researchers Tuesday are presenting a paper (PDF) describing software that they say could make data center networks massively scalable. The researchers say their PortLand software will enable Layer 2 data center network fabrics scalable to 100,000 ports and beyond; they have a prototype running at the school's Department of Computer Science and Engineering's Jacobs School of Engineering. 'With PortLand, we came up with a set of algorithms and protocols that combine the best of layer 2 and layer 3 network fabrics,' said Amin Vahdat, a computer science professor at UC San Diego. 'Today, the largest data centers contain over 100,000 servers. Ideally, we would like to have the flexibility to run any application on any server while minimizing the amount of required network configuration and state... We are working toward a network that administrators can think of as one massive 100,000-port switch seamlessly serving over one million virtual endpoints.'"

It's all about address management

[Animats]

The paper is about adding a layer of addressing so that IP and Ethernet addresses can be moved from one machine to another as instances of virtual machines are migrated around. It's not about the problems of physically building a very large switch. The switch components are mostly stock items.

Re:You still need isolation

[amorsen]

What do you do about broadcast storms?

In the paper they detail how they handle ARP. All other broadcasts you can get away with dropping these days; use multicast instead. (Yes, that will break NETBIOS broadcast name lookups. So sad.)

How do you prevent some clown from anywhere in that 100,000 machine cloud from poaching another machine's IP address (either maliciously or by an accidental typo)?

That is a solved problem if you use decent switches. You can apply pretty much any policy you like.

Re:You mean

[Anonymous Coward]

33,334 = 100,000 / 3 = 4 port hubs, 3 ports going to servers and one port daisy chaining to the next switch.

Re:How many LEDs is that? How much power in LEDs?

[hhedeshian]

There is only one reason that I can think of for driving at that power: temperature. This doesn't really apply to switches because they're going to (hopefully) stay in air conditioned rooms, however: Once you get past 40C the effective brightness of the LED starts to approach 0 very rapidly. This is critical for equipment spec'd to run in industrial conditions. Because LEDs are so non-linear, designing a product with wide temp ranges with LEDs can be a real PITA. There is a good explanation here [marktechopto.com].

Re:Anyone remember Cabletron?

[Anonymous Coward]

USC has this sort of single domain campus-wide Cabletron network. It's a disaster, violates RFC894 wrt ARP behavior, and is one of the reasons (the other being low-skill cable monkeys) faculty can only get 10 Mbps ports. 100 Mbps ports are too expensive on that technology.

100,000-port ethernet is a problem not a solution.

Re:You still need isolation

[guitaristx]

A no-broadcast policy breaks Wake-on-LAN.

Re:Oh no...

[TheThiefMaster]

If you have the tools it's possible to crimp one plug to both ends of a loop of wire, so that the port's own send and receive lines are joined. This confuses a router even more than a loop between two ports.

Have you even read the proposal?!

[jeko]

They're not reducing complexity. They're proposing sandwiching another layer between two and three. It's not going to make things easier to design and troubleshoot. It's going to end up causing more trouble than it's worth. The only people who like this idea are salesguys like you who will have a new buzzword to sell.

But hey, by all means, implement this scheme. You're going to end up needing twice the network engineers you do now. The network explosions it will cause will be epic, the stuff of legend like Mt. St. Helens.

And for the love of Mike, I'm currently working 60-70 hours a week. We're not the Maytag repairmen. Most of us would LOVE to find a better way to do things. I have no doubt that 100 years from now, computer networking will make current schemes look slow and stupid. But those future protocols will still need to connect to the node -- layer one, identify the node -- layer two, and group the nodes together to make them easier to address -- layer three.

Look, I have no doubt you spend your week with your SE wildly gesticulating at you and shouting. I know by the time those frantic shouts get through your ears, it sounds like Charlie Brown's schoolteacher.

Show him some patience. He's trying to wedge some understanding between your ears.

He's not having much luck, apparently.