Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×
Networking IT Technology

10GbE: What the Heck Took So Long? 295

Posted by Soulskill from the i-blame-the-schools dept.
storagedude writes "10 Gigabit Ethernet may finally be catching on, some six years later than many predicted. So why did it take so long? Henry Newman offers a few reasons: 10GbE and PCIe 2 were a very promising combination when they appeared in 2007, but the Great Recession hit soon after and IT departments were dumping hardware rather than buying more. The final missing piece is finally arriving: 10GbE support on motherboards. 'What 10 GbE needs to become a commodity is exactly what 1 GbE got and what Fibre Channel failed to get: support on every motherboard,' writes Newman. 'The current landscape looks promising. 10 GbE is starting to appear on motherboards from every major server vendor, and I suspect that in just a few years, we'll start to see it on home PC boards, with the price dropping from the double digits to single digits, and then even down to cents.'"
This discussion has been archived. No new comments can be posted.

10GbE: What the Heck Took So Long? More

10GbE: What the Heck Took So Long?

Comments Filter:

  • Meanwhile (Score:2, Informative)

    by Anonymous Coward

    Everyone's still running off of ancient Cat3 wiring laid down when telephones were still analog.

    • Re: (Score:2)

      by lgw ( 121541 )

      Sounds like my home network may jump from 1Gb to 10Gb sooner than I expected, but it's still behind 3Mb DSL as my only non-Comcast option. Yay?

      • Re: (Score:2, Insightful)

        by homey of my owney ( 975234 )
        What the hell do you do at home that would require a 10GbE network?

        • stream hd video to multiple nodes frome a network file server

          • Depending on the level of compression a full HD (1080p) stream requires between 400KBytes/sec and ~2 MBytes/sec of bandwidth. That is, approximately 4MBits-20MBits.

            Needless to say, even 100MBit ethernet has no problem with a couple of those, let alone existing 1-gigabit ethernets.

            At 2160p (which is what people call 4K, for 3840x2160), perhaps ~1 MByte/sec to ~5 MBytes/sec depending on the level of compression and the complexity of the video. That is, somewhere north of 50 MBits on the top end. Despite ha

          • Ok #1 who does that? I mean that is not a very "home user" application in general. However #2 is gig is plenty for that. 1920x1080 24/30fps AVCHD PH video is 24mbps max. Blu-rays can in theory be 50mbps (between audio and video) mostly for MPEG-2 though in practice it is usually more like 25mbps AVC. Youtube is 6mbps for 1920x1080.

            So even with the max Blu-ray rate you are good for two streams. Realistically you can do 4 streams at most data rates. Even when 4k stuff starts to happen, it'll be fine to do one

            • Ok #1 who does that? I mean that is not a very "home user" application in general.

              you mean you have never heard of a home with people watching television in different rooms? at my home there are tiem when everyone is watching a different show/movie and using network resources with their portable devices such as laptops and tablets so it is not that hard to swamp a 1 gig nic.

              • If people are running these TVs in different rooms, you only need the really fast uplink from the media center to the switch, its probably a ton cheaper to simply have 2 NICs in the server. 10GbE runs about $200 per port on the switch, and a ton more for NICs.

        • Re: (Score:2)

          by lgw ( 121541 )

          Until sustained R/W speed on disk passed 10MB/s, I had no use for 1 GbE either! But it looks like it won't be too long before the network is back to being the bottleneck on network file copies/backups again, even on simple non-RAID volumes.

      • The current high-cost item is the 10GbE switch. Those things are way too expensive (10+K range, and the plus goes way up)

        Also, for flexibility, you want SFP+ ports and adapters for each port. None of those are cheap.

        • Re: (Score:3)

          by lgw ( 121541 )

          That's been true at this point at each jump in speeds (well, other than the details of the connection). The Ethernet chip-on-motherboard heralds the price fall on the switch - at the scale the 10GbE chips will soon be made, their price will fall (and thus the price of port-specific electronics in the switch will fall too), and then reasonably-priced unmanaged switches from low-end vendors follow soon after.

    • Re: (Score:2)

      by hurfy ( 735314 )

      Parent may be flamebait (atm) but i found it terribly funny.....i just installed an analog phone system from the mid-80's. The phone rings, we answer it. I didn't feel the need to buy a new shiny to do that. Spent $25 (dsl splitters) to set up a 3 line and 13 handset system and only used half the system :)

      Personally I don't have a real need for this, but seems the logical progression. I know lots of you do move mountains of stuff locally at home and/or work.

  • I think its a combo of the crappy economy, but then again maybe the need for wide adaptation just wasn't there. I would think it is like any other thing, if the demand was there the supply would have ramped up and the costs would have gone down.

    • Re:The real reason (Score:5, Insightful)

      by redmid17 ( 1217076 ) on Friday June 07, 2013 @04:44PM (#43941117)
      Biggest reason I can remember from when we were looking at upgrading SAN and LAN equipment in our data center was the price/performance point. We didn't need 10 GbE performance yet and the price was pretty far above what we were using. That was 3 years ago though, so I'd have to poke around some of the newer equipment to see if we have any boxes with it. I just took a gander through the HP and Dell offerings and it's not even an option on anything but the top tier equipment. I think that pretty much explains the situation itself.

      • And if you've ever looked at a NIC, you can see why. You get a modern gig server class NIC and it has this tiny little ASIC on it that does everything and draws less than a watt. Heck it'll probably drive two ports, if the hardware is on it. Then you get a 10gig NIC and it has a much larger ASIC with a big heatsink on it, and perhaps another chip as well. Guess what? That extra silicon cost extra money, as well as all the other related shit. And it just gets more and more expensive as you want more ports, l

  • Cost (Score:3, Insightful)

    by Anonymous Coward on Friday June 07, 2013 @04:43PM (#43941099)

    10GE Motherboards are still pointless when 10G routers & switches are still way too expensive.

    • It's a case of demand. There's no demand for those routers and switches because motherboards don't have 10GbE ports on them. Motherboards don't have 10GbE on them because there's no cheap routers or switches. Something has to give eventually and the motherboard probably makes the most sense to give in first.

    • Chicken and Egg, Bob. If I have a bunch of devices held back only by a few switches that can easily be replaced, the switches, when they drop a little in price, are getting replaced.

      It's totally different when I need to rip out every single component, down to the wiring in the walls, to upgrade the network.

    • 10GE Motherboards are still pointless when 10G routers & switches are still way too expensive.

      Absolutely true. You can get a single-port 10Gb card that uses Cat6 cabling for less than $300, but the cheapest switch with more than eight 10Gb ports is around $8000. You can piece together a switch with 6-8 10Gb ports (using modules) for around $4000.

      So, the reality is that you will pay 1x-3x the cost of the 10Gb NIC for a port to plug it into. Although that is less than the relative cost per port for high-end 1Gb managed switches, that's because the cost of a 1Gb NIC is basically pennies.

      • Well, if you are ok with going totally no-frills, you can get a 8*10G switch for under 800€ from netgear:

        http://direkt.jacob-computer.de/_artnr_1491948.html?ref=103 [jacob-computer.de]

        • Well, if you are ok with going totally no-frills, you can get a 8*10G switch for under 800€ from netgear:

          Yes, I am, and thanks for the pointer.

          I have two dual port 10Gb cards in the machine that is my SAN so that I can connect to 4 servers back-to-back, and the switch would allow me to replace them with single port cards and still have failover. I could then sell the two dual port cards for almost the cost of the switch.

  • My idea of the perfect cable:

    Four strands, two copper, two fiber.
    The two fiber strands enable redundancy (ring topology all the way to the end-point);
    The two copper strands for being able to provide power to devices.

    That's it. That's all that's needed.

    • Re:My idea of the perfect cable (Score:4, Insightful)

      by D1G1T ( 1136467 ) on Friday June 07, 2013 @04:59PM (#43941197)
      What you describe exists. It's not uncommonly used for IP cameras outside the 100m limit of TP Ethernet (on perimeter fences, etc.). The problem with fibre is that it's a bitch to terminate compared to copper, and therefore quite a bit more expensive to install on a large scale. Fibre still only makes sense when you need the long cable runs.

      • Re: (Score:2)

        by Nethead ( 1563 )

        Also a lot of big box stores. Loews stores are mostly fiber and they just did a huge upgrade from 10Mb/s to 1Gb/s last year.

        Side note distance story:
        Had a trouble ticked for a Home Depot where we found that one of the printers up front was wired all the way back to the data center in the opposite corner, about 550 feet. Out temporary fix was to drop the port down to 10Mb/s until we could get a lift in to run a line to the IDF by the printer.

    • Re: (Score:3)

      by jon3k ( 691256 )
      For what? What's the application? Way too expensive to run to my IP Phone or Desktop PC (could juse use fiber or copper, why both?). Unnecessary in the datacenter (we don't need PoE). What's the use case?

      • For what? What's the application? Way too expensive to run to my IP Phone or Desktop PC (could juse use fiber or copper, why both?). Unnecessary in the datacenter (we don't need PoE). What's the use case?

        Purists demand that One Cable Rule Them All. This naturally leads to a One True Cable that is wildly overengineered and expensive for the keyboards and mice and IP phones of the world, while still failing to support common, but in some way unusually demanding, edge scenarios.

        • Re: (Score:2)

          by Nethead ( 1563 )

          But you have to admit that Cat5e with RJ45s on the end sure came close to that "one true cable" for a long time.

    • . . . along with fusion splicers dropping from thousands to under $200

  • Commodity (Score:4, Insightful)

    by rijrunner ( 263757 ) on Friday June 07, 2013 @04:55PM (#43941163)

    Of course its growth was going to be lower.

    The primary use of 10GbE is virtualization. The use of network cards are a function of the number of chassis, not the number of hosts. Numerically, 10GbE is not 10 1GbE cards. You can split the 10GbE between a lot of hosts. You can easily double, triple, or even quadruple that to making that 10 GbE card the equivalent of 1 GbE cards on 40 servers, depending of their load and use. Instead of buying 40 servers and associated cards, you're buying one larger chassis with larger pipes. In a large farm environment, and it makes sense.

    Throw in the fact that network is only as fast as its narrowest choke point, there is no reason to put in a 10 GbE card behind a 7MB DSL connection.

    What 10GbE needs to become a commodity is a) end of any data caps, b) data to put down that pipe, and c) a pipe that can handle it.

    Show me fiber to my door and then, it will be a commodity.
     

    • Re: (Score:2)

      by jon3k ( 691256 )
      That logic doesn't really hold. We moved from 100Mb Fast-E to 1GbE and residential broadband speeds had nothing to do with it.

      • Most people use issued DSL or Cable modems for networking. Commodity use is directly tied to broadband. And those modems shipped based on the tech supported by the ISP. Switching to 1GbE on the switch side tracks to when companies implemented DOCSIS 2.0. When they move to DOCSIS 3.0, then you'll see an upgrade in networking layer in residential use.

    • there is no reason to put in a 10 GbE card behind a 7MB DSL connection.

      And yet, there was apparently a reason to put GbE cards behind that same 7Mbit DSL connection, or else we'd still be on 100BaseTx.

  • The bottlenecks are elsewhere (Score:4, Insightful)

    by AdamHaun ( 43173 ) on Friday June 07, 2013 @04:58PM (#43941191) Journal

    Ten gigabits per second is 1,250 megabytes per second. High-end consumer SSDs are advertising ~500 MB/sec. A single PCIe 2.0 lane is 500 MB/sec. Then there's your upstream internet connection, which won't be more than 12.5 MB/sec (100 megabits/sec), much less a hundred times that. I guess you could feed 10GbE from DDR3 RAM through a multi-lane PCIe connection, assuming your DMA and bus bridging are fast enough...

    I'm sure a data center could make use of 10GbE, but I don't think consumer hardware will benefit even a few years from now. Seems like an obvious place to save some money in a motherboard design.

    • Re: (Score:2)

      by jon3k ( 691256 )
      We're years (probably a decade+) away from any significant demand (read: more than low single digit percentage) for 10Gb for personal use.

    • Re:The bottlenecks are elsewhere (Score:5, Insightful)

      by Guspaz ( 556486 ) on Friday June 07, 2013 @05:29PM (#43941475)

      You're looking at things backwards. If you've got a 500 MB/s SSD, then you shouldn't look at 10GigE and say "that's twice as fast as I need, it's useless". You should look at the existing GigE and say "my SSD is four times faster, one gigabit is too slow"...

      Even a cheap commodity magnetic hard disk can saturate a gigabit network today. The fact that lots of computers use solid state drives only made that problem worse. Transferring files between computers on a typical home network these days, I think the one gigabit per second network limitation is going to be the bottleneck for many people.

      • Re:The bottlenecks are elsewhere (Score:4, Insightful)

        by AdamHaun ( 43173 ) on Friday June 07, 2013 @06:07PM (#43941805) Journal

        You're looking at things backwards. If you've got a 500 MB/s SSD, then you shouldn't look at 10GigE and say "that's twice as fast as I need, it's useless". You should look at the existing GigE and say "my SSD is four times faster, one gigabit is too slow"...

        If I want to copy tons of large, sequentially-read files every day, maybe. (Assuming that 500 MB/sec actually hits the wire instead of bottlenecking in the network stack.) But I'm not sure why I would do that. If I have a file server, my big files are already there. If I have a media server, I can already stream because even raw Blu-ray is less than 100 Mbps. If I'm working on huge datasets, it's faster to store them locally. If I really need to transfer tons of data back and forth all the time, I'm probably not a typical home network user. ;-)

        • Eventually it's just going to be cheaper to include the new standard instead of the old due to manufacturing efficiency, and you'll get it whether you need it or not.

      • Re: (Score:3)

        by Kjella ( 173770 )

        Transferring files between computers on a typical home network these days, I think the one gigabit per second network limitation is going to be the bottleneck for many people.

        Real world calling, most home networks have gone wireless and most use laptops, tablets or other portable devices that don't get plugged in more than they need to. Even if you have a family server or one of the kids is a gamer with a desktop it still won't go any faster. The GigE cap is only if you need to move huge amounts of data between two wired - or at least plugged in for the occasion - boxes in the same house, which is quite rare. That anybody feels speed is a limitation is rarer still, cables are mo

      • You should look at the existing GigE and say "my SSD is four times faster, one gigabit is too slow"...

        Actually, I'd say "two bonded/teamed/aggregated GbE NICs is good enough". That's half the throughput of your SSD, but you're probably not maxing out your SSD constantly, and you've got headroom for plenty of local disk I/O while you're at it. You could go for 4 bonded GbE NICs, and that'll cost far less than even a single 10GbE port.

        If we're talking about a SAN, sure, you probably want (multiple) 10GbE po

    • Re: (Score:2)

      by jgrahn ( 181062 )

      Ten gigabits per second is 1,250 megabytes per second. High-end consumer SSDs are advertising ~500 MB/sec. A single PCIe 2.0 lane is 500 MB/sec. Then there's your upstream internet connection, which won't be more than 12.5 MB/sec (100 megabits/sec), much less a hundred times that. I guess you could feed 10GbE from DDR3 RAM through a multi-lane PCIe connection, assuming your DMA and bus bridging are fast enough...

      More importantly, you can't make an IP stack consume or generate 10Gbit on any hardware I know of, even if the application is e.g. a TCP echo client or server where the payload gets minimal processing. The only use case is forwarding, in dedicated hardware, over 1Gbit links. 10Gbit is router technology, until CPUs are 5--10 times faster than today, i.e. forever.

      • Re: (Score:2)

        by adri ( 173121 )

        Netflix OpenConnect pushes 20GBit+ on a FreeBSD-9 base with nginx and SSDs. Over TCP. To internet connected destinations.

        Please re-evaluate your statement.

      • You are so wrong it isn't even funny.

        We're running app stacks at full line rate on 40GbE using today's hardware. A dual-socket sandy bridge server (I.e. HP DL380) has no problem driving that kind of bandwidth. Look up Intel DPDK or 6Windgate if you want to learn a thing or two.

        It's real, it works, and we're getting ready to start 100GbE testing.

    • For many things you do, you find 1gbit is enough. More doesn't really gain you anything. It is enough to stream even 4k compressed video, enough such that opening and saving most files is as fast as local access, enough that the speed of a webpage loading is not based on that link but something else.

      Every time we go up an order of magnitude, the next one will matter less. There will be fewer things that are bandwidth limited and as such less people that will care about the upgrade.

      As you say, 10gbit, or eve

    • I'm sure a data center could make use of 10GbE, but I don't think consumer hardware will benefit even a few years from now.

      10GbE would mean you could move your storage off your local machine to your NAS, since those remote disks would be as fast as the average local disk. There are a lot of uses for this, like saving money by only having programs/data on one set of disks, but still having very fast access.

      No, not every home user could benefit from this, but not every home user benefits from 1GbE, either.

    • So storage access is already 5x faster than 1GbE.

      Sounds to me like 10GbE is already overdue.

      For the cluster I develop for at work we have a 40GB infiniband LAN. For serious IT I'd skip 10GbE now and go to IB.

    • Seems like an obvious place to save some money in a motherboard design.

      Savings are only available right now. 10Mbit chips are actually more expensive now than 10/100. Older style cards even more so. It's all about economies of scale. Given enough years 10Gbit may become the standard and it may be too expensive to produce slower boards.

      • Re: (Score:2)

        by AdamHaun ( 43173 )

        Given enough years 10Gbit may become the standard and it may be too expensive to produce slower boards.

        That was kinda my point, although I didn't say it very well. I think 10GbE will become common on home systems when it's about the same price as 1GbE.

    • Re: (Score:2)

      by Bengie ( 1121981 )
      And memresistors are right around the corner and can run at main memory speeds. How long until 4GB/s cheap SSDs?

      • Re: (Score:2)

        by AdamHaun ( 43173 )

        And memresistors are right around the corner and can run at main memory speeds.

        That will be great, but I think "right around the corner" is a little ambitious. It takes a long time to implement a new memory technology at the scale needed for PC hard drives. I'd expect memristor USB drives long before SSDs.

        How long until 4GB/s cheap SSDs?

        My guess? Never. Shrinking flash makes reliability harder (fewer electrons on the floating gates). And manufacturers are already pushing TLC SSDs for density. Both of those affect read and write speeds. And again, you have to look at the overall picture. SATA3 is 600 MB/sec, so for

        • Re: (Score:2)

          by Bengie ( 1121981 )
          "Right around the corner" as HP and Hynix have already stated that they are currently in the process of replacing a large portion of their production for memresistors for later this year. They were planning to do memresistors early this year, but they realized that memresistors are so much better than FLASH and memory, that they're going to eat into their own markets, so the postponed for Q4 '13.

          They will be selling both DDR3 and SSDs by 2015.

    • One 6Gb SAS drive (defacto local and network standard in 2013 Datacenters) can do 3-600 MB/s per port (a good deal faster than older 6Gb SAS drives). It's pretty easy to saturate a 10Gb ethernet connection under the right conditions with the standard 2 ports found on an HP, DELL or IBM low end x86 solution.

  • Don't count on the price of 10gigE dropping to cents. Unlike gigE, 10gigE has really very little 'enterprise' competition technologies. Fibre channel, infiniband, etc. - if you want more than gigE speeds, it's going to cost you. Those were costly technologies then - but back then, they offered significantly more performance (and thus value) than gigE. With 10gigE, there is no financial incentive to drop costs.

    • Re: (Score:2)

      by Bengie ( 1121981 )
      Intel stated the 10Gb is trivial to implement using 22nm tech and expect to be integrated into low-end chipsets when it happens. The problem is 22nm is current tech and CPUs get priority. We need to wait a bit before 22nm trickles down to chipsets.

      • Re: (Score:2)

        by CAIMLAS ( 41445 )

        So in other words, this is probably 4-5 years off from chipset implementation - for Intel boards. This leaves all the other boards implementing Broadcom, Realtek, etc. out in the rain...

  • Most of my customers are still running 100base-T and see little reason to upgrade since their networks primarily exist to distribute Internet access. What took so long? Nobody seems to really want it. Slashdot crowd not withstanding.

  • I would argue that part of the issue is that 10GigE connections have limited use. Not that they're not useful, but at this point, with the amount of data we're moving around, most people aren't going to see a huge benefit over existing solutions. It's a little like why desktop computer sales have slowed in general: what people have now is kind of working "well enough".

    Of course, part of the problem is that a lot of what people are doing now is over the Internet, which means that you're bottlenecked by yo

  • Expensive (Score:3)

    by Dwedit ( 232252 ) on Friday June 07, 2013 @05:25PM (#43941429) Homepage

    The best reason I can think of not to buy a 10-gigabit Ethernet card is simple: The cheapest ones go for $351 on Newegg. Want an Ethernet switch to go with that? That will be $1036.

    So once again, the answer is simple, and it has to do with a dollar sign.

    Gigabit equipment got really cheap fairly quickly, but not so much for the 10-gigabit equipment.

  • Please stop talking like your desktop defines IT. 10 Gb ethernet has been around for years for Sun/Oracle servers, IBM servers, Cisco switches, storage arrays, etc. Hell, I could even get 10Gb for my Mac. It hadn't made it into the PC world yet due to office wiring to the desk still being Cat 5. It's hard enough to get 1 Gb connections for the general user.

  • So nice and fast (if you can afford it) (Score:3)

    by Sarusa ( 104047 ) on Friday June 07, 2013 @05:34PM (#43941519)

    We have some of these at work where we do have the need for moving massive volumes of data around. We can get about 99.6% of theoretical throughput in actual use, thanks to the hardware offloading and large frame support. Besides the 10x faster to start with, that's way above any efficiency we get from the 1 GbE ports, though I expect if 10 GbE went commodity you'd lose all the hardware support and you'd be back to 80-90% range.

    Note to sustain a data feed to one of these you at least need two SATA 6 gbps SSD drives in RAID0. On the receiving end we're not writing to disk, or you'd need ~3-4 RAIDed.

    In our case we're feeding 4 10GbE ports on the same machine and using a 10 SSD RAID0 to supply the data with some headroom (we don't care if we lose the data if one fails, these aren't the master copies). We're just using software RAID, but thanks to all the DMA and offloading the CPU usage is quite low.

    Now do I need this at home? Well, SSD speeds are far above the ~85 MB/sec 1GbE delivers, but so far the cost hasn't made it worth it. If I'm copying a gigabyte it takes 12 seconds, which I can live with.

  • Cisco and most other vendors have made 10Gb ports too expensive and/or don't have a backplane that can effectively support 10Gb across all the ports. This is pretty ridiculous given how cheap processors have gotten. Even when they do support it, the licensing and maintenance costs can be crazy.

    For that reason we're currently deploying several 1Gb connections to our VM servers through various switches (depending on costs per port, reliability needed and location).

    I've been hoping that late 2013 is when 10Gb

  • Cost and complexity (Score:3)

    by AaronW ( 33736 ) on Friday June 07, 2013 @08:11PM (#43942777) Homepage

    I think the main reason is cost. I have been working with 10Gbe for several years writing drivers for PHYs and MACs. I've worked with a number of PHYs and 10Gbe is a lot more complex. For example, the SFP+ cables and modules each have a serial EEPROM that contains parameters needed to program the PHY. It's not just a simple RJ45 CAT5/CAT6 cable. As someone who has worked in 10Gbe drivers there's a lot more complexity. With some PHYs I have to query the serial EEPROM to make changes based on things such as cable length and whether or not it's active or passive or if it's copper or optical. Distances over copper are also usually limited to much shorter distances unless active cabling is used.

    In terms of cost, a 1 meter copper cable is around $43 from www.cablesondemand.com. A 12 meter cable is $189. It's not like gigabit where you just plug in a CAT5 cable and go.

  • Well, my hard drive can't write data at 1250MB/s so why send it over the network that quickly? So...there's that.

  • Most people I know barely benefit from gigabit Ethernet. Most people I know are not running Exchange servers and huge file sharing projects on their LANs but hosting their data on their local PC and using their network for E-mail and the web.

    While 10-100Mbit made a huge difference to peoples' networking abilities, and going from that to gigabit helps with smooth transfers of larger files, there's still a lot of people running 100Mbit and quite happy with it because modern switches are pretty good at what t

Slashdot Top Deals

IOT trap -- core dumped

Close