Stories
Slash Boxes
Comments
Search

News for nerds, stuff that matters

Slashdot Log In

Log In

Create Account  |  Retrieve Password

Ethernet The Occasional Outsider

Posted by Zonk on Thu May 25, 2006 02:18 PM
from the popular-kid-gets-snubbed dept.
Communications Networking IT
coondoggie writes to mention an article at NetworkWorld about the outsider status of Ethernet in some high-speed data centers. From the article: "The latency of store-and-forward Ethernet technology is imperceptible for most LAN users -- in the low 100-millisec range. But in data centers, where CPUs may be sharing data in memory across different connected machines, the smallest hiccups can fail a process or botch data results. 'When you get into application-layer clustering, milliseconds of latency can have an impact on performance,' Garrison says. This forced many data center network designers to look beyond Ethernet for connectivity options."
+ -
story
This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Ethernet The Occasional Outsider 50 Comments More /

 Full
 Abbreviated
 Hidden
More
Loading... please wait.

  • Long Live! (Score:5, Funny)

    by Anonymous Coward on Thursday May 25 2006, @02:20PM (#15404153)
    Long Live the Token Ring!

    One Ring to rule them all

  • My idea: a vat of salt water & CAT5 (Score:5, Funny)

    by Anonymous Coward on Thursday May 25 2006, @02:23PM (#15404181)
    In our Data Center, we have a great big vat of steaming salt water and we drop one end of the cat5 cables from each server into the vat....those packets that can't figure out where they're going just drop to the bottom and die ...we have to drain this packet-goo out once a month. (but we do recycle it...we press it into CDs and sell them on Ebay)

    (Seriously, haven't people heard cut-through switches which just look at the first part of the header and switch based on that... store-and-forward switches are so "1990s")

    TDz.
    • that was my thought exactly (not the salt vat.. althought i like it)

      we have a small office ~20 computers and 3 servers.. and i refuse to buy switchs that can't do cut through.. store and forward is slow..and very memory entisive for switchs on high speed networks..
    • There are only TWO reasons to use Store & Forward.

      #1. You're running different speeds on the same switch (why?).

      #2. You really want to cut down on broadcast storms (just fix the real problem, okay?)

      Other than that, go for the speed! Full duplex!
      • People run different speeds on the same switch all the time, and for not necessarily poor reasons: If you have a SMB (in this case, that's small or medium business) with maybe one big fileserver, you don't need to run gigabit to everyone... You can run 100Mbps to the clients, and run gig to the switch only. Of course, since just about everything but laptops is coming with gig now (and probably some of them) this is becoming less valuable.
        • There's plenty of hardware out there that doesn't come gig-e equipped. Hell, I still deploy RS232 terminal concentrators at 10 megs now and then.
          • While that's true, most of the time those kind of devices would be happiest on their own subnet for security and management reasons - or at least, I'd be happiest with them there. Therefore they can live on different router interfaces, whether the router's from cisco, or a PC from fry's with linux on it. The only time it's really necessary to mix speeds on the same switch is when you have multiple clients accessing a resource and their aggregate speeds make it useful.

        • For performance, run the same speed. (Score:5, Interesting)

          by khasim (1285) <brandioch.conner@gmail.com> on Thursday May 25 2006, @03:34PM (#15404805)
          People run different speeds on the same switch all the time, and for not necessarily poor reasons: If you have a SMB (in this case, that's small or medium business) with maybe one big fileserver, you don't need to run gigabit to everyone...
          What's with the "need to"?

          I'm talking performance. Store & Forward hammers your performance. In my experience, you get better performance when you run the server at 100Mb full duplex (along with all the workstations) and use Cut Through than if you have the server on a Gb port, but run Store & Forward to your 100Mb workstations.
    • (Seriously, haven't people heard cut-through switches which just look at the first part of the header and switch based on that... store-and-forward switches are so "1990s")

      Even still - low 100 ms for store-and-forward ethernet switches? That seems really, really high. I would've said more like single milliseconds, which is still high, but it isn't 100 ms.

      I know from experience that I've used store-and-forward ethernet switches with much, much better latency than 100 ms.
  • The NSA's network sniffer, recently discovered at an AT&T broadband center, can only sniff up to 622MB [slashdot.org]. Sounds to me like if you use an InfiniBand switch, that would effectively make the output of the NSA's network sniffers complete gibberish.

  • 100ms ethernet latency? (Score:5, Informative)

    by victim (30647) on Thursday May 25 2006, @02:23PM (#15404183) Homepage
    I don't think I need to read anymore, well, I did verify that the number really appears in the article.
    This author does not understand the subject material.

    (I suppose you could deliberatly overload a switch enough to get this number, maybe, but that would be silly, and your switch would need 1.25Mbytes of packet cache.)
  • Ultra-low latency networking is a minor interest of mine, but one I've never had the chance to really pursue. Can anyone familiar with the landscape recommend some low-cost options for experimenting with this stuff? Or maybe just let me down gently. "No, Sammy, there are no low-cost options. And there's no Santa Claus."
    • Define low cost? Myrinet [myrinet.com] with less than 10 microsecond latency is normally considered to be the least expensive option. You can check their price lists, but an 8 port solution [myrinet.com] (with 8 HBA's) will set you back over $8k, not including the fiber.

      For some people, that's cheap. If not, sorry.

  • Not an Auspicious Start (Score:5, Informative)

    by Anonymous Coward on Thursday May 25 2006, @02:25PM (#15404203)
    From the article, three paragraphs in:
    "(By comparison, latency in standard Ethernet gear is measured in milliseconds, or one-millionth of a second, rather than nanoseconds, which are one-billionth of a second)"

    That would be one-thousandth, not millionth (aka micro second). Not a good start...

    • Well that and ethernet gear is measured in miliseconds? That doesn't seem useful. If I run a traceroute, the time is listed as "1ms" for all the internal hops. There are 5 internal hops, all ethernet. In my experience, all modern ethernet gear adds less than a millisecond of latency. Traceroute programs only report milliseconds because there's a useful measure for Internet traffic and anything under 1ms can be safely called "really fast" for normal work.

      Seems to me you'd need to measure ethernet gear in mic

  • When you get to many hops (Score:5, Funny)

    by with_him (815684) on Thursday May 25 2006, @02:25PM (#15404209)
    I just blame it on the ether-bunny.

  • Software design (Score:3, Interesting)

    by nuggz (69912) on Thursday May 25 2006, @02:26PM (#15404213) Homepage
    The origional post makes some comments that
    sharing memory ... the smallest hiccups can fail a process or botch data results.
    Sounds like bad design, or a known design trade off.
    Quite reasonable, when on a slow link, until I know better assume the data I have is correct, if it isn't throw it out and start over. Not wildly different than branch prediction or other approaches to this type of information.

    'When you get into application-layer clustering, milliseconds of latency can have an impact on performance,'
    Faster is faster, not really a shocking concept.
    • what it looks like to me is.. ok so they set something up using normal 100/1000 ethernet and then realized something was slow and that if they use gbic 30gb ports things run faster... can someone please sent them a cookie?

  • Did you mean "microseconds"? (Score:4, Interesting)

    by pla (258480) on Thursday May 25 2006, @02:26PM (#15404218) Journal
    The latency of store-and-forward Ethernet technology is imperceptible for most LAN users -- in the low 100-millisec range.

    I don't know what sort of switches you use, but my home LAN, with two hops (including one over a wireless bridge) through only slightly-above-lowest-end DLink hardware, I consistantly get under 1ms.



    When you get into application-layer clustering, milliseconds of latency can have an impact on performance

    Again, I get less than 1ms, singular.



    Now, I can appreciate that any latency slows down clustering, but the ranges given just don't make sense. Change that to "microseconds", and it would make more sense. But Ethernet can handle single-digit-ms latencies without breaking a sweat.
    • I wonder if his messed up numbers come from his mistaken belief that a millisecond is three orders of magnitude smaller than it is.
    • Sure, for an 8 port switch, where all the computers have a direct connection. Consider the issues involved for a router with a 128 machines all trying to cross-communicate. Or larger collections of computers that might need to use multiple sets of switches to span the entire system.

      On a Force10 switch, with 2 nodes on the same blade:
      tg-c844:~ # ping tg-c845
      PING tg-c845.ncsa.teragrid.org (141.142.57.161) from 141.142.57.160 : 56(84) bytes of data.
      64 bytes from tg-c845.ncsa.teragrid.org (141.142.57.161):

  • On my planet, a millisecond is a full thousandth of a second, not just one millionth.

    Oh, well. People tell me I'm just slow.

  • Ethernet's strength is it's flexiblity, not it's speed per se. It can handle changing network environments where hardware or software is added and removed continually, and you never know quite where the bandwith is most needed. You just plug it all in, and ethernet does a decent job of neotiating who gets to use the bandwidth.

    But it's never been a really high speed protocol. It's easy to beat, speed-wise, as long as you know what the network use looks like ahead of time.

    Which of course is a killer for m
      • You're right; the terminology there is weak. I mean that it usually can't get the full potential out of the underlying physical network that others can. 'High speed' is relative, of course, to whatever you are comparing it to.

        Basically, at a given level of tech, you should be able to build a network that is faster than an ethernet network at that level of tech. But it will be more complicated to set up and maintain.

  • No kidding (Score:5, Interesting)

    by ShakaUVM (157947) on Thursday May 25 2006, @02:46PM (#15404389) Homepage Journal
    Er, yeah. No kidding.

    When I was writing applications at the San Diego Supercomputer Center, latency between nodes was the single greatest obstacle to getting your CPUs to running at their full capacity. A CPU waiting to get its data is a useless CPU.

    Generally speaking, clusters who want high performance used something like Myrnet instead of ethernet. It's like the difference between consumer, prosumer, and professional products you see in, oh, every industry across the board.

    As a side note, how many parallel apps solve the latency issue is by overlapping their communication and computation phases, instead of having them in discrete phases, this can greatly reduce the time a CPU is idle.

    The KeLP kernel does overlapping automatically for you if you want: http://www-cse.ucsd.edu/groups/hpcl/scg/kelp.html [ucsd.edu]
  • The article's worth reading, if you're not already familiar with currently popular cluster interconnects, but the title of "Data center networks often exclude Ethernet" is totally bogus.

    I guess "Some Tiny Percentage of Data Centers use Something Faster than Ethernet in addition to Ethernet" didn't fit on the page.

  • --- malin.vidarlo.net ping statistics --- 15 packets transmitted, 15 received, 0% packet loss, time 14003ms rtt min/avg/max/mdev = 0.310/0.347/0.375/0.019 ms 2 hops, over 100Mb ethernet with a cheapass switch (8 port unmanaged hp). Seems like he got no grip on numbers...
  • Just had a quick ping to the beeb... via a wireless hop onto my ethernet network, two hops to my adsl router, then 6 hops around Nildram's network (ATM into their network then god knows, probably some form of gigabit ethernet) and a couple more hops to the bbc.

    Average latency is around 20ms.

    Now I know this isn't as plain as straight ethernet but I'd have guessed the latency if anything on ATM + the change from 802.11g to ethernet to atm to ethernet to whatever would have been worse.

    So either someone is usin

  • The worst post! (Score:3, Informative)

    by Anonymous Coward on Thursday May 25 2006, @03:22PM (#15404708)
    I wonder what's happening to slashdot. That's as bad as technical news can get. Ethernet latency -- 100ms?? Typical Ethernet latencies are around a few hundred microseconds. Even the ping round-trip time from my machine to google.com is about 20ms.

    $ ping google.com
    PING google.com (64.233.167.99) 56(84) bytes of data.
    64 bytes from 64.233.167.99: icmp_seq=1 ttl=241 time=20.1 ms
    64 bytes from 64.233.167.99: icmp_seq=2 ttl=241 time=19.6 ms
    64 bytes from 64.233.167.99: icmp_seq=3 ttl=241 time=19.5 ms

    What a shame that such a post is on the front page of slashdot! Someone please s/milli/micro.

  • Slashdot summary wrong, actual article is better (Score:4, Interesting)

    by m.dillon (147925) on Thursday May 25 2006, @03:45PM (#15404920) Homepage
    The slashdot summary is wrong. If you read the actual article the author has it mostly correct except for one comment near the end.

    Ethernet latency is about 100uS through a gigE switch, round-trip. A full-sized packet takes about 200uS (micro seconds), round-trip. Single-ended latency is about half of that.

    There are proprietary technologies that have much faster interconnects, such as the infiniband technology described in the article. But the article also mentions the roadblock that a proprietary technology respresents over a widely-vendored standard. The plain fact of the matter is that ethernet is so ridiculously cheap these days it makes more sense to solve the latency issue in software, for example by designing a better cache coherency management model and by designing better clustered applications, then it does with expensive proprietary hardware.

    -Matt

  • Tolkien ring (Score:4, Funny)

    by Shabazz Rabbinowitz (103670) on Thursday May 25 2006, @04:37PM (#15405394)
    I had recently considered using this Tolkien ring until I found out that deinstallation is very difficult. Something about having to take it to a smelter.

  • For those who don't understand... (Score:4, Informative)

    by bill_kress (99356) on Thursday May 25 2006, @05:21PM (#15405696)
    Most (all?) Ethernet hardware reads in an entire packet, looks at it, then sends it on to a destination. This makes building routers and switching hardware fairly easy but extremely slow.

    If you go to a high-speed network, what you get is a packet being forwarded as it's being read. By the time the first few bits are through the switch, it should be able to figure out the next hop and have the packet moving in that direction. Phone companies have huge problems with the delays in Ethernet. This is why the ATM protocol was invented, it's hard to use, awkward and not too graceful, but it can fly through a switching network like nobody's business.

    Ethernet is also extremely sloppy--Any switch along the way is allowed to throw a packet away and wait for the originator to resend causing a HUGE hiccupp in the communication stream (Most if not all routers do this whenever an address is not in it's forwarding table yet).

    IIRC the faster protocols see a "Routing" packet in the stream and set up forwarding hardware before getting the actual packet/stream, then wait until the end of the packet (or entire stream) to tear the route down again.

    Ethernet, however, due to it's simplicity is bridging the gaps. It's a pretty crappy protocol in general, but we keep throwing better, smarter hardware at it in an effort to brute-force it into the parameters we require. (I work for a company that makes Ethernet over fiber hardware, and have worked for companies based around ATM, SONET and other interesting solutions).

    I guess the point of the article was to remind a world that is coming to believe that ethernet is the end-all be-all of networking that it was always just the simplest hack available and therefore the easiest to deal with.

    Just like SNMP.
    • This is a NUMA (non-uniform memory access) cluster. Basicly a bunch of computers woring together that occasionally need to access the same data. If the last process to need that data happens to be on another computer, it needs ot be transfered. The trick to these clusters is writing software so that transfer need is minimal, and that the same data set stays on the same processor, to the best of your ability.
    • Maybe I should RTFA...
      Either that, or you should take the class [gatech.edu] that I took this past semester. There's a bunch of links to research papers and lecture slides about distributed shared memory (and other kinds of parallel/shared computing issues), if you care to read them.
    • I think the article just has a typo. 'Imperceptible' is definitely not how I'd describe 100ms latency on a switched LAN. It's also true that switches do not necessarily have to store and forward, and cut-through switching used to be a lot more popular. I believe it's probably less popular now because store and forward performance is more than adequate, and because it offers a handful of advantages (verify FCS before forwarding, for example).

    • Re:Channel Bonding (Score:5, Funny)

      by kjs3 (601225) on Thursday May 25 2006, @03:47PM (#15404943)
      So you have an environment with requirements totally unlike the ones described in the article and needing none of the solutions illustrated in the article. Hey...thanks for letting us know. Maybe the other million Slashdot users with environments irrelevant to the post can let us know what they have as well.

All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2009 Geeknet, Inc.