Stupid Data Center Tricks 305
jcatcw writes "A university network is brought down when two network cables are plugged into the wrong hub. An employee is injured after an ill-timed entry into a data center. Overheated systems are shut down by a thermostat setting changed from Fahrenheit to Celsius. And, of course, Big Red Buttons. These are just a few of the data center disasters caused by human folly."
Re:bad article is bad (Score:1, Informative)
Let me help out a bit
Printable Version [computerworld.com]
Re:Network meltdown due to hub cross-connects (Score:4, Informative)
This should work quite OK with hubs. A hub, after all, sends the packet to every port except the one where it came from. So two hubs in a loop should just forward the same packet back and forth all the time.
Re:Network meltdown due to hub cross-connects (Score:5, Informative)
Oh yes, it works quite well for sabotaging a network.
It used to be a constant issue at LAN parties where "pranksters" would do it before going to sleep... Usually we never found them but when we did we flogged them with cat5 cables stripped of insulation :p
Re:Not using Cisco ACLs (Score:5, Informative)
Hours?
You get something on the network which has an IP from the offending DHCP server, use ARP to establish what that DHCP servers' MAC address is then lookup the switches' own tables to figure out which port that MAC is plugged into and switch that port off and wait for the equipment owner to start complaining. Takes about 3-5 minutes to do by hand, and some switches can do it automatically.
Re:bad article is bad (Score:3, Informative)
Re:Quad Graphics 2000 (Score:1, Informative)
Someone needs a Molly-guard
Re:Quad Graphics 2000 (Score:4, Informative)
Hmm, if only someone could invent some kind of cover [wiktionary.org] to prevent accidental use...
I think a compounding issue is that the facilities guy (or higher up) is a cheapass.
Re:Not using Cisco ACLs (Score:2, Informative)
Cisco switches have a wonderful feature called dhcp snooping.
ip dhcp snooping
Followed by
ip dhcp snooping trust
on your port that supplies DHCP to the network. This ensures that only the trusted port can hand out dhcp addresses, and as a bonus, the switch tells you which MAC has which IP.
show ip dhcp snooping binding
Re:Network meltdown due to hub cross-connects (Score:3, Informative)
Can this really happen easily? I thought for really ugly things to happen, you need to have switches (without working STP, that is).
Spanning tree can not deal with the situation where there is a loop on a single port, which you can do easily by attaching a consumer grade switch. There are various workarounds (such as BPDU protection) but they aren't standard, and require manual configuration. Once your network gets big enough, you probably can't afford not to use them, though.
Re:Power strips (with on/off buttons) are bad (Score:2, Informative)
The power switch on an extension strip is not a surge protector. The surge protection mechanism is usually an internal varistor. It's safe to disable the switch mechanism.
Don't disable the breaker or fuse, however. That's suicidal.
Big red button: a true story (Score:3, Informative)
My mother, who is a database admin for a county office (and has been for a long time), was getting a tour of a brand new mainframe server in the basement of her department's building back in the early 80's. At some point during the tour a large red button was pointed out that controlled the water-free fire suppression system. When pressed it activated a countdown safety timer that could be deactivated when the button was pulled back out.
Always wanting to try things for herself, she went to the red button at the end of the tour and pressed it. No timer was activated, instead a noticeable shutting down sound was heard as the buzzing of the mainframe died down. She accidentally hit the manual power-off button for the mainframe which was situated very close to the fire suppression button and happened to look similar.
All the IT staff of that building got to go home early that day because the mainframe took several hours to reboot and it was already lunch. She was very embarrassed and I have heard that story many times.
Re:bad article is bad (Score:3, Informative)
To this day most cheap switches still can't handle a network cable with the 2 ends plugged in to the same switch. As a telco company technician I can't count the number of times I've solved someone's Internet connectivity problem by unplugging said cable. (I sort of understand it when there's a big mess of cables and you can't see where they all go, but I've also seen some really ridiculous ones where the troublesome cable is less than a foot long and therefore extremely obviously out of place!)
And before someone says it, yes I'm positive these are switches and not hubs. (I don't think I've actually seen a hub in use in a couple of years now... can you still even buy them?)
Now I hope that high end switch gear is better, but I have to admit that I haven't tried the experiment to find out (most high end switch gear I have access to is in use for mission critical stuff, no matter how remote the chances of actually taking it down, I'm not going to try the experiment)
Re:Network meltdown due to hub cross-connects (Score:3, Informative)
There is such a thing as a Layer 3 switch. They have routing functionality built-in, mostly to reduce latency for inter-vlan routing across a single switch. Cisco makes devices called Layer 3 switches, which are different from routers.
Re:Not using Cisco ACLs (Score:5, Informative)
Cisco switches have a wonderful feature called dhcp snooping.
Not supported on many of the lower end Cisco edge switches. It believe it also interferes with DHCP relaying.
Another great tool is "ip verify source vlan dhcp-snooping
" which can be used to block traffic from IPs/macs that did not obtain their IP from the DHCP server. This nicely prevents users from statically assigning addresses and/or spoofing their mac address.
There is. Tubes. (possibly before you were born) (Score:3, Informative)
Re:Network meltdown due to hub cross-connects (Score:3, Informative)
The physical difference is pretty much the key. The Layer 3 switch will have a bunch of Ethernet ports, but generally no serial ports (other than the console and auxiliary, of course). The layer 3 switch tends to push most of the work logic off onto ASICs rather than doing it in software on CPU time, too. That way you don't suffer much performance loss when routing between VLANs, but you wouldn't put on at a WAN uplink or network border.
Re:Know your colo contracts (Score:2, Informative)
Not only was the notion of cleaning the cable end bizarre -- what, wipe it on his t-shirt? -- and never fully explained,
There are in fact, standard procedures for cleaning fibre optic cable. [cisco.com]
Re:Know your colo contracts (Score:3, Informative)
After the usual confusion it was finally determined that one of the ISP's staff had "noticed a cable not quite seated" while working on the data center floor. He had apparently followed a "standard procedure" to remove and clean the cable before plugging it back in. It was a fiber cable and he managed to plug it back in wrong (transposed connectors on a fiber cable). Not only was the notion of cleaning the cable end bizarre -- what, wipe it on his t-shirt? -- and never fully explained, but there was no followup check to find out what that cable was for and whether it still worked. It didn't, for nearly a week.
Actually there's nothing odd about cleaning a fiber connection at all and it is a very exacting process (see link below). Apparently exacting in this case just didn't include re-inserting the ends in the right holes.
Inspection and Cleaning Procedures for Fiber-Optic Connections
http://www.cisco.com/en/US/tech/tk482/tk876/technologies_white_paper09186a0080254eba.shtml [cisco.com]
Re:Know your colo contracts (Score:3, Informative)
That's what I was getting at -- it's not as if it's a simple case of blowing on the end to clear out some fluff. Detailed procedures, including not least unplugging the other end of said cable to make sure it's unlit, which would include finding said other end. And likely go and get various the items required for the cleaning procedure. Which would add up at least to a conversation or two, and perhaps one with us the customer discussing the topic. I'm not disagreeing with cleaning of fiber cables sometimes being necessary, but I didn't for a moment believe all that had actually gone on.
Re:Ah, the memories! And lessons, too. (Score:3, Informative)
but only on the drives which were oriented north-south; those oriented east-west were not affected. So came the directive that all drives, henceforth, needed to be oriented north-south.
That seems counter-productive. They were oriented into the less optimal position?
Yes, I blew that one... Oops! But let me take this opportunity to point out something that I realized only after posting the GP post... That I was able to deduce the problem I had with the PBX, because I applied what I learned from the situation with the cleaning staff using a slot on a rack's outlet strip to plug in their vacuum cleaner.
IOW, although some of these stories seem funny in retrospect, they can also prove to be great learning opportunities, too! I'm looking forward to reading the other posts in this thread. I should probably head over to the "daily wtf" web site, again, too.
Re:Network meltdown due to hub cross-connects (Score:5, Informative)
I'm CCNP, taking my CCIE lab next month, I'll give this a shot.
Yes, the "cow goes moo" level definitions you get are "hub = L1, switch = L2, router = L3" but the reality is more complex.
A hub is essentially a multi-port repeater. It just takes data in on one port and spews it out all the others.
A switch is a device that uses hardware (not CPU/software) to consult a simple lookup table which tells it which port(s) to forward the data, and does so very fast (if not always wire-speed). Think like the GPU/graphics card in your PC. Something specific super fast.
A router is a device that understands network hierarchy/topology (in the case of IP, this is mainly about subnetting, but there are plenty of other routed protocols) and can traverse that hierarchy/topology to determine the next hop towards a destination.
Now, because of the protocol addressing in Ethernet and IP, these lend themselves easily to hub/switch/router = L1/L2/L3, but they're not really defined that way.
These days, most Cisco switches (3560, 3750, 6500, etc) run IOS, the software which can do routing, and which uses CEF. CEF in a nutshell takes the routing table (which would best be represented as a tree) and compiles it into a "FIB", which is essentially a flat lookup-table version of that same (layer 3, IP) table. It also caches a copy of the L2 header that the router needs to forward an L3 packet. The hardware (ASICs) in the switches hold this FIB, and thus allow them to "switch" IP/L3 packets at fast rates and without CPU intervention, thus making them still "switches", even if they run a routing protocol and build a routing table.
Meanwhile, when Cisco refers to a "router" in marketing terms, they're talking about a device with a (relatively) powerful CPU, which can not only perform actual routing, but also usually more CPU-intensive inter-network tasks like Netflow and NBAR.
Magic/More Magic (Score:3, Informative)
I can't believe no one's posted Guy Steele's Magic/More Magic story, yet:
http://everything2.com/user/Accipiter/writeups/Magic [everything2.com]
Re:Network meltdown due to hub cross-connects (Score:1, Informative)
a hub can also be a switch. I have worked with people who referred to both switches and repeaters as hubs
A repeater is a two-port device, traditionally they are half-duplex but you can find full-duplex versions. The only real use for them is to extend the length of an ethernet span- most ethernet networks don't need repeaters anymore, so other than some special commercial-grade ethernet extenders they're pretty hard to find.
A hub is simply a multi-port repeater. Also traditionally half-duplex, although you can find them in full duplex now-a-days. Data in one port echos out all other ports. Which makes a mess of your network really fast as you add machines to it. These days they are primarily used as a "poor man's" port-mirror or as paperweights. Most of the ones for sale in retail stores are going to still be a 10 or a 10/100 but in half duplex. Usually if you want full duplex you'll have to order online.
A switch is a device which will only send the data to the correct port (unless you have another port mirroring it, in which case both get it). Most switches off the shelf are at least 10/100 full duplex, and it's pretty common to see them in 10/100/1000 full duplex these days. And they usually aren't much more expensive than a hub.. so most networks don't bother with hubs at all anymore.
So you can make a switch operate as a hub if you want, but you can't make a hub act like a switch.
In addition, managed switches these days often have layer 3 or higher functions built in, so what you get off the shelf might not really be a 'pure' switch. The line between switches and routers is certainly starting to blur these days.
Cheap deep-packet inspection (using an old hub and Wireshark) ?
If it's an old hub, see my above point regarding half vs. full duplex. In any event, if you chip out an extra $10 or $15 bucks over a hub you can get a managed switch with port mirroring capability, which will work a lot better and give much more fine-grained control over what traffic you want to sniff. I'm talking maybe 50 bucks max (quick google search shows some cheap full-duplex 10/100 managed switches with mirroring for under $45US)
DISCLAIMER:
It's been a while since I saw the CCNA exam, but I think Cisco officially considers the difference between a switch and a hub to be that the hub is half-duplex repeater, and switches are full-duplex single-port forwarding devices. Just be aware that in the real world there is a lot more gray area than on the Cisco exams.
Re:Network meltdown due to hub cross-connects (Score:2, Informative)
Here, let me google this for you (Score:2, Informative)
Why do people type "*nix" instead of spelling it out?
http://en.wikipedia.org/wiki/*nix [wikipedia.org]