Stupid Data Center Tricks

jcatcw writes "A university network is brought down when two network cables are plugged into the wrong hub. An employee is injured after an ill-timed entry into a data center. Overheated systems are shut down by a thermostat setting changed from Fahrenheit to Celsius. And, of course, Big Red Buttons. These are just a few of the data center disasters caused by human folly."

Router Plugged Into Itself

[Anonymous Coward]

Where I work a couple years ago one of the non-technical people decided to plug a router into itself. Ended up bringing down the whole network for ~25 people in a company which depended on the Internet (Internet marketing company).

Unfortunately one of the tech guys figured it out literally as everyone was standing by the elevator waiting for it to take us home. We were that close to freedom :(

Re:bad article is bad

[Timex]

the summary text is, verbatim, the first part of the article.

It is my personal observation that this seems to be the best way to get anything on the front page: using the article text as the "summary". Isn't it nice to see that Slashdot submitters are so original in their writing skill? :D

Obligatory: The Etherkiller

[Anonymous Coward]

The Etherkiller [fiftythree.org]

Video

[AnonymousClown]

Here's a video of a tech worker explaining why these things happen. [youtube.com]

It's very disturbing and you'll see why these things happen.

Re:Quad Graphics 2000

[FictionPimp]

Well, where I work some maintenance genius decided that the location of the red button (near the entrance door) was too risky. They said people coming in the door could hit it while trying to turn on the lights.

Their solution? They moved it to behind the racks. So every time I bend down to move or check something I have to be conscious not to turn off the power to the entire room with my ass.

Re:I got a good one too!

[Yvan256]

192.168.x.x? That's amazing. I've got the same IPs on my luggage.

Re:Not using Cisco ACLs

[contrapunctus]

I have done this error before :)

What surprised me was that the linksys router assigned IP numbers up thorough the uplink connection. I thought that was impossible, guess not.

Power strips (with on/off buttons) are bad

[gavving]

So I'm working in this company's datacenter on their networking equipment. But it's installed is such a crappy way that there's a floor tile pulled right next to the rack and the cables are run down into that hole. I'm working around on the equipment and step down into the hole by accident, at that point I notice that it's suddenly alot quieter where I'm standing, I look down and realize I'd just stepped on the power button of a power strip that most of the networking equipment was plugged into. Oh Sh!t. At the time the room was empty except for me, I quickly turn the strip back on. About the time the switches are just finishing coming back up one of the companies IT guys comes in and asks if anything's going on. I look at him a little confused and say "I'm not sure, what's up?". The network's back up by the time they noticed it.... I probably should have admitted it, but no harm, no foul. :)

data centers 101

[ei4anb]

Those data centers in the article sound huge, some may even have up to ten servers!

Re:bad article is bad

[Sponge Bath]

I only got a 200 on my English SAT. I's got no writin' skills.

You has done been promoted to /. editor. Collect your "Grammer be important!" t-shirt at the door.

Re:Network meltdown due to hub cross-connects

[pushing-robot]

Ah, yes, what network technician hasn't felt the sting of the old "cat5 o' eight tails"?

Ah, the memories! And lessons, too.

[martyb]

Ah, the memories! Here are some of the stories I've heard and or witnessed over the years.

1. Orientation: As a co-op student at DEC in 1980, I was told this (possibly apocryphal) story. On seemingly random occasions, a fixed-head disk drive would crash at the main plant in Maynard, Massachusetts. Not all of the drives, just a couple. Apparently the problem was isolated when someone was midway between the computer room and the loading dock. They heard the bump of a truck backing hard into the loading dock followed very shortly by a curse from the computer room! It apparently caused enough of a jolt to cause platters to tilt up and hit the heads... but only on the drives which were oriented north-south; those oriented east-west were not affected. So came the directive that all drives, henceforth, needed to be oriented north-south.
2. Hot Stuff: Seems that a mini-computer developed a nasty tendency to crash in the early afternoon. But only on some days. Diagnostics were run. Job schedules were checked and evaluated. All the software and hardware checked out A-OK. This went on for quite a while until someone noticed that there was a big window to the outside and that in the early afternoon the sun's light would fall upon the computer. This additional heat load was enough to put components out of expected operational norms and caused a crash.
3. Cool!: A friend of mine was a field engineer for DEC back in the day when minicomputers had core memory. He was called into a site where their system had some intermittent crashes. He ran diagnostics. All seemed to be within spec. He replaced memory boards. Still crashed. Replaced mother boards. Reloaded the OS from fresh tapes. Still crashed. He finally noticed that one of the fans on the rack was not an official DEC fan. Though it WAS within spec for airflow and power draw, it was NOT within spec for magnetic shielding... it would sporadically cause bit flips in the (magnetic) core memory. Swapping out the fan solved the problem.
4. This sucked: Another place had a problem with a computer that would sometimes crash in the early evening after everyone went home for the day. Well, not everyone. The cleaning staff apparently noticed a convenient power strip on a rack and plugged their vacuum cleaner into it. The resulting voltage sag took down the server!
5. Buttons: Every couple years, IBM would hold an open house where anyone in the community could come in and get a tour of the facility (Kingston, NY). This was back in 1984, IIRC. PCs were just starting to make an impact at this time... big iron was king. We're talking about a huge raised-floor area with multiple mainframes, storage, tape drives... MANY millions of dollars per system. A few hundred users on a system was quite an accomplishment back then and these boxes could handle a thousand users. We were also in the midst of a huge test effort of the next release of VM/SP. I had come in that Sunday afternoon to get several tests done (death marches are no fun). All of a sudden the mainframe I was on crashed. Hard. I'd grown accustomed to this as we were at a point where we were "eating our own dog food"; the production system was running the latest build of the OS. But, an hour later and it was STILL down. Apparently, a tour guide had led a group to one of the operator consoles and a child could not resist pressing buttons. Back in those days, booting a mainframe meant "re-IPL" Initial Program Load. Unless the computer was REALLY messed up and wouldn't boot. Only then would someone re-IML the system. Initial Microcode Load. Guess which button the kid pressed? It left the system in such a wonky state that it had to be reloaded from tape. All the development work of that weekend was lost and had to be recreated and rebuilt. (It was a weekend and backups were only done on weekday nights.) It took us a week to get things back to normal.
6. Drivers: A friend of mine at IBM told me of an

Obligatory

[garyisabusyguy]

THE WEBSITE'S DOWN!!!

http://www.youtube.com/watch?v=W8_Kfjo3VjU [youtube.com]

Re:Network meltdown due to hub cross-connects

[ColdWetDog]

When you're 16 working at a LAN party you get somewhat motivated when an 18 year old girl wearing duct-tape clothing (skimpy at that :p) wields such a tool :p

Yes, and now look at you. Years later, life wasted. Posting to Slashdot on a weekend.

If only you had listened to your mother and gone into welding.

Re:Network meltdown due to hub cross-connects

[Anonymous Coward]

Ah, yes, what network technician hasn't felt the sting of the old "cat5 o' eight tails"?

You're thinking of the Cat5 o' Nine Tails -- or maybe you just lost count because you were on the receiving end of one?

<--- Joke

O<-- You

CAT5 cable has eight conductors inside it... hence the joke.

Re:My favourite human error - a true story

[dirk]

I have a better AC story. We had a second AC unit installed in server room, as the first was cranking 24/7 and was just barely keeping up, with the thought that the 2 of them in tandem could handle the load. A few days after it was installed, we noticed the room was hot when we got in in the morning. Not enough to cause alarms, but hotter than it should be. As the day went on, it dropped, so we chalked it up to a one time fluke. This happened a time or 2 more throughout the week, but it always dropped during the day. Finally the weekend came, and it got hot enough to cause an alarm. We got in and the AC units kicked on without us actually doing anything, and the room started to cool down. We called out AC guys and they checked both system and couldn't find anything wrong with either of them. Well, the same thing happened again that night. Finally, someone was there late, trying to see if they could see what was going on. Everything was fine throughout the evening, so they finally decided to leave. Luckily, they noticed as they walked out the door and flipped off the lights that the AC units both turned off. HE went back in to verify, and when he turned the lights back on, the AC units both started again. Turned the lights off, and they both shut off again. The genius (lowest bid) company that we hired to install the new AC unit had wired both units into the wall switch for the lights! So when we were there checking, we had the lights on and everything worked perfectly. We went home for the day and turned off the lights, and the AC units. Needless to say, that company isn't even allowed inside out building anymore!

Re:My favourite human error - a true story

[Linker3000]

More AC fun - all in the same room - as refurbed into a computer room in the 1980s by the in-house maintenance team:

1) They re-lined the walls, but also boxed in the radiators without turning them off - we had numerous AC engineers turning up and scratching their heads while they re-did their thermal load calculations until we realised our walls were warm to the touch.

2) They put the AC stat on a pillar by the windows so in the summer, the heat radiation falling on the stat from outside made the AC run harder than required, which we really didn't notice as the room was 'cold', but come winter, the AC often wouldn't kick in until we had the stat relocated.

Best cock-up I saw was a computer room with a 4ft under-floor void. There should have been a 4 inch void, but there was a major cock-up between architects and builders. The floor panels sat on some spookily-sized pillars (which must have been specially made) and the IT staff actually put some servers under the floor.

Re:Quad Graphics 2000

[drsmithy]

One of my co-workers got curious about the unlabeled big red button in the server room. Because he lied about hitting it [...]

At a previous job we had one of these (albeit with a "Do not push this, ever" label above it) that did nothing more than set off a siren and snap a photo of the offender with a hidden camera. Much amusement was had by all when some new employee's curiosity inevitably got the better of them.

Re:Network meltdown due to hub cross-connects

[Anonymous Coward]

It's a "cat5 o' NINE tails"

Which is how you achieve 5 9's reliability, once you take it to your vendor's sales rep.

Re:None of us are innocent.

[afidel]

Wait, this was a 700 person company and they had single power source servers? Yeah the root cause of that one was not your cape =)

Re:My favourite human error - a true story

[internewt]

Best cock-up I saw was a computer room with a 4ft under-floor void. There should have been a 4 inch void, but there was a major cock-up between architects and builders. The floor panels sat on some spookily-sized pillars (which must have been specially made) and the IT staff actually put some servers under the floor.

Was Nigel Tufnel the architect?

Re:bad article is bad

[Anonymous Coward]

Suck-start a shotgun. FTFY has never been funny.

FTFY