Slashdot Log In
Sprint Routers Stolen; NYC Internet Outage Ensues
Posted by
timothy
on Mon May 03, 2004 07:41 PM
from the it's-not-what-kind-of-bike-it's-the-kind-of-lock dept.
from the it's-not-what-kind-of-bike-it's-the-kind-of-lock dept.
cbnet2004 writes "This story on eWEEK reports that late Sunday night a number of Sprint's DS-3 network cards were stolen from a Verizon colocation center at 38th St in Manhattan. Some customers apparently have service back but a number remain down -- it could be a while. The latest rumor on this situation is that some fiber optic cables were cut as well; this could put the affected customers out for days more."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Was it easy? Why was it not major? (Score:5, Interesting)
This quote sort of caught me off-guard as I imagine some customers might disagree:
Fleckenstein said that the outage was "not major," and not large enough to require a report to the Federal Communications Commission.
The beginning of the article states:
A handful of corporate customers were left without e-mail and Internet access Monday after the theft of networking equipment from a New York City office late Sunday.
So, I would guess that the "handful" of corporate customers who lost service probably felt it was major to them. I understand the notion that it was not major in the sense of being more widespread, I just think his comment could have been worded better.
Happy Trails!
Erick
Re:Was it easy? Why was it not major? (Score:5, Insightful)
Parent
Re:Was it easy? Why was it not major? (Score:5, Interesting)
Ar....that remind me of my days in a research lab.
Security guards downstair would be 'notified' whenever someone is attempting to reboot those SGI workstations at night. The problem was that SGI hanged up quite often. When this happen, we should either move to another workstation, wait til tomorrow morning. Sometime we had no choice but to trouble those security guards when we ran out of unhanged SGI.
Initially those security guards were nice to us as we didn't do reboot very often - until someone decided to replace all those SGI workstations with NT Alpha. You imagine how irritating to have been called 2-4 times every night.
Soon after the SGI were replaced by NT Alpha, those reboot-alarms were removed for obvious reason.
Parent
Re:Was it easy? Why was it not major? (Score:5, Insightful)
Parent
How can they not figure out who did it? (Score:5, Informative)
Sure, I could have brought in a stick and poked at lots of other customer's gear, thru the chainlink cubicles, but I mean, I was signed in and on camera.
Check the log - when did the affected net go down and who was there at the time?
It has got to be a short list of visitors and guards or somebody is really stoopid.
Parent
Re:How can they not figure out who did it? (Score:5, Informative)
The phrase collocation facilities could cover a wide range of facility types. It could cover a datacenter like you are thinking of, manned 24/7, or it could be a small switching center where Sprint has leased a rack from Verizon that is normally unmanned unless someone from Sprint or Verizon actually is working on the equipment.
There is one of the later near my apartment. Basically, a building smaller than a small house, with equipment from the ILEC and maybe one or two CLECs. I assume they have an alarm or security cameras, but someone could break the door, and be in and out, long before any police could arrive.
Parent
Unmanned Vz facilities (Score:5, Informative)
Parent
Re:Was it easy? Why was it not major? (Score:5, Funny)
Then again, I suppose it does take some network connectivity to build a Beowulf cluster...
Parent
Re:Was it easy? Why was it not major? (Score:5, Interesting)
We lost our direct extension phone dialing ability to them (could use the full 10 numbers though) and they completely disappeared from our network. They still had regular internet connectivity through a 100mbit cogent line and were able to access other company resources through our other offices Citrix metaframe farms [note 1] with almost full capacity but we still recieved numerous calls at our office as only the road warriors were actually used to using that method for access. We have the licences, horsepower, redundancy, and data sharing ability for this exact reason, well actually in case of another terrorist attack but it works for this too
[note 1]
One thing stood out above all of this. About a year ago, a discussion at a network/desktop meeting lead to a disagreement but eventually a gadget VBS workstation AD weenie created a script on the pc's to "automatically" select connections to our fellow offices Citrix servers through the internal network if you were plugged into the internal network. It was to "eliminate" any http or https confusion as you technically did not need https if you were already on the company WAN, I guess the KISS approach was not a challenge. That was all fine and dandy until today when the route was down. They eventually pushed out an undo so you could connect either way but I wanted to call up and laugh and say I told ya so but I decided not too. What comes around goes around.
Parent
Potential Suspects (Score:5, Funny)
Re:Potential Suspects (Score:5, Funny)
I think we found them [slashdot.org] already....
Parent
Be on the lookout for... (Score:5, Funny)
Parent
Re: Be on the lookout for... (Score:5, Funny)
I don't see anything unusual yet [ebay.com]
Parent
That's odd .... (Score:5, Funny)
How many people were affected by this?
Re:That's odd .... (Score:5, Funny)
Good question, but now I guess we have a pretty good idea where all the FP trolls are from.
Parent
Haha (Score:5, Funny)
Who'd you sell it to? Dude will be busted. Someone walks up to you in an alley and say "wassup cuz you wanna buy a ds3 innernet?" it raises eyebrows.
Re:Haha (Score:5, Funny)
We are talking NYNY here, right? Here in the Midwest, our impression is that no one would blink if you walked up to them and tried to sell them an oil drilling rig or an Aegis cruiser.
Parent
Re:Haha (Score:5, Funny)
Imagine some druggie snorting coke off of a WinXP CD with the ds3 card, "Dude this is way better than a mirror and razorblade..."
Parent
Re:Haha (Score:5, Funny)
Parent
Hmm (Score:4, Funny)
Story from the mysterious future... (Score:5, Funny)
NIC thieves busted! Traced by MAC Address when the stolen components were plugged in.
Re:Story from the mysterious future... (Score:5, Funny)
Parent
Re:Story from the mysterious future... (Score:5, Funny)
Parent
Disgruntled? (Score:5, Funny)
I'm surprised it didn't happen sooner.
Re:Disgruntled? (Score:5, Insightful)
Parent
That's the same combination on my luggage! (Score:5, Funny)
Poor security or inside job? (Score:5, Interesting)
It'll be interesting to see where this investigation goes.
Re:Poor security or inside job? (Score:5, Insightful)
I have shopped around for a data center more than once. The people who take you on the tours are so eager for your business (at least nowdays) that they show you just about everything. One company even took me into a place where pretty much all the connectivity in Seattle passes through (a level 3 node or something, I can't recall the name.) This place was secured by two locked doors with no guards and street level access. I have seen plenty more 'security' that would be pretty easy to bypass. If you were a terrorist, it would be pretty damn easy to destroy many of these places.
Parent
CASE SOLVED (Score:5, Funny)
"late Sunday night a number of Sprint's DS-3 network cards were stolen from a Verizon colocation center at 38th St in Manhattan"
This can also be read:
"late Sunday night a number of MAJOR TELCO's DS-3 network cards were stolen from a RIVAL MAJOR TELCO colocation center at 38th St in Manhattan"
The reward money can be sent to my spam-obfuscated email addy.
Re:CASE SOLVED (Score:5, Interesting)
VZ, especially in NYC, has a bad reputation for these antics.
Parent
Physical Security (Score:5, Insightful)
Most people spend way too much time on thinking of attacks from the Internet or employees, but usually don't look at someone who wants to sabotage the equipment. Computer rooms usually contain all of the proprietary data in a company, and most companies don't put that much effort into patrolling computer rooms for people who shouldn't be there. Executives should make sure that physical security is part of the I.T. plan from the beginning and not an afterthought.
I'm assuming in this case it was in a Verizon C.O. which are usually somewhat secure, but something like this could happen anywhere, computer sabotage I think will become more and more common in the future as businesses rely more and more on them.
Joint terrorism task force??? (Score:5, Insightful)
That's just great. Not that I don't hope they find the crooks to walked off with this stuff - but once the word "terrorism" pops up, all of the sudden I'm thinking Patriot Act.
These thieves might have gotten themselves some kick butt network hardware - but I bet they won't get themselves due process
Re:Joint terrorism task force??? (Score:5, Insightful)
I see a problem. Last year the Patriot act was only for infringing the constitutional rights of terrorists. Last week, it was only for terrorists and drug smugglers. Today it's terrorists, drug smugglers, and network card theives. Who's up next for loss of due process? People accused of robbery? Fraud? Speeding? Keeping overdue library books?
If you start denying due process to anyone it erodes the rights of everyone. Now we're seeing that slippery slope in action.
Parent
Re:Joint terrorism task force??? (Score:5, Insightful)
If we take away rights from whoever is unpopular with the government today, then we don't have rights, period. The whole IDEA of rights are to protect you from the government and other citizens. The worst scumbags in the world have rights. In fact, it's probably the scumbags who most SHOULD have them, because they NEED them the most.
The measure of your rights is what you retain when your your government hates you and wants you dead. Rights that you have only when popular aren't rights, but privileges, which are revocable.
Any group of people that values life over freedom is easy to enslave.
Parent
Well, that explains that... (Score:5, Funny)
No wonder eWeek was the first on the story, even though it took them a while to publish it.
Working in the NOC (Score:5, Interesting)
Re:How they did it (Score:5, Insightful)
What's amazing (and it may not be the case, as we don't know all of the details, I'm sure) is that a simple correlation of the start time of the network down event and the sign-in log and security cameras (if any) hasn't been done to ID who did it. These facilities aren't particularly heavily trafficed by people on Sunday evenings, and they usually aren't all that big.
Parent
Routers Missing - Happens a lot. (Score:5, Interesting)
To be fair, I hadn't connected them yet, so they were just in the cabinet not powered up, and I was going to bring them up the next time I returned to that location, which was going to be in about 3 months. All to often, in a production environment, when there is an emergency, anything not powered up is often considered fair game. I'm sure that the routers are still in use at the company, I just can't find them.
Most colo space in our company is pretty secure. You'd have a pretty tough time getting in if you weren't supposed to be there. Even if you did get in to the colo space, most customers keep the stuff that they manage themselves in locked cages, inside the already secured colo space. Perhaps it was Verizon employees just trying to screw over Sprint. Or perhaps Sprint didn't secure their stuff properly.
------Can you hear me now?.
I am so NOT surprised. (Score:5, Funny)
But Quietly.. (Score:5, Insightful)
Stolen or damaged? (Score:5, Interesting)
My business initally heard stolen equipment but we were later told that it was caused by damaged equipment from a "Verizon union employee".
I was not on the call but that type of information is VERY specific and there is no gray area or room for interpetation there. I assume this is either totally 100% completely false or someone else knows something more.
FP !!! (Score:5, Funny)
Switch room ops (Score:5, Interesting)
The physical security is usually pretty good. About on par with a normal Fortune 500 company, where you scan into areas that you have a reason to be in. The switch room is usually a little harder to get in, especially since 9/11. At Nextel, they actually hired armed guards for a short while when we almost hired an alleged Felon. A competitors security guard recognized him and tipped off our security. Turns out he was supposedly part of a crew that carted off entire racks of telecom equipment.
Getting back on topic. The cards sound like they are the DS3 that pop into a larger fiber demark, like an OC12, 48 or 192. The cards are pretty small and just have coax-looking DS3 plugs on the front (in, out, and monitor). These aren't cards you could really ever use anywhere else. It almost sounds like someone accidentally yanked the wrong cards during maintenence. Although, most telecoms are very religious about not doing maint during the day (if the outage started at night, tho, I'd say it was a switch tech who screwed up).
The reason I'd assert this is the theft was too small to be of any other value. Three DS3 cards aren't going to fetch much, and they're tainted goods. If you're malicious, you're not going to just grab 3. If you're damaging a competitor, grabbing 3 cards is somewhat silly. We commonly have a backhaul path in preparation for things such as this. For example, when I worked at Nextel a fiber dig broke a couple DS3s we had going through PacBell. Within 4 or 5 hours, we swung the traffic over to other DS3s that bypassed the carrier and area with the break.
On a side note, it was also an eye opener that the "Protected, Redundant" Ring-topology that we were paying extra for was not being provided by the Telco. Let's just say there were some very colorful conversations going on between companies at the VP level.
Terrifying 911 call from incident (Score:5, Funny)
Slashdot user: I just lost all Internet!!!
911: Excuse me?
Slashdot user: I was just about to first post to Slashdot, and I clicked "Submit," and nothing happened. I tried to ping them to see if I was dreaming, but got nothing, so I tracerouted and found out I couldn't get past localhost.
911: I don't understa...
Slashdot user: My god, this first post would have done wonders for my karma! And now I've been beaten by a goatsex troll...
911: Sir, this line is used strictly for emerencies...
Slashdot user: THIS IS AN EMERGENCY, DAMN IT!!! Send ambulences! DSL repairmen! Cowboyneal! I won't leave this computer until I get my Internet back, and I only have half a can of Mountain Dew to live on till then!
Hollywood Movie (Score:5, Funny)
Vixie gives it an allow ACL.
Cisco's "Packet" magazine calls it "this season's most secure flick".
NANOG calls it "an interface to remember".
(ignore the creative liberties. I was out of ways to tie things together...)
Re:This is really bad. (Score:5, Funny)
Then, if your hosting company isn't full of morons, you will restore it from the multiple backups.
Parent
Re:This is really bad. (Score:5, Funny)
I don't need to worry about that because most of my equipment is steel already. Except my Powerbook, which is aluminum.
What if a terrorist had got in there and blew up all our data.
That would be terrible. I remember one time when I spilled all of my data on the floor. I was cleaning it up for days; it's almost impossible to get data out of a carpet once it dries!
For terrorists this would be a major blow to interest banking which they so abbhor.
I'm sure Osama bin Laden is at this very moment plotting to destroy those infidel bankers that are keeping his billions secure and earning him a nice revenue stream through his investments.
Parent
Re:vulnerability (Score:5, Interesting)
Parent
Re:Fiber Optics Cable Dynamics (Score:5, Interesting)
I imagine fiber repair guys earn their pay, especially when it's out in some muddy hole and they have a couple hundred strands to fuse together while everyone is breathing down their necks.
Just wondering out loud... It would be really cute if some of the fiber repair guys worked for the NSA. "Of course there's a blip in the TDR, that's where we repaired it." "Okay, nevermind."
Parent
Re:Reading the Article and (Score:5, Funny)
:)
Parent