Multiple Fiber Cuts In San Francisco Area 368
georgewilliamherbert writes "Multiple news reports, mailing list posts, blogs, and tweets are pointing out two overnight acts of sabotage in the San Francisco Bay area, with long distance fiber network cables being cut in two locations in the early morning hours. The first cut, around 1:30 AM, affecting landline and cell phone service and 911 calls in the communities of Morgan Hill, Gilroy, and parts of Santa Cruz counties, was on an AT&T fiber alongside Monterey Highway near Blossom Hill Road, in San Jose. A second cut, around 3:30 AM, in San Carlos, affected Sprint fiber and has significantly disrupted services at the 200 Paul datacenter in southern San Francisco. Rumor says that this may be related to a AT&T communications workers contract having just expired — but no evidence has been published yet in the media, and this could be an intentional act of sabotage by someone unrelated to the company's workers."
BOFH (Score:2, Informative)
Re:Just curious... (Score:5, Informative)
...but how do you repair a fiber optic cable that has been cut? What is the magic process for sticking it back together?
splicing it together.
http://en.wikipedia.org/wiki/Fusion_splicing [wikipedia.org]
It's like getting two copper wires and just heating the copper to such a high temperature that they melt and re-form one strand.
Re:Just curious... (Score:5, Informative)
Cut the fiber carefully and cleanly back from the cut, which has ragged ends. Usually a few feet in each direction.
Bring in a fiber patch section.
Go in with fiber polishing gear, to every individual fiber on one side, polish end, test end, polish again until it's smooth enough. Identify what fiber ID that fiber is. patch it together with the patch cable. Repeat on the other side of the patch.
Cross-test to ensure that you didn't cross any fibers in the reattachment - if so, pick one end as new ground truth, and repatch or logically reroute the other to match new physical reality.
Once the whole bundle has been repolished, patched, and tested on both sides, you wrap the patch sections up with new covering (armored section, flexible covering, depends on the cable and location). Apply waterproofing goop.
Put the manhole cover back on. Consider locking it down in place, this time...
This is tedious work, requires careful attention to detail to properly polish the cut fiber ends and repatch them, and for large fiber bundles takes forever. You can start running data through a fiber once its two ends are repatched - you don't have to get the whole bundle back for that - but the whole process can take 24-48 hours depending on how many fibers are involved and how much space there is to work in the trench or down the manhole. In many cases, there's only enough space for 1 or maybe 2 people to be working at any given time, which makes the repairs take forever...
Status from earlier in the day... (Score:5, Informative)
Activity Type Code Desc: PROGRESS COMMENTS
Activity Type Code: PROG
OTDR readings were taken by AT&T West and a cut was located 1600 ft from
the San Jose, CA central office. AT&T West technicians are onsite
working to isolate the exact location of the cut. There are 4 cables
impacted. AT&T Mobility has 61 GSM and 45 co-located UMTS sites out of
service off of Santa Clara Base Station Controllers 15 & 23, and Santa
Clara Radio Network Controller 4. E911 has 52 Location Measuring Units
down. The AT&T West Santa Cruz 11 central office (41,803 ATNs) is
experiencing an SS7 isolation and the San Martin central office (11,904
ATNs) lost it's umbilical and is isolated at this time. The Bailey
remote site (4,973 ATNs) is also isolated. Scott's Valley has 3 out of 4
SS7 links down. The Santa Cruz 01, Aptos, Scott's Valley, Felton,
Boulder Creek, Ben Lomand, San Jose 11, San Jose 13, San Jose 21 central
offices have trunks impacted such that all lines are busy and incoming
calls are receiving trouble messages. The Santa Cruz County SO (178,040
ATNs), Scott's Valley PD (12,007 ATNs) and the UC Santa Cruz PD (14,909
ATNs) are all without ALI at this time. The Gilroy PD PSAP and the
Morgan Hill PD and CDF have been rerouted with ALI/ANI. The Felton CDF
has not been rerouted. There are 17 DSLAMS and 4 ATMS out of service
impacting DSL service. There are 3 SMDI Links down impacting voicemail
service. Verizon's Morgan Hill and Gilroy central offices are currently
isolated. There have been 224,865 blocked calls.
Re:Why our infrastructure is vulnerable (Score:5, Informative)
Re:Just curious... (Score:2, Informative)
No you wont. This is not 62.5/50micron multi-mode fiber. A coupler in a single-mode fiber causes a great deal of signal loss. I have never seen anyone terminate SM fiber anywhere but a termination point (i.e. at the equipment, repeater, or patch pannel inside a building.) "Just install a repeater" is laughable... those things are not free and require power that isn't found in most ditches.
Today, we have very good equipment for making fusion splices -- to the point it's almost automatic. The real time consuming process is getting to the fiber to fix it in the first place. Followed closely behind by the tediousness of getting each strand connected to the correct other half.
Re:Your optic fu is strong (Score:5, Informative)
I can tell from your UID and the description that you know the old school ways.
I never want to see another polishing puck again.
The new fusion splicers really do make it easy as now it is just strip the insulation back a quarter inch for the 62.5 (MM)or more probably 9 (SM), get a good cleave, and let the fusion splicer rip. Have seen a 24 strand cut fixed in about two hours, with about a quarter of the splices at 0.0dB loss (yes, I do mean ZERO) and the rest 0.05 to 0.1.
I think Corning Cable Systems (Siecor) also has a ribbon cable splicer for instant pigtails up to 72 strand, its been a few years since this happened, so not really up on the latest.
Re:Scrappers (Score:4, Informative)
Re:Why our infrastructure is vulnerable (Score:3, Informative)
I would assume so, presumablly you would have to work along the cable fixing each cut and then taking a new measurement.
Quite a good way to inflict a lot of financial damage on a telco really. Afaict fixing fibers is FAR more expensive than cutting them.
Re:This just in (Score:5, Informative)
Copper theft is at record highs right now.
A year ago, yes, not anymore. The prices for scrap has fallen through the floor, thanks to the global recession.
Reward has been posted (Score:3, Informative)
AT&T Offers $100k reward for capture of vandals. [gizmodo.com]
at&t using twitter to post updates on restorat (Score:2, Informative)
http://twitter.com/attnews [twitter.com]
Re:The Arpanet was supposed to survive nukes. (Score:3, Informative)
Blame the Tier-1 & Tier-2 backbone providers and telcos for skimping on SONET implementations; UPSRs (Unidirectional, Path-Switched Rings) do not have the line-fault switching capabilities that a BLSR (Bi-directional, Line-Switched Ring) because of the single-direction design of a UPSR. Since UPSR networks are cheaper (1/2 the fiber-lay costs) than BLSR, many large telcos and backbone providers play fast and loose with fiber capacity and provisioning...which, in this case, apparently came back to bite them.
The original ARPANet, as it was designed at that time in history, *was* redundant and met the needs for the spec. The ARPANet / NSFNet is as distant from today's Internet as a Blue Whale is from granite.
During "The Great Internet Build-out" of the late 90's, outages similar to this were more common than what you have been led to believe; the reason why people heard virtually nothing about those outages was because (a such outages weren't "visible" to those outside of the telco industry, and there wasn't such a demand 10 years ago for such high capacity circuits, and (b circuits were more carefully planned-out and used BLSR as much as possible. Now, where stockholders go crazy if their investment in a given telco doesn't grow by 10%, those telcos scrimp and cut corners wherever they can - including running SONET networks with inherently unsafe ring topologies.
For more about the differences in SONET topologies, please visit:
http://www.hill2dot0.com/wiki/index.php?title=2F-BLSR [hill2dot0.com]
--ScottKin