Chinese ISP Hijacks the Internet (Again) 171
CWmike writes "For the second time in two weeks, bad networking information spreading from China has disrupted the Internet. On Thursday morning, bad routing data from a small Chinese ISP called IDC China Telecommunication was re-transmitted by China's state-owned China Telecommunications, and then spread around the Internet, affecting Internet service providers such as AT&T, Level3, Deutsche Telekom, Qwest Communications, and Telefonica. 'There are a large number of ISPs who accepted these routes all over the world,' said Martin A. Brown, technical lead at Internet monitoring firm Renesys. Brown said the incident started just before 10 am Eastern and lasted about 20 minutes. During that time the Chinese ISP transmitted bad routing information for between 32,000 and 37,000 networks, redirecting them to IDC instead of their rightful owners. These networks included about 8,000 US networks, including those operated by Dell, CNN, Starbucks, and Apple. More than 8,500 Chinese networks, 1,100 in Australia, and 230 owned by France Telecom were also affected."
Accident (Score:5, Insightful)
Re:Accident (Score:5, Insightful)
Twice is a coincidence.
Three times is enemy action."
-- Gen. Douglas MacArthur
cut out the middleman (Score:5, Funny)
Re: (Score:3, Interesting)
now you can order iPad direct from china through apple.com
Nothing new here. When I ordered this Macbook Pro last year, I was able to follow online its progress from the warehouse in Shanghai to my porch. Apple is now effectively a delivery and customer-support service for Asian manufacturers.
Maybe eventually they will cut out the middleman, as IBM did a while ago with its Thinkpad laptops. Now you order them directly from Lenovo, which is a Chinese firm. The pretense that they were an IBM product has en
Re: (Score:2)
You're basically saying that if I put together a jigsaw puzzle I invented it.
Not unintentional (Score:5, Interesting)
Re: (Score:1)
Re: (Score:2, Informative)
and add to that a Chinese CA certificate inside Firefox and even SSL could be sniffed
Re: (Score:2)
If the networks that your traffic is being routed to doesn't simply melt sure.
This has happened before quite a few times, it's a side of the internet which is surprisingly fragile.
Almost Certainly Unintentional (Score:5, Informative)
Limited-scope attacks like the Pakistani YouTube diversion are much more likely to be a deliberate attack; broad-spectrum attacks are obviously either mistakes (or really clever DDOS.) Advertising that you're the best route to half the world isn't exactly un-stealthy enough for intelligence gathering - and China doesn't have the bandwidth to handle that much traffic, either inside their entire country's network or especially across the Pacific; the only carriers with a chance of absorbing some fraction of AT&T's plus Level3's traffic are Verizon or possibly Google, and they're both competent enough not to do that.
This kind of thing happens occasionally with BGP, which was designed to be run in a relatively trusted environment by relatively-to-extremely-competent people, which means that it only explodes occasionally and most major carriers do a good job of filtering routing announcements that look seriously wrong, and detecting when other people advertise bogus information about their networks. The typical cause used to be bad conversions between external BGP routes and internal OSPF or RIP routes, especially back when some random customer would have left autosummarization on so they'd take their two Class C subnets, combine them into the Class A that they're both in, and announce to everybody in the world that they were the best route to reach the Tier 1 carrier who's their upstream (or who's the upstream of their local ISP, who wasn't bothering to filter their BGP announcements.)
The first time this happened in a big way was a bit of a surprise, as some little ISP announced that their T1 line was the best way to reach all of MAE-EAST (i.e. half the world), so suddenly there were gigabits of traffic headed that direction, at least until their self-DDOS killed off most of the BGP sessions and somebody fixed it. Since then, if you try to advertise being the best route to some large carrier who has a /8, you'll find they're also advertising a pair of /9s (which win), and that they'll be calling your upstream carrier within a couple of minutes to get your BGP session shut down. On the other hand, if this happens, it also means your upstream carrier wasn't filtering your BGP announcements for sanity, so they may also not be good at having somebody who can answer the phone and quickly resolve that level of problem.
Re: (Score:2)
they'll be calling your upstream carrier within a couple of minutes to get your BGP session shut down
I wonder what would happen if there were no voice circuits anymore and everybody used VOIP? Would network operators use dedicated radio circuits to coordinate operations? I have this vision of them pulling up their own 80 metre antennas to ensure voice communication or maybe RTTY.
Re: (Score:3, Interesting)
Re: (Score:2)
Google versus China (Score:2, Funny)
nuff said. Ok, I will ellaborate, but that shouldnt be neccecary. Do you really need to read more?
This may be a cyberwar between a multinational corporation and China. Google will of course win this war. The war is secret, and not fought with bullets. Oh, you want to know even more? That is hardly neccecary, but I will go on.
Also, we will need to equip an army of female acrobatic tech-warriors wearing tight-fitted latex with large open cleavages. That can probably keep the kung-fu chinese hackers at bay. No
Re: (Score:2)
The next time one of these stories comes around, then you can jump to conclusion. Right now, well....
Blacklist 'em (Score:5, Interesting)
Re:Blacklist 'em (Score:5, Informative)
Blacklisting China's IP ranges would do nothing to protect you against bad routing - something you as an end user don't have any control over.
Re: (Score:2)
I second the AC above: If someone has a link for all Chinese Internet-routable subnets in order to drop, that'd be cool.
No, it won't protect against malicious fake routes, but it protects against attacks/scans/connections from legitimately Chinese networks.
Re: (Score:2)
http://www.blockbycountry.com/ [blockbycountry.com] can give you the list, though it'll come in the form of an htaccess block list that'll generate for you. Short work to convert it into a list of apf rules, though. But, as you are aware, this still won't protect against most-specific route advertisements to BGP peers.
oops, bad link; sorry (Score:3, Informative)
I mistyped the link. The proper URL is http://www.blockacountry.com/ [blockacountry.com]
Re: (Score:2)
I found this site [okean.com] that has Chinese and Korean lists in several formats
Re: (Score:3, Interesting)
Of course, you are right about the routing. But since giving in to my baser impulses and blacklisting the entire country on my one humble web server, I've had a remarkable decrease in my annoyance factor in terms of crap like port scans, login attempts, comment spam in the blogs, and even a respite from the damned Baidu spiders who won't observe anybody's robots.txt file. Along about the fall of last year, I began observing what looked like attempts at ddos attacks--all originating from China. None of them
Re: (Score:2, Informative)
Baidu's real spiders obey robots.txt. However there are plenty of malicious spiders out there who pretend to be Baidu in their User-agent string - giving Baidu a bad name in this area.
Re: (Score:2)
Sorry, but if Baidu wants us to believe that their spiders behave lawfully, they should arrange to receive lawful communications regarding them. As for me, I'm enjoying the respite--since the spiders stopped when I terminated the communication.
Re:Blacklist 'em (Score:5, Interesting)
Until China learns how to act as responsible Internet citizens, I'll continue to blackhole as many of Chinese subnets as I can find both at work and home. Spam, malware, and every kind of crap comes from China, and I don't do business with any Chinese, so it's a no-brainer
Well, since more SPAM comes from the US I assume you'll block those subnets too? http://www.spamhaus.org/statistics/countries.lasso [spamhaus.org]
Also, in March the US was the source of most malware, but since you already have that blocked for SPAM you should also block Korea who for some reason in the month of April took the lead. http://www.infosecurity-us.com/view/8547/korea-reigns-as-king-of-malware-threats-/ [infosecurity-us.com]
In regard to China learning how to act as responsible Internet citizens, you are not leading by example.
Re: (Score:2, Insightful)
Spam is not the only factor to consider.
It certainly DOES make sense to blacklist China in its entirety unless you're doing business with them.
Re: (Score:2)
Re: (Score:2, Insightful)
Dude pull your head out of the sand. The US government doesn't trust its own citizens too - that's why they datamine and wiretap your ass.
The sheer hypocrisy and the little fantasies Americans tell themselves to feel better about themselves - need a new 'cold war' enemy to fight against, sandal-wearing dipshit? Was 'Al-Qaeda' as the big bad 'bogeyman' not doing enough to wet your sado-masochistic 'warrior' fantasies?
I've never seen a more clear-cut example of 'pot calls kettle black'. America has been the n
Re: (Score:2)
Wow, that is infallible logic. Completely repudiated.
I wish to subscribe your newsletter, hmmm?
Re: (Score:2)
yes than your traffic can get router there anyway when the start advertising American, and European subnets.
Re: (Score:2, Funny)
The Great Firewall of China, works two ways (Score:3, Funny)
Good walls work both ways. To "help" China from being tainted by the evil ways of us westerners let's just cut them off completely.
Access to Zebra, Re:Blacklist 'em (Score:3, Insightful)
While at it, I offer you to query my own Zebra server, I guarantee to only return the best available routes ;-))
http://www.gnu.org/software/zebra/ [gnu.org]
Contact me off-line if you are interested.
Seriously, I have some friends who do like you, they start by blocking China, then Korea, then end up blocking half of the world to enhance their security.
In my humble opinion, this is not a valid security approach, I actually use some requests or connection attempts from these countries to test and strengthen my security.
Re: (Score:2)
Re: (Score:2)
> 100% of my thousands of failed SSH attempts come from
> (Chinese) APNIC address space
I call bs on this and I have logs and automated complaint reports to prove it. Also, I have other additional means to deal with this issue.
There are compromised machines on every network, most of these attempts are done by botnets without the knowledge of the IP owner as I found exchanging with remote network admins. ! ;-))
Here is one report, I edited out my own IP for obvious reasons.
Date: Tue, 13 Apr 2010 15:27:10
the latest hot status symbol: (Score:2)
The ??AA can suck it, too!
Re: (Score:2)
You sir are a fucking moron.
Re: (Score:2)
To be fair, the American auto companies were behind the phony reports of Toyota accelerator problems. A lot of Americans were just fooled by the media.
Re: (Score:2)
Oh, let me guess, 9/11 was an inside job too!
And Obama's birth certificate is FAKE!
Did I miss any? I actually hadn't ever heard that new conspiracy theory before. Thanks, I'll add it to the collection.
Re:Blacklist 'em (Score:5, Informative)
I use http://www.countryipblocks.net/ [countryipblocks.net] -- they seem to do a pretty decent job of keeping their database up-to-date. It will also provide the output in varying formats (net/mask, CIDR, ip range, etc).
Re: (Score:3, Informative)
I also like http://www.ipdeny.com/ [ipdeny.com]
An old saying... (Score:4, Insightful)
"Once is an Accident, twice is a Coincidence, and three times is a Pattern."
Re:An old saying... (Score:4, Informative)
Three times is enemy action.
Re: (Score:2)
You'd be funnier too if you had any sense of history.
http://thinkexist.com/quotation/once_is_happenstance-twice_is_coincidence-three/220863.html [thinkexist.com]
Re: (Score:2)
Re: (Score:2)
Re:An old saying... (Score:5, Informative)
The correct quote is:
Re:An old saying... (Score:5, Funny)
Re:An old saying... (Score:5, Funny)
Confucius say, Man who walk through airport turnstile sideways is going to Bangkok.
*GONG*
Re: (Score:2)
"A communications disruption can mean only one thing... invasion!" - some shitty Star Wars movie
Re: (Score:2)
"Once is an Accident, twice is a Coincidence, and three times is" ...enemy action, I think it was. Appropriately.
Re: (Score:2)
Uh, oh! Did anyone notice The Observer in the vicinity?
Wiskey Tango Foxtrot (Score:5, Insightful)
Any sufficient level of Incompetence is indistinguishable from Malice.
Solution however is exactly the same.
Re: (Score:2)
What about signing & certificates? (Score:5, Interesting)
Re:What about signing & certificates? (Score:4, Informative)
Re: (Score:2)
That's because there are better solutions, including LISP (Not the (()))()(), but http://en.wikipedia.org/wiki/Locator/Identifier_Separation_Protocol [wikipedia.org] - which has just been implemented by Cisco.
built to spill (Score:1, Insightful)
... faulty by design.
Fall guy (Score:4, Interesting)
Why can one "small" ISP do this? I mean from a technical point of view how can they spread routing information for endpoints their network doesn't own? While they have clearly dropped the ball, I struggle to understand how they could accomplish this even if they tried, that is if everyone else's equipment is configured correctly *cough*
Re:Fall guy (Score:5, Informative)
Re:go back to old school principles (Score:2)
IP V6 everywhere
static herarchical routing everywhere based on geographical IP addresses prefixex.
like in the old telecom way.
Fat Chance that IPv6 actually fixes this problem (Score:5, Interesting)
By "old-school principles", you did mean "pre-ARIN IPv4 Swamp Addresses", didn't you? :-)
Yeah, the people who designed IPv6 hoped that by having a big enough address space with no pre-existing reservations, they could make routing simpler and cleaner and delay the problem of routers running out of special route table memory and routing protocol horsepower, but that was pretty much a pipe dream:
so the IPv6 world's going to be a non-hierarchical mess just like the IPv4 world.
Trust but Verify (Score:2)
As several other people have commented, the ISPs they connect to are responsible for doing some sanity filtering on the routes they announce. It's not universal, especially for connections between ISPs (as opposed to connections from end-user customers that use BGP for multi-homing, where ISPs usually do a better job), and there's nothing close to universal agreement about address range registration systems or how to validate BGP information.
Re: (Score:2)
>>there's nothing close to universal agreement about address range registration systems or how to validate BGP information.
Given this same problem happened before back in the 90s, you'd think that they'd at least not allow negative route lengths to be propagated.
Re: (Score:2)
However, why should a network be able to advertise routes for subnets that are out of its control? Even if we accept multiple levels of peering relationships, there should be some safeguards against overly broad routes and "hijacking" of networks known to be authoritatively announced by other peers.
(Note: I'm genuinely asking, as I'm fairly ignorant of the design of BGP - I'm much more LAN than WAN.)
The whole idea of "trust" on the network is something of an anachronism. The internet is not the secure, safe
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
It reminds me of a scenario we had at work. We come in one day and find that about half the computers in the building are getting bad IP addresses, and as such, weren't able to connect to the email servers or the internet. We found out it was a rogue router on the network, dishing out 192.168.1.x/24 addresses when that specific building was under 172.21.30.x/20. We were lucky that it was obviously a default linksys setup, we were able to log into it once we found the IP and disable DHCP. Then we had to go t
Re: (Score:2)
One of the joys of being on the 172.x.x.x network scheme internally is finding devices like that. Everyone knows about 192.168.x.y and 10.x.y.z, but the 172 range tends to be overlooked.
Re: (Score:2)
Re: (Score:3, Insightful)
The small ISP can't do this if the big ISP would've done it's job properly.
Re: (Score:2)
If you are expecting a router to pass GOOD data, how hard is it to believe that someone can trick you into accepting BAD data?
This is no different than you downloading a Windows Update that bluescreens your computer. Clearly your equipment isn't configured correctly.
In actuallity, in order to route things through China, you have to trust China, and yes, that sucks, and yes, I'm using way too many comas.
Re: (Score:2)
I'm using way too many comas.
If you were putting your comas to good use, you wouldn't have enough consciousness left to over-use the commas.
Chinese bashing? (Score:1)
Re:Chinese bashing? (Score:5, Interesting)
Re: (Score:3, Insightful)
Are you saying this is truly a selection bias, or are the Chinese screwups more global in scope? Seems like propagating a small ISP to a large ISP to the entire Internet would be something
Re: (Score:3, Interesting)
Don't get me wrong, this was a really big mistake. It doesn't happen often at this scale, but it does happen.
In this case the prefixes what were mis-broadcast were sequential for the most part and covered several networks and countries, not a specific target. The bulk of the misrouted addresses were actually in China. They also didn't leak the routes (as in the Pakistan incident) but re-originated the prefixes, pre-pending their AS number to the announcement. This means "origin AS" based filters would have
Re: (Score:2)
Thanks again.
Corrections - every couple of years, and Pakistan (Score:2)
It was actually Pakistan, not Iran, and significant problems are more like every couple of years - and most ISPs have enough filtering to prevent most accidental screwups from getting very far, at least for very long. But yeah, it's not rare, and it only takes multi-party incompetence, not malice.
Gotta Build A Fence (Score:5, Funny)
Does Narus do business with China? (Score:3, Interesting)
Chinese fire drill? (Score:4, Funny)
Filter BGP updates? (Score:3, Insightful)
does this imply large scale packet sniffing ? (Score:2, Interesting)
So while this was going on could the chinese save off the network traffic? They have the infrastructure Cisco routers, etc. ...
Could they decrypt SSL packets ? It may take awhile but they're not doing this real-time.
Go through any interesting attachments ? Spreadsheets, documents,
I think I'll read up more on asymmetric warfare and the Red Army officer's paper on the subject.
How to protect against this (Score:2)
This is unlikely to be the last this will happen. What can be done to protect against this sort of issue?
Failure of Tier 1 ISP's (Score:2)
Tier 1 & 2 ISP's should really be filtering all subnets they own. A lot of them do, but also a lot of them do not or think their Tier 2's are handling it. I've seen a company who was assigned a /24 misstype a number and suddenly they're claiming a /16 and disrupt a bunch of our customers.
Unfortunately many companies are ill equipped to detect this type of error, internally they may see everything is fine, but it's external traffic that's being detected.
It's easy if you can setup a server to check who'
Re: (Score:2)
Oh yeah?
What ISP owns 17.0.0.0/8? In fact, how does any ISP know what other ISP is allowed to advertise that prefix or a subnet of it?
Re: (Score:2)
In which case, the Tier 1 should filter announcements for anything other than:
And to answer your other question, Apple owns 17.0.0.0/8
OrgName: Apple Inc.
OrgID
Slashdot hit by China... (Score:2)
It seems like Slashdot has been hit hit by China.
If I try:
http://slashdot.org/firehose [slashdot.org]
or
http://slashdot.org/~ls671/ [slashdot.org]
I have been getting this for the past half hour:
Error 503 Service Unavailable
Service Unavailable
Guru Meditation:
XID: 147127282289
Varnish
Thank $deity for signed prefix announcements (Score:2)
RIPE is pushing to have all route announcements signed by 1.1.11 and the other four RIRs are following suit. Personally, I can't wait for this to happen :)
Re: (Score:2)
Re: (Score:2)
You know where they are coming from, but it seems the tier 1 & 2 ISPs are not willing to filter incoming routes from China. With signed announcements, they would have something to filter on.
While that won't help if they just strip all intermediary AS numbers out of the routes (unless upstreams are also verified, at some point), it will still improve the overall situation.
Re:Why the FUCK does china still have internet acc (Score:5, Interesting)
Our Grand Communist Party of the Great Nation of China plan to get the rest of the world to leave us alone about our glorious firewall, and desire, nay, duty to protect our citizens:
Step 1: Push out Google
Step 2: Muck up their internet
Step 3: They kick us off "their" internet
Step 4: Setup our own, national, internet
Step 5: Be praised by the lesser nations for staying off their internet, rather than chastised for walling ourselves off and keeping their realfacts out
Step 6: Spread propaganda, er... goodfacts about our Grand Communist Party of the Great Nation of China
Step 7: Unlimited, eternal power to do whatever we please
Re: (Score:2, Insightful)
Racist garbage spoken like a true uninformed dickhead. Meanwhile crap like this continues to get modded up on slashdot. I'm tied of seeing almost daily china threads started on /. accompanied by racist or boarderline racist rants in the threads.
Re: (Score:2)
I'm tired of seeing almost daily china threads started on
Re: (Score:2, Funny)
'cause we created it. Thanks.
Close enough (Score:3, Informative)
ISPs use BGP to talk to each other, but internally they may use iBGP or EIGRP or OSPF or (once upon a time) RIP, and they usually have a complex routing structure internally and a small number of border routers that announce a simplified set of routes to their upstream carriers or peers. Badly-automated conversions between OSPF/etc and BGP are the easiest place to make a big mistake like that, though some operators are clever enough to break their routing purely by hand.
Re: (Score:2)
It's sort of how the network 'works', at a fundamental level, and it works really well if everybody basically trusts their peers and knows what they're doing.
You left out "...and are not malicious."
Re: (Score:2)
configuration error??
Sometimes diplomacy is required, like when 'getting shot' is referred to as lead poisoning.
Re: (Score:2)