Microsoft's GitHub Under Fire For DDoSing Crucial Open Source Project Website (theregister.com) 51
The servers used by the GMP project, an open source arithmetic library at the heart of GCC and other programs, slowed to a crawl earlier this month due to a large amount of network traffic originating from Microsoft servers. The Register reports: Torbjorn Granlund, principal author of GMP, raised the alarm in a note to the project's mailing list. "The GMP servers are under attack by several hundred IP addresses owned by Microsoft Corporation," he wrote. "We do not know if this is made with malice by Microsoft, if it is some sort of mistake, or if [it is one] of their cloud customers ... running the attack. The attack targets the GMP repo, with thousands of identical requests. The requests are cleverly chosen as to cause heavy system load. "We're firewalling off all of Microsoft's IP addresses as an emergency response."
The following day, Mike Blacker, director of threat hunting, operations, and response at Microsoft's GitHub, had identified the culprit: a GitHub Actions Workflow that clones a Mercurial repo and has been forked more than 700 of times. "Microsoft and GitHub have investigated the issue and determined that a GitHub user updated a script within the FFmpeg-Builds project that pulled content from https://gmplib.org," explained Blacker. "This build was configured to run parallel simultaneous tests on 100 different types of computers/architectures. This activity does not appear to be nefarious. [GMP] appears to have limited infrastructure that could not sustain the limited, yet simultaneous requests." [...]
As of last week, the excessive traffic was still an issue. "Our servers are fully available again, but that's the result of us adding all participating Microsoft network ranges to our firewall," the GMP project explains on its webpage. "We understand that we are far from the first project to take such measures against Github." The Register asked Granlund whether he was satisfied with Microsoft-GitHub's response, and he told us he had only heard once from Blacker. "I blocked about 40 IP ranges from accessing our web server," he explained. "A week after this started, there was still intensive traffic from the same IP addresses, perhaps 100 different Microsoft addresses all in all, belonging to about 40 ranges. The difference was that that traffic just caused minuscule load, and a log line in the firewall." "Problem solved. I cannot care less if they no longer can access gmplib.org. I find it interesting how little responsibility Github/Microsoft assume here. They seem to think that they are entitled to bash away at smaller sites."
The following day, Mike Blacker, director of threat hunting, operations, and response at Microsoft's GitHub, had identified the culprit: a GitHub Actions Workflow that clones a Mercurial repo and has been forked more than 700 of times. "Microsoft and GitHub have investigated the issue and determined that a GitHub user updated a script within the FFmpeg-Builds project that pulled content from https://gmplib.org," explained Blacker. "This build was configured to run parallel simultaneous tests on 100 different types of computers/architectures. This activity does not appear to be nefarious. [GMP] appears to have limited infrastructure that could not sustain the limited, yet simultaneous requests." [...]
As of last week, the excessive traffic was still an issue. "Our servers are fully available again, but that's the result of us adding all participating Microsoft network ranges to our firewall," the GMP project explains on its webpage. "We understand that we are far from the first project to take such measures against Github." The Register asked Granlund whether he was satisfied with Microsoft-GitHub's response, and he told us he had only heard once from Blacker. "I blocked about 40 IP ranges from accessing our web server," he explained. "A week after this started, there was still intensive traffic from the same IP addresses, perhaps 100 different Microsoft addresses all in all, belonging to about 40 ranges. The difference was that that traffic just caused minuscule load, and a log line in the firewall." "Problem solved. I cannot care less if they no longer can access gmplib.org. I find it interesting how little responsibility Github/Microsoft assume here. They seem to think that they are entitled to bash away at smaller sites."
This isn't (Score:4, Interesting)
Problem solved. I cannot care less if they no longer can access gmplib.org. I find it interesting how little responsibility Github/Microsoft assume here. They seem to think that they are entitled to bash away at smaller sites.
Github/Microsoft is not accessing gmplib.org. A Github user's script is, which executes as a Github action (for testing). Blocking the IP ranges on the firewall is only going to cause a problem for the user with the FFmpeg-Builds repo. It's not going to impact Microsoft or Github in any way. Blame the user, not the service provider.
The attack targets the GMP repo, with thousands of identical requests. The requests are cleverly chosen as to cause heavy system load.
It was a poorly written script by a contributor to the FFmpeg-Builds repo. The requests were not cleverly chosen at all.
Torbjorn Granlund sounds like a paranoid idiot who doesn't understand how this happened, and is misplacing blame.
Re: This isn't (Score:1)
Re:This isn't (Score:4, Insightful)
Re:This isn't (Score:4, Insightful)
"Blocking the IP ranges on the firewall is only going to cause a problem for the user with the FFmpeg-Builds repo."
Why won't it impact anyone else using github actions accessing gmplib.org?
Or if github is hosted on Azure (likely), and an entire azure data center/site or multiple of them is firewalled, the collateral damage could be any azure cloud user who access gmplib and the site? While that's not a Microsoft problem per se... it potentially affects a lot of Microsoft customers, and is absolutely problem for microsoft.
I don't think Torbjorn is entirely off base at all here. I agree "blaming microsoft" for the attack isn't quite right, but microsoft does have a responsiblity here, to ensure its cloud services isn't abused accidentally or on purpose.
Re:This isn't (Score:4, Insightful)
Torbjorn's soapbox is eye-rolling bullshit.
MS has a customer misbehaving, and needs to take care of that problem if they want to avoid people taking defensive actions against MS customers as a class. But accusing MS of "beating up on smaller sites" or what-the-fuck-ever was claimed is just some asshole with an ax to grind.
Re: (Score:2, Insightful)
But that is exactly what the Microsoft representative said:
So yes, Torbjorn has a legitimate axe to grind. Microsoft's response is irresponsible and flippant. (To them, the answer is to move GMP's hosting to azure resources and for the project to spend 10x as much money on hos
Re: (Score:2)
But that is exactly what the Microsoft representative said:
Bullshit. Shame on you.
So yes, Torbjorn has a legitimate axe to grind. Microsoft's response is irresponsible and flippant. (To them, the answer is to move GMP's hosting to azure resources and for the project to spend 10x as much money on hosting that is not necessary.)
Again, bullshit.
Microsoft's response is that the behavior does not constitute abuse, and this is a perfectly reasonable stand for MS/GH to take. Just as it is perfectly reasonable to disagree with them, and block them.
But the fact is, the behavior is not malicious, and is therefor not abuse.
The fact is, MS is right- GMP has a shit server that can't handle the load that their project demands.
They have a right to have such a setup, and the ecosystem as a whole will learn to work aro
Re: (Score:3)
"[GMP] appears to have limited infrastructure that could not sustain the limited, yet simultaneous requests." does not equal "They seem to think that they are entitled to bash away at smaller sites."
It equals, "We are entitled to nothing. We provide a service to people, and people are using said service. You provide a service, and you don't like how the people using our service are using you
Re: (Score:2)
Some guy is giving out free water on a hot day. Microsoft owns and operates a tour bus, he stop at the side of the road because someone on the bus sees water guy and everyone go out and get some free water.
Half the bus gets water, the rest get nothing and don't know why. Half the bus is pissed, the guy providing the water is pissed, everyone else in the neighborhood who usually got a single water from the guy are all also pissed, and then Microsoft has the gall to say 'Don't blame me, that guy just didn't h
Re: (Score:2)
Replace Microsoft with a family cruising in their RV with 10 kids.
You can't blame the carrier. It's fucking lazy. It is again, like blaming the USPS for your junk mail.
Re: (Score:2)
Indeed. MS does need to police their Azure customers if the want to stay off blacklists. They already have to do that for Spam anyways. Of course, MS has a history of both arrogance and incompetence, and things seem to be getting slowly worse, so they may take a while to even understand what they need to do here.
Re: (Score:2)
The behavior happening here isn't abusive, it's damaging.
If, for example, I had a residential customer doing something that I didn't consider abusive behavior, and someone on the internet gets a hold of me via my ARIN abuse contact, I'm not going to shut down my customer automatically based on that complaint. I will decide myself if my customer is being abusive, or if the problem is between my customer and th
Re:This isn't (Score:5, Insightful)
I think he understands what's happening, it's just that as far as he's concerned it doesn't matter. He can't see any difference between who caused the requests, only that they're originating from Microsoft's Github service. I'd do the same thing in his place: firewall the source and let them worry about what caused it on their side.
Microsoft here has a responsibility since it's their service being used here. Their responsibility is to identify the cause on their side (their user) and deal with it (eg. by suspending that user's access to Github Actions until he fixes his script). If they don't want to, then they can deal with the fallout from none of their users being able to access the GMP library site.
Re: (Score:3)
So, I'll grant you that in his shoes I'd do the same thing. And I spend a lot of time in his shoes. It's part of my day job, after all.
So I don't fault him for that at all.
However, he's using this as a bizarre soapbox to attack the common carrier here. And that's fucking stupid.
The corrective action taken is correct. The placement of blame and accusations of intent are ridiculous and ignorant.
Re: (Score:2)
Github is not a common carrier in any way, shape, or form.
Re: (Score:2)
Github is not a common carrier in any way, shape, or form.
Wrong.
They are not a common carrier in the legal sense, but they are in the analogous sense here.
They are merely indiscriminately allowing their customers to use their service.
Accusing MS, the owner of GitHub, of attacking you because a customer is doing something that any other customer could do on that service is analogous to accusing the USPS of attacking you with junk mail.
It is fucking stupid.
Re: (Score:3)
Let's try an analogy. An engineer wrote a script to call his mother every morning to remind her to take her meds. She has poor hearing in some frequency ranges to he scripted the reminder with 100 different pitched voices to find which one she could hear best.
The problem occurred when the engineer forgot his mother's new phone number. She had replaced her cell phone when the 3G service was discontinued and he didn't write down her new number but he remembered she lived in Oregon. So the engineer had his
Re: (Score:2)
Re: (Score:2)
Github is not a common carrier in any way, shape, or form.
Wrong.
They are not a common carrier in the legal sense, but they are in the analogous sense here.
No, they are not. Period. There is literally no way in which they act like a common carrier. Look up what those words mean, I'm tired of explaining them here.
Re: (Score:2)
I work for a common carrier (and also a private carrier- you can be both)
Since I explained the way it was meant, and since the term "common carrier" has evolved constantly for as long as it has been in use, and since we literally sit upon the cusp of hosting services being regulated as common carriers, it is appropriate to point out the ways in which they are analogous- and in this case, free-registrations organizations like GitHub that provide a service without contract or dis
Re: (Score:2)
> Github/Microsoft is not accessing gmplib.org. A Github user's script is,
And now, the 64000 question:
Who runs a Github user's script?
> Torbjorn Granlund sounds like a paranoid idiot who doesn't understand how this happened, and is misplacing blame.
You sound like an idiot who doesn't understand what it takes to run an Internet-facing server.
Re: (Score:2)
Torbjorn Granlund sounds like a paranoid idiot
If you've ever tried to report a GMP issue then you'll realise how true this observation is. He seems to treat everything, including bug reports, as a personal attack on him.
Re: (Score:3)
Github/Microsoft is not accessing gmplib.org.
No the blame is placed correctly. They're providing a service which can be induced by accident to DDOS some poor sod's server. They're also the only ones with the access to prevent it. Individual users of github don't know if they're participating in a DDOS. Github however knows how often it as a system is hitting particular URLs.
Blaming GitHub for an FFMPEG problem (Score:4, Insightful)
This is like the /. Effect. Blaming /. Because/. Users used to swamp interesting websites with traffic.
Re:Blaming GitHub for an FFMPEG problem (Score:5, Interesting)
Back in the 90s I had a website that got slashdotted.
Cost me hundreds of dollars in 90s money. Damn right I blamed /.
Re: (Score:1, Insightful)
Pain is temporary, the USD decreases in value forever.
Re: (Score:2)
Don't worry--if you had held on to that hundreds of dollars until today it would have decreased in value by nearly 60%.
Fortunately, he invested it in NFTs.
Re: (Score:3)
Re: (Score:2)
You had a site that wouldn't throttle or shut down that you didn't want to get traffic?
Re: (Score:1)
You had a site that wouldn't throttle or shut down that you didn't want to get traffic?
Many people are interested in just sharing a small piece of something, and not at all interested in becoming famous or the next great internet thing.
Also 90s, ... almost no sites throttled. Slashdot took down a shitload of the internet.
Re: (Score:2)
Dude. 1990s. It was a pentium 75 , under a gig of ram and a blog. Nobody knew what they where doing back then, let alone me We had a scandal on campus regarding something to do with Sony wanting to sue students or something like that over piracy, and someone on the club server wrote an article that got linked here and we got steamrolled.
Throttling. Did Apache back then even have that? Was it even a thing then?
Mercurial (Score:2)
MDoS (Score:2)
1. He failed.
2. He attempted to blame the victim.
3. His choice of words was derogatory and insulting.
I'd be leaving the M$ IP ranges firewalled until an apology and rectification was forthcoming.
Re: (Score:2)
Re: (Score:2)
The problem is that it could be - and if you run a web service that could flog someone else's website, you have a duty of care to make sure that any events reported to you are managed.
Blaming someone else for their website's inability to handle your sloppy code is lazy at best.
Insulting them along the way is just bullying. That never used to be M$'s style, but it is these days, and people need to call them out on it.
Re: (Score:2)
It becomes an attack when it is still going after a day or two after they have been notified of the problem.
Re: (Score:2)
Essentially, this is just some as-expected incompetent MS person not understanding the problem and hence saying stupid things. The problem is that while MS is not the root cause here, some customer of MS is. And while overloading a site by accident can happen, a sustained overload originating from Azure needs MS to take measures to rectify the problem, no excuses. Yes, a user is misbehaving on their servers, bit these are still their servers.
I had a vserver of mine used in a reflector attack. My provider ga
This is a diseasese of bad CI (Score:4, Interesting)
I've seen so many inept CI pipelines that always re-fetch artifacts, which, by their very nature, are immutable, from primary sources: going back to github repos, maven central, npm, or wherever without a local persistent cache or some kind of org-level artifact cache server. It's bad for build repeatability, bad for the integrity of the build process, and puts unnecessary load on small projects which might not have the budget for a bunch of mirrors or CDN hosting.
It's not Microsoft's fault per se, but it would be nice if they policed their users a little more aggressively about repeated egress requests from github actions.
Re: (Score:3)
I don't blame inept CI frameworks for this, I blame the developers that build those tools. Maven started this crazy idea you should always download the package by default and too many people drank the Koolaid.
I loathe maven as well, BUT:
Maven caches artifacts locally after fetching them once. ~/.m2/repository is the default location. The problem is many developers don't bother to learn how to use their CI system. The just have it execute mvn package or whatever and call it a day. Since most CIs run builds in a sandbox (a good thing), this results in everything being re-fetched by default since the maven cache will be empty. However, there is always a way to mitigate this. Either by pre-populating the local
Cache the resource (Score:3)
Instead of downloading a single resource for every build variation the script should download it once for all variations. As the resource doesn't even change between most of the build runs. They could have used GitHub's cache functionality: https://docs.github.com/en/act... [github.com]
Re: (Score:2)
The github workflows & runners infrastructure is so badly designed, badly documented (and practically impossible to debug other than by trial and error) that being able to effectively use those cache actions turns into a project by itself.
Been there, done that. Maybe it's me who am stupid. In any case, what's clear is that it's EXTRA EFFORT and most of the workflows examples that everybody is copy-pasting around don't even bother to cache anything, ever.
And BTWl, even when caching, what they practically
Undo (Score:2)
Undo moderation
Haters B Losers (Score:3)
I'm sure there all kinds of issues coming out of a higher education network like AS1653, his head would explode if Microsoft blocked all of AS1653 because of this -- but he thinks it is OK to block all of GitHub.
Github down (Score:1)