Microsoft Teams Went Down After Microsoft Forgot To Renew a Critical Certificate (theverge.com) 72
An anonymous reader quotes a report from The Verge: Microsoft Teams went down this morning for nearly three hours after Microsoft forgot to renew a critical security certificate. Users of Microsoft's Slack competitor were met with error messages attempting to sign into the service on Monday morning, with the app noting it had failed to establish an HTTPS connection to Microsoft's servers. Microsoft confirmed the Teams service was down just after 9AM ET today, and then later revealed the source of the issue. "We've determined that an authentication certificate has expired causing users to have issues using the service," explains Microsoft's outage notification. Microsoft then started rolling the fix out at 11:20AM ET, and by 12PM ET the service was restored for most affected users.
The error is... (Score:3, Insightful)
that Teams is forced on every computer with Windows 10.
Re: (Score:2)
I thought that Skype was still the messenger of choice that they were trying to cram down everyone throats by installing it by default in Windows 10.
Do they do the same thing with Teams now as well, or does that one just come along for the ride with an Office 365 installation?
Re: The error is... (Score:2)
Not that killing Slack would be a bad thing. (Score:3)
The only question would be: *How* brutally? :D
But yeah. Microsoft being Microsoft.
Makes it hilarious when kids always say that we should stop mistrusting MS or that they've changed.
Corporate culture doesn't change that quickly. Tgat takes centuries! Or massive actual changes like being bought and everybody sustaining that culture being fired.
I hope MS kills itself in the process. (Let a man dream. :)
Re: (Score:3)
Teams is supposedly replacing Skype for Business. Which sucks, since as buggy as Skype is, at least it's useful when it does work unlike Teams.
Re: (Score:3)
My company did the changeover. There was a 3-day flurry of questions along the lines of "How do I... like in Skype?", then it went quiet. Once a week or so, someone posts a cat giphy, but business is back to email and phone calls.
Re: (Score:2)
We had some meetings with Teams, they didn't go well.
Re:The error is... (Score:4, Insightful)
that Teams is forced on every computer with Windows 10.
no, it isn't.
Re: (Score:3, Insightful)
Re:The error is... (Score:4, Informative)
That must be caused by AD group policy or something. MS Teams is not installed on any of my Windows 10 machines.
Re: The error is... (Score:1)
Re:The error is... (Score:4, Informative)
Attempting to uninstall it
It's never installed in the first place. It's not part of Windows 10, it's not a default app for Office 365. It sounds like you are using a business machine with a pushed application.
Across all of my Windows 10 machines running various versions (home, pro, LTSB, enterprise) the only system which has teams installed is the enterprise one and on that I had to download it manually (though since then my company has added it to the default software list meaning it gets pushed).
Re: (Score:2)
Let's be honest... (Score:5, Insightful)
In the real world, it was never secure in the first place. Here's a list of entities you have to trust absolutely blindly, to trust certificates: :D)
* Whoever has access to the hardware/building.
* Your hardware manufacturers and the sourcr you got it from.
* Your OS maker, and the source you got it from.
* Your browser maker, and the site you got it from. (The latter being trusted by OS-supplied root cert list.)
* The CA. (Trusted by browser-supplied root cert list.)
* The website (Trusted by the CA.)
* Your sanity. (Probably not sane if you think that the above result in a trustworthy chain.
Re:The error is... (Score:5, Informative)
No it's not. Teams is not part of any version of Windows 10.
Teams is part of a couple of Office subscriptions though, but even then all you get is an account, you still need to chose to install it.
It was clearly posted (Score:5, Funny)
Re: It was clearly posted (Score:2)
Re: It was clearly posted (Score:2)
Re: (Score:1)
They gave the responsibility to some contractor who was sent out on to their 90 day "no workie for Microsoft" plan on Christmas eve.
Re: (Score:1)
Some code in the future. Some code in the distance past.
I have teams installed, but never have used it (Score:1)
Meanwhile... (Score:3)
I got an email reminder yesterday from Let's Encrypt that my certificate was expiring in 20 days, and proceeded to SSH into the machine that handles the reverse proxy and renew it, the whole process taking less than 10 minutes.
Why does this seem to be a repeat issue with this larger companies, that have SSL certs with longer lifespans and calendaring products that could remind them of renewal?
Re: (Score:3)
Meanwhile, I run certbot from cron and never have to deal with it at all.
Re: (Score:3)
Sometimes that's not an option. For instance, I use DNS validation with my home server because (1) Cox blocks inbound traffic on port 80 and (2) Let's Encrypt doesn't allow HTTP validation to proceed on other ports.
Re: (Score:2)
You're still doing it wrong then - that too can be done from automated scripts. I have a home server that uses DNS validation for a cert - it's a custom script that can follow a CNAME to a zone that takes RFC2136 dynamic updates.
At work, I have over 120 Let's Encrypt certs, mostly with DNS validation, all managed from a single server (it renews them and then pushes them out to the various places they need to be, using Ansible playbooks or custom scripts for some odd-ball things that don't map well onto Ansi
Re: (Score:2)
I tried doing that when I first got the certificate set up, and the certbot task didn't seem to work for some reason. The dry run and logs all implied the task was running right, but when the actual renewal came due it didn't get renewed. I'm sure I can get the problem figured out if I revisit it, but since it only comes up once every 90 days I've been lazy.
Re: Meanwhile... (Score:2)
... or cert-manager in your k8s clusters...
Re: (Score:3)
Or if it is automated, the automation somehow failed.
Re: (Score:2)
If they's been in AWS, Amazon would have renewed it automatically for them ;-)
Re:Meanwhile... (Score:5, Interesting)
Because the longer lifecycles mean the process does not get used much. People change jobs or leave the company and now the email address is no longer valid. Or the mail goes to multiple people and each assumes someone else is taking care of it. Lot's of reasons things in little-used processes fall through the cracks.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Why does this seem to be a repeat issue with this larger companies, that have SSL certs with longer lifespans and calendaring products that could remind them of renewal?
Because they're too big to even have a BOFH anymore.
Re: (Score:2)
That's the problem, right there. "Oh, the cert's not going to expire for 10 years. We've got plenty of time to set things up for the renewal." Then they forget about it.
Re:Meanwhile... (Score:5, Interesting)
Re: (Score:1)
This isn't just HTTPS for a public website, there are multiple types of SSL certificates and LetsEncrypt doesn't solve every situation.
Yes, they should've automated this but most likely this was a self-signed SSL cert 10 years ago and the person responsible has long left the organization and never documented it.
Re: (Score:2)
and proceeded to SSH into the machine that handles the reverse proxy and renew it
Oh that's dangerous. It sounds like you have a setup that would lead exactly to the kind of error that can land you with an expired certificate. User interaction to renew the certificate? Horrible process for anything even remotely important.
Why have you not automated it?
Re: (Score:2)
Why have you not automated it?
I tried and it didn't work out. I got the email saying my certificate was due for renewal and I waited, expecting the chron job to work properly.
I'll give it another shot at some point here, but considering it would take longer for me to hunt down the reason the automated task didn't work than it does to just manually renew it's not exactly a priority.
Complexity. (Score:2)
This is analogous to the cube-square law and to multitasking: The bigger a system gets, the more resources are wasted simply on administration. Or more precisely: A graph's edges between nodes scale faster than the nodes themselves, in any natural system.
That is why big companies wast so much of of their time with meetings and such. And why big states grow disorganized. The figurative left hand literally doesn't know what the figurative right hand does.
The solution is a fractal compartmentalization. Managab
12PM ET? (Score:1, Troll)
What time is that? There is no such time.
It's either 12 noon or 12 midnight.
A guy got out of a parking ticket because the sign said no parking between 8 AM and 12 PM and he argued, correctly, there is no such time. AM (Ante meridiem) means before noon. PM (Post meridiem) means after noon. Therefore, there cannot be a time as 12 AM or 12 PM.
Re: (Score:1)
Thank you Mr. Pedantic Asshole.
In normally accepted usage by pretty much everyone, 12 Noon is 12pm, and 12 midnight is 12 am.
Re: (Score:1)
That the judge wasn't sure and so tossed the ticket is not surprising. Happens every day. So if your story was told correctly, it would be reasonable.
However, you're simply wrong; this is not undefined. 12PM is noon, 12AM is midnight.
And etymology does not determine the meaning, so blathering about post meridiem is dumb.
This looks *really* wrong to me. (Score:2)
Forgive my German origins, but what you are saying, sounds to me, as if your hours were ordered as 12 1 2 3 4 5 6 7 8 9 10 11 for both AM and PM. With 12 AM being 00:00 and 12 PM being 12:00.
But the number 12 clearly belongs to the end of the series! So 12 PM should come after 11 PM! Not after 11 AM and before 1 PM.
Or: Just replace 12 with 0 (zero), for it to make sense. ;) *ducks* :D
Or use a 24-hour clock like civilized people.
(Don't look at me. I have a binary-coded hexadecimal clock showing the fraction
Re: (Score:2)
Don't blame Germany, there are smart Germans all over the world which disproves that explanation for what you said.
Re: (Score:2, Informative)
Re: (Score:1)
In other words, 12 < 1. This makes four discontiguous periods within a single day. Yay.
Re: (Score:2, Interesting)
What time is that? There is no such time.
It's either 12 noon or 12 midnight.
A guy got out of a parking ticket because the sign said no parking between 8 AM and 12 PM and he argued, correctly, there is no such time. AM (Ante meridiem) means before noon. PM (Post meridiem) means after noon. Therefore, there cannot be a time as 12 AM or 12 PM.
A guy? Provide a citation please...
Re: (Score:2, Informative)
A guy? Provide a citation please...
It was an article in the paper I cut out right around the time I started to drive and kept it with me for decades, but apparently threw out at some point because it's not where I thought it should be. The basic story was the guy got a parking ticket. He contested the ticket because the sign said no parking between 8 AM - 12 PM. He argued there is no such time as 12 PM, only 12 noon or simply noon. The judge agreed, dismissed the ticket and the city had to have new signs
Re: (Score:1)
Re: (Score:1)
What time is that? There is no such time.It's either 12 noon or 12 midnight.A guy got out of a parking ticket because the sign said no parking between 8 AM and 12 PM
What other things do you believe?
Re: (Score:2)
What time is that? There is no such time.
It's either 12 noon or 12 midnight.
No. 12 noon = 12:00PM. 12 midnight = 12:00AM
You not understanding common terminology used in the entire world which uses 12 hour clocks does not mean that something doesn't exist. Do you not own a watch? Or do you not own a computer? Both of those will happily show you the time does in fact exist, twice a day even.
Re:12PM ET? (Score:4, Informative)
From Europe, I am looking amused at how you manage to divide a 24-hour day in 12 hours, and proceed to fight about how to label the first and second batch of hours, instead of employing the obvious solution: 0:00 is midnight, 12:00 is noon.
This has to be added to the list after Imperial units and before month-day-year notation.
Does this mean... (Score:2)
Re: (Score:2)
Very strange that I have been using Windows 10 for some time now and never got this pop up. I've never even heard of "Microsoft Teams" until I came across this article.
Cloud is not cathartic (Score:1)
One "problem" with the cloud is that if the hoster bleeps up, you can't really fire them. If an employee did this and it brought an org to its knees, they'd be fired and all the executives would feel better.
Why is there expiration in the first place? (Score:3, Insightful)
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
Certs could have been handled with pinning, web of trust, out of band authentication, and revocation.
But that wouldn't have allowed a scarce resource to be created out of thin air and wouldn't have created an artificial need for that resource.
Re: (Score:3, Interesting)
Without expiration, the whole certificate system (PKI) would become useless.
With indefinitely valid certificates in play, you will never be able to be sure it isn't compromised. One successful social engineering attack and it won't take that much effort anymore to impersonate the certificate owner.
With expiring certificates, keys are getting renewed, which makes you able to trust it again.
In my experience, if you buy a certificate from a vendor, it is usually valid for a year. Well, 365 days, so a leap year
Re:Why is there expiration in the first place? (Score:5, Interesting)
Forgive me, as my understanding of PKI is admittedly quite limited, but I was under the impression that was what revocation was for. If someone gets your signing key, you let the CA know and they revoke the key and you get a new one. An expiration date wouldn't really solve the problem, either, at least not until the expiration date hits. That's still up to a year where someone might be using your signing key, which isn't ideal. I guess it does limit how long they can impersonate you if you aren't paying attention for some reason, but that's really all I can think of.
Re: (Score:2)
that was what revocation was for.
Yes, if you ever figure out that you're compromised and have someone who knows what to do about it, can get the revokation published properly, all the clients properly honour the revokation and you're not stuck in "someone needs to sign this but nobody knows who" hell for two weeks.
Like many of the "after time X" things in security (passwords, policy reviews, etc.) certificate expiration is the thing we put in place because we know that too many things can and will go wrong for revocation to really work as
Re: (Score:2)
This all falsely implies ... (Score:2)
... that CAs are trustworthy in the first place.
Have you ever even met a CA? (He said, acting as if it was a single person.)
Or the browser that simply trusted them without asking you.
The only case where it makes sense, is for internal CAs and root certs at a company.
Re: (Score:2)
I don't see any reason why a valid certificate would be invalid after some time. Is your password expiring?
There's good technical reasons for it. Comparing it to passwords is leading you down an incorrect through process.
Passwords expiring were to lead people to change passwords as these get used, guessed, shared and the thought was by rotating them there are less people likely to know it at any given moment. It was an incorrect assumption.
For certificates the reason they expire is to force currency (not currency as in money). Encryption standards change and evolve continuously. If we had 20 year certificates y
Aaand everybody deserved it. (Score:1)
It's simply natural selection. Everybody mentally retarded enough to fall for the "cloud", or makes himself dependend on outsourcing in general, will suffer and die out.