Catch up on stories from the past week (and beyond) at the Slashdot story archive

 



Forgot your password?
typodupeerror
×
Microsoft Communications Security Social Networks The Internet

Microsoft Teams Went Down After Microsoft Forgot To Renew a Critical Certificate (theverge.com) 72

An anonymous reader quotes a report from The Verge: Microsoft Teams went down this morning for nearly three hours after Microsoft forgot to renew a critical security certificate. Users of Microsoft's Slack competitor were met with error messages attempting to sign into the service on Monday morning, with the app noting it had failed to establish an HTTPS connection to Microsoft's servers. Microsoft confirmed the Teams service was down just after 9AM ET today, and then later revealed the source of the issue. "We've determined that an authentication certificate has expired causing users to have issues using the service," explains Microsoft's outage notification. Microsoft then started rolling the fix out at 11:20AM ET, and by 12PM ET the service was restored for most affected users.
This discussion has been archived. No new comments can be posted.

Microsoft Teams Went Down After Microsoft Forgot To Renew a Critical Certificate

Comments Filter:
  • The error is... (Score:3, Insightful)

    by hiroshimarrow ( 5489734 ) on Monday February 03, 2020 @05:14PM (#59686860)

    that Teams is forced on every computer with Windows 10.

    • by leonbev ( 111395 )

      I thought that Skype was still the messenger of choice that they were trying to cram down everyone throats by installing it by default in Windows 10.

      Do they do the same thing with Teams now as well, or does that one just come along for the ride with an Office 365 installation?

      • It comes along with office 365, Theyâ(TM)re doing the same thing they always do... giving stuff away for free to kill a competitor, in this case slack.
        • The only question would be: *How* brutally? :D

          But yeah. Microsoft being Microsoft.
          Makes it hilarious when kids always say that we should stop mistrusting MS or that they've changed.
          Corporate culture doesn't change that quickly. Tgat takes centuries! Or massive actual changes like being bought and everybody sustaining that culture being fired.

          I hope MS kills itself in the process. (Let a man dream. :)

      • Teams is supposedly replacing Skype for Business. Which sucks, since as buggy as Skype is, at least it's useful when it does work unlike Teams.

        • by sasdrtx ( 914842 )

          My company did the changeover. There was a 3-day flurry of questions along the lines of "How do I... like in Skype?", then it went quiet. Once a week or so, someone posts a cat giphy, but business is back to email and phone calls.

    • Re:The error is... (Score:4, Insightful)

      by Kwirl ( 877607 ) <kwirlkarphys@gmail.com> on Monday February 03, 2020 @06:17PM (#59687078)

      that Teams is forced on every computer with Windows 10.

      no, it isn't.

      • Re: (Score:3, Insightful)

        by troff ( 529250 )
        Attempting to uninstall it results in it being reinstalled on the next reboot. You have to uninstall it AND the unannounced bootstrapper in order to get rid of it. That sounds forced to me.
        • Re:The error is... (Score:4, Informative)

          by _merlin ( 160982 ) on Monday February 03, 2020 @07:44PM (#59687426) Homepage Journal

          That must be caused by AD group policy or something. MS Teams is not installed on any of my Windows 10 machines.

        • Re:The error is... (Score:4, Informative)

          by thegarbz ( 1787294 ) on Tuesday February 04, 2020 @02:19AM (#59688106)

          Attempting to uninstall it

          It's never installed in the first place. It's not part of Windows 10, it's not a default app for Office 365. It sounds like you are using a business machine with a pushed application.

          Across all of my Windows 10 machines running various versions (home, pro, LTSB, enterprise) the only system which has teams installed is the enterprise one and on that I had to download it manually (though since then my company has added it to the default software list meaning it gets pushed).

    • The error is that certificates are treated as if they're 100% totally secure and valid at 23:59:59 and totally insecure and compromised at 00:00:01 even though nothing at all has changed except that the CA's yearly permission-to-continue-running fee hasn't been paid. Given the endless PKI-based outages we've seen coming up again and again and again, you'd think the people writing the PKI software would implement a soft-fail rather than cutting things off dead when you forget to pay your CA for permission t
      • Let's be honest... (Score:5, Insightful)

        by BAReFO0t ( 6240524 ) on Tuesday February 04, 2020 @08:22AM (#59688706)

        In the real world, it was never secure in the first place. Here's a list of entities you have to trust absolutely blindly, to trust certificates:
        * Whoever has access to the hardware/building.
        * Your hardware manufacturers and the sourcr you got it from.
        * Your OS maker, and the source you got it from.
        * Your browser maker, and the site you got it from. (The latter being trusted by OS-supplied root cert list.)
        * The CA. (Trusted by browser-supplied root cert list.)
        * The website (Trusted by the CA.)
        * Your sanity. (Probably not sane if you think that the above result in a trustworthy chain. :D)

    • Re:The error is... (Score:5, Informative)

      by thegarbz ( 1787294 ) on Tuesday February 04, 2020 @02:16AM (#59688088)

      No it's not. Teams is not part of any version of Windows 10.

      Teams is part of a couple of Office subscriptions though, but even then all you get is an account, you still need to chose to install it.

  • by fahrbot-bot ( 874524 ) on Monday February 03, 2020 @05:15PM (#59686868)
    "Renew Security Certificate" was clearly posted on the calendar for Feb 3, 2020 in the Security Team channel -- oh wait...
  • Forced installed of course, I went to school with a guy that worked on developing teams, he was canned shortly after it's launch, something tells me he wasn't alone
  • by SeaFox ( 739806 ) on Monday February 03, 2020 @05:20PM (#59686894)

    I got an email reminder yesterday from Let's Encrypt that my certificate was expiring in 20 days, and proceeded to SSH into the machine that handles the reverse proxy and renew it, the whole process taking less than 10 minutes.

    Why does this seem to be a repeat issue with this larger companies, that have SSL certs with longer lifespans and calendaring products that could remind them of renewal?

    • Meanwhile, I run certbot from cron and never have to deal with it at all.

      • by ncc74656 ( 45571 ) *

        Meanwhile, I run certbot from cron and never have to deal with it at all.

        Sometimes that's not an option. For instance, I use DNS validation with my home server because (1) Cox blocks inbound traffic on port 80 and (2) Let's Encrypt doesn't allow HTTP validation to proceed on other ports.

        • by Burdell ( 228580 )

          You're still doing it wrong then - that too can be done from automated scripts. I have a home server that uses DNS validation for a cert - it's a custom script that can follow a CNAME to a zone that takes RFC2136 dynamic updates.

          At work, I have over 120 Let's Encrypt certs, mostly with DNS validation, all managed from a single server (it renews them and then pushes them out to the various places they need to be, using Ansible playbooks or custom scripts for some odd-ball things that don't map well onto Ansi

      • by SeaFox ( 739806 )

        I tried doing that when I first got the certificate set up, and the certbot task didn't seem to work for some reason. The dry run and logs all implied the task was running right, but when the actual renewal came due it didn't get renewed. I'm sure I can get the problem figured out if I revisit it, but since it only comes up once every 90 days I've been lazy.

      • ... or cert-manager in your k8s clusters...

    • They most likely get renewal reminders, but they have quite a lot of certificates to manage (most likely in the thousands), so if this is a manual process, once in a while, one is missed.

      Or if it is automated, the automation somehow failed.
    • Re:Meanwhile... (Score:5, Interesting)

      by bws111 ( 1216812 ) on Monday February 03, 2020 @05:33PM (#59686942)

      Because the longer lifecycles mean the process does not get used much. People change jobs or leave the company and now the email address is no longer valid. Or the mail goes to multiple people and each assumes someone else is taking care of it. Lot's of reasons things in little-used processes fall through the cracks.

      • I got to see the groundskeeper "to-do" book at Oxford University once. It lists the tasks that the groundskeeper needs to do each year (like "replace roof on building X" or "inspect plumbing for building Y"). It's a handwritten book on a 100-year cycle. If a new task gets added that will need to be dealt with decades hence, the current groundskeeper writes it in the book on the pages for that future year. Seems like a company should be able to do even better these days, no matter how large the corporation g
        • by ffkom ( 3519199 )
          You would be surprised how many companies nowadays delete all records after 3 years just out of fear they could be used against them is some later legal confrontation. Having a 100-year audit trail of things that should have been done, might have been missed, and now somebody sues because a root brick fell on his head - that is the nightmare of today's MBAs, they do not care whether the root leaks, though.
    • Why does this seem to be a repeat issue with this larger companies, that have SSL certs with longer lifespans and calendaring products that could remind them of renewal?

      Because they're too big to even have a BOFH anymore.

    • Why does this seem to be a repeat issue with this larger companies, that have SSL certs with longer lifespans

      That's the problem, right there. "Oh, the cert's not going to expire for 10 years. We've got plenty of time to set things up for the renewal." Then they forget about it.

    • Re:Meanwhile... (Score:5, Interesting)

      by ffkom ( 3519199 ) on Monday February 03, 2020 @06:32PM (#59687156)
      I can say that from my first hand experience, certificate renewal worked just fine when we were a company of 20 to 200 people. But about two years ago, with then several thousand people (and still the same number of certificates for many web sites), the one person who was previously responsible for certificate management got laid off to improve profitability, and ever since certificate renewal has been a mess and missed the dead line several times. Since no customer was able to claim compensation for any outages, though, management says this was the right decision - lower quality, less cost, more profit. The same kind of decision is taken every day in yet another company.
    • by guruevi ( 827432 )

      This isn't just HTTPS for a public website, there are multiple types of SSL certificates and LetsEncrypt doesn't solve every situation.

      Yes, they should've automated this but most likely this was a self-signed SSL cert 10 years ago and the person responsible has long left the organization and never documented it.

    • and proceeded to SSH into the machine that handles the reverse proxy and renew it

      Oh that's dangerous. It sounds like you have a setup that would lead exactly to the kind of error that can land you with an expired certificate. User interaction to renew the certificate? Horrible process for anything even remotely important.

      Why have you not automated it?

      • by SeaFox ( 739806 )

        Why have you not automated it?

        I tried and it didn't work out. I got the email saying my certificate was due for renewal and I waited, expecting the chron job to work properly.

        ...and I ended up suddenly unable to connect to my Airsonic server, among other things, due to an expired certificate.

        I'll give it another shot at some point here, but considering it would take longer for me to hunt down the reason the automated task didn't work than it does to just manually renew it's not exactly a priority.

    • This is analogous to the cube-square law and to multitasking: The bigger a system gets, the more resources are wasted simply on administration. Or more precisely: A graph's edges between nodes scale faster than the nodes themselves, in any natural system.

      That is why big companies wast so much of of their time with meetings and such. And why big states grow disorganized. The figurative left hand literally doesn't know what the figurative right hand does.

      The solution is a fractal compartmentalization. Managab

  • 12PM ET? (Score:1, Troll)

    by quonset ( 4839537 )

    What time is that? There is no such time.

    It's either 12 noon or 12 midnight.

    A guy got out of a parking ticket because the sign said no parking between 8 AM and 12 PM and he argued, correctly, there is no such time. AM (Ante meridiem) means before noon. PM (Post meridiem) means after noon. Therefore, there cannot be a time as 12 AM or 12 PM.

    • Thank you Mr. Pedantic Asshole.

      In normally accepted usage by pretty much everyone, 12 Noon is 12pm, and 12 midnight is 12 am.

    • That the judge wasn't sure and so tossed the ticket is not surprising. Happens every day. So if your story was told correctly, it would be reasonable.

      However, you're simply wrong; this is not undefined. 12PM is noon, 12AM is midnight.

      And etymology does not determine the meaning, so blathering about post meridiem is dumb.

      • Forgive my German origins, but what you are saying, sounds to me, as if your hours were ordered as 12 1 2 3 4 5 6 7 8 9 10 11 for both AM and PM. With 12 AM being 00:00 and 12 PM being 12:00.
        But the number 12 clearly belongs to the end of the series! So 12 PM should come after 11 PM! Not after 11 AM and before 1 PM.

        Or: Just replace 12 with 0 (zero), for it to make sense.
        Or use a 24-hour clock like civilized people. ;) *ducks* :D

        (Don't look at me. I have a binary-coded hexadecimal clock showing the fraction

        • Don't blame Germany, there are smart Germans all over the world which disproves that explanation for what you said.

    • Re: (Score:2, Informative)

      by troff ( 529250 )
      "12:00PM" refers to the entire minute that happens after the instant of it becoming post meridiem. Bullshit, "There is no such time".
    • Re: (Score:2, Interesting)

      by blahbooboo ( 839709 )

      What time is that? There is no such time.

      It's either 12 noon or 12 midnight.

      A guy got out of a parking ticket because the sign said no parking between 8 AM and 12 PM and he argued, correctly, there is no such time. AM (Ante meridiem) means before noon. PM (Post meridiem) means after noon. Therefore, there cannot be a time as 12 AM or 12 PM.

      A guy? Provide a citation please...

      • Re: (Score:2, Informative)

        by quonset ( 4839537 )

        A guy? Provide a citation please...

        It was an article in the paper I cut out right around the time I started to drive and kept it with me for decades, but apparently threw out at some point because it's not where I thought it should be. The basic story was the guy got a parking ticket. He contested the ticket because the sign said no parking between 8 AM - 12 PM. He argued there is no such time as 12 PM, only 12 noon or simply noon. The judge agreed, dismissed the ticket and the city had to have new signs

    • No, due to quantum uncertainty there is no such thing as exactly 12 noon or exactly 12 midnight.
    • What time is that? There is no such time.It's either 12 noon or 12 midnight.A guy got out of a parking ticket because the sign said no parking between 8 AM and 12 PM

      What other things do you believe?

    • What time is that? There is no such time.

      It's either 12 noon or 12 midnight.

      No. 12 noon = 12:00PM. 12 midnight = 12:00AM

      You not understanding common terminology used in the entire world which uses 12 hour clocks does not mean that something doesn't exist. Do you not own a watch? Or do you not own a computer? Both of those will happily show you the time does in fact exist, twice a day even.

    • Re:12PM ET? (Score:4, Informative)

      by orzetto ( 545509 ) on Tuesday February 04, 2020 @07:13AM (#59688564)

      From Europe, I am looking amused at how you manage to divide a 24-hour day in 12 hours, and proceed to fight about how to label the first and second batch of hours, instead of employing the obvious solution: 0:00 is midnight, 12:00 is noon.

      This has to be added to the list after Imperial units and before month-day-year notation.

  • Does this mean that computers with Windows 10 no longer have an annoying-ass popup imploring people to get Teams whenever they log in?
    • Very strange that I have been using Windows 10 for some time now and never got this pop up. I've never even heard of "Microsoft Teams" until I came across this article.

  • One "problem" with the cloud is that if the hoster bleeps up, you can't really fire them. If an employee did this and it brought an org to its knees, they'd be fired and all the executives would feel better.

  • by JcMorin ( 930466 ) on Monday February 03, 2020 @07:07PM (#59687310)
    I don't know who came up with this solution but it's probably for making more money selling more certificate. I don't see any reason why a valid certificate would be invalid after some time. Is your password expiring? Study show it's even usage to force people to change their password? Is PGP key expiring? All that is pure crap and waste of time!
    • Why expiration? So they can rip you off hundreds of dollars per certificate each and every year just to sign a few bits.
    • Older certs support older and more easily compromised encryption standards.
    • by sjames ( 1099 )

      Certs could have been handled with pinning, web of trust, out of band authentication, and revocation.

      But that wouldn't have allowed a scarce resource to be created out of thin air and wouldn't have created an artificial need for that resource.

    • Re: (Score:3, Interesting)

      by Shades72 ( 6355170 )

      Without expiration, the whole certificate system (PKI) would become useless.

      With indefinitely valid certificates in play, you will never be able to be sure it isn't compromised. One successful social engineering attack and it won't take that much effort anymore to impersonate the certificate owner.

      With expiring certificates, keys are getting renewed, which makes you able to trust it again.

      In my experience, if you buy a certificate from a vendor, it is usually valid for a year. Well, 365 days, so a leap year

      • by twocows ( 1216842 ) on Tuesday February 04, 2020 @08:40AM (#59688750)

        One successful social engineering attack and it won't take that much effort anymore to impersonate the certificate owner.

        Forgive me, as my understanding of PKI is admittedly quite limited, but I was under the impression that was what revocation was for. If someone gets your signing key, you let the CA know and they revoke the key and you get a new one. An expiration date wouldn't really solve the problem, either, at least not until the expiration date hits. That's still up to a year where someone might be using your signing key, which isn't ideal. I guess it does limit how long they can impersonate you if you aren't paying attention for some reason, but that's really all I can think of.

        • by Tom ( 822 )

          that was what revocation was for.

          Yes, if you ever figure out that you're compromised and have someone who knows what to do about it, can get the revokation published properly, all the clients properly honour the revokation and you're not stuck in "someone needs to sign this but nobody knows who" hell for two weeks.

          Like many of the "after time X" things in security (passwords, policy reviews, etc.) certificate expiration is the thing we put in place because we know that too many things can and will go wrong for revocation to really work as

        • by ei4anb ( 625481 )
          It is to ensure that key lengths are re-evaluated every year or two to keep pace with Moore's law of key cracking and also to ensure revenue stream for the vendor :-)
      • ... that CAs are trustworthy in the first place.

        Have you ever even met a CA? (He said, acting as if it was a single person.)

        Or the browser that simply trusted them without asking you.

        The only case where it makes sense, is for internal CAs and root certs at a company.

    • I don't see any reason why a valid certificate would be invalid after some time. Is your password expiring?

      There's good technical reasons for it. Comparing it to passwords is leading you down an incorrect through process.

      Passwords expiring were to lead people to change passwords as these get used, guessed, shared and the thought was by rotating them there are less people likely to know it at any given moment. It was an incorrect assumption.

      For certificates the reason they expire is to force currency (not currency as in money). Encryption standards change and evolve continuously. If we had 20 year certificates y

  • It's simply natural selection. Everybody mentally retarded enough to fall for the "cloud", or makes himself dependend on outsourcing in general, will suffer and die out.

With your bare hands?!?

Working...