Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
Technology

A Spotify Publisher Was Down Monday Night. The Culprit? A Lapsed Security Certificate (npr.org) 41

Posted by msmash from the closer-look dept.
On Monday night, some Spotify users went to download their favorite podcasts and were met with an error. By Tuesday morning, the issue was resolved. What was the source of the massive disruption impacting some of the platform's biggest producers? An expired security certificate. From a report: The SSL security certificate is what keeps a website secure by enabling encryption, giving it the "s" in HTTPS. For Megaphone, the podcast advertising and publishing platform Spotify acquired in 2020, the certificate expired Monday evening. Shortly thereafter, publishers and listeners for Megaphone-hosted podcasts experienced service disruptions. "Megaphone experienced a platform outage due to an issue related to our SSL certificate," a Spotify spokesperson told NPR. "During the outage, clients were unable to access the Megaphone CMS and podcast listeners were unable to download podcast episodes from Megaphone-hosted publishers. Megaphone service has since been restored." The entire outage lasted for about nine hours, with Megaphone publishing real-time updates of the issue. Some podcast publishers took to Twitter to express their frustration business implications of the outage, according to Verge.
This discussion has been archived. No new comments can be posted.

A Spotify Publisher Was Down Monday Night. The Culprit? A Lapsed Security Certificate More

A Spotify Publisher Was Down Monday Night. The Culprit? A Lapsed Security Certificate

Comments Filter:

  • This wouldn't have happened if they were using Let's Encrypt.

    • Let's Encrypt is not a guarantee and the required 90 day renewal cycle is overly aggressive.

      The last time a cert expired on me, I wrote a little script that gets the expiry date of certs and sends an email reminder 7 days prior to expiry if any of them are about to expire. One of my Let's Encrypt certs almost expired one time after that. My script saw the upcoming expiring cert and notified me. No one else noticed since no outages occurred but there would have been. Let's Encrypt is an improvement but i

      • Re: (Score:2)

        by lsllll ( 830002 )
        I guess if you have your certificate on a platform that doesn't support auto-renewal, then yeah, it sucks having to update it every 85 days. But you get what you can for free. LE has great automation tools if your platform supports it.
        • Auto-renewals can be custom scripted as well. They document their APIs for such use-cases. The aggressive expiry ensures that websites that lose access to their respective domains are not able to pose as that site for long, and private keys that have been compromised are quickly mitigated.

      • This problem keeps happening, and it is somewhat baffling. When you get the cert, you know when it expires. If your company operations depend upon this, then add a note to your calendar to renew it. And don't just have a few days reminder, you can have a one month or two of overlap. Especially true if these are self-published certificates that you generate in-house, I'm baffled how those can be allowed to expire only to cause a scramble.

        This isn't really a technological problem, it's an organizational p

        • Re: (Score:2)

          by ahodgson ( 74077 )

          You can even monitor expiry dates with nagios.

        • Axway Validation Authority can monitor for expired, fraudulent, or revoked certificates. It can also be configured to monitor for soon-to-expire certificates and send email notifications to the responsible group. The defaults are 14 days before the certificates expire and notifications sent every 60 minutes, but these intervals can be configured as desired.

          Axway Certificate Expiration Warnings [axway.com]

      • Just use Certbot. You're not the first person to need LE certificates automatically renewed. Others had the same problem and it's been solved for a while.

      • Re: (Score:1)

        by room34 ( 1557423 )

        I was certainly being a bit glib in my original post, but my point was that there are really simple solutions to this problem, and yet it keeps happening through a combination of IT staff incompetence and the established CAs making cert registration and renewal way more of a convoluted process than it needs to be.

      • Re: (Score:3)

        by netik ( 141046 )

        The 90 day renewal cycle is "overly aggressive" to make sure your renewal scripts actually work!

    • Re: (Score:2)

      by lsllll ( 830002 )

      +1! I'm doing Nginx on OpnSense with ACME Client tied into my account on Let's Encrypt and don't even have to worry about renewing certs. Setting up a new server is a breeze, too: set up my DNS, create the reverse proxy in Nginx, create a record in ACME Client, have it get the cert, and bam! All I'm missing is to throw salt on it over my elbow!

      But, in reality, you can blame every mofo fear monger out there who pushed SSL, especially Apple, with their browsers not trusting any certificate that's over 13 m

    • Re: (Score:2)

      by bws111 ( 1216812 )

      Except that anyone can go to their website (https://www.megaphone.fm/) and look at the certificate, and lo-and-behold it is issued by Lets Encrypt.

    • This wouldn't have happened if they were using Let's Encrypt.

      ITYM:

      This wouldn't have happened if they weren't using certificates.

  • Major companies do this. Not some silly podcast, but things that cause other things to collapse, etc.

    /. where you hear it first for the millionth time, with more less.

  • Where everything was text based html and no 's'

  • It never ceases to amaze me that some IT people cannot even get basic stuff right. Obviously people that would have struggled to get burger-flipping right, but somehow managed to land an IT job. (With apologies to everybody that can competently flip a burger.)

  • This type of problem can exist only because somebody is making money from it not being solved. If a certificate was valid when it was pinned, there's no reason to not keep accepting it for some period after expiry. (Passports are often accepted for six months post the expiration date as an example.)

    Others have mentioned Lets Encrypt with an API and ninety day renewal which is nice.

    Of course if you pin a certificate and it gets updated, things also break. One would think that there would be a way

    • Re: (Score:2)

      by Nkwe ( 604125 )

      If a certificate was valid when it was pinned, there's no reason to not keep accepting it for some period after expiry. (Passports are often accepted for six months post the expiration date as an example.)

      It depends on what you are using the certificate for. At a high level certificates provide a public / private key pair which is used for both encryption and for identity assurance. Proper control of the private key assures that the encryption is solid and that the holder of the private key is who they say they are. If the private key is exposed or compromised, you are no longer assured that encryption will keep the communication private or that the entity on the other end of the network is who they say they

      • I can't speak for the (non-financial) concerns of the CAs, but I think there's a legitimate reason that the certificates only last a year or so. People abandon entire entities. i.e. I open up WhizBangSuper.example.com and expect to get filthy rich. A year later, I've brought in zero revenue so I don't renew the domain and end up in bankruptcy court. My servers are sold off for pennies on the dollar. Now a new entity could register the same site and use my abandoned certificate and cause calamity.

        Http

    • Re: (Score:3)

      by vux984 ( 928602 )

      "(Passports are often accepted for six months post the expiration date as an example.) "

      If you want a certificates to last for 6 months beyond its expiration date... just issue certificates that last 6 months longer in the first place.

      All a 'grace period does' on pinned certificates does is make everything more complicated (code, maintainance, validation) for no real benefit except that the date you set it to expire is not the date it "really" expires now in some cases.

      Right now a certificate is either vali

      • The *client* can pin multiple certificates but the server can't *present* multiple certificates as far as I know. i.e. here's my old certificate expiring in a week and here's the new one that will replace it. You can only present one entity certificate in the handshake.

        • Re: (Score:2)

          by vux984 ( 928602 )

          Right, ok... I'm not seeing why you want the server to present two?

          Are you thinking the client should 'update' its pinning automatically, if the server presents a pinned cert, and then says ... here's the coming replacement, update your pin list.

          That's an interesting suggestion, but I'm not sure many clients that use certificate pinning would want to risk the possibility of automatically pinning a malicious cert -- what if the server had been compromised and the hackers added their own certs -- you wouldn't

          • The pinning doesn't have to be automatic. Right now, when the new cert is presented, the connection fails. Then the client is forced to make a decision and the only available information is the *new* cert. If the new cert is presented as part of a handshake where the known-good certificate actually secures the connection, the client has better information upon which to base a decision. Right now you when the pinned cert updates, the client sees a new certificate that's valid for the host, and somebody ha

            • Re: (Score:2)

              by vux984 ( 928602 )

              " the client has better information upon which to base a decision"

              Information yes, but not necessarily good information.

              You know you are talking to the same server, but you don't know if its compromised and offering you "evil" certs.

              Most pinning isn't appropriate anyway in my opinion and can do as much harm as good. And where pinning is appropriate the pin list is coming from a secondary channel (e.g. as part of the client software updates) rather than directly from the server offering the certificate.

              • Right but the certificate pinning isn't about whether or not the server in question has been compromised. If it has, all bets are kind of off. The point of certificate pinning is to ensure that nobody can fraudulently obtain a certificate for the target server and use that to somehow trick you. If they actually compromise the target server, that's a bigger payload.
              • And yes the pinning comes through a third-party channel. But it wouldn't have to if the server had a way to present updates which was kind of my point.

  • "Sorry, your toaster oven isn't working this morning. We didn't patch the security hole in it's remote access protocol and someone was mining crypto with it, so we remotely disabled all the toasters. Maybe you'd like to try cereal this morning?"

Slashdot Top Deals

news: gotcha

Close