DigiCert Revoking Certs With Less Than 24 Hours Notice (digicert.com) 61
In an incident report today, DigiCert says it discovered that some CNAME-based validations did not include the required underscore prefix, affecting about 0.4% of their domain validations. According to CA/Browser Forum (CABF) rules, certificates with validation issues must be revoked within 24 hours, prompting DigiCert to take immediate action. DigiCert says impacted customers "have been notified." New submitter jdastrup first shared the news, writing: Due to a mistake going back years that has recently been discovered, DigiCert is required by the CABF to revoke any certificate that used the improper Domain Control Validation (DCV) CNAME record in 24 hours. This could literally be thousands of SSL certs. This could take a lot of time and potentially cause outages worldwide starting July 30 at 19:30 UTC. Be prepared for a long night of cert renewals. DigiCert support line is completely jammed.
Crowdstrike #2? (Score:2)
Is this going to be like Crowdstrike, but with webservers?
Re: (Score:1)
Probably not. There's much more competition in the digital certificate market and DigiCert's slice of it is miniscule.
Re: (Score:2)
Thanks - I certainly hope so. :-)
Re: (Score:2)
More like secure boot. PKI is kind of going to hell lately because of all the bean counters who modified the principles on how to apply it properly and took all kinds of silly shortcuts thinking; nobody will ever notice the difference.
As an early adopter of RSA in ~1980:
https://en.wikipedia.org/wiki/... [wikipedia.org]
I kind of saw that coming ~2000 since same shortcuts were taken in projects I was working on, despite the architect team wishes.
"In a public-key cryptosystem, the encryption key is public and distinct from th
Re: (Score:3)
Worth noting that Crowdstrike has plenty of competition and their market share is also miniscule. The question is, whether their market share happens to include something critical that affects a lot of people (like say... Airlines). I suspect if flights weren't affected by Crowdstrike it would have been just as newsworthy as Crowdstrike's previous issues (which were reported only in tech rags rather than every news agency on the planet).
Re: (Score:3)
Crowdstike has something like 80% of the F500. The market share might be small if you define the market place as all EDR/XDR products, but if you look at EDR/XDR for the large enterprise, CS is approaching natural monopoly status.
Re: (Score:2)
Crowdstike has something like 80% of the F500. The market share might be small if you define the market place as all EDR/XDR products, but if you look at EDR/XDR for the large enterprise, CS is approaching natural monopoly status.
This brings an interesting discussion. We mostly look at DR/BCP risk and mitigation within a single corporation or enterprise, but should we be considering the aggregate risk and impact with the entire F500 being an entity to evaluate risk? What is the greater impact if F500 as a whole is impacted by an outage?
Maybe these are the kinds of discussions that need to happen in order to open up the desktop to something other than Microsoft and Microsoft. Which is THE eggs-in-one-basket problem.
Re: (Score:2)
Re: (Score:2)
No. The numbers are not comparable and machines are still accessible. Also, it is pretty easy to get a cert from another source.
Re: (Score:2)
Depends how the certs were distributed. Ideally the update should be automatic.
single source is fine (Score:3)
Well, we've known for over 30 years that having multiple parties attest to a key's owner's identity is really easy (PGP could do it) so I'm sure one little provider's problem won't cause anything widespread. Surely a modern, fundamental protocol like HTTPS doesn't rely on anything with quaint limitations.
Re:single source is fine (Score:4, Interesting)
This has been a head-scratcher for me since SSL was invented. Why does it have just one root key, and no more? Why can't I get a certificate for my TLS terminator (be it a reverse proxy, load balancer, or the actual program) that is signed by multiple vendors? This way, if one vendor has issues, it will still work. This redundancy would save businesses a ton of time and man-hours. Especially if something happens and someone forgets to renew.
We have redundant subsystems for everything else... but only one root CA per SSL/TLS cert.
This also will make life easier, as I could have the cert renewed often via LE/ACME, but have it signed by a commercial company, perhaps with extra vetting for an EV certificate to show that the web server is working.
Of course, what also would be nice is better availability of HSMs, so I can generate the private key for the webserver on a PCIe card, perhaps back it up or sync it among HSMs using m of n security procedures, which ensures the bad guys can't just copy off the public key.
Re:single source is fine (Score:4, Interesting)
An interesting idea. Cross-signing sort of achieves this.
How would certificate revocation work with multiple signers? If one signer said the certificate was revoked, but the other said it wasn't, which would you trust?
Re: (Score:2)
That is where things get hairy. In GPG, one can set a threshold. For this, if one revocation cert is issued, the key should be marked invalid. If a CA is sending out fake revocation certs without proper authorization, it gets sacked, just like a CA that issues broken or bogus certificates.
Re: (Score:2)
That is where things get hairy. In GPG, one can set a threshold. For this, if one revocation cert is issued, the key should be marked invalid. If a CA is sending out fake revocation certs without proper authorization, it gets sacked, just like a CA that issues broken or bogus certificates.
That defeats the whole purpose of multiple signers, though. In this particular case, for example, the validation process itself was flawed, and DigiCert sent out revocation without the customer's authorization. So DigiCert should be sacked (and I would argue that for the affected customers, that should absolutely be the outcome), but that doesn't avert the crisis.
IMO, the best answer is probably a plurality of responding/reachable certificate authorities, erring towards validity in the case of an exact ti
Re: (Score:3)
If one signer says their certification is still valid, and if you trust that signer (as people always completely do with X.509; there's no such concept as "moderately trusted") then it's a valid cert.
That someone else revoked it, doesn't mean anything is wrong with it. For whatever reason, they're just no longer attesting that it's correct. not-known-to-be-correct != incorrect. As long as someone you trust says it's correct, then you should believe it's correct. (Either that, or rethink how much you trust t
Re: (Score:2)
That is a bit problematic though. Normally if a certificate is revoked this is because it should not be trusted, period - it was cracked, compromised, fraudulent, etc. Like a counterfeit bill, if one teller says it is fake but the teller to the left and right say it's good, then the default should still be to doubt it.
Re: (Score:2)
That is a bit problematic though. Normally if a certificate is revoked this is because it should not be trusted, period - it was cracked, compromised, fraudulent, etc. Like a counterfeit bill, if one teller says it is fake but the teller to the left and right say it's good, then the default should still be to doubt it.
Not necessarily. In this case, if I understand correctly, these certs are being revoked because their standards weren't followed correctly in constructing temporary DNS records for the purpose of verifying the identity of the company requesting the certs. And the details of that validation are specific to DigiCert, so that would not affect the validity of any other certificate authority's validation.
So DigiCert isn't saying that the certs *can't* be trusted, but rather that DigiCert can no longer vouch fo
Re: (Score:2)
But do the browsers know this? Does the browser or certificate authority distinguish "revoked due to legalistic reasons" versus "revoked because something's wrong with it", and take different actions in the code? Or does the user still have to manually click on the "trust this cert anyway?"
Re: (Score:2)
But do the browsers know this? Does the browser or certificate authority distinguish "revoked due to legalistic reasons" versus "revoked because something's wrong with it", and take different actions in the code? Or does the user still have to manually click on the "trust this cert anyway?"
Depends on how they marked it in the revocation list. I'm going to guess "privilegeWithdrawn". That can include such vague things as "CA terms of service violation", and is the closest to a plausible revocation reason code that could reasonably still allow trust from other signers..
Re: (Score:2)
Why does it have just one root key, and no more?
You're likely well aware of this already, but just to clarify - your cert has but one root cert, but there are multiple root certs out there. PGP/GPG rely on numbers of people trusting a cert to built a web of trust with implied authority. SSL/TLS certs rely on established authorities, and all users keep a list of those they trust, which is an inherently different structure.
Why can't I get a certificate for my TLS terminator ... that is signed by multiple vendors? This way, if one vendor has issues, it will still work.
IMHO, there is an easy way to get a very similar result - obtain multiple certs for each site, each from a different vendor. IE: rather
Re: (Score:2)
Re: (Score:2)
There are indeed methods and patends to do this. Ie, authentication involving N parties with mutual distrust. I cited something similar in a patent. I'm not a web developer though, so I really don't know if the state of the art is actually being used there.
A snag though is if a CRL goes out invalidating a cert, then that cert is presumably no good anymore. At what point is the certificate no good in the web? Do 2 out of 3 votes of "it's good" make it work, or does a single "don't trust it!" suffice to in
Re: single source is fine (Score:2)
Re: (Score:2)
This may be not doable, depending on the HSM. Storage on those devices can be limited, and storing multiple private keys may not be doable. Of course, this is definitely an option, but it isn't really a mainstream one.
Re: (Score:2)
IMO, the reason getting multiple certs isn't common is the same sort of reason as to why most do not have a mirror of their entire infrastructure mirrored to an offsite location.
For example, in the old datacenter way, getting space in at least two data centers with duplicate hardware in both, both fully configured and ready to run, and able to cut over to the cold spare with minimal effort. In the cloud, duplicating your whole setup at a region facility. It's too much overhead financially and in man hours f
This seems like a horrible way to let people know (Score:2)
Why would the CABF, whatever that is, be very strict and require revocation in less than 24 hours? There's no middle ground? Suspension maybe? Hell, when I don't pay my hosting bills, the website is parked for a while until I pay and the DNS resolves again, but the site isn't "revoked" or removed.
Re: (Score:2)
Re:This seems like a horrible way to let people kn (Score:4, Interesting)
Please be gentle, I'm asking in ignorance. Why would creating a DNS CNAME record for _wp8y3yrq3 be meaningfully more secure than wp8y3yrq3 ?
Thanks!
Re: (Score:2)
With an underscore, which isn't allowed in hostnames, this guarantee it doesn't match a hostname even though the changes are infinitesimally small of that happening.
Still, it's against the policy, so we have to do this.
Re: This seems like a horrible way to let people k (Score:4, Interesting)
You can revoke with a reason code of certificateHold. The behavior is identical as for other reason codes, except you can unrevoke the certificate later if the revocation was in error. It doesn't appear that it applies in this case.
Re: (Score:3)
It makes sense in the context of an actual fraudulent or potentially fraudulent cert. i.e. it puts the onus on them to not leave questionable certs sitting out there giving clients a false sense of security.
That said, in THIS case it appears to be silly and unnecessary, of the type "well the contract was signed in blue ink and our rules say black" sort of way.
But compliance / security field, as practiced, isn't generally known for balanced evaluations of risk... it's mostly legalistic rule following.
Re: (Score:2)
Why would the CABF, whatever that is, be very strict and require revocation in less than 24 hours?
To minimise fraud you need to quickly address the mechanism using it. In this case it happens to be innocent, but the reality is the rules probably don't apply different times to different revocation reasons.
Re: This seems like a horrible way to let people k (Score:2)
Re: (Score:2)
Unless your client or vendor happens to use a DigiCert certificate that's impacted. Then, they're screwed. For those clients, this is going to be like CrowdStrike all over again.
Re: (Score:2)
This all reads like autistic screeching of the "it says right here in the rules!!!" type.
As if there is a security problem.
As if CABF will revoke DigiCert's authority for taking care of their customers.
As if nobody takes a 2-week vacation in the summer.
As if SMB's don't just have one guy who knows how to deal with certs.
As if they aren't knowingly going to cause (at least a few) outages
As if they aren't going to get their asses sued.
Digicert is killing its reputation for customer service.
Digicert is going t
Re: (Score:2)
Apparently the CABF never realized how its rigidity cheapens the whole concept of this mechanism for fraud prevention. Using revocations as a form of digital Wite-Out sends a different message altogether. I used to think cert revocations were reserved for only serious, security related issues. In the past I would probably trust a signed cert that had simply expired recently, but no way would I trust a revoked cert! Well the joke's on me, now I know better.
I guess the message is to go ahead and accept wh
Re: (Score:2)
Apparently the CABF never realized how its rigidity cheapens the whole concept of this mechanism for fraud prevention. Using revocations as a form of digital Wite-Out sends a different message altogether. I used to think cert revocations were reserved for only serious, security related issues. In the past I would probably trust a signed cert that had simply expired recently, but no way would I trust a revoked cert! Well the joke's on me, now I know better.
I guess the message is to go ahead and accept whatever cert is presented by whatever server. There's no way of knowing the significance of a revocation when they can be revoked for any trivial reason their perseverating hiveminds can conjure up. It could be anything from the validity period having one extra second, to missing an obscure field (cPSuri) that nobody ever looks at, or even knows of its existence.
Since it's almost always going to be a corrective type revocation, who actually cares? I'm hearing it lound and clear. CABF says just click Continue and go about your day.
If this happens more than a tiny number of times on more than a tiny number of websites, yeah, that's exactly what people are going to start doing.
But IMO, the real problem was the "HTTPS everywhere" nonsense. A website that is purely informational and doesn't take logins doesn't really need HTTPS. Nobody is going to bother pretending to be a website to give fake information about something. So except when the content of the website is potentially sensitive (i.e. sites that you don't want other people to
Re: (Score:2)
See, the CA/Browser Forum is run by Google, and maybe the other Browser out there too, and the CAs, which are beholden to Google because Google Knows Best. They have no concept of the real world. This is why they have an argument about what "within 24 hours" means, and why they push everyone towards fragile automation that ultimately reduces security rather than improves it.
That being said, if I had an improperly issued certificate on one of my servers, even one I wanted and requested, I would expect it to
Sensationalism at its best (Score:2)
Ok
"This could literally be thousands of SSL certs."
Wake me when it's 8.5 million.
"This could take a lot of time and potentially cause outages worldwide starting July 30 at 19:30 UTC"
"Be prepared for a long night of cert renewals. DigiCert support line is completely jammed."
Why do you need to call them to renew a cert? Just use the self-service web interface.
I'm expecting a total nothingburger fr
Re: (Score:2)
Why do you need to call them to renew a cert? Just use the self-service web interface.
If you've ever worked in support, you know that there will always be people needing to call. I've spoke to many smart people, that are complete idiots under the wrong circumstances.
Re: (Score:2)
This might cause an issue for a few small sites, but I'll be surprised if anyone besides their own close circles notice.
Re: (Score:3)
Maybe 'those' people shouldn't be managing domain certs in the first place XD
Maybe the UI on the self-serve web portals shouldn't be complete unusable shit. (Not maligning DigiCert here -- I've never seen theirs. Well, I probably have used it actually just not lately and I don't remember at all. Just a general comment on those "stupid users".)
I've been programming since 1973, in all the most advanced environments, and the first thing I tell people about computers is: "They tell you this is user friendly. It is not. It is crap. It's not your fault you think it's hard to use. It is.
Re:Sensationalism at its best (Score:4, Insightful)
Renewing Web certs can be an exercise in frustration with some providers. How hard is it to kick up a .csr, get a .crt back?
Where the real pain point comes in are various apps. Each of them want the certificate file with the private key in a completely different format. Some want the .key file as a separate item. Others want it catted to the .crt file. Still others want the entire cert chain from root to intermediaries to the end cert, with the private key dangling along for giggles. Still others, you just say hell with it, slap an nginx or Apache proxy on front of it, and call the job done, because restarting a proxy after a cert renewal is a lot easier than hoping edits to the their weird key storage thing actually work.
Also, do the relatively short lives of 6-12 months for non-LE certs actually increase security? I'm not seeing this.
How about a compromise on this. If the certificate is generated in a HSM, and the cert has an attestation certificate showing it was generated in a HSM, then allow it to have up to a ten year lifetime. This way, if hardware security is involved, and this could be even separated from low level HSMs to ones that have a ton more security and anti-tamper measures, then the worry about the private key material escaping to the wild is not an issue.
Re:Sensationalism at its best (Score:4, Insightful)
Some companies might have so many certificates that they need a dedicated person to manage them, but plenty of companies will have one certificate which needs renewing once per year, and that's not frequent enough to become second nature. On top of that, the timing is such that some companies are undoubtedly finding that the one person who's done it before is on holiday and some other muggins needs to figure out how to do it under extreme time pressure. I would expect that at least some of the calls are people checking "Do I really need to do this? Are you sure you can't fix things without me doing anything?"
Re: (Score:3)
"affecting about 0.4% of their domain validations" "This could literally be thousands of SSL certs."
Wake me when it's 8.5 million.
These are not client certs. they are servers. So if "thousands" means 8,500 sites with just 1,000 users each, there ya go.
But a lot more people than 8.5 million were affected by CrowdStrike. Several orders of magnitude more, I suspect. A lot of the affected machines were not servers, but things like management or customer service rep PCs. Even if not every PC someplace had CrowsStrike on it, just some, there are major ramifications because it's exactly the ones they thought were sensitive enough to enroll.
K
Re: (Score:2)
Have you ever had the misfortune of renewing a certificate from DigiCert? For some odd reason, they randomly flag cert renewals and require the requestor to provide proof that they own the business that they're representing. Which, if you're an entry level IT person with no easy access to accounting or the C suite, is pretty damn hard to do.
Re: (Score:2)
It just needs to take down a few big web sites that serve a few million users, and this could be another big outage.
I'd imagine that a lot of big companies who use DigiCert certificates are bureaucratic monsters who can't even approve a password change in 24 hours. They'll get hit by this hard.
PKI ~= Broken Trust (Score:2)
My organization is probably going to get hit by this for a few of the certificates. I'm away right now so I can cross-check their entire list of certificates to see which ones are going to be revoked, but I have a feeling it's going to be a few .
Unfortunately, my organization keeps promoting and assigning responsibilities to people who are loyal to management and tow the company line instead of the people that are technically and security conscious So we end up with people who have no background in x.509 or
Re: PKI ~= Broken Trust (Score:2)
I apologize for the terrible grammar on the post above. I was trying to be as eloquent as possible enunciating every word correctly and slowly to Google while I was dictating that but it seems like it has failed completely trying to understand my speech to transcribe it into text.
Re: (Score:2)
Re: (Score:2)
I really love the fact that whole point of short certificate lifetimes is "because bad guys might compromise a system and be there for a long time with a valid certificate" while certbot and the ACME process allows for a compromised machine that never had a certificate to be automatically issued a certificate without any human oversight. But sure, once the compromise is discovered three years later the certicate will automatically die in 90 days, so I guess that's a good thing?
Re: (Score:2)
I really love the fact that whole point of short certificate lifetimes is "because bad guys might compromise a system and be there for a long time with a valid certificate" while certbot and the ACME process allows for a compromised machine that never had a certificate to be automatically issued a certificate without any human oversight. But sure, once the compromise is discovered three years later the certicate will automatically die in 90 days, so I guess that's a good thing?
The whole model is garbage. And heaven help you if you start running into validation issues because one of your secondary domain name servers doesn't respond. The LetsEncrypt name servers apparently cache failures to respond, and don't retry with the other name servers on the list, so if anything is even slightly wrong (for example, because of a secondary being down), you'll spend hours trying to get it to work, waiting for the DNS changes in the domain record to propagate to all of the caching root-adjac
Re: PKI ~= Broken Trust (Score:2)
Hope no one is having a day off (Score:1)
This is just dumb (Score:2)
Am I understanding this correctly?
Sure, fix the poli
Re: (Score:2)
Re: (Score:2)
To the best of my understanding, yes, that is correct. And this is utterly the most idiotic reason for screwing over site admins with a 24-hour panic attack that I've ever heard in my life. If anybody still uses DigiCert after this week, it will be a miracle.