Google API Keys Remain Active After Deletion (darkreading.com) 9
Aikido Security found that deleted Google API keys can continue authenticating for a median of about 16 minutes and as long as 23 minutes, despite Google Cloud's UI claiming that once a key is deleted it can no longer make API requests. Dark Reading reports: Joe Leon, researcher at Belgian startup Aikido Security, recently analyzed the revocation window -- the time between a key's deletion and its last successful authentication -- for the cloud giant's API keys. In a blog post published today, Leon said Google Cloud Platform (GCP) customers expect API access to end immediately after the key is deleted, but this is not the case. In a series of tests, Leon found that the median revocation window was around 16 minutes, while the longest window was up to 23 minutes, "an incredibly long time" for API keys to continue authenticating successfully, he said.
And these windows have serious repercussions for organizations. "An attacker holding your deleted key can keep sending requests until one reaches a server that has not caught up. If Gemini is enabled on the project, they can dump files you have uploaded and exfiltrate cached conversations," Leon said. "The GCP console will not show the key, and it will not tell you the key is still working. You are trusting Google's infrastructure to eventually catch up."
[...] Leon tells Dark Reading the revocation windows for Google's API keys, as well as the unpredictable authentication success rates, complicate matters for incident response teams that are dealing with a potential breach. "This breaks the mental model IR teams have when responding to leaked credentials," he says. "It's assumed that when you click 'Delete' or 'Revoke' that the credential no longer works. Now IR teams need to remember that for GCP credentials, a window exists when that 'Deleted' credential still works for attackers."
To that end, Aikido recommended that security teams and IR personnel use a 30-minute window for Google API key deletions. Additionally, organizations should monitor their API requests by credential through the "Enabled APIs and services" portion of the GCP console, and review API requests by credential. "If you see unexpected usage from that credential after deletion, someone could be actively exploiting it," Leon wrote. Aikido reported the findings to Google, but the company closed the report as "won't fix," according to the blog post.
And these windows have serious repercussions for organizations. "An attacker holding your deleted key can keep sending requests until one reaches a server that has not caught up. If Gemini is enabled on the project, they can dump files you have uploaded and exfiltrate cached conversations," Leon said. "The GCP console will not show the key, and it will not tell you the key is still working. You are trusting Google's infrastructure to eventually catch up."
[...] Leon tells Dark Reading the revocation windows for Google's API keys, as well as the unpredictable authentication success rates, complicate matters for incident response teams that are dealing with a potential breach. "This breaks the mental model IR teams have when responding to leaked credentials," he says. "It's assumed that when you click 'Delete' or 'Revoke' that the credential no longer works. Now IR teams need to remember that for GCP credentials, a window exists when that 'Deleted' credential still works for attackers."
To that end, Aikido recommended that security teams and IR personnel use a 30-minute window for Google API key deletions. Additionally, organizations should monitor their API requests by credential through the "Enabled APIs and services" portion of the GCP console, and review API requests by credential. "If you see unexpected usage from that credential after deletion, someone could be actively exploiting it," Leon wrote. Aikido reported the findings to Google, but the company closed the report as "won't fix," according to the blog post.
Propagation takes time! (Score:2)
/nothingburger
Re: (Score:2)
But its also an exploit/security hole if a key has been compromised, see git-repos as of this week.
So nothingburger all you want, if someone has keys and can use them, they will.
Re: (Score:2)
I was just looking at that very part of the dashboard yesterday. It says this is how it works. Very specifically.
Re: Propagation takes time! (Score:1)
Parent parent (Score:1)
Re: (Score:2)
Propagation delays in an auth table did not cause the keys to leak.
The root cause remains the root cause, whether it is accidentally posting the keys on your github repository or a disgruntled former employee acting maliciously.
I once pushed a BGP update with an error in it. I caught it right away and pushed a correction. But thanks to propagation delays... it caused problems for hours. Lesson learned.
We are as secure as we want you to believe (Score:2)
The fact that companies want to trade insecurity for a known cost (which isnt, as they always go up in cost), over a KNOWN 3-5 year single purchase, is amazing to me.
Accountants can assume a hardware cost based on policies of the company, this isnt the excuse hill they should die on and in
At Scale (Score:2)
at scale, EVERYTHING is an "eventual consistency" model, especially for global cloud providers. nothing is instant. it takes time to issue all the requests behind the scenes to all the targets and to get everything synced up properly.
should there be faster syncing for auth tokens? yeah, probably. but anyone who has done any admin work in any cloud environment probably knows that a ton of requests are no where near instant. you issue a command, and job workers async pull those requests and then act upon them
Cloud environment (Score:2)
This isn't unusual for a cloud environment where services are distributed across multiple servers for performance and resilience. For read/write data the propagation window necessarily has to be short, but for read-only or read-mostly data like authentication tokens the architecture usually favors speed of authentication and resistance to infrastructure failures over fast propagation of changes. Eg., using a pull-based "changes since the last time I checked" process instead of setting up everything for a re