If you didn’t hear about this incident from last summer, that’s probably because the hackers who broke into Microsoft’s Cosmos DB service were not cybercriminals. They were researchers at Wiz, a cloud security startup. The researchers gave the vulnerability a memorable name, “ChaosDB,” and reported it to Microsoft. The “cross-tenant” issue was fixed before any actual attackers could exploit it. Crisis averted.
But the stunning finding made researchers at Wiz and several other vendors curious to find out how prevalent this new class of cross-tenant vulnerability actually is. That led to the discovery of another scary bug in an Azure service a month later. Then another. Then three more — for a total of six critical Azure vulnerabilities in as many months.
Including ChaosDB, five of the critical vulnerabilities demonstrated the possibility of breaching large numbers of different cloud environments, or tenants, in one fell swoop. A cross-tenant flaw like ChaosDB is “the most severe vulnerability that could be found in a cloud service provider,” said Shir Tamari, head of Research at Wiz.
The Wiz research team was not out looking for this type of vulnerability, and only found ChaosDB by accident, Tamari said. The finding was a revelation to researchers that this type of issue is even possible in the public cloud, he said.
Security researchers would go on to discover a pair of critical vulnerabilities in AWS too. But the lion’s share of the most severe vulnerabilities over the past year have been found in Azure, researchers say. To some security researchers and industry analysts, this series of issues raises questions about Microsoft’s approach to securing its Azure services.
“It’s concerning. And it is a pattern,” said Rich Mogull, CEO at independent security research firm Securosis and a longtime security industry analyst. “And so the question is: Do we believe that that’s because they’re under greater scrutiny? Or is it that they have more problems? It might be a little bit of both.”
At cloud security firm Orca Security, whose researchers have found two of the cross-tenant vulnerabilities in Azure services, the issues strongly suggest that Azure is not withstanding the pressure applied by researchers to the same degree as AWS and Google Cloud, according to Orca CTO Yoav Alon.
“I think that in the cloud space right now, compared to other vendors, they might be a bit behind [on security],” Alon said.
First contacted over a week ago to discuss the reporting in this story, Microsoft declined to make a representative available to comment on the record.
“Security is foundational for Azure. Customers trust Microsoft’s multi-layered security provided across physical datacenters, infrastructure, and operations with cyber security experts actively monitoring to protect organizations’ data. We are continually engaged both internally and also externally through bug bounties with researchers to find and remediate security issues, and we are proactive in sharing updates and guidance,” the company said in a statement.
Other researchers and analysts told Protocol they don’t think these findings point to any weakness in Microsoft’s approach to securing its Azure services as compared to AWS or Google Cloud.
“It’s not what we see,” said Ami Luttwak, co-founder and CTO at Wiz, which has discovered three of the six critical Azure vulnerabilities. “I don’t think it’s true.
The reality is that cloud infrastructure is so complex that security issues like these are inevitable, said Kevin Beaver, an independent information security consultant. “I don’t think this is anything egregious,” Beaver said.
A new breed of bug
What many security experts who spoke with Protocol do agree on, however, is that the issues point to a new type of vulnerability that customers are largely not factoring into their understanding of their cloud risks. After all, prior to these discoveries, even security researchers didn’t think tenant isolation in the cloud was in question.
“I think it is safe to say that a lot of people would just assume this is all sorted when they go to use a cloud provider — and might be a bit surprised to find out it’s not,” said Casey Ellis, founder and CTO at bug bounty platform Bugcrowd. “It does violate a fundamental assumption about cloud security that people have. And if that assumption is in play, then you may need to re-address that assumption.”
Frank Dickson, program vice president for Security and Trust at IDC, said he considers the public cloud providers to be “fundamentally extremely secure,” much more so than an on-premises data center environment.
At the same time, “does this illuminate a new class of vulnerabilities? Sure it does,” Dickson said.
The pandemic-accelerated shift to the cloud has also made it so that many more customers now rely heavily on AWS, Azure and Google Cloud than just a year or two ago. And that magnifies the potential impact of a cross-tenant vulnerability being exploited, security experts told Protocol.
“The amount of trust that we give to AWS, Microsoft or Google is immense,” said Patrick Orzechowski, a former penetration tester for a U.S. intelligence agency, who is now a founder and vice president at cybersecurity firm Deepwatch. “You’re counting on them to make sure there are no major vulnerabilities in the infrastructure.”
Spending on cloud infrastructure services surged to nearly $53 billion in the first quarter of the year, up 34% from a year earlier, according to Synergy Research Group. Azure captured 22% of the market during the quarter, trailing Amazon’s 33% market share but remaining well ahead of Google Cloud’s 10% share, Synergy reported.
Hacking the cloud
In mid-2019, an attack against Capital One’s AWS cloud environment exposed the data of 106 million customers. The breach of one of the largest U.S. banks served as a wake-up call in the world of cloud security, showing what’s possible when an attacker targets the public cloud.
But as bad as it was, the breach only impacted one company. Thanks to the architecture of the cloud, every organization’s data is kept isolated and invisible from the others. An attacker who breaches a single customer’s environment cannot gain access to the rest.
The amount of trust that we give to AWS, Microsoft or Google is immense.
Or at least, they’re not supposed to be able to. The series of public cloud vulnerabilities discovered since last August have pointed to the possibility of an attack at an unprecedented scale: If such an attack were to be successful, it could be the Capital One breach times 100 or 1,000, researchers say.
Thankfully, that’s not something we’ve seen happen so far. But it also means that the issue hasn’t gotten a large amount of attention either, even in the security community, a number of cybersecurity experts told Protocol.
“It will get more attention if something devastating happens,” said Alon. “We’re trying to prevent that.”
An analogy could be made to software supply chain attacks, which are frightening on similar grounds: Compromising a single application can lead to the breach of many end customers, as in the SolarWinds attack in 2020. The public cloud has yet to have a SolarWinds moment, but it might’ve been a different story if attackers had discovered even just one of these recent vulnerabilities first, according to researchers.
A ‘critical’ concern
Between August 2021 and January 2022, security researchers discovered a total of eight vulnerabilities with “critical” severity on the major cloud service platforms, according to Scott Piper, a cloud security specialist and former consultant who’s compiled a lengthy list of the issues on GitHub. Azure services have had six “critical” vulnerabilities discovered since last summer, compared to two on Amazon’s cloud platform and none on Google Cloud, Piper’s tally shows.
Since public cloud vulnerabilities are typically excluded from the Common Vulnerabilities and Exposures system, most of these issues were not awarded an official severity rating. Based on his own assessment, Piper ascribed a severity rating to a total of 53 public cloud issues that have been disclosed to date, ranging from “low” to “critical” severity. Wiz has pointed to Piper’s work as the authoritative documentation of these public cloud issues.
Outside of the researchers and vendors themselves, Piper is probably as familiar as anyone with the recent string of public cloud vulnerabilities. And to him, what stands out is that researchers have repeatedly proven the ability to achieve cross-tenant access on Azure services.
By contrast, with the two recently discovered critical vulnerabilities affecting AWS, “it seemed like they might have been able to get [cross-tenant access] if they dug deeper,” Piper said. “But with Azure, they actually proved it.”
After ChaosDB, the next critical Azure vulnerability to be found was a cross-account takeover vulnerability on the Azure Container-as-a-Service platform. It was discovered by researchers at Palo Alto Networks’ Unit 42 group, and dubbed “Azurescape.”
“Cross-account vulnerabilities are often described as a ‘nightmare’ scenario for the public cloud,” Yuval Avrahami, principal security researcher at Palo Alto Networks, wrote in a September blog post. “Azurescape is evidence that they’re more real than we’d like to think.”
Since then, the evidence for this has continued to pile up. The Azurescape bug was followed by three additional cross-tenant vulnerabilities in Azure services in the following months. Orca Security researchers discovered “AutoWarp,” which affected the Azure Automation service, and “SynLapse,” which impacted the Azure Synapse analytics service. Wiz researchers discovered “ExtraReplica,” which affected Azure Database for PostgreSQL Flexible Server.
Another critical Azure vulnerability, discovered by Wiz and dubbed “OMIGOD,” did not enable cross-tenant access. But it was far more of an immediate problem for customers than the other bugs: The vulnerability affected an array of Azure services for customers and saw wide exploitation by attackers last fall. By contrast, the other recent Azure vulnerabilities are not known to have been exploited.
Orca Security’s researchers have found two of the cross-tenant vulnerabilities in Azure services.GIF: Orca Security
‘You had one job’
When it comes to preventing cross-tenant vulnerabilities in the cloud, there’s not much that a cloud customer’s security team can do. Such issues “are clearly beyond the scope of any customer to address,” said Lee Sustar, a principal analyst at Forrester. “They’re relying on their cloud provider.”
All the major public cloud platforms use some form of a “shared responsibility” model, which divides up the duties for security between the vendor and customers. Under the scheme, vendors commit to securing the underlying infrastructure. Customers are responsible for securing their own data and applications.
Cross-tenant vulnerabilities, Sustar noted, fall unambiguously on the vendor’s side of the shared responsibility model. By contrast, for past cloud incidents such as the Capital One breach, the accountability was mainly on the customer’s side.
Ensuring tenant isolation ranks at the top of the list for security responsibilities that fall to the cloud service providers, according to Mogull, who is also senior vice president of Cloud Security at cybersecurity firm FireMon. “It’s like, ‘You had one job,’” he said. “There is no greater risk or concern for any cloud provider.”
Still, while the cloud platforms are not perfect on security, in many cases they’re still going to be more secure than using a data center, he said. “I don’t want to lose that at all. I’m a huge proponent of cloud,” Mogull said. “I think everybody should move everything they can up there.”
For Wiz, a goal of the vulnerability research is simply to help customers understand that isolation issues do exist in the public cloud, said Luttwak, who was formerly the CTO of Microsoft’s Israel R&D unit before co-founding the cloud security startup in 2020.
“It’s something that was not in our perception enough” in the past, he said. “As users of a cloud provider, we do need to ask them questions like, ‘What’s the isolation model? How can you ensure it?'”
Given that the isolation of customer data is “absolutely a core tenet” of secure cloud computing, the industry is just now discovering a need for documentation about the isolation architecture being used by the cloud providers, said Jasmine Henry, field security director at cybersecurity firm JupiterOne. Cloud providers must find a way to prove isolation without giving away too much and creating further security risks, Henry said.
As an industry, “I think these are things that we’re very much learning,” she said.
A matter of perspective
The series of discoveries of cross-tenant vulnerabilities in Azure can even be seen as a good thing, according to Luttwak. Microsoft encourages research of this kind, he said, and offers up to a $60,000 bounty for the discovery of Azure bugs.
“The fact that there are vulnerabilities [doesn’t mean] the entire platform is not secure,” Luttwak said. “I don’t think that you can claim that Azure is more or less secure than the others. It’s a very good question [to ask], but I don’t think anyone has the data to say that.”
The vulnerabilities that have been surfacing in Azure would appear to be more of an issue of Microsoft “being down on its luck and being the target” of security research, said Beaver, who founded his consultancy Principle Logic LLC in 2001 and is on the faculty at the prominent research firm IANS.
“These things can happen to any business at any time,” he said. “They still have human beings running the show. And negative events are going to occur.”
In its statement to Protocol, Microsoft said that its security teams “work around the clock to identify and mitigate potential security issues. We also work with the security research community through coordinated vulnerability disclosures to ensure potential security issues are identified and mitigated before they are disclosed publicly.”
Still, the four-person research team at Wiz and the two-person public cloud vulnerabilities research team at Orca Security both say they’ve been devoting a similar amount of time to looking for issues in AWS, Azure and Google Cloud — and serious cross-tenant vulnerabilities are predominantly showing up in Azure.
Serious cross-tenant vulnerabilities are predominantly showing up in Azure.GIF: Wiz
No one would suggest that security is being left out of Microsoft’s design considerations for Azure services, of course. But based on the vulnerability research so far, it would appear that certain usage scenarios allowing a user to bypass tenant isolation were missed, according to multiple cloud security experts.
Issues with authentication and authorization are a running theme through many of the critical Azure vulnerabilities. “There’s too much trust built into the system,” said Scott Walsh, owner and principal researcher at threat research firm Invisible Threat.
With ChaosDB, it appears that Microsoft was checking the first step of authorization but then “assuming trust” afterward, Walsh said, and that enabled attackers to gain access to other tenants after the initial stage of authorization.
“It basically looked like, if you’re trusted enough to get into this [cloud] instance, you’re trusted enough for everything in this instance. Which in multi-tenant, shared environments, is not good enough,” said Walsh, who is also a senior security engineer at cyber insurance firm Coalition.
By exploiting a series of misconfigurations in the Cosmos DB service, Wiz researchers reported they were able to attain unrestricted access to network access and acquire a number of “secrets” — private keys and certificates — which could be utilized to manage the database service. “We should’ve never had those,” Tamari said. “It’s crazy.”
While researchers say that Microsoft has usually responded quickly to address critical Azure vulnerabilities, that was not the case with SynLapse, according to Orca Security. The firm’s researchers first discovered the flaw in the Azure Synapse service in January, but the researchers were still able to bypass tenant separation after Microsoft deployed patches in March, and then again in April, co-founder and CEO Avi Shua wrote in a blog post on May 9.
“We think that the architecture contains underlying weaknesses that should be addressed with a more robust tenant separation mechanism,” Shua said in the post.
Customers unfazed
So far, the discovery of the cross-tenant Azure vulnerabilities have not prompted major customer concerns, said Navisite CEO Mark Clayman, whose firm provides cloud strategy and migration services and holds the top-tier Azure Expert MSP certification. “Customers are not pausing” in terms of adopting Azure, Clayman said, at least in part because the discoveries are relatively recent and there hasn’t been a cross-tenant breach so far.
Another factor could be that customers who are “a little bit more tech-savvy” are more likely to focus on AWS or Google Cloud anyway, he said. “Companies that are a bit more traditional, a bit more conservative — I just tend to find them leaning more toward Azure,” said Clayman, whose firm partners with all of the major cloud platforms.
Dickson of IDC said the customers he’s been speaking with lately have not expressed any concerns about the critical Azure vulnerabilities either. Customer pressure on Microsoft over the issue would thus appear to be minimal.
Still, while customers may have seen cross-tenant issues to be a “low-level risk” in the past, “now they may reassess that” as they digest the recent findings, said Forrester’s Sustar.
At Microsoft today, there seems to be a disparity between the amount of money it’s generating and the types of security promises it’s making, Orca Security’s Alon said.
“If you have a platform that generates billions of dollars in revenue, and promises security, and under-delivers — you can move some of those dollars into doing more security,” he said. “This is not a mom-and-pop shop. They are [one of the] biggest corporations in the world. And they can do amazing things if they choose to do it, and put it as the highest priority.”
Addressing the underlying issues would likely entail, first, getting a better handle on where the boundaries are supposed to be in these Azure services, according to Walsh.
Then, Microsoft would probably want to “slowly, iteratively change the architecture” of the services over time to bring a more cloud-native style of identity and access management, Walsh said.
“It may be bumpy for a while yet,” he said.
Trustworthy Computing 2.0?
Alon expressed confidence that Microsoft will be able to improve the situation around vulnerabilities on Azure services. Microsoft has a “great track record in turning security around,” he said, going back to the famous “Trustworthy Computing” memo in 2002. In the internal memo to Microsoft’s staff, Bill Gates committed to placing a higher priority on security for Microsoft products, leading to “very big strides for many years to improve security,” Alon said.
Microsoft may now be approaching a similar moment with the security of the cloud, he said, adding that he suspects that the company has already made changes internally to head off future Azure security issues. One key indicator, Alon said, was the arrival last fall of AWS veteran Charlie Bell to head Microsoft’s security effort as an executive vice president reporting to CEO Satya Nadella.
“But change takes time. And we will see more of it in the next months and years,” Alon said.
It’s also encouraging that Microsoft has been proving how committed it is to cybersecurity with a series of very positive moves in recent years, Mogull said.
For example, Microsoft deserves much credit for taking down “global nation-state level botnets” in recent years, he said. In April, Microsoft disclosed its role in taking down the ZLoader botnet, a network of infected computers that cybercriminals were using to carry out attacks including ransomware.
On the whole, “they have excellent security,” Mogull said. But when it comes to the security of Azure, he said, “I just want it to be better. And I think it can be.”
