In mid-2017, Russian state-sponsored attackers installed a malicious worm in a Ukrainian financial software package. When businesses updated their software, it became infected. The worm, NotPetya, spread quickly, doing billions of dollars of damage around the world. The White House called it “the most destructive and costly cyberattack in history.”
Three years later, Russia-linked attackers hijacked the software upgrade process of another piece of enterprise software, SolarWinds’ Orion network monitoring toolset. Again, the impact was widespread.
“Having access to the software development pipelines gives them a chance to reach networking infrastructure and get access to intellectual property,” says Viktor Gazdag, senior security consultant at NCC Group, a global cybersecurity advisory firm.
Attacks on DevOps pipelines are increasing
It can be tempting to say that attacks like this are isolated and depend on highly motivated and skilled attackers. In fact, the DevOps pipeline has become a popular target not just for state actors but criminal gangs.
According to a study released last month by Argon, an Aqua Security Company, attacks on the software supply chain grew by more than 300% compared to 2020. Common tactics include planting malicious code in popular open-source packages or exploiting vulnerabilities that are already there, compromising CI/CD pipeline tools, and taking advantage of hard-coded credentials and other misconfigurations and security issues. The open-source component channel was a particularly popular target.
Attacks on the open-source software supply chain increased 650% last year compared to 2020, according to a study by Sonatype released in September. The attack surface is vast. More than 37 million components and packages are in the top four open-source ecosystems, according to Sonatype. Open-source software downloads hit 2.2 trillion last year, up 73% compared to 2020.
Why DevOps pipelines are vulnerable
Software developers often have high permission levels and access privileges, Gazdag says. If the software being produced is designed for external consumption, the impact can be dramatically greater. “The attackers also have the opportunity to get a foothold in the final application,” he says.
So, the DevOps pipelines should have higher levels of security in place. Instead, they have a lot of weak security practices and exposed infrastructure and credentials. “If you use Shodan and search for [development tool] ‘Jenkins’ you can see a lot of Jenkins infrastructure available and accessible on the internet,” GazDag says.
Too often, the CI/CD infrastructure doesn’t get the same level of attention as other areas of the enterprise, Gazdag says. With modern development practices, the situation is getting worse.
“As organizations move to DevOps, there is a tendency to relax some of the controls we have in place around development,” says Gartner analyst Dale Gardner. “We want to be flexible and the whole DevOps method is we’re trying to get code out quickly. Limits and controls get in the way of that.”
Types of attacks on DevOps pipelines
According to David Wheeler, director of open-source supply chain security at Linux Foundation, the three most common kinds of attacks are dependency confusion, typosquatting, and malicious code injection.
Dependency confusion, also known as namespace confusion, is when an attacker figures out the names of proprietary enterprise software packages and creates open-source packages with the same names and later release dates. Certain pipeline tools automatically try to download the latest version of a software package and wind up getting the one with the malicious payload.
Typosquatting is when an attacker creates an open-source software package with a name almost identical to a real one, hoping that a programmer will make a typo and use the wrong library.
Code injection is where attackers add malicious code to a legitimate open-source project. They can do it by either stealing a project maintainer’s credentials and uploading the code under their name, volunteering to work on the project themselves, or tampering with open-source developer tools.
Vulnerabilities in open-source components
Then there’s the issue of known vulnerabilities in open-source components — vulnerabilities that attackers are quick to exploit. In April, application security testing company Synopsys reviewed the code of more than 1,500 enterprise software projects, both internal and commercial, and found that 98% of them contained some open-source code. For an average application, 75% of the codebase was open source.
Here’s the scary part: In Synopsys’ analysis, 84% of the codebases had at least one vulnerability. That’s before the Log4J vulnerability came to light, which security researchers called the most dangerous Java exploit in years. In addition, 91% percent of the open-source components used hadn’t seen any maintenance in the past two years.
More than 28,000 new vulnerabilities were disclosed in 2021, a record high, according to a report released this month by Flashpoint’s Risk Based Security. Of those, more than 4,000 were remotely exploitable, with both a public exploit and documented solution information.
The Log4j vulnerability was particularly dangerous, surpassing all others in impact, the report said. The library was found in more than 6,200 other software products, and the number of vendor advisories continues to climb.
How to protect software development pipelines
What should companies be doing to protect their software development pipelines? It starts with education and training for the developers, instituting best practices like two-factor authentication and code reviews, and installing monitoring tools to flag suspicious activities.
It starts with the developers
At managed services provider Ensono, David Gochenaur, senior director of cybersecurity, says that both in-house developers and third-party software shops need oversight when it comes to the security of the code development and deployment process. The two groups of developers need to be approached in different ways.
Ensono doesn’t sell software, but it needs custom software to run its customer portals. The security of these portals is of paramount importance. “We manage systems for a lot of clients and collect data about the status of those systems and put it into a portal,” Gochenaur says.
That means that Ensono’s tools have access to those customer systems, which makes Ensono a high-value target for attackers. “Because there are so many clients, you want to make sure that client A can’t get into client B’s data,” Gochenaur says.
The stakes are high, Gochenaur says. “Some of our clients are very sensitive from a national security perspective and privacy perspectives,” he says. So the first challenge is to be very stringent when it comes to vetting vendors. “You have to get to know them very well,” he says. “The SolarWinds incident is a good example and there are many other examples out there of third parties that didn’t secure themselves very well and were used as entry points for threat actors.”
That includes outside software development firms. “When we use third parties, we vet them pretty hard to make sure they have processes and controls in place to make sure that whatever we’re getting from them is secure,” Gochenaur says. That includes reviewing their testing procedures and the security controls they have in place in their development environment. “And we build defect penalties into the contracts,” he adds.
Then, for the company’s own developers, the biggest issue is not to use publicly accessible code repositories “because anything could be out there,” Gochenaur says. “There might be code that just looks awesome. Probably is awesome for many reasons. It does awesome things for you, but it allows threat actors access to whatever you have going on.”
Developers could be taking many other measures to help them produce more secure code, Gochenaur says. One strategy that has helped provide both security training and motivation is to run penetration tests by third parties and by in-house teams. “It will make a huge difference in the quality of the product that’s developed,” he says.
In fact, when Gochenaur’s running pen-tests on the company software, developers have always asked to sit in on the tests and watch the white-hat hackers do their work. “They wanted to understand what they were doing and learn from the vulnerabilities that the pen-testers found,” he says. “It gave the developers a different way to think. Now, when I bring in a third party, this is one of my requirements — that the technical teams can look over their shoulders and see what’s going on and learn from them.”
Use proper tools and controls
To help the company’s developers make good decisions, and to help keep them safe, Ensono has several security controls in place. For example, multi-factor authentication helps keep outsiders from accessing the DevOps pipeline. The company uses private code libraries so that developers can pick from code that’s already been reviewed and approved.
Ensono also has teams dedicated to patching systems, to ensure that everything that is deployed is current and up to date. “We scan our entire environment regularly looking for vulnerabilities,” Gochenaur says.
Companies can do other things to help lock down their development pipeline that are often missed, says Venky Chennapragada, DevOps architect at Capgemini Americas. For example, companies should have separate pipelines for the non-production staging environment and for production — and to limit the people who have access to both systems. To lock down the access even further, companies should be using enterprise-grade access management systems, like Active Directory or LDAP.
Many companies have a separate user database for the software development teams or use built-in user management tools. It is easier to have a separate system.
“If I’m integrating with Active Directory or LDAP there’s going to be a security audit,” Chennapragada says. “Some engineers might want to bypass the security audit because they didn’t install things properly.”
Role-based access is another control that developers might chafe at. “It’s always easy to give full access, and not have to create user groups and roles,” Chennapragada says, “but it’s bad practice.”
Finally, Chennapragada recommends that organizations carefully track all the components that go into their software, especially the open-source libraries. “Developers have a tendency to include open-source code in their software and it can have bugs and security vulnerabilities,” he says.
External libraries need to undergo security scanning and code reviews, and developers should be limited to only using certified dependencies. It’s not just libraries that developers might want to grab off the internet. Other attractive tools include operating system variants and plugins.
Linux, for example, comes in millions of different flavors. “Make sure any version they use is hardened, is the latest and up to date,” Chennapragada says. The popular development tool Jenkins, an open-source automation server, comes with a variety of plugins. “There are plugins for everything,” he says, “but the plugins can be really vulnerable. People can put malicious code in the plugin that can take over your system.”
Many security controls and processes are available that don’t cost a lot and don’t create too much overhead, but do require some thoughtful planning or training, says Ilia Kolochenko, CEO at cybersecurity vendor ImmuniWeb. For example, AWS offers built-in security controls and tools that are not expensive or even free, he says. “People don’t go for them because they’re unaware, don’t think they need them, or it’s too difficult to dig into them and leverage them.”
The cloud makes it easier to deploy tools like continuous security monitoring and incident response, he says. “You can detect suspicious activity, immediately stop it, replace it with a clean image, and continue your operations without going offline. The cloud provides many great opportunities to automate your continuous security monitoring and incident response, but people don’t use it.”
Request SBOMs but also scan for vulnerabilities
Many in the industry have been pushing for a software bill of materials (SBOM). Last May, President Biden issued an executive order requiring SBOMs from vendors that provide software to the federal government. Two days later, the Cloud Native Computing Foundation released a best-practices white paper recommending that all vendors provide an SBOM where possible, with clear and direct links to dependencies.
An SBOM would help companies find instances of vulnerable components in their environment. For example, Log4j was patched in December, but, as of February 11, 40% of all downloads were still of the vulnerable version.
“If you buy a loaf of bread, it’s got the list of ingredients written on the side,” says Kayne McGladrey, IEEE senior member and cybersecurity strategist at Ascent Solutions, a technology consulting firm. “Having that for software allows organizations to make informed risk decisions.”
